QuantizationYou want to serve a 66-billion-parameter model with every matmul in INT8, weights and activations both, and the weights quantize with almost no loss.
A few activation channels are about a hundred times larger than the rest in nearly every token, so with one INT8 scale per tensor the normal channels get squeezed into a handful of levels and accuracy collapses.