What would you do to recover most of the lost quality?

1. AKeep the 1% of weights with the largest magnitude in FP16, quantize the rest.
2. BFind the 1% of channels with the largest activations and scale them up first.
3. CUse one INT3 scale per tensor instead of per group of 128 weights.
4. DQuantize the activations to INT8 as well so the whole layer runs in integers.