Distributed trainingMultiple choiceMediumCopy prompt

How would you cut memory per GPU without changing the model or the batch?

1. ARecompute activations during the backward pass instead of storing them.
2. BPartition the optimizer states, gradients and weights across the 64 GPUs.
3. CSwitch from Adam to SGD with momentum to drop one of the two moments.
4. DStore Adam's two moment estimates in 8-bit precision on every GPU.