OptimizationMultiple choiceMediumCopy prompt

How would you set the learning rate so the big-batch run matches the small one?

1. AScale the learning rate by $\sqrt{32}$ and keep it fixed.
2. BKeep the learning rate and accumulate gradients over 32 small batches.
3. CScale the learning rate by 32 and ramp up to it over the first epochs.
4. DScale the learning rate by 32 and use it from the very first step.