What would you change in your optimizer?

1. AClip the gradient norm at 1.0 before it goes into the moment estimates.
2. BDivide $m_t$ by $1-{\beta }_1^t$ and $v_t$ by $1-{\beta }_2^t$ before the update.
3. CSwitch to SGD with momentum 0.9 and keep the same value for the learning rate.
4. DCut the learning rate tenfold for the whole run to absorb the spikes.