Mistral AITraining stabilityYour Transformer applies layer norm after each residual addition (Post-LN). Without learning-rate warmup it ends far worse, and the warmup length is one more knob to tune.
At initialization, gradients of parameters near the output are much larger than in earlier layers.