Mistral AITraining stabilityFreeYou move a training run from fp32 to fp16 to halve activation memory and use tensor cores; the forward pass looks fine, but the loss eventually diverges.
Many backward-pass activation gradients are below about , the smallest positive fp16 value, so they round to zero.