How would you compute the loss so neither happens?

1. ACompute $\log p_y$ as $z_y-m-\log {\sum }_i{e}^{z_i-m}$, where $m={\max }_i{z}_i$.
2. BClamp each probability to at least $10^{-7}$ before taking its logarithm.
3. CCompute the softmax in bf16 instead, to get more headroom in the exponent.
4. DDivide every logit by 100 before the softmax so that all the values stay small.