Bank
49 / 87
Training stabilityShort answerHard

You're writing a fused cross-entropy kernel for a language model whose logits come out in fp16 over a 128K-token vocabulary.

It has to return the loss and the gradient with respect to the logits.

How would you compute both without overflow or underflow, and how would you test it?