You're writing a fused cross-entropy kernel for a language model whose logits come out in fp16 over a 128K-token vocabulary.
It has to return the loss and the gradient with respect to the logits.
You're writing a fused cross-entropy kernel for a language model whose logits come out in fp16 over a 128K-token vocabulary.
It has to return the loss and the gradient with respect to the logits.