What do you conclude, and what would you do?

1. AThe init is off: the logits start far too large, so scale them down at init.
2. BNothing is wrong: the loss on random weights is arbitrary and warmup absorbs it.
3. CThe vocabulary is too large, so cut it to 8K to bring the starting loss down.
4. DThe learning rate is too high, so halve it and the first loss will come down.