AttentionYour small Transformer LM reaches training loss near zero within an hour on next-token prediction, and validation loss on held-out text is just as low. Generating one token at a time, the output is gibberish.
The data pipeline and the tokenizer both check out.