You replace every LayerNorm in a Transformer language model with BatchNorm. It trains with 8 sequences per GPU, padded to the longest one, and at inference it serves one request at a time.

Validation loss computed one sequence at a time is higher than in-batch validation loss, and training gets unstable on batches with many short, padded sequences.