A one-layer transformer trained with AdamW and weight decay learns $(a+b)\mathrm{mod}97$ from 30% of all 9,409 pairs and is tested on the rest.

By step 300 training accuracy is 100% and test accuracy under 1%; test accuracy stays under 10% for the next 24,000 steps, then climbs to 100% between steps 24,000 and 31,000.