AdderBoard asks for the smallest transformer that adds two ten-digit numbers with 99% accuracy: set by hand, ten weights suffice, while the smallest trained model so far needs 36.

Picture the loss as a landscape where height is error, and training as a marble dropped at a random spot that rolls downhill.