You train a small decoder-only transformer on addition with operands of up to 20 digits. Each problem is a single sequence of digit tokens, such as 481+903= followed by the answer's digits, and each token's position embedding encodes only its index in that sequence.

Test accuracy is close to 100% for operands up to 20 digits and close to 0% at 100 digits.