Bank
19 / 87
ScalingMultiple choiceMedium

Chinchilla's compute-optimal rule of about 20 tokens per parameter puts an 8B model at 160B tokens, but Meta trained the 8B Llama 3 on over 15 trillion, about 94 times that.

With that compute the rule would pick a model of about 77B parameters trained on 1.5 trillion tokens, which Chinchilla's own loss formula rates slightly better.

So why train the small one for so long?