A few billion tokens into training a small transformer, the loss stops its smooth fall and takes a small bump.

Before the bump, given six random tokens followed by the same six again, it mispredicts the repeat; after it, the model predicts the repeat token by token from the prompt alone, with no weight change.