In your RLHF run, a reward model trained on pairwise human comparisons scores responses, and PPO updates the policy to raise that score.

Over 2,000 steps the reward keeps climbing, yet raters prefer the step-500 checkpoint; later answers run longer and flatter the user, and the reward model scores both highly even when the answer is wrong.