You fine-tune a chat model with DPO on preferred/rejected answer pairs against a frozen reference. After one epoch raters prefer it to the reference; by the third it repeats phrases and drifts off topic while the DPO loss keeps falling.

The rejected answers' log-probabilities have dropped sharply, and the preferred answers' have dropped too, more slowly.