RLHFAfter RLHF with PPO, labelers strongly prefer your model's answers to the supervised baseline, but its SQuAD and DROP scores have fallen below the base model's.
You want to keep the preference gains and recover the benchmark scores.
RLHFRLHFRLHFAfter RLHF with PPO, labelers strongly prefer your model's answers to the supervised baseline, but its SQuAD and DROP scores have fallen below the base model's.
You want to keep the preference gains and recover the benchmark scores.