RLVRYou run RL on a 7B math model with a rule-based reward: 1 if the final answer matches, 0 otherwise.
PPO also trains a value model, usually policy-sized, to predict reward from each token onward; that second 7B model is a large memory and compute cost, and with reward only at the final token its per-token predictions are hard to make accurate.