You run GRPO on math problems with a 0/1 verifiable reward. For each problem you sample a group of 8 responses, and each response's advantage is its reward minus the group's mean reward, shared by all of its tokens.

Each response's loss is the mean of its per-token losses over response tokens, then averaged over the batch. Accuracy rises slowly, but incorrect responses grow in length much faster than correct ones.