GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm introduced by the DeepSeek team in DeepSeekMath, and DeepSeek-R1 used it to develop long-form reasoning without a reward model. The core idea is simple: sample a group of answers to the same question at once, and let the answers rank each other within the group, instead of training a separate reward model to score them.
How GRPO works: sample, group, compare
During training, the model generates a group of answers to the same question, say 8 to 16 of them. Each answer earns a reward by a rule: a math question is scored on whether the final answer is right, a coding question on whether the tests pass. The average reward of the group then serves as the baseline: answers above the average are pushed up, answers below it are pushed down, and the policy moves toward the better answers. To keep the model from drifting too far in a single update, the update is constrained to stay near the old policy, with a penalty for straying too far. The whole pipeline drops a reward model, and also drops the critic model that PPO uses to estimate value, so training maintains fewer models and the memory bill shrinks.
Where it differs from PPO
PPO also learns from rewards, but it needs a value model to estimate how good the current state is before computing an advantage; GRPO replaces that estimate with the group average. The upside is a simpler structure, less memory, and fewer hyperparameters. The downside is that when every answer in a group is right, or every answer is wrong, the rewards do not separate and the learning signal vanishes. Training R1 therefore depended heavily on the difficulty mix of the questions: questions so easy that the whole group gets them right, or so hard that the whole group fails, teach nothing.
Two misconceptions, and where it applies
The first misconception is that GRPO uses no reward. What GRPO drops is the reward model; the reward itself is still there, just computed by verifying correctness against a rule. The second is treating it as a universal training method. Open-ended writing and creative tasks have no reliable automatic judge, so the reward cannot be pinned down and GRPO has nothing to work with. To judge whether a scenario fits GRPO, ask one question: can the answer be verified automatically, without relying on the model itself? Math, code, and formatting tasks with clear right answers qualify; open-ended Q&A should not be forced into it.
