GRPO and RL for reasoning

Why reasoning rewards are different

For open-ended chat, "better" is often subjective. For reasoning tasks, you can sometimes check the outcome. Did the final answer match? Did the code pass tests? Did the SQL query return the expected rows? Did the plan obey the constraints?

That makes RL more attractive. Instead of asking a human which answer sounds better, you can sample several attempts and reward the ones that actually solve the problem. This is still not easy, but the reward is less squishy.

Verifiable rewards in practice

Outcome rewards work best when the checker is cheap, deterministic, and hard to game. Examples:

• Math: Parse the final answer and compare to ground truth. A GSM8K-style checker gives +1 for a matching number, 0 for wrong format, −1 for incorrect.
• Code: Run the generated function against a hidden test suite in a sandbox. Reward is pass rate; partial credit only if tests are independent.
• Structured extraction: Validate JSON against a schema. Reward 1 if all required fields match, 0 otherwise.
• SQL: Execute the query and compare result rows to expected output on a fixed database snapshot.

Each example shares a pattern: sample N completions, score each with the verifier, then update weights based on relative performance within the group. That is the GRPO loop in production.

Group-relative policy optimization

GRPO samples a group of outputs for the same prompt. Each output receives a reward. The algorithm then compares each output against the group's average or normalized reward. Outputs that beat the group get pushed up. Outputs that lag get pushed down.

The practical appeal is that GRPO does not need a separate value model or critic. That can save memory and simplify training, especially when tuning large language models where every extra model copy is expensive. Hugging Face TRL ships a GRPOTrainer that handles group sampling and relative advantage normalization.

The DeepSeek-R1 era

GRPO moved from a niche PPO variant to mainstream practice when DeepSeek-R1 (2025) used group-relative RL with verifiable math and code rewards to push open reasoning models without a separate critic. The recipe: start from a strong base model, generate groups of chain-of-thought completions, score outcomes with rule-based checkers, and iterate RL on the relative winners.

The lesson for practitioners is not "run GRPO on everything." It is that when you have cheap verifiers and exploration helps, group-relative RL can unlock behaviors imitation alone misses. Chat personality tuning is still a DPO problem. Multi-step math and code are GRPO problems.

Verifiable rewards are not free

A verifier can be wrong or incomplete. Unit tests can miss edge cases. A math checker can validate the final number while ignoring a brittle chain of reasoning. A sandbox can be too slow or too permissive. Outcome rewards are stronger, but they still need design.

For many production teams, the best use of reasoning RL is narrow: domains where success can be checked cheaply and repeatably. Coding tasks, math tasks, constrained planning, data extraction with exact schemas, and tool workflows with clear end states are better fits than broad chat personality.

Where GRPO fits in the stack

Use supervised fine-tuning to teach the model the format and basic task. Use DPO when you have offline human preferences. Reach for GRPO-style RL when you can score multiple attempts and the score captures real success.

That does not mean every app needs GRPO. If a smaller prompt fix, RAG change, eval suite, or DPO run solves the problem, take the simpler route. RL becomes worth it when exploration reveals better solutions than imitation alone.

Quick check