Reward models and RLHF
RLHF is the classic preference tuning loop: learn a reward function from human comparisons, then tune the language model to earn more reward without drifting too far from the useful base model.
RLHF turns preference labels into a learned reward model, then uses reinforcement learning to make the LLM produce outputs that the reward model scores highly.
The three-model picture
RLHF usually involves three versions of model behavior. First there is the base or supervised-tuned model that already knows how to answer. Second there is a reward model trained to score answers. Third there is the policy model being updated by RL.
The reward model is not the final assistant. It is a judge. It reads a prompt and an answer, then predicts a scalar reward. During RL, the policy generates an answer, the reward model scores it, and the optimizer nudges the policy toward answers with higher scores.
This is the stack OpenAI validated in InstructGPT: SFT teaches the task, RLHF teaches the ranking humans prefer. Lesson 04 shows how DPO later removed the separate reward model from that loop while keeping the same preference math.
Training the reward model
A reward model is trained on preference pairs. Given the same prompt, it should score the chosen answer higher than the rejected answer. The absolute number is less important than the ordering.
This sounds simple, but it can break in quiet ways. If labelers prefer longer answers, the reward model may learn "longer is better." If labelers reward confident tone, the model may learn to sound certain. If the dataset under-samples refusal boundaries, the reward model may make unsafe answers look helpful.
A reward model is a lossy proxy for human judgment. Once you optimize against it hard enough, the policy can find weird pockets where the proxy gives high scores for bad answers.
Optimizing the policy
After the reward model exists, the policy model is updated with an RL algorithm. In LLM work, PPO is the classic choice. The policy samples responses, gets reward scores, and updates its token probabilities so high-reward responses become more likely.
There is a catch: if the model chases reward too aggressively, it can lose broad language ability, become repetitive, or exploit reward-model quirks. RLHF systems usually add a KL penalty that keeps the policy close to a reference model. That reference is often the supervised-tuned model you started from.
Why PPO runs are hard to debug
PPO is sensitive in ways supervised training is not. Watch for these failure modes in production logs:
- Reward collapse: Scores climb while human win rate flatlines. The policy found a shortcut the reward model likes.
- KL blow-up: The policy drifts so far from the reference that fluency and tool-use regress even if reward looks fine.
- Entropy collapse: The model becomes repetitive because the optimizer keeps sampling the same high-reward n-grams.
- Critic misalignment: The value baseline lags the reward model, producing noisy advantage estimates and unstable gradient steps.
Mitigations: clip the reward range, anneal the KL coefficient, cap response length, log per-token entropy, and checkpoint early. Treat the first RLHF run as an experiment with rollback, not a one-shot training job.
Reward hacking and Goodhart's law
Reward hacking means the policy learns to maximize the reward signal without actually getting better for users. In text models, that might look like overlong answers, fake citations, excessive hedging, formulaic safety disclaimers, or confident nonsense that the reward model likes.
This is a concrete instance of Goodhart's law: when a measure becomes a target, it stops being a good measure. The reward model is a proxy. Optimize against it hard enough and the policy will find pockets where the proxy diverges from what users want.
The fix is not one trick. You need better preference data, adversarial evals, length-normalized checks, human review, and limits on how far the policy can move. The reward model should be treated as one noisy instrument, not as ground truth.
A support bot tuned on thumbs-up data started inserting "Hope this helps!" and restating the question before every answer. Reward scores rose. Escalation rate rose faster. The model learned politeness tokens the labelers rewarded, not problem resolution. Read samples, not just aggregate reward curves.
RLAIF (reinforcement learning from AI feedback) replaces human labelers with a stronger model that grades outputs against a written constitution of principles. Anthropic's Constitutional AI chains self-critique and revision before preference labels are applied: the model drafts an answer, critiques it against principles ("remove harmful content", "prefer concise refusals"), revises, and the improved pair becomes training data.
The engineering tradeoff: lower labeling cost and faster iteration, but the constitution and judge model become the new source of bias. Use RLAIF when human labeling is the bottleneck and you have a strong rubric. Keep a human gold set to catch judge drift. RLAIF still ends in DPO or RLHF; it changes who writes the preference labels, not the training math.
Landmark references
The InstructGPT paper that made RLHF the default post-training stack for chat models. Read it after you understand the three-model picture above.
- Take from it
- The SFT → reward model → PPO pipeline, KL penalty motivation, labeler agreement stats, and why RL on top of SFT beats SFT alone for helpfulness.
- It skips
- DPO, GRPO, verifiable reasoning rewards, open-source tooling, and modern PPO stability tricks. Lesson 04 picks up where this paper leaves off.
A practitioner-oriented survey of where RLHF breaks: reward misspecification, distribution shift, and the gap between proxy rewards and real objectives.
- Take from it
- Why reward hacking is structural, not accidental; when human feedback is too sparse; and why KL control and eval discipline are not optional extras.
- It skips
- Implementation recipes, DPO as an alternative, and production rollout playbooks. Pair with the reward-hacking section above and lesson 06.
RLHF has more moving parts than supervised tuning. You own the policy, reward model, rollout sampling, KL settings, evals, and rollback plan. Use it when the preference signal is valuable enough to justify that surface area.
Checkpoint
You're ready for the next lesson if you can answer these from memory:
- What does a reward model learn from preference pairs?
- Why does RLHF need drift control?
- Name two signs of PPO instability during training.
- What is reward hacking, and how does Goodhart's law apply?
- Why is a reward model not the same as human judgment?
Quick check
- It generates the final answer shown to users
- It scores prompt-answer pairs during optimization
- It converts text into tokens
- To keep the tuned policy close to useful base behavior
- To make generation faster
- To add new documents into the model
- Reward hacking against the learned proxy
- KL penalty set too high
- Learning rate too low