What is preference tuning?

Why imitation is not enough

Supervised fine-tuning works well when the target answer is fairly clear. Extract this field. Classify this ticket. Rewrite this paragraph in a known style. The dataset gives the model one good answer and training pushes probability mass toward that answer.

But many LLM behaviors are not that clean. Two answers can both be factually correct while one is more useful. One answer may be shorter, safer, clearer, more honest about uncertainty, or better aligned with the product's voice. It is easier for a human reviewer to compare two candidates than to write the perfect target answer from scratch.

Preference tuning turns that comparison into training signal. The raw unit is usually a prompt, a chosen answer, and a rejected answer. Over many examples, the model learns what kinds of outputs win.

The shape of a preference example

A preference row is small, but it carries a lot of meaning:

• Prompt: the user request and any system context.
• Chosen answer: the response the reviewer preferred.
• Rejected answer: the response the reviewer preferred less.
• Rubric metadata: why the chosen answer won, who labeled it, and whether the pair was close.

The Bradley-Terry model

Most preference tuning methods assume a simple probabilistic model behind the labels. In the Bradley-Terry model, each answer has a latent score. Given two answers y_w (winner) and y_l (loser) for the same prompt, the probability that a human prefers the winner is:

P(y_w > y_l | x) = σ( r(x, y_w) − r(x, y_l) )

Here r(x, y) is a reward function over prompt-answer pairs, and σ is the logistic sigmoid. The model does not need to know the absolute score. It only needs the difference between the two answers to predict which one humans pick.

That framing explains why pairwise data is enough. RLHF learns an explicit reward model r and optimizes against it. DPO reparameterizes the same preference likelihood directly on the policy. GRPO uses group-relative scores instead of human pairs. The shared assumption is that preferences reflect a ranking, not a single perfect target string.

What preference tuning changes

Preference tuning is usually about behavior at the margin. It nudges the model toward answers that humans or automated rubrics prefer. That can mean better helpfulness, cleaner formatting, fewer unsupported claims, better refusals, stronger reasoning traces, or a house style that is hard to express as a single instruction.

It is not magic reasoning juice. If the base model cannot solve the task with a good prompt and examples, preference tuning may only teach it to sound confident while still being wrong. The base capability still matters.

Why RL shows up

Classical RLHF uses reinforcement learning because the final thing we care about is not next-token likelihood. We care about a score: did humans prefer the whole answer? A reward model estimates that score, then an RL algorithm updates the model to produce answers with higher estimated reward.

Newer methods like DPO avoid a separate RL loop for many use cases. They train directly on chosen and rejected answers with a simpler objective. That is why modern "preference tuning" often includes both RLHF-style methods and non-RL methods. The shared idea is preference signal.

Paper lineage: how the methods connect

This course follows the arc the field actually took. Each paper solves a problem the previous stack exposed:

1. InstructGPT (2022): Proved SFT plus RLHF beats SFT alone for helpfulness. Introduced the reward model + PPO loop covered in lesson 03.
2. DPO (2023): Showed the reward model can be folded into a closed-form preference loss. Same Bradley-Terry objective, no online PPO. Covered in lesson 04.
3. DeepSeek-R1 (2025): Brought group-relative RL with verifiable rewards to open reasoning models. Covered in lesson 05.

You do not need to read every paper before tuning a model. You do need to know which problem each method was built to solve, so you pick the right tool instead of the trendiest one.

SFT, DPO, RLHF, or GRPO?

Preference tuning is not one technique. The right choice depends on your data and reward signal:

• Supervised fine-tuning: One target answer per prompt. Use this first to teach task shape.
• DPO: Offline chosen/rejected pairs. Lowest ops after SFT. Start here for chat alignment.
• RLHF: Online rollouts against a learned reward model. Use when exploration during training matters and you can own PPO.
• GRPO: Verifiable outcome rewards (math, code, schema checks). Use when you can score many samples per prompt.

Lesson 06 has the full shipping checklist. The quizzes below test whether you can pick the right method, not just name it.

Quick check