DPO: direct preference optimization
DPO is popular because it removes a lot of RLHF machinery. You still need good preference data, but you do not need to train a separate reward model and run an online RL loop.
DPO trains the model to make chosen answers more likely than rejected answers, while a reference model keeps the update from drifting too far.
What DPO skips
Classic RLHF has two stages after supervised tuning: train a reward model, then optimize a policy against that reward model. DPO folds the preference signal directly into the model update. The dataset still contains prompts, chosen answers, and rejected answers.
Rafailov et al. showed this is not a different goal from InstructGPT's RLHF. It is the same Bradley-Terry preference objective from lesson 01, with the reward model replaced by an implicit function of the policy and reference model. If you understood the three-model loop in lesson 03, DPO is what happens when you delete the standalone reward model and PPO rollout.
This makes DPO feel closer to supervised fine-tuning from an engineering point of view. You can train offline on a static dataset. There is no rollout loop where the policy samples fresh answers and asks a reward model to score them during training.
The DPO objective
DPO does not train a separate reward model. Instead it defines an implicit reward from the gap between the policy πθ and a frozen reference πref. For a prompt x, chosen answer yw, and rejected answer yl, the loss is:
L_DPO = −E[ log σ( β · ( log π_θ(y_w|x)/π_ref(y_w|x) − log π_θ(y_l|x)/π_ref(y_l|x) ) ) ]Read it in pieces. The term log πθ(y|x) − log πref(y|x) is how much more likely the tuned model makes answer y compared with the reference. DPO pushes that gap wider for the chosen answer and narrower for the rejected one. The sigmoid σ turns the gap into a preference probability, same as the Bradley-Terry model from lesson 01.
The scalar β (beta) controls how strongly DPO enforces the preference relative to staying near the reference. Low β: small updates, safer but slow. High β: aggressive preference fitting, higher risk of overfitting label noise or collapsing diversity. Typical starting values are 0.1–0.5; tune on a held-out preference set.
DPO is like saying: compared with the old model, make this answer win more often than that answer. β decides how hard you push. The reference model is the anchor.
The reference model still matters
DPO compares the tuned model against a reference model, usually the supervised-tuned model before preference training. The goal is not simply "make the chosen answer likely at any cost." The goal is to prefer the chosen answer relative to the rejected one without destroying broad behavior.
Without the reference term, DPO reduces toward ordinary likelihood training on chosen answers only. That drifts faster and forgets capabilities the preference data never covers.
Why teams often start here
DPO is a practical first preference-tuning method because it has fewer moving parts than RLHF. You can reuse much of the supervised fine-tuning setup: dataset loading, batch training, validation, checkpoints, and adapter training. That lowers the operational burden.
It also fits many product tuning problems: style preference, answer helpfulness, format discipline, refusal tone, and domain-specific ranking. If the data is mostly offline human preference pairs, DPO is often the simplest serious baseline.
Most teams run DPO through existing tooling rather than writing the loss from scratch. Hugging Face TRL's DPOTrainer wraps the objective, reference model handling, and β scheduling for open models. The same library's GRPOTrainer covers group-relative RL in lesson 05. On hosted APIs, OpenAI's preference fine-tuning endpoint accepts chosen/rejected pairs without you managing a reward model or PPO loop.
from trl import DPOTrainer, DPOConfig
from datasets import load_dataset
dataset = load_dataset("json", data_files="prefs.jsonl") # prompt, chosen, rejected
trainer = DPOTrainer(
model=model,
ref_model=ref_model, # or peft_config for LoRA
args=DPOConfig(beta=0.1, ...),
train_dataset=dataset["train"],
processing_class=tokenizer,
)
trainer.train()You still own the preference dataset, eval set, and β sweep. TRL handles the loss, reference model sync, and checkpointing.
Where DPO is not enough
DPO is not a free replacement for every RL setup. It depends on the quality and coverage of static preference pairs. If the behavior needs long exploration, tool feedback, verifiable multi-step outcomes, or rewards that depend on running code, a pure offline pairwise method may miss the important signal.
It can also inherit label bias. If reviewers consistently prefer a shallow signal, such as verbosity or confidence, DPO can teach the model that shortcut efficiently. Simpler training does not remove the need for careful evals.
IPO (identity preference optimization) replaces the sigmoid with a squared loss that is less sensitive to noisy or contradictory pairs. Use it when labelers disagree often or when chosen/rejected gaps are tiny.
KTO (Kahneman-Tversky optimization) works from binary good/bad labels per answer instead of strict pairs. Useful when you have thumbs-up/down logs but not full pairwise comparisons.
Both still need a reference model and careful evals. They are not magic fixes for bad data, but they reduce training instability when standard DPO overfits label noise.
The paper that made DPO the default alternative to RLHF for offline preference tuning.
- Take from it
- The DPO loss derivation, the implicit reward parameterization, β's role, and proof that DPO optimizes the same Bradley-Terry objective as RLHF without online sampling.
DPO is attractive because it is easier to run, not because it removes alignment risk. Treat it like a strong baseline. Keep RLHF or outcome-based RL for cases where static preference pairs do not capture the reward you need.
Checkpoint
You're ready for the next lesson if you can answer these from memory:
- What parts of the RLHF pipeline does DPO skip?
- What does the DPO loss compare between chosen and rejected answers?
- What does the β parameter control?
- When might static preference pairs be too weak?
Quick check
- It skips the separate reward model and online RL loop
- It does not need labeled data
- It removes inference cost in production
- It improves held-out preference win rate
- The training method has fewer moving parts, so evals matter less
- It passes regression tests on important cases
- Increase β to 1.0
- IPO or a stricter rubric that discards near-ties
- Switch to full RLHF immediately
- Standard DPO
- KTO
- GRPO