What is model distillation?
Distillation is the move from "this big model works" to "can a smaller model copy the parts we actually need?" It is one of the cleanest ways to cut latency and cost when the task is stable.
Model distillation trains a smaller student model on signals produced by a larger teacher model. The goal is not to clone every capability. The goal is to transfer the behavior your product needs.
The teacher-student setup
In normal supervised fine-tuning, examples usually come from people or existing product data: prompt in, expected answer out. In distillation, the expected answer often comes from a stronger model. The large model acts as the teacher. The smaller model is the student.
For LLMs, the teacher might be a frontier API model, a large open model, or an internal model that is too slow for the final product. You ask the teacher to solve many examples, clean those outputs, then train the student to produce similar answers.
This is useful because the teacher output can include choices that are hard to write by hand: tone, reasoning style, refusal behavior, tool call shape, extraction judgment, and edge-case handling. The student does not become the teacher. It learns a cheaper approximation over the distribution you show it.
What gets distilled?
Distillation methods fall into two families that often get mixed up: response distillation and logit distillation.
Response distillation trains the student on the teacher's final text or label. The student sees the input and the teacher answer, then learns to imitate the answer. This is the default for most LLM product work: classification, extraction, rewrite, routing, support drafts, and tool-call shapes. You do not need access to the teacher's internal weights or logits. A frontier API that returns only text is enough.
Logit distillation trains the student to match the teacher's output probability distribution, not just the winning token or class. Geoffrey Hinton called the extra information in those soft probabilities dark knowledge: the teacher may assign 0.55 to class A and 0.40 to class B, which tells the student the case is ambiguous in a way a hard "A" label cannot. Classic image-classifier distillation uses a temperature-scaled softmax over logits. For LLMs, you get the same idea when you own both models and can pass teacher logits during training. When the teacher is a closed API, you approximate logit distillation with multiple samples, rankings, or rubrics instead.
Some setups also distill intermediate behavior. A teacher might produce a rationale, a plan, a tool call trace, or rejected alternatives. That extra signal can help, but it can also teach the student to copy noisy reasoning. Use it only when the intermediate text maps to behavior you want at inference time.
The paper that named knowledge distillation and dark knowledge. Read it after the teacher-student picture above clicks, before you build a dataset in Lesson 02.
- Take from it
- Soft targets carry more information than hard labels; temperature softens the teacher distribution; the student learns from what the teacher almost chose, not only what it chose.
- It skips
- LLM response distillation from APIs, dataset curation, quantization, and every production concern in this course. Those are the engineering layers on top of the core idea.
A hard label says "this is the answer." A soft target carries more shape: confidence, alternatives, or multiple acceptable outputs. Logit distillation is the classic way to pass soft targets. Response distillation is the practical LLM default when you only have teacher text.
Distillation is not the same as quantization
Distillation trains a new or adapted model. Quantization changes how numbers inside a trained model are stored and computed. Both can make inference cheaper, but they act at different layers.
If you distill a 70B teacher into a 7B student, you are changing the model. If you quantize a 7B model from 16-bit weights to 4-bit weights, you are changing the representation of the same model. In production, teams often combine them: pick or train a smaller student, then quantize it for the target hardware.
The mistake is treating all compression as one knob. It is not. Distillation changes behavior. Quantization changes precision. Pruning changes structure. The right mix depends on which constraint is hurting: memory, latency, throughput, quality, privacy, or device limits.
When distillation is worth it
Distillation shines when the task is narrow, repeated, and expensive at scale. Think high-volume classification, extraction, moderation, routing, structured summarization, customer support drafts, or a voice-agent turn where every 100 milliseconds matters.
It is weaker when the product needs open-ended reasoning across many domains. A small student can copy a teacher's style on examples, but it will not inherit the full breadth of a much larger model. The training data bounds the behavior. Outside that zone, the student may fail sooner and more confidently.
Distillation only pays off if the new model will serve enough traffic to repay the work. You are buying lower inference cost with training cost, data work, eval work, and ongoing ownership of another model artifact.
Checkpoint
You're ready for the next lesson if you can answer these from memory:
- What is the difference between response distillation and logit distillation?
- What did Hinton mean by dark knowledge?
- How is distillation different from quantization?
- What kind of production task is a good fit for distillation?
Quick check
- Generate teacher-labeled examples and train a smaller student
- Add a longer prompt to the large model
- Build a vector database first
- The training examples the model learns from
- The numeric precision used to store or compute model weights
- Which model receives each request