Build a teacher-student distillation dataset

Start from traffic, not imagination

The best prompts for distillation come from the task the small model will actually serve. Use production logs when you can. If you do not have traffic yet, use eval cases, support tickets, documents, or hand-written cases that match the expected product shape.

Coverage matters more than raw count at the start. A thousand near-duplicate examples teach less than a few hundred examples that span easy cases, hard cases, empty inputs, policy boundaries, weird formatting, and user mistakes. Distillation is not magic averaging. It is pattern copying.

Make the target narrow. "Be a smaller general assistant" is not a dataset spec. "Classify these ticket messages into this taxonomy and return JSON with confidence and escalation reason" is a dataset spec.

Generate teacher outputs deliberately

The teacher prompt should describe exactly what you want the student to learn. If the final product needs strict JSON, make the teacher produce strict JSON. If the product needs short answers, do not let the teacher write essays. If refusals matter, include refusal cases in the teacher job.

For ambiguous tasks, generate more than one teacher answer or ask the teacher to grade its own answer against a rubric. You are not doing this because the teacher is always right. You are doing it because disagreement shows you where the boundary is soft.

Keep the teacher prompt, model version, sampling settings, and output filters with the dataset. Six months later, you should be able to explain why an example exists and which teacher produced it.

API teacher distillation

Many teams never run the teacher on their own GPUs. The teacher is a hosted API: OpenAI, Anthropic, Google, or an internal gateway. The workflow is the same as in-house distillation, but the teacher is accessed over HTTP and you usually get response distillation, not logits.

OpenAI documents a distillation-oriented fine-tuning path: generate training rows with a stronger model such as gpt-4o or gpt-4o-mini, store prompt and completion pairs, then fine-tune a smaller hosted model on that dataset. The product pattern is familiar: a frontier model labels data, a cheaper model serves traffic. You still own filtering, eval splits, schema normalization, and version metadata. The API does not remove dataset quality work.

The same pattern appears outside OpenAI. Any teacher API can label rows if the output shape is stable and you can afford the labeling bill. Treat API distillation as a data pipeline with a meter attached: log teacher model ID, prompt hash, token cost per row, and filter status so you can trace regressions later.

Watch synthetic diversity

When prompts are generated by another model instead of pulled from real traffic, diversity becomes the main risk. A synthetic prompt generator often reuses the same sentence shapes, entity types, and difficulty bands. The student learns those patterns and looks strong on eval, then fails on messy production wording.

Mix sources on purpose: real logs, hand-written edge cases, adversarial phrasing, and synthetic fill for rare labels. Measure slice coverage, not only row count. If 80 percent of rows share one template family, you have volume without distribution.

Also watch teacher monoculture. If every label comes from one teacher version on one day, the student inherits that snapshot. When the teacher API updates, your student may be optimizing for yesterday's behavior. Keep a golden eval set from real traffic so you notice when synthetic data drifts away from what users actually send.

Filter harder than you generate

Teacher output is not ground truth. Large models can be verbose, inconsistent, overconfident, or just wrong. Bad teacher labels are worse than missing labels because the student will train toward them.

Use automatic filters first: valid JSON, schema match, length bounds, forbidden phrases, citation presence, allowed labels, and duplicate detection. Then sample by slice for human review. Do not only review random rows. Review the weird rows: low-confidence cases, long outputs, rare labels, failed parses, and cases where two teacher samples disagree.

If people correct the teacher, keep the corrected answer as the label and record that it was edited. Those edited rows are valuable because they mark places where the teacher alone was not enough.

Split by source and difficulty

Random train and test splits can leak near-duplicates. If the same customer issue appears in both train and test with tiny wording changes, your student looks smarter than it is. Split by source, account, document, time window, or task family when possible.

Hold out hard cases on purpose. A compressed model often looks fine on easy examples and collapses on edge cases. Your eval split should include rare labels, long contexts, short contexts, ambiguous inputs, adversarial wording, and cases where the safe answer is "I cannot answer from the given context."

Also keep a small golden set that never enters training. That set is the judge you come back to after each dataset refresh, model change, or quantization pass.

Distill the output shape you need

Students learn habits. If the teacher output includes rambling rationales, the student may ramble. If the teacher sometimes returns `"category"` and sometimes `"type"`, the student may drift. If the teacher says "probably" in labels, the student may emit uncertainty where your parser expects a fixed enum.

Before training, normalize the answer shape. Use stable keys, stable label names, clear refusal text, and a consistent amount of explanation. This is boring data work, but it is exactly what turns distillation from a demo into a product asset.

Quick check