Deploy small, fast models

Choose the serving role

A small model can play several roles. It can be the primary model for a narrow task, a pre-filter before a larger model, a router that chooses tools or prompts, a local privacy-preserving model, or a fallback when the expensive model is unavailable.

Each role has a different risk profile. A router can be wrong but still send the request to a larger model later. A primary model owns the answer users see. An on-device model may trade quality for privacy and low latency. Be explicit about the role before tuning infrastructure.

For many products, the best first deployment is a shadow run. The small model receives the same requests as production, but its answers are logged rather than shown. This gives you traffic-shaped eval data without user exposure.

Route by confidence and task shape

Routing rules can be simple. Send short, common, well-formed cases to the small model. Send long, rare, high-risk, or low-confidence cases to the large model. If the small model fails schema validation, returns low confidence, or hits a policy boundary, escalate.

Do not hide fallback rate. It is one of the core economics of the system. If 95 percent of requests stay on the small model, you probably have a strong cost win. If only 40 percent stay there, the blended latency and cost may not justify the extra complexity.

Version everything together

A compressed model artifact is more than weights. It includes the base model, distillation data, teacher model, training recipe, quantization method, runtime, prompt template, tokenizer, safety rules, and eval report. Changing any of those can change behavior.

Version the whole bundle. A rollback should restore the model and the prompt template together. If you roll back weights but keep a new parser or routing policy, you may not have rolled back the system users experience.

Keep the release note short but concrete: what changed, which eval gates passed, known weak slices, fallback policy, and the expected cost or latency gain.

Monitor for boundary drift

The safe zone for a compressed model can drift. Product traffic changes, users discover new prompts, policies change, and the large fallback may improve. Monitor the slices you used during evaluation: parse failures, fallback rate, latency percentiles, refusal rate, escalation rate, user corrections, and sampled quality review.

When fallback rate climbs, ask whether the router got stricter, traffic got harder, or the small model is stale. When quality drops in one slice, add examples to the next distillation dataset instead of blindly retraining on everything.

Know when not to deploy it

Do not deploy a compressed model just because it is technically impressive. Skip it if the quality budget is unclear, eval coverage is weak, fallback rate destroys the savings, or the task is so broad that the small model will constantly hit its limits.

Also skip it when the product is early and traffic is low. Compression work has fixed cost: data, training, evals, serving, monitoring, and incident response. It is often smarter to pay the API bill until usage proves the bottleneck is real.

The best compressed models feel boring in production. They handle a known slice cheaply, escalate when uncertain, and give you clear numbers showing why they exist.

Quick check