Deploy small, fast models

Choose the serving role

A small model can play several roles. It can be the primary model for a narrow task, a pre-filter before a larger model, a router that chooses tools or prompts, a local privacy-preserving model, or a fallback when the expensive model is unavailable.

Each role has a different risk profile. A router can be wrong but still send the request to a larger model later. A primary model owns the answer users see. An on-device model may trade quality for privacy and low latency. Be explicit about the role before tuning infrastructure.

For many products, the best first deployment is a shadow run. The small model receives the same requests as production, but its answers are logged rather than shown. This gives you traffic-shaped eval data without user exposure.

Route by confidence and task shape

Routing rules can be simple. Send short, common, well-formed cases to the small model. Send long, rare, high-risk, or low-confidence cases to the large model. If the small model fails schema validation, returns low confidence, or hits a policy boundary, escalate.

Do not hide fallback rate. It is one of the core economics of the system. If 95 percent of requests stay on the small model, you probably have a strong cost win. If only 40 percent stay there, the blended latency and cost may not justify the extra complexity.

Latency tricks beyond compression

Quantization and distillation shrink the model, but serving latency has other levers. Speculative decoding runs a small draft model to propose tokens and a larger model to verify them, cutting time-to-first-token and tokens-per-second on some workloads without changing the final model weights. It pairs well with a compressed draft model plus a larger verifier. See Speculative decoding in the Latency course for when that pattern pays off.

Compression and speculative decoding solve different problems. Quantization attacks memory and per-token math cost. Speculative decoding attacks serial token generation. Teams sometimes quantize the draft model, keep the verifier at higher precision, and still route hard requests to a full fallback.

A/B quant variants in production

Do not ship the first quant that fits in memory. Run a short multi-quant A/B: same prompts, same runtime, two or three artifacts such as Q4_K_M, Q5_K_M, and fp16 baseline. Compare slice quality, p95 latency, and memory headroom on real traffic shape.

Pick the smallest quant that clears your acceptance budget from Lesson 05, not the most aggressive one that still passes a chat demo. Log which artifact served each request so you can compare fallback rate and user corrections across variants.

For fleet economics after you pick a winner, model cost with the Serving & Economics course and edge constraints with Edge and on-device serving.

Version everything together

A compressed model artifact is more than weights. It includes the base model, distillation data, teacher model, training recipe, quantization method, runtime, prompt template, tokenizer, safety rules, and eval report. Changing any of those can change behavior.

Version the whole bundle. A rollback should restore the model and the prompt template together. If you roll back weights but keep a new parser or routing policy, you may not have rolled back the system users experience.

Keep the release note short but concrete: what changed, which eval gates passed, known weak slices, fallback policy, and the expected cost or latency gain.

Monitor for boundary drift

The safe zone for a compressed model can drift. Product traffic changes, users discover new prompts, policies change, and the large fallback may improve. Monitor the slices you used during evaluation: parse failures, fallback rate, latency percentiles, refusal rate, escalation rate, user corrections, and sampled quality review.

When fallback rate climbs, ask whether the router got stricter, traffic got harder, or the small model is stale. When quality drops in one slice, add examples to the next distillation dataset instead of blindly retraining on everything.

Know when not to deploy it

Do not deploy a compressed model just because it is technically impressive. Skip it if the quality budget is unclear, eval coverage is weak, fallback rate destroys the savings, or the task is so broad that the small model will constantly hit its limits.

Also skip it when the product is early and traffic is low. Compression work has fixed cost: data, training, evals, serving, monitoring, and incident response. It is often smarter to pay the API bill until usage proves the bottleneck is real.

The best compressed models feel boring in production. They handle a known slice cheaply, escalate when uncertain, and give you clear numbers showing why they exist.

Deployment scenarios

Compression choices do not end at conversion. How you serve the artifact matters as much as how you made it.

Quick check