Quantization: int8, int4, and GGUF

Why lower precision helps

A model is mostly numbers. In many LLM serving setups, the weights are stored in 16-bit floating point. A 7B model at 16 bits needs roughly 14 GB just for weights before runtime overhead. Move the weights to 8 bits and the weight memory roughly halves. Move them to 4 bits and it roughly quarters.

That memory cut matters twice. First, the model may fit on a smaller GPU, a laptop, or a phone. Second, inference often waits on memory bandwidth. If the hardware can move less data per token, it may generate faster.

The catch is that real speed depends on kernels, hardware support, batch size, context length, and whether the runtime can compute efficiently in that format. A tiny file is not automatically a fast server.

Post-training quantization and calibration

Post-training quantization takes an already-trained model and converts weights to a lower precision format. It is popular because it is fast and does not require a full retraining run. You start with a checkpoint, run a quantization tool, then test the result.

Simple quantization maps a range of floating point values into a smaller set of integers. For example, many nearby weight values may collapse into the same int4 bucket. Good methods pick scales per tensor, per channel, per group, or per block so the rounding damage is not spread evenly across sensitive and insensitive parts of the model.

Calibration is how those scales get chosen for aggressive formats like int4. The quantizer runs a small set of representative inputs through the model, records activations at each layer, and picks scale factors that minimize error on the values that actually fire during inference. Calibration does not retrain the model. It is a short measurement pass, often a few hundred to a few thousand prompts. The calibration set should look like production traffic: similar length, domain, and task mix. A calibration set of tiny chat prompts can produce a quant that looks fine in a demo and fails on long JSON extraction.

GPTQ, AWQ, and bitsandbytes

Production LLM quantization is not one algorithm. Three names show up constantly, each with a different default use case.

GPTQ (GPT Quantization) is a post-training weight quant method that compresses weights to 4-bit (sometimes 3-bit) using layer-wise error compensation. It is widely used for GPU serving when you want a smaller Hugging Face-compatible checkpoint and your runtime supports the format. Expect an offline conversion step and a calibration dataset.

AWQ (Activation-aware Weight Quantization) also targets 4-bit weights, but it protects weights that matter more given real activations. The intuition: not every weight is equally sensitive, and sensitivity depends on what the model actually computes. AWQ often recovers a bit more quality than naive int4 at the same bit width, at the cost of another calibration-aware recipe to own.

bitsandbytes is the practical path for loading models in 8-bit or 4-bit inside PyTorch training and inference stacks, often paired with LoRA fine-tuning. It is convenient for experimentation and QLoRA workflows covered in the Fine-tuning LoRA lesson. It is not the same thing as shipping a GGUF file for llama.cpp.

Worked example: size and rough speed

Numbers shift by architecture and hardware, but the orders of magnitude are stable enough to plan with. Suppose you distill a 70B teacher into an 8B student, then deploy the student locally.

Weight memory scales roughly with parameters times bytes per weight. An 8B model at fp16 needs about 16 GB for weights alone (8 billion parameters times 2 bytes). The same 8B model in a Q4_K_M GGUF file is often around 5 GB on disk because K-quants mix bit widths across blocks. That is the difference between "does not fit on a laptop" and "runs on a consumer machine."

Throughput is a separate measurement. On a single consumer GPU, an 8B fp16 model might land near 40 tokens per second while a Q4_K_M build of the same model might reach 80 to 120 tokens per second because less weight data moves per token. Exact numbers depend on context length, batch size, and runtime. The point is: quantization can cut memory by a large factor and often improves decode speed when the stack has efficient kernels. Measure on your target box; do not assume from file size alone.

For GPU fleet serving and cost modeling, carry these measurements into the GPU cost per token lesson once your quant candidate passes eval gates.

KV cache and activation precision

Weight quantization gets most of the attention, but serving also stores the KV cache: past key and value tensors for every layer and every token in context. Long contexts make the KV cache large. Some production stacks quantize KV cache entries to fp8 or int8 to save memory and bandwidth during decode.

KV quant is a serving decision, not a file-format decision. A model can ship as int4 weights while the runtime keeps fp16 KV entries, or compress both. The tradeoff is again slice-specific: long-context recall, multi-turn chat, and RAG with big prompts are the places KV precision shows up first.

int8, int4, and where quality drops

Int8 is often a conservative first step. Many models survive it well, especially when the runtime and hardware support it cleanly. Int4 is more aggressive. It can be the difference between "does not fit" and "runs locally," but it is more likely to hurt long-context behavior, reasoning-heavy tasks, rare tokens, multilingual tasks, and precise formatting.

Do not judge quality from one chat. Compression failures are often slice-specific. A quantized model may answer easy questions fine but fail on tool-call JSON, math, rare labels, or safety boundaries. That is why the eval set from the Fine-tuning dataset lesson matters here too.

What GGUF is for

GGUF is a model file format used heavily in the llama.cpp ecosystem for local and CPU-friendly inference. A GGUF file stores quantized weights plus metadata the runtime needs to load the model. You will see names like `Q4_K_M`, `Q5_K_M`, or `Q8_0`, which describe quantization variants.

For an engineer, the important point is not memorizing every suffix. The important point is that the format and quantization type are part of the deployment target. A model that is great as an fp16 Hugging Face checkpoint may behave differently after conversion to a GGUF quant.

If your target is local inference, edge inference, or CPU fallback, test the exact GGUF file you plan to ship. File format, runtime, prompt template, and sampling settings all affect behavior.

Quantization-aware training

Post-training quantization is the quick path. Quantization-aware training is the heavier path: during training or fine-tuning, the model is exposed to the effects of lower precision. The model learns to be robust to the rounding noise.

This can recover quality, especially for aggressive formats, but it costs more. It also gives you another training recipe to own. For many product teams, the practical order is simple: try post-training quantization, evaluate hard, then reach for quantization-aware training only if the cheaper method fails and the deployment target is worth it.

Pick a compression method

Each scenario has a first reasonable move. The wrong answer is often technically possible but fights your bottleneck.

Quick check