Pruning, sparsity, and low-rank compression

Pruning removes parts of the model

Pruning means removing weights, neurons, attention heads, layers, or blocks that seem less important. The simplest version sets small weights to zero. More structured versions remove whole chunks so the model graph itself changes.

Unstructured pruning can create many zeros, but general-purpose hardware may still do almost the same dense matrix multiplication unless the runtime uses sparse kernels. Structured pruning is easier to accelerate because a removed head or channel is genuinely absent from the computation.

The risk is simple: importance is task-dependent. A head that looks useless on one calibration set may matter for a rare language, a safety case, or a long-context pattern. Pruning needs evals by slice, not only a single score.

Sparsity is a hardware contract

Sparsity means many values are zero or skipped. It is attractive because a sparse model seems like it should do less work. In practice, sparsity has to match what the hardware and kernels support.

A random sparse pattern can be awkward. The runtime has to store indices, branch around missing values, and keep memory access efficient. General-purpose GEMM kernels on NVIDIA and AMD GPUs are still mostly dense. Unless you hit a path built for sparsity (NVIDIA 2:4 structured sparsity on some Ampere+ chips, specialized research kernels, or vendor-specific formats), the GPU may multiply by the zeros anyway.

Some accelerators support fixed patterns, such as 50 percent sparsity inside small blocks. Those patterns are less flexible, but they are easier to execute quickly. The lesson from production: a checkpoint with 40 percent zeros is not automatically 40 percent faster. Measure decode latency on the exact runtime and GPU generation you plan to ship.

So the real question is not "how sparse is the model?" The real question is "can my serving stack exploit this exact sparsity pattern on this hardware?"

Low-rank compression and LoRA

Large neural networks are full of big matrices. Low-rank compression replaces a large matrix with two smaller matrices whose product approximates the original. If the rank is much smaller than the original dimensions, the model stores fewer parameters and may do less work.

This shares math with LoRA, but the goal is different. LoRA, covered in Fine-tuning Lesson 03, adds low-rank update matrices beside a frozen base model so you can adapt behavior cheaply. Low-rank compression tries to replace an existing weight matrix with a cheaper factorized form. They are complementary tools, not duplicates:

• LoRA: keep the base model, train a small adapter for a new task, swap adapters at serve time.
• Low-rank compression: bake the approximation into the weights you ship, aiming for a smaller permanent artifact.
• Together: distill or fine-tune with LoRA, then quantize the merged checkpoint for deployment. Each step attacks a different bottleneck.

The tradeoff is approximation error. Push the rank too low and the compressed layer loses information. Some layers tolerate it. Others do not. Good compression recipes often treat different layers differently instead of applying one global setting everywhere.

Layer dropping and smaller architectures

Another path is to remove whole layers or train a smaller architecture directly. This can be easier to serve because the final model is dense and regular. There are fewer tricks for the runtime to understand.

Layer dropping usually needs recovery training or distillation after the cut. A transformer is not a stack of independent parts. Removing layers changes how representations flow through the network, so the remaining model needs time to adapt.

For production, a purpose-built small model is often cleaner than a heavily hacked large model. If a 3B model trained well beats a pruned 7B model on your evals and runs faster, choose the boring option.

Pick structural compression for the bottleneck

If you are memory-bound, quantization may be enough. If you are decode-latency-bound and the runtime cannot exploit sparsity, structural pruning may not help. If you need a model that fits a specific accelerator pattern, structured sparsity can make sense. If you are shipping one artifact to many devices, a dense smaller student may be easier to operate.

Do not choose pruning because it sounds more scientific. Choose it because it maps to a measured bottleneck and your serving stack can benefit.

Quick check