Why LLM inference is memory-bound

Compute-bound vs memory-bound

A workload is compute-bound when arithmetic is the slow part. Give it more math throughput and it gets faster. A workload is memory-bound when moving data is the slow part. Give it more math units and they sit around waiting for bytes.

LLM inference has both modes, but decode is often memory-bound. For each generated token, the server has to read a large amount of model weight data and use the current KV cache. If the batch is small, there may not be enough arithmetic per byte loaded to keep the GPU fully busy.

A useful mental model is the roofline: plot arithmetic intensity (FLOPs per byte moved) against hardware peaks. If your point sits below the ridge, more FLOPs will not help because the GPU is waiting on memory. Decode with batch size 1 sits there for most model sizes. Prefill with a long prompt often sits higher because many tokens share each weight load.

On recent datacenter GPUs, HBM bandwidth is the number to remember. An NVIDIA A100 80GB delivers about 2.0 TB/s of HBM bandwidth. An H100 pushes about 3.35 TB/s. A 7B model in FP16 is roughly 14 GB of weights. Loading those weights once per decode step means you need to move on the order of 14 GB through memory per token before you even count the KV cache. At 2 TB/s that is already tens of milliseconds of memory traffic per step if the path is not perfectly overlapped, which is why decode feels memory-bound even when the FLOP counter looks idle.

Why decode has low reuse

In training and large prefill batches, the same model weights can be used across many tokens at once. That creates a lot of arithmetic for each weight load. The hardware has a better chance of staying busy.

In single-request decode, you generate one token for one user. The model still has to touch a huge set of weights, but it only gets one token's worth of work from that read. That is poor reuse. Batching multiple decode steps from different users improves reuse, which is why serving systems care so much about batching.

Model size is speed pressure

A bigger model usually means more weights to read per token. More weights can improve quality, but they also raise memory traffic and memory footprint. If your task does not need the larger model, you pay for unused capacity on every request.

Quantization helps because lower precision weights take fewer bytes. A 4-bit model can be much easier to fit and move than a 16-bit model. But quantization is not magic. It can affect quality, and speed depends on whether the hardware and kernels are optimized for that format.

The KV cache joins the fight

Weights are not the only memory traffic. Decode also reads from the KV cache so the current token can attend to previous tokens. As the context grows, the cache gets larger. As concurrency grows, there are more caches alive at the same time.

This creates a practical serving tradeoff: long contexts can improve capability, but they reduce how many requests fit comfortably on the same hardware. A model might have a 128k context window on paper and still be too expensive to serve at that length for normal traffic.

KV_bytes ≈ 2 × 32 × 32768 × 8 × 128 × 2 ≈ 4.0 GB per request

What improves a memory-bound path

The usual fixes reduce bytes moved, reuse bytes better, or use hardware with faster memory. Smaller models, quantization, grouped-query attention, cache paging, better kernels, and batching all target some part of this.

There is no single universal fix because the bottleneck changes with model size, prompt length, output length, batch size, hardware, and traffic shape. That is why benchmark numbers without workload shape are hard to trust.

Quick check