Learn/How Inference Works/Lesson 03
Lesson 03

KV cache explained

The KV cache is the reason decode is practical. It saves the attention state for previous tokens so the model does not have to recompute the whole prompt and every generated token from scratch at every step.

The one idea

The KV cache stores the keys and values produced by attention layers for tokens the model has already seen. Decode appends one new entry per layer per generated token, trading memory for speed.

What gets cached

In transformer attention, each token produces three useful vectors: a query, a key, and a value. The current token's query looks at keys from previous tokens to decide which values matter. That is how the model uses context.

During prefill, the model computes keys and values for all prompt tokens. During decode, the previous keys and values do not change. Instead of recomputing them, the server stores them in memory. That stored state is the KV cache.

When a new token is generated, the model computes that token's new key and value, appends them to the cache, and uses the whole cache for the next step.

After prefill Every prompt token owns a key and value entry. prompt 1 K V prompt 2 K V prompt 3 K V prompt 4 K V decode appends, never rewrites old cells During decode prompt 1 K V prompt 2 K V prompt 3 K V prompt 4 K V out 1 K V out 2 K V new K V
The cache starts with prompt tokens and grows with each generated token. Longer conversations use more cache memory.

Why this speeds up decode

Without the KV cache, every decode step would rerun attention for the entire prefix: original prompt plus all generated tokens so far. Generating a 500-token answer would repeatedly redo old work.

With the cache, the model only computes the new token's fresh state and reads the cached states for previous tokens. That does not make decode free. It still has to run model layers and read cache memory. But it avoids a huge amount of repeated computation.

Step through it below. Set a prompt length, then generate tokens one at a time. The cache row grows by one cell per step, the new cell is the only fresh work, and the counter shows how much you would be recomputing on every step if the cache were not there.

The cache is not tiny

The cost of the KV cache grows with context length, number of layers, attention heads, hidden size, precision, and active requests. This is why long-context serving is a memory problem, not just a model-quality feature.

Every active request needs its own cache because each user has a different prompt and generated output. A server handling many long requests at once can run out of cache memory before it runs out of raw compute.

Common trap

A model that fits in GPU memory at load time may still fail under real traffic. The weights fit, but the active KV caches for concurrent requests may not.

Size the cache in bytes

Production engineers estimate KV cache memory with a formula, not vibes. For each request:

KV_bytes = 2 × num_layers × seq_len × num_kv_heads × head_dim × bytes_per_element

The leading 2 is keys plus values. seq_len is the live context: prompt plus every generated token so far. num_kv_heads is the number of key/value head groups. With full multi-head attention, that equals the query head count. With grouped-query attention (GQA), several query heads share one key/value set, so num_kv_heads is smaller and the cache shrinks. See Transformer architecture for how GQA is wired.

Typical 7B-class config: 32 layers, 8 KV heads (GQA), head dimension 128, sequence length 8,192, FP16 (2 bytes per element).

KV_bytes = 2 × 32 × 8192 × 8 × 128 × 2
         = 1,073,741,824 bytes ≈ 1.0 GB per request

That is cache alone for one active request at 8k context. Ten concurrent 8k requests need roughly 10 GB of KV memory before you count weights, activations, or batching overhead. Double the context to 16k and this term doubles.

Some serving stacks store the KV cache in FP8 instead of FP16. That cuts cache bytes roughly in half at the cost of a small quality risk and hardware/kernel support requirements. It is a common lever when long context is the memory bottleneck but you cannot add GPUs. Pair it with eval on your actual prompts before you ship it.

Before IO-aware attention kernels, materializing full attention matrices for long sequences blew up memory traffic during prefill. FlashAttention fuses attention so the GPU reads and writes less HBM per token. It does not change the KV cache size formula above, but it makes long-context prefill and decode practical on hardware that would otherwise choke on bandwidth. Modern serving stacks use FlashAttention-style kernels by default.

Why cache management is serving logic

Inference servers spend a lot of effort managing KV cache blocks. They allocate cache memory when a request starts, append blocks as decode continues, reuse freed blocks when requests end, and sometimes evict or swap when memory pressure gets bad.

This is one reason libraries like vLLM, TGI, TensorRT-LLM, and llama.cpp matter. The model architecture is only part of serving. The cache scheduler, memory layout, batching behavior, and paging strategy can decide whether the same model feels fast or unusable. vLLM's PagedAttention stores KV cache in fixed-size blocks allocated on demand, like virtual memory, so sequences of different lengths do not waste GPU RAM. That design is what makes continuous batching practical; see Continuous batching in the Latency course.

Blog
Easy, Fast, and Cheap LLM Serving with PagedAttention (vLLM team, 2023)

The post that connected paged KV cache blocks to a production serving engine. Read it after the byte formula above clicks, before you tune a vLLM deployment.

Take from it
Why reserving worst-case KV memory per slot wastes GPU RAM, how block-based allocation works, and why throughput under mixed traffic depends on the scheduler plus cache layout together.
It skips
Prefill/decode theory, TTFT/TPOT definitions, chunked prefill tuning, and cost modeling. Those are this course and Serving & Economics.
Engineering reality

The KV cache is a production capacity limit. If you allow every user to send giant prompts and request giant answers, you are reserving memory for those choices. Put real limits on input tokens, output tokens, and concurrent long-context requests.

Checkpoint

You're ready for the next lesson if you can answer these from memory:

  • What do the K and V in KV cache refer to?
  • Why does the cache help decode?
  • Why does cache memory grow during generation?
  • Write the KV cache byte formula and name each term.
  • How does GQA change num_kv_heads in that formula?
  • Why can concurrency make KV cache memory the bottleneck?

Quick check

  • Reuse previous tokens' attention state
  • Store the tokenizer vocabulary
  • Store a second copy of all model weights
  • The weights grow for each user
  • Each active request needs its own KV cache
  • Temperature uses extra memory