Prefill vs decode

Prefill reads the prompt

Before the model can generate a useful first token, it has to read the prompt: system message, user message, chat history, tool results, retrieval chunks, and any hidden scaffolding the product adds. This is the prefill phase.

In prefill, the model can process many input positions in parallel. A transformer layer can compute representations for the prompt tokens together, because all prompt tokens are already known. This makes prefill chunky: a lot of work happens before the first visible token appears. While it works, prefill fills the KV cache with the keys and values for every prompt token, so decode can reuse them instead of recomputing the prompt for each new token. The next lesson goes deep on that cache.

Because it handles many tokens at once, prefill is a big, dense pile of matrix multiplication. It keeps the GPU's math units busy, which means prefill is usually compute-bound: the limit is how fast the chip can multiply, and the cost scales with the number of prompt tokens. Double the prompt and you roughly double the prefill work.

That first visible delay is called time to first token, or TTFT, and it is not all prefill. TTFT is queue time plus prefill time. Under load a request can sit waiting for a slot before any prefill starts, so a long silence can mean a long prompt, a busy server, a cold start, or all three. When you debug TTFT, separate the wait-in-line part from the read-the-prompt part before you reach for a fix.

Decode writes the answer

Decode starts after prefill. The model predicts one next token, the sampler picks it, and that token is appended to the context. Then the model predicts the next token. The answer grows one token at a time.

This phase is sequential. You cannot generate token 200 until token 199 exists. The server can still batch multiple users together, but each individual response has a dependency chain through its own generated tokens.

Each decode step does almost no math: it processes a single new token. But to do that, the GPU still has to read the entire model's weights out of memory, plus the growing KV cache. So decode is usually memory-bound: the limit is how fast the chip can move data, not how fast it can multiply. The weights get loaded again for every single token, which is why one decode step for one request leaves most of the GPU's math units idle. That waste is exactly what batching and the other tricks in the Latency & Throughput course exist to recover.

Drag the sliders, then press Play. Prefill is one block; decode ticks appear one token at a time. The vertical marker is where TTFT ends.

Chunked prefill for long prompts

A very long prompt does not have to monopolize the GPU in one giant prefill step. Modern schedulers can slice prefill into chunks and interleave those chunks with ongoing decode work from other requests. Instead of freezing every in-flight stream while a 30,000-token prompt loads, the server processes a slice of prefill, runs a decode step for someone already streaming, then returns to the next prefill slice.

Chunked prefill trades a slightly higher TTFT for the newcomer against smoother inter-token latency (ITL) for everyone already decoding. That is usually the right trade under mixed traffic. The scheduling details live in Scheduling and tuning throughput in the Latency course.

Why users feel the phases differently

A slow prefill feels like silence. The user has clicked send and nothing is visible yet. A slow decode feels like a weak stream: text appears, but slowly. Both can produce the same total request time, but they feel different.

This is why streaming helps perceived latency. If prefill is short and decode is long, the user sees progress quickly. If prefill is long, streaming cannot show anything until the first token exists.

One is a sprint, the other is a drip

Put the two profiles side by side. Prefill is compute-bound and parallel: it chews through the whole prompt in a burst, and its cost tracks the number of input tokens. Decode is memory-bound and sequential: it drips out one token at a time, reloading the weights for each, and its cost tracks the number of output tokens. They are different kinds of work, and they respond to different fixes. Shrinking the prompt speeds up prefill. A smaller or faster-served model, or better batching, speeds up decode.

This split is why a single "it's slow" complaint is never enough to act on. A request that is slow to start and a request that is slow to stream have almost nothing in common under the hood.

Prompt shape matters

Two prompts with the same visible user message can have very different prefill cost. One might include a short system prompt. Another might include 20 turns of chat history, five retrieved documents, tool traces, and a JSON schema. The model sees all of it as input tokens.

That hidden context is often where latency comes from. RAG systems are a common example. Retrieval can improve quality, but every chunk inserted into the prompt has to be read during prefill. Bigger context is not free.

The phases even cost different amounts

The split is not only about speed, it shows up on the bill. Most hosted APIs price input tokens and output tokens separately, and output tokens are usually several times more expensive than input tokens. That is the prefill-versus-decode story in dollars: input tokens are handled in the cheap, parallel prefill pass, while each output token is a full sequential decode step that reloads the weights.

So when you trim a prompt you are mostly buying latency and a little cost, and when you cap the output length you are mostly buying cost and a little latency. Knowing which phase a token lives in tells you which budget it spends. The Serving & Economics course turns this into concrete cost-per-token math.

Quick check