How LLMs generate text
An LLM answer is not produced in one shot. The model predicts one token, appends it to the context, then predicts the next one. The whole response is that loop repeated until something tells it to stop.
LLMs generate by repeatedly turning the current context into a probability distribution over the next token, choosing one token, adding it to the context, and running the model again.
Training teaches the next-token game
During pretraining, the model sees a huge amount of text and learns a simple task: given the previous tokens, predict the next token. It does this over and over at every position. That task sounds small, but solving it well forces the model to learn grammar, facts, style, code patterns, reasoning traces, and many other regularities in text.
At inference time, we use the same skill differently. We give the model a prompt, ask for the next token, append that token, then ask again.
From logits to probabilities
The model's final layer outputs one score for every token in the vocabulary. These raw scores are called logits. A softmax turns them into probabilities that add up to 1. High-probability tokens are the model's best guesses for what should come next.
Greedy decoding vs sampling
The simplest strategy is greedy decoding: always pick the highest-probability token. This is deterministic and often useful for narrow tasks, but it can make text dull, repetitive, or brittle. Sampling instead treats the probabilities as a distribution and randomly chooses from them. More likely tokens are picked more often, but lower-probability tokens still have a chance.
That controlled randomness is why the same prompt can produce different good answers. The model is not "changing its mind." The sampling process is choosing different paths through the probability tree.
Temperature, top-k, and top-p
Temperature reshapes the probability distribution before sampling. Divide logits by temperature before softmax. Low temperature (0.1–0.3) sharpens the distribution: the top token wins more often. Good for extraction, classification, code edits, and JSON where you want the same input to give the same shape of answer. High temperature (0.8–1.2) flattens it: unlikely tokens get a real chance. Good for brainstorming and creative drafts. Temperature does not make the model smarter. It changes how adventurous the sampler is.
Top-k keeps only the k highest-probability tokens and zeroes the rest before sampling. Simple guardrail against sampling a bizarre tail token. Typical values: 40–100 for chat, lower for structured tasks.
Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to p (often 0.9–0.95), then samples inside that set. It adapts to how peaked the distribution is: when one token is obviously right, the nucleus is tiny; when several are plausible, it stays wider.
For structured output: low temperature, low top-p, consider greedy or top-k=1. For open-ended chat: temperature around 0.7, top-p around 0.9. For evals and regression tests: temperature 0 (greedy) so runs are reproducible. Tune on your task with a golden set, not by vibes.
Here are temperature and top-p on one distribution. Drag temperature to sharpen or flatten the bars, drag top-p to cut the tail, then sample to draw a real token from whatever survives. Notice how a low top-p makes the long tail unreachable no matter how high you push temperature.
Logprobs, stop sequences, and tool tokens
Most APIs can return logprobs: the log probability the model assigned to each candidate token at a step. You use these for evals (did the model assign high probability to the right answer?), confidence thresholds (refuse when the top token is only barely ahead), and debugging sampling bugs. They are not calibrated probabilities of truth, but they are useful signals.
Stop sequences tell the server to halt generation when a string appears: </json>, a newline after a code block, or a closing XML tag. Without them the model may ramble past the format you wanted.
Tool-calling models add special tokens or structured spans for function names and arguments. The generation loop is the same, but stop conditions and parsers must match the model's training format. That is why "just ask for JSON" often fails without schema enforcement and validation downstream.
Stopping is part of generation
The loop needs a stop condition. It can stop when the model emits a special end-of-sequence token, when it hits a maximum output length, or when the serving layer sees a configured stop sequence. If your product expects JSON, SQL, Markdown, or a tool call, stop conditions and validation matter as much as the prompt.
Streaming does not make the model compute the answer all at once. It sends tokens to the client as they are generated. This improves perceived latency, but the server still runs the decode loop token by token. Long answers are slow because every output token requires another model step.
Go deeper (optional)
Karpathy builds a minimal GPT from scratch in one long video. It connects tokenization, causal attention, the transformer block, and the generation loop in code. Watch it after lessons 01–04 if you want the pieces to snap together.
Checkpoint
You're ready for the next lesson if you can answer these from memory:
- Why is LLM generation a loop?
- What are logits, and how do they become probabilities?
- When would you prefer greedy decoding over sampling?
- What does temperature change, and what does it not change?
- When would you use top-k vs top-p?
- What are logprobs useful for in production?
- Why does streaming help perceived latency but not remove decode cost?
Quick check
- A raw score for a possible next token
- The hidden prompt stored by the model
- The numeric ID of the chosen token
- It makes the model more factually correct
- It makes sampling more varied and less locked to the top token
- It reduces the number of input tokens
- Each output token requires another model step
- They use rarer words
- The tokenizer has to relearn the vocabulary