Attention: how tokens look at each other
A token's meaning depends on the tokens around it. Attention is the transformer mechanism that lets every token pull in the parts of the context that matter for the next representation.
Attention lets each token build a weighted summary of other tokens. It asks "which previous tokens matter to me right now?" and mixes information from them.
The same word can mean different things
Take the word bank. In "the river bank was muddy," it means land near water. In "the bank approved the loan," it means a financial institution. The token is the same, but the context changes its meaning. A useful language model needs every token representation to absorb nearby clues.
Older sequence models pushed information forward step by step. Transformers do something more direct: each token can look across the sequence and decide which tokens to read from. That is attention.
Queries, keys, and values
The standard attention explanation has three names: query, key, and value. The names sound more mysterious than the idea.
- Query: what this token is looking for.
- Key: what each other token offers as a match.
- Value: the information each token will contribute if it gets attended to.
The model compares a token's query with other tokens' keys. Better matches get higher weights. Then it takes a weighted mix of the values. That mixed vector becomes the information this token carries forward.
Scaled dot-product attention
A normal lookup grabs one exact item by key. Attention is softer: it compares against many keys, assigns a score to each, turns those scores into probabilities, then blends the values. If one previous token is clearly relevant, it dominates. If several tokens matter, the model can mix them.
The original transformer paper calls this scaled dot-product attention. For one head, the math is:
Attention(Q, K, V) = softmax( Q K^T / sqrt(d_k) ) VRead it left to right. Q is the matrix of queries (one row per token position). K is keys. The dot product Q K^T scores how well each query matches each key. Divide by sqrt(d_k) (the key dimension) so those scores stay in a stable range as models get wider. Softmax turns scores into weights that sum to 1. Multiply by V (values) to get the weighted mix.
You do not need to implement this by hand. You do need the picture: each token asks a question, every allowed token offers a key, the model scores the matches, then pulls a blend of values.
Click through the example below. Each row is one token's query; each column is a key it is allowed to read (causal mask blocks the upper triangle). Brighter cells mean higher weight.
Causal attention only looks backward
LLMs that generate text are usually trained with causal masking. When predicting the next token, position 5 can look at positions 1 through 5, but not position 6 or 7. During training, the model may process a whole sequence at once, but the mask hides the future so the task stays honest.
This is why generation can run left to right. The model builds each next token from the prompt and the tokens it already generated. It never gets to peek at the answer that has not been written yet.
Multi-head attention
One attention pattern is not enough. A token might need grammar from one place, factual context from another, and formatting clues from a third. Multi-head attention runs several smaller attention passes in parallel. Each head has its own Q, K, and V projections, so it can learn a different routing pattern. The head outputs are concatenated and projected back to the model width.
Think of it as several specialists looking at the same sentence at once. One head might track which noun a pronoun refers to. Another might focus on the token right before the cursor. Another might pick up delimiter tokens in a JSON blob. They do not have fixed human labels, but the split lets the block attend in more than one way per layer.
Do not over-interpret individual heads. Some learn visible behaviors, like copying names or tracking punctuation. Many are harder to explain. What matters is that multiple heads give the block several routes for moving information.
The paper that introduced the transformer. Worth skimming once you have the high-level picture from this lesson and the architecture lesson next.
- Take from it
- The scaled dot-product attention equation, multi-head attention, encoder/decoder layout, and why self-attention replaced recurrence for long sequences.
- It skips
- Decoder-only LLMs, RoPE, GQA, modern activations, serving cost, and everything after GPT-2 era design. Those are what the rest of this course covers.
Attention is powerful, but it scales badly with sequence length. A sequence with twice as many tokens has roughly four times as many token pairs to score during the full prompt pass. This is one reason long context is expensive, even before you ask the model to generate anything.
Checkpoint
You're ready for the next lesson if you can answer these from memory:
- What problem does attention solve for token representations?
- What are queries, keys, and values in plain English?
- Why does a causal language model mask future tokens?
- What is the scaled dot-product attention formula, in words if not symbols?
- Why does long context make attention more expensive?
- What does multi-head attention buy you over a single attention pass?
Quick check
- How strongly one token reads information from another token
- How common a token is in the training data
- How expensive a token is to process
- To hide private training examples
- To stop each position from looking at future tokens
- To ignore punctuation tokens
- It generates multiple final answers at once
- It gives the model several learned ways to route information
- It splits text into tokens more accurately