Transformer architecture: the LLM block
A modern LLM is mostly one transformer block repeated many times. Once you understand the block, the giant model stops looking like magic and starts looking like a deep stack of the same few operations.
A transformer block alternates between mixing information across tokens with attention and transforming each token's representation with an MLP. Residual paths and normalization keep the deep stack trainable.
From token IDs to vectors
The tokenizer gives the model token IDs. The first learned table in the model turns each ID into a vector, called a token embedding. If the sequence has 200 tokens, the model now has 200 vectors, one per position.
The model also needs position information. Without it, "dog bites man" and "man bites dog" would contain the same token vectors in a different order, but attention by itself does not know order. Early GPT-style models added fixed sinusoidal vectors to each position. That worked, but it does not extrapolate well past the length the model saw during training.
The repeating block
After embeddings, the sequence flows through a stack of transformer blocks. A simplified decoder-only LLM block looks like this:
Attention mixes across tokens
The attention sublayer is where tokens exchange information. The representation at one position can pull from previous positions, so a pronoun can connect to a noun, a function call can connect to its name, and the last token can summarize what the prompt asked for.
In a decoder-only LLM, causal masking keeps this left-to-right. Each token can use the past and itself, not the future.
The MLP works position by position
The feed-forward network, often called the MLP, is different. It does not mix tokens with each other. It applies the same learned transformation to each token position independently. You can think of it as the part that refines what each position knows after attention has gathered context.
Many of the model's parameters live in these MLP layers. Attention gets the fame because it routes information, but the MLPs hold a lot of the model's learned transformations.
Residuals and normalization keep the stack alive
Deep networks are hard to train if every layer must rewrite everything from scratch. Residual connections give each block a shortcut: the block adds a change to the existing representation instead of replacing it completely. That makes it easier for information and gradients to move through dozens of layers.
Normalization keeps vector values in a stable range as they pass through the stack. GPT-2 used post-norm (normalize after the sublayer). Most modern decoder-only LLMs use pre-norm (normalize before attention and before the MLP). Pre-norm tends to train more reliably at depth. You will see RMSNorm more often than classic LayerNorm in recent open models like Llama: same stabilizing job, slightly cheaper compute.
When people talk about "model size," they usually mean parameter count across embeddings, attention projections, MLPs, and output layers. More parameters can store more patterns, but serving cost also rises. Bigger models need more memory, more bandwidth, and often more aggressive batching or quantization to run cheaply.
What modern LLMs actually use
The block diagram above is the right mental model. Production models from the last few years swap several internals while keeping the same rhythm: attention mixes across tokens, MLP transforms each position, residuals carry information forward.
- RoPE (rotary position embeddings). Instead of adding a position vector, RoPE rotates query and key vectors based on position. Relative distance shows up in the attention score. Models like Llama 3 and Qwen use it. It extrapolates to longer contexts better than absolute sinusoidal encodings, though very long windows still need training or finetuning tricks (covered in the context lesson).
- GQA / MQA. Standard multi-head attention stores separate key and value vectors per head. Grouped-query attention (GQA) shares one key/value set across several query heads. Multi-query attention (MQA) goes further: one key/value set for all heads. Fewer KV tensors means a smaller KV cache at inference, which matters when you serve long contexts to many users.
- SwiGLU MLPs. Llama-class models often replace the simple two-layer MLP with a gated SwiGLU feed-forward. More parameters in the MLP, better quality per FLOP in practice.
- Decoder-only stack. Translation-era transformers had separate encoder and decoder towers. Chat LLMs are decoder-only: one causal stack, no cross-attention to a source encoder. Simpler to train for "continue this text."
Some models like Mixtral and DeepSeek-MoE do not run every parameter on every token. They keep several "expert" MLPs and a router that picks a small subset per token. Active parameters per forward pass stay lower than total parameter count, so you can get large-model quality with smaller per-token compute. Serving is trickier: you need enough GPU memory for all experts, and routing can imbalance load across devices. Worth knowing when comparing a 8x7B MoE to a dense 70B on cost spreadsheets.
Bigger models learn faster, but data and compute matter too. The Chinchilla work showed many models were undertrained: for a fixed compute budget, a smaller model trained on more tokens often beats a larger model trained on fewer. In practice that is why a well-trained 7B can feel surprisingly close to an older 13B, and why a 70B class model behaves differently than 7B on hard reasoning, not just because of a bigger number on the spec sheet.
The best single visual walkthrough of the original encoder-decoder transformer. Read it after this lesson to see the data flow through one block.
- Take from it
- How embeddings, attention, and the feed-forward layer connect in a full forward pass, with diagrams you can stare at until it clicks.
- It skips
- Decoder-only LLMs, RoPE, GQA, sampling, serving, and everything after the 2017 paper. Use this course for the modern stack.
Where the output comes from
After the last transformer block, the model has a final vector for each token position. For text generation, we care about the final position: given everything so far, what token should come next? A final projection maps that vector to a score for every token in the vocabulary. Those scores become probabilities in the generation step.
Checkpoint
You're ready for the next lesson if you can answer these from memory:
- What happens between token IDs and the first transformer block?
- What does attention do that the MLP does not?
- Why do transformer blocks use residual connections?
- Why do modern LLMs use RoPE instead of absolute positional encodings?
- What does GQA save at inference time?
Quick check
- Self-attention
- The MLP only
- Layer normalization
- Because token IDs already include all word order
- Because order changes meaning, and attention alone has no built-in order
- Because the API bills tokens by position
- Deleting unimportant tokens
- Letting information flow around a sublayer and adding an update
- Changing token IDs back into text