Part 3 · Language Models
The Transformer
A Transformer lets every word look at every other word and decide which ones matter most.
Key concepts
- Self-attention lets each token gather information from other tokens.
- Each token creates a query, a key, and a value.
- Multiple heads learn different relationship patterns in parallel.
- Positional information preserves word order.
- A Transformer reads context in parallel, but usually generates one token at a time.
- The context window limits how many tokens it can consider at once.
This interactive figure requires JavaScript.
Formal view and evidence
- Established. Scaled dot-product attention is . Decoder masks prevent access to future tokens during training. Naive attention costs in sequence length.
- Debated. Whether attention weights explain model decisions. Attention is informative, but it is not by itself a faithful explanation.
- Open problem. The most efficient architecture for very long context, and how models use information placed in different context positions.
Common misconception
Common belief
"Attention means the model understands like a person."
In practice
Attention is a learned weighting mechanism. It does not by itself prove comprehension.
Key takeaways
- Self-attention lets each token draw information from all the others.
- Positional information preserves word order.
- Attention weights are informative, but they are not a complete explanation.