Cognitive

Part 3 · Language Models

The Transformer

A Transformer lets every word look at every other word and decide which ones matter most.

3 min readBuilds on Recurrent Networks and Sequences, Embeddings

Each token computes how strongly to attend to every other token.

Key concepts

  • Self-attention lets each token gather information from other tokens.
  • Each token creates a query, a key, and a value.
  • Multiple heads learn different relationship patterns in parallel.
  • Positional information preserves word order.
  • A Transformer reads context in parallel, but usually generates one token at a time.
  • The context window limits how many tokens it can consider at once.

This interactive figure requires JavaScript.

Figure 15.1. Select a token to see which other tokens it attends to. Changing the final word changes what "it" refers to. The weights are illustrative and do not come from a trained model.
Layers of attention and feed-forward networks refine each token's representation.

Formal view and evidence

  • Established. Scaled dot-product attention is Attention(Q,K,V)=softmax(QK⊤dk)V. Decoder masks prevent access to future tokens during training. Naive attention costs O(n2) in sequence length.
  • Debated. Whether attention weights explain model decisions. Attention is informative, but it is not by itself a faithful explanation.
  • Open problem. The most efficient architecture for very long context, and how models use information placed in different context positions.

Common misconception

Common belief

"Attention means the model understands like a person."

In practice

Attention is a learned weighting mechanism. It does not by itself prove comprehension.

Key takeaways

  • Self-attention lets each token draw information from all the others.
  • Positional information preserves word order.
  • Attention weights are informative, but they are not a complete explanation.

Explore next