Part 3 · Language Models
Interpretability
Interpretability researchers open up models to discover what their internal numbers actually represent.
Key concepts
- A feature is a pattern inside a model that represents something meaningful.
- Probes test what information can be read from activations.
- Activation patching swaps internal values to test what causes a behavior.
- Sparse autoencoders separate mixed signals into more readable features.
- Circuits are groups of features and connections that work together.
Activation patching tests whether an internal signal causes a behavior.
Formal view and evidence
- Established. Neurons are often polysemantic. The superposition hypothesis proposes that networks represent more features than dimensions by using nearly orthogonal directions, and toy models demonstrate this behavior.
- Debated. Whether features found by sparse autoencoders are the model's true units of computation. Saliency methods can also fail basic sanity checks.
- Open problem. Complete, scalable, and faithful explanations of frontier models. A model's stated reasoning may also differ from the computation that produced its answer.
Common misconception
Common belief
"If a model explains its reasoning, that must be how it decided."
In practice
Its explanation is generated text and may not faithfully reflect the internal process.
Key takeaways
- Researchers study features and circuits inside trained models.
- Causal tests provide stronger evidence than correlation alone.
- A model's stated explanation may not match its internal computation.