Cognitive

Part 3 · Language Models

Interpretability

Interpretability researchers open up models to discover what their internal numbers actually represent.

3 min readBuilds on The Transformer, Large Language Models

Researchers look for internal features that respond to related concepts.

Key concepts

  • A feature is a pattern inside a model that represents something meaningful.
  • Probes test what information can be read from activations.
  • Activation patching swaps internal values to test what causes a behavior.
  • Sparse autoencoders separate mixed signals into more readable features.
  • Circuits are groups of features and connections that work together.

Activation patching tests whether an internal signal causes a behavior.

Figure 19.1. Activation patching tests whether an internal signal causes a behavior.
Circuits are groups of features that work together to produce a behavior.

Formal view and evidence

  • Established. Neurons are often polysemantic. The superposition hypothesis proposes that networks represent more features than dimensions by using nearly orthogonal directions, and toy models demonstrate this behavior.
  • Debated. Whether features found by sparse autoencoders are the model's true units of computation. Saliency methods can also fail basic sanity checks.
  • Open problem. Complete, scalable, and faithful explanations of frontier models. A model's stated reasoning may also differ from the computation that produced its answer.

Common misconception

Common belief

"If a model explains its reasoning, that must be how it decided."

In practice

Its explanation is generated text and may not faithfully reflect the internal process.

Key takeaways

  • Researchers study features and circuits inside trained models.
  • Causal tests provide stronger evidence than correlation alone.
  • A model's stated explanation may not match its internal computation.

Explore next