Part 3 · Language Models
Sampling and Decoding
A model scores every possible next token, and a sampling method chooses one.
Key concepts
- The model outputs a score for every token in its vocabulary. Softmax turns the scores into probabilities.
- Temperature sharpens or flattens the distribution.
- Top-k and top-p limit choices to likely candidates.
- Structured output restricts tokens so the result follows a required format.
- Caching, batching, and speculative decoding make generation faster and cheaper.
This interactive figure requires JavaScript.
Formal view and evidence
- Established. Temperature transforms logits with . As the choice approaches greedy decoding, and as it approaches uniform. Speculative decoding uses a smaller draft model while preserving the target distribution through rejection sampling.
- Debated. The best truncation strategy across tasks and temperatures.
- Open problem. Reliable, low-overhead watermarking of generated text that withstands paraphrasing.
Common misconception
Common belief
"Temperature 0 always gives the same answer."
In practice
It reduces sampling randomness, but hardware, batching, and model updates can still change the result.
Key takeaways
- Softmax turns model scores into probabilities.
- Temperature, top-k, and top-p shape which tokens can be chosen.
- Different sampling settings suit different tasks.