Cognitive

Part 3 · Language Models

Sampling and Decoding

A model scores every possible next token, and a sampling method chooses one.

3 min readBuilds on Probability and Uncertainty, Large Language Models

Temperature sharpens or flattens the probabilities over next tokens.

Key concepts

  • The model outputs a score for every token in its vocabulary. Softmax turns the scores into probabilities.
  • Temperature sharpens or flattens the distribution.
  • Top-k and top-p limit choices to likely candidates.
  • Structured output restricts tokens so the result follows a required format.
  • Caching, batching, and speculative decoding make generation faster and cheaper.

This interactive figure requires JavaScript.

Figure 18.1. Next-token probabilities after "The capital of France is", computed with softmax at temperature T. The logits are illustrative.
Sampling chooses one token, then the process repeats.

Formal view and evidence

  • Established. Temperature transforms logits zi with pi=exp(zi/T)∑jexp(zj/T). As T→0 the choice approaches greedy decoding, and as T→∞ it approaches uniform. Speculative decoding uses a smaller draft model while preserving the target distribution through rejection sampling.
  • Debated. The best truncation strategy across tasks and temperatures.
  • Open problem. Reliable, low-overhead watermarking of generated text that withstands paraphrasing.

Common misconception

Common belief

"Temperature 0 always gives the same answer."

In practice

It reduces sampling randomness, but hardware, batching, and model updates can still change the result.

Key takeaways

  • Softmax turns model scores into probabilities.
  • Temperature, top-k, and top-p shape which tokens can be chosen.
  • Different sampling settings suit different tasks.