Cognitive

Part 3 · Language Models

Tokenization

AI does not read letters or whole words. It reads chunks called tokens.

3 min readBuilds on Data Representation

Text becomes a sequence of tokens, each mapped to an integer.

Key concepts

  • A tokenizer splits text into pieces and maps each to an ID from a fixed vocabulary.
  • In English, a token averages roughly ¾ of a word, but this varies widely.
  • Spaces, punctuation, capitals, and rare words change how text splits.
  • Models see tokens rather than letters, which helps explain why counting letters can be difficult.
  • Usage limits and pricing are usually measured in tokens.

How text becomes the integer IDs that a language model receives.

Figure 13.1. How text becomes the integer IDs that a language model receives.
Some languages need more tokens to express the same content.

Formal view and evidence

  • Established. Byte-pair encoding, adapted to neural translation by Sennrich et al., repeatedly merges frequent symbol pairs. Tokenization is a modeling decision that affects cost and performance.
  • Established. Tokenizers can require substantially more tokens for the same content in some languages, creating unequal cost and latency.
  • Open problem. Tokenization that serves all languages fairly, and models that operate well directly on bytes or characters.

Common misconception

Common belief

"One token equals one word."

In practice

A token can be a word, part of a word, punctuation, or a space-prefixed fragment.

Key takeaways

  • Tokenizers convert text into integer IDs from a fixed vocabulary.
  • Token count affects cost, speed, and how much fits in context.
  • The same content can require more tokens in some languages.

Explore next