Part 3 · Language Models
Large Language Models
A large language model learns to predict the next token so well that it can explain, translate, code, and converse.
Key concepts
- Pre-training teaches next-token prediction across enormous amounts of text and code.
- Knowledge is stored in the model's weights, not in a database of documents.
- The model's knowledge stops at a training cutoff, unless tools provide fresher information.
- More data, compute, and parameters have generally improved capability, with rising costs.
- Open-weight models publish their weights. Closed models are accessed through a product or API.
Pre-training repeats one objective at enormous scale: predict the next token.
Formal view and evidence
- Established. Pre-training minimizes the average negative log-likelihood . Empirical scaling laws relate loss to parameters, data, and compute. Chinchilla found earlier large models were undertrained, and that parameters and tokens should scale roughly together for a fixed compute budget.
- Debated. Whether apparently sudden "emergent" abilities are real or partly an artifact of metrics. Whether statistical language modeling can amount to understanding is also contested.
- Open problem. How far scaling continues, and how to attribute behavior to specific training data.
Common misconception
Common belief
"The model looks up answers in a giant database."
In practice
It generates from learned weights, which is why it can sound fluent and still be wrong.
Key takeaways
- Pre-training minimizes error when predicting the next token.
- Knowledge is stored in weights rather than in a document database.
- Capability has improved with scale, but scale also raises cost.