Part 1 · Foundations
Machine Learning Paradigms
Machine learning finds patterns in examples so a computer can handle cases nobody explicitly programmed.
Key concepts
- Supervised learning: examples come with correct answers.
- Unsupervised learning: the machine finds structure without answers.
- Reinforcement learning: the machine learns from rewards and penalties (Chapter 12).
- Self-supervised learning: the data supplies its own practice questions, such as hiding the next word and predicting it. This is central to language models.
- Training adjusts the model. Inference uses the trained model.
The training loop shared by most supervised learning systems.
Formal view and evidence
- Established. Supervised learning is commonly framed as empirical risk minimization, . PAC theory bounds the gap between training and true risk in terms of capacity and sample size. No-free-lunch theorems show no learner is best across all possible problems, so inductive bias is unavoidable.
- Debated. Classical capacity measures poorly explain overparameterized deep networks (Chapter 9).
- Open problem. Learning efficiently from few examples, as humans often do.
Common misconception
Common belief
"Machine learning programs write down rules we can read."
In practice
Most models store learned numbers, not readable rules.
Key takeaways
- Learning paradigms differ mainly in the feedback they receive.
- Supervised learning commonly minimizes average loss on examples.
- No learner is best for every possible problem, so assumptions matter.