Cognitive

Part 1 · Foundations

Machine Learning Paradigms

Machine learning finds patterns in examples so a computer can handle cases nobody explicitly programmed.

3 min readBuilds on Data Representation, Probability and Uncertainty

Unsupervised learning discovers structure without labels.

Key concepts

  • Supervised learning: examples come with correct answers.
  • Unsupervised learning: the machine finds structure without answers.
  • Reinforcement learning: the machine learns from rewards and penalties (Chapter 12).
  • Self-supervised learning: the data supplies its own practice questions, such as hiding the next word and predicting it. This is central to language models.
  • Training adjusts the model. Inference uses the trained model.

The training loop shared by most supervised learning systems.

Figure 5.1. The training loop shared by most supervised learning systems.
Supervised learning fits a decision boundary to labeled data.

Formal view and evidence

  • Established. Supervised learning is commonly framed as empirical risk minimization, f^=arg minf∈ℱ1n∑i=1nℓ(f(xi),yi). PAC theory bounds the gap between training and true risk in terms of capacity and sample size. No-free-lunch theorems show no learner is best across all possible problems, so inductive bias is unavoidable.
  • Debated. Classical capacity measures poorly explain overparameterized deep networks (Chapter 9).
  • Open problem. Learning efficiently from few examples, as humans often do.

Common misconception

Common belief

"Machine learning programs write down rules we can read."

In practice

Most models store learned numbers, not readable rules.

Key takeaways

  • Learning paradigms differ mainly in the feedback they receive.
  • Supervised learning commonly minimizes average loss on examples.
  • No learner is best for every possible problem, so assumptions matter.

Explore next