Cognitive

Part 2 · Neural Networks

Generalization and Overfitting

The real test of learning is how well a model handles examples it has never seen.

3 min readBuilds on Gradient Descent, Backpropagation

The widening gap between training and validation error signals overfitting.

Key concepts

  • Overfitting: great on training data, poor on new data.
  • Underfitting: too simple to capture the pattern.
  • Regularization discourages memorization. Examples include dropout, weight decay, data augmentation, and early stopping.
  • Watching validation loss reveals when memorization begins.

Model selection relies on data the training process did not see.

Figure 9.1. Model selection relies on data the training process did not see.
Double descent: test error can fall again after the interpolation threshold.

Formal view and evidence

  • Established. For squared error, expected error decomposes into bias², variance, and irreducible noise. Zhang et al. showed standard networks can fit randomly assigned labels, so capacity alone cannot explain their generalization on real data.
  • Debated. The mechanisms behind double descent, in which test error can fall again beyond the interpolation threshold, and implicit regularization from optimization.
  • Open problem. Tight, predictive generalization bounds for practical deep networks, and a full account of grokking, where generalization appears long after training accuracy saturates.

Common misconception

Common belief

"Bigger models always overfit more."

In practice

Classical intuition suggests this, but very large networks can generalize well.

Key takeaways

  • The goal of training is low error on data the model has not seen.
  • Regularization and validation help detect and limit overfitting.
  • Large networks can generalize well in ways classical theory does not fully explain.

Explore next