Part 2 · Neural Networks
Gradient Descent
Training a model means searching for parameters that minimize a loss. Gradient descent does this by repeatedly moving against the local slope.
The loss landscape
A model's behavior is determined by its parameters . The loss is a single number that measures how poorly those parameters perform on the training data. Every setting of the parameters has a loss, so training can be pictured as moving across a surface whose height is the loss.
A real network has millions or billions of parameters, so this surface cannot be seen directly. The optimizer only has local information: the value and the slope at its current position.
The update rule
The gradient points in the direction of steepest increase. Gradient descent takes a step in the opposite direction, scaled by the learning rate :
The learning rate is usually the most important setting to tune. If it is too small, progress is slow. If it is too large, the iterates overshoot the minimum, oscillate, or diverge.
This interactive figure requires JavaScript.
Established. If is convex and -smooth, then with the suboptimality after steps is bounded by:
Neural network losses are non-convex, so this guarantee does not carry over to them.
Stochastic gradients
Computing the gradient over an entire dataset is expensive. Stochastic gradient descent estimates it from a small random mini-batch . With uniform sampling, the estimate is unbiased:
Each step is noisier, but many more steps can be taken for the same amount of computation.
Momentum and adaptive methods
Momentum accumulates a decaying sum of past gradients. This damps oscillation across narrow valleys and builds speed along consistent directions:
Adam combines momentum with a separate adaptive step size for each parameter, based on running estimates of the first and second moments of the gradient.
This interactive figure requires JavaScript.
Open questions
Debated. Flat minima are often linked to better generalization, but sharpness depends on how a network is parameterized, so simple flatness measures can mislead.
Open problem. Why large non-convex networks train so reliably is not fully understood. In high dimensions, saddle points appear to be a more prominent obstacle than poor local minima.
Common misconception
"Training finds the single best set of parameters."
Training finds a good low region of the loss surface. Different runs can reach different solutions that perform similarly well.
Key takeaways
- Training repeatedly moves parameters against the gradient of the loss.
- The learning rate controls the balance between speed and stability.
- Mini-batches, momentum, and adaptive methods make training practical at scale.