How Gradient Descent Finds Minima

Imagine standing blindfolded on hilly terrain, trying to reach the lowest point. You can't see the landscape, but you can feel the slope under your feet: which way is downhill and how steep it is. Gradient descent repeats one move: take a small step downhill, feel the slope again, and step again.

In machine learning, the "terrain" is a loss surface: a function that measures how wrong your model is. The horizontal axes are the model's parameters. The vertical axis (or color, in a map view) is the loss. Training a model means finding parameters that make the loss as small as possible.

The update rule. At each step, we nudge the parameters a small distance η against the loss gradient.

The Loss Landscape

With two parameters, a loss surface reads like a topographic map: contour lines join points of equal loss, valleys are low loss and ridges are high. Where a walker starts matters. Some reach the global minimum; others settle into a local minimum, with a deeper valley past the ridge.

0 walkers

Click or tap the contour map to drop walkers, or press Random start

Figure 1. A 2D loss surface with multiple minima. Each walker follows the negative gradient from its starting point. Color encodes loss (viridis: bright = low, dark = high).

Each walker stops where the ground is flat. The update rule uses only the local slope, so it cannot distinguish a local minimum from the global one.

How Far to Step

The gradient tells you which direction to step, but not how far. That distance is the learning rate. Too small, and you need hundreds of steps to reach the minimum. Too large, and you overshoot the valley, bouncing across it or flying off entirely. In between, you reach the bottom in a handful of steps.

Small rate Medium rate Large rate

Learning rates:

Small 0.020

Medium 0.500

Large 2.500

Figure 2. Three gradient descent runs of 80 steps from the same start on the same one-minimum loss curve, one learning rate per panel. Each step moves the dot by the gradient times the learning rate, and the word over each panel is read off its run. At the defaults the small rate is still short of the minimum when the 80 steps run out (it needs about 300), the medium rate settles in 10, and the large rate bounces between the two walls without ever settling. Push the large rate past about 3.1 and the bounces grow wider than the chart, so the dot leaves it while still bouncing; from about 3.8 it diverges and flies off for good.

The best learning rate depends on the shape of the loss surface, which changes as you move through it. This curve bends twice as sharply at the bottom as far up its walls, which is why the large rate can bounce forever: its steps are too long to settle near the bottom but not long enough to throw the dot out. Modern optimizers adapt the step size during training for this reason.

Momentum and Adam

Plain gradient descent struggles in long, narrow valleys. The gradient points mostly across the steep walls instead of along the floor, and a learning rate large enough to make progress along the floor overshoots across it. The path zigzags from wall to wall and makes slow progress toward the minimum.

Momentum adds a velocity term that keeps a fraction β of the previous step. Across the valley the gradient flips sign from step to step, so those pushes partly cancel in the velocity; along the floor they point the same way and add up. Too much momentum brings the oscillation back: at the learning rate used below, β = 0.9 is still swinging across the valley after 20 steps. Adam keeps running averages of each parameter's gradient (the first moment, its mean) and of its squared gradient (the second raw moment, the mean squared gradient), and divides each parameter's step by the square root of the second, so every parameter moves at a pace set by the learning rate rather than by how steep its own direction is.

Momentum carries a running velocity v through the gradient field; β sets how much of the previous step to keep.

Gradient descent (full gradient, rate 0.158)

Momentum (rate 0.158, β = 0.5)

Adam (rate 0.25)

Click or tap the map (or focus it and use the arrow keys) to set the starting point, or use Race to compare all three

Figure 3. Three optimizers, 100 steps each, from the same start (open circle) on a quadratic valley tilted away from both parameter axes; the minimum is the dark cross at (1, 1). From the default start, gradient descent zigzags from wall to wall for about twenty steps and is still far down the valley at the end. Momentum, at the same learning rate, stops zigzagging within about five steps and ends much closer to the minimum. Adam heads off diagonally, swings below the floor in wide loops and then wobbles along it: its scaling works per parameter, and the steep and shallow directions of this valley each mix both parameters.

Adam is not always the best choice, as this valley shows. On some problems, well-tuned stochastic gradient descent with momentum (the mini-batch version described next) also generalizes better than adaptive methods, and the choice is often made by trying several.

Learning from Mini-Batches

Everything above assumes you compute the gradient using your entire dataset. Real datasets are large, so instead you estimate the gradient from a small random subset, a mini-batch. The estimate is noisy: a single sample points roughly the right way but wanders, and a larger batch is cleaner but costs more. Using mini-batch estimates is what makes it stochastic gradient descent (SGD). Figure 4 fits a straight line, ŷ = w x + b, to 64 simulated data points, so the two parameters are the slope w and the intercept b.

Mean squared error over a mini-batch B: the loss compares each true value against the model's prediction.

Batch size 1 / 64

Distance from the minimum after 50 steps -

Loss contours

Walker path (w, b)

Least-squares minimum

Figure 4. The mean squared error of the line over (w, b), for 64 simulated points drawn around w = 0.7, b = 1.2 with noise; shading deepens as the loss rises. The green dot is the least-squares fit to this sample, the true minimum of the plotted loss, which the noise has moved slightly off the generating line. From the open circle the walker takes fifty steps at learning rate 0.1, each on a fresh random mini-batch. At batch size 1 it zigzags into the neighborhood of the minimum (in about fifteen steps, averaged over many runs) and then keeps jumping around it, so the final distance changes with every Resample and averages about 0.4. At 64 every batch is the whole dataset, so it runs almost straight in and stops on the minimum.

The noise can also help, by bouncing the optimizer out of a shallow local minimum into a better one. Keskar et al. (2017) found that small-batch training tends to land in flatter minima that generalized better than the sharp minima large batches found, and attributed it to the gradient noise. Whether flatness itself explains generalization is debated: Dinh et al. (2017) showed that a network can be reparametrized to make a minimum arbitrarily sharp without changing what it computes.

Every step above needs the gradient of the loss with respect to each parameter. For the line in Figure 4 that is two partial derivatives. A neural network has many layers of weights, and Part 2 shows how backpropagation computes the gradient for all of them.