Training a model means making its loss small. Gradient descent does it by repeatedly stepping downhill, and the step size and choice of optimizer shape the path it takes.
Imagine standing blindfolded on hilly ground, trying to reach the lowest point. You can feel the slope under your feet, so you take a small step downhill, feel again, and repeat. That is gradient descent.
In machine learning the ground is a loss surface: a function that measures how wrong your model is, over axes that are the model's parameters. Training means finding parameters where the loss is as small as possible.
With two parameters, a loss surface reads like a topographic map, with contour lines joining points of equal loss. Where a walker starts matters. Some reach the global minimum; others settle into a local minimum with a deeper valley past the ridge.
Each walker stops where the ground is flat. The update rule sees only the local slope, so it cannot tell a local minimum from the global one.
The gradient gives the direction of a step but not its length. That length is the learning rate η. Too small and the walk takes hundreds of steps; too large and it overshoots, bouncing across the valley or flying off.
How large is too large depends on how sharply the loss bends. Near a minimum the loss is a bowl, and along each of its principal directions a step multiplies the distance to the bottom by 1 − ηλ, with λ the curvature there. That shrinks the distance only while η < 2/λ, so the steepest direction sets the limit. Figure 2's curve has curvature 1 at the bottom and about half that up its walls, so rates between 2 and 4 bounce forever: unstable at the bottom, stable up the walls.
The inputs in Figure 3 share an offset of 1, which makes one direction very steep: all parameters moving together. Each parameter alone could take a rate above 0.4; together they diverge past 0.046. Centring the inputs raises the limit about tenfold.
In a long, narrow valley the steep walls cap the learning rate, and at that rate progress along the shallow floor is slow. Momentum keeps a fraction β of the previous step as a velocity: across the valley the gradient flips sign and the pushes partly cancel, along the floor they add up. Adam steps each parameter by a running average of its gradient divided by a running root-mean-square of it, so every parameter moves at a pace set by the learning rate rather than by its own steepness.
How much momentum buys depends on the valley's condition number κ, its steepest curvature over its shallowest. At its best rate gradient descent shrinks the error by (κ − 1)/(κ + 1) per step, so its step count grows like κ. Tuned momentum shrinks it by (√κ − 1)/(√κ + 1), and its count grows like √κ.
Everything above computes the gradient from the entire dataset. Real datasets are large, so in practice it is estimated from a small random subset, a mini-batch, which makes the method stochastic gradient descent (SGD). Figure 6 fits a line, ŷ = w x + b, to 64 simulated data points, so the parameters are the slope w and the intercept b.
Near the minimum the noisy steps never settle. The walker keeps wandering at a distance set by the batch size and the learning rate, and quadrupling the batch only about halves it.
The noise can also help, by bouncing the optimizer out of a shallow local minimum. Keskar et al. (2017) found that small-batch training tends to land in flatter minima that generalize better. Whether flatness explains it is debated: Dinh et al. (2017) showed that a network can be reparametrized to make a minimum arbitrarily sharp without changing what it computes.
Every step above needs the gradient with respect to each parameter. For the line in Figure 6 that is two partial derivatives. A neural network has many layers of weights, and Part 2 shows how backpropagation computes the gradient for all of them.