Gradient descent needs the loss gradient for every weight in a network. Backpropagation gets them all in one backward pass, which also explains why deep sigmoid networks stall.
A neural network starts with random weights. Training adjusts them by gradient descent from the previous article, which needs the gradient of the loss with respect to every one. Backpropagation computes them all by tracing the error at the output backward through every connection.
Take a small network: two inputs, three hidden nodes, one output. Each node computes a weighted sum of its inputs, then squashes the result through a sigmoid function to produce an activation between 0 and 1. Positive weights pass the signal along and negative weights flip it.
Figure 1 slows the forward pass down, from the inputs to the prediction.
Figure 1. The forward pass through a 2-3-1 network, one step at a time. From step 2, edge width shows weight size and color its sign (blue positive, red negative); from step 3, node shading shows the activation. The panels follow hidden node h2 and the output a.
The loss is a single number for the network's mistake. It is zero when the prediction matches the target and grows as they diverge.
The loss gives the total error but not which weights caused it. The backward pass works that out.
For each weight, the backward pass measures how much the loss would change if that weight were slightly different. That sensitivity is the gradient.
The chain rule makes this tractable. At each node, the incoming gradient gets multiplied by the local gradient, which is the derivative of whatever that node computed. For a sigmoid node, the local gradient is the slope of the sigmoid curve at the current operating point. Gradients compose by multiplication as they flow backward through the network.
Figure 2. The backward pass, at the inputs 0.60 and 0.40 with target 0.85. Red pulses carry gradients from the output back toward the inputs. Edge labels show each weight's gradient dL/dw; the label above each hidden node is dL/dz, the gradient at that node's weighted sum.
Each node only needs its own local derivative. Multiply that by the gradient arriving from downstream and you have the gradient to pass upstream.
The definition of a gradient gives a check on the backward pass: nudge one weight by a small ε, rerun the forward pass, and divide the change in the loss by ε. Large nudges feel the curve bend and tiny ones drown in rounding error; nudging both ways (a two-sided difference) cancels most of the bending.
Figure 3. Relative error between backprop's gradient and the nudge estimate for one weight of Figure 2's network. Both errors fall, bottom out and rise again as rounding takes over: one-sided best near √u, two-sided near ∛u, where u is the rounding unit.
Nudging costs a forward pass for every weight, two for the two-sided version. Carrying derivatives forward alongside the values (forward-mode differentiation) also takes one pass per weight. The backward pass starts from the single loss and reuses each node's gradient for everything upstream, so one pass delivers them all.
Figure 4. Multiplications for one full gradient of a sigmoid network with 4 inputs and three hidden layers, counted as the code runs, as the width grows. Backpropagation stays near three forward passes; the other two grow in proportion to the parameter count.
Once every weight has a gradient, each one moves against it in proportion to its size: a weight the loss is sensitive to gets a big adjustment, one it barely notices hardly moves. The learning rate η sets the step size.
One forward pass, backward pass and update make one step of learning. Below, the network repeats that cycle to learn XOR, a 2D pattern no single straight line can separate.
Figure 5. Training on the XOR pattern for up to 200 epochs. From left to right (top to bottom on a phone): the loss curve, the network with its weights updating, and the 2D decision boundary evolving as the network learns (blue where the output is near 1, red near 0).
Figure 2 multiplied the gradient by the sigmoid's slope σ(z)(1−σ(z)) at every node on the way back. That slope is at most 0.25, at z = 0. The gradient reaching layer k picks up one factor of weights times slopes for every layer above it, and with weights of ordinary size each factor is around a quarter: ten layers give about one millionth.
Figure 6 measures this in a stack of 16-unit layers with random weights and a linear readout, backpropagating a squared-error loss over a batch of 64 random inputs and targets.
Figure 6. Root-mean-square entry of the batch-loss gradient with respect to each layer's weights, from layer 1 (next to the input) to the last hidden layer. Simulated: weights, inputs and targets are random draws, with weights from a normal distribution with standard deviation scale/√16. The dashed gray line is the other activation function with the same settings.
With sigmoid units at weight scale 1 the gradient falls by a factor of four or five per layer going back, so gradient descent barely moves the early layers. A larger weight scale enlarges the factor from the weights but pushes the weighted sums onto the flat parts of the sigmoid. Near scale 8 the shrink falls to about 1.4 per layer on a typical draw, and for some draws the gradient does not shrink at all, but by then about 60% of the units sit within 0.05 of 0 or 1, where they barely respond.
ReLU has slope exactly 1 wherever a unit is active, and its profile stays roughly flat at every weight scale: without biases, scaling every weight scales each layer's signal alike. The scale sets the overall size instead. At √2, the scaling proposed by He et al., 2015, going from 2 layers to 20 usually changes layer 1's gradient by less than thirtyfold either way; at scale 1 it shrinks it a hundredfold or more, and at scale 2 grows it by about as much. This is one reason deep networks moved from sigmoid to ReLU units.
Forward pass, backward pass and update are all a network needs to learn from examples. Part 3 uses them to train an attention layer, the building block of transformers, on the page.