Gradient descent needs the loss gradient for every weight in a network. Backpropagation gets them all in one backward pass, which also explains why deep sigmoid networks stall.
A neural network starts with random weights, so its outputs have nothing to do with the task. Training adjusts the weights with gradient descent from the previous article, which needs the gradient of the loss with respect to every one of them. Backpropagation computes those gradients by tracing the error at the output backward through every connection, working out how sensitive the loss is to each weight.
Take a small network: two inputs, three hidden nodes, one output. Each node computes a weighted sum of its inputs, then squashes the result through a sigmoid function to produce an activation between 0 and 1. Positive weights pass the signal along and negative weights flip it.
Figure 1 slows the forward pass down, from the inputs to the point where the prediction meets the target.
Figure 1. The forward pass through a 2-3-1 network, one step at a time. From step 2 on, edge width encodes weight magnitude and color encodes sign (blue positive, red negative); from step 3, node shading shows the activation. The output node is labeled a, the prediction in the equations. The three panels follow the middle hidden node h2 and the output. Move the input sliders to change what flows through.
The loss is a single number for the network's mistake. It is zero when the prediction matches the target and grows as they diverge.
The loss gives the total error but not which weights caused it. The backward pass works that out.
For each weight, the backward pass measures how much the loss would change if that weight were slightly different. That sensitivity is the gradient.
The chain rule makes this tractable. At each node, the incoming gradient gets multiplied by the local gradient, which is the derivative of whatever that node computed. For a sigmoid node, the local gradient is the slope of the sigmoid curve at the current operating point. Gradients compose by multiplication as they flow backward through the network.
Figure 2. The backward pass, at the inputs 0.60 and 0.40 with target 0.85. Red pulses carry gradients from the output back toward the inputs. Edge labels show each weight's gradient dL/dw; the label above each hidden node is dL/dz, the gradient at that node's weighted sum.
Each node only needs its own local derivative. Multiply that by the gradient arriving from downstream and you have the gradient to pass upstream.
Once every weight has a gradient, each one shifts against it, in proportion to its size. A weight with a large gradient has a big effect on the loss and gets a big adjustment; a weight with a small gradient barely moves. The learning rate η sets the step size, with the same trade-offs as in gradient descent.
One forward pass, backward pass and update make one step of learning. Below, the network repeats that cycle to learn XOR, a 2D pattern no single straight line can separate.
Figure 3. Training on the XOR pattern for up to 200 epochs. From left to right (top to bottom on a phone): the loss curve, the network with its weights updating, and the 2D decision boundary evolving as the network learns (blue where the output is near 1, red near 0).
Figure 2 multiplied the gradient by the sigmoid's slope at every node on the way back. That slope is σ(z)(1−σ(z)). It is 0.25 at z = 0, its largest value, and smaller everywhere else. In a network with many layers, the gradient reaching layer k has passed through every layer above it, picking up one factor of weights times slopes per layer. With weights of ordinary size, each factor is around a quarter, and ten layers at a quarter each give about one millionth.
Figure 4 measures this. It builds a stack of 16-unit layers with random weights and a linear readout, runs a batch of 64 random inputs forward, gives each input one random target, and backpropagates the squared-error loss averaged over the batch. The plot shows the typical size of that gradient for each layer's weights (the root-mean-square of its entries), on a log scale.
Figure 4. Root-mean-square entry of the batch-loss gradient with respect to each layer's weights, from layer 1 (next to the input) to the last hidden layer. Simulated: weights, inputs and targets are random draws, with weights from a normal distribution with standard deviation scale/√16. The dashed gray line is the other activation function with the same settings.
With sigmoid units at weight scale 1, the gradient falls by roughly a factor of four or five per layer going back, as the slope argument predicts, and the first layer's gradient is several orders of magnitude smaller than the last layer's. Gradient descent then changes the early layers very slowly. Raising the weight scale increases the multiplier from the weights, but it also pushes the weighted sums away from zero, onto the flat parts of the sigmoid, and the average slope in the readout drops. Near scale 8 the two effects come close to cancelling: the shrink falls to about 1.3 per layer, and for some draws of the weights (try New random weights) the gradient does not shrink at all. That is not a cure, because by then about 60% of the units sit within 0.05 of 0 or 1, where small changes in their input barely change their output.
ReLU has slope exactly 1 wherever a unit is active, and its profile stays roughly flat at every weight scale on the slider. Without biases, multiplying every weight by a constant scales each layer's signal by that constant, so the scale cannot change how the gradient varies from layer to layer. What it changes is the overall size, which the readout prints for layer 1. Near scale √2, the scaling proposed by He et al., 2015, signals keep roughly the same size from layer to layer, and going from 2 layers to 20 shrinks layer 1's gradient only about fivefold. At scale 1 the same change shrinks it about two thousandfold, and at scale 2 it grows about ten thousandfold. This is one of the reasons deep networks moved from sigmoid to ReLU hidden units.
Forward pass, backward pass and update are all a network needs to learn from examples. Part 3 uses them to train an attention layer, the building block of transformers, on the page.