Module 4 — Backpropagation explained step by step

Here is the algorithm that makes deep learning possible. Its reputation for obscurity is largely undeserved: backpropagation is nothing but the chain rule, applied methodically and reusing intermediate computations.

The problem to solve

After the forward pass we have a loss $\mathcal{L}$, a single number. To improve it we need to know which way to push each parameter, that is, compute $\partial \mathcal{L} / \partial w$ for each weight — sometimes millions of them.

The naive approach would be to nudge each weight slightly and observe the effect on the loss. That would require one full forward pass per parameter: for a million weights, a million passes. Unworkable.

Backpropagation obtains all gradients in a single backward pass whose cost is of the same order as the forward pass. That speedup factor changes everything.

The chain rule, the only prerequisite

If $y$ depends on $u$ which depends on $x$, then:

$$\frac{\partial y}{\partial x} = \frac{\partial y}{\partial u} \cdot \frac{\partial u}{\partial x}$$

A network is precisely a composition of functions: the input traverses layer 1, then layer 2, up to the loss. The derivative of the loss with respect to a deep weight is therefore the product of local derivatives along the path connecting them.

This reading immediately gives the intuition for module 6: if every factor in that product is below 1, the product of many factors tends to zero. That is the vanishing gradient, and it is inscribed in the very structure of the algorithm.

The backward pass, layer by layer

The central quantity is a layer's local error, written $\delta^{(\ell)}$: the sensitivity of the loss to the pre-activation $z^{(\ell)}$.

We start at the last layer. With a softmax followed by cross-entropy, the result simplifies remarkably:

$$\delta^{(L)} = \hat{y} - y$$

The output layer's error is simply the gap between prediction and truth. That elegance is no accident: it is the pairing of softmax with cross-entropy that produces it, and one more reason to respect the couples of module 2.

We then move up one layer at a time:

$$\delta^{(\ell)} = \left(W^{(\ell+1)\top} \delta^{(\ell+1)}\right) \odot f'\left(z^{(\ell)}\right)$$

Two operations, each with a clear meaning. The product by $W^{(\ell+1)\top}$ redistributes the next layer's error back to the current one, in proportion to the weights: a neuron that contributed a lot receives a proportional share of the error. The element-wise multiplication by $f'(z^{(\ell)})$ filters that error by the local sensitivity of the activation. If the output is saturated, $f'$ is near zero and the error no longer flows back.

The parameter gradients follow directly:

$$\frac{\partial \mathcal{L}}{\partial W^{(\ell)}} = \delta^{(\ell)} a^{(\ell-1)\top}, \qquad \frac{\partial \mathcal{L}}{\partial b^{(\ell)}} = \delta^{(\ell)}$$

The first formula deserves careful reading: a weight's gradient is the product of the downstream error by the upstream activation. A weight is corrected only if both are non-zero. If the input neuron was inactive, that weight does not move — which concretely explains the dying neuron of module 2.

A minimal implementation

def backward_pass(activations, pre_activations, weights, y_true):
    """Return the gradients of a ReLU network with softmax output."""
    grads_W, grads_b = [], []
    # Output layer: the softmax + cross-entropy pairing simplifies.
    delta = activations[-1] - y_true

    for i in reversed(range(len(weights))):
        grads_W.insert(0, delta.T @ activations[i])
        grads_b.insert(0, delta.sum(axis=0))
        if i > 0:
            # Redistribute through the weights, then filter by the ReLU derivative.
            delta = (delta @ weights[i]) * (pre_activations[i - 1] > 0)

    return grads_W, grads_b

The ReLU derivative reduces to the test > 0, which is 1 for positive inputs and 0 elsewhere. That is one of this activation's practical advantages.

What libraries actually do

You will never write this code in production, and that is fortunate. PyTorch and TensorFlow perform automatic differentiation: every operation of the forward pass is recorded in a computation graph, and a call to loss.backward() walks it backwards applying each operation's known local derivative.

Two consequences are worth keeping. First, automatic differentiation is neither symbolic computation nor a finite-difference approximation: the gradients obtained are exact, to machine precision. Second, the graph must retain intermediate activations for the backward pass, which explains why training consumes far more memory than inference — and why reducing batch size is the first reflex when facing GPU memory exhaustion.

Why understand an algorithm you will not write

Because the symptoms of a failing training run are readable in these formulas. A stalled loss may come from gradients filtered by a saturated activation. Zero gradients in early layers signal a product of factors that are too small. A loss going NaN usually comes from an exploding gradient. Without the mental model of backpropagation, these diagnoses are superstition; with it, they become reading.

Summary

• Backpropagation obtains all gradients in one backward pass, where the naive approach would need one per parameter.
• It is only the chain rule: a deep weight's gradient is the product of local derivatives along the path.
• The backward pass redistributes the error through $W^{\top}$ then filters it by $f'(z)$; a saturated activation therefore blocks the flow.
• A weight's gradient is the product of the downstream error by the upstream activation; libraries automate all of this, at the cost of heavy memory use during training.

Next module: optimizers, which decide what to do with these gradients once computed.