#backpropagation — Neural networks

The computation graph replayed backwards: each node receives ∂L/∂(itself) and the chain rule does the rest.

What you'll play with

Welcome to #backpropagation. On screen: a computation graph. Two blue inputs x1, x2, two pink hidden neurons h1, h2 (tanh), a green output o (sigmoid) and, on the far right, the yellow loss L = ½(o − y)². Every connection carries a weight. First we compute left to right, then we differentiate right to left: each node receives ∂L/∂(itself) and the chain rule does the rest.
Start with the forward pass. Type /forward: values light up from left to right, from the inputs to the loss, and the connections crossed turn blue.
The loss is known. Now we go back. First backward step: ∂L/∂o. Type /step.
One node further: ∂L/∂zo = ∂L/∂o · σ'(zo), with σ'(zo) = o(1 − o). This is the entire chain rule: at every node crossed backwards, one multiplication by the local derivative. Type /step.
The pattern is always the same, so let the gradient flow all the way back to the inputs in one shot: /backward. Notice the thickness of the red connections: the gradients of the first-layer weights (w11, w12, w21, w22) are smaller than the output ones, because they went through more successive multiplications.
We now know ∂L/∂w for each of the nine parameters. Let's update them: w ← w − lr · ∂L/∂w. Type /update and compare the loss before/after in the response and the right panel.
Change the target: /target 0. The network must now predict 0 instead of 1. Redo /forward then /backward freely: the gradients flip sign, gradient descent pushes the weights the other way.
This is exactly what PyTorch does when you write loss.backward(): it replays the computation graph in reverse, one local derivative per node, and optimizer.step() applies the /update. Play freely: /input 1.5 -1, /lr 1.5, /reset. Next up: the #overfitting channel, where we'll see what happens when we take too many steps.

Channel commands

/step — Reveals the next graph operation (forward then backward).
/forward — Runs the whole forward pass up to the loss.
/backward — Runs the whole backward pass: every gradient in one shot.
/input <x1=-2..2> <x2=-2..2> — Changes the two inputs and recomputes the graph.
/target <0|1> — Changes the target y (0 or 1) and recomputes the graph.
/update — Applies one gradient-descent step (w ← w − lr · ∂L/∂w) and recomputes.
/lr <0.01..2> — Sets the learning rate used by /update.
/reset — Resets inputs, target, weights and cursor to their initial value.

Glossary

Backpropagation: Algorithm that computes the gradient of the loss with respect to every weight by propagating the error from output back to input, using the chain rule. Efficient: a single backward pass gives every gradient.
Forward pass: Computing the network output from the input, layer by layer. Activations are cached: the backward pass will need them.
Backward pass: The return trip: starting from the output error, we compute gradients layer after layer in reverse.
Chain rule: Derivative of a composition: (f∘g)' = f'(g)·g'. Backpropagation is just this rule applied to thousands of nested functions.
Delta: The gradient of the loss with respect to a neuron's pre-activation, δ = ∂L/∂z. It flows from layer to layer; a weight's gradient is δ × input activation.
Local gradient: Derivative of a node with respect to its immediate inputs (for example the derivative of the activation). The global gradient is the product of local gradients along the path.
Computational graph: Representation of operations as a directed graph. PyTorch and TensorFlow build it automatically to differentiate any model (automatic differentiation).
Weight update: Once the gradients are known: w ← w − η·∂L/∂w. One gradient-descent step per weight, all at once.

Other channels in Neural networks

#neuron — Let us break a neuron apart: inputs, weights, sum, activation.
#dropout — Dropout regularization: the best friend of deep networks.
#activation — Why an activation? ReLU, sigmoid, tanh, Leaky ReLU, GELU and the vanishing gradient.
#gradient-descent — Loss is a landscape. The gradient says which way is up, so we step the other way.
#backpropagation — The computation graph replayed backwards: each node receives ∂L/∂(itself) and the chain rule does the rest.
#overfitting — A big model on few data points: the decision boundary twists until it memorizes the noise.
#cnn-filters — An image is a grid of numbers. A 3×3 filter slides over it, multiplies, adds up: that is a convolution.
#is-it-einstein — Is it Einstein? Two faces go through the scanner of a network that learned only Einstein: Haythem → NO, Einstein → YES. Watching is free; touching is Premium.
#embeddings-3d — A word becomes a vector: close in space = close in meaning, and you can do math on them.