Module 2 — Activation functions: ReLU, sigmoid, tanh, GELU

The activation function is the most unobtrusive component of a network and one of the most decisive. This module explains first why it is structurally indispensable, then why the default choice changed in the mid-2010s.

Without non-linearity, depth is pointless

Here is the demonstration to know, because it justifies the very existence of these functions. Suppose a two-layer network without activation. The first computes $h = W_1 x + b_1$, the second $y = W_2 h + b_2$. Substituting:

$$y = W_2(W_1 x + b_1) + b_2 = (W_2 W_1)x + (W_2 b_1 + b_2)$$

The product $W_2 W_1$ is a matrix, and $W_2 b_1 + b_2$ a vector. The two-layer network therefore is a one-layer network, with different parameters. Stack a hundred of them and the result remains a linear transformation.

The consequence is stark: without non-linear activation, depth is a computational cost with no gain in capacity. It is the activation function, and it alone, that makes stacking fruitful.

The three historical activations

Use the buttons to isolate a curve. What sets ReLU apart is visible at a glance: it does not saturate on the positive side, where sigmoid and tanh flatten out at both ends.

The sigmoid squashes any input into the interval $]0, 1[$:

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$

It dominated the 1990s for its probabilistic interpretation. Two flaws pushed it out of hidden layers. First saturation: as soon as $|x|$ exceeds 4 or 5, the curve is nearly flat, so its derivative is close to zero and the gradient stops flowing — the subject of module 6. Second, its output is not centered at zero, which makes weight updates systematically biased in one direction and slows convergence.

The hyperbolic tangent fixes the second flaw by producing outputs in $]-1, 1[$, centered at zero. It is strictly preferable to the sigmoid in hidden layers, but it still saturates at both ends.

ReLU, and why it changed everything

ReLU is disarmingly simple:

$$\text{ReLU}(x) = \max(0, x)$$

It became the default choice, and its advantages are concrete. It does not saturate on the positive side: its derivative there is exactly 1, so the gradient passes through undiminished, which makes very deep networks practical. It is trivial to compute — one comparison, where the sigmoid requires an exponential. And it produces sparse activation: neurons with negative output are inactive, yielding leaner representations.

Its flaw has a name: the dying neuron. If the weights lead a neuron to systematically receive negative inputs, its output is always zero, its derivative always zero, and it never updates again. It is permanently lost. Too high a learning rate produces them in quantity.

Hence the variants, all of which let a trickle of gradient through on the negative side:

Variant	Behavior for $x < 0$	Benefit
Leaky ReLU	small fixed slope, often $0.01x$	removes the dying neuron
PReLU	slope learned during training	more flexible, one extra parameter
ELU	exponential decay toward $-1$	centered outputs, more costly
GELU	smooth weighting by a normal law	the transformer standard

GELU deserves particular attention: it is the activation of modern language architectures, including the transformers of course 12. Instead of cutting sharply to zero, it weights the input by the probability that a normal variable falls below it, giving a smooth transition around zero. In practice it brings a consistent but modest gain, at slightly higher computational cost.

The output layer is not chosen, it is deduced

An important point of method: for hidden layers, the activation is a hyperparameter; for the output layer, it is dictated by the task.

Task	Output activation	Matching loss
regression	none (linear output)	mean squared error
binary classification	sigmoid, 1 neuron	binary cross-entropy
multiclass classification	softmax, 1 neuron per class	categorical cross-entropy
multi-label classification	sigmoid on each neuron	binary cross-entropy

The distinction between the last two rows is a frequent source of errors. Softmax normalizes outputs to sum to 1: it assumes classes are mutually exclusive. If an observation can carry several labels at once — a photograph containing a dog and a bicycle — you need an independent sigmoid per label, never a softmax.

What to choose in practice

Start with ReLU in hidden layers: it is the sensible default, fast and proven. If you observe dying neurons, move to Leaky ReLU. On a transformer-based architecture, use GELU, which is the field's convention. Use sigmoid and tanh in hidden layers only for a specific reason — a recurrent network gate, for instance, the subject of course 11. And remember that the gain from optimizing the activation stays small next to that of a better architecture or better data.

Summary

• Without non-linear activation, a stack of layers reduces to a single layer: depth then contributes nothing.
• Sigmoid and tanh saturate at the extremes, which blocks the gradient; tanh is at least centered at zero.
• ReLU does not saturate on the positive side, computes instantly and produces sparse activations, at the risk of the dying neuron that Leaky ReLU and its variants correct.
• The output activation is deduced from the task; softmax for mutually exclusive classes, independent sigmoids for multiple labels.

Next module: the forward pass and loss computation, that is, exactly what happens when data traverses the network.