#activation — Neural networks

Why an activation? ReLU, sigmoid, tanh, Leaky ReLU, GELU and the vanishing gradient.

What you'll play with

Welcome to #activation. On screen, the sigmoid curve on [-5, 5] and a pink point at x = 2.5. A neuron computes f(Σ w·x + b). Without f, stacking ten layers is the same as composing ten linear functions… that is, a single linear function: a regression, unable to separate XOR. The activation is what makes the network non-linear.
Push the point further right: type /x 4. Look at where f(x) lands, and above all by how much it moved.
For learning, what matters is not f(x) but f'(x), the slope. This is what backpropagation multiplies layer after layer to send the error back. Show it: /derivative.
Switch to ReLU: /function relu. Right of zero, the slope is 1 whatever the value of x: no saturation. This detail made it possible to train AlexNet in 2012 and relaunched all of deep learning.
But ReLU has an Achilles heel. Place the point on the left: /x -2.
Overlay all five functions: /compare. Spot GELU (light blue): smooth like sigmoid around zero, linear like ReLU on the right, slightly negative (dip at -0.17). This is the activation of transformers: BERT, GPT and most LLMs.
Your turn: /function gelu then /x -0.75 (GELU dip), /function tanh then /x 0 (max slope = 1), /derivative to compare all five derivatives, /reset to start over. Next step: the #gradient-descent channel, where this slope becomes the direction weights move in.

/function <relu|sigmoid|tanh|leaky|gelu> — Change the activation function being plotted.
/x <value -5..5> — Move the pink point along the curve.
/derivative — Show or hide the derivative curve f'(x).
/compare — Overlay the five functions, each in its own color.
/reset — Back to sigmoid, x = 2.5, no derivative and no compare.

Activation function: A non-linear function applied to a neuron's weighted sum. Without it, stacking layers collapses to a single linear function: this is what gives the network its power.
ReLU: max(0, x). Slope 1 to the right of zero (no saturation), 0 to the left. Fast and effective: the default activation of deep networks since 2012.
Sigmoid: 1 / (1 + e^(−x)), output between 0 and 1. Saturates at both ends: the slope tends to 0 there, which starves the gradient in deep networks. Still useful as an output for a probability.
Tanh: Hyperbolic tangent, output between −1 and 1, centered at zero (better than sigmoid for hidden layers), but it saturates too.
Leaky ReLU: ReLU with a small slope (0.01) to the left of zero: a neuron is never fully "dead", it can always wake up.
GELU: A smooth activation x·Φ(x) (Φ = Gaussian CDF): close to ReLU on the right, slightly negative near zero. Used in BERT, GPT and most transformers.
Saturation: A zone where the function flattens: the derivative is almost zero and the learning signal stops flowing. Sigmoid and tanh saturate at the extremes.
Vanishing gradient: In backpropagation, derivatives multiply from layer to layer. If each is 0.1, ten layers give 10⁻¹⁰: the earliest layers no longer learn.
Dead neuron: A ReLU neuron whose pre-activation is always negative: output 0, gradient 0, it will never wake up. Leaky ReLU and a good initialization limit the problem.

#neuron — Let us break a neuron apart: inputs, weights, sum, activation.
#dropout — Dropout regularization: the best friend of deep networks.
#activation — Why an activation? ReLU, sigmoid, tanh, Leaky ReLU, GELU and the vanishing gradient.
#gradient-descent — Loss is a landscape. The gradient says which way is up, so we step the other way.
#backpropagation — The computation graph replayed backwards: each node receives ∂L/∂(itself) and the chain rule does the rest.
#overfitting — A big model on few data points: the decision boundary twists until it memorizes the noise.
#cnn-filters — An image is a grid of numbers. A 3×3 filter slides over it, multiplies, adds up: that is a convolution.
#is-it-einstein — Is it Einstein? Two faces go through the scanner of a network that learned only Einstein: Haythem → NO, Einstein → YES. Watching is free; touching is Premium.
#embeddings-3d — A word becomes a vector: close in space = close in meaning, and you can do math on them.