#dropout — Neural networks

Dropout regularization: the best friend of deep networks.

What you'll play with

Welcome to #dropout. On screen, a small MLP 2-8-8-1. At every forward pass, a random fraction of the hidden neurons is switched off: this is dropout, one of the most effective regularizations ever invented.
Apply a rate of 50%. Type /dropout 0.5.
Draw a new mask: /resample. The draw changes at every forward pass. That is what keeps the network from leaning too much on any single neuron.
Pick a dataset: /dataset moons, /dataset circles, or /dataset xor. The noise will make the network overfit unless it has a safety net.
Let us train without dropout to watch overfitting happen. Set the rate to zero /dropout 0, then run /train 60. The right panel plots train loss (blue) and validation loss (orange).
Now set /dropout 0.3 and rerun /train 60. The train/val gap closes: dropout regularizes the network without changing the architecture at all.
Play freely: /dropout 0.7 (too strong, underfitting), /dataset xor, /reset, /resample. The #gradient-descent channel will show you why it works mathematically.

Dropout: During training, each hidden neuron is switched off with probability p for every example. The network can no longer rely on a specific neuron: it learns redundant features and generalizes better.
Dropout rate: The probability p of muting a neuron (typically 0.2 to 0.5). Too high, the network stops learning; too low, the effect disappears.
Mask: The binary draw (0 or 1 per neuron) applied to a layer for a given example. A new mask is drawn on every pass.
Inverted dropout: During training, the remaining activations are divided by 1 − p so that the output scale matches inference, where nothing is muted.
Regularization: Any technique that limits a model's ability to memorize the training data: dropout, L2 penalty, early stopping, data augmentation.
Implicit ensemble: Each mask defines a different sub-network; at inference the full network behaves like the average of all those sub-networks. A free "ensemble".
Validation loss: Error measured on data never seen during training. If it climbs while the training loss still drops, the model is overfitting.
Co-adaptation: When neurons learn to correct each other's mistakes and become useless alone. Dropout breaks it by making every neighbour unreliable.

#neuron — Let us break a neuron apart: inputs, weights, sum, activation.
#dropout — Dropout regularization: the best friend of deep networks.
#activation — Why an activation? ReLU, sigmoid, tanh, Leaky ReLU, GELU and the vanishing gradient.
#gradient-descent — Loss is a landscape. The gradient says which way is up, so we step the other way.
#backpropagation — The computation graph replayed backwards: each node receives ∂L/∂(itself) and the chain rule does the rest.
#overfitting — A big model on few data points: the decision boundary twists until it memorizes the noise.
#cnn-filters — An image is a grid of numbers. A 3×3 filter slides over it, multiplies, adds up: that is a convolution.
#is-it-einstein — Is it Einstein? Two faces go through the scanner of a network that learned only Einstein: Haythem → NO, Einstein → YES. Watching is free; touching is Premium.
#embeddings-3d — A word becomes a vector: close in space = close in meaning, and you can do math on them.