Module 7 — Regularization: dropout, L2 penalty, data augmentation

A deep network has enormous capacity. Well-known work showed that a large network can memorize images paired with entirely random labels: pure memorization is within its reach. Regularization is therefore what forces it to generalize rather than memorize.

Dropout: randomly disabling neurons

Dropout, proposed in 2012, zeroes a randomly drawn fraction of neurons at each training pass.

A dropout rate of 0.4 on the hidden layers. Click “New draw” to see another pass: the disabled neurons change every time, so the network can never rely on any one neuron in particular.

Why does this work? Three complementary readings, and the second is the most illuminating.

First, dropout prevents co-adaptation. Without it, neurons specialize into teams that only work together, each correcting the others' errors — a fragile arrangement that does not survive new data. Since a neuron can vanish at any moment, each must become useful independently.

Second, dropout amounts to training an ensemble of networks. Each draw defines a different architecture; with $n$ neurons there are $2^n$ possible sub-networks, all sharing their weights. Inference approximates the average of that enormous ensemble, recovering the benefit of the ensemble methods from the supervised learning course for the price of a single model.

Third, dropout injects noise, pushing the network toward more robust solutions.

The technical point everyone misses

Dropout behaves differently at training and at inference, and that asymmetry is the most common source of error.

At training, a fraction $p$ of neurons is disabled. At inference, no neuron is disabled: we want a deterministic prediction using the whole network. But if nothing were corrected, the sum received by the next layer would be larger at inference than at training, since more neurons contribute — the scale of activations would change between the two regimes.

The solution libraries adopt is inverted dropout: during training, surviving activations are divided by $1-p$. The expected value of the sum is thus preserved, and inference has nothing left to adjust.

model.train()   # dropout active, batch norm in batch-statistics mode
# ... training loop ...

model.eval()    # dropout disabled, normalization in running-average mode
with torch.no_grad():
    predictions = model(X_test)

Forgetting model.eval() before evaluation is a classic. Predictions become noisy and irreproducible, metrics degrade for no apparent reason, and no error is raised. The next module will show that batch normalization suffers the same omission, and worse.

In practice, a rate of 0.2 to 0.5 on fully connected layers is common usage. Note that dropout is markedly less used in recent architectures: layer normalization and large data volumes have largely taken over.

The L2 penalty

Adding to the loss a term proportional to the squared weights pushes them toward zero:

$$\mathcal{L}_{\text{total}} = \mathcal{L} + \lambda \sum_i w_i^2$$

This is the same idea as Ridge regression: smaller weights produce a smoother function, hence one less likely to fit noise. In deep learning vocabulary this is called weight decay, and module 5 flagged the caveat: with Adam, this penalty must be applied separately, which is what AdamW does. Values from $10^{-4}$ to $10^{-2}$ cover most cases.

Early stopping and data augmentation

Early stopping is the cheapest regularization in existence: monitor the validation loss and interrupt training when it stops improving, keeping the weights of the best pass. A patience of a few epochs avoids stopping on a fluctuation. Module 9 will show how to read these curves.

Data augmentation acts at the root of the problem. Rather than constraining the model, it multiplies examples through label-preserving transformations: rotations, crops, brightness shifts for images; synonym replacement for text. A cat rotated ten degrees is still a cat, but it is a new observation for the network.

The rule to remember fits in one sentence: the transformation must preserve the label. A horizontal flip suits animal photographs, but it destroys information on handwritten digits or text.

In which order to act

Facing overfitting, the order of effectiveness is almost always the same. More data first, real or augmented, because it is the only answer that adds information. Then early stopping, free and immediate. Then a moderate weight penalty. Then dropout on dense layers. Reducing network size comes last: a large well-regularized network generally beats a small unregularized one.

Summary

• A large network can memorize random labels: regularization is what forces it to generalize.
• Dropout prevents co-adaptation and is equivalent to training an ensemble of $2^n$ weight-sharing sub-networks.
• Inverted dropout divides surviving activations by $1-p$ at training; forgetting eval() at inference silently degrades results.
• L2 penalty, early stopping and data augmentation complete the arsenal; augmentation is only valid if the transformation preserves the label.

Next module: batch and layer normalization, which transformed the stability of training.