#batch-normalization — Deep learning

Batch normalization: keeping activations in the right range.

What you'll play with

Welcome to #batch-normalization. On screen, four blue columns: one per hidden layer of a small network (4 layers of 16 neurons, activation tanh). Each column is the histogram of the layer's activations when a batch of 64 points goes through: 16 bars between −1 and +1, a yellow bar for the mean, a shadow for ± standard deviation. Scan left to right: the histogram already tightens (σ 0.64 → 0.29) even though the initialization is 'correct'. It is like a chain of photocopiers each set slightly too bright: by the sixth copy, the page is white. Each layer receives a distribution the previous one has distorted — the internal covariate shift. Batch normalization recentres and reduces each layer on the fly, then lets it readjust with two learned parameters, γ and β.
Let's caricature the drift. Multiply the initial weights by 6: type /init large.
Does a deeper network help? Add two layers: /layers 6.
Turn batch normalization on: /bn on. For each neuron, we compute the mean μ and standard deviation σ of its pre-activation over the batch, then replace z with (z − μ) / σ before applying tanh.
BN doesn't freeze everything: after normalization, the layer multiplies by γ and shifts by β, two parameters it learns. Try /gamma 2.
Turn BN off to prepare the next step: /bn off.
The other pathology: weights that are too small. Type /init small (weights × 0.25).
Let's see what this costs the training. The same network (same starting weights) will be trained twice on the 'moons' set, once without BN, once with: /train 40.
Your turn: /activation relu then /bn on and /beta -2 to kill almost every ReLU neuron; /batch 8 to see noisy batch statistics; /propagate for another draw of weights and points; /init normal then /train 60 to compare with a good initialization; /reset to start over. Next: channel #rnn-lstm, where the signal no longer crosses layers but time — and drifts the same way.

Channel commands

/layers <2..6> — Number of hidden layers (16 neurons each).
/activation <tanh|relu|sigmoid> — Activation function of the hidden layers.
/init <small|normal|large> — Initial weight scale: small (× 0.25), normal (Xavier / He), large (× 6).
/bn <on|off> — Turn batch normalization on or off on each hidden layer.
/gamma <0.1..3> — Scale factor γ applied after normalization (pre-activation standard deviation).
/beta <-2..2> — Shift β applied after normalization (pre-activation mean).
/batch <8..128> — Batch size that flows through the network and is used to compute μ, σ.
/propagate — New draw (initial weights and batch), recomputes every histogram.
/train <5..60> — Trains the same network with and without BN on "moons" and traces both losses.
/reset — Back to 4 tanh layers, normal init, BN off, γ = 1, β = 0, batch 64.

Glossary

batch normalization: Layer that recentres and reduces each pre-activation over the current batch, ẑ = (z − μ) / σ, then rescales it with two learned parameters, y = γ·ẑ + β. It stabilizes the distribution each layer receives, allows larger learning rates, and makes the network less sensitive to initialization.
internal covariate shift: The fact that the distribution of a layer's inputs changes over the course of training and with depth, because the earlier layers keep moving. Each layer then has to re-learn against a moving target; this is the problem batch normalization was designed to mitigate.
batch mean and standard deviation: Statistics μ and σ computed, for each neuron, over the examples of the current batch. They are what BN normalizes with during training: the smaller the batch, the noisier they are, and the less reliable BN becomes.
gamma and beta: The two learned parameters of batch normalization: γ multiplies the normalized value (its standard deviation), β shifts it (its mean). They give the layer the freedom to choose its range — including undoing normalization if that is useful.
saturation: Zone of an activation where the slope is nearly zero: the edges of tanh (±1) or of the sigmoid (0 and 1). A saturated activation barely transmits any gradient: this is the source of the vanishing gradient.
dead neurons: ReLU neurons whose pre-activation is negative for (nearly) every example: their output is 0, their gradient too, and they no longer relearn. A too-negative mean (β ≪ 0) or bad weights can kill entire layers.
layer normalization: Variant that normalizes each example over its own neurons (one mean and one standard deviation per example, not per neuron) instead of normalizing over the batch. Independent of batch size, it is standard in transformers and recurrent networks.
running averages: At inference time, there is no batch anymore: batch normalization uses means and variances accumulated during training (exponential moving averages of the batch statistics). The network then behaves as a fixed function, example by example.
weight initialization: Choice of the initial standard deviation of the weights. Xavier / Glorot (1/√n, for tanh and sigmoid) and He (√(2/n), for ReLU) are calibrated to preserve variance from one layer to the next. Too large: saturation; too small: the signal collapses.
batch: Subset of examples processed together at each step of gradient descent (8 to 128 here). Batch normalization computes its statistics over that batch: its size is therefore a hyperparameter that changes the network's behaviour, not just its speed.

Other channels in Deep learning

#optimizers — SGD, Momentum and Adam: the race to the minimum.
#batch-normalization — Batch normalization: keeping activations in the right range.
#rnn-lstm — RNN and LSTM: remembering a sequence.
#autoencoder — Autoencoder: compress then reconstruct.
#transfer-learning — Transfer learning: start from an already-trained network.
#gan — GAN: a forger against an inspector