Module 8 — Batch and layer normalization

The feature engineering course insisted on scaling the inputs. Batch normalization extends that idea to the inside of the network, and its introduction in 2015 probably did more for training stability than any other technique of that period.

The problem: every layer sees its input distribution move

A hidden layer receives the previous layer's activations. But that previous layer's weights change at every update, so the distribution of what our layer receives shifts continuously during training.

This is uncomfortable: the layer must adapt to a moving target. The method's authors named this phenomenon internal covariate shift. You should know that this explanation has since been contested: later work attributes the method's effectiveness instead to a smoothing of the loss landscape, which permits higher learning rates. The mechanism remains debated; the practical effectiveness does not.

The computation, and above all the two learned parameters

For each activation feature, we normalize over the batch's observations:

$$\hat{x} = \frac{x - \mu_{\text{batch}}}{\sqrt{\sigma^2_{\text{batch}} + \epsilon}}, \qquad y = \gamma \hat{x} + \beta$$

The first step centers and scales. The second is the one often forgotten, although it is essential: $\gamma$ and $\beta$ are learned parameters that let the network rescale and shift the result.

Why normalize and then allow denormalization? Because imposing zero mean and unit variance is a strong constraint that can hurt. With a sigmoid, forcing inputs into a narrow range around zero confines the activation to its linear portion and strips it of its purpose. The parameters $\gamma$ and $\beta$ make the transformation optional: the network can, if it needs to, relearn the identity. We offer it a possibility, we do not impose a regime.

The observed benefits are substantial: higher learning rates become usable, sensitivity to initialization drops markedly, and a regularizing effect appears, due to the noise of batch statistics — to the point that dropout can often be reduced.

The two regimes, and the pitfall that follows

Like dropout, batch normalization behaves differently depending on the phase, but the stakes are higher here.

At training, it uses the mean and variance of the current batch, and updates running averages along the way. At inference, those batch statistics are unusable: the prediction would depend on the other observations present in the batch, which is absurd — and for a single example, the variance would be zero. It therefore uses the running averages accumulated during training.

Hence the practical consequence: forgetting model.eval() with batch normalization does not merely degrade results, it makes predictions dependent on the batch composition. The same example gets different predictions depending on what accompanies it. This is a formidable bug because it raises no error and disappears when you test one example at a time.

The limits, and layer normalization

Three situations put batch normalization in difficulty. With small batches, statistics become unreliable — below 8 or 16 observations the method loses its point. With variable-length sequences, as in language processing, per-position statistics lose their meaning. And its asymmetric behavior between training and inference is a source of production incidents.

Layer normalization answers all of this with a change of axis: instead of normalizing one feature across the batch's observations, it normalizes all the activations of a given observation.

	Batch normalization	Layer normalization
Computation axis	across the batch	across one observation's features
Batch-dependent	yes	no
Distinct regimes	yes	no
Preferred domain	vision, convolutional networks	transformers, sequences

The decisive advantage is there: layer normalization does not depend on the batch. It therefore behaves identically at training and inference, works with a batch of one observation, and accommodates sequences of any length. For these reasons it equips every transformer, and course 12 will find it in each block.

import torch.nn as nn

# Vision: batch normalization, after the layer and before the activation.
vision_block = nn.Sequential(
    nn.Linear(256, 128),
    nn.BatchNorm1d(128),
    nn.ReLU(),
)

# Sequences: layer normalization, insensitive to batch size.
sequence_block = nn.Sequential(
    nn.Linear(256, 128),
    nn.LayerNorm(128),
    nn.GELU(),
)

Where to place normalization

The historical convention places normalization after the linear transformation and before the activation, as above. Recent transformer architectures often adopt the so-called pre-norm arrangement, which normalizes before the block: it markedly stabilizes the training of very deep networks. Note finally that the linear layer preceding a batch normalization needs no bias, since the $\beta$ parameter already plays that role — hence the bias=False you encounter in large-model code.

Summary

• Normalization addresses the distribution shift each layer experiences; the original explanation is debated, the effectiveness is not.
• The learned parameters $\gamma$ and $\beta$ make the transformation optional: the network can relearn the identity if it needs to.
• Batch normalization has two regimes; forgetting eval() makes predictions dependent on batch composition.
• Layer normalization does not depend on the batch, behaves identically in both phases, and equips every transformer.

Next module: reading learning curves, the diagnostic tool that ties all the previous modules together.