Module 6 — ResNet and residual connections

Take the small VGG of module 5 and stack another six blocks on top. Intuition says the network should do at least as well: the added layers could always learn to be identity mappings if they had nothing to contribute. In practice, the training error goes up, not down. He and colleagues at Microsoft named this the degradation problem in 2015, and proposed a fix so simple that it read like a hack. It won the ImageNet competition that year with a 152-layer network and has been the default architecture ever since.

The degradation problem

Depth was supposed to be a free lunch. Instead, past twenty or thirty layers, plain stacks of VGG-style convolutions plateau in training accuracy and eventually get worse, both on train and on test. This is not overfitting: the training error itself grows. It is not vanishing gradients either, at least not entirely, because batch normalisation (introduced a year earlier) had already fixed most of that.

The right diagnosis: identity mappings, which should be trivial to learn, are actually hard for a stack of non-linear layers to represent. To output the same tensor as the input, the last layer has to fine-tune weights against a moving target coming from every layer below. The network struggles to do nothing.

The residual block: a shortcut around the layers

The fix is to add a shortcut connection that carries the input directly around a block of layers, and to make the block predict the residual, that is, the difference between the input and the desired output. If the block has nothing to contribute, it can zero out its convolutions and the shortcut passes the input through unchanged.

The residual block: the shortcut carries x unchanged over the convolutions, so the layers only need to learn the residual y - x.

Formally, a plain block computes $y = F(x)$, where $F$ is a small stack of layers. A residual block computes $y = F(x) + x$. The gradient of the loss $L$ with respect to $x$ is:

$$\frac{\partial L}{\partial x} \;=\; \frac{\partial L}{\partial y}\left(1 + \frac{\partial F}{\partial x}\right).$$

The 1 guarantees that gradient flows to earlier layers even when $\partial F / \partial x$ is small — the vanishing gradient problem practically disappears.

Implementing the block

Here is the minimal residual block in Keras. Note the two conventions everyone follows: batch normalisation between the convolution and the ReLU, and no ReLU immediately after the addition until the next block.

import tensorflow as tf
from tensorflow.keras import layers

def residual_block(x, filters, downsample=False):
    strides = 2 if downsample else 1
    shortcut = x
    y = layers.Conv2D(filters, 3, strides=strides, padding="same", use_bias=False)(x)
    y = layers.BatchNormalization()(y)
    y = layers.ReLU()(y)
    y = layers.Conv2D(filters, 3, padding="same", use_bias=False)(y)
    y = layers.BatchNormalization()(y)
    # If the shortcut shape does not match, project it with a 1 x 1 conv.
    if downsample or x.shape[-1] != filters:
        shortcut = layers.Conv2D(filters, 1, strides=strides, use_bias=False)(x)
        shortcut = layers.BatchNormalization()(shortcut)
    out = layers.Add()([y, shortcut])
    return layers.ReLU()(out)

Two subtleties are worth flagging. First, when the block downsamples or changes the number of channels, the shortcut cannot be a pure identity — the shapes would not match. A 1 by 1 convolution on the shortcut aligns them at almost no cost. Second, use_bias=False before batch normalisation: the bias would be added and immediately cancelled by BN's mean subtraction, so the framework saves the parameter.

The bottleneck block: fewer FLOPs at large channel counts

At 256 or 512 channels, a 3 by 3 convolution becomes expensive: $3 \times 3 \times 512 \times 512 = 2.4 \text{M}$ multiply-adds per output pixel. ResNet50 and deeper variants use the bottleneck block instead: 1 by 1 down-project to 64 channels, 3 by 3 at 64 channels, 1 by 1 up-project back to 256. The overall channel count is preserved, but the expensive 3 by 3 operates on 8 times fewer channels.

Block type	Channels	Layers	Multiply-adds per output pixel
Basic	256 → 256	3×3, 3×3	~1.2 M
Bottleneck	256 → 64 → 64 → 256	1×1, 3×3, 1×1	~0.2 M

That is the reason ResNet50 is faster than ResNet34 despite having more layers.

The ResNet zoo, in one table

Variant	Depth	Block type	Parameters	Top-1 ImageNet
ResNet-18	18	Basic	11 M	~70 %
ResNet-34	34	Basic	21 M	~73 %
ResNet-50	50	Bottleneck	25 M	~76 %
ResNet-101	101	Bottleneck	44 M	~78 %
ResNet-152	152	Bottleneck	60 M	~78.5 %

Diminishing returns beyond ResNet50 are the rule, not the exception. For most transfer learning use cases — including our waste-sorting case study in modules 8 and 9 — ResNet50 is the default: enough capacity to win, small enough to fine-tune on a laptop with a modest GPU.

Pre-activation ResNet (v2)

A follow-up paper reordered the block as BN → ReLU → Conv → BN → ReLU → Conv, so the shortcut adds a pure sum of pre-activated outputs. This variant, sometimes called ResNet-v2, trains slightly deeper models more stably. When a library offers both, v2 is a safe default; the numerical difference on ImageNet is small but reliable.

Bringing it back to the red thread

A tiny ResNet on CIFAR-10 — three stacks of two basic blocks each at 32, 64 and 128 channels — reaches 88-89 % test accuracy in 30 epochs, comfortably above our small VGG. That is more than enough for the last stretch of the red thread; from module 7 onwards we start comparing architectures on compute efficiency, and from module 8 we switch to real-world images where the extra depth pays off differently.

def small_resnet_cifar():
    inputs = tf.keras.Input(shape=(32, 32, 3))
    x = layers.Conv2D(32, 3, padding="same", use_bias=False)(inputs)
    x = layers.BatchNormalization()(x)
    x = layers.ReLU()(x)
    for _ in range(2): x = residual_block(x, 32)
    x = residual_block(x, 64, downsample=True)
    x = residual_block(x, 64)
    x = residual_block(x, 128, downsample=True)
    x = residual_block(x, 128)
    x = layers.GlobalAveragePooling2D()(x)
    out = layers.Dense(10, activation="softmax")(x)
    return tf.keras.Model(inputs, out)

Batch normalisation is a hidden state, not just weights

BatchNorm learns two parameters per channel (scale, shift) plus two running statistics (mean, variance) that are updated only at training time. Its behaviour differs between training and inference — the same trap that will hit us in module 9 when we freeze a pretrained ResNet50 and forget to pass training=False.

In summary

• Plain stacks of layers degrade past a certain depth: training accuracy drops. The cause is the difficulty of representing identity mappings, not merely vanishing gradients.
• A residual block adds a shortcut around a small stack of layers and predicts the residual, which makes the identity trivial to represent and lets gradients flow.
• The bottleneck block uses 1 by 1 convolutions to shrink and expand channels around a 3 by 3, cutting compute at high channel counts.
• ResNet50 is the default architecture for most transfer learning tasks: enough capacity, small enough to fine-tune on modest hardware. Pre-activation (v2) is a safe upgrade when the library offers it.

Next module: Inception and MobileNet, two architectures that trade the residual idea against different structural constraints — parallel branches and depthwise separable convolutions.