Module 5 — VGG and the power of stacking small filters

AlexNet used 11 by 11, 5 by 5 and 3 by 3 kernels mixed together, out of tradition and hardware constraint. In 2014, Simonyan and Zisserman published VGG, an architecture that made a single, radical claim: everything can be done with 3 by 3 filters, provided you stack them deep enough. The paper — one of the shortest to change a field — quantified the trade-off, and the answer flipped every subsequent design.

Two 3x3 are worth a 5x5, for less

Consider the receptive field. A 5 by 5 convolution covers 25 input pixels per neuron. Two stacked 3 by 3 convolutions cover 3 + (3 - 1) = 5 pixels along each axis, so also 25 input pixels per neuron. Same effective window. What about parameters?

• One Conv2D(C, 5) on $C$ input channels: $5 \times 5 \times C \times C = 25 C^2$ weights.
• Two Conv2D(C, 3) stacked: $2 \times (3 \times 3 \times C \times C) = 18 C^2$ weights.

The stack has 28 % fewer parameters for the same receptive field. It also inserts a non-linearity in the middle, giving the network two ReLUs instead of one across the same spatial extent — a strictly richer function class. Three stacked 3 by 3 layers cover a 7 by 7 receptive field with $27 C^2$ weights, versus $49 C^2$ for a single 7 by 7 layer.

That is the entire theoretical argument, and it is enough to explain why every architecture from 2015 onwards defaults to 3 by 3 kernels, and often to 1 by 1 kernels for channel mixing.

The VGG block: repeat and downsample

VGG's structural idea is equally uniform: define a block made of two or three 3 by 3 convolutions with same padding, followed by one 2 by 2 max pool that halves the resolution. Then repeat the block, doubling the number of filters each time the resolution halves. That doubling keeps the number of activations per block roughly constant, which is a healthy balance between spatial detail and channel richness.

import tensorflow as tf

def vgg_block(x, filters, n_conv=2):
    for _ in range(n_conv):
        x = tf.keras.layers.Conv2D(filters, 3, padding="same", activation="relu")(x)
    return tf.keras.layers.MaxPooling2D(2)(x)

def small_vgg_cifar():
    inputs = tf.keras.Input(shape=(32, 32, 3))
    x = vgg_block(inputs, 64,  n_conv=2)   # 16 x 16
    x = vgg_block(x,      128, n_conv=2)   #  8 x 8
    x = vgg_block(x,      256, n_conv=3)   #  4 x 4
    x = tf.keras.layers.GlobalAveragePooling2D()(x)
    out = tf.keras.layers.Dense(10, activation="softmax")(x)
    return tf.keras.Model(inputs, out)

small_vgg_cifar().summary()

This small VGG on CIFAR-10 climbs to around 84 % test accuracy with basic augmentation, on a laptop, in an hour. The original VGG16 (13 conv layers + 3 dense) and VGG19 on ImageNet 224 by 224 hold 138 M and 144 M parameters respectively; they will not fit training-side on a modest GPU without gradient checkpointing.

Where the mass goes: parameters versus activations

VGG makes the true cost of a CNN visible for the first time. Look at a summary of full VGG16 and split the accounting:

Zone	Parameters	Activations per image (float32)
Convolutional body	~15 M	very large: 224×224×64 alone is 12.8 MB
Dense head (4096, 4096, 1000)	~123 M	tiny

Parameters live in the dense head, activations live in the body. That distinction shapes every downstream trade-off:

• To reduce parameters, kill the dense head — that is what modules 3 and 6 do with global average pooling.
• To reduce memory during training, reduce the input resolution or the number of filters in the early layers, because those are the layers where the feature maps are largest.

Feature maps use most of the training memory

A backward pass has to store the activations of every layer to compute gradients. A batch of 64 images at 224 by 224 in the first VGG block produces $64 \times 224 \times 224 \times 64 \times 4$ bytes = 820 MB of feature maps for one layer alone. That is why real training rarely goes above batch 32 on a mid-range GPU with VGG-scale bodies.

The 1x1 convolution, quietly introduced

VGG did not use 1 by 1 convolutions, but the observation that made them possible comes from the same era. A 1 by 1 convolution acts pixel by pixel across channels: it is a per-position dense layer on the channel axis. It is used for two things you will meet in the next two modules:

1. Reduce or expand the number of channels before an expensive 3 by 3 operation (this is the bottleneck of ResNet, module 6, and the projection of Inception, module 7).
2. Mix channels without touching spatial structure, which is what depthwise separable convolutions do in MobileNet (module 7).

# 1 x 1 conv: dense layer applied at every spatial position.
x = tf.keras.Input(shape=(28, 28, 128))
y = tf.keras.layers.Conv2D(32, 1)(x)     # 28 x 28 x 32, cheap

Why VGG is still a reference baseline

Ten years later, VGG shows up almost every week in a paper. Why? Three reasons.

• Its feature maps are easy to interpret. Because no residual connections or branches mix activations, the ordering of layers matches the depth of features neatly. Perceptual losses in style transfer, image quality metrics like LPIPS, and many teacher networks for distillation still use VGG features.
• It is a fair comparison target. Every paper claims to beat ResNet50; comparing against a plain VGG is a sanity check that the gain does not come from a training trick.
• It exposes the two axes that later work optimises. Every architecture from VGG onwards can be described as "same VGG idea, but with $X$ to reduce parameters or activations". ResNet reduces optimisation depth, Inception reduces FLOPs, MobileNet reduces multiply-adds. Knowing VGG is knowing the baseline they all improve against.

PyTorch equivalent

The Keras block above translates to PyTorch line for line with nn.Conv2d, nn.MaxPool2d and nn.ReLU. The one detail that matters: in PyTorch, nn.Conv2d(in_channels, out_channels, kernel_size) takes the input channel count explicitly, whereas Keras infers it. That helps when debugging: a shape mismatch is caught at layer construction rather than at first call.

In summary

• Two stacked 3 by 3 convolutions cover the same 5 by 5 receptive field as a single 5 by 5, with fewer parameters and one extra non-linearity; three stacked 3 by 3 cover 7 by 7.
• VGG repeats a block of two or three 3 by 3 convs followed by a 2 by 2 max pool, doubling the number of filters at each downsampling.
• Parameters live in the dense head, activations live in the body; halving one does not halve the other.
• VGG remains a reference baseline for perceptual losses, feature extraction and paper-to-paper comparisons, even though its parameter count is now considered excessive.

Next module: ResNet, which asks a deeper question — why does simply stacking more VGG blocks make the training error worse, and what is the trick that makes a 152-layer network trainable.