Module 4 — LeNet and AlexNet: the first successes

Modules 1 to 3 built every ingredient of a CNN by hand. This module and the next three use those ingredients to reproduce four historical architectures, in order of appearance. The point is not nostalgia: every trick that survives in today's models — ReLU, dropout, augmentation, GPU training — was introduced in one of these papers, and understanding why they were needed is faster than reading the news.

LeNet-5: the 1998 recipe that still works

LeCun and colleagues designed LeNet-5 to read handwritten digits on bank cheques. It was trained on 28 by 28 grayscale digits (MNIST) and reached less than 1 % error at a time when most people were still solving MNIST with hand-designed features. The recipe was the whole modern CNN in miniature: convolution, subsampling, convolution, subsampling, then a small dense head.

Let us reproduce it in Keras, on MNIST, exactly as it appeared:

import tensorflow as tf

def lenet5():
    x = tf.keras.Input(shape=(28, 28, 1))
    h = tf.keras.layers.Conv2D(6, 5, padding="same", activation="tanh")(x)   # C1
    h = tf.keras.layers.AveragePooling2D(2)(h)                               # S2, 14 x 14
    h = tf.keras.layers.Conv2D(16, 5, activation="tanh")(h)                  # C3, 10 x 10
    h = tf.keras.layers.AveragePooling2D(2)(h)                               # S4, 5 x 5
    h = tf.keras.layers.Flatten()(h)
    h = tf.keras.layers.Dense(120, activation="tanh")(h)                     # F5
    h = tf.keras.layers.Dense(84, activation="tanh")(h)                      # F6
    out = tf.keras.layers.Dense(10, activation="softmax")(h)                 # OUTPUT
    return tf.keras.Model(x, out)

lenet5().summary()

Read the summary: about 60 000 parameters, six convolutional filters at the first layer, sixteen at the second. On a modern laptop it trains to 99 % test accuracy on MNIST in less than a minute. On the 32 by 32 colour images of CIFAR-10, however, it barely reaches 55 % — the receptive field is too small, the number of filters too tiny, and tanh gradients die in deeper stacks. That failure is exactly what motivated the next fourteen years of research.

Reading a layer table

Papers describe architectures in a table. Here is LeNet-5, rewritten in the format the CNN literature uses ever since:

Layer	Type	Kernel	Stride	Padding	Output shape	Parameters
Input	—	—	—	—	28×28×1	0
C1	Conv	5×5	1	same	28×28×6	156
S2	AvgPool	2×2	2	valid	14×14×6	0
C3	Conv	5×5	1	valid	10×10×16	2 416
S4	AvgPool	2×2	2	valid	5×5×16	0
F5	Dense	—	—	—	120	48 120
F6	Dense	—	—	—	84	10 164
Out	Dense	—	—	—	10	850

Where does the count come from? For C1, one 5 by 5 kernel on a single input channel has 25 weights, plus one bias, times six output filters: $6 \times (5 \times 5 \times 1 + 1) = 156$. For C3, six input channels: $16 \times (5 \times 5 \times 6 + 1) = 2\,416$. Get comfortable checking this by hand — module 5 will exploit the same arithmetic to compare 3 by 3 stacks with 5 by 5 kernels.

AlexNet: four decisions that unlocked ImageNet

Fourteen years later, at ImageNet 2012, AlexNet cut the top-5 error rate from 26 % to 15 % and changed the field overnight. The architecture was five convolutional layers and three dense ones — bigger than LeNet, but conceptually identical. What made the difference was a bundle of engineering choices, four of which we still use today:

1. ReLU instead of tanh or sigmoid. ReLU is $\max(0, x)$: it has no upper saturation, so gradients do not shrink to zero in deep stacks, and it is trivially cheap to compute. Training was six times faster than with tanh at the same accuracy. Every model in this course from now on uses ReLU or one of its variants.
2. Dropout in the dense layers. With 60 million parameters and only 1.2 million training images, AlexNet had to fight overfitting. Dropout randomly zeroes half of the activations at training time and forces the network to learn redundant representations. It disappears from convolutional layers in later architectures (module 6 explains why with batch normalisation), but survives in the head.
3. Data augmentation. Random crops, horizontal flips, and colour jitter — the augmentation techniques of module 8 — were introduced systematically by AlexNet as a training regime, not an afterthought.
4. Two GPUs. The 60 million parameters did not fit on a single 3 GB GPU of the era. Krizhevsky split the model across two cards, communicating between them at specific layers. The technique is obsolete; the lesson — that CNN training scales with hardware — is not.

def alexnet_like(num_classes=1000):
    # Simplified, single-GPU version. Real AlexNet uses 224 x 224 inputs.
    x = tf.keras.Input(shape=(224, 224, 3))
    h = tf.keras.layers.Conv2D(96, 11, strides=4, activation="relu")(x)
    h = tf.keras.layers.MaxPooling2D(3, strides=2)(h)
    h = tf.keras.layers.Conv2D(256, 5, padding="same", activation="relu")(h)
    h = tf.keras.layers.MaxPooling2D(3, strides=2)(h)
    h = tf.keras.layers.Conv2D(384, 3, padding="same", activation="relu")(h)
    h = tf.keras.layers.Conv2D(384, 3, padding="same", activation="relu")(h)
    h = tf.keras.layers.Conv2D(256, 3, padding="same", activation="relu")(h)
    h = tf.keras.layers.MaxPooling2D(3, strides=2)(h)
    h = tf.keras.layers.Flatten()(h)
    h = tf.keras.layers.Dropout(0.5)(h)
    h = tf.keras.layers.Dense(4096, activation="relu")(h)
    h = tf.keras.layers.Dropout(0.5)(h)
    h = tf.keras.layers.Dense(4096, activation="relu")(h)
    out = tf.keras.layers.Dense(num_classes, activation="softmax")(h)
    return tf.keras.Model(x, out)

Do not train AlexNet on your laptop: 60 M parameters and 1.2 M ImageNet images take a full day on a mid-range GPU. The point of the code above is to count parameters and shapes and see where the mass lives. Run alexnet_like().summary() and notice that the two 4 096-wide dense layers dominate: 40 million parameters out of 60. VGG will inherit this problem; ResNet will finally solve it.

ReLU does die too

A large gradient can push a weight so far that the neuron's pre-activation is negative for every input in the training set. The neuron then outputs zero forever, and its gradient is zero forever: it is dead. Symptom: many silent zeros in intermediate activations. Fix: a smaller learning rate, or a variant like Leaky ReLU or GELU.

Bringing it back to the red thread

Trained on our CIFAR-10 red-thread problem, LeNet-5 reaches about 55 % test accuracy and AlexNet-scale networks are overkill and overfit. Our small CNN from module 3 sits in between at around 75 %. The takeaway: architecture choices depend on the input resolution and the size of the dataset, not on the reputation of the model. Modules 5 and 6 will show two ways to close the gap that still make sense today.

In summary

• LeNet-5 already contains the modern CNN skeleton — convolution, subsampling, dense head — and still trains to 99 % on MNIST in under a minute.
• AlexNet did not invent CNNs; it added ReLU, dropout, augmentation and GPU training to make them work at ImageNet scale.
• A layer table describes an architecture unambiguously; predicting parameter counts and output shapes by hand is the fastest way to spot a wrong summary.
• Bigger is not always better: LeNet is too small for CIFAR-10, AlexNet is too big, and the right size depends on the data.

Next module: VGG's demonstration that a stack of small 3 by 3 filters beats a single large one, and why that observation shaped every architecture that came after.