Module 8 — Data augmentation for images

From this module onwards, the red thread changes. We move away from CIFAR-10 and onto a realistic case study: classifying photographs of household waste into six classes — cardboard, glass, metal, paper, plastic, general trash. Roughly 500 images per class, taken on plain backgrounds, sometimes at different angles. That is a hundred times less data than ImageNet, and the network we plan to fine-tune in module 9 has 25 million parameters. Without augmentation, it will overfit before the first epoch is over. This module makes the augmentation pipeline explicit, and shows the four traps that turn a helpful augmentation into a label-corrupting one.

The core toolbox

Six operations cover 95 % of image augmentation in practice. Each one preserves the object identity while changing its appearance.

Operation	What it changes	Typical parameter
Horizontal flip	mirror around the vertical axis	50 % probability
Random crop	crop a random sub-window, resize back	crop 80 to 100 % of the area
Colour jitter	brightness, contrast, saturation, hue	±10 to 20 % on each
Rotation	small angle, ideally under ±20 degrees	random within a range
Translation	small horizontal or vertical shift	±10 % of the size
Additive noise	Gaussian or salt-and-pepper	very small standard deviation

The purpose is not to invent new content; it is to teach the network the irrelevant variations it should be robust to. A photograph of a plastic bottle should still be classified as plastic if the light is warmer, if it is slightly tilted, if the camera moved five centimetres to the left. Augmentation encodes those invariances into the training signal.

Cutout and Mixup: two ideas that survived

Beyond the basics, two augmentations that appeared around 2017 keep showing up in state-of-the-art recipes.

Cutout masks a random rectangle of the image with zeros. The network learns not to rely on any single region: if the eye of the cat is masked, it has to use the ears, the fur pattern and the paw. On our waste dataset, Cutout forces the model to look at multiple parts of a bottle rather than latch onto the cap. Typical rectangle size: 20 to 40 % of the image side.

Mixup takes two training images and their one-hot labels, and linearly combines both. A 60/40 mix of a bottle and a can produces an image that is 60 % bottle plus 40 % can, with a soft label of 0.6 bottle and 0.4 can. The network learns smoother decision boundaries, which reduces overconfidence and often improves calibration by more than accuracy.

import tensorflow as tf

def mixup(images, labels, alpha=0.2):
    beta = tf.compat.v1.distributions.Beta(alpha, alpha)
    lam = beta.sample([])
    idx = tf.random.shuffle(tf.range(tf.shape(images)[0]))
    mixed_images = lam * images + (1.0 - lam) * tf.gather(images, idx)
    mixed_labels = lam * labels + (1.0 - lam) * tf.gather(labels, idx)
    return mixed_images, mixed_labels

CutMix is a hybrid: paste a random rectangle from image B onto image A, and use a soft label proportional to the pasted area. It combines the localisation regularisation of Cutout with the label mixing of Mixup, and empirically outperforms both on ImageNet.

Augmentation as part of the model

Traditionally, augmentation was written in the data pipeline: read the file, decode, augment, batch. Modern Keras suggests a different pattern — make the augmentation a set of layers inside the model itself:

data_augmentation = tf.keras.Sequential([
    tf.keras.layers.RandomFlip("horizontal"),
    tf.keras.layers.RandomRotation(0.05),        # ~ ±18 degrees
    tf.keras.layers.RandomZoom(0.1),
    tf.keras.layers.RandomContrast(0.15),
], name="augmentation")

# Build a model that includes augmentation on the training side only.
inputs = tf.keras.Input(shape=(224, 224, 3))
x = data_augmentation(inputs)                    # active only when training=True
x = tf.keras.applications.resnet50.preprocess_input(x)
backbone = tf.keras.applications.ResNet50(weights=None, include_top=False)
x = backbone(x)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
outputs = tf.keras.layers.Dense(6, activation="softmax")(x)
model = tf.keras.Model(inputs, outputs)

Three benefits: augmentation runs on the accelerator (GPU or TPU), avoiding CPU-bound preprocessing bottlenecks; the augmentation layers are exported with the model so serving code cannot forget them; and the model automatically disables augmentation at inference, because Keras propagates training=False.

Order matters, and so does randomness

The order of augmentation operations changes the distribution of samples. Rotate then crop and you may cut off the object; crop then rotate and the corners fill with black. Colour jitter after normalisation shifts pixel intensities in the wrong space. A safe order for most pipelines:

1. Geometric transforms (flip, rotate, crop, translate) on the raw image.
2. Photometric transforms (brightness, contrast, colour jitter) on the raw image.
3. Framework-specific preprocessing (mean subtraction, scaling to [-1, 1]) after augmentation.

Randomness is uniform, but reproducibility is a real concern: set a seed on each augmentation layer so that runs are comparable, and log the augmentation configuration alongside the model checkpoint.

What you must never augment

The one lesson that saves the most projects: augmentation must preserve the label. Here are four traps that everyone falls into once.

• Digits and letters. A horizontal flip turns 6 into a mirror image that is not 9 but also not 6; a rotation turns 6 into 9 and vice versa. On MNIST, RandomFlip("horizontal") collapses class 6 and its mirror into a single unlabelable blob.
• Medical images with side markers. Chest X-rays are labelled left/right by the radiologist. Flipping them silently swaps sides and creates false labels — a scandal that took years to be documented in the deep-learning-for-medicine literature.
• Documents and receipts. Rotations beyond a few degrees change the reading order for downstream OCR; augmenting a scan by 90 degrees teaches the model that upside-down text is normal, which then breaks in production.
• Colour-dependent classes. On our waste dataset, a strong colour jitter that turns a brown cardboard box into a greenish one is fine; but on a task like traffic sign classification, red-to-green colour jitter converts a stop sign into a go sign and destroys the label.

Augment the input, never the label

The invariant is: whatever you do to the input must be something the human labeller would still call the same class. Everything else — flip on symmetric objects, small rotations, small crops, small colour jitter — is fair game. Everything else — big rotations on asymmetric objects, flips on directional objects, colour jitter on colour-defined classes — is a mislabelling in disguise.

Bringing it back to the red thread

On the waste dataset, a sensible pipeline is: horizontal flip (bottles look the same from either side), small rotation (±15°), random zoom (±10 %) and mild colour jitter (±15 % contrast). No vertical flip — a bottle standing up and a bottle lying down are not the same photograph in a bin. No large rotation — pieces of waste on a conveyor belt are usually upright. With this pipeline, our ResNet50 in module 9 will train without overfitting for the twenty epochs the case study needs.

Look at your augmented images

Before every training run, sample 16 augmented images and display them. Ninety percent of "the model does not converge" bug reports on new datasets are actually augmentation configurations that produce unrecognisable inputs. One plt.imshow grid saves hours.

In summary

• Augmentation encodes irrelevant variations into the training signal; it does not create new data, it teaches invariance.
• Cutout, Mixup and CutMix are three augmentations from 2017-2019 that keep appearing in state-of-the-art recipes for their regularisation effect.
• Prefer in-model augmentation layers so that augmentation runs on the accelerator and is exported with the model.
• Never augment away a label: no flips on digits and directional objects, no rotations on documents, no strong colour jitter when colour defines the class. Look at your augmented images every time.

Next module: transfer learning on the waste-sorting case study, and the two-phase fine-tuning procedure that unlocks the accuracy of a pretrained ResNet50 without destroying it.