Module 9 — Annotation, augmentation and dataset quality

Modules 2 to 8 spent almost all their time on models. In practice, on the crossroads project as on any real vision problem, model work is a fraction of the effort. Most of it goes into labels and their integrity: choosing an annotation tool, agreeing on a labelling protocol, measuring how much annotators disagree, augmenting without corrupting the labels, and detecting the near-duplicate images that quietly leak from the training set into the test set. This module packages that hidden foundation.

Annotation tools, briefly

Half a dozen tools cover 95 % of vision annotation work. A rough map, biased towards what supports the running crossroads example well.

Tool	Formats	Strength	When to use
CVAT	COCO, YOLO, VOC, mask	Web, teams, video	Multi-annotator projects with review
Label Studio	COCO, YOLO, JSON, mask	Very flexible, extensible	Mixed data types, ad hoc taxonomies
Labelme	JSON, VOC	Simple desktop, polygons	Small solo projects
Roboflow	COCO, YOLO, mask	Managed, augmentation baked in	Fast prototyping, no infra to run
SAM-based tools	Mask	Interactive mask refinement	Instance segmentation at scale

For the crossroads, CVAT is the reference choice: it supports video annotation with propagation between frames, review workflows, and both boxes and polygons.

Inter-annotator agreement

Give the same image to two annotators. They will not draw identical boxes. Some disagreement is legitimate — a partially occluded vehicle can be labelled or ignored — some is noise. Measuring the disagreement matters because it caps model performance: no model can score better than the labels it was trained against.

For detection, the practical measure is agreement IoU: for each object matched between the two annotators, compute IoU of their boxes; take the median. A dataset with median agreement IoU below 0.85 is noisy and no model above that ceiling will train stably.

For classification decisions (car versus truck, pedestrian versus cyclist), Cohen's kappa on the class labels quantifies agreement corrected for chance:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

where $p_o$ is observed agreement and $p_e$ expected under independence. A kappa below 0.6 signals that the labelling protocol is unclear; the fix is not to hire more annotators but to rewrite the guidelines.

Writing a labelling guide

A guideline document that fits on one page eliminates 80 % of annotator disagreement. Three sections, in order:

1. Class definitions with edge cases. Do not just write "car" — write "any four-wheeled motor vehicle up to 3.5 tonnes; motorbikes are separate; delivery vans belong to car unless larger than a Sprinter". Cover the confusing cases explicitly.
2. Box conventions. Tight around visible pixels, or include the occluded parts? Include wing mirrors or not? Cut off at the image edge or extrapolate? Every project answers these differently; write the answer.
3. Reject conditions. When to skip an image entirely (too dark, license plate unreadable at the required resolution, ambiguous object). Skipping is a valid label and should be recorded, not left implicit.

Attach two or three worked-out examples to each rule. Ambiguous cases resolve faster against an example than against a definition.

Augmentation, and the box problem

Augmentation multiplies dataset diversity without new annotation. On classification, torchvision.transforms is enough. On detection and segmentation, every transformation that moves pixels must also move the corresponding labels, or the boxes end up on empty background. torchvision.transforms.v2 and Albumentations both handle this; Albumentations dominates the ecosystem.

import albumentations as A
from albumentations.pytorch import ToTensorV2

transform = A.Compose(
    [
        A.RandomResizedCrop(size=(640, 640), scale=(0.6, 1.0)),
        A.HorizontalFlip(p=0.5),
        A.RandomBrightnessContrast(p=0.3),
        A.MotionBlur(blur_limit=7, p=0.2),
        A.CLAHE(p=0.2),
        ToTensorV2(),
    ],
    bbox_params=A.BboxParams(
        format="coco",              # [x, y, w, h] in pixels
        label_fields=["labels"],
        min_visibility=0.3,         # drop boxes cropped below 30 % visible
    ),
)

augmented = transform(image=image, bboxes=bboxes, labels=labels)

Two augmentations to think twice about on the crossroads:

• HorizontalFlip flips traffic directions. If your model uses the direction of motion (module 8's counting line distinguishes "in" and "out"), a horizontally flipped training image trains the model to accept both directions symmetrically. Usually fine for a detector, potentially wrong for a downstream classifier that reads direction.
• VerticalFlip flips gravity. Cars become upside down. Almost always harmful for real-world imagery; useful for medical or satellite images where up has no meaning.

Mosaic and mixup, for detection

Two augmentations were designed specifically for detection.

Mosaic (YOLOv4) tiles four training images into one, then resizes back. The composite contains four times as many objects, in every scale and every position, at every corner boundary. It is the single most impactful augmentation for detection since its introduction, and it is on by default in Ultralytics YOLO.

Mixup linearly blends two images and their labels. It works less well for detection than for classification, because a half-transparent car is not something you ever want the model to declare with high confidence; recent detectors either disable it or use a variant.

Turn mosaic off in the last epochs

Mosaic distorts real image statistics: real crossroads images do not contain four half-images tiled together. Training on mosaic all the way to the end shifts the model away from the real distribution. YOLOv8 disables mosaic in the last 10 epochs by default, and you should keep this behaviour when writing a custom training loop.

Near-duplicate leakage between train and test

The most dangerous data-quality bug in vision is invisible at model training time: images that appear in both train and test, sometimes exactly, more often as near-duplicates. On the crossroads, this happens when frames 100 and 101 of the same video end up on opposite sides of the split. The two frames are 40 milliseconds apart; the model sees frame 100 during training and is tested on frame 101, which is essentially the same image.

Reported mAP on such a split is inflated by 10 to 30 points versus true generalisation.

The fix is a video-aware split: entire video clips go into a single split, never split within a clip. When the source is a stream, sample at wide intervals (one frame per 30 seconds) and record the source frame index to detect proximity later.

For image collections without video structure, compute a perceptual hash (imagehash or CLIP embeddings) and cluster before splitting; every cluster goes entirely on one side.

from imagehash import phash
from PIL import Image
from collections import defaultdict

hashes = defaultdict(list)
for path in all_paths:
    h = str(phash(Image.open(path), hash_size=16))
    hashes[h].append(path)

clusters = list(hashes.values())    # near-duplicates share the same hash

A dataset card catches problems no metric will

Write a one-page card for every dataset you build: source, capture conditions, class distribution, split strategy, known biases, annotation protocol, agreement IoU. The mere exercise of writing it exposes gaps — "we do not know how the split was made", "night images are 3 % of train and 40 % of test" — that no downstream metric would ever surface until deployment.

In summary

• Annotation quality caps model performance: a dataset with median inter-annotator IoU below 0.85 or Cohen's kappa below 0.6 needs a better labelling guide, not more data.
• Albumentations and torchvision.transforms.v2 handle box- and mask-aware augmentation; label-unaware augmentation silently moves objects off their boxes.
• Mosaic is the single most impactful detection augmentation; disable it in the last epochs, never leave VerticalFlip on for scenes with gravity.
• Near-duplicate leakage between train and test inflates reported mAP by tens of points; split videos entire, or cluster near-duplicates by perceptual hash before splitting image collections.

Next module: the final project — the whole crossroads pipeline, from a custom dataset to an evaluated, exported detector.