Module 1 — The three families of image tasks
Course 10 taught what a convolutional network computes on a single image. This course teaches what you ask that network to output. And the first question, before any architecture, is which of the three families of image tasks your problem actually belongs to.
The running example throughout the course is a traffic crossroads camera: every module contributes one piece to the pipeline that will, by module 10, count vehicles and pedestrians in real time. This first module chooses the labels.
Classification, detection, segmentation
Three families, three answers of increasing granularity.
Classification answers what is in the image, and nothing more. One label per image. A crossroads photo becomes "crowded" or "empty". A cell scan becomes "healthy" or "abnormal". The output is a probability vector over a fixed set of classes.
Object detection answers what, and where. For every object of interest the model outputs a bounding box and a class. Our crossroads camera returns car (0.94) at pixels (x1, y1, x2, y2), then pedestrian (0.72) at another box, and so on. This is the family that lets us count.
Segmentation answers what, and which pixel belongs to it. Each pixel receives a label. It divides further:
- Semantic segmentation labels the class of every pixel but does not distinguish objects of the same class. All car pixels form one region called "car".
- Instance segmentation distinguishes them:
car_1,car_2,car_3. Every object has its own pixel mask. - Panoptic segmentation unifies both: countable objects (cars, people) get instance masks, while non-countable regions (road, sky, grass) get a single semantic label.
The crossroads uses all three: detection to count vehicles (module 2 to 5), semantic segmentation to identify the road surface and pavements (module 6), instance segmentation when two cars overlap (module 7).
Choosing between the three
Cost and information do not grow linearly. A useful rule of thumb:
| Task | Output per image | Annotation cost, relative | When it is enough |
|---|---|---|---|
| Classification | 1 label | 1 | You only need presence, not location |
| Detection | N boxes + N classes | 8 to 15 | You need to count, or to localise coarsely |
| Semantic segmentation | 1 mask per pixel | 30 to 80 | You need surface area or fine contact |
| Instance segmentation | 1 mask per object | 40 to 100 | You need both count and shape |
A frequent mistake is to reach for segmentation because the demo videos look impressive. If the downstream system only needs to know "how many pedestrians crossed", detection is enough and costs an order of magnitude less to annotate.
Before you choose a task family, write the exact decision your system will make: an alarm, a count, an area in square metres, a route. If the decision only reads one number per image, classification is enough. If it needs a location, detection. If it needs a shape, segmentation. This one exercise saves months of over-specified annotation.
Annotation formats: COCO, Pascal VOC, YOLO
Three formats dominate, and the choice matters because it constrains the tools you can chain later.
COCO stores everything in a single JSON. Images, categories, annotations (boxes, masks, keypoints) live in three top-level arrays linked by image_id and category_id. This is the format the module 5 metrics assume, and the one the crossroads pipeline exports.
import json
with open("annotations/instances_train.json") as f:
coco = json.load(f)
print(len(coco["images"])) # 5000
print(coco["categories"][:2]) # [{'id': 1, 'name': 'car'}, {'id': 2, 'name': 'pedestrian'}]
print(coco["annotations"][0])
# {'image_id': 42, 'category_id': 1, 'bbox': [312.0, 108.0, 84.0, 60.0], 'area': 5040.0, ...}
Bounding boxes in COCO are [x, y, width, height] in pixels, with the origin at the top left.
Pascal VOC uses one XML file per image, boxes as [xmin, ymin, xmax, ymax] in pixels. It is verbose, harder to inspect at scale, but well supported by torchvision.
YOLO uses one plain-text file per image, one line per object: class_id cx cy w h, where every coordinate is normalised between 0 and 1 relative to the image size. It is the format Ultralytics tools consume natively (module 3 and 10).
The annotation cost trap
The visible cost of annotation is money. The hidden cost is time, and it dominates. A rough figure to keep in mind:
- Classification: 5 to 10 seconds per image
- Detection: 30 to 60 seconds per image with 3 to 5 objects
- Instance segmentation: 5 to 15 minutes per image with a polygon tool
Multiply by ten thousand images and instance segmentation becomes a six-figure line item. This is why the pipeline in module 7 relies heavily on Segment Anything to produce mask candidates that annotators only need to accept or correct, rather than draw from scratch.
The other hidden cost is inter-annotator disagreement. Ask three people to draw a bounding box around a car partially hidden behind a sign, and you will get three different boxes. Module 9 turns this into a metric: any dataset where the same object receives incompatible labels from two annotators will limit model performance by that same margin. No architecture recovers from noisy labels.
Converting from COCO to YOLO is trivial with a script; converting halfway through a project, when part of the dataset uses one convention and the rest another, silently doubles the bug surface. Choose one at project start and enforce it in the loader.
In summary
- Three families of tasks form an increasing hierarchy of information: classification says what, detection says what and where, segmentation says which pixel belongs to what.
- The crossroads pipeline in this course uses all three: detection to count, semantic segmentation for the road surface, instance segmentation when objects overlap.
- Three annotation formats dominate: COCO (JSON, pixels), Pascal VOC (XML, pixels), YOLO (text, normalised); choose one and never mix them within a project.
- Annotation cost grows by roughly an order of magnitude at each family; reverse-engineering the label from the actual decision your system will make is the single most effective way to avoid over-specified data.
Next module: two-stage detection with Faster R-CNN, our first serious detector on the crossroads dataset.