Lesson 3 — The four tasks
"We need computer vision" is not a specification. Four distinct tasks sit behind that phrase, they differ by an order of magnitude in annotation cost, and choosing the wrong one is one of the more expensive mistakes available in this field.
Task 1 — Classification: what is in this image?
Input: one image. Output: one label, with a confidence.
This X-ray shows signs of pneumonia. This product photograph is a shoe. This weld is defective.
The simplest and cheapest task by a wide margin. Annotation means someone looking at an image and choosing a category, so a few thousand images can be labelled in an afternoon.
Choose it when one decision per image is enough. Note that this covers far more real problems than teams initially assume — "is there a defect anywhere in this photograph?" is a classification problem, and answering it does not require knowing where.
Watch out for: images containing several relevant things. Classification returns one answer and will pick one arbitrarily. If that matters, you need detection.
Task 2 — Detection: what is in this image, and where?
Input: one image. Output: a list of boxes, each with a label and a confidence.
Four cars, two pedestrians, one traffic light, with a rectangle around each.
Detection is what you need to count things, to locate them, or to crop them out for further processing. Annotation means drawing a rectangle around every relevant object in every image — perhaps ten to thirty times slower than classification, and considerably more error-prone, because annotators disagree about partially visible objects.
Choose it when you need counts, positions, or several objects handled independently.
Watch out for: crowded scenes and small objects. Fifty overlapping objects, or objects twenty pixels wide, are where detection quality collapses and where you should test before promising anything.
Task 3 — Segmentation: which pixels belong to what?
Input: one image. Output: a label for every pixel.
Two variants, and the difference matters:
- Semantic segmentation: every pixel gets a class. All cars are "car". You learn how much of the image is road, and not how many cars there are.
- Instance segmentation: every pixel gets a class and an object identity. Car 1, car 2, car 3, each with its exact shape.
Choose it when the precise shape or area matters: tumour boundaries, crop coverage, surface area of corrosion, background removal.
Watch out for the annotation cost, which is the real constraint. Outlining objects pixel by pixel takes minutes per image rather than seconds. A thousand segmentation masks is a serious project. This is why the right question is usually "can a box do the job?" before "can we afford masks?".
Task 4 — Tracking: is that the same object as before?
Input: a video. Output: object identities maintained across frames.
Vehicle #7 entered at 0:12, crossed the line at 0:19, exited at 0:24.
Tracking is detection plus identity over time, and identity is where it gets hard. Objects disappear behind others and reappear, cross paths, change appearance as they turn. A tracker that swaps two identities produces a plausible-looking output that is quietly wrong, which makes tracking failures unusually easy to miss.
Choose it when you need trajectories, dwell time, or counts of unique objects rather than counts per frame.
What it costs, side by side
| Task | Annotation per image | Output | Typical use |
|---|---|---|---|
| Classification | 2–5 seconds | One label | Pass/fail, sorting, triage |
| Detection | 30–120 seconds | Boxes and labels | Counting, locating, cropping |
| Semantic segmentation | 2–10 minutes | Pixel classes | Area measurement, background removal |
| Instance segmentation | 5–20 minutes | Pixel classes plus identity | Medical imaging, precise metrology |
| Tracking | Detection cost plus identity linking | Trajectories | Traffic, retail analytics, sport |
The pattern is consistent: each step up the ladder multiplies the labelling budget. Since labelling is usually the largest line item in a vision project, the discipline of solving the problem with the cheapest sufficient task is worth real money.
Measuring these honestly
Each task has its own metrics, and using the wrong one hides failure.
Classification. Accuracy is misleading whenever classes are imbalanced — with 2 % defects, a model that always says "fine" scores 98 %. Use precision and recall per class, and look at the confusion matrix to see which classes get mixed up.
Detection. The standard metric is mean average precision, built on intersection over union: the overlap between predicted and true boxes divided by their union. A threshold, commonly 0.5, decides what counts as a hit. Two consequences worth knowing: a single number hides whether the model misses small objects, and a model can score well while being useless at the size range you care about.
Segmentation. Also intersection over union, computed over pixels. Beware that large regions dominate the average, so a model can look excellent while consistently missing the small structures that motivated the project.
Tracking. You need both detection quality and identity quality. A tracker with excellent boxes and frequent identity swaps looks fine frame by frame and produces nonsense trajectories.
Whatever the metric, compute it on images collected the way deployment images will be collected — same cameras, same lighting, same time of day, same operators. A benchmark score on clean public data tells you almost nothing about your problem, and a hundred honestly-collected examples tell you a great deal.
Tasks beyond the four
Worth knowing they exist, and rarely the starting point:
- Pose estimation — locating body joints, for sport analysis, physiotherapy and ergonomics.
- Depth estimation — inferring distance from a single image.
- Optical character recognition — reading text from images, effectively solved for printed text and still hard for handwriting.
- Image retrieval — finding visually similar images, which powers visual search and duplicate detection.
- Anomaly detection — flagging anything unlike the training data, valuable precisely when you cannot enumerate the defects in advance.
In three sentences
Computer vision splits into classification (one label per image), detection (boxes with labels), segmentation (a label per pixel) and tracking (identity across video frames), and each step up that ladder multiplies annotation cost by roughly an order of magnitude. Since labelling is normally the largest expense in a vision project, the discipline that saves money is asking whether a cheaper task solves the actual problem — many "we need segmentation" requirements turn out to be classification questions. Each task also needs its own honest metric, computed on images collected under deployment conditions, because a single averaged score routinely hides failure on exactly the small or rare cases that motivated the work.