Lesson 5 — Where it still fails
Benchmark results make vision look finished. Production tells a different story, and the failures follow a small number of recognisable patterns. Knowing them is the difference between a project that ships and one that quietly gets shelved.
1. Domain shift, the most common cause of failure
A model performs well in testing and poorly in production because production images differ from training images in some way nobody wrote down.
Documented cases follow the same shape repeatedly. A pneumonia detector trained on scans from several hospitals learned to use the scanner model as a signal, because one hospital had sicker patients and a distinctive machine; on a new hospital's scanner, performance dropped sharply. A skin lesion classifier learned that dermatologists place a ruler beside lesions they suspect are malignant. A cattle detector failed when the animals stood on sand rather than grass.
The mechanism has a name: shortcut learning. A model has no notion of what you meant to detect. It finds whatever correlates with the label in the training data, and if the background, the equipment or an annotation artefact correlates more reliably than the object, it will use that instead. Nothing in the training process penalises it — the shortcut works on the test set too, because the test set shares the same artefacts.
What actually helps:
- Collect training data across the full range of real conditions: every camera, every lighting condition, every site, every shift.
- Hold back a test set from a different source than the training data — a different site, a different month, a different camera. This is the single most informative thing you can do.
- Look at what the model attends to, not only at its score. Attention and saliency maps regularly reveal a model reading the background.
- Monitor production inputs for drift and retrain when conditions change. Cameras get dirty, lighting is replaced, suppliers change packaging.
2. Adversarial examples
Small, deliberate changes to an image, often imperceptible to a person, can flip a model's prediction with high confidence. Printed patterns on a sticker can cause a detector to miss a stop sign. Patterned clothing can prevent a person detector from registering a person.
Two things are worth separating here.
The security implication is real but situational. It matters where an adversary has an incentive and access: content moderation, biometric access, automated payment or inspection systems that someone would like to fool. It matters much less for an inspection camera inside a factory.
The scientific implication is broader. That such small perturbations work at all indicates these models rely on features that are predictive in the training distribution but not the ones humans use. It is the same underlying issue as shortcut learning, in a more dramatic form.
Defences exist — adversarial training, input preprocessing, ensembles — and none is complete. If your application has an adversary, treat robustness as an ongoing engineering commitment rather than a box to tick.
3. Bias, which is measurable and consequential
Vision datasets reflect who assembled them, and the resulting disparities are documented rather than theoretical.
Commercial gender classification systems audited in 2018 showed error rates under 1 % for lighter-skinned men and over 30 % for darker-skinned women. Pedestrian detection has been shown to perform less well on darker-skinned pedestrians. Medical imaging models trained predominantly on one population transfer poorly to others.
The causes are mundane and therefore fixable in principle: training images skewed towards particular populations, annotators applying inconsistent judgement, and evaluation reported as a single aggregate number that averages away the failure.
What actually helps:
- Report accuracy broken down by subgroup, always. An aggregate figure conceals precisely the disparity you need to see.
- Audit datasets for representativeness before training, not after an incident.
- Ask what a false positive and a false negative cost for each affected group, since they are rarely symmetric.
- Accept that some applications should not be built. Face recognition for policing has an error profile that falls hardest on people least able to contest it, which is a design fact rather than a bug awaiting a patch.
This is treated at length in Ethics of AI.
4. Small objects, rare events and long tails
Models perform well on what is common and large in the frame, and poorly on what is small and rare — which is frequently the point of the system.
A defect covering twenty pixels, an event occurring once in ten thousand frames, an unusual failure mode absent from the training set: all are hard, and all are exactly what an inspection system exists to catch. Averaged metrics hide this completely, because the common cases dominate.
Mitigations: work at higher resolution or tile the image; oversample rare classes during training; consider anomaly detection, which flags anything unlike normal rather than requiring examples of each defect; and always report performance separately for the rare classes.
5. Evaluation that flatters
Beyond the metric issues in lesson 3, three habits produce optimistic numbers that do not survive contact with production:
Random splits with correlated images. If your dataset contains several frames of the same object and you split randomly, near-duplicates land in both training and test sets. The score measures memorisation. Split by object, by session or by site — never by row.
Tuning on the test set. Comparing twenty configurations on the same test set and reporting the best is selection on noise. Keep a genuinely untouched set for the final number.
Public benchmark scores as a proxy. Widely used datasets contain label errors, and models have been optimised against them for years. A leading benchmark score says little about your images.
Two hundred images collected the way production images will be collected, labelled carefully, and never used for tuning. That number will be lower than your validation score and it will be true, which makes it the only one worth quoting to anyone who has to make a decision.
What vision genuinely cannot do yet
Worth stating plainly, since it is often oversold:
- Open-ended understanding. "Tell me if anything unusual is happening" has no reliable implementation. Narrow, specified anomalies do.
- Reliable causal or physical reasoning. Models predict what tends to co-occur, not what causes what. Vision-language models have improved at describing scenes and still fail on questions requiring physical inference.
- Robust generalisation from few examples. Humans recognise a new object category from a handful of views. Models need far more, or a pre-trained representation that already covers something similar.
- Knowing when they do not know. Confidence scores are poorly calibrated, and a model asked about something entirely outside its training will often answer confidently. Rejecting uncertain inputs has to be engineered deliberately.
In three sentences
The dominant cause of production failure is domain shift combined with shortcut learning: a model latches onto whatever correlates with the label — the scanner, the background, an annotation artefact — and since the test set shares those artefacts, the failure is invisible until deployment, which is why a test set from a genuinely different source is the most informative thing you can build. Bias is measurable and consequential rather than hypothetical, with audited error-rate gaps of thirty points between demographic groups, and it stays hidden as long as accuracy is reported as a single aggregate number. Small objects, rare events, adversarial inputs and poorly calibrated confidence complete the picture, and the only performance estimate worth quoting is one computed on a couple of hundred images collected exactly as production images will be and never used for tuning.
Next — Lesson 6: recap and FAQ →