Lesson 6 — Recap and frequent questions
The five lessons in five sentences
- An image reaches a computer as a grid of millions of numbers, and the difficulty is that lighting, viewpoint, scale and background change nearly all of them while the correct answer stays fixed.
- Convolution slides small learned filters across the image, which handles position for free and stacks into a hierarchy — edges, then shapes, then parts, then objects — that emerges from training rather than design.
- Four tasks sit behind "computer vision", and each step from classification to detection to segmentation to tracking multiplies annotation cost by roughly an order of magnitude.
- Architectures progressed by fixing specific obstacles, transformers traded convolution's helpful assumption for long-range attention, and foundation models changed the workflow more than the accuracy.
- Production failures come mostly from domain shift and shortcut learning, bias is measurable and hidden by aggregate metrics, and the only trustworthy score comes from images collected exactly as deployment images will be.
The pipeline, end to end
Note where the arrows return: to data collection, not to architecture selection. That is where the gains are.
A decision guide for a vision project
| Question to answer first | Why it decides the project |
|---|---|
| What decision does the output drive? | Determines the accuracy actually required, which is otherwise unknowable |
| What does each type of error cost? | False positives and false negatives are almost never equally expensive |
| Can a human do it from the image alone? | If not, no model can either, and you need different data |
| Is a single label per image enough? | Saves an order of magnitude in annotation |
| Under what conditions will images be captured? | Defines what training data must cover |
| Who reviews the output, and how often? | Human-in-the-loop lowers the accuracy bar dramatically |
| What happens when the model is unsure? | Rejection handling must be designed, not improvised |
Twelve frequent questions
1. How many images do I need?
Fine-tuning a pre-trained model typically needs a few hundred to a few thousand labelled images per class. The more useful framing is that representativeness dominates quantity: five hundred images spanning every camera, lighting condition and shift will outperform fifty thousand from one ideal setup. Start with a few hundred, measure, and let the error analysis tell you what to collect next.
2. Can I use images scraped from the internet?
For pre-training and feasibility testing, yes and it is standard. For your actual task, they are a weak substitute, because internet photographs are well-lit, centred and unlike a factory camera at 3 a.m. There are also licensing questions worth checking before anything ships.
3. What is the difference between a CNN and a vision transformer, practically?
The convolutional network assumes nearby pixels matter most and patterns can appear anywhere, which is free information that helps when data is scarce. The transformer assumes nothing and relates any patch to any other directly, which wins when data is plentiful. With a few thousand images, start convolutional; with millions, transformers.
4. Does my model need to run in real time?
Less often than teams assume. "Real time" usually means fast enough that the answer arrives before it is needed, which for a quality gate might be two seconds and for a driving system twenty milliseconds. Establishing the real number early prevents a lot of unnecessary optimisation.
5. Can a vision model run on a phone or a small device?
Yes. Quantisation, pruning and distillation routinely fit useful models into a few megabytes running on phone or microcontroller hardware. Accuracy costs a little, and the gains in latency, privacy and offline operation frequently outweigh it.
6. How do I know whether the model is looking at the right thing?
Saliency and attention visualisations highlight the image regions that most influenced the prediction. They are approximations rather than explanations, and they reliably catch the embarrassing cases — a model reading the background, the timestamp or a ruler placed in the frame.
7. My model works in the lab and not in the field. What now?
Almost certainly domain shift. Collect a few hundred images from the field, look at them next to your training images, and the difference is usually obvious within minutes — different camera, different lighting, dirtier lens, different angle. Add field images to training and retrain. Do not start by trying a bigger model.
8. Is facial recognition accurate?
Under controlled conditions with cooperative subjects and good images, extremely. In uncontrolled conditions — surveillance footage, poor lighting, angles — considerably less, and the error rate varies substantially by demographic group. Since the applications proposed for it are usually the uncontrolled kind, quoted laboratory accuracy is close to meaningless as a guide to deployment.
9. Can I detect something I have almost no examples of?
Sometimes, by inverting the problem: train on what normal looks like and flag anything unlike it. Anomaly detection avoids needing examples of each defect, and it produces more false alarms and cannot tell you what it found. For rare but known defects, synthetic data and heavy augmentation are worth trying.
10. What is data augmentation and does it help?
Creating additional training examples by transforming existing ones — rotating, cropping, adjusting brightness, adding noise. It helps meaningfully and cheaply, and only for variations that genuinely occur: flipping images horizontally is sensible for animals and wrong for text or for anything where left and right carry meaning.
11. Do I need to label images myself?
Someone with domain knowledge must define the guidelines and label the difficult cases, and that part cannot be outsourced usefully. Bulk labelling can be, and assisted tools now pre-generate boxes and masks for correction, which cuts the effort substantially. Inconsistent labelling is a common and expensive failure — two annotators disagreeing about what counts as a defect puts a ceiling on the model.
12. Where should I start if I want to build this?
Take a hundred of your own images, fine-tune a pre-trained classifier, and look carefully at what it gets wrong. That exercise teaches more than any amount of reading, and the error analysis will tell you whether the project is about data, about task definition or about the model. The premium catalogue covers the full path with notebooks, datasets and deployment.
Vocabulary you can now use precisely
| Term | Meaning in one line |
|---|---|
| Pixel | One position in the image grid, holding one number per channel |
| Channel | One numerical layer of an image, such as red, green or depth |
| Convolution | Sliding a small learned filter over the image to detect a local pattern |
| Feature map | The output of a filter, showing where its pattern occurs |
| Pooling | Shrinking a feature map by keeping the strongest local response |
| Receptive field | The region of the original image influencing one unit |
| Residual connection | Adding a block's input back to its output, making great depth trainable |
| Transfer learning | Starting from a pre-trained model and adapting it to your task |
| Intersection over union | Overlap between predicted and true regions, divided by their union |
| Domain shift | Production images differing from training images in some relevant way |
| Shortcut learning | Relying on an incidental correlation instead of the intended signal |
| Quantisation | Storing weights at lower numerical precision to shrink and speed up a model |
And now?
You have the map. Choosing the right task, judging feasibility, and knowing which failure to look for first will serve you in most conversations about a vision project.
Validate it: take the 5-question quiz →
Continue free: Generative AI covers producing images rather than reading them.
Go deeper: the premium catalogue covers convolutional networks, detection, segmentation and edge deployment hands-on, with a verifiable certificate after a 40-question examination.