Lesson 1 — What a computer sees
Look at a photograph of a cat. You know it is a cat before you have finished registering that you are looking at anything. The recognition is so fast and so effortless that it takes deliberate effort to appreciate that anything happened at all.
Now consider what the computer received.
An image is a grid of numbers
A digital image is a rectangular grid of pixels, and each pixel is a number describing brightness. In a greyscale image, one number per pixel, conventionally 0 for black through 255 for white. In a colour image, three numbers per pixel: how much red, how much green, how much blue.
A modest 1000 × 1000 colour photograph is therefore three million numbers. That is the entire input. No labels, no outlines, no hint that some of those numbers belong to a cat and others to a sofa.
| What you see | What the computer receives |
|---|---|
| A cat on a sofa | 3,000,000 integers between 0 and 255 |
| The cat's ear | A region where values change abruptly |
| "That's fur" | A statistical texture in local pixel variation |
| The edge of the sofa | A line along which values shift |
Everything meaningful — objects, boundaries, depth, identity — has to be reconstructed from those numbers. Nothing about the format helps.
Channels, and why they matter less than you expect
The three colour channels are the standard case, and vision extends well beyond them:
- Greyscale: one channel. Often enough — many industrial inspection systems discard colour deliberately, because it varies with lighting and adds nothing to the task.
- RGBA: four channels, the fourth being transparency.
- Medical imaging: a CT scan is a stack of slices, effectively a three-dimensional grid.
- Satellite imagery: often ten or more channels, including infrared bands invisible to the eye and highly informative about vegetation and water.
- Depth cameras: one channel where the value is distance rather than brightness.
The practical point is that a vision model does not care what the channels mean. It receives a grid of numbers with some depth and learns whatever patterns predict the label. This is why the same architecture handles photographs, X-rays and satellite passes with only minor adjustment.
Resolution is a trade-off, not a virtue
Higher resolution means more detail and more computation, and the cost grows with the square of the dimension: doubling the width quadruples the pixel count.
Most models resize inputs to something modest — 224 × 224 and 384 × 384 are common — which is a deliberate choice, not a limitation. A cat is recognisable at 224 pixels wide. A hairline crack in a weld is not, which is why inspection systems often work at much higher resolution, or tile a large image into patches and process each one.
Before designing anything, shrink your images to the resolution the model will see and look at them yourself. If you cannot perform the task on the downsized image, the model will not either, and you have learned something important for the cost of thirty seconds.
Why this is genuinely hard
The difficulty is not the volume of numbers. It is that the numbers change enormously while the answer stays the same.
Photograph the same cat twice and essentially every pixel value differs, because of:
Lighting. Sunlight, tungsten, fluorescent and shade all shift colour and contrast. The same white shirt reads as cream, blue-grey or amber.
Viewpoint. Move the camera and the object's outline, size and visible surfaces change completely.
Scale. A cat filling the frame and a cat forty metres away share a label and share very few pixel values.
Occlusion. Half the object is behind something else. Humans complete the missing part without noticing.
Deformation. Cats stretch, curl and rotate. So do people, clothes and cables.
Background. The same object in a kitchen and in a forest produces two mostly-different images.
The technical name for the requirement is invariance: the output should not change when the input changes in ways irrelevant to the task. And note that some changes must not be ignored — a model reading a resistor's colour bands must be invariant to lighting while remaining exquisitely sensitive to hue. Which variations to ignore is part of the problem definition, not a detail.
Why hand-written rules failed
The pre-2012 approach was to design features by hand: edge detectors, corner detectors, texture descriptors, histograms of gradient orientations. Skilled engineers spent careers on this and produced genuinely clever methods.
It plateaued, for a reason that reads as obvious now. Hand-designed features encode what an engineer believes distinguishes categories, and for anything beyond simple, controlled settings that belief is incomplete. Nobody can write down what makes a photograph contain a cat.
The alternative was to stop specifying features and learn them from examples — which is lesson 2.
In three sentences
An image reaches a computer as a grid of numbers, three million of them for a modest colour photograph, with no indication of where objects are or what they are. The difficulty is not the volume but the variability: lighting, viewpoint, scale, occlusion, deformation and background all change nearly every pixel value while leaving the correct answer untouched, so a useful model must be invariant to exactly those changes and sensitive to the rest. Hand-designed features could not capture that distinction, because nobody can state in advance what makes an image contain a cat.
Next — Lesson 2: convolution →