Lesson 3 — CNNs: how machines see
Convolutional networks were the architecture that made the 2012 breakthrough, and they remain the standard for images. Their design comes from two observations about pictures that a plain network cannot exploit.
Why a plain network fails on images
Take a modest colour photograph of 224 by 224 pixels. That is 224 × 224 × 3 colour channels, roughly 150,000 numbers.
Connect that to a plain layer of 1,000 neurons and you need 150 million weights — for one layer. Two problems follow, and the second is more fundamental than the first.
It is far too many parameters. You would need an implausible amount of data to fit them without memorising.
It throws away the structure of the image. A plain layer treats the 150,000 inputs as an unordered list. It has no notion that two pixels are adjacent, and having learned to recognise a cat's ear in the top-left corner, it would have to learn the same ear from scratch for every other position, since each position has entirely separate weights.
Both problems come from ignoring two facts that are true of every image.
The two facts CNNs exploit
Locality. A meaningful pattern is made of nearby pixels. An edge, a corner, a texture — all are local. Pixels at opposite corners of a photograph are almost never part of the same small feature.
Translation invariance. A pattern means the same thing wherever it appears. An eye is an eye in the top-left or the bottom-right. A vertical edge is a vertical edge everywhere.
A convolutional layer is designed around exactly these two facts, and that is the entirety of the idea.
What convolution does
A filter — also called a kernel — is a tiny grid of weights, typically 3 by 3. It is placed over one small patch of the image, multiplied element-wise with those pixels, and the results are summed to give one output number. Then it slides to the next patch and repeats, across the whole image.
The same filter, with the same weights, is applied at every position. That is weight sharing, and it delivers both benefits at once:
- Few parameters. A 3 × 3 filter on a colour image has 27 weights plus a bias, regardless of image size. Compare with 150 million.
- Position independence. Because the identical filter is applied everywhere, a pattern it detects is found wherever it occurs. Learn it once, apply it everywhere.
A layer contains many filters, each learning to detect something different, and each producing its own output map. A first layer with 64 filters produces 64 maps, each highlighting where its particular pattern was found.
Pooling: seeing less to understand more
Between convolutional layers sits pooling, which shrinks the maps — typically taking the maximum value in each 2 by 2 square, halving width and height.
Discarding information deliberately sounds wrong, and it does three useful things:
It reduces computation substantially as the network deepens.
It enlarges the receptive field. After pooling, a 3 × 3 filter covers a larger area of the original image, which is what allows deeper layers to see whole objects rather than small patches.
It adds tolerance to small shifts. Taking the maximum in a neighbourhood means a feature moving by one pixel produces the same output. Small translations stop mattering.
The combination — convolve to detect, pool to abstract, repeat — is what produces the hierarchy from lesson 2: edges become textures, textures become parts, parts become objects.
What CNNs are used for
Beyond classifying whole images, the same machinery supports a family of tasks:
| Task | Question answered | Example |
|---|---|---|
| Classification | what is in this image | is this X-ray normal |
| Detection | what, and where | locate every pedestrian in this frame |
| Segmentation | which pixels belong to what | outline the tumour precisely |
| Pose estimation | where are the key points | track joints for motion analysis |
| Super-resolution | what would a sharper version look like | restore detail in a low-quality scan |
And beyond images entirely, wherever data has local structure worth exploiting: audio spectrograms, time series, and even short text, where a filter sliding over words detects local phrases.
Since 2020, vision transformers have matched or exceeded CNNs on large-scale tasks by cutting an image into patches and treating them like words. They typically need more data to reach that point, which keeps CNNs the pragmatic choice for smaller datasets and constrained hardware. Both are current, and neither has retired the other.
Why you rarely train one yourself
The final and most practical point. A network trained on millions of general images has already learned the edge, texture and part detectors that any vision task needs. Those early layers are not specific to the original categories.
So the standard approach is transfer learning: take a pre-trained network, replace its final layer with one matching your categories, and train only that — or gently adjust the rest. It routinely works with a few hundred images per class and outperforms anything you would train from scratch on that data.
Lesson 5 goes into this properly, because it is the single most useful practical fact in the course.
In three sentences
A plain network fails on images because it needs a hundred million weights per layer and treats pixels as an unordered list, so a pattern learned in one corner has to be relearned everywhere else. A convolutional layer slides a small grid of shared weights across the image, which gives it few parameters and makes detection independent of position, while pooling shrinks the maps to enlarge what deeper filters can see. Stacking convolution and pooling produces the edge-to-texture-to-part-to-object hierarchy, and in practice you take that hierarchy pre-trained rather than learning it yourself.