Lesson 4 — Modern architectures
You do not need to memorise architecture names to work with vision. You do need to understand what each generation fixed, because that tells you which one suits your constraints — and vendors quote model names as though they were arguments.
The convolutional line
AlexNet (2012) was eight layers and won by a margin that ended the hand-designed-features era. Its real contribution was demonstrating that depth plus data plus GPUs worked; the specific design was superseded quickly.
VGG (2014) showed that stacking many small 3 × 3 filters beat using large ones, establishing a convention that still holds. It was also enormous and slow, which is why it survives mainly as a teaching example.
ResNet (2015) solved the problem blocking further depth. Beyond roughly twenty layers, deeper networks were performing worse — not from overfitting, but because the training signal degraded as it propagated back through many layers. ResNet added residual connections: each block learns an adjustment to its input rather than a replacement for it, and the input is added back to the output. This gives the training signal a short path through the whole network, and depths of a hundred layers and beyond became trainable. Residual connections are now in essentially every deep architecture, including transformers.
EfficientNet (2019) asked how to scale depth, width and input resolution together rather than one at a time, and produced a family of models with markedly better accuracy per unit of compute. It remains a sensible default when inference cost matters.
YOLO and its successors reframed detection: instead of proposing candidate regions then classifying each one, predict all boxes in a single pass over the image. This made real-time detection practical on modest hardware and is why the name appears in nearly every camera-based product.
Then transformers arrived
Convolution's built-in assumption — nearby pixels are related, patterns can appear anywhere — is a genuine advantage when data is scarce and a genuine limitation when it is not. Relating two distant parts of an image requires many stacked layers.
The vision transformer dropped the assumption. Cut the image into patches, typically 16 × 16 pixels, treat each patch as a token, and apply the same attention mechanism that transformed language processing. Every patch can attend to every other patch in a single layer, so long-range relationships are available immediately.
The trade-off is direct: with abundant data, transformers win; with modest data, convolutional networks often still win. The convolutional prior is information the architecture supplies for free, and when you cannot supply it from data, you want it. Hybrid designs that use convolution early and attention later have become common precisely because they get both.
Foundation models changed what "training" means
Two developments matter more than any single architecture.
CLIP was trained on hundreds of millions of image–caption pairs to place images and text in the same vector space, so an image of a dog and the words "a photo of a dog" land near each other. The consequence is zero-shot classification: you can classify into categories the model was never explicitly trained on, simply by writing them out. Accuracy is below that of a properly fine-tuned model, and it is available immediately with no labelled data at all, which makes it an excellent way to test whether a project is worth pursuing.
Segment Anything produces high-quality masks for objects in an image without task-specific training, from a click or a box as a hint. Since segmentation annotation was the dominant cost in lesson 3's table, a model that generates most of a mask and leaves a human to correct it changes the economics substantially. Assisted annotation is now the normal workflow rather than a novelty.
Choosing, in practice
| Your situation | Reasonable starting point |
|---|---|
| Classification, a few thousand images | Fine-tune a pre-trained EfficientNet or ResNet |
| Classification, no labelled data yet | CLIP zero-shot, to test feasibility before investing |
| Detection, real time on modest hardware | A recent YOLO variant |
| Detection, accuracy over speed | A transformer-based detector |
| Segmentation, few masks available | Segment Anything for assisted annotation, then fine-tune |
| Mobile or embedded deployment | A small efficient model, quantised |
| Millions of images available | A vision transformer |
Architecture choice typically moves accuracy by a few percentage points. Data quality and representativeness move it by tens. Teams spend weeks comparing architectures and days on data, and the returns run firmly the other way — which is lesson 5.
Efficiency, because deployment has a budget
A model that only runs on a rented GPU limits what you can build. Three techniques make small deployment feasible, all covered in depth in the premium track:
Quantisation stores weights at lower precision, typically 8-bit integers instead of 32-bit floats. Roughly four times smaller and often faster, with accuracy loss that is frequently negligible.
Pruning removes weights that contribute little, exploiting the fact that trained networks are substantially redundant.
Distillation trains a small model to imitate a large one's outputs, which transfers more than training the small model on labels alone.
Together these routinely fit a useful vision model onto a phone or a microcontroller — the reason on-device photo classification and offline inspection tools exist.
In three sentences
The convolutional line solved successive obstacles — AlexNet proved depth plus data worked, ResNet's residual connections made great depth trainable, YOLO made detection real time — and vision transformers then replaced convolution's assumption that nearby pixels matter most with attention that relates any patch to any other, winning when data is abundant and losing when it is scarce. Foundation models changed the workflow more than the architectures did: CLIP allows classification into categories never trained on, which makes feasibility testing free, and Segment Anything generates most of a segmentation mask, which collapses the annotation cost that used to decide whether a project was viable. Architecture choice moves accuracy a few points while data quality moves it tens, so the sensible order of effort is data first, architecture second.