Lesson 2 — Why depth changes everything
A single layer of neurons can, in theory, approximate almost any function — a result known as the universal approximation theorem. So why stack dozens of layers? The theorem says such a network exists; it says nothing about how wide it would have to be or whether you could ever train it. In practice depth is enormously more efficient than width, and the reason is worth understanding.
The hierarchy that builds itself
The defining property of deep learning is that the network invents its own intermediate representations. This is not a metaphor; it is observable by inspecting what the layers respond to after training.
Take an image network trained only on labelled photographs, with no instruction about visual structure. Examine what each layer has learned to detect:
| Layer depth | What it responds to |
|---|---|
| First layers | edges, corners, patches of colour, orientations |
| Early middle | simple textures, repeated patterns, curves |
| Middle | recognisable parts: an eye, a wheel, a leaf, a doorway |
| Deeper | assemblies: a face, a car, a plant, a building |
| Final layer | the categories you asked for |
Nobody programmed that progression. Nobody told the network that edges combine into textures, or that eyes and noses appear together in faces. It emerged because that hierarchy is genuinely present in the data, and building it was the most efficient way to reduce the training error.
Lesson 1 of the Introduction to AI course described Polanyi's paradox: you cannot write down the rules you use to recognise a cat. Deep learning does not solve that problem by extracting the rules from you. It sidesteps it entirely by discovering its own — which is why it works precisely where feature engineering is impossible.
Why depth beats width
Consider building a face detector.
Shallow and wide: one enormous layer that must recognise every possible face directly from raw pixels. Each unit has to handle every combination of position, lighting, angle and expression independently. The number of units required explodes, and there is no sharing of effort.
Deep and narrow: the first layers learn edges, which are useful for every face and indeed for every object. The next layers assemble edges into eyes and mouths, reusing the edge detectors. The next assemble those into faces, reusing the part detectors.
Depth allows reuse. Each layer builds on the abstractions below it instead of starting from pixels, so the same edge detector serves a face, a car and a tree. That compositional efficiency is why a deep network with a million parameters can outperform a shallow one with a hundred million.
What depth cost, and what fixed it
Depth was understood to be desirable long before it was achievable. Networks past about twenty layers trained worse than shallower ones, which was baffling: a deeper network can always in principle imitate a shallower one by making the extra layers do nothing, so it should never be worse.
The culprit was the vanishing gradient, described in the maths course. Backpropagation multiplies rates of change along the chain from output back to input. Multiply thirty numbers smaller than one and you get something microscopic, so the earliest layers received essentially no signal and stopped learning.
Two changes resolved it:
ReLU activations. Their slope is exactly 1 for positive inputs, so gradients pass through undiminished rather than being repeatedly shrunk by a squashing function.
Residual connections. Introduced in 2015 with ResNet, and the more important of the two. A residual connection adds a shortcut that lets the input of a block skip past it and be added to its output. The gradient can then travel back along the shortcut without being attenuated by the layers.
The effect was immediate: networks jumped from around 20 usable layers to over 150, with better results. Residual connections are now in essentially every deep architecture, including every transformer, which makes them one of the most consequential small ideas in the field.
What depth does not fix
Being clear about the limits, since depth is often proposed as a solution to problems it cannot touch:
Insufficient data. A deeper network has more parameters and therefore needs more data, not less. Depth on a small dataset produces confident memorisation.
Bad labels. The network will learn your labelling errors faithfully and with great precision.
Missing information. If the signal is not in the input, no architecture recovers it. Depth discovers features within the data; it does not invent data.
Tabular data. Adding layers does not make a neural network competitive with gradient boosting on a spreadsheet. The hierarchy that depth exploits exists in pixels and language; it is largely absent from a table of columns that were designed by a human in the first place.
When a model underperforms, "make it deeper" is rarely the answer. Check the data volume, the label quality, and whether the input actually contains the signal. Depth amplifies what is there, including the problems.
Two ideas you will see everywhere
Two techniques accompany depth so consistently that they are worth naming.
Batch normalisation rescales the values flowing between layers so they stay in a well-behaved range. Deep networks otherwise suffer from values growing or shrinking as they propagate, which destabilises training. It made deep networks train faster and tolerate a wider range of learning rates.
Dropout randomly switches off a fraction of neurons during each training step. It sounds destructive and it is a regularisation technique: the network cannot rely on any single unit, so it is pushed towards redundant, robust representations rather than brittle memorisation. It is switched off at prediction time.
In three sentences
Depth matters because each layer builds on the abstractions below it, so a network discovers a reusable hierarchy — edges, then textures, then parts, then objects — that nobody programmed and that makes deep networks far more efficient than wide ones. That hierarchy is exactly why deep learning succeeds where features cannot be hand-designed, and why it offers little on tabular data whose columns a human already designed. Depth was unusable until ReLU and residual connections let gradients travel back through many layers, which is what took networks from twenty layers to hundreds.