Module 1 — From the perceptron to the multilayer network

The previous course closed on one idea: a model only learns what the feature space makes expressible, which is why we built those features by hand. Deep learning proposes the exact opposite — letting the network build its own representations. This module starts from the elementary computing unit to show where that ability comes from.

The computation of a neuron, in full

An artificial neuron does only two things: a weighted sum, then a non-linear transformation. Nothing more.

The three inputs are weighted then summed with the bias. The activation function then applies the non-linearity without which depth would be pointless.

Formally, for inputs $x_1, \dots, x_n$:

$$y = f\left(\sum_{i=1}^{n} w_i x_i + b\right)$$

Each term plays a distinct role worth naming. The weights $w_i$ measure the influence of each input, and they are what training adjusts. The bias $b$ shifts the firing threshold: without it, the neuron would be forced through the origin, preventing it from representing otherwise simple relationships. The activation function $f$ introduces non-linearity, and the next module is devoted entirely to it.

You may recognize a familiar structure: with $f$ equal to the sigmoid, this neuron is a logistic regression. That is a useful landmark — a neural network is not a conceptual break, it is a stack of objects you already know.

Why a single neuron is not enough

The perceptron, proposed by Frank Rosenblatt in 1957, runs into a limitation that Marvin Minsky and Seymour Papert formalized in 1969: it can only draw a linear boundary.

The canonical example is the XOR function, which is 1 when the two inputs differ:

$x_1$	$x_2$	XOR
0	0	0
0	1	1
1	0	1
1	1	0

Place these four points in the plane and look for a straight line separating the 1s from the 0s. None exists: the two positive points occupy opposite corners. No setting of the weights solves this, because the obstacle is not in the learning but in the form of the model.

That demonstration was costly. It contributed substantially to the first AI winter, a decade of disinterest in neural networks — while the solution was within reach.

What depth changes

The solution is to stack layers. One hidden layer of two neurons suffices for XOR: the first learns one boundary, the second another, and the output layer combines them into a non-linear region.

A fully connected network. Each layer transforms the representation it receives, and it is this stack of transformations that replaces the feature engineering work of the previous course.

The vocabulary settles here. The input layer computes nothing, it exposes the features. The hidden layers produce intermediate representations — "hidden" because we never directly observe their contents. The output layer produces the prediction, and its size is dictated by the task: one neuron for regression or binary classification, as many neurons as classes for multiclass classification.

Depth is the number of layers, width the number of neurons per layer.

Universality, and why it does not say what people think

The universal approximation theorem establishes that a network with one single hidden layer can approximate any continuous function to any desired precision, given enough neurons.

This result is often misquoted, when what matters most is its silences. It states that such a network exists, without saying how to find it — which is precisely the job of training. It does not bound the number of neurons required, which can be astronomical. And it promises nothing about generalization: perfectly approximating the training data is the very definition of overfitting.

Hence the question that really matters: if one layer suffices in theory, why stack? Because depth is more efficient. Some functions that a deep network represents with a modest number of neurons would require exponentially more neurons in a single layer. Depth allows hierarchical composition: early layers capture simple patterns, later ones combine them into more abstract ones. In vision this hierarchy is almost visible to the naked eye — edges, then textures, then object parts, then objects — and the convolutional networks course will return to it in detail.

What deep learning actually contributes

The previous course consisted of building the right features by hand, from domain knowledge. A deep network learns those representations from raw data. That explains its dominance wherever features are hard to formulate: pixels, sound, text. On tabular data, by contrast, feature engineering combined with gradient boosting very often remains superior — and that is a trade-off to make clear-headedly, not a matter of fashion.

Summary

• A neuron computes a weighted sum plus a bias, followed by an activation function; with a sigmoid, that is exactly a logistic regression.
• The perceptron alone draws only a linear boundary and fails on XOR: the obstacle is the model's form, not the learning.
• Stacking hidden layers enables non-linear boundaries; depth and width denote the number of layers and of neurons respectively.
• Universality guarantees a one-layer solution exists, while saying nothing about its size, findability or generalization; depth is worth it for its representational efficiency.

Next module: activation functions, the non-linearity without which a deep network would collapse into a plain linear model.