Lesson 1 — What a neuron computes
The unit at the heart of every neural network, from a small classifier to a model with hundreds of billions of parameters, is startlingly simple. Understanding it precisely is worth more than any amount of architectural vocabulary, because everything else is arrangement.
The operation
An artificial neuron does three things:
- Multiply each input by a weight
- Add the results together, plus a constant called the bias
- Pass the total through a simple non-linear function called the activation
That is the entire computation. In pseudo-code:
total = (input1 × weight1) + (input2 × weight2) + ... + bias
output = activation(total)
The weights and the bias are the learned parameters. They begin as random numbers and are adjusted by training, exactly as described in the Introduction to AI course. Everything a network knows is stored in these numbers and nowhere else.
A weight expresses how much this input matters, and in which direction. A large positive weight means the input pushes the output up; a large negative weight means it pushes down; a weight near zero means the input is being ignored. Training is the process of discovering which inputs matter and by how much.
The bias deserves a word, because it looks like an afterthought and is not. It shifts the threshold at which the neuron responds. Without a bias, every neuron would be forced to output the same value when all its inputs are zero, which severely constrains what the network can express. It is a small term that buys a lot of flexibility.
Why the activation function is the whole point
This is the part that is skipped most often, and it is the reason depth works at all.
Suppose you remove the activation function, so each neuron only computes a weighted sum. Stack a hundred layers of that. What have you built?
A single weighted sum. A chain of linear transformations collapses, mathematically, into one linear transformation. Your hundred-layer network is exactly as expressive as one layer, which cannot even solve the trivial problem of deciding whether exactly one of two inputs is true.
The non-linear function between layers is what breaks that collapse. It is what allows each additional layer to add genuinely new expressive power, and therefore what makes "deep" mean anything.
The functions you will meet
| Activation | What it does | Where it is used |
|---|---|---|
| ReLU | negatives become zero, positives pass through unchanged | the default in hidden layers, almost everywhere |
| Sigmoid | squashes any number into 0 to 1 | output layer for binary classification |
| Softmax | turns several scores into probabilities summing to 1 | output layer for multi-class classification |
| Tanh | squashes into -1 to 1 | older recurrent networks |
| GELU | a smoother relative of ReLU | transformers, including language models |
ReLU deserves attention because its dominance is instructive. It is the simplest function on the list — if the input is negative, output zero, otherwise output the input — and it replaced the mathematically elegant sigmoid across the field for two reasons. It is extremely cheap to compute, which matters when you evaluate it billions of times. And its slope is exactly 1 for positive inputs, which means gradients pass through it undiminished, largely avoiding the vanishing gradient problem that had limited network depth for years.
A great deal of progress in deep learning has this shape: a simpler component that trains better beats a sophisticated one that does not.
From neuron to layer to matrix
One neuron is not useful. A layer is a group of neurons all receiving the same inputs, each with its own weights, each learning to respond to something different.
And here is where the maths course connects. Computing a whole layer means, for each neuron, multiplying inputs by weights and summing. Performed for every neuron at once, that is precisely a matrix multiplication:
layer_output = activation(W · inputs + b)
W holds every weight in the layer, one row per neuron. b holds every bias. This is why deep learning is fundamentally matrix arithmetic, and why graphics processors — hardware built to multiply matrices for video games — turned out to be exactly the right machine.
Counting parameters
Being able to count parameters is a practically useful skill, because it tells you how much data and memory a model will need.
A layer with 100 inputs and 50 neurons has 100 × 50 weights plus 50 biases: 5,050 parameters. Add a second layer of 50 neurons feeding 10 outputs and you add 510 more.
Scale that up and the numbers become the ones you read in announcements. A large language model with 70 billion parameters holds 70 billion of these individual learned numbers, which is why it needs many gigabytes just to be loaded into memory, before any computation happens.
It would be possible to design a more sophisticated neuron. Deep learning went the other way: keep the unit trivial, keep the operation something hardware can do billions of times per second, and get capability from scale and arrangement rather than from cleverness in the component. That trade has held for a decade.
In three sentences
A neuron multiplies each input by a learned weight, sums them with a learned bias, and applies a simple non-linear function — and the weights and biases are the entirety of what a network knows. The activation function is not a detail: without it a hundred stacked layers collapse mathematically into one, so it is what makes depth meaningful. Computing a whole layer is a matrix multiplication, which is why deep learning is matrix arithmetic and why graphics processors turned out to be the right hardware.