Lesson 3 — Calculus
Calculus has exactly one job in machine learning: it tells the model which way to move in order to be less wrong. Everything in this lesson serves that single purpose.
A derivative is a slope
Forget the notation for a moment. A derivative answers one question:
If I nudge this input a little, how much does the output change, and in which direction?
That is it. A derivative is a rate of change, and geometrically it is the slope of a curve at a point.
- Large positive derivative: increasing the input increases the output sharply
- Small derivative: the input barely matters here
- Negative derivative: increasing the input decreases the output
- Zero derivative: you are at a flat spot, a peak, a valley or a plateau
Why this is what training needs: your model has an error, the error depends on the parameters, and you want the error smaller. The derivative of the error with respect to a parameter tells you exactly which way to nudge that parameter, and roughly how much it will help.
The gradient is a derivative for many parameters
A real model does not have one parameter. It has thousands, millions, sometimes hundreds of billions. So instead of one slope you need one slope per parameter, and the collection of them is the gradient.
The gradient is a vector — back to lesson 2 — with one entry per parameter, each entry saying how the error responds to that parameter. Read as a direction in the space of all parameters, the gradient points the way the error increases fastest.
Which gives training its rule: move in the opposite direction.
The fog analogy, made precise
You are on a hillside in thick fog and want to reach the valley. You cannot see the bottom, but you can feel the slope under your feet. So you step downhill, feel again, step again.
Each element of that picture maps onto something real:
| In the analogy | In training |
|---|---|
| the landscape | the error as a function of every parameter |
| your position | the current parameter values |
| the slope you feel | the gradient |
| the size of your step | the learning rate |
| the fog | you only know the slope locally, never the whole surface |
| the valley floor | a set of parameters where the error is low |
| a small dip that is not the true bottom | a local minimum |
The fog is the honest part of the analogy. The optimiser has no map. It knows the slope where it currently stands and nothing else, which is why training does not always end in the same place twice.
The learning rate: the one setting you will always tune
The learning rate scales how far you step. It is the single most consequential number in deep learning, and getting it wrong looks like a broken model rather than a badly chosen setting.
Too small: you creep. Training takes a hundred times longer than necessary, and may stall in the first shallow dip it meets.
Too large: you overshoot the valley entirely and land higher up the opposite slope. The error jumps around or grows without limit, and you see loss values that become meaningless numbers. This is the classic beginner symptom.
Roughly right: the error falls quickly at first, then more slowly, then flattens.
Modern optimisers such as Adam adapt the effective step size per parameter, which makes them far more forgiving than plain gradient descent. They reduce the problem; they do not remove it. You will still tune the learning rate more often than anything else.
A loss that explodes or becomes not-a-number almost always means the learning rate is too high. A loss that barely moves usually means it is too low. A loss that falls then plateaus far above zero points at the model or the data rather than the step size. This one habit will save you hours.
Backpropagation, without the algebra
A deep network is a chain: input, layer, layer, layer, output, error. To improve the first layer you need to know how it influences the final error — but it does so only indirectly, through every layer that follows.
The chain rule of calculus handles exactly this. It says that when one thing affects another through an intermediate, you multiply the rates of change along the path. Its practical form: to find the slope at the start of a chain, work backwards from the end, multiplying as you go.
Backpropagation is that idea applied to a network. Compute the error at the output, then walk backwards layer by layer, and at each step work out how much that layer contributed. One backward pass produces the gradient for every parameter in the network, at a cost comparable to one forward pass. That efficiency is the reason deep networks are trainable at all.
Two consequences you will meet by name:
Vanishing gradients. Multiplying many small numbers along a long chain gives something microscopically small. The early layers then receive almost no signal and effectively stop learning. This limited network depth for years, until residual connections — shortcuts that let the gradient bypass layers — made very deep networks practical.
Exploding gradients. The opposite: multiply many large numbers and the gradient becomes enormous, producing a step so wild it destroys everything learned so far. The standard fix, gradient clipping, simply caps the size of the step.
PyTorch and TensorFlow compute gradients automatically. Every operation records what it did, and the framework replays that record backwards. This is automatic differentiation, and it is arguably the single most important engineering feature of both libraries. You need to understand what it computes, never how to perform it.
Convexity, and why deep learning works anyway
A convex error surface is a single smooth bowl: one bottom, and following the slope always finds it. Simple models such as linear and logistic regression have convex surfaces, which is why they train reliably and reproducibly.
Deep networks are emphatically not convex. Their error surface is a vast, complicated landscape with countless local dips, flat plateaus and saddle points. In theory this should be a disaster, since gradient descent has no guarantee of finding the best solution.
In practice it works, and the reason is genuinely interesting: in very high-dimensional spaces, most of the dips turn out to be about as good as each other. Being trapped in a poor local minimum requires the surface to curve upward in every one of millions of directions at once, which is vanishingly unlikely. Almost every flat spot is a saddle — uphill in some directions, downhill in others — and there is always a way out.
So the theoretical worry is real, and empirically the landscape is kinder than it has any right to be.
In three sentences
A derivative is the slope that tells you which way to nudge a parameter to reduce the error, and the gradient is one such slope per parameter, pointing where the error rises fastest so training moves the other way. The learning rate scales that step and is the setting you will tune most, since too large explodes the loss and too small stalls it. Backpropagation applies the chain rule backwards through the network to get every gradient in one pass, which is what makes deep networks trainable at all.