Module 6 — Gradient descent and the learning rate

Module 5 gave the instrument — the gradient. This module turns it into an algorithm: gradient descent, the method that trains the vast majority of learning models, from the simplest linear regressor to the largest neural networks.

The algorithm, in four repeating steps

Gradient descent is a disarmingly simple loop:

1. Predict with the current parameters and compute the cost (the error).
2. Compute the gradient of the cost with respect to each parameter.
3. Update each parameter by stepping away from the slope.
4. Repeat until the cost stops falling.

$$w \leftarrow w - \eta \cdot \nabla L$$

for epoch in range(n_epochs):
    y_pred = X @ w                       # 1. predict
    error = y_pred - y
    cost = (error ** 2).mean()           #    mean squared cost
    gradient = 2 * X.T @ error / len(y)  # 2. gradient
    w = w - learning_rate * gradient     # 3. update

This loop is the beating heart of machine learning. Everything else — architectures, regularization, sophisticated optimizers — is a refinement of it.

The learning rate: the setting that tips everything

The learning rate $\eta$ sets the step size. It is the most decisive hyperparameter of training, and it is tuned like a tightrope walker:

• Too small: the model learns, but excruciatingly slowly; it may settle in a mediocre trough.
• Too large: steps overshoot the target, the cost oscillates or shoots off to infinity — training diverges.
• Well tuned: the cost falls steadily toward a minimum.

The first diagnostic reflex, when training misbehaves, is almost always: "what if I changed the learning rate?"

Three flavors: full batch, stochastic, mini-batch

Over how much data do we compute the gradient at each step? Three answers:

Variant	Data per step	Character
Full batch	the whole dataset	accurate steps but slow and memory-heavy
Stochastic (SGD)	a single observation	noisy steps but very fast
Mini-batch	a small packet (32, 64…)	the trade-off, standard everywhere

Mini-batch gradient descent dominates in practice: it combines the stability of full batch and the speed of stochastic, while exploiting GPU parallelism. When you see batch_size=32 in training code, this is the choice being made.

Beyond raw gradient: modern optimizers

The formula $w \leftarrow w - \eta \nabla L$ is the basic version. Modern optimizers improve it: momentum keeps impetus to cross small troughs, and Adam — the most used — automatically adapts the step to each parameter. You don't code them yourself; you pick Adam in TensorFlow or PyTorch. But all rest on this module's idea: follow the slope downward.

Local minima: less serious than feared

In two dimensions, we imagine the hiker trapped in a trough that isn't the deepest (a local minimum). In very high dimension — the real case — such traps are rare: it is almost always possible to descend in one direction among thousands. The real obstacle isn't the local minimum but plateaus (flat regions where the gradient is near zero) and saddle points, which momentum and Adam precisely help to cross.

Summary

• Gradient descent repeats: predict, measure the error, compute the gradient, update parameters opposite to the slope.
• The learning rate sets the step size: too small = slow, too large = divergence; it's the first setting to question.
• Three variants by data per step; mini-batch is the standard, combining stability, speed and GPU parallelism.
• Modern optimizers (momentum, Adam) refine the basic formula to cross plateaus and saddle points.

Next module: probability — independence and conditional probability, the language of uncertainty at the heart of every predictive model.