Skip to main content

Module 5 — Derivatives and gradient: the slope that guides learning

The introductory course stated it: a model learns by adjusting its parameters to reduce a cost function. The derivative is the instrument of that adjustment. At each point it says in which direction and by how much to change a parameter to lower the error. This module gives the intuition, without needless formalism.

The derivative: a local slope​

The derivative of a function at a point is the slope of its curve there. It answers a precise question: "if I nudge the input up a little, how does the output change?"

  • Positive derivative: the function is rising; increasing the input increases the output.
  • Negative derivative: the function is falling.
  • Zero derivative: flat ground — a peak, a trough, or a plateau.

For learning, this last case is crucial: where the derivative of the cost vanishes lie the minima, the points where the error can no longer fall locally. All of training is about reaching them.

Going downhill: the intuition of optimization​

Picture a cost function as a hilly landscape and the model as a hiker looking for the lowest point in the fog. It cannot see the whole valley, but it feels the slope underfoot: the derivative. The obvious strategy — take a step downhill, repeat — is exactly the gradient descent of module 6.

new parameter=old parameter−η⋅derivative\text{new parameter} = \text{old parameter} - \eta \cdot \text{derivative}

The minus sign encodes "go down": you move opposite to the slope. The factor η\eta (the learning rate) sets the step size.

The gradient: the slope in many dimensions​

A real model has thousands, even billions of parameters. The derivative with respect to one parameter (the others held fixed) is a partial derivative. The gradient collects all partial derivatives into a vector:

∇L=(∂L∂w1,∂L∂w2,…,∂L∂wn)\nabla L = \left( \frac{\partial L}{\partial w_1}, \frac{\partial L}{\partial w_2}, \dots, \frac{\partial L}{\partial w_n} \right)

This vector has a remarkable property: it points toward the direction of steepest ascent of the cost. To lower the error, you therefore move in the opposite direction — hence "gradient descent." The gradient reconciles the two halves of the course: it is a calculus object, laid out as a linear-algebra vector.

The chain rule: the engine of backpropagation​

How do you compute the gradient through a network's many layers? Thanks to a single rule: the chain rule. For nested functions, the derivative of the whole is the product of the derivatives of each step:

ddxf(g(x))=f′(g(x))⋅g′(x)\frac{d}{dx} f(g(x)) = f'(g(x)) \cdot g'(x)

A network being a long composition of functions (each layer applied to the previous one), the chain rule lets you compute the effect of each weight on the final cost by going back layer by layer. This is precisely what we call backpropagation — module 5 of the introductory course gave the result; here we hold the exact mechanism.

You won't differentiate by hand

Good news: modern libraries (TensorFlow, PyTorch) compute gradients automatically through automatic differentiation. You describe the "forward" computation and the gradient is derived for you. Understanding the derivative and the chain rule is thus not required to code, but essential to diagnose: exploding gradients, vanishing gradients, stalled learning — failures that only make sense with this intuition.

Summary​

  • The derivative is the local slope of a function: it says which way and how much to change an input to move the output.
  • Where the cost's derivative vanishes lie the minima, the target of training.
  • The gradient collects all partial derivatives; it points toward steepest ascent, so you descend in the opposite direction.
  • The chain rule propagates derivatives through nested functions: it is the mechanism of backpropagation, computed automatically by libraries.

Next module: gradient descent in detail — how the learning rate turns this slope into a concrete training algorithm.