Module 2 — Autograd: dynamic graph and gradient computation

The previous module described the tensor as a container for numbers and shapes. This module explains the machinery that makes a tensor remember what happened to it, so that its derivatives can be computed at the end. That machinery is called autograd. Understanding it in ten minutes prevents whole categories of bugs that would otherwise take hours: gradients that never zero out, tensors that refuse to be exported, evaluation runs that consume five times the memory of training.

The Fashion-MNIST classifier does not appear directly in this module: we work on tiny expressions that would fit on a whiteboard. But every mechanism introduced here — accumulated gradients, no_grad inference, detach before extraction — will be applied verbatim when we assemble the full training loop in module 5.

Autograd builds a graph while you compute

PyTorch's computation graph is dynamic: it is constructed on the fly, one operation at a time, as the forward pass proceeds, and then discarded after the backward pass. This contrasts with TensorFlow's original static graphs, and it is the design decision that explains most of PyTorch's ergonomic advantages — you can use ordinary Python control flow, print intermediate tensors, insert a breakpoint anywhere.

A tensor participates in the graph as soon as it is created with requires_grad=True, or as soon as it depends on one that does:

import torch

w = torch.tensor(3.0, requires_grad=True)
x = torch.tensor(2.0)                    # no gradient
y = w * x + 1                            # y inherits the graph

The tensor y now carries a hidden reference to a grad function — MulBackward0 chained into AddBackward0 — that describes how to propagate a gradient back to w. The chain is destroyed the first time .backward() is called on y, unless we explicitly keep it with retain_graph=True.

backward computes derivatives; the result lands in .grad

y.backward()          # walks the graph from y back to w
print(w.grad)         # tensor(2.) — the derivative of y w.r.t. w

backward is a walk backwards through the graph, applying the chain rule at each node. It writes the result on the .grad attribute of every leaf tensor with requires_grad=True. A leaf is a tensor that was created directly by the user, not the output of an operation on other tensors.

Two subtleties matter from the start. First, .grad is a tensor, not a number, and it has the same shape as the leaf. Second, .grad is not overwritten by the next call to backward: it is accumulated. That accumulation is the root of the single most common mistake in PyTorch, and the one this module exists to prevent.

The accumulation trap: zero_grad is not optional

w = torch.tensor(3.0, requires_grad=True)

for _ in range(3):
    y = w * 2
    y.backward()
    print(w.grad)     # 2., then 4., then 6.

Every call to backward adds to .grad rather than replacing it. Without an explicit reset, the gradient of the third batch is the sum of the gradients of the three batches, the fourth is the sum of four, and the optimiser takes steps that are wildly too large. Training does not diverge cleanly with an error message: it slowly degrades, the loss climbs by a few percent per epoch, and the culprit is invisible in any dashboard.

The fix is one line, called at the start of each iteration:

w.grad = None          # or: optimizer.zero_grad(set_to_none=True)

The optimizer.zero_grad() call, which we will use from module 5 onwards, does exactly that for every parameter in one shot. Setting to None is slightly faster than filling with zeros; the modern API accepts both and defaults to set_to_none=True.

This is the number one PyTorch bug

Every PyTorch codebase has been burnt by a forgotten zero_grad. The symptom is deceptively benign — training just underperforms — and the diagnosis is delayed by the fact that everything else looks correct. Make the pattern reflexive: zero_grad, forward, loss, backward, step, in that order, and never remove one of the five.

no_grad and detach: turning the graph off

Not everything you compute deserves a graph. Evaluating the model on the validation set, generating predictions for a report, sampling from a trained network: none of these need gradients, and building the graph anyway wastes memory and cycles.

model.eval()
with torch.no_grad():
    predictions = model(val_batch)

Inside a torch.no_grad() block, PyTorch skips graph construction entirely. Memory usage drops sharply — often by half — and computation speeds up. It is the standard wrapper around every validation and inference call.

detach() produces a new tensor that shares memory but is severed from the graph. It is used when a value computed under gradients must feed into a computation that should not carry gradients back:

loss_display = loss.detach().cpu().item()   # for logging
target = model_teacher(x).detach()          # frozen teacher in distillation

.item() alone would raise on a tensor with a grad function; .detach().cpu().item() is the safe extraction idiom, and worth learning as a single mental unit.

no_grad versus detach versus eval: three distinct switches

These three often get conflated. They are not.

Mechanism	What it changes
with torch.no_grad():	temporarily disables graph construction for the block
.detach()	severs one specific tensor from the graph
model.eval()	switches BatchNorm and Dropout to inference mode

eval() does not turn off gradients: it is entirely about layer behaviour. no_grad() does not switch BatchNorm to running statistics: it is entirely about the graph. Evaluating without both means either wasting memory or feeding batch statistics into what should be inference — the second is a silent accuracy killer we return to in module 5.

Second derivatives, retained graphs and edge cases

The graph is destroyed on the first backward. Two consecutive .backward() calls on the same loss without retain_graph=True raise RuntimeError: Trying to backward through the graph a second time. Higher-order derivatives require both retain_graph=True and create_graph=True so that the backward pass itself is differentiable. These features exist for research code — meta-learning, adversarial training — and hardly matter in a supervised classifier.

Comparison with course 08 in one paragraph

TensorFlow builds the graph explicitly with tf.function, then executes it repeatedly. PyTorch builds a fresh graph on every forward pass and discards it after the backward pass. That is what "dynamic graph" means: the topology of the computation may change between iterations without any recompilation. The cost is that the graph is rebuilt each time; the benefit is that Python if, while, list comprehensions and prints all work as usual. Both frameworks have converged since — TensorFlow adopted eager execution, PyTorch added torch.compile for optimisation — but the mental model still differs.

In summary

• Autograd builds a dynamic graph as the forward pass proceeds, and destroys it after the first backward unless you keep it explicitly.
• .grad accumulates on every backward call; forgetting to reset it is the number one PyTorch bug and looks like generic underperformance rather than an error.
• Use with torch.no_grad(): for evaluation and inference — it disables graph construction — and .detach() to sever one tensor from a running graph, typically before .cpu().item().
• no_grad, detach and eval do three different things and cannot substitute for each other; validation code needs both eval() and no_grad().

Next module: nn.Module, the container that stitches parameters and forward code together in the way autograd expects.