Module 3 — Backpropagation through time

Module 2 introduced the recurrent neuron and its hidden state, and closed by noting that the unrolled network behaves like a deep feedforward network for the gradient. This module makes that statement precise: it derives backpropagation through time, counts what it costs in memory, and shows the two escape valves — truncated BPTT and stateful RNNs — that make long series trainable on ordinary hardware.

The electricity series still runs beneath the code. The 168-hour lookback used in module 2 was chosen partly for a physical reason (one week captures the useful cycles) and partly because 168 is short enough that a laptop can hold the full unrolled graph in memory. Doubling it to 336 hours already starts to hurt.

What "through time" really unrolls

For one training example, the RNN produces a sequence of hidden states $h_1, h_2, \ldots, h_T$ and a loss $\mathcal{L}$ computed on one of them (or on all of them, in a sequence-to-sequence setting). The gradient of $\mathcal{L}$ with respect to the shared weight $W_h$ is:

$$\frac{\partial \mathcal{L}}{\partial W_h} = \sum_{t=1}^{T} \frac{\partial \mathcal{L}}{\partial h_t} \cdot \frac{\partial h_t}{\partial W_h}$$

But $h_t$ depends on $W_h$ at every previous step, because $h_t$ is a function of $h_{t-1}$, which is a function of $h_{t-2}$, and so on. Expanding the chain rule gives:

$$\frac{\partial h_t}{\partial W_h} = \sum_{k=1}^{t} \left( \prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}} \right) \cdot \frac{\partial h_k^{\text{local}}}{\partial W_h}$$

The product of Jacobians $\prod \partial h_i / \partial h_{i-1}$ is the source of every difficulty in the next module. For now, it is enough to notice that computing this sum requires storing every intermediate hidden state $h_1, \ldots, h_T$, because each one is needed in the backward pass.

Memory cost: linear in length, linear in width

Every recurrent framework — TensorFlow, PyTorch, JAX — keeps the sequence of hidden states in memory during the forward pass so the backward pass can read them. For a batch of $B$ examples, a sequence of length $T$ and a state of dimension $H$, the recurrent activations alone cost:

$$\text{memory} \approx B \times T \times H \times 4 \text{ bytes (float32)}$$

For $B = 64$, $T = 168$, $H = 32$: about 1.4 MB per layer, negligible. For $T = 4\,000$ (a full six-month window at hourly resolution) and $H = 256$: 260 MB per layer, and if there are two stacked layers plus attention scratch buffers, the laptop starts swapping.

This is why raw RNNs, even with modern hardware, are rarely trained on sequences longer than a few thousand steps. The memory cost is the harder constraint, and it is very unforgiving: doubling the sequence length exactly doubles the memory.

import tensorflow as tf

def unrolled_memory_MB(batch, time, hidden, layers, bytes_per_scalar=4):
    return batch * time * hidden * layers * bytes_per_scalar / 1024**2

print(unrolled_memory_MB(64, 168, 32, 1))    # 1.31 MB
print(unrolled_memory_MB(64, 4000, 256, 2))  # 500 MB

Truncated BPTT: shorten the backward pass, not the forward one

The classical fix is truncated backpropagation through time. The idea: continue running the forward pass over the whole sequence, but only back-propagate the gradient over the last $k$ steps. The gradient becomes biased — it ignores dependencies longer than $k$ — but the memory cost drops from proportional to $T$ to proportional to $k$.

In practice, on the electricity series, this often means splitting a long 8 760-hour year into overlapping windows of, say, 168 hours: the forward pass runs on 168 hours, the backward pass runs on the same 168 hours, and the next batch starts a few hours later. The full year never lives in one graph.

lookback = 168      # forward and backward span
stride   = 24       # start of the next window, one day later

The window itself is short enough that BPTT is exact within it. Truncation only bites when we would have wanted to train on a single window covering multiple weeks or months.

Stateful RNNs: keep the state, drop the gradient

Keras exposes a second mechanism, stateful=True, that carries the hidden state from the end of one batch to the beginning of the next, without carrying the gradient. The forward pass therefore sees an arbitrarily long history, but each backward pass only covers one batch worth of steps.

from tensorflow.keras import layers, Sequential

model = Sequential([
    layers.Input(batch_shape=(32, 168, 1)),
    layers.SimpleRNN(64, stateful=True),
    layers.Dense(24),
])

# Between epochs, the state must be reset by hand
model.reset_states()

The two constraints of a stateful RNN are exactly what makes it uncomfortable:

• The batch size is fixed at construction time (the state must have a stable shape).
• The order of batches matters: batch $i+1$ must contain the continuation of batch $i$'s sequences.
• The state must be reset whenever a batch does not follow the previous one (a new epoch, a new city in a multi-building dataset).

Getting any of those wrong silently produces a model that learns from a random state at every step. Stateful RNNs are worth learning but should not be reached for on a first project.

The memory-versus-truth trade-off in one table

Strategy	Forward span	Backward span	Bias	Practical use
Full BPTT on the whole series	$T$	$T$	none	small or synthetic problems
Truncated BPTT (windowed)	$L$	$L$	ignores dependencies beyond $L$	almost every production RNN
Stateful RNN	$T$	$L$ per batch	ignores gradient beyond $L$	very long sequences with careful batching
Chunked with gradient checkpointing	$T$	$T$	none	rare with RNNs, standard with Transformers

A stateful RNN with a shuffled DataLoader is a broken model

The most common misuse of stateful=True is combining it with a shuffled data pipeline. Each batch resets the effective context to a random point in the year, and the hidden state carries information from an unrelated sequence into the next update. The training loss will look normal — the model still learns to average — but the recurrent structure is doing nothing.

What this means for the rest of the course

• Modules 5 and 6 (LSTM and GRU) address the gradient quality on long sequences.
• Module 9 addresses the memory and batching side by teaching how to build a stable data pipeline.
• Module 10 puts the two together on the electricity forecasting project.

If a project ever grows to hundreds of thousands of time steps per example — high-frequency finance, long biosignal recordings — the recurrent approach has essentially been replaced by Transformers with attention (course 12) and by state-space models (course 13). RNNs remain the correct tool for the small-to-medium sequence regime that covers most operational forecasting problems.

Profile memory before you profile speed

A recurrent model that trains slowly because it is swapping to disk looks like a compute problem. tf.config.experimental.get_memory_info("GPU:0") or torch.cuda.memory_allocated() tell the true story in one line, and the fix is usually a shorter lookback, not a bigger machine.

In summary

• Backpropagation through time applies the chain rule across the unrolled sequence; computing it requires storing every hidden state of the forward pass.
• Memory scales linearly with sequence length, batch size and hidden width; doubling any of them doubles the cost, which is unforgiving on long series.
• Truncated BPTT limits the backward span to $k$ steps and accepts a biased gradient in exchange for a bounded memory cost; this is the standard practice.
• Stateful RNNs carry the hidden state across batches without the gradient; they demand fixed batch size, ordered batches and explicit state resets.

Next module: what actually happens to the product of Jacobians inside BPTT, and why long sequences make the gradient vanish or explode.