Module 2 — The recurrent neuron and its hidden state

Module 1 established the problem: a table of features throws away order, and adding lagged columns only works when the useful past is fixed. The recurrent neuron is the smallest object that carries information across time steps without a fixed horizon. This module unpacks its equation, watches it unroll over the electricity series, and pins down the tensor shapes that decide whether a layer runs or throws.

One equation, three ingredients

A recurrent neuron reads one input $x_t$ at each time step and updates a hidden state $h_t$ using the previous hidden state $h_{t-1}$:

$$h_t = \tanh(W_x x_t + W_h h_{t-1} + b)$$

$W_x$ transforms the current input, $W_h$ transforms the previous state, and $b$ is a bias. The activation is traditionally $\tanh$ because it keeps values bounded in $[-1, 1]$, which matters when the state is fed back into itself indefinitely; a $\operatorname{ReLU}$ can grow without bound and destabilise the recurrence in seconds.

The key point, and the one that separates recurrent networks from every architecture you have seen so far: the same $W_x$, $W_h$ and $b$ are used at every time step. A sequence of length 168 does not have 168 sets of weights; it has one set applied 168 times. That is what makes the model handle sequences of variable length and what keeps the parameter count independent of the sequence length.

Unrolling in time: a picture and a warning

An RNN is often drawn in two ways: the rolled form shows one neuron with a self-loop; the unrolled form draws one copy per time step, connected left to right. Both describe exactly the same object with the same weights; the unrolled diagram is only a visual aid for backpropagation, which the next module covers.

The warning: the unrolled network looks like a deep feedforward network 168 layers tall. It is not — the weights are tied — but for the gradient it behaves that way. That is why long sequences hit the gradient problems of module 4, and why LSTM and GRU exist.

Tensor shapes: (batch, time, features)

Keras and PyTorch agree on the same convention for recurrent layers: the input tensor has shape (batch, time, features). For the electricity series with a lookback of 168 hours and only the consumption as a feature, the shape is (batch_size, 168, 1) — the trailing 1 is easy to forget and produces obscure error messages.

import numpy as np

series = np.loadtxt("electricity_hourly.csv").astype("float32")
x = series[:8000]

lookback = 168
x_windows = np.lib.stride_tricks.sliding_window_view(x, lookback)[::1]
x_windows = x_windows[:, :, None]           # add the feature axis
print(x_windows.shape)                      # (N, 168, 1)

Adding a second feature — say the hour of the day encoded as a scalar — changes the trailing dimension to 2. Adding one-hot day-of-week columns turns it into (N, 168, 9). The time axis stays the middle one; only the last axis grows with the number of variables at each step.

return_sequences: hidden state at every step or only the last one

Every recurrent layer emits a hidden state at every time step. What the framework returns to the next layer depends on one flag.

• return_sequences=False (the default in Keras): the layer returns only the last hidden state, shape (batch, units). This is what a classifier or a single-value regressor wants.
• return_sequences=True: the layer returns the state at every step, shape (batch, time, units). This is required when the next layer is another recurrent layer (module 7) or when the task is sequence-to-sequence (module 8).

from tensorflow.keras import layers, Sequential

model = Sequential([
    layers.Input(shape=(168, 1)),
    layers.SimpleRNN(32, return_sequences=False),  # last step only
    layers.Dense(24),                              # forecast next 24 hours
])

The same model with return_sequences=True produces an output of shape (batch, 168, 32) which a Dense(24) cannot consume directly. Stacking a second recurrent layer or a TimeDistributed(Dense) layer becomes mandatory.

Weight sharing keeps the parameter count small

A SimpleRNN(32) on (batch, 168, 1) inputs has:

• $W_x$ of shape (1, 32) = 32 parameters
• $W_h$ of shape (32, 32) = 1 024 parameters
• $b$ of shape (32,) = 32 parameters

Total: 1 088 parameters, independent of the 168 time steps. A dense network reading the same 168 hours as a flat vector would need 168 × 32 = 5 376 weights in its first layer alone, and it would tie each weight to a specific time position — a Monday feature would live at a different index than a Tuesday feature. Weight sharing makes the recurrent network translation-invariant in time, which is the reason it generalises to sequences of unseen length.

Input(shape=(None, 1)): accepting variable length

Because the same weights are applied at every step, the layer does not need to know the sequence length at construction time. Passing None on the time axis is legal and produces a model that trains on 168-hour windows and can predict on any other length without re-instantiating:

model = Sequential([
    layers.Input(shape=(None, 1)),
    layers.SimpleRNN(32),
    layers.Dense(24),
])
model.summary()   # 1 088 + 33 * 24 parameters

This is not a cosmetic detail. Module 9 will use it to feed batches padded to variable lengths without recompiling the graph.

A missing feature axis produces an unhelpful message

Feeding a (batch, time) tensor instead of (batch, time, features) triggers a shape error deep inside the recurrent cell. The fix is a single x[..., None] or np.expand_dims(x, -1); the diagnosis takes fifteen minutes if you do not know the trick.

A first working RNN on the electricity series

import numpy as np
from tensorflow.keras import layers, Sequential

series = np.loadtxt("electricity_hourly.csv").astype("float32")
mu, sigma = series[:6000].mean(), series[:6000].std()
series_s = (series - mu) / sigma

def make_xy(s, lookback=168, horizon=24):
    x, y = [], []
    for t in range(len(s) - lookback - horizon + 1):
        x.append(s[t:t + lookback])
        y.append(s[t + lookback:t + lookback + horizon])
    return np.array(x)[..., None], np.array(y)

x_train, y_train = make_xy(series_s[:6000])
x_val,   y_val   = make_xy(series_s[6000 - 168:])

model = Sequential([
    layers.Input(shape=(168, 1)),
    layers.SimpleRNN(32),
    layers.Dense(24),
])
model.compile(optimizer="adam", loss="mse")
model.fit(x_train, y_train, validation_data=(x_val, y_val),
          epochs=10, batch_size=64, verbose=2)

Nine thousand training examples, twenty seconds per epoch on a laptop CPU, and a model that already beats the naive baseline on a well-behaved building. The interesting question — why it will collapse on longer sequences — is the subject of the next two modules.

Start with 16 or 32 units and increase only if the loss stalls

Doubling the units quadruples the recurrent matrix. On the electricity series, 32 units already saturate what a simple RNN can learn; going higher wastes training time and, worse, hides the fact that the architecture is the limitation, not the capacity.

In summary

• A recurrent neuron updates a hidden state $h_t = \tanh(W_x x_t + W_h h_{t-1} + b)$ using the same weights at every time step, so parameter count does not grow with sequence length.
• Input tensors have shape (batch, time, features); missing the trailing feature axis is the single most common error when starting.
• return_sequences=False returns only the last state, return_sequences=True returns all of them; the choice depends on whether the next layer is dense, recurrent or sequence-to-sequence.
• Weight sharing makes the model translation-invariant in time and allows a variable-length input axis with Input(shape=(None, features)).

Next module: how gradients are actually computed through this loop, and why unrolling 168 steps looks a lot like training a 168-layer network.