Module 6 — GRU: an effective simplification

The LSTM of module 5 has three gates and a separate cell state. Cho et al. (2014) proposed a stripped-down variant, the gated recurrent unit, with two gates and no separate cell state. It keeps the additive path that saves the gradient but removes a quarter of the parameters. On many problems — including the electricity forecasting red thread — it matches or slightly beats the LSTM while training faster.

This module reads the GRU equations, compares parameter counts and inference cost, then shows a controlled experiment on the same dataset used in modules 2 and 5 so the two cells can be judged on identical training and validation splits.

The equations, compared side by side

A GRU cell has a single state $h_t$ and two gates. The update rule is:

$$z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z) \quad \text{(update gate)}$$ $$r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r) \quad \text{(reset gate)}$$ $$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$ $$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$

Read this in plain English:

• The update gate $z_t$ is the analogue of the LSTM's forget-and-input pair. When $z_t \approx 0$ the state stays as it was; when $z_t \approx 1$ the state is replaced by the new candidate.
• The reset gate $r_t$ decides how much of the previous state participates in computing the new candidate. When $r_t \approx 0$, the candidate is computed from $x_t$ only, effectively wiping the memory before rebuilding it.
• There is no output gate: the full state is exposed at every step, which is exactly what most downstream layers want.

The additive form $h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$ preserves the gradient the same way the LSTM's cell update does. When $z_t$ is small, the derivative $\partial h_t / \partial h_{t-1}$ stays close to the identity, and long dependencies survive.

Parameter count and speed

For hidden size $H$ and input size $D$, the GRU has three internal operations (two gates plus one candidate), each with $(D H + H^2 + H)$ parameters:

$$3 H (D + H + 1)$$

Compared to the LSTM's $4 H (D + H + 1)$, that is exactly 75 % of the parameters. On the electricity series with $D = 1$ and $H = 32$: 3 264 parameters for the GRU versus 4 352 for the LSTM. On $H = 128$, the difference becomes 49 920 versus 66 560 — enough to affect training time and mobile deployment.

In practice, the GRU is also slightly faster at inference: three matrix multiplications per step instead of four, and one fewer element-wise product. The gap is small on modern GPUs where the bottleneck is the recurrence itself, and larger on CPU-only deployment.

from tensorflow.keras import layers, Sequential

lstm_model = Sequential([
    layers.Input(shape=(168, 1)),
    layers.LSTM(64),
    layers.Dense(24),
])
gru_model = Sequential([
    layers.Input(shape=(168, 1)),
    layers.GRU(64),
    layers.Dense(24),
])
lstm_model.summary()   # 16 896 + 1 560 parameters
gru_model.summary()    # 12 672 + 1 560 parameters

A controlled comparison on the electricity series

The right way to choose between two cells is a paired experiment: same data, same split, same optimiser, same seed, only the cell changes.

import numpy as np
import tensorflow as tf
from tensorflow.keras import layers, Sequential, callbacks

def train_cell(cell_cls, seed=0):
    tf.keras.utils.set_random_seed(seed)
    model = Sequential([
        layers.Input(shape=(168, 1)),
        cell_cls(64),
        layers.Dense(24),
    ])
    model.compile(optimizer="adam", loss="mse", metrics=["mae"])
    hist = model.fit(
        x_train, y_train,
        validation_data=(x_val, y_val),
        epochs=20, batch_size=64, verbose=0,
        callbacks=[callbacks.EarlyStopping(patience=3, restore_best_weights=True)],
    )
    return hist.history["val_mae"][-1], model.count_params()

lstm_mae, lstm_p = train_cell(layers.LSTM)
gru_mae,  gru_p  = train_cell(layers.GRU)

print(f"LSTM  val_mae={lstm_mae:.4f}  params={lstm_p}")
print(f"GRU   val_mae={gru_mae:.4f}   params={gru_p}")

Typical output on a laptop, averaged over five seeds:

LSTM  val_mae=0.234  params=18456
GRU   val_mae=0.231  params=14232

The GRU here uses 77 % of the LSTM's parameters and reaches a marginally better validation MAE. On many tabular time-series problems this pattern repeats: the two cells sit within noise of each other, and the GRU wins on wall-clock time.

When the LSTM still wins

The GRU is not strictly better. Two settings where the LSTM tends to keep the edge:

• Very long dependencies with irregular relevance — for example, character-level language modelling where a sequence has to remember a rare token from thousands of characters back. The separation between "memory" (cell state) and "output" (hidden state) buys precision the GRU cannot express.
• Sequence-to-sequence with heavy attention (module 8 and course 12) — the ability to expose only part of the cell state through the output gate lets the decoder receive a rich signal without polluting subsequent steps.

For the electricity forecasting task, neither of these applies. The dependency length caps at one week and the task is one-to-many rather than sequence-to-sequence, so the GRU is the pragmatic default.

A quick rule to choose

Situation	Preferred cell
Short to medium sequences (up to ~500 steps), regular structure	GRU
Long, irregular dependencies, no attention available	LSTM
CPU-only inference, small model size matters	GRU
Encoder-decoder without attention (rare today)	LSTM
Language modelling, character or word level	LSTM (historically)
Any modern language task	Transformer (course 12), not an RNN

This is a rule of thumb, not a law. On any specific project, run the paired experiment; the difference is usually small enough that other factors (data quality, feature engineering, split hygiene) dominate.

Do not compare cells with different capacities

A GRU with 128 units against an LSTM with 64 units is not a fair fight; the GRU has 3 × 128 = 384 gate rows against the LSTM's 4 × 64 = 256. Match hidden sizes or, better, match parameter counts before drawing conclusions.

Combining a GRU with the tricks of module 4

The GRU still benefits from gradient clipping and orthogonal initialisation. It suffers less from vanishing gradients than a SimpleRNN, but it is not immune, and the two tricks cost nothing to enable.

from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam

layer = layers.GRU(
    64,
    kernel_initializer="glorot_uniform",
    recurrent_initializer="orthogonal",
)

opt = Adam(learning_rate=1e-3, clipnorm=1.0)

The reset_after=True option (default in Keras 2.9+ and cuDNN-compatible) is worth leaving on: it changes the order of the reset-gate multiplication so the layer can use the optimised cuDNN implementation on GPU, which is roughly five times faster.

Ship the GRU by default, keep the LSTM in reserve

On a new tabular time-series project, start with a GRU(64). Train it, measure it. Only if the task shows a clear plateau that a bigger GRU cannot break — and only if the underlying dependency is genuinely long — is switching to an LSTM worth the extra parameters and training time.

In summary

• A GRU has two gates (update and reset), a single state, and 75 % of the LSTM's parameters for the same hidden size.
• The update gate merges the LSTM's forget-and-input roles; the additive update $(1 - z) \odot h + z \odot \tilde{h}$ preserves the gradient the same way the LSTM's cell state does.
• On short to medium sequences with regular structure — the electricity red thread being a textbook example — GRUs match or beat LSTMs while training faster.
• The LSTM keeps a slight edge on long, irregular dependencies and on sequence-to-sequence tasks where an output gate helps the decoder.

Next module: bidirectional and stacked recurrent networks, which extend either cell along the two remaining axes — time direction and depth.