Module 7 — Bidirectional and stacked networks

Modules 5 and 6 stayed with a single recurrent layer moving forward in time. Two extensions push the architecture further: stacking several recurrent layers to build depth, and running an additional layer backward in time so every step sees both its past and its future. Both are cheap in code and consequential in results — but one of them has a very specific requirement that ruins forecasting projects when ignored.

Bidirectional networks: two passes, one output per step

A bidirectional recurrent layer runs two independent recurrences over the same input: one forward in time, one backward. At each step $t$, the two hidden states are concatenated (by default) into a vector of size $2 H$:

$$h_t^{\text{bi}} = [\, \overrightarrow{h_t} \,;\, \overleftarrow{h_t} \,]$$

The value at $t$ therefore summarises the whole sequence, past and future. Named-entity recognition, part-of-speech tagging, sentiment classification of a completed review — any task where the entire sequence is available before prediction — benefits from bidirectionality.

from tensorflow.keras import layers, Sequential

model = Sequential([
    layers.Input(shape=(None, 100)),   # token embeddings, variable length
    layers.Bidirectional(layers.LSTM(64, return_sequences=True)),
    layers.TimeDistributed(layers.Dense(9, activation="softmax")),  # NER tags
])

The output shape of Bidirectional(LSTM(64)) on (batch, T, D) inputs is (batch, T, 128), and the parameter count is exactly twice that of a plain LSTM — there are two full sets of weights, one per direction.

The one situation where bidirectional is forbidden

Forecasting is not one of them. In a real-time forecasting problem — the electricity red thread of this course — the model must predict future values from past values only. A bidirectional layer applied on the lookback window is fine if it stays inside the window (the whole past week is available at prediction time), but it is often confused with a broken pipeline that lets the backward pass see the target itself.

The critical rule: the model must never see a value from a time later than the target it is predicting. On the electricity series, that means:

• Bidirectional on the lookback of past 168 hours: acceptable.
• Bidirectional on a window that includes the horizon: catastrophic leakage.

The mistake is easy to make with a poorly-written data pipeline. If the sliding window function accidentally returns x[t:t+lookback+horizon] instead of x[t:t+lookback], the model sees the answer during training, reports 99 % accuracy in validation, and collapses in production.

A bidirectional RNN reaching 99 % on a forecasting task is almost always a leakage bug

Real-time forecasting problems bounded by autocorrelation top out at 90 to 95 % of the naive-baseline improvement. Anything drastically higher is the model reading the answer through the pipeline. Trace back to the windowing function before celebrating.

Stacking: depth in a recurrent network

Stacked recurrent networks apply several recurrent layers one on top of the other. The output of the first layer (a tensor of hidden states over time) becomes the input to the second layer, and so on. This gives the model a hierarchy: the lower layers capture short patterns, higher layers combine them into longer motifs.

model = Sequential([
    layers.Input(shape=(168, 1)),
    layers.LSTM(64, return_sequences=True),   # must return sequences!
    layers.LSTM(64),                          # returns only the last state
    layers.Dense(24),
])

The return_sequences=True on the first layer is mandatory: the second recurrent layer needs one input per time step, not a single vector. Forgetting it produces an error message about the wrong number of dimensions.

Two or three stacked layers is the practical maximum for a raw RNN, LSTM or GRU. Deeper stacks train slowly and rarely help beyond depth three, because each layer already integrates over the whole sequence. Residual connections between recurrent layers (rare in Keras, common in research code) can push the depth further.

Recurrent dropout: not the same as plain dropout

Standard dropout applied between two recurrent layers is fine — it drops values in the feed-forward direction. But applying it inside the recurrence — that is, dropping values as they pass from $h_{t-1}$ to $h_t$ — is more subtle, because a naive implementation drops different units at every time step, which destroys the state.

Gal and Ghahramani (2016) proposed the correct form, now standard: use the same dropout mask across all time steps of one sequence. Keras exposes it as recurrent_dropout:

layer = layers.LSTM(
    64,
    dropout=0.2,             # applied to the input at each step
    recurrent_dropout=0.1,   # applied to the state, same mask across time
)

The catch: recurrent_dropout > 0 disables the cuDNN kernel on GPU, and the layer becomes five to ten times slower. On a CPU it does not matter. On a GPU, prefer plain dropout between stacked layers unless there is measurable overfitting inside a single recurrent layer.

A stacked, bidirectional NER model in one code block

To see everything together on a task where bidirectionality is legitimate:

from tensorflow.keras import layers, Sequential

vocab_size = 20_000
max_len    = 60
tag_count  = 9

model = Sequential([
    layers.Input(shape=(max_len,), dtype="int32"),
    layers.Embedding(vocab_size, 100, mask_zero=True),
    layers.Bidirectional(layers.LSTM(128, return_sequences=True, dropout=0.2)),
    layers.Bidirectional(layers.LSTM(64,  return_sequences=True, dropout=0.2)),
    layers.TimeDistributed(layers.Dense(tag_count, activation="softmax")),
])
model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"],
)

The mask_zero=True on the embedding is essential: it tells the downstream LSTMs to ignore the padding tokens introduced when sequences are aligned to max_len. Module 9 covers padding and masking in detail; this is a preview.

Depth for the electricity forecast: usually not worth it

On the red thread, a two-layer stacked LSTM sometimes squeezes an additional 5 to 10 % out of the validation MAE. Beyond that, the model overfits the training seasonality and generalises poorly to the following year. Simpler wins:

• A better feature encoding (hour of day, day of week as sinusoidal features, temperature if available) helps far more than depth.
• A longer lookback with a GRU sometimes captures the signal that depth would try to synthesise.
• A dropout of 0.1 to 0.2 between two stacked layers is the sweet spot to prevent memorisation.

model = Sequential([
    layers.Input(shape=(168, 4)),                     # consumption + 3 calendar features
    layers.LSTM(64, return_sequences=True, dropout=0.2),
    layers.LSTM(32, dropout=0.2),
    layers.Dense(24),
])

Four features and two layers already push the LSTM into the regime where the electricity forecast really improves. This is the shape of the model used in the module 10 project.

For sequence classification, bidirectional first; for forecasting, forward-only first

The two paradigmatic mistakes are opposite: adding bidirectionality to a forecast (leakage) and forgetting it on a classification (leaving performance on the table). Decide once, at project start, whether the whole sequence is available before prediction.

In summary

• Bidirectional RNNs run two passes, forward and backward, and expose $[h^{\rightarrow}; h^{\leftarrow}]$ at every step; use them when the whole sequence is available at prediction time.
• Bidirectional on a forecasting task is a leakage red flag; the model must never see values later than the target, and a bidirectional layer that spans the horizon is a bug, not a feature.
• Stacked RNNs require return_sequences=True on every layer except the last; two to three layers is the practical maximum before diminishing returns.
• Recurrent dropout uses the same mask across time steps and disables the cuDNN kernel on GPU; between stacked layers, plain dropout is a faster substitute.

Next module: the encoder-decoder architecture, which uses two recurrent networks and steps outside the electricity red thread to translate short sentences.