Module 9 — Splitting, padding and batching sequences in practice

Modules 1 to 8 focused on the model. This one is entirely about the pipeline: how the sliding windows of module 1 get combined into batches, how sequences of different lengths are aligned without polluting the loss, and how normalisation is applied without leaking future information. On a real project, more RNN failures come from a bad pipeline than from a bad architecture.

The examples use the electricity red thread again and give the concrete tf.data and PyTorch DataLoader equivalents side by side, because both are common and behave slightly differently on the same problem.

Windowing done efficiently

The naive Python loop of module 1 works, but it copies data. Both frameworks offer streaming windowers that share memory.

TensorFlow, with tf.data.Dataset.window:

import tensorflow as tf

def make_dataset(series, lookback, horizon, batch_size, shuffle_buffer=1024):
    ds = tf.data.Dataset.from_tensor_slices(series)
    ds = ds.window(lookback + horizon, shift=1, drop_remainder=True)
    ds = ds.flat_map(lambda w: w.batch(lookback + horizon))
    ds = ds.map(lambda w: (w[:lookback, tf.newaxis], w[lookback:]))
    ds = ds.shuffle(shuffle_buffer).batch(batch_size).prefetch(tf.data.AUTOTUNE)
    return ds

PyTorch, with stride_tricks.sliding_window_view and a custom Dataset:

import numpy as np
from torch.utils.data import Dataset, DataLoader

class WindowDataset(Dataset):
    def __init__(self, series, lookback, horizon):
        w = np.lib.stride_tricks.sliding_window_view(series, lookback + horizon)
        self.x = w[:, :lookback, None].astype("float32")
        self.y = w[:, lookback:].astype("float32")
    def __len__(self): return len(self.x)
    def __getitem__(self, i): return self.x[i], self.y[i]

dl = DataLoader(WindowDataset(series, 168, 24), batch_size=64, shuffle=True)

The sliding_window_view returns a view into the array — no copy — so the memory cost is a single contiguous buffer, not the naive $N \times L$ duplicate.

Padding: aligning variable-length sequences

When sequences have variable length (translation, NER, arbitrary-length event logs), a batch cannot be a rectangular tensor unless the short ones are padded to the longest. Padding on the right with a special token — usually 0 — is the standard choice.

from tensorflow.keras.utils import pad_sequences

sentences = [[3, 8, 1], [5, 2], [9, 7, 4, 6]]
padded = pad_sequences(sentences, padding="post", value=0)
# array([[3, 8, 1, 0],
#        [5, 2, 0, 0],
#        [9, 7, 4, 6]])

Padding on the right (padding="post") is preferable to left-padding for recurrent networks: the useful signal comes first, the padding trails behind, and the hidden state at the last real token is easy to extract.

Masking: telling the loss to ignore padding

A padded batch is a lie the model is not supposed to be fooled by. Masking tells the recurrent layer to skip the padding steps and tells the loss to exclude them. Keras handles both automatically when the mask is set at the source:

from tensorflow.keras import layers

# Method 1: an Embedding layer with mask_zero=True propagates the mask downstream
model.add(layers.Embedding(vocab_size, 128, mask_zero=True))
model.add(layers.LSTM(64))    # correctly ignores masked steps

# Method 2: a Masking layer, useful when there is no Embedding
model.add(layers.Masking(mask_value=0.0, input_shape=(None, features)))

PyTorch's pack_padded_sequence and pad_packed_sequence do the same job manually:

import torch
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

packed = pack_padded_sequence(x, lengths, batch_first=True, enforce_sorted=False)
out_packed, h = rnn(packed)
out, _ = pad_packed_sequence(out_packed, batch_first=True)

Forgetting to mask produces a subtle bug: the padded zeros participate in the hidden-state update, and the model learns to output whatever gives the lowest loss on a long trailing run of zeros. On sentiment classification, this typically pulls every prediction toward the majority class.

Masking must reach both the layer and the loss

A masked recurrent layer that feeds an unmasked loss still counts the padding in the average. Use sample_weight in Keras or a masked loss in PyTorch, or the mean is diluted by the padding steps.

Sorting by length: fewer wasted computations

Padding is wasteful when sequences in a batch have very different lengths: the batch pays for the longest one. Two mitigations:

• Bucketing: group sequences by length before batching so each batch has similar lengths. tf.data.Dataset.bucket_by_sequence_length does this in TensorFlow; a Sampler does it in PyTorch.
• Sorting: sort the whole dataset by length once and iterate in order. Faster batches, but breaks true randomness — pair it with a modest shuffle within a bucket.

ds = ds.bucket_by_sequence_length(
    element_length_func=lambda x, y: tf.shape(x)[0],
    bucket_boundaries=[16, 32, 64, 128],
    bucket_batch_sizes=[64, 64, 32, 16, 8],
)

On a translation corpus, bucketing cuts wall-clock training time by 30 to 50 % for the same number of gradient steps, at no cost to convergence.

Per-window normalisation without leakage

Module 1 fitted a global scaler on the training portion. That is the correct default. Two variations show up in practice, and both have a leakage trap.

The first is per-window normalisation: subtract each window's mean, divide by its standard deviation. This works well on drifting series (a slow trend confuses a global scaler), but the statistics must be computed on the lookback only, never on the horizon:

def normalise_window(x_window, y_window):
    mu, sigma = x_window.mean(), x_window.std() + 1e-6
    return (x_window - mu) / sigma, (y_window - mu) / sigma

The second is rolling statistics: use a rolling mean and standard deviation computed causally up to time $t$. Slightly more expensive, more principled, and identical in intent — no future values leak into the normalisation.

Building the training pipeline end to end

Pulling everything together, the pipeline for the electricity red thread — used in module 10 — looks like this:

import tensorflow as tf
import numpy as np

series = np.loadtxt("electricity_hourly.csv").astype("float32")
lookback, horizon = 168, 24

n = len(series)
train, val, test = series[: int(0.7 * n)], series[int(0.7 * n) - lookback : int(0.85 * n)], series[int(0.85 * n) - lookback :]

mu, sigma = train.mean(), train.std()
train, val, test = (train - mu) / sigma, (val - mu) / sigma, (test - mu) / sigma

def to_dataset(series, batch_size=64):
    ds = tf.data.Dataset.from_tensor_slices(series)
    ds = ds.window(lookback + horizon, shift=1, drop_remainder=True)
    ds = ds.flat_map(lambda w: w.batch(lookback + horizon))
    ds = ds.map(lambda w: (w[:lookback, tf.newaxis], w[lookback:]))
    return ds.batch(batch_size).prefetch(tf.data.AUTOTUNE)

train_ds, val_ds, test_ds = to_dataset(train), to_dataset(val), to_dataset(test)

Three details make this pipeline honest:

• The split points are contiguous in time; no shuffle across the boundary.
• Validation and test both include a -lookback offset at the start so their first window does not consume training data.
• Normalisation uses only training statistics.

A checklist before hitting fit

[ ] Split is chronological, no shuffle across the boundary
[ ] Scaler was fitted on training only, never on validation or test
[ ] Windows never overlap the split boundary
[ ] Padding uses `padding="post"` and a mask flows to the loss
[ ] Lookback is short enough that BPTT fits in memory
[ ] A naive baseline (persistence, seasonal average) has been computed for comparison
[ ] The pipeline yields a batch the model can consume: `for x, y in ds.take(1): print(x.shape, y.shape)`

Running through this list once at the start of every recurrent project prevents the majority of costly bugs. Modules 1, 4, 5 and 7 have each fixed one specific failure mode; this pipeline is where they compound if the setup is wrong.

A small .take(1) inspection is worth ten paragraphs of documentation

Before every long training run, iterate once through the dataset and print the shapes, the dtypes and the min/max of the first batch. Ninety percent of "the model won't train" bugs show up in that one line.

In summary

• Streaming windowers (tf.data.window, sliding_window_view) turn a series into overlapping examples without duplicating memory.
• Padding on the right aligns variable-length sequences; a mask must reach both the recurrent layer and the loss for the padding not to poison training.
• Bucket by length to avoid paying for the longest sequence in every batch; the wall-clock gain on translation is 30 to 50 %.
• Normalise on training statistics only and, when using per-window normalisation, restrict the statistics to the lookback so no future value leaks in.

Next module: the full end-to-end project on the electricity series, combining LSTM, GRU, quantile prediction intervals and the naive baseline into a report-ready pipeline.