Module 10 — Project: forecasting a series of measurements

Modules 1 to 9 established every piece needed to build a real forecasting system. This module assembles them on the red-thread dataset: hourly electricity consumption of a small office building over one year, with the operational question "what will consumption be during each of the next 24 hours?". The deliverable is not just a single number per hour but a prediction interval obtained from quantile losses, because a point forecast without a spread is dangerous in operations.

Framing the problem

• Input: a rolling window of 168 hours (one week) of past consumption, plus three calendar features (hour of day, day of week, is-weekend) so the model does not have to reinvent the calendar from scratch.
• Output: the next 24 hours of consumption, as a point forecast and an 80 % prediction interval (10th and 90th quantiles).
• Metric: mean absolute error (MAE) on the point forecast, plus the pinball loss on the quantile forecasts to check interval calibration.
• Baseline to beat: yesterday's 24 hours as the naive forecast.

Loading and splitting the data

import numpy as np
import pandas as pd

df = pd.read_csv("electricity_hourly.csv", parse_dates=["timestamp"])
df = df.sort_values("timestamp").reset_index(drop=True)

series = df["consumption_kw"].to_numpy(dtype="float32")
hour   = df["timestamp"].dt.hour.to_numpy(dtype="float32") / 23.0
dow    = df["timestamp"].dt.dayofweek.to_numpy(dtype="float32") / 6.0
weekend = (df["timestamp"].dt.dayofweek >= 5).to_numpy(dtype="float32")

features = np.stack([series, hour, dow, weekend], axis=1)   # (N, 4)

lookback, horizon = 168, 24
n = len(features)
train_end = int(0.7 * n)
val_end   = int(0.85 * n)

train = features[: train_end]
val   = features[train_end - lookback : val_end]
test  = features[val_end - lookback :]

mu    = train[:, 0].mean()
sigma = train[:, 0].std()

Only the consumption channel is normalised — the calendar features are already in $[0, 1]$.

The naive baseline, computed first

def naive_forecast(x_lookback):
    # Predict the same 24 hours as yesterday
    return x_lookback[-24:, 0] * sigma + mu

def make_windows(feats):
    x, y = [], []
    for t in range(len(feats) - lookback - horizon + 1):
        x.append(feats[t : t + lookback])
        y.append(feats[t + lookback : t + lookback + horizon, 0])
    return np.array(x), np.array(y)

x_test, y_test = make_windows(test)
x_test[:, :, 0] = (x_test[:, :, 0] - mu) / sigma

naive_pred = np.array([naive_forecast(w) for w in x_test])
naive_mae  = np.abs(naive_pred - (y_test * sigma + mu)).mean()
print(f"Naive MAE: {naive_mae:.2f} kW")

On a typical office building the naive baseline sits around 2.5 to 3.5 kW MAE. Any LSTM or GRU that does not beat that by a clear margin is not learning anything the series does not already say.

Two candidate models: LSTM and GRU

import tensorflow as tf
from tensorflow.keras import layers, Sequential, callbacks

def build_model(cell):
    return Sequential([
        layers.Input(shape=(lookback, 4)),
        cell(64, return_sequences=True, dropout=0.2),
        cell(32, dropout=0.2),
        layers.Dense(horizon),
    ])

def train_model(model, x_train, y_train, x_val, y_val, epochs=25):
    model.compile(
        optimizer=tf.keras.optimizers.Adam(1e-3, clipnorm=1.0),
        loss="mse", metrics=["mae"],
    )
    return model.fit(
        x_train, y_train,
        validation_data=(x_val, y_val),
        epochs=epochs, batch_size=64, verbose=2,
        callbacks=[callbacks.EarlyStopping(patience=4, restore_best_weights=True)],
    )

x_train, y_train = make_windows(train)
x_train[:, :, 0] = (x_train[:, :, 0] - mu) / sigma

x_val, y_val = make_windows(val)
x_val[:, :, 0] = (x_val[:, :, 0] - mu) / sigma

lstm_model = build_model(layers.LSTM)
gru_model  = build_model(layers.GRU)

train_model(lstm_model, x_train, y_train, x_val, y_val)
train_model(gru_model,  x_train, y_train, x_val, y_val)

Both use gradient clipping (module 4), stacking with plain dropout instead of recurrent dropout for GPU speed (module 7), and orthogonal initialisation by default (module 4 again). On a modern laptop each model trains in 5 to 10 minutes.

Evaluation on the test set:

def test_mae(model):
    pred = model.predict(x_test, verbose=0) * sigma + mu
    return np.abs(pred - (y_test * sigma + mu)).mean()

print(f"LSTM test MAE: {test_mae(lstm_model):.2f} kW")
print(f"GRU  test MAE: {test_mae(gru_model):.2f} kW")
print(f"Naive baseline: {naive_mae:.2f} kW")

On a well-behaved building the two models land within a few percent of each other and reduce the naive MAE by 20 to 40 %. The GRU is usually 10 to 20 % faster to train for that outcome, which is why module 6 recommended it as the default.

Quantile intervals: from a point forecast to a range

A point forecast alone is a false promise. The pinball loss trains one model per quantile:

$$\mathcal{L}_\tau(y, \hat{y}) = \max\left( \tau (y - \hat{y}), (\tau - 1)(y - \hat{y}) \right)$$

For $\tau = 0.5$, the pinball loss is proportional to MAE and produces a median forecast. For $\tau = 0.1$ it produces the 10th percentile, for $\tau = 0.9$ the 90th percentile. Together, the pair defines an 80 % prediction interval.

def pinball_loss(tau):
    def loss(y_true, y_pred):
        e = y_true - y_pred
        return tf.reduce_mean(tf.maximum(tau * e, (tau - 1) * e))
    return loss

def build_quantile_model(tau):
    m = build_model(layers.GRU)
    m.compile(optimizer=tf.keras.optimizers.Adam(1e-3, clipnorm=1.0),
              loss=pinball_loss(tau))
    return m

q10 = build_quantile_model(0.1)
q50 = build_quantile_model(0.5)
q90 = build_quantile_model(0.9)

for m in (q10, q50, q90):
    m.fit(x_train, y_train, validation_data=(x_val, y_val),
          epochs=25, batch_size=64, verbose=0,
          callbacks=[callbacks.EarlyStopping(patience=4, restore_best_weights=True)])

Interval quality is checked by coverage: the fraction of test targets that fall inside [q10, q90]. If the models are well-calibrated, coverage is close to 80 %.

lower  = q10.predict(x_test, verbose=0) * sigma + mu
upper  = q90.predict(x_test, verbose=0) * sigma + mu
actual = y_test * sigma + mu
coverage = np.mean((actual >= lower) & (actual <= upper))
print(f"80 % interval coverage: {coverage:.1%}")

Under-coverage (say 60 %) means the interval is too narrow: the model is overconfident. Over-coverage (say 95 %) means the interval is too wide: the model hedges too much. Both are honest to report; a single point forecast never is.

Mistakes to avoid on a forecasting project

• Skipping the naive baseline. A hundred-thousand-parameter LSTM that ties with y_pred = x[:, -24:, 0] has learned nothing useful.
• Shuffled splits or overlapping windows. Reviewed in module 1; still the top cause of overestimated performance.
• Bidirectional layers on the lookback that peek at the horizon. Reviewed in module 7; a silent leakage bug.
• Fitting the scaler on the whole series. Reviewed in module 9; leaks the future into the training normalisation.
• Reporting a single MAE without an interval. Sends a false signal of certainty to operations, which then plans capacity on the wrong assumption.
• Training many models on many seeds and reporting only the best one. Cherry-picking; report the median and the spread over seeds.

A single random seed is a promise you cannot keep

Small recurrent networks on modest datasets often show 5 to 10 % variance across seeds. Any comparison — LSTM vs GRU, one layer vs two, dropout 0.1 vs 0.3 — that does not average over at least three seeds is an anecdote, not a result.

Package the pipeline as three functions and freeze the interfaces

load_and_split(), train_models(), evaluate_with_intervals(). When the building changes or the horizon shifts from 24 to 48 hours, only the arguments move. This is the smallest engineering discipline that turns a notebook into a system.

In summary

• The naive baseline (yesterday's 24 hours) is the reference every model must beat by a clear margin.
• Stacked LSTM or GRU with modest dropout and gradient clipping reduces MAE by 20 to 40 % on a typical office-building series; the GRU is usually the pragmatic default.
• Quantile regression with the pinball loss produces prediction intervals; coverage on the test set tells you whether the intervals are honest.
• Every failure mode from modules 1, 4, 7 and 9 shows up again on this project — a good pipeline is what keeps them out.

Next module: the recap and the 40-question exam that closes the course and prepares the ground for attention and Transformers in course 12.