Module 8 — Deep approaches: LSTM and temporal Transformers
Gradient boosting won the M5. Deep learning did not. That is the empirical fact this module starts from, and it is not the whole story: on many related series, with rich external regressors, and long histories, deep models — especially temporal Transformers — do earn their cost. This module trains an LSTM and a temporal Transformer on the pharmacy data, gives a fair comparison against the LightGBM of module 7, and names the situations where the choice tilts either way.
Windowing: turning a series into a network input
Recurrent and attention-based models expect a tensor of shape (batch, timesteps, features) — the same convention we saw in the RNN course. Producing it from a series of daily values means slicing rolling windows.
import numpy as np, pandas as pd
def windows(series: np.ndarray, lookback: int, horizon: int):
X, y = [], []
for i in range(len(series) - lookback - horizon + 1):
X.append(series[i : i + lookback])
y.append(series[i + lookback : i + lookback + horizon])
return np.array(X), np.array(y)
series = pd.read_parquet("pharmacy_daily.parquet").set_index("date")["units"].to_numpy()
X, y = windows(series, lookback=90, horizon=28)
The lookback of 90 days lets the network see roughly three months, enough to observe both the weekly cycle and short-scale trend. Going deeper (365) helps only if we also give the model the volume of data to justify it; for a single-store four-year series, 90 to 180 is the sweet spot.
Chronological split. As always, we split by time, not by shuffle. On the M5, a shuffled window split makes any model look 30 % better than it is.
An LSTM baseline
We reuse the RNN course's building blocks — an LSTM followed by a dense head producing the 28 outputs at once.
import tensorflow as tf
from tensorflow.keras import layers, Model
def build_lstm(lookback: int, horizon: int, features: int = 1) -> Model:
inputs = layers.Input(shape=(lookback, features))
x = layers.LSTM(64, return_sequences=True)(inputs)
x = layers.LSTM(64)(x)
x = layers.Dropout(0.2)(x)
outputs = layers.Dense(horizon)(x)
return Model(inputs, outputs)
model = build_lstm(lookback=90, horizon=28)
model.compile(optimizer=tf.keras.optimizers.Adam(1e-3), loss="mae")
We scale the input to zero mean and unit variance using the training window's statistics only (never the full series — module 1 warned about this). Training for 30 epochs with early stopping on a validation block lands the LSTM at MAE 7.4 on the test window.
Two things a beginner often forgets. First, the output layer produces a 28-element vector, one per horizon step; there is no recursive rollout at inference. Second, the loss is L1 (mae); using MSE trains the model to be over-cautious in the tails and hurts the interval later.
The temporal Transformer
Transformers with causal attention (no peeking into the future) are today's default deep architecture on time series. Two flavors matter for us.
Vanilla temporal Transformer. Same encoder-decoder as in NLP, with position embeddings replaced by an explicit time-of-day and day-of-week embedding, and causal masks that prevent the decoder from attending to future positions. Simple, powerful, hungry for data.
Temporal Fusion Transformer (TFT) (Lim et al., 2021). Combines LSTM encoders, self-attention, and gated variable-selection heads that let the model turn features on and off per time step. It handles known-future regressors natively (they enter at every time step of the encoder), which matches exactly the distinction from module 7.
We show a small vanilla version.
from tensorflow.keras import layers, Model
def temporal_transformer(lookback, horizon, features=1, d=64, heads=4):
inp = layers.Input(shape=(lookback, features))
x = layers.Dense(d)(inp)
x = x + layers.Embedding(lookback, d)(tf.range(lookback))
for _ in range(2):
a = layers.MultiHeadAttention(num_heads=heads, key_dim=d // heads,
use_causal_mask=True)(x, x)
x = layers.LayerNormalization()(x + a)
f = layers.Dense(d * 4, activation="relu")(x)
f = layers.Dense(d)(f)
x = layers.LayerNormalization()(x + f)
z = layers.GlobalAveragePooling1D()(x)
out = layers.Dense(horizon)(z)
return Model(inp, out)
Trained on a single four-year store's series, this model underperforms — it overfits, and validation MAE stalls around 8.2. Trained on the combined data of 200 stores with a store-id embedding, the same model reaches MAE 6.7, edging LightGBM. That is not accidental.
Foundation models: a preview
A new class of models — time-series foundation models — deserves a mention because it changes the trade-offs on the horizon of the next two years.
Models such as TimeGPT, Chronos, Lag-Llama and Moirai are Transformers pretrained on hundreds of thousands of series across domains. At inference time they take a lookback window and produce a probabilistic forecast without any per-user training. For our pharmacy problem, the zero-shot MAE of a small Chronos model is around 8.5 — worse than a properly fit local model, but roughly on par with SARIMA and Prophet, with no fitting effort.
The interesting question is not "are they best today?" (they are usually not on a single well-studied series) but "when they are close enough, does the operational simplicity of not maintaining a model justify a small metric loss?". On a small team with hundreds of low-value series, that answer increasingly is yes.
When deep learning wins
Three conditions predict a deep model's win.
Many related series. The M5 covered 30 000 series. LightGBM won there too, but the gap to deep models was single-digit percentage points, and Transformers dominated on the sub-tasks with the largest number of series. On our 200-store chain, a single deep model has enough data to learn cross-store patterns.
Rich external regressors. Weather, prices, calendar, mobility. TFT handles them cleanly; LightGBM needs manual feature engineering. When the number of features climbs above a hundred with meaningful interactions, deep models catch up and often pass.
Long, high-frequency series. Hourly or sub-hourly data over years of history stresses a tree-based model's memory in ways an attention-based model handles better. Electricity, network traffic, and stock-tick data are the poster examples.
When deep learning loses
Three conditions predict a deep model's loss.
Few series, short history. A single store with two years of history has roughly 700 daily points. That is far too few to fit a Transformer without overfitting, and a well-tuned SARIMA or LightGBM will win by a wide margin. Deep learning starts to earn its cost around 5 000 to 10 000 observations per series, or across many series.
Very stable seasonality with few external effects. If the series is essentially "weekly seasonality plus a slow trend", a Holt-Winters or SARIMA is inside 5 % of the best deep model at 5 % of the operational cost. This is the case for many operational series in retail and services.
Small operations team. Deep models require training infrastructure, monitoring for training drift, and specialized engineers. On a team of two data scientists with 200 series, LightGBM on a nightly job is the pragmatic answer for years.
Results on our pharmacy chain
| Model | MAE (units/day) | Notes |
|---|---|---|
| naive | 24.6 | |
| seasonal_naive_7 | 9.8 | |
| SARIMA(1,1,1)(0,1,1) | 8.4 | one seasonality only |
| Holt-Winters | 8.9 | |
| Prophet with FR holidays | 7.6 | |
| LightGBM (lags + calendar) | 6.9 | winner on single-store view |
| LightGBM + planned promotions | 6.3 | |
| LSTM (single store) | 7.4 | |
| Temporal Transformer (global, 200 stores) | 6.5 | needs the global setup |
| Chronos (zero-shot) | 8.5 | no fit, useful floor |
The Transformer edges LightGBM only when trained on the pooled 200-store data. Single-store LSTM is worse than plain gradient boosting. That is the honest result on this class of series.
Every retail forecasting talk in the last five years opens with a slide saying "deep learning transforms forecasting" and closes with a case study won by gradient boosting. The truth is closer to a rule than a preference: deep models earn their cost on many related series with rich features and long history, and they lose otherwise. Pick according to the shape of your problem, not the fashion of the conference.
Summary
- Deep models expect a windowed input
(batch, timesteps, features); the split remains chronological and scaling uses training-window statistics only. - LSTM and temporal Transformers are the two main architectures; TFT specifically handles known-future regressors natively.
- Foundation models (TimeGPT, Chronos, Moirai) offer zero-shot probabilistic forecasts and are a genuine floor to consider on low-value series.
- Deep wins on many series, rich regressors, long history; loses on single series, short history, stable seasonality. On our pharmacy chain, a global Transformer edges LightGBM only in the global setup.
Next module: rolling-origin validation and the metrics that fit each situation. Everything above was measured on a single test window; module 9 replaces that with the honest evaluation any deployment requires.