Module 10 — Project: demand forecasting with intervals

Nine modules have built up a comparison of forecasting methods on the pharmacy chain's daily demand. This module consolidates the results, defends the model we deploy, produces quantile intervals, plans the retraining cadence, and — the honest part of any forecasting project — decides how to communicate the remaining uncertainty to the procurement team.

The final results table

The rolling-origin backtest of module 9 gave us a stable ranking. Here it is, on 24 rolling origins spaced one week apart over the last six months of history:

Model	MAE	MASE	80 % coverage	Interval score	Fit time
naive	24.6	2.51	—	—	0
seasonal_naive_7	9.8	1.00	—	—	0
SARIMA	8.6	0.88	84 %	42.1	3 s
Holt-Winters	8.8	0.90	79 %	44.7	1 s
Prophet + FR holidays	7.7	0.79	68 %	48.9	6 s
LightGBM + calendar	7.0	0.71	—	—	4 s
LightGBM + planned promos + quantile	6.4	0.65	81 %	31.8	12 s
LSTM (single store)	7.5	0.77	74 %	41.2	90 s
Transformer (global)	6.7	0.68	82 %	30.9	20 min
Chronos zero-shot	8.6	0.88	76 %	39.4	0 s (no fit)

Two winners emerge: quantile-regressed LightGBM on the local, one-store view, and the global temporal Transformer on the fleet view. Both beat the seasonal-naïve by roughly 35 %; both produce intervals whose coverage is within 1 point of the nominal 80 %.

The Transformer's slightly better interval score does not close the operational gap. Its 20-minute fit time and its need to be trained across the whole fleet make it a stronger candidate for a central platform than for a per-store weekly job. LightGBM's 12-second fit means the pharmacy team can refit every store overnight on a modest server.

Deployment decision. We ship LightGBM with quantile regression, and revisit the choice in three months. If the fleet-wide Transformer stabilizes and the platform team is willing to own the training infrastructure, we reconsider.

Quantile regression for real intervals

Rather than relying on parametric intervals (SARIMA's normal approximation, Prophet's Monte Carlo), we produce three separate quantile models: $\tau = 0.1, \tau = 0.5, \tau = 0.9$.

import lightgbm as lgb
import pandas as pd

feat = pd.read_parquet("features.parquet")
cutoff = feat.index[-28]
train, test = feat[feat.index < cutoff], feat[feat.index >= cutoff]

feature_cols = [c for c in feat.columns if c != "units"]
predictions = {}

for q in [0.1, 0.5, 0.9]:
    m = lgb.LGBMRegressor(
        objective="quantile", alpha=q,
        n_estimators=800, learning_rate=0.03, num_leaves=31,
    )
    m.fit(train[feature_cols], train["units"])
    predictions[q] = m.predict(test[feature_cols])

low = predictions[0.1]
median = predictions[0.5]
high = predictions[0.9]

objective="quantile" with alpha=q fits the pinball loss (module 9) at quantile $q$. Three fits produce a 10th, 50th, and 90th percentile prediction. The band [low, high] is our 80 % interval.

Two important consequences.

No normality assumption. Any non-Gaussian shape in the residuals — a long right tail on promotional days, for example — is captured by the quantile regressor without any transformation of the data.

Non-crossing guarantee. The three quantiles are trained independently, and nothing prevents the model from returning low > high on rare examples. In practice we sort the three per row (np.sort) before using them, and we log the crossing rate as a diagnostic.

Communicating the interval

An interval is only useful if the reader knows what to do with it. We pair every forecast with three numbers per day: the median, the 10th percentile, the 90th percentile. That maps to three procurement decisions.

• Order for the median and accept some risk of stockout.
• Order for the 90th percentile to guarantee no stockout except on outliers.
• Order for the 10th percentile to minimize waste on perishable items.

A dashboard shows the three curves side by side and the historical coverage. In addition, we produce a shortfall probability for each day: the empirical probability, computed from the quantile spread, that the true demand exceeds a reference level (say, the current stock plus one delivery). Procurement can then act on a probability directly rather than a scalar estimate.

Handling holidays and promotions explicitly

The features from module 7 already include a is_holiday flag, days_to_holiday, and a promotion regressor whose future values are known from the marketing calendar. Two additions matter at the project level.

Regional holidays. France has national holidays but Alsace-Moselle and some overseas territories have extra local ones. If the store operates in those regions, the flag must be region-aware; otherwise the model over-predicts on days when the store is unexpectedly closed and under-predicts otherwise.

Zero-demand days. On days when the store is closed (Sundays for most stores, some public holidays), demand is exactly zero. A regression model will happily predict small negative or small positive numbers. The right handling is a two-stage model: a classifier decides "open or closed", and the regressor predicts demand only on open days. On our data, this cleans up MAE by 0.4 units and dramatically improves the interval near closures.

The retraining plan

A model that ships once and never refits is a model whose accuracy will decay silently. Our plan has three components.

Scheduled retraining. Weekly refit on the last 4 years of data, every Monday at 03:00, using the same hyperparameters. On 200 stores, the LightGBM refit fits in an hour on a laptop-class server. The Transformer refit, if adopted, is quarterly rather than weekly, because 20 minutes per fit compounds badly across 200 stores.

Drift-triggered retraining. We monitor the rolling MASE in production: if the last four weeks' MASE exceeds 1.0 (worse than seasonal-naïve), an alert fires and the team investigates. That is a much stronger signal than the raw MAE, which fluctuates with seasonality.

Feature-drift monitoring. For each numerical feature, we compute a Population Stability Index between the training distribution and the last week's live distribution. PSI above 0.25 on any feature is investigated: it usually means the world moved (a competitor opened next door, a promotion changed the customer mix) and the model may need architectural changes, not just retraining.

The MLOps course (course 20) laid out the general machinery; the specificity here is that our drift metric is temporal, not just distributional.

Communicating uncertainty

The last, hardest, and most-often-skipped step of a forecasting project is telling the business what "confidence" means.

Three lines are enough for procurement.

• "The median is what we expect."
• "80 % of the time, the true demand lands between the 10th and 90th percentiles."
• "20 % of the time, it does not. Half of that, it exceeds the 90th; half, it falls below the 10th."

Every stakeholder we have talked to reacts to those three sentences by asking the same follow-up: "how often has the model been right in the past?". That is exactly what the rolling coverage number in the results table answers. Publish the coverage next to the intervals; never publish an interval without its historical coverage.

The final rule: on the day the model fails badly — a delivery truck strike, a heat wave, an unforeseen promotion — the team must be able to say why. The retraining plan does not save you from unmodeled events; a documented audit log of the top features per day does. When something breaks, the log tells the team which feature moved and by how much, and the conversation with the business moves from "the model failed" to "the world moved outside what the model has seen".

Ship the interval, not just the point

A single-number forecast forces the business to guess its own uncertainty. The most valuable deliverable of this project is not the 6.4 MAE — it is the 80 % interval whose coverage the team can quote from memory. That is what turns a forecast from an estimate into a decision-support tool.

What we learned across the course

• Baselines are non-negotiable. Seasonal-naïve gave us 9.8 units of MAE, and every metric above is honest only because the baseline number sits next to it.
• Complex models are worth their cost when the shape of the problem fits them. Gradient boosting was the right pick here; deep learning would have been the pick with a much larger fleet.
• Uncertainty is the deliverable. A point forecast without an interval is a half-answered question. Quantile regression is the simplest tool to produce honest intervals.
• Evaluate rolling, not once. A single-window MAE lies half the time; rolling backtests turn a single number into a distribution the business can act on.

Summary

• The final table names LightGBM with quantile regression as our deployed model on a per-store view, with the global Transformer as the platform-level contender we revisit in three months.
• Quantile regression at $\tau = 0.1, 0.5, 0.9$ replaces parametric intervals with data-driven ones; three separate fits, one pinball loss each.
• The retraining plan is weekly-scheduled plus drift-triggered, monitored by rolling MASE and per-feature PSI.
• Communicating uncertainty is the deliverable: three sentences, one coverage number, and an audit log for the day the world moves outside the training distribution.

Next module: the recap of the course and the 40-question exam that decides whether the certificate is issued.