Module 9 — Rolling validation and suitable metrics

Every MAE we wrote in modules 4 to 8 came from a single test window at the end of the series. That is a starting point, not a conclusion. Two questions have been unanswered so far. First, would the ranking hold if we had cut the series a different day? Second, is MAE the right metric for a business that cares about stockouts more than about small daily errors? This module answers both.

Rolling-origin evaluation

The proper backtest for a forecasting model is a rolling origin. We pick an origin time $T_1$, produce a horizon-$h$ forecast, measure the error, move the origin forward by one step (or by a batch), refit, and repeat. Each origin yields one forecast; averaging across origins gives us a metric that resembles what the model would produce in production over months.

import numpy as np, pandas as pd
from sklearn.metrics import mean_absolute_error

def rolling_backtest(series, model_factory, initial_train, horizon, step=7):
    errors = []
    origin = initial_train
    while origin + horizon <= len(series):
        train = series[:origin]
        test = series[origin : origin + horizon]
        model = model_factory().fit(train)
        pred = model.predict(horizon)
        errors.append(mean_absolute_error(test, pred))
        origin += step
    return np.array(errors)

We move the origin by 7 days rather than by 1: for a 28-day horizon, a step of 1 makes 27 out of 28 predictions per origin overlap with the previous origin's, which is fine statistically but wastes computation. A step of 7 gives us weekly-refreshed forecasts and matches the operational cadence of a pharmacy team.

The initial training window must be long enough for the model to see its seasonalities: at least two full periods of the longest cycle. For our yearly seasonality, that means two years — a comfortable minimum before we start rolling.

Multiple horizons in one report

We rarely care equally about "day 1" and "day 28". A pharmacy team plans differently: order tomorrow's items in the afternoon, order next Monday's stock from a wholesaler over the weekend, plan a promotion for day 28. The evaluation should mirror that.

We report MAE per horizon step, not just a single scalar.

def per_horizon_error(series, model_factory, initial_train, horizon, step=7):
    errors = np.zeros(horizon)
    n = 0
    origin = initial_train
    while origin + horizon <= len(series):
        train = series[:origin]
        test = series[origin : origin + horizon]
        model = model_factory().fit(train)
        pred = model.predict(horizon)
        errors += np.abs(test - pred)
        n += 1
        origin += step
    return errors / n

On our pharmacy series, the LightGBM MAE grows from 4.1 at horizon 1 to 8.7 at horizon 28. That gradient shape is honest: the further out you predict, the less signal the past carries. A model whose MAE is flat across horizons is either miraculous or leaking.

MAE and RMSE

The two point-forecast metrics you already know.

MAE — Mean Absolute Error. Reports errors in the units of the target. Insensitive to outliers.

RMSE — Root Mean Squared Error. Penalizes large errors more (quadratically). In the same units.

For a warehouse, MAE is often the right business metric: an average of 6.9 units per day off is easier to reason about than a "root mean square". RMSE matters when large errors have a disproportionate business impact — say, a stockout costs 100 times as much as an overstock — and pushes the model to be more conservative on peaks.

MAPE and its zero-value trap

MAPE — Mean Absolute Percentage Error — is the most requested metric by non-technical stakeholders because "off by 5 %" is intuitive.

$$\text{MAPE} = \frac{100}{n} \sum_{t=1}^{n} \left| \frac{y_t - \hat{y}_t}{y_t} \right|$$

It has two well-known problems.

Division by zero. If $y_t = 0$ (a closed store on Sundays, an idle day), MAPE is undefined. Pandas silently returns inf; a naive average returns nan.

Asymmetry. For a true value of 10 and a prediction of 20, MAPE returns 100 %. For a true value of 20 and a prediction of 10, MAPE returns 50 %. The same absolute error is penalized twice as heavily when the true value is small — which biases model selection toward over-predicting on low-volume days.

We report MAPE only after filtering out zeros, and we always report MASE alongside it.

MASE: scaled to a baseline

MASE — Mean Absolute Scaled Error — divides the model's MAE by the MAE of a naïve baseline on the training set:

$$\text{MASE} = \frac{\text{MAE}(\hat{y}, y)}{\text{MAE}_{\text{baseline}}(y)}$$

For a series with seasonality, the baseline in the denominator is the seasonal-naïve on the training window; for a non-seasonal series it is the plain naïve. A MASE below 1 means "better than baseline"; a MASE close to 1 means "the model is equivalent to copying the appropriate past".

MASE is the metric to use when we want to compare across series (different scales) or across models. It also handles zeros without complaint: they enter the numerator normally.

def mase(y_true, y_pred, y_train, m=7):
    naive = np.abs(np.diff(y_train, m)).mean()
    return np.abs(y_true - y_pred).mean() / naive

On the pharmacy data:

Model	MASE
seasonal_naive_7	1.00
SARIMA	0.86
Prophet	0.78
LightGBM	0.70
Transformer (global)	0.66

Every model does beat the baseline; the gaps we saw in MAE are preserved. MASE is our default headline metric from module 10 onwards.

Interval evaluation

A forecast is more than a point. Modules 5, 6 and 10 all produce intervals, and evaluating them is not the same as evaluating the point.

Coverage. For an 80 % prediction interval, roughly 80 % of the true values should fall inside. Backtesting produces an empirical coverage: if we get 68 %, the interval is over-confident; if we get 92 %, it is over-wide.

Interval score. A proper score for a $(1 - \alpha)$ interval $[l, u]$:

$$S = (u - l) + \frac{2}{\alpha}(l - y) \mathbb{1}[y < l] + \frac{2}{\alpha}(y - u) \mathbb{1}[y > u]$$

The score penalizes width and missed observations. Lower is better. Two forecasts with the same coverage can differ dramatically in interval score if one is narrow-with-a-few-big-misses and the other wide-with-no-misses.

Pinball loss. For a single quantile $\tau$, the pinball loss

$$L_\tau(y, \hat{q}) = \max\left(\tau (y - \hat{q}),\ (\tau - 1)(y - \hat{q})\right)$$

is what quantile regression minimizes. Reporting pinball at $\tau = 0.1$ and $\tau = 0.9$ gives a compact interval quality summary.

Putting it all together

The evaluation dashboard for a serious forecasting project shows, for each candidate model:

• MAE and MASE, per horizon step (1, 7, 14, 28)
• 80 % interval coverage and average width
• Interval score
• Runtime per refit

A model that improves MAE by 3 % but takes ten times longer to fit is a different trade than one that also improves interval score. Presenting all four columns is what lets a stakeholder make an operational decision.

The reversal we sometimes see

On rolling backtests, a model whose single-window MAE was strongest can turn out to be inconsistent — it is the best in some months and mediocre in others. LightGBM often produces a very stable MAE across origins; deep models are more variable, with lower means and higher variance.

For a business that cares about consistency (procurement wants predictable errors), the stable model is often preferred at equal average performance. That is not a matter of mathematics but of risk preference, and the rolling backtest is what surfaces it.

A single-window metric hides the variance

"Our LSTM beat the LightGBM by 5 %" — on which origin? If it was the last 28 days of the series, that measures one draw of a random variable. Rolling backtests measure the distribution. The choice between two models on the mean of their MAEs may reverse when you look at the 90th percentile.

Summary

• Rolling-origin backtests replace the single-window test with dozens of origins, giving us a distribution of errors rather than a point; step size matches operational cadence.
• Report per-horizon MAE, not one scalar. A flat MAE across 1 to 28 days is a red flag.
• MAPE is treacherous with zeros and asymmetric with small values; MASE is the right scale-free default.
• Evaluate intervals with coverage, interval score, and pinball loss — a good point forecast with a bad interval is not deployable in a procurement setting.

Next module: the project. We consolidate the results table, pick a deployable model, produce a forecast with quantile intervals, and plan the retraining schedule.