Module 6 — AutoML: relevant uses and limits

AutoML is a submitted job like any other: same workspace, same compute, same data assets, same MLflow logging. It just replaces "your training script" with "a search over dozens of algorithms and preprocessing pipelines". On the demand-forecast problem, that search is worth running — and then honestly compared against the LightGBM baseline from Module 5.

What AutoML actually does

An AutoML forecasting job explores three things in parallel: feature engineering (lags, rolling means, calendar decompositions), models (Prophet, Auto-ARIMA, ExponentialSmoothing, ForecastTCN, LightGBM, XGBoost), and hyperparameters for each. It ranks candidates by a metric you pick, on a time-series cross-validation that respects the temporal order of the data.

The output is the same shape as Module 5's output: an MLflow model in the run's artifacts, ready to be registered and deployed. What differs is that dozens of runs happened under the hood, each visible in Studio.

The forecast job spec

For the grocery demand problem, on the same sales-2y data asset, converted to an MLTable (Module 3):

# automl-forecast.yml
$schema: https://azuremlschemas.azureedge.net/latest/autoMLJob.schema.json
type: automl
task: forecasting
display_name: automl-forecast-8w
experiment_name: demand-forecast
compute: azureml:cpu-cluster-forecast

training_data:
  type: mltable
  path: azureml:sales-mltable:1
target_column_name: units

forecasting:
  time_column_name: week_start
  time_series_id_column_names: [store_id, sku_id]
  forecast_horizon: 8
  frequency: W
  country_or_region_for_holidays: FR

primary_metric: normalized_root_mean_squared_error
n_cross_validations: auto

limits:
  timeout_minutes: 240
  max_trials: 40
  max_concurrent_trials: 4
  enable_early_termination: true

Two settings deserve their own paragraph.

time_series_id_column_names: [store_id, sku_id] tells AutoML there is not one series but tens of thousands — one per store per SKU. Getting this wrong is the top source of nonsense results: with a single series assumed, AutoML aggregates across the whole chain and returns forecasts that are useless per-store.

forecast_horizon: 8, frequency: W matches the business question exactly. AutoML's cross-validation slices the history into rolling windows of the same horizon, so the metric it reports is the metric you will see in production.

When AutoML wins

Three conditions make AutoML clearly worth its cost.

A new problem with no strong prior. The team has never forecast demand before, and the shape of a good model is not obvious. AutoML's search covers algorithm families you might not have considered and returns a solid starting point in a few hours — often within 10 % of an expert's manual baseline.

Many small models to train. A per-store model, a per-region model, a per-category model. Hand-tuning each is unrealistic; AutoML's search per group is cheaper than a data scientist's time.

A regulatory need for an audit trail on the search space. AutoML's Studio view lists every model tried, with its metrics and preprocessing. Reproducing that record with hand-run experiments takes discipline that most teams do not sustain.

When AutoML loses

Three conditions where the hand-built baseline is the right choice.

You already have a strong prior. If the forecast team knows that LightGBM on lag features beats every neural approach on this data, AutoML burns 4 hours of compute to confirm what a two-line command job proves in 15 minutes. Module 5's baseline achieved a validation MAPE of 12 % with mlforecast in this project; AutoML's best on the same data settled at 11.5 %, i.e. a 0.5 point gain for eight times the compute.

Latency budget in production matters. Some AutoML winners are stacked ensembles of five or six models. They score slowly. Module 8's online endpoint has a 200 ms budget per call; a stack that scores in 400 ms is unusable even if it wins on MAPE by a hair.

Features encode domain knowledge AutoML cannot infer. A promotion calendar, a weather forecast, a store-remodel flag — you know they matter and how to encode them. AutoML's own feature engineering does not.

Reading an AutoML run

In Studio, an AutoML job exposes the child runs view: one row per algorithm-plus-preprocessing combination it tried, sorted by primary metric. Two panes matter.

The Explanations tab gives per-feature importance for the winner. On the forecast, it typically shows the top lag features (lag-1, lag-52), day-of-week and holiday flags dominating; that alone tells you whether the model is behaving sensibly.

The Best model tab exports the pipeline as an MLflow model, ready for Module 7's registry. Register the AutoML winner and the manual baseline side by side, then let Module 8 traffic-split between them to compare in production.

The cost, honestly

An AutoML job with max_trials: 40 and a timeout_minutes: 240 on four concurrent Standard_DS3_v2 nodes bills roughly 16 CPU-hours. At around €0.20 per CPU-hour, that is €3 per run — cheap once, expensive daily. Cap max_trials and timeout_minutes deliberately: the search almost always converges long before either limit, and the early termination flag helps.

The classic AutoML mis-use to avoid

Running AutoML on a problem where a domain expert has already spent a week engineering features, hoping AutoML will "find something better", almost never pays off. AutoML's search space is broad but shallow on custom features. It shines on problems where nobody has looked closely yet, not on ones where somebody has looked very closely.

Summary

• AutoML runs as a submitted job and produces the same MLflow-model output as a hand-written run.
• The two critical settings for time series are time_series_id_column_names and forecast_horizon + frequency; getting either wrong invalidates the whole search.
• AutoML wins on new problems, many small models, and audit trails; the hand-built baseline wins on strong priors, tight latency budgets, and domain-specific features.
• Cap max_trials and timeout_minutes; measure the cost against the metric gain before adopting the AutoML winner.

Next module: registering the winner — from AutoML or from the baseline — as a versioned model with full lineage back to its data and its training job.