Skip to main content

Module 9 — Pipelines and scheduling

Modules 5 to 8 produced a training job, an AutoML job, a registration step, and two endpoint deployments — each launched by hand. In production the same sequence must run every Sunday night, unattended, with the winning model automatically promoted to next week's forecast. That is what an Azure ML pipeline is for.

Components: the reusable step​

A pipeline is a graph of components. A component is a self-contained step with a stable interface — declared inputs, declared outputs, an environment, a command — that can be reused across pipelines. Turning Module 5's training job into a component is one YAML file:

# component-train.yml
$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
name: train_forecast_lgbm
version: 3
type: command
display_name: Train LightGBM forecast
inputs:
  sales:
    type: uri_folder
  learning_rate:
    type: number
    default: 0.05
  num_leaves:
    type: integer
    default: 63
outputs:
  model:
    type: mlflow_model
  metrics:
    type: uri_file
code: ./src
environment: azureml:forecast-env:3
command: >-
  python train.py
  --sales ${{inputs.sales}}
  --learning-rate ${{inputs.learning_rate}}
  --num-leaves ${{inputs.num_leaves}}
  --model-out ${{outputs.model}}
  --metrics-out ${{outputs.metrics}}
az ml component create --file component-train.yml

The component is now in the workspace registry, versioned, discoverable, and reusable. The _prep, _evaluate, _register and _batch_score components follow the same template.

The weekly forecast pipeline​

The pipeline wires those components in order and passes outputs of one as inputs of the next:

# pipeline-weekly.yml
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
experiment_name: demand-forecast
display_name: weekly-forecast-pipeline
compute: azureml:cpu-cluster-forecast

inputs:
  raw_sales:
    type: uri_folder
    path: azureml:sales-2y@latest

jobs:
  prep:
    type: command
    component: azureml:prep_sales:2
    inputs:
      raw: ${{parent.inputs.raw_sales}}

  train:
    type: command
    component: azureml:train_forecast_lgbm:3
    inputs:
      sales: ${{parent.jobs.prep.outputs.clean_sales}}

  evaluate:
    type: command
    component: azureml:evaluate_forecast:2
    inputs:
      model: ${{parent.jobs.train.outputs.model}}
      sales: ${{parent.jobs.prep.outputs.clean_sales}}

  register:
    type: command
    component: azureml:register_if_better:1
    inputs:
      model: ${{parent.jobs.train.outputs.model}}
      metrics: ${{parent.jobs.evaluate.outputs.metrics}}
      champion_name: forecast-lgbm

Two things are worth noticing.

The ${{parent.jobs.prep.outputs.clean_sales}} syntax passes an artifact directly between steps — no manual serialisation, no shared blob path. Azure ML mounts the previous step's output as the next step's input, and the graph in Studio draws the arrows.

The register_if_better component is a conditional gate: it registers a new version of forecast-lgbm only if this week's validation MAPE beats the current champion by more than a configured threshold. That single component turns the pipeline from "always train" into "train and promote only when it helps", which is what MLOps really means in practice.

Submit the pipeline like any other job:

az ml job create --file pipeline-weekly.yml

Studio shows the pipeline as a DAG, each node clickable to its own logs, metrics and artifacts.

Scheduling: cron and recurrence​

Running the pipeline once is trivial. Running it every Sunday at 02:00 needs a schedule:

# schedule-weekly.yml
$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: weekly-forecast-schedule
display_name: Sunday 02:00 UTC weekly forecast
trigger:
  type: cron
  expression: "0 2 * * 0"
  time_zone: UTC
  start_time: "2026-09-06T00:00:00"
create_job: pipeline-weekly.yml
az ml schedule create --file schedule-weekly.yml

The type: recurrence variant is friendlier for simple cases (frequency: week, interval: 1, week_days: [Sunday], hours: [2]). Both produce a schedule visible in Studio's Schedules tab, where you can disable, edit, or delete it.

Two operational habits pay off. Store the pipeline YAML in git, then re-create the schedule from CI. A schedule created interactively in Studio is invisible outside it. Use UTC in the cron expression and let the display layer localise — daylight saving time is a nightmare in cron and it will eventually skip or duplicate a Sunday.

Event-based triggers: when new data lands​

Sometimes the trigger is not calendar time but the arrival of new data. When the point-of-sale team drops the week's Parquet into sales/history/2026-09-XX/, we want the pipeline to launch within minutes, not to wait for the next Sunday.

Azure ML integrates with Azure Event Grid: an event on the storage account (Microsoft.Storage.BlobCreated filtered on a prefix) triggers a Logic App, which triggers the pipeline. The end-to-end setup is more moving parts than a cron schedule and it is worth it only when the arrival timing is genuinely unpredictable. For the retail forecast, whose data lands like clockwork Saturday at midnight, a Sunday 02:00 cron is simpler and equally effective.

Retries, resumes and reruns​

A pipeline that fails at step 4 of 5 can be cloned from Studio: a new pipeline run inherits the same components and inputs, and you can toggle "reuse previous run" per step. Steps whose inputs did not change are skipped and their previous outputs reused — often turning a hour-long pipeline into a few-minute rerun.

Automatic retries live on each component with retry_settings in the job spec: max: 3, timeout: 60. Use them on flaky steps (a transient network glitch on data upload), not on steps whose failure needs investigation (a training run that diverged).

Compose small components, not one big one

The temptation is to put prep, train, evaluate and register in one Python script and be done. Then every rerun re-trains, every debug means reading a 300-line log, and no step is reusable. Small components are more YAML upfront and less pain forever.

Summary​

  • Components are reusable, versioned steps; a pipeline wires them by piping outputs into inputs.
  • A conditional registration component turns "train weekly" into "train weekly and promote if better", which is the actual production requirement.
  • Cron schedules in UTC, stored in git, are the standard trigger; Event Grid covers the rare cases where data arrival timing is unpredictable.
  • Step reuse on rerun turns most debug loops into minutes, not hours.

Next module: closing the loop — monitoring the deployed model for drift, watching cores quotas, and putting a cost tag on every resource.