Skip to main content

Module 4 — Environments and container images

The compute and the data are in place. What remains is telling the cluster which Python, which libraries, which system packages to run under. Azure ML calls that an environment, and every training job, batch scoring job, and online deployment binds to exactly one, by name and version.

What an environment actually is​

Under the hood, an environment is a Docker image plus optional metadata. When a job starts, Azure ML pulls that image onto the compute node, mounts your code and data into it, and runs your entry point inside. When you register an environment, three things can happen:

  • Curated environment: you reference an image Microsoft prebuilt and maintains. No build.
  • Custom, conda-based: you provide a base image plus a conda.yml. Azure ML builds a new image on top by running conda env create, pushes it to your workspace's ACR, and tags it with the environment version.
  • Custom, Dockerfile-based: you provide a full Dockerfile. Azure ML runs docker build and pushes.

The first is instant. The second takes 3 – 10 minutes on first build, seconds thereafter. The third is the escape hatch when the second is not enough.

Curated environments: use them when you can​

Microsoft publishes a large family of curated images under names like AzureML-sklearn-1.5:{version}, AzureML-lightgbm-4.5:{version}, AzureML-tensorflow-2.16:{version}. They are already patched, already cached on compute nodes, and their versions match tested combinations of CUDA, cuDNN and Python.

For the forecast project's baseline training with LightGBM, the curated environment is enough:

# job.yml
command: python train.py --data ${{inputs.sales}}
environment: azureml://registries/azureml/environments/AzureML-lightgbm-4.5/versions/8
compute: azureml:cpu-cluster-forecast

No conda.yml, no build, no ACR push. When Microsoft releases a security update, you bump the version tag and get a fresh image, in one line.

Custom conda: the mainstream choice​

The moment your code depends on a library outside the curated set — say, mlforecast for lag features and holidays for calendar variables — a custom environment is the right answer. The recipe is a small conda file layered on a curated base:

# env-forecast.yml
$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json
name: forecast-env
version: 3
description: LightGBM plus mlforecast for weekly demand forecasting
image: mcr.microsoft.com/azureml/curated/lightgbm-4.5:latest
conda_file: conda.yml
# conda.yml
name: forecast
channels: [conda-forge]
dependencies:
  - python=3.11
  - pip=24.2
  - pip:
    - lightgbm==4.5.0
    - mlforecast==0.15.1
    - holidays==0.56
    - mlflow==2.17.0
    - pandas==2.2.3
    - pyarrow==17.0.0

Pin every version. lightgbm>=4 looks harmless until 4.6 changes a default parameter and last month's model no longer reproduces. Reproducibility, not novelty, is what the environment buys you.

Register it with:

az ml environment create --file env-forecast.yml

The first job that references azureml:forecast-env:3 triggers the image build; every subsequent job pulls the cached image in seconds.

Dockerfile: when conda is not enough​

Some things are outside conda's reach: a compiled tool, a system library at a specific version, an OS package. In that case, provide a full Dockerfile:

# Dockerfile
FROM mcr.microsoft.com/azureml/curated/lightgbm-4.5:latest

# system: needed by a legacy geospatial dependency
RUN apt-get update && apt-get install -y --no-install-recommends \
    libgdal-dev=3.6.4+dfsg-1~jammy0 \
 && rm -rf /var/lib/apt/lists/*

COPY requirements.txt /tmp/
RUN pip install --no-cache-dir -r /tmp/requirements.txt
# env-forecast-geo.yml
name: forecast-env-geo
version: 1
build:
  path: .
  dockerfile_path: Dockerfile

The trade-off is honest: full control on the image, longer builds, and you own the security patching that curated environments hand you for free. Use it only when you must.

The build cache and why it matters​

Azure ML caches image layers on each compute node. A cluster that ran a job with forecast-env:3 yesterday will start today's job in seconds, not minutes. But the cache is per node: the first time a fresh node pulls the image, the pull cost is real (a few minutes for a 4 GB image). Two consequences follow.

First, avoid rebuilding the environment for every trivial change to train.py. Code goes in the job's code: folder, uploaded per job; the environment stays stable across many jobs.

Second, bumping the environment version invalidates the cache — deliberately. A hotfix to a pinned package should be a new version (4), not a silent overwrite of 3. Overwriting is not even possible: environment versions are immutable once registered.

Versioning discipline for the forecast project​

The habit that scales: one environment version per meaningful change. Version 1 was the initial LightGBM setup. Version 2 added holidays. Version 3 pinned pandas to 2.2.3 after a merge bug in 2.1. Each version is dated in its description, tagged with the run it was first used in, and never modified.

The job spec always references an explicit version, never latest:

environment: azureml:forecast-env:3

latest is convenient in a demo and ruinous in production: a job that trained a model in June against latest cannot be re-run in September because latest now points at version 5, with three package upgrades in between.

The most common failure: unversioned environments

The exam bank will hit this pattern. A team that references environments by name only — no version, no build hash — will one day find that a job that used to converge no longer does, and they will not be able to say why. Pin the version at the call site, always.

Summary​

  • Environments are Docker images plus metadata, pinned to a name and version, referenced by every job and endpoint.
  • Curated environments require no build and get security updates from Microsoft; use them when your dependencies fit.
  • Custom conda is the mainstream choice; pin every package version and layer on a curated base image.
  • Dockerfiles are the escape hatch for system-level dependencies, at the cost of owning security patches.
  • Never reference an environment by latest in a job spec; a version tag is a reproducibility contract.

Next module: submitting training jobs — command jobs, MLflow tracking, and hyperparameter sweeps for the forecast baseline.