Skip to main content

Module 7 — Continuous integration and delivery pipelines

Everything so far — reproducibility, tracking, versioning, registry, container — is only worth its trouble if a single event (a merged pull request, a fresh dataset, a nightly cron) can retrain, validate, promote, and deploy without a human. That is the job of a CI/CD pipeline for ML. The differences with regular CI/CD are entirely on the testing side.

Four kinds of tests, in order​

A serious ML pipeline runs, in this order:

  1. Code tests: unit tests on feature transformations, on the API, on the loading code. Same as any Python project.
  2. Data tests: does the incoming dataset satisfy the assumptions the training code makes?
  3. Model tests: does the newly trained model meet the quality thresholds?
  4. Container and integration tests: does the built image start, respond correctly, and stay within the latency budget?

Failing early is the point. A code test that fails in five seconds is cheaper than a model test that fails after two hours of training.

Data tests​

The pipeline never trains on data it has not validated. Great Expectations, Pandera or a hand-written test module all work; the checks are the same:

# tests/test_data.py
import pandas as pd

def test_schema(df: pd.DataFrame):
    expected = {"subscriber_id", "tenure_months", "monthly_charges",
                "contract_type", "payment_method", "churned"}
    assert set(df.columns) == expected, f"Schema drift: {set(df.columns) ^ expected}"

def test_types(df):
    assert df["tenure_months"].dtype.kind == "i"
    assert df["monthly_charges"].dtype.kind == "f"

def test_no_leakage(df):
    # 'churned' is the target: it must never appear in the feature set at inference
    assert "churned" not in feature_columns()

def test_target_balance(df):
    rate = df["churned"].mean()
    assert 0.10 < rate < 0.40, f"Churn rate {rate:.2%} outside expected range"

The test_target_balance check catches a data pipeline bug that would otherwise look like a model regression: a filter accidentally kept only churners, the target rate jumped to 80 %, the model trains but predicts nonsense. That is a data problem masquerading as a model problem, and only a data test finds it before training burns two GPU-hours.

Model tests with thresholds​

Model tests answer "is the freshly trained model good enough to be promoted?". They compare against two references: an absolute floor (business SLA) and the current champion (the model that is live). Both must be beaten.

# tests/test_model.py
CHAMPION = "models:/churn-classifier@production"
CANDIDATE = "runs:/{RUN_ID}/model"

def test_absolute_floor():
    auc = evaluate(CANDIDATE, holdout).roc_auc
    assert auc >= 0.82, f"Below absolute floor: {auc:.3f}"

def test_beats_champion_on_recent_data():
    recent = load_recent_slice(days=14)
    champ = evaluate(CHAMPION, recent).roc_auc
    cand = evaluate(CANDIDATE, recent).roc_auc
    assert cand >= champ - 0.005, (
        f"Regression on recent data: candidate {cand:.3f} vs champion {champ:.3f}"
    )

def test_no_drift_on_predictions():
    # Sanity: candidate should not deviate wildly from champion on a fixed sample
    p_c = predict(CHAMPION, canary_sample)
    p_n = predict(CANDIDATE, canary_sample)
    assert abs(p_c.mean() - p_n.mean()) < 0.05

Two lessons compressed here. The absolute floor protects against a champion that has silently degraded — comparing only to a bad champion would ratify decline. The recent data slice protects against the module 5 pitfall: training-time metrics are not production metrics. The tolerance of 0.005 is a hedge against noise, not against real regressions — set it from a bootstrap on multiple seeds.

The pipeline in GitHub Actions​

The full workflow is longer than a screen, but the essential stages are readable:

name: churn-mlops
on:
  push:
    branches: [main]
  schedule:
    - cron: "0 3 * * *"  # nightly retraining

jobs:
  code:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install --require-hashes -r requirements-dev.txt
      - run: ruff check . && mypy src/ && pytest tests/unit -q

  data-and-train:
    needs: code
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: dvc pull
      - run: pytest tests/data -q
      - run: python -m src.train  # logs to MLflow, prints RUN_ID
        env:
          MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_URL }}
      - run: pytest tests/model -q

  build-and-scan:
    needs: data-and-train
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t churn:${{ github.sha }} .
      - uses: aquasecurity/trivy-action@master
        with: { image-ref: churn:${{ github.sha }}, severity: HIGH,CRITICAL, exit-code: 1 }

  promote:
    needs: build-and-scan
    runs-on: ubuntu-latest
    environment: production   # requires manual approval on GitHub
    steps:
      - run: python -m src.promote --alias production --run-id $RUN_ID
      - run: kubectl rollout restart deploy/churn-serving

The environment: production clause is the module 5 promotion gate made concrete. A protected environment in GitHub requires one or two named reviewers to click "Approve deployment" before promote runs. That is the human on the critical path, deliberately placed at the last step, where their judgment matters — everything before is automated.

The manual approval, and when to remove it​

Early on, the reviewer catches things the tests forgot: a spike in false positives on a specific segment, a comment on the pull request explaining "we're expecting seasonal churn to look like drift for a week". Every time the reviewer clicks approve without hesitation, the automation is one step from being able to promote itself. When six months pass without a reviewer catching anything the tests missed, the gate can be moved from the alias switch to a canary deployment (module 10) — reviewers still catch problems, but from real traffic, not from a green checkmark.

A green pipeline is not a valid model

Tests pass by defaulting to the previous behavior when data is missing. A pytest tests/model that silently skips because the test data was not downloaded is a green pipeline promoting an unvalidated model. Every test file should end with assert counts or --strict-markers; a zero-test run is a pipeline failure.

Summary​

  • Four test stages, in order: code, data, model, container — each fails cheaper than the next.
  • Model tests compare against an absolute floor and the current champion on recent data; the recent slice defeats the "training-time metric" trap.
  • The pipeline drives the whole lifecycle: pull data with DVC, log the run to MLflow, build a scanned image, promote through a protected environment.
  • Keep a manual approval at promotion until the pipeline has demonstrated, for months, that it never needs one — then move it to a canary deployment.

Next module: serving — the choice between batch, online and streaming, and why "just build an API" is not always the right answer.