Module 10 — Automated retraining and rollback

Monitoring surfaces drift. This module closes the loop: what to do about it, without a human in the middle. Automated retraining sounds like an aspirational feature; it is really a discipline of triggers, validation gates, and one-command rollback. Every mature ML system converges to some version of what follows.

Triggers: when to retrain

Retraining on a schedule is not enough, and retraining on every drift is too much. Three triggers, layered:

1. Cadence: nightly or weekly, regardless of any signal. This catches slow drift that never breaches a threshold and keeps the pipeline warm — a pipeline that has not run in two months will fail when you need it.
2. Drift-triggered: when module 9's alerts fire on features the model cares about, plus a proxy signal on predictions or residuals. This catches the sharp shifts a cadence would miss for days.
3. Volume-triggered: after N new labeled examples arrive. Useful for young models where every week of new data materially improves quality.

For the churn project, the concrete setup is nightly cadence, with drift-triggered runs allowed to preempt the queue. Trigger 3 is unnecessary given the churn dataset's size.

The retraining pipeline is the CI/CD pipeline

There is not a separate "retraining system" alongside CI/CD. The pipeline of module 7 already trains, tests, builds, and promotes; retraining is that same pipeline invoked by schedule or by a webhook from the monitoring stack. The workflow reuses the same tests, the same thresholds, the same protected environment.

The one addition specific to retraining: the champion's metrics are recomputed on the same recent slice used to test the candidate, right before the comparison. Otherwise you compare a fresh candidate to a stale champion metric captured at its promotion time, and the champion always looks worse than it actually is.

Validation before promotion: what module 7 becomes

The gate is tightened when running under automation. The manual approval is replaced by a canary deployment: the candidate is promoted to @candidate, and a small fraction of live traffic (1 %, 5 %, then 25 %) is routed to it. Its metrics are watched for 30 minutes to an hour before it is allowed to become @production.

# promote_canary.py
client = mlflow.MlflowClient()
client.set_registered_model_alias("churn-classifier", "candidate", version=NEW_VERSION)
# ... traffic router sends 5 % to @candidate ...
# ... wait, measure, then either ...
if canary_ok():
    client.set_registered_model_alias("churn-classifier", "production", version=NEW_VERSION)
else:
    # keep production as-is; open ticket with the diff
    ...

Canary metrics that matter: predicted-probability distribution against @production (should be close), realized errors when labels arrive fast, latency and error rate of the service, and — if applicable — a business KPI that the model influences (retention offer take-up, false-positive complaints).

Rollback in one command

The full promise of the registry becomes concrete here. Rollback is one command:

mlflow models set-alias churn-classifier production 7  # was 8

Or, when the model is baked into the image (module 6): kubectl set image deploy/churn-serving churn=churn:v7 && kubectl rollout undo deploy/churn-serving. Both take under a minute.

Two conditions make rollback fast in practice, and their absence makes it slow:

• The previous version's artifacts are intact. MLflow's Archived versions are still fetchable. Never garbage-collect the previous champion; the cost of keeping it is negligible next to the cost of not having it during an outage.
• The serving layer resolves the alias at runtime, not at build time. If the container caches the alias-to-version resolution at startup, rollback requires a redeploy — which extends recovery from seconds to minutes. Pull the resolution behind a short TTL (30–60 s) or expose an operator endpoint that forces a reload.

Progressive deployment strategies

Beyond canary, two patterns are common. Blue/green keeps two full production environments; a switch flips traffic wholesale after the new environment passes its checks. It is expensive (twice the capacity for the transition) but rollback is instantaneous.

Shadow traffic sends the same live requests to both @production and @shadow, using the production's answer while comparing. It catches problems that would not show under 1 % of traffic and adds no user-visible risk. The compute cost is real (the shadow does full inference on every request); shadow is used for the tricky promotions, not for every one.

Cost and governance

Automated retraining has a bill. A nightly training that costs $50 in compute is $18 000 a year. That is fine when the model brings meaningful value; it is theatre when the drift-corrected model recovers 0.001 in ROC AUC. Two disciplines keep costs in line:

• Trigger retraining only when the trailing signal is real, not on every cadence tick. Skip nightly runs when neither drift nor volume has moved.
• Log the per-run cost alongside metrics; a retraining that improved ROC AUC by 0.0002 for $50 is not a win.

Governance concerns are equally practical. In regulated industries, an autonomous promotion needs an audit trail: who (which service account) promoted what (which version) with what evidence (the test results linked to the run). The trail exists implicitly in MLflow and GitHub; the discipline is to make it queryable — a single dashboard listing every promotion in the last quarter, with links to the run, the diff, and the canary metrics.

Automation does not remove the on-call

An autopromoted model that regresses at 2 a.m. still needs someone to notice and to trigger the rollback. Automated retraining shifts the human's role from "gatekeeper of every change" to "responder to the ones that go wrong" — a smaller job, not a job that no longer exists.

Summary

• Retrain on cadence + drift + volume, layered; a schedule alone misses sharp shifts, a drift-only trigger lets pipelines rot.
• The retraining pipeline is the CI/CD pipeline; the only addition is canary deployment replacing the manual approval.
• Rollback is one command if the previous version's artifacts are kept and the alias is resolved at runtime.
• Watch cost and audit trail: skip retrainings that would move nothing, and keep every promotion queryable.

Next: the recap and the 40-question exam.