Module 2 — Reproducibility: seeds, environments, version pinning

Reproducibility is not a philosophical goal. It is the concrete capability to rerun a training six months from now and land on the same model, byte for byte, so that a regression report or a compliance audit can point at one model rather than a cloud of near-identical variants. This module pins every source of variability that would otherwise let a rerun drift.

What is not reproducible in the churn notebook

Rerun churn.ipynb twice in the same afternoon and the ROC AUC will move by two or three thousandths. Rerun it tomorrow, after pip install --upgrade was run somewhere, and it may move by a full percentage point. The variability has three families of causes:

1. Randomness in the algorithm: train/test split, batch shuffling, initial weights, dropout mask, subsampling in gradient boosting.
2. The environment: the exact versions of Python, NumPy, pandas, scikit-learn, the boosting library, and their transitive dependencies.
3. The hardware and its drivers: BLAS backends, GPU non-determinism, thread counts.

Each family needs its own treatment.

Seed every source of randomness

A seed argument that is set only for the model is not enough. Randomness enters through many APIs, and any one of them left unseeded is enough to make results vary. The correct pattern sets every RNG once, near the top of the entry point:

import os, random
import numpy as np

def seed_everything(seed: int = 42) -> None:
    random.seed(seed)
    np.random.seed(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)
    # Framework-specific:
    # import torch; torch.manual_seed(seed); torch.cuda.manual_seed_all(seed)
    # import tensorflow as tf; tf.random.set_seed(seed)

Then propagate the seed to every function that accepts one: train_test_split(random_state=seed), the model's random_state, DataLoader workers via worker_init_fn, cross-validation splitters. A single unseeded call is enough to break reproducibility.

Pin every dependency, transitively

requirements.txt with unpinned versions is a reproducibility trap. Six months later pip install will resolve differently: a minor bump in a transitive dependency, a new default in the boosting library, a numpy API tightening. The goal is a lock file, not a wish list.

Two workflows are common. Poetry takes a pyproject.toml with constraints and produces a poetry.lock pinning every transitive dependency down to its hash. pip-tools does the same with pip-compile requirements.in > requirements.txt, generating pinned versions and hashes. Whichever you choose, commit the lock file next to the code and install from it exclusively in CI and in production.

Freeze the runtime in a training container

Even with a lock file, the local Python is not the CI Python is not the production Python. A container closes that last gap. A minimal training Dockerfile for the churn project:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --require-hashes -r requirements.txt
COPY src/ src/
ENTRYPOINT ["python", "-m", "src.train"]

--require-hashes refuses to install any wheel whose hash was not pinned in the lock file: a supply-chain attack that swaps a package is caught here. The container is now the ground truth: what runs in CI, what runs in production, what a colleague pulls to reproduce a bug — all the same bytes.

Sources of non-determinism on GPU

Two identical PyTorch or TensorFlow runs on the same GPU, with all seeds set, can still produce different weights. The culprit is the parallel reduction: floating-point addition is not associative, and many CUDA kernels sum partial results in a non-deterministic order for performance.

You control this at a cost. In PyTorch:

import torch
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

The three lines together force deterministic kernels, disable cuDNN's autotuner (which picks different kernels depending on load), and pay in throughput — typically 10 to 30 % slower on convolutions. For a research paper, an audit trail, or a regression that must be reproduced, the cost is worth it. For high-throughput training where results are averaged over many runs, it usually is not.

What "reproducible" actually means for the churn project

For the rest of this course, a reproducible training means: given the same commit hash, the same data version (module 4), and the same container image (module 6), the training produces the same model artifact whose SHA-256 matches. That is what MLflow will log in module 3 and what the registry will promote in module 5. If any of those three inputs drifts, the model does too — and you want to know which one.

The illusion of the fixed seed

Setting one seed and rerunning the notebook is not reproducibility, it is a demonstration that the algorithm accepts a seed. True reproducibility requires all three axes — code, data, environment — pinned together. Fixing only one gives false confidence.

Summary

• Non-reproducibility comes from three families: algorithmic randomness, environment drift and hardware non-determinism; treat all three.
• Seed every RNG (Python, NumPy, framework, dataloaders, splitters); one unseeded call breaks the chain.
• Pin dependencies transitively with a lock file and install from it exclusively; a container closes the last gap.
• GPU determinism is opt-in and costs 10–30 % throughput; enable it when a run must be replayable, accept variance when it must not.

Next module: MLflow tracking — recording every run so you can compare them and pick the winner without relying on memory.