Skip to main content

Module 9 — Containerization and deployment

The service now runs on a laptop with uvicorn app.main:app. This module packages it as a container — small, non-root, reproducible, with a health probe the orchestrator understands — and then chooses how many worker processes and replicas to run behind a load balancer. The container built here is what CI/CD (course 20, module 7) publishes and what the Streamlit dashboard (course 38) will call in staging.

What the image must and must not contain​

The rule from course 20 module 6 is worth repeating because it changes what you write in the Dockerfile:

  • Must contain: the Python runtime, the pinned runtime dependencies, the inference code, and either the model file or the credentials to fetch it at startup.
  • Must not contain: the training code, notebooks, training data, developer tools (git, curl, compilers), or any secret hard-coded in the image.

A .dockerignore file is the cheapest guard against accidentally shipping the wrong things:

# .dockerignore
.git
.venv
__pycache__
*.pyc
tests/
notebooks/
data/
mlruns/
*.md
.env

Every entry above has cost a team an incident somewhere: a mlruns/ directory shipped inside the image (60 MB of MLflow files served to the internet), a .env with a real API key baked in, a notebooks/ folder including a printout of a customer table.

A multi-stage Dockerfile​

Multi-stage builds separate the layers used to install dependencies from the runtime layer, which produces a smaller final image and, more importantly, one that does not contain the tools used to build it (a common source of vulnerabilities). Save the following as Dockerfile:

# syntax=docker/dockerfile:1
FROM python:3.11-slim AS builder

ENV PIP_NO_CACHE_DIR=1 PIP_DISABLE_PIP_VERSION_CHECK=1
WORKDIR /build

# Install into a virtual environment we can copy over as one directory.
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

COPY requirements.txt .
RUN pip install --require-hashes -r requirements.txt


# ---- runtime ----
FROM python:3.11-slim AS runtime

# Fixed non-root user for defense in depth.
RUN useradd --create-home --uid 10001 app
WORKDIR /home/app

# Copy the site-packages only, not the build tools.
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH" PYTHONUNBUFFERED=1

# Then the application code.
COPY --chown=app:app app/ app/

USER app
EXPOSE 8080

HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
  CMD python -c "import urllib.request, sys; \
    sys.exit(0 if urllib.request.urlopen('http://localhost:8080/healthz').status == 200 else 1)"

CMD ["gunicorn", "-k", "uvicorn.workers.UvicornWorker", \
     "-w", "2", "-b", "0.0.0.0:8080", "--timeout", "60", "app.main:app"]

Every line encodes a decision:

  • python:3.11-slim trades several hundred megabytes of build tools for a smaller attack surface. alpine is smaller still but the wheels of scikit-learn, numpy and friends are built for glibc; the musl mismatch on Alpine costs painful compile-from-source builds.
  • --require-hashes enforces that every wheel matches the SHA recorded when the lock file was created. This is the supply-chain protection from course 20 module 2.
  • useradd --uid 10001 app creates a fixed non-root user. Container escape techniques that assume root-in-container are neutralized. UID 10001 is high enough not to collide with host UIDs on typical Linux distributions.
  • HEALTHCHECK targets /healthz (the liveness probe from module 8). The orchestrator will restart the container if the probe fails three times.
  • gunicorn -k uvicorn.workers.UvicornWorker -w 2 runs Uvicorn under Gunicorn's process manager, with two worker processes. Sizing is discussed below.

Copying dependencies before code means the dependency layer is cached across code changes: a one-line fix in app/main.py rebuilds in seconds instead of minutes.

The requirements.txt and its lock​

A requirements.txt with ranges (fastapi>=0.115) is a wish list, not a lock. Produce a hashed lock with pip-compile --generate-hashes (from pip-tools) or with poetry export --with-hashes:

# terminal
pip install pip-tools
pip-compile --generate-hashes --output-file=requirements.txt requirements.in

requirements.in holds the direct dependencies (fastapi, uvicorn[standard], gunicorn, pydantic, mlflow, pandas, scikit-learn, pyjwt, slowapi, prometheus-fastapi-instrumentator). requirements.txt holds every transitive dependency, pinned to an exact version and to a set of wheel hashes. Two rebuilds of the same tag now produce the same bytes.

Environment variables carry every deployment difference​

The image is the same across environments. What changes is a handful of environment variables:

# .env.example — checked into git, values are placeholders
MODEL_URI=models:/churn-classifier@production
MLFLOW_TRACKING_URI=https://mlflow.internal
API_KEY_STORE_URL=redis://redis:6379/0
LOG_LEVEL=INFO
UVICORN_WORKERS=2

Two anti-patterns to avoid:

  • Baking values into the image (ENV MODEL_URI=... in the Dockerfile). Now staging and production need different images, and rollback is not "redeploy image X" but "rebuild image X for the target".
  • Reading a .env file at runtime in production. Fine on a laptop, wrong in production: secrets belong in the orchestrator's secret store (Kubernetes Secret, ECS Parameter Store, systemd credentials), which mounts them as env vars at start.

Workers: the right number is small​

Uvicorn's --workers flag (or gunicorn -w N) forks N worker processes. Each worker holds its own copy of the model in memory. A 400 MB model with 4 workers is 1.6 GB of RAM — before the framework, the interpreters, and the request pools.

Two rules of thumb from experience, refined in module 10:

  • CPU-bound scoring: workers = number of CPU cores available to the container. Two workers on a 2-vCPU pod, four on a 4-vCPU pod.
  • I/O-bound scoring (feature store fetches dominate the request time): workers = 2 × cores; the extra workers cover the I/O wait.

Both numbers are starting points. Load-testing with Locust in module 10 replaces them with a measured value for your specific model and traffic.

Building and running​

Locally:

# terminal
docker build -t churn-api:0.9.0 .
docker run --rm -p 8080:8080 \
  -e MODEL_URI="models:/churn-classifier@production" \
  -e MLFLOW_TRACKING_URI="https://mlflow.internal" \
  churn-api:0.9.0

# In another terminal:
curl -H "X-API-Key: dev-key-do-not-use" \
     -X POST http://localhost:8080/predict \
     -H "content-type: application/json" \
     -d '{"subscriber_id":"sub-000123","tenure_months":12,"monthly_charges":75.5,
          "total_charges":900,"contract_type":"month_to_month",
          "payment_method":"credit_card","is_senior":false}'

If the model loads (module 3) and the route returns a JSON body (module 4) with the expected shape, the container is healthy.

Deployment shapes​

Three shapes cover 95 % of ML API deployments:

  • A single VM with docker compose. One churn-api container behind Nginx. Fine for a demo, a proof of concept, or a low-traffic internal tool. Rollback is docker compose up -d --wait <previous-image-tag>.
  • A managed container service (AWS ECS Fargate, Google Cloud Run, Azure Container Apps). No cluster to run; you push an image and specify CPU, memory, min and max replicas. Cloud Run in particular is well-suited for spiky ML traffic because it scales to zero when idle.
  • A Kubernetes cluster. A Deployment with spec.replicas: 3, a Service for the load balancer, and probes wired to /healthz and /ready. Worth the complexity only when you have multiple services or need custom auto-scaling. The MLOps course (module 8) walks through this shape.

Whichever you pick, the container from this module is the deployment unit. Only the surrounding YAML or JSON changes.

Scanning before push​

A vulnerability scanner in CI is the last cheap protection. Trivy and Grype both scan images for known CVEs; a policy that fails the build on any HIGH or CRITICAL finding catches the vast majority of drive-by vulnerabilities without paging on informational ones. Scanning after deploy is closing the barn door.

A 4 GB ML image is a smell, not a badge

An image that lands at 3–5 GB usually contains something it should not: a full CUDA runtime on a CPU-only service, a datasets/ directory, unused deep-learning libraries, dev tools left in the runtime stage. Aim for 400–800 MB on a scikit-learn service and 1.5–2 GB on a small PyTorch service. Anything above 2 GB deserves a docker history audit before it ships.

In summary​

  • Build with a multi-stage Dockerfile: install into a venv in builder, copy only the venv and the code into runtime. Run as a non-root user.
  • Pin every dependency with a --require-hashes lock; keep secrets and per-environment values in environment variables injected by the orchestrator.
  • Choose workers = cores for CPU-bound scoring, 2 × cores for I/O-bound; measure in module 10 and adjust.
  • Wire the container's HEALTHCHECK to /healthz, and the orchestrator's probes to both /healthz (liveness) and /ready (readiness). Scan the image in CI before push.

Next module: load testing and sizing — running a Locust scenario against the container of this module, reading percentiles, and deciding how many workers and replicas to run for a target QPS and p95.