Skip to main content

Module 10 — Serving an ONNX model

Every previous module built toward this one. We exported a model in modules 2 and 3, verified its numerical parity in 4, optimized it in 5, quantized it in 6, picked an execution provider in 7, measured its performance in 8, and worked around its exports quirks in 9. What is left is turning that artifact into a service — a process that accepts HTTP requests, runs the model, and returns predictions. This module builds a minimal but production-shaped FastAPI service around the ResNet18, points at the pitfalls that ruin the numbers of module 8, and closes with a preview of ONNX Runtime Web for in-browser inference. Course 40 pushes deployment further; this module makes sure the serving code does not undo everything the previous nine chapters achieved.

One session per process, shared across requests​

The single most consequential architectural decision is when the InferenceSession is created. A session carries the loaded graph, the optimized rewrites, the CUDA or TensorRT engine cache, and dozens of megabytes of allocator state. Creating one on every request adds hundreds of milliseconds to each response and undoes every speedup earned earlier.

The right pattern is one session shared across all requests, initialised at process startup:

from contextlib import asynccontextmanager
from fastapi import FastAPI
import onnxruntime as ort
import numpy as np

@asynccontextmanager
async def lifespan(app: FastAPI):
    so = ort.SessionOptions()
    so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
    so.intra_op_num_threads = 4
    app.state.session = ort.InferenceSession(
        "resnet.onnx",
        so,
        providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
    )
    active = app.state.session.get_providers()
    assert active[0] == "CUDAExecutionProvider", f"fell back to {active}"
    yield

app = FastAPI(lifespan=lifespan)

Two production-shaped details are worth pausing on. The lifespan handler is FastAPI's recommended way to run startup code; it guarantees the session is ready before the first request is served. The assertion on get_providers() is the module 7 check applied at startup: a server that silently fell back to CPU refuses to start rather than serving slow requests silently.

InferenceSession is thread-safe — the session's .run can be called from multiple threads simultaneously. FastAPI runs one worker per process by default; scaling up means adding more workers via uvicorn --workers N, each carrying its own session. This is standard practice and does not need per-request locking.

Preprocessing identical to training​

The service's job is more than calling session.run. It must take a raw image, run exactly the same preprocessing the training script used, and only then feed the tensor. Any drift between training-time and serving-time preprocessing is a silent bug: the model produces plausible outputs, none of which match the training-time distribution, and accuracy on real data collapses without any error message.

The recipe for Fashion-MNIST via the fine-tuned ResNet18 uses ImageNet normalisation constants:

from PIL import Image
from io import BytesIO

MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
STD  = np.array([0.229, 0.224, 0.225], dtype=np.float32)
LABELS = ["T-shirt", "Trouser", "Pullover", "Dress", "Coat",
          "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot"]

def preprocess(raw_bytes: bytes) -> np.ndarray:
    img = Image.open(BytesIO(raw_bytes)).convert("RGB").resize((224, 224))
    arr = np.asarray(img, dtype=np.float32) / 255.0
    arr = (arr - MEAN) / STD                  # HWC, values in normalised range
    arr = np.transpose(arr, (2, 0, 1))        # HWC -> CHW
    return arr[None, ...].astype(np.float32)  # add batch axis

The three lines that produce the tensor must be byte-for-byte the same as in the training script. Never reimplement them from memory. A common practice is to store the preprocessing code in a shared package used by both training and serving, so a change ripples to both.

The prediction endpoint​

With the session in app.state and preprocessing in place, the endpoint is short:

from fastapi import UploadFile, HTTPException
import numpy as np

@app.post("/predict")
async def predict(file: UploadFile):
    if file.content_type not in {"image/jpeg", "image/png"}:
        raise HTTPException(415, f"unsupported type {file.content_type}")
    x = preprocess(await file.read())
    logits = app.state.session.run(None, {"input": x})[0][0]
    probs = np.exp(logits - logits.max())
    probs = probs / probs.sum()
    top = int(probs.argmax())
    return {"label": LABELS[top], "confidence": float(probs[top])}

np.exp(logits - logits.max()) computes a numerically stable softmax; subtracting the max before exponentiating prevents overflow on large logits and costs nothing. The content_type check rejects payloads that would raise deeper in the stack and produce cryptic 500 errors.

The endpoint accepts one file per request. Batching multiple requests into a single session.run — batching that would exploit the dynamic axis of module 2 — is a serving-layer optimisation, not a per-request one, and belongs in a queue-based worker rather than in the endpoint. Course 40 explores that path.

Health and readiness​

Any production service needs endpoints Kubernetes, HAProxy or a load balancer can probe. Two are enough:

@app.get("/health")
def health():
    return {"status": "ok"}

@app.get("/ready")
def ready():
    if not hasattr(app.state, "session"):
        raise HTTPException(503, "model not loaded")
    return {"providers": app.state.session.get_providers()}

/ready returning the active providers is the smallest introspection endpoint that catches "the model is loaded but running on CPU because CUDA drivers are missing" in staging. That single JSON line has saved many silent-fallback incidents.

Thread configuration on shared hardware​

On a CPU-only host with multiple concurrent workers, intra_op_num_threads deserves attention. Suppose 4 uvicorn workers on a 16-core machine. If every worker sets intra_op_num_threads=16, the workers fight over cores, context-switch constantly, and total throughput collapses. Set it to total_cores / num_workers (4 in this case) and every worker gets its share.

inter_op_num_threads = 1 is the right default for most models. Higher values help only when the graph has independent branches that can run in parallel — a rare case with ResNets and transformers.

ONNX Runtime Web, a preview​

The same .onnx file that runs behind FastAPI on the server also runs in the browser via onnxruntime-web. The JavaScript package weighs a few hundred kilobytes, downloads the WebAssembly runtime lazily, and executes the model client-side with no network round trip for inference:

import * as ort from "onnxruntime-web";

const session = await ort.InferenceSession.create("./resnet.onnx");
const input = new ort.Tensor("float32", preprocessedPixels, [1, 3, 224, 224]);
const results = await session.run({ input });
console.log(results.logits.data);

The model runs on the user's CPU (WebAssembly with SIMD) or GPU (WebGL, WebGPU on newer browsers). For classification on small images, the latency is comparable to a server round trip, and the entire inference happens on the user's device — which changes the privacy story, the offline story and the operational cost. Not every model fits this profile; the quantized version of the ResNet is a strong candidate.

Putting the numbers back on the table​

The module 8 benchmarks assumed a session created once and reused. A serving-layer implementation that respects this — one session per process, thread config aware of concurrency, providers asserted at startup, preprocessing identical to training — reproduces those numbers within measurement noise. When production latency disagrees with the benchmark by more than 20 %, the answer is almost always in this module's checklist rather than in another optimization.

Instrument the endpoint from day one

Wrap the session.run call with a timer, emit the per-request latency to your metrics stack, and add a percentile alert on P95. The moment provider changes, driver upgrades or a new deployment silently regress performance, the alert fires with concrete numbers — and the module 8 benchmark becomes your reference for what "normal" looks like.

In summary​

  • Create one InferenceSession per process at startup (via lifespan), share it across all requests, and assert get_providers() matches the expected provider so silent CPU fallback fails loudly.
  • Preprocessing must be byte-for-byte identical to training; keep it in a shared package so both scripts import the same code.
  • A prediction endpoint is short (preprocess, session.run, softmax, return); batching across requests is a queue-worker concern for course 40, not an in-endpoint one.
  • ONNX Runtime Web runs the same file client-side in the browser via WebAssembly, WebGL or WebGPU — a real option for small quantized models where privacy or offline usage matter.

Next module: the recap of the ten modules and the 40-question exam.