Module 8 — Performance benchmarking

Modules 5, 6 and 7 promised speedups. This module verifies them. Deep learning benchmarks are famously unreliable: the same model on the same machine can report a 5x speedup or a 5x slowdown depending on how the measurement was performed. A missing warmup phase, a laptop dropping to a low-power state, thermal throttling on the third run, a Python allocator warming its cache — each can move the number by an order of magnitude. This module fixes a protocol, applies it to the ResNet18 and the text encoder, and produces the comparison table that the rest of the course has been referring to.

The protocol is short enough to remember and strict enough to defend. Anyone reproducing it on the same hardware should reach the same numbers within 5 %.

The five rules of a defensible measurement

Warm up before you time. The first inference is always slow: memory allocators grow their pools, CUDA compiles kernels, TensorRT populates its per-shape cache, the CPU frequency scales up from idle. Run 10 to 50 iterations before starting to record. Nothing before the warmup counts.

Measure the same operation, many times. One inference tells you nothing; timing measurements have noise. Record at least 100 iterations per configuration. On fast models, run 1 000 or more.

Report percentiles, not means. The mean is dragged around by outliers — a GC pause, an OS scheduler decision, a background process. The median (P50) is the typical latency; the P95 is what your worst 5 % of users see; the P99 is what a lot of monitoring alerts fire on. A mean without percentiles hides everything that matters.

Match the shape you serve. Serving one request at a time is different from serving a batch of 32. GPU throughput often multiplies with batch, but latency worsens; CPU throughput is much flatter. Report both single-request latency and batched throughput.

Isolate the host. Close other applications, disable GPU sharing, plug the laptop into power, disable Turbo Boost variability if the OS lets you. On a shared machine, the numbers are entertainment; on a dedicated one, they can be defended.

The Python benchmark loop

The pattern encapsulates all five rules in a small utility:

import numpy as np
import time
from statistics import median

def bench(session, feed_fn, n=200, warmup=20):
    for _ in range(warmup):
        session.run(None, feed_fn())
    ts = []
    for _ in range(n):
        feed = feed_fn()
        t0 = time.perf_counter()
        session.run(None, feed)
        ts.append((time.perf_counter() - t0) * 1000.0)
    p50 = median(ts)
    p95 = float(np.percentile(ts, 95))
    p99 = float(np.percentile(ts, 99))
    thr = 1000.0 / p50
    return {"p50_ms": p50, "p95_ms": p95, "p99_ms": p99, "throughput_ips": thr}

feed_fn is a callable that returns a fresh input dictionary on each call, which avoids ONNX Runtime caching the same input tensor across iterations (a subtle way of measuring cache hits instead of real inference).

For the ResNet18 at batch 1:

import onnxruntime as ort

def feed_single():
    return {"input": np.random.rand(1, 3, 224, 224).astype(np.float32)}

sess = ort.InferenceSession("resnet.onnx", providers=["CPUExecutionProvider"])
print(bench(sess, feed_single))

For the same model at batch 32, feeding a shape-varying input relies on the dynamic axis declared in module 2:

def feed_batch(bs):
    return lambda: {"input": np.random.rand(bs, 3, 224, 224).astype(np.float32)}

print(bench(sess, feed_batch(32)))

onnxruntime_perf_test, the reference tool

ONNX Runtime ships a native command-line benchmark called onnxruntime_perf_test, distributed with the ONNX Runtime package. It runs the measurement outside the Python interpreter, avoiding the interpreter's noise, and produces percentiles by default.

onnxruntime_perf_test \
    -e cuda \
    -r 1000 \
    -c 4 \
    -m times \
    -o resnet_bench.json \
    resnet.onnx

• -e cuda selects the CUDA execution provider (cpu, cuda, tensorrt, openvino are supported).
• -r 1000 sets the number of iterations.
• -c 4 uses four concurrent sessions, which matters for throughput measurements.
• -m times reports total time and per-run distribution.
• -o writes a JSON result file.

The tool is authoritative because it is written by the same team that writes the runtime; when a Python benchmark disagrees with onnxruntime_perf_test, the Python benchmark is usually wrong.

A comparison table for the ResNet18

Applying the protocol to the ResNet18 on a laptop with an Intel i7-12700H CPU and an RTX 3070 GPU produces a table shaped like this — the exact numbers depend on the hardware, but the ratios are typical:

Configuration	P50 (ms)	P95 (ms)	Throughput (ips)	Speedup vs PyTorch CPU
PyTorch, CPU, batch 1	22.4	25.1	45	1.0x
ONNX Runtime, CPU, batch 1	12.8	14.0	78	1.75x
ONNX Runtime, CPU, INT8 static	6.2	7.1	161	3.6x
PyTorch, CUDA, batch 1	3.9	5.2	256	5.7x
ONNX Runtime, CUDA, batch 1	3.1	3.6	322	7.2x
ONNX Runtime, TensorRT, FP16	1.4	1.8	714	16.0x
ONNX Runtime, TensorRT, INT8	0.9	1.2	1 111	24.9x

Two lessons emerge every time.

Batching matters more than the last provider tweak. Going from batch 1 to batch 32 on the same TensorRT INT8 configuration typically drops per-image latency by another 3-5x. Systems that serve request-by-request leave that throughput on the table.

INT8 without a hardware accelerator is a size optimisation, not a speed one. On the CPU without VNNI, the INT8 speedup collapses to marginal. On TensorRT, it stacks with FP16 fusion for a compound win. The provider decides whether quantization pays.

The text encoder tells a different story

The transformer is dominated by matrix multiplication. The measurement often shows a different pattern:

Configuration	P50 (ms, seq=128)	Speedup vs PyTorch CPU
PyTorch, CPU	58.0	1.0x
ONNX Runtime, CPU, dynamic INT8	18.3	3.2x
ONNX Runtime, CUDA, FP32	4.6	12.6x
ONNX Runtime, TensorRT, FP16	1.9	30.5x

Dynamic INT8 on CPU is unusually effective on transformers because their bulk is MatMul. On the ResNet, dominated by Conv, dynamic INT8 wins less than static INT8. This is why one benchmark table per model is the right amount, and "our results" without a table is not.

A benchmark without percentiles is a marketing number, not an engineering one

"The model runs in 5 ms" — mean or median? On what percentile? Any measurement that produces a single scalar is a claim, not evidence. Reviewers who accept single-number benchmarks are the reason production launches ship with P99 latencies three times what the demo showed.

In summary

• A defensible benchmark has warmup, at least 100 timed iterations, percentiles (P50/P95/P99) rather than means, and matches the batch shape of production serving.
• onnxruntime_perf_test is the reference tool distributed with ONNX Runtime; when Python numbers and the tool disagree, trust the tool.
• Expect an ONNX Runtime CPU speedup of about 1.5–2x over PyTorch CPU; CUDA around 5–10x; TensorRT FP16 15–20x; TensorRT INT8 stacks further to 20–30x — on hardware that supports them.
• The model shape decides the winners: convolutions favour static INT8 with per-channel scales; transformers favour dynamic INT8 on CPU and TensorRT FP16 on GPU.

Next module: what to do when the export fails because an operator is not supported.