Skip to main content

Module 7 — Execution providers: CPU, GPU, TensorRT

An execution provider in ONNX Runtime is the code that actually runs a subgraph on a specific piece of hardware. CPUExecutionProvider runs everything on the processor. CUDAExecutionProvider runs it on an NVIDIA GPU. TensorRTExecutionProvider compiles it into a TensorRT engine, CoreMLExecutionProvider runs on Apple Silicon Neural Engine, OpenVINOExecutionProvider on Intel iGPU and VPU. The same ONNX file dispatches to any of them by changing a single argument. This module explains what "changing a single argument" hides — and, especially, how to notice when the runtime silently falls back to CPU because the requested provider was not installed or not compatible.

That silent fallback is the single most expensive bug of this course. A team requests CUDAExecutionProvider, the session creation succeeds, inference runs — but on the CPU, and a service that was supposed to answer in 4 ms answers in 40. The metric to check is not "does it work" but "which provider actually took the graph".

Requesting providers​

The list of providers passed to InferenceSession is a preference order, not a hard requirement. ONNX Runtime walks the list from left to right, tries each, and uses the first one it can initialise:

import onnxruntime as ort

session = ort.InferenceSession(
    "resnet.onnx",
    providers=[
        "TensorRTExecutionProvider",
        "CUDAExecutionProvider",
        "CPUExecutionProvider",
    ],
)

CPUExecutionProvider at the end of the list is the safety net: if TensorRT and CUDA both fail to initialise, the session still runs. It is also exactly what causes the silent-fallback bug. Never rely on the ordering to mean "TensorRT is available"; verify it after session creation.

Detecting the silent fallback​

The provider that actually took the graph is available via session.get_providers(). On startup, log it and assert against the expected value:

active = session.get_providers()
print(f"providers active: {active}")
assert active[0] == "TensorRTExecutionProvider", f"expected TensorRT, got {active[0]}"

The list order in get_providers() reflects which providers accepted the graph, most preferred first. A graph running under CUDA returns ['CUDAExecutionProvider', 'CPUExecutionProvider']. A graph that silently fell back to CPU returns ['CPUExecutionProvider'].

A second sanity check catches a subtler variant: partial fallback. TensorRT may accept most nodes and delegate the unsupported ones to CUDA, and CUDA in turn to CPU. The result is a session that reports TensorRT as the primary provider yet runs, silently, at a fraction of its potential speed because a critical node crossed device boundaries three times per inference. Enable verbose logging once, look for NodeAssignments lines, and confirm the assignment matches your expectations:

so = ort.SessionOptions()
so.log_severity_level = 0    # verbose
so.log_verbosity_level = 1
session = ort.InferenceSession("resnet.onnx", so, providers=[...])

The output shows each subgraph and which provider ran it. Nodes that fell back to CPU appear with an explicit assignment. Fix them at the source — either by re-exporting to a supported opset, or by using a different provider option — before shipping.

providers=None on a GPU-labelled machine is not "use the GPU"

Omitting providers gives you CPUExecutionProvider. Full stop. The runtime will not scan the machine for accelerators; it uses what you tell it to use. This is by design — a service must not pick up an unexpected accelerator — and it is why every production InferenceSession should pass providers explicitly, even when running on CPU.

Session options that matter​

Beyond providers, three session options change the picture on real hardware.

intra_op_num_threads controls how many threads a single operator can use. On a 32-core server running one request at a time, set it to 32. Running many concurrent sessions on the same machine, set it lower to leave cores for other sessions.

inter_op_num_threads controls how many independent subgraphs can run in parallel within one inference. For most models this is 1; for models with independent branches it can go higher.

enable_cpu_mem_arena pools CPU memory allocations. Enabled by default; disabling it can help debugging leaks but never in production.

so = ort.SessionOptions()
so.intra_op_num_threads = 8
so.inter_op_num_threads = 1
so.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL
session = ort.InferenceSession("resnet.onnx", so, providers=["CPUExecutionProvider"])

The CUDA provider​

The CUDA provider runs the graph on an NVIDIA GPU using cuDNN and cuBLAS kernels. It is the right choice for medium-scale inference where you want the same graph running on any CUDA card without a per-model compilation step. Provider options let you point it at a specific device:

providers = [
    ("CUDAExecutionProvider", {
        "device_id": 0,
        "arena_extend_strategy": "kNextPowerOfTwo",
        "cudnn_conv_algo_search": "EXHAUSTIVE",
    }),
    "CPUExecutionProvider",
]
session = ort.InferenceSession("resnet.onnx", providers=providers)

The first inference is slower — cuDNN is running its algorithm search to pick the fastest convolution implementation for the shapes it sees. Subsequent inferences are stable. The warmup step of module 8's benchmark exists to absorb this.

The TensorRT provider​

TensorRT is NVIDIA's specialised inference compiler. It reads the ONNX graph, fuses aggressively, picks INT8 or FP16 kernels where possible, and produces a compiled engine specialised for the input shapes and the target GPU. On modern GPUs this typically delivers 2 to 5x more throughput than CUDA on the same graph, with the same numerical outputs.

The trade-off is compilation time. Building the engine can take 30 seconds to several minutes on a large transformer. TensorRT can cache the result:

providers = [
    ("TensorRTExecutionProvider", {
        "device_id": 0,
        "trt_max_workspace_size": 4 * 1024 ** 3,             # 4 GB
        "trt_fp16_enable": True,
        "trt_engine_cache_enable": True,
        "trt_engine_cache_path": "./trt_cache",
    }),
    "CUDAExecutionProvider",
    "CPUExecutionProvider",
]

trt_engine_cache_enable=True writes the compiled engine to trt_engine_cache_path after the first build, and reuses it on subsequent runs. The cache key includes the ONNX file hash and the input shapes, so re-exporting the model or changing the shape recompiles. Ship the cache directory alongside the model on stable infrastructure, or accept the first-run cost.

trt_fp16_enable=True casts activations to FP16 internally when the operator supports it. Rerun the module 4 parity check with widened tolerances; on classifiers, top-1 agreement above 99 % is achievable.

Ordering providers for a real deployment​

The provider list encodes the deployment contract. Three patterns cover almost every case.

CPU-only host, no accelerator installed:

providers=["CPUExecutionProvider"]

GPU host, latency-sensitive: TensorRT if available, CUDA otherwise, and assert one of them took the graph:

providers=["TensorRTExecutionProvider", "CUDAExecutionProvider"]
active = session.get_providers()
assert "CPUExecutionProvider" not in active, "fell back to CPU"

GPU host, best-effort with graceful degradation (a dev environment, not production):

providers=["TensorRTExecutionProvider", "CUDAExecutionProvider", "CPUExecutionProvider"]

The difference between the second and the third is one line and an assertion, but it is the difference between a service that fails loudly and one that runs silently on CPU during an outage.

In summary​

  • Execution providers are a preference-ordered list; ONNX Runtime picks the first one that initialises, and CPUExecutionProvider at the end is the safety net that also causes the silent-fallback bug.
  • Always call session.get_providers() and assert the expected provider took the graph; enable verbose logs to detect partial fallback where individual nodes cross device boundaries.
  • The CUDA provider is a general-purpose accelerator; the TensorRT provider is a specialised compiler that produces a cached engine and typically wins 2–5x more on top of CUDA.
  • A production InferenceSession always passes providers explicitly; omitting the argument gives you CPU, regardless of the hardware present.

Next module: how to measure the difference between all these providers with a benchmarking protocol you can defend.