Skip to main content

ONNX Runtime: an exchange format to serve models anywhere

ONNX is the format that severs the link between the workshop where a model is trained and the machine where it is served. An ONNX artifact loads into a C++, Java, JavaScript or Go inference engine, runs on CPU, CUDA, TensorRT or a mobile processor, and no longer needs PyTorch, TensorFlow or a Python interpreter. This course teaches you how to make that switch without silently breaking your model.

Course duration: 4h​

What you will learn​

  • Read an ONNX graph, understand operator sets (opsets) and their versions.
  • Export a model from PyTorch with torch.onnx.export, handling dynamic axes and input names.
  • Export a TensorFlow or Keras model with tf2onnx, and cope with the NHWC vs NCHW convention.
  • Numerically verify the equivalence between the source model and its ONNX export.
  • Apply the graph optimization levels of ONNX Runtime and measure the actual gain.
  • Quantize a model to INT8, dynamically or statically with a calibration dataset.
  • Pick and order execution providers: CPU, CUDA, TensorRT; detect a silent fallback.
  • Compare the performance of PyTorch, ONNX Runtime CPU, CUDA and TensorRT under a reproducible protocol.
  • Diagnose an unsupported operator and work around it without retraining the model.
  • Serve an ONNX model behind FastAPI, with a shared session and preprocessing identical to training.

Prerequisites​

  • PyTorch (course 09) or TensorFlow / Keras (course 08). You do not need both: modules 2 and 3 are independent.
  • Python, NumPy and a bit of command-line experience.
  • Deep learning fundamentals (course 07) to understand what a computation graph, a tensor and a convolution layer are.

The running example​

Two models travel together throughout the course, sharing the same tensors and the same measurements: the ResNet18 fine-tuned in module 9 of the PyTorch course on Fashion-MNIST, and a small transformer text encoder for sentence classification (inspired by course 13). The first walks the classical path — CNN, well-supported operators, dramatic gains from TensorRT; the second reveals the subtler traps — token embeddings, attention mask, one stubborn operator that will need a workaround. By the end of the course, both models have been exported, quantized, benchmarked and served behind a small HTTP API.

Course modules​

  1. The ONNX format: graph, operators, versions.
  2. Exporting from PyTorch.
  3. Exporting from TensorFlow.
  4. Verifying numerical equivalence.
  5. Graph optimization.
  6. ONNX quantization.
  7. Execution providers: CPU, GPU, TensorRT.
  8. Performance benchmarking.
  9. Unsupported operators and workarounds.
  10. Serving an ONNX model.

Assessment and certificate​

The course ends with a 40-question exam covering all ten modules, from choosing an opset to reading an ONNX Runtime latency profile. On success, a certificate of completion is issued immediately; its number can be verified by any third party on the platform.

Free courses, by contrast, end with a 5-question quiz and a certificate preview, without certification.