Recap and final exam
Ten modules to move two models — a Fashion-MNIST ResNet18 and a small text encoder — from their training frameworks into an interoperable artifact and then into a running service. This chapter condenses each module into what to retain, draws the threads that run through the whole course, and closes with the 40-question exam that gates the certificate.
The course at a glance
| Module | What to retain |
|---|---|
| 1. ONNX format | A .onnx file is a static computation graph in Protocol Buffers, versioned by opset; use Netron to read it. |
| 2. Export from PyTorch | torch.onnx.export, model.eval() mandatory, dynamic_axes on the batch axis, deliberate input_names. |
| 3. Export from TensorFlow | tf2onnx.convert.from_keras, serving signature with training=False, --inputs-as-nchw to standardise layout. |
| 4. Numerical equivalence | np.testing.assert_allclose with atol=1e-4, check on diverse inputs, top-1 agreement on the validation set. |
| 5. Graph optimization | Constant folding, operator fusion, ORT_ENABLE_ALL in production, optimized_model_filepath to cache. |
| 6. Quantization | Dynamic for transformers, static + per_channel=True + QDQ for CNNs; calibration data must be production-realistic. |
| 7. Execution providers | Ordered list, assert get_providers() after startup, watch out for silent CPU fallback and partial fallback. |
| 8. Benchmarking | Warmup, 100+ iterations, P50/P95/P99, onnxruntime_perf_test as the reference; batch shape matters. |
| 9. Unsupported operators | Raise opset first, try dynamo, rewrite Python control flow into tensor ops, custom op only as a last resort. |
| 10. Serving | One InferenceSession per process, thread config aware of workers, preprocessing identical to training. |
Four threads running through the course
A .onnx file is a contract, not a translation. Everything about the format — its explicit graph, its opset versioning, its portability across runtimes — exists so that the training team and the serving team can hand off a single artifact without also handing off the training environment. The moment a hidden dependency creeps in — a Python-side preprocessing step, a custom operator library, an implicit opset — the contract breaks. Module 3's NHWC-to-NCHW flag, module 9's advice to keep preprocessing outside the graph, and module 10's rule about shared preprocessing code all serve the same principle: an artifact must be self-contained, or it is not a production artifact.
The parity check is the single non-negotiable step. Module 4 established it, and every subsequent module rebuilt the pipeline around it. After graph optimization, rerun the parity check. After static quantization, rerun the parity check with widened tolerances and validation-set accuracy. After changing an execution provider, rerun the parity check. Ignoring any of these is how a service silently ships a model that agrees with its source on the sanity input and disagrees on real traffic. The habit that separates a model that works in the demo from one that works in production is the habit of comparing outputs after every transformation.
Silence is the enemy. Three of the course's costliest bugs — training-mode dropout leaking into the export, providers=None on a GPU host, TensorRT partially falling back to CPU on a few nodes — share one property: they do not raise. The service starts, the responses come back, and the metrics look normal on synthetic tests. The remedy is always explicit assertion. Assert model.eval() before export. Assert get_providers()[0] matches the expected provider. Assert the parity difference stays under the recorded tolerance. Every explicit assertion turns a silent bug into a loud one, which is the only kind you can fix before it reaches users.
Measure before optimizing, then measure again. Modules 5 to 8 each introduce a lever — graph optimization, quantization, execution providers, benchmarking. The temptation to enable them all and skip to a comparison is what turns a real 10x speedup into a claimed 40x that no one can reproduce. The discipline is boring: benchmark with the module 8 protocol before every change, benchmark after, and record both. Then the conversation with a reviewer is "we moved from 22 ms to 1.4 ms because we quantized statically and switched from CUDA to TensorRT FP16", and the numbers can be re-run at any point on the same hardware.
The final exam
The exam has 40 questions drawn from a pool of 48, covering all ten modules. Expect questions on: how an opset differs from a framework version and what happens when either is wrong, why declaring dynamic_axes at export time is what enables batched serving, why NHWC-versus-NCHW is the recurring TensorFlow-to-PyTorch trap, what tolerance is realistic in assert_allclose and why 1e-6 is a naive target, how constant folding and operator fusion differ, when dynamic INT8 wins over static and vice versa, how to detect a silent CPU fallback, what a defensible benchmarking protocol contains, how to work around an unsupported operator without registering a custom one, and why serving needs one session per process rather than one per request.
Several questions present situations to diagnose: a service that quietly runs 10x slower than the benchmark, an ONNX file that classifies correctly on the sanity image but drifts on the real dataset, a TensorRT build that keeps recompiling on every restart, a transformer export that fails on opset 11 but succeeds on 17. What is assessed is judgment, not memorised API signatures.
On success, your certificate of completion is issued immediately; its number is verifiable on the platform by any third party.
Reread the "what to retain" table and, for each row, ask yourself the diagnostic question: what symptom would I see in production if I got this wrong? If you can explain why a missing dynamic_axes collapses GPU throughput, why an incorrect provider ordering hides a 10x slowdown, and why a "5 ms" benchmark without percentiles is a marketing number rather than an engineering one, you are ready. Good luck!
Final exam
Ready to validate this course?
40 questions drawn at random from the course bank · passing score 70% · verifiable PDF certificate issued immediately on success.
Start the examYou need to be signed in to your InSkillML account with an active subscription. You can also start the exam from My courses.