Module 4 — Verifying numerical equivalence

The most treacherous export failure is not the one that raises. It is the one that produces a .onnx file that loads, runs, delivers plausible outputs, and disagrees with the source model by a percentage point here, a class boundary there. Trained on the same data, evaluated on the same test set, the two report different accuracies. Nobody notices until the ONNX version ships and the monitoring dashboards drift. This module puts a mandatory checkpoint between "the export succeeded" and "the export is correct".

The check is short. It runs in less than a second on the two running models. It is worth writing as a reusable function and calling it every single time an export is produced — in the export script, in continuous integration, and again at deployment time from the artifact repository. The rest of the course assumes it is in place.

The two-line check that catches most bugs

Given a model, an example input, and an .onnx file, the entire verification is a comparison of two numeric outputs:

import numpy as np
import torch
import onnxruntime as ort

def verify_onnx(model, example, onnx_path, atol=1e-4, rtol=1e-3):
    model.eval()
    with torch.no_grad():
        torch_out = model(example).cpu().numpy()

    sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])
    input_name = sess.get_inputs()[0].name
    onnx_out = sess.run(None, {input_name: example.numpy()})[0]

    diff = np.abs(torch_out - onnx_out)
    print(f"max abs diff: {diff.max():.2e}, mean abs diff: {diff.mean():.2e}")
    np.testing.assert_allclose(torch_out, onnx_out, atol=atol, rtol=rtol)
    return diff.max()

# For the fine-tuned ResNet18
example = torch.randn(1, 3, 224, 224)
verify_onnx(model, example, "resnet.onnx")

np.testing.assert_allclose raises with a readable message when the two tensors disagree by more than the tolerance. That message is where debugging starts, not where it ends: it tells you the disagreement exists and its magnitude, not its cause.

The same function serves for TensorFlow with a small adaptation — replace the torch.no_grad() block by model(example, training=False).numpy(). The rest is identical.

Choosing tolerances that mean something

The default tolerances in tutorials — 1e-5 or 1e-6 — are strict enough for hand-written arithmetic and wrong for real neural networks. Fused operators, reordered summations, and slightly different Conv implementations between frameworks produce differences an order of magnitude larger without indicating any bug.

The practical scale to memorise:

Order of magnitude	Interpretation
1e-6 to 1e-5	Expected float32 rounding noise.
1e-4	Normal for a deep model with many fused kernels.
1e-3	Suspicious; look at --opset, .eval(), and dtype casts in the graph.
1e-2 and above	Real bug; a layer is missing or misconfigured.

For classification, the tolerance that matters is on the argmax, not the raw logits. A logit difference of 1e-3 typically leaves the top class unchanged; a difference of 1e-1 starts flipping classes near the decision boundary. Add an argmax check alongside the numeric one:

top1_match = (torch_out.argmax(-1) == onnx_out.argmax(-1)).mean()
print(f"top-1 agreement: {top1_match:.4f}")
assert top1_match >= 0.999, "argmax disagrees more than expected"

A run over the validation set is the strongest test. Agreement above 99.9 % on 10 000 examples is the number that convinces reviewers.

Cover more than one input

A single example does not exercise the graph. If the model has data-dependent behaviour — an if on a tensor, an attention mask that changes shape — the traced path was the one taken on that one input. The check must therefore run on a diverse batch:

inputs = [
    torch.randn(1, 3, 224, 224),
    torch.zeros(1, 3, 224, 224),
    torch.ones(1, 3, 224, 224),
    torch.randn(8, 3, 224, 224),                    # different batch size
    torch.randn(1, 3, 224, 224) * 10,               # unusual value range
]
for i, x in enumerate(inputs):
    diff = verify_onnx(model, x, "resnet.onnx")
    print(f"input {i}: diff {diff:.2e}")

A batch of 8 exercises the dynamic axis declared in module 2. Zero-valued and one-valued inputs stress activation functions differently. Multiplying by 10 pushes values outside the training distribution and can reveal a normalisation that lives inside the model versus outside. Any of them can uncover a bug that the single-example check missed.

For a text transformer, the diversity is over sequence length. Include a length-1 sequence, a length-32 sequence, and one that exceeds the training median. The attention mask logic often behaves differently at the extremes.

onnx.checker: catch structural issues before running

Before the numerical check, run the structural one. onnx.checker.check_model parses the graph and reports any violation of the ONNX specification: missing initializers, mismatched shapes between a node's declared output and its consumer's declared input, invalid attribute values.

import onnx
onnx_model = onnx.load("resnet.onnx")
onnx.checker.check_model(onnx_model)

It is fast, mandatory, and often catches problems before they become opaque runtime errors. A model that fails check_model will not load reliably in any runtime; a model that passes may still disagree numerically, which is the reason both checks live in the same pipeline.

Half-precision drift and the checks that catch it

Modules 6 and 7 introduce float16 execution — through quantization or the CUDA provider — and both cost tolerance. A model that agrees to 1e-4 in float32 easily disagrees by 1e-2 or 1e-1 after conversion to float16, because the intermediate accumulations lose several decimal digits of precision.

Two habits keep this under control. First, run the parity check again after every conversion step, not only after the initial export. The float16-cast model is a new artifact and deserves its own verification. Second, evaluate the top-1 accuracy on the actual validation set — never trust that "the logits are close enough" translates to "the accuracy is close enough" when the model was pushed into a lower-precision regime.

Verification numbers belong next to the model file

Store, alongside every deployed .onnx, the max absolute difference against its source, the opset used at export time, the tolerance thresholds, and the top-1 agreement on a validation set. Six months later, when someone asks whether the version in production drifted, those three numbers answer the question in seconds instead of hours.

Diagnosing a numerical disagreement

When assert_allclose fires, the graph is where the answer lives. Two techniques cover most cases.

Layer-by-layer comparison. Add hooks in PyTorch (or tf.keras.Model submodel calls) that capture the output of each layer. Load the same input into ONNX Runtime with all intermediate tensors exposed as extra outputs — modify the ONNX graph to declare them as outputs. Compare layer by layer; the first layer where the difference explodes points at the offending operator.

Netron inspection. Open the file in Netron. Look for unexpected nodes: a Cast where none should be, a fused LayerNormalization that decomposed into ReduceMean and friends, a Constant with a value that does not match the trained weight. These visual patterns are faster to spot than to enumerate in code.

In summary

• Every export is followed by a numerical parity check: run the source model and the ONNX file on the same input, compare with assert_allclose, and record the max absolute difference.
• Tolerances must be calibrated to reality: 1e-6 is unreachable, 1e-4 is normal, above 1e-3 there is a real bug.
• Cover diverse inputs — zeros, ones, large batches, unusual value ranges — and, for classifiers, check the argmax agreement on a validation set, not only the raw logits.
• onnx.checker.check_model catches structural violations before they become opaque runtime errors, and is faster and more informative than starting from a failed inference session.

Next module: graph optimization, where ONNX Runtime rewrites the graph into a faster equivalent.