Module 6 — ONNX quantization

The ResNet18 and the text encoder both weigh their fully-connected and convolutional weights in float32: four bytes per parameter. On the ResNet18 that is 45 MB; on a small transformer with 30 million parameters, 120 MB. Both fit fine on a server, but the moment the model has to reach a mobile phone, ship over a cellular network, or run at scale on CPUs paying for RAM by the gigabyte, the size becomes the constraint. Quantization compresses the weights — and often the activations — from float32 to INT8. The file shrinks roughly 4x, inference on CPU speeds up by 2 to 4x, and the accuracy drops by fractions of a percent when done well and by unusable amounts when done wrong. This module explains the two flavours ONNX Runtime offers, when to use each, and how to keep the drop below "fractions of a percent".

The theory in one paragraph

An INT8 value stores 256 discrete levels between two float endpoints, min and max. Converting a float tensor to INT8 means picking min and max such that the rounding error is tolerable across the tensor's real distribution, then storing q = round((x - min) / scale) where scale = (max - min) / 255. Multiplying two INT8 tensors is much cheaper than multiplying two float32 tensors — modern CPUs and GPUs have dedicated instructions (VNNI on Intel, dp4a on NVIDIA, ARM DOT product) for INT8 matrix multiply. The trick is picking min and max. Everything about quantization comes down to that choice.

Dynamic quantization: the easy path

Dynamic quantization computes the min and max of the activations at runtime, on every request. Only the weights are quantized offline, once. The API is one call:

from onnxruntime.quantization import quantize_dynamic, QuantType

quantize_dynamic(
    model_input="text_encoder.onnx",
    model_output="text_encoder.int8.dynamic.onnx",
    weight_type=QuantType.QInt8,
)

The file shrinks by about 4x. Inference on CPU speeds up on all MatMul- and Gemm-heavy layers — which is exactly the shape of a transformer. Convolutional layers benefit less because dynamic quantization does not always quantize Conv; the CPU cost of estimating activation ranges per batch can eat the win.

The gain distribution across models is not uniform:

Model	File size	Inference speedup (CPU)	Top-1 drop
ResNet18 (Fashion-MNIST, our fine-tune)	45 → 12 MB	1.2x	around 0.1 %
Text encoder (transformer, sentence classifier)	120 → 31 MB	2.5–3x	around 0.3 %

Dynamic quantization is the safest first attempt for transformers and models dominated by matrix multiplies. When the numbers above are good enough, stop here.

Static quantization: the calibration path

Static quantization quantizes both weights and activations offline. The min and max of every activation tensor get computed once on a small dataset — the calibration set — and stored inside the model. No runtime estimation, no per-request overhead, more speedup, especially on convolutions.

The calibration set is not the training set and not the validation set. It is a representative sample of the inputs the model will see in production: 100 to 500 examples that cover the input distribution. Too small and the ranges miss rare-but-real values, saturating activations at inference time. Too large is not wrong but wastes calibration time.

import os
import numpy as np
from onnxruntime.quantization import quantize_static, QuantType, CalibrationDataReader

class ResNetCalibrator(CalibrationDataReader):
    def __init__(self, images):
        self.iter = iter(images)

    def get_next(self):
        try:
            img = next(self.iter)
            return {"input": img.astype(np.float32)}
        except StopIteration:
            return None

# 200 preprocessed Fashion-MNIST images in the shape the model expects
calib = [np.random.rand(1, 3, 224, 224) for _ in range(200)]

quantize_static(
    model_input="resnet.onnx",
    model_output="resnet.int8.static.onnx",
    calibration_data_reader=ResNetCalibrator(calib),
    quant_format="QDQ",
    activation_type=QuantType.QInt8,
    weight_type=QuantType.QInt8,
    per_channel=True,
)

Three arguments have visible impact on the outcome. quant_format="QDQ" — QuantizeDequantize — inserts explicit QuantizeLinear and DequantizeLinear nodes around every quantized operator. It is more verbose in the graph but the format understood by every hardware backend, and it is the recommended default. The older QOperator format uses fused quantized ops directly and is less portable across runtimes. per_channel=True computes a separate min/max per output channel of Conv and MatMul, dramatically improving accuracy on wide layers at a negligible cost.

A weak calibration set is the silent killer

A calibration set built from a training-time data generator (with augmentation, random cropping, and normalisation) can produce activation ranges that never occur in production, saturating half the activations on the first real request. Feed the calibrator with production-realistic preprocessed inputs, then verify with module 4's parity check on held-out inputs it never saw.

QDQ versus QOperator

Two representations exist and understanding the difference saves debugging time.

QOperator: the quantized Conv is a single ONNX node called QLinearConv, taking already-quantized inputs and producing quantized outputs. Compact, but only a small set of runtimes execute those operators efficiently.

QDQ: the graph keeps its original Conv, wrapped by QuantizeLinear at the input and DequantizeLinear at the output. Optimizers on modern runtimes fold that pattern into an efficient INT8 kernel at load time. The graph reads like a float32 graph with quantization annotations, which is much easier to inspect in Netron.

Recent ONNX Runtime releases and every hardware-specific optimizer (TensorRT, OpenVINO) prefer QDQ. Choose it unless a target runtime documents QOperator support explicitly.

Verify, always

The temptation with quantization is to check "does it still classify a cat?" and move on. That is not enough. Run the module 4 parity check with widened tolerances (atol=5e-2, then check top-1 agreement on the validation set):

import onnxruntime as ort, numpy as np

sess_fp = ort.InferenceSession("resnet.onnx",         providers=["CPUExecutionProvider"])
sess_q  = ort.InferenceSession("resnet.int8.static.onnx", providers=["CPUExecutionProvider"])

# 500 preprocessed images with their labels
correct_fp, correct_q, agree = 0, 0, 0
for x, y in val_set[:500]:
    p_fp = sess_fp.run(None, {"input": x})[0].argmax(-1)[0]
    p_q  = sess_q.run(None,  {"input": x})[0].argmax(-1)[0]
    correct_fp += (p_fp == y)
    correct_q  += (p_q  == y)
    agree      += (p_fp == p_q)

print(f"float32 acc: {correct_fp/500:.3f}  int8 acc: {correct_q/500:.3f}  agree: {agree/500:.3f}")

Acceptable numbers: int8 accuracy within 0.5 percentage points of float32, agreement above 99 %. When the gap exceeds one percentage point, the calibration set is the first thing to review; when the gap exceeds three, per_channel=True is often missing.

Hardware compatibility, the practical constraint

Quantization is only fast if the hardware executes INT8 natively. The table condenses the state you can rely on today:

Hardware	INT8 speedup	Notes
Modern Intel CPU (Cascade Lake and later, VNNI)	2–4x	Dynamic and static both win.
ARM v8.4-A CPUs (mobile, Apple M-series)	2–3x	Static preferred.
NVIDIA GPU (Turing and later) via TensorRT	3–5x	Static, per-channel, QDQ format.
Older CPUs without VNNI	Marginal	Quantize for size, not for latency.

The recipe is: quantize once for size and version compatibility, deploy on hardware that accelerates INT8, and verify the throughput number with module 8's benchmark before promising it to stakeholders.

In summary

• Dynamic quantization is a single call, quantizes weights offline and activation ranges at runtime, and shines on matrix-multiplication-heavy models like transformers.
• Static quantization quantizes both weights and activations offline using a calibration dataset of 100–500 representative inputs; use per_channel=True for Conv and MatMul.
• The QDQ format wraps original operators with QuantizeLinear / DequantizeLinear nodes; it is portable across runtimes and preferred over QOperator.
• Every quantized model needs the parity check rerun with widened tolerances and top-1 accuracy on the validation set; a weak calibration set is the recurring root cause.

Next module: execution providers, where the same ONNX file routes to CPU, CUDA or TensorRT depending on the host.