Module 4 — Pruning and quantization

The base model chosen in module 2 weighs about 6 GB in fp16. A work laptop typically has 8 to 16 GB of RAM, most of which is already taken by the browser, the ticketing app, an IDE and half a dozen background services. Making the model fit on the target device is not a bonus — it is the deployment gate. Two techniques do it: pruning removes parameters, quantization shrinks the number of bits per parameter. In practice, in 2026, quantization does most of the work.

Pruning: two flavours

Unstructured pruning. Zero out individual weights below a magnitude threshold. Fifty percent of the weights can be zeroed on many networks with minor quality loss. The catch: unless the runtime supports sparse matmul (most don't for LLMs on CPU), the zeroed weights still cost memory and compute. On real hardware, unstructured pruning of an SLM in 2026 gives you very little.

Structured pruning. Remove whole rows, columns, attention heads or MLP neurons. The runtime sees a smaller dense model, so the compute savings are real. But the quality loss is often steep — twenty percent of heads pruned might cost five points of accuracy on the ticket task — and reversing a bad pruning decision means retraining. Structured pruning is a research technique that has not yet become the default for shipping SLMs.

For most 2026 projects, the honest answer is: you probably do not need pruning. Quantization alone reaches the footprint target without a retraining detour.

Quantization: the workhorse

Quantization stores each parameter with fewer bits. fp16 uses 16, int8 uses 8, int4 uses 4. The parameter count is unchanged; the memory footprint scales linearly with the bit width.

Precision	Bits/param	3B model size	Typical quality loss
fp16	16	~6 GB	0 (baseline)
int8	8	~3 GB	< 0.5 point on most benchmarks
int4 (GPTQ, AWQ, Q4_K_M)	4	~1.8 GB	1–3 points on hard benchmarks, often invisible on narrow tasks
int3, int2	3–2	~1.2–0.8 GB	starts to break down; avoid for shipping

Two things are worth internalising. 8-bit is essentially free: the quality loss on a chat SLM is below the noise of most evaluations. 4-bit is the practical sweet spot: two to three points on hard reasoning, often zero measurable loss on a narrow task like ticket classification.

Two families of 4-bit methods

GPTQ and AWQ are "post-training quantization" methods: take a trained fp16 model, run it on a small calibration set (~128 examples) to figure out where quantization error hurts most, and produce a quantized model without any retraining. This is the technique behind the int4 weights you download from Hugging Face.

QLoRA-style methods train adapters in fp16 on top of a int4 frozen base — but that is module 7's business (fine-tuning), not this module's. Here, we quantize a model we plan to use as-is.

Measuring the quality loss you actually pay

The mistake to avoid: assuming that "quantization costs 2 points on MMLU" translates to "2 points on my task". It might cost zero on your task. It might cost five. Measure on your own evaluation set, always.

The recipe on the ticket assistant, taking the fine-tuned Qwen 2.5 3B from module 3:

# 1) Convert to GGUF at three precisions, using llama.cpp tools
./llama-quantize model.f16.gguf model.Q8_0.gguf Q8_0
./llama-quantize model.f16.gguf model.Q4_K_M.gguf Q4_K_M
./llama-quantize model.f16.gguf model.Q4_0.gguf Q4_0

# 2) Rerun the 200-ticket eval from module 1 on each
python eval.py --model model.f16.gguf     > eval_f16.json
python eval.py --model model.Q8_0.gguf    > eval_q8.json
python eval.py --model model.Q4_K_M.gguf  > eval_q4km.json

Typical readings on this exact task:

Precision	Category accuracy	Summary score	RAM at inference
f16	92.0 %	4.20	~6.2 GB
Q8_0	91.5 %	4.20	~3.3 GB
Q4_K_M	91.5 %	4.18	~2.1 GB
Q4_0	90.5 %	4.10	~1.9 GB

Q4_K_M is essentially free on this narrow task, and it fits comfortably alongside the other applications a support agent runs. That is why it is the format shipped in module 8.

When quantization is enough — and when it is not

Quantization alone is enough when the task is narrow and the base model has some margin above the target accuracy. On the ticket assistant, we have both, and no distillation-plus-quantization is needed beyond what modules 3 and 4 already do.

Quantization alone is not enough when the base model is already at its ceiling on your task in fp16. Then quantization drops you below the threshold and you need to either fine-tune (module 7) or step up to a bigger base (module 2). The measurement above is what tells you which situation you are in — assuming, and not measuring, is what turns "we quantized to int4" into "the assistant regressed and nobody knows why".

In summary

• Pruning removes parameters; on 2026 LLM runtimes it rarely pays off, structured or not. Skip it unless you have specific hardware support.
• Quantization shrinks the bit width per parameter and is the compression workhorse: int8 is essentially free, int4 (Q4_K_M, GPTQ, AWQ) is the sweet spot, int3 and below break down.
• Always measure the quality loss on your own evaluation set — MMLU tells you nothing about ticket classification, and quantization losses are task-dependent.
• On narrow tasks with margin, quantization alone is enough; on tasks already at the ceiling, quantization pushes you below and you need fine-tuning or a bigger base.

Next: efficient runtime formats — GGUF, ONNX, MLX — and how to pick the one that matches the target hardware without repeating the quantization dance for every runtime.