Module 3 — Knowledge distillation and compression

Module 2 picked a base — Qwen 2.5 3B Instruct — that can do the ticket task reasonably out of the box. This module is about closing the gap the head-to-head of module 1 revealed (88 % vs 93 % category accuracy) by teaching the small model with the large one as a teacher. The technique is called knowledge distillation and it comes in two flavours that both apply to language models in 2026.

Teacher and student

The mental picture: a large teacher model that solves the task well but is too expensive to ship, and a small student model that does not solve the task well enough but is cheap to run. Distillation transfers behaviour from teacher to student. The student is what you deploy; the teacher is used only during training.

Two things are being transferred, and they are worth naming separately.

The teacher's decisions on real inputs. For every input in a data set, the teacher produces an answer, and the student is trained to reproduce that answer. This is "distillation by outputs" or "hard-label distillation" and, for a chat model, it is the workhorse.

The teacher's underlying reasoning. For inputs the teacher labels along with an intermediate reasoning trace, the student is trained to reproduce both. The reasoning acts as a scaffold and often improves the student's generalisation. This is "chain-of-thought distillation" and it is what pushes Phi-3 and Phi-4 above their weight class.

For classical fine-tuning of small models on classification tasks, the teacher can also expose its soft labels — the full probability distribution over classes — and the student is trained to match that distribution. This is "response-based distillation" in the older sense and is not usually what you use for a chat SLM in 2026; hard-label plus reasoning is the modern default.

Synthetic-data distillation for the ticket assistant

Where do you get labelled tickets? A team of humans annotating 5 000 examples costs weeks of work. A large model annotating the same 5 000 examples costs about a euro. This is the practical form of distillation in most projects: use the large model as a cheap annotator on a large pool of unlabelled real inputs.

The recipe, applied to the ticket assistant:

# distill.py -- one-shot script, runs overnight on 5 000 tickets
import json
from openai import OpenAI  # the teacher, gpt-4o-mini

client = OpenAI()
SYSTEM = "You are a support-desk analyst. Return strict JSON."
USER = """Ticket:
{ticket}

Return a JSON object with:
- category: one of billing, technical, account, other
- summary: two sentences, at most 40 words
- reasoning: two sentences on why you chose this category"""

with open("tickets.jsonl") as f, open("labelled.jsonl", "w") as out:
    for line in f:
        ticket = json.loads(line)["text"]
        rep = client.chat.completions.create(
            model="gpt-4o-mini", temperature=0,
            messages=[{"role":"system","content":SYSTEM},
                      {"role":"user","content":USER.format(ticket=ticket)}],
            response_format={"type":"json_object"})
        out.write(json.dumps({"text": ticket,
                              "label": json.loads(rep.choices[0].message.content)}) + "\n")

Cost: ~5 000 requests, ~600 000 tokens output, ~$0.20 at 2026 prices for gpt-4o-mini. Runtime: an hour or two.

The dataset it produces is what module 7 fine-tunes Qwen 2.5 3B on. The categories the teacher labelled are the ones the student learns; the reasoning it generated becomes a training signal that lifts accuracy by a few points beyond category-only fine-tuning.

The two ways this fails

Teacher errors become student ceilings. If the teacher mis-classifies 5 % of tickets, the student will not learn to do better than 95 %. Distillation transfers behaviour, mistakes included. The mitigation is a small human-audited slice — 200 tickets manually labelled — that lets you both measure the teacher's error rate and cap the student's evaluation to what is achievable.

Distribution drift between synthetic and real. If the teacher was prompted on tickets from month one and the student deploys against tickets from month twelve, the vocabulary and topic mix will have shifted. Refresh the distillation set every quarter, or as often as the ticket topics move.

What distillation buys you

On the running example, a Qwen 2.5 3B fine-tuned by LoRA (module 7) on 5 000 distilled tickets typically closes the gap of module 1 to roughly:

Metric	Base 3B	+5 000-ticket distillation	GPT-4o-mini teacher
Category accuracy	88 %	92 %	93 %
Summary score	3.9	4.2	4.3
Reply score	3.5	4.0	4.1

Not zero-gap. But close enough that the local model becomes a rational choice on the three axes of module 1 — cost, latency and privacy — while giving up ~1 % of accuracy.

Distillation is not compression on its own

A common confusion: distillation is often bundled under "compression" but it does not shrink a model. It teaches a smaller model to imitate a larger one. Actual shrinking — reducing the parameter count of a fixed model — is the job of pruning and quantization, which module 4 covers next. Distillation, pruning and quantization are complementary: you distill from a big teacher, prune what turns out to be dead, and quantize the survivor to fit on the target device.

In summary

• Distillation transfers behaviour from a teacher (large, expensive) to a student (small, cheap), typically via synthetic-data annotation in 2026.
• On the ticket assistant, a large model as a cheap annotator produces ~5 000 labelled examples for a few euros in a few hours, and closes most of the gap from module 1.
• Two failure modes: the student inherits the teacher's mistakes, and synthetic-real drift ages the dataset — audit a small slice, refresh regularly.
• Distillation ≠ compression: it teaches a smaller model, it does not shrink one. The next module, on pruning and quantization, is what actually reduces the memory footprint.

Next: pruning and quantization — from fp16 to int8 to int4, when each step costs quality and when it comes free.