Recap and final exam
Ten modules to go from "just call GPT" to a 2 GB, fine-tuned, offline ticket assistant that matches a hosted API on the target metric at zero per-request cost. Here is the course condensed, then the threads that run through it, then the exam.
The course at a glance
| Module | The essential point |
|---|---|
| 1. Why size is not always the answer | Narrow vs open tasks; cost, latency, privacy as the three axes; first head-to-head 88 % vs 93 % on 200 tickets |
| 2. Landscape of open small models | Phi, Gemma, Qwen, Llama, Mistral — license and language coverage dominate over benchmark scores |
| 3. Knowledge distillation | Large model as cheap annotator; 5 000 labels for ~$0.20 in ~90 min; teacher errors become student ceilings |
| 4. Pruning and quantization | Pruning rarely worth it in 2026; int8 is free, Q4_K_M is the sweet spot; measure on your own eval set |
| 5. Efficient runtime formats | GGUF/llama.cpp for CPU, ONNX for Windows/Intel/mixed-language stacks, MLX for Apple silicon; cross-format smoke test after every conversion |
| 6. Measuring latency and throughput | Report TTFT, TPS-out and wall time; prefill is compute-bound, decode is bandwidth-bound; five rules of a reproducible protocol |
| 7. Cheap fine-tuning of a small model | LoRA r=16, lr=2e-4, 2 epochs; ~50 min RTX 4060, ~$0.25; closes the gap from 88 % to 93 % |
| 8. Local and offline execution | Ollama wraps llama.cpp for the fleet; Modelfile binds template + parameters; ~2.5 GB RAM at inference; pull for versioned updates |
| 9. Privacy through local execution | Local protects the model call, not the store, the logs, the machine, or the update channel; tcpdump test + subject-access-request test |
| 10. End-to-end desktop assistant | Three local processes; matches API on accuracy at 0 % cost; four failure modes with named fixes; route 6–9 % to the large model |
The threads running through the course
Task fit before technique. The single most important decision is the one from module 1: is my task narrow enough that a 3B model can plausibly solve it? Everything else — distillation, quantization, LoRA — makes a marginally better version of an answer that was either possible or impossible from the start. Distillation cannot turn a large-scale open-ended reasoning task into a small-model task. Diagnose task fit before spending a week on training.
Measure on your own data or you are guessing. Every module ends with numbers, and every number came from the same 200-ticket evaluation set. MMLU tells you nothing about ticket classification; a benchmark on someone else's task tells you nothing about yours. The 200 tickets, the bench.py protocol and the reproducible-protocol rules of module 6 are the discipline that lets you compare a local model against a hosted API honestly and detect a regression next quarter.
Local is a systems argument, not just a model argument. The reason to run a 3B model on the workstation is a compound of cost, latency, privacy and the fact that the model fits. Any one of those in isolation would not justify the operational overhead of Ollama, quarterly retraining and fleet updates. Together, on a narrow high-volume task with sensitive data, they compound into a deployment that is genuinely better than the API alternative.
Local does not mean uncontrolled. The management pitch of module 9 and the routing rule of module 10 are the two disciplines that keep a local deployment credible. Being able to name what local does not protect, and being able to route the hard 6–9 % to the big model, are what distinguish an honest deployment from an ideological one.
The final exam
The exam has 40 questions covering all ten modules: narrow versus open tasks and the three decision axes, the five model families with their licenses and language coverage, teacher-student distillation and its failure modes, pruning versus quantization at int8 and int4, GGUF versus ONNX versus MLX with conversion pitfalls, TTFT and TPS-out and the reproducible protocol, LoRA hyperparameters and the fine-tune budget, Ollama Modelfile and memory sizing, what local execution protects and does not, and the end-to-end assistant with its four failure modes and its routing rule.
Several questions present situations to diagnose: a team quantizing to int4 without an eval set and reporting a regression they cannot explain, a distillation pipeline where the student is stuck at the teacher's error rate, a benchmark reading that contradicts a blog post's number for suspicious reasons, a local deployment whose "offline" claim is contradicted by a tcpdump capture, and a routing rule that leaks confidential tickets to the hosted API. Judgement on the pitfalls that surface in production is what gets assessed, not memorised commands.
On success, your certificate of completion is issued immediately; its number can be verified by any third party on the platform.
Re-read the table above and, for each row, ask yourself "what symptom would I see if I got this wrong?". If you can explain why a distillation set ages, why Q4_K_M might cost zero points on your task and three on someone else's, why a TTFT of 40 ms tells you nothing without the prompt length, why a Modelfile without a version tag is a footgun, and why a local model does not exempt you from the DPIA on your ticket store, you are ready. Good luck!
Final exam
Ready to validate this course?
40 questions drawn at random from the course bank · passing score 70% · verifiable PDF certificate issued immediately on success.
Start the examYou need to be signed in to your InSkillML account with an active subscription. You can also start the exam from My courses.