Module 1 — Why size is not always the answer
The default reflex in 2026 is still "pipe it to GPT-4o". It usually works. It also usually costs more, runs slower and leaks more data than the same task deserves. This module is about the taste for choosing between a 1 to 4 billion parameter open model on the workstation and a hundred-billion API model in the cloud — and about the first honest measurement of that trade-off on the running example.
The running example is a support-ticket assistant for a small team of agents. It classifies each incoming ticket into {billing, technical, account, other}, writes a two-sentence summary and drafts a suggested reply. We will measure the same task with a small local model and a large API model on the same 200 tickets, and read the numbers together.
Narrow tasks versus open tasks
Not all tasks are equal in front of a small model.
Narrow tasks have a small output space, a clear success criterion and stable phrasing. Classifying a ticket into four buckets, extracting a two-sentence summary, deciding whether a message is polite — these tasks live on a small manifold of the input space and a well-trained 3B model reaches human-level accuracy on them after some fine-tuning. The extra capacity of a 200B model does not translate into extra accuracy; it translates into extra dollars.
Open tasks have an unbounded output space, no single right answer and reward broad world knowledge. Writing a legal brief on an obscure jurisdiction, translating literary prose, doing multi-step mathematical reasoning across a fresh paper — the gap between a 3B and a 200B model here is real and often decisive. No amount of fine-tuning a small model produces the wide-knowledge behaviour of a large one.
The first question when someone proposes a small model is: is my task narrow or open? Ticket classification is narrow. Legal drafting is open. Most support-desk work sits comfortably in the narrow half.
The three axes that decide
Beyond task fit, three axes push the decision toward local or toward API.
Cost per request. A large API call costs a few tenths of a cent; a self-hosted 3B model on hardware you already own costs the electricity to run it, which is fractions of a cent when the machine is on anyway. That difference is trivial per call and enormous at 50 000 tickets a day.
Latency and dependency. A local model answers in 200 to 800 ms on a laptop CPU and does not depend on the network. An API call is 400 to 1 500 ms round-trip plus every outage of the provider between you and it. For a call-centre agent whose workstation must respond in real time, that difference is felt.
Privacy and regulation. Data that never leaves the workstation cannot be logged by a third party. For tickets that contain names, addresses and payment details, that is not a preference — it is what makes the assistant deployable at all in some jurisdictions. Module 9 pushes on the limits of this argument.
The first head-to-head on the running example
Take 200 real tickets — anonymised — with human labels for category, summary quality (1 to 5) and reply usefulness (1 to 5). Run the same prompt against qwen2.5:3b-instruct on the workstation via Ollama, and against gpt-4o-mini via the API. Measure category accuracy, average summary score and average reply score.
Typical numbers on this exact setup (order of magnitude, not guarantee):
| Metric | Qwen 2.5 3B, local | GPT-4o-mini, API |
|---|---|---|
| Category accuracy | 88 % | 93 % |
| Summary score, mean | 3.9 | 4.3 |
| Reply score, mean | 3.5 | 4.1 |
| Latency, p50 | 380 ms | 720 ms |
| Cost per 1 000 tickets | ~0 (electricity) | ~$0.60 |
The API wins on quality. The local model wins on latency and cost. The choice is not obvious — and it will move again once module 7 fine-tunes the small model on the same 200 tickets.
Reading the trade-off honestly
Three habits keep the comparison intellectually honest.
Same prompt on both sides. A prompt tuned for one model that the other has never seen inflates the winner. The prompt used in the table above was written once and applied verbatim.
Same evaluation set on both sides. Not a hand-picked demo — the same 200 anonymised tickets, scored by the same rubric. Module 6 turns this into a reproducible protocol.
Report the numbers you saw, not the ones you hoped for. A small model that reaches 88 % category accuracy on tickets is a real success; pretending it reached 93 % helps nobody, and the day production disagrees is the day the project loses credibility.
In summary
- Narrow tasks with a small output space and stable phrasing are where small models earn their keep; open tasks with unbounded outputs still favour large models.
- The three decision axes are cost per request, latency and network independence, and privacy and regulation — a task that wins on two of them is a candidate for a small local model.
- The first honest measurement is a head-to-head on the same evaluation set with the same prompt; on the ticket assistant, a 3B local model reaches ~88 % category accuracy against 93 % for a large API model, and wins clearly on latency and cost.
- The choice is not "small versus large" as an ideology — it is a per-task, per-workload arithmetic that this course will teach you to run.
Next: the open small model landscape — Phi, Gemma, Qwen, Llama, Mistral — with their licenses and language coverage, so you can pick the right base before spending a week fine-tuning it.