Module 10 — Project: a specialized desktop assistant
Nine modules of ingredients. This module assembles them into the assistant that the running example promised — a specialized, offline, on-device support-ticket triager that fifteen agents actually use in production. It is also the module where we measure the full system against the API baseline, compute a one-year cost, and — importantly — draw the line where a ticket should still be routed to the big model.
The full system, on one diagram
┌───────────────────────────────────────────────────────────────┐
│ Agent laptop (Windows or macOS, 16 GB RAM) │
│ │
│ ticketing-web-app ───POST───▶ assistant-service (FastAPI) │
│ │ │
│ ▼ │
│ ollama server (:11434) │
│ │ │
│ ▼ │
│ ticket-assistant:v3 (GGUF Q4_K_M) │
│ fine-tuned Qwen 2.5 3B, 2.0 GB │
└───────────────────────────────────────────────────────────────┘
│
▼
local structured log (JSONL), PII-redacted
│
▼
nightly push to central sink
Three processes on the laptop, no network dependency at inference time, one artefact (qwen-ticket.Q4_K_M.gguf) that carries the entire task-specific behaviour.
The two hundred tickets, one more time
The evaluation set that has followed us since module 1 is the honest yardstick. Same 200 tickets, same rubric, same protocol from module 6.
| Configuration | Category acc. | Summary | Reply | p50 latency | Cost / 1 000 tickets |
|---|---|---|---|---|---|
Module 1: Qwen 3B base, Q4_K_M, local | 88 % | 3.9 | 3.5 | 380 ms | ~$0 |
Module 7: + LoRA on 5 000 distilled, Q4_K_M, local | 93 % | 4.2 | 4.0 | 400 ms | ~$0 |
| Module 1: GPT-4o-mini via API | 93 % | 4.3 | 4.1 | 720 ms | ~$0.60 |
| Reference: GPT-4o via API | 96 % | 4.5 | 4.4 | 1 100 ms | ~$8.00 |
The story in one sentence: the fine-tuned 3B local model matches the small hosted API on the target metric, at zero per-request cost, half the latency, and with the privacy story of module 9. The big hosted API is measurably better; whether the extra three points of accuracy justify its cost is a business question, not a technical one.
The one-year cost picture
Fifteen agents, five ticket assistants per agent per hour, eight hours a day, 220 working days: 132 000 tickets per year.
| Approach | One-year cost |
|---|---|
| GPT-4o for every ticket | ~$1 060 |
| GPT-4o-mini for every ticket | ~$80 |
| Local Qwen 3B fine-tuned, on existing agent laptops | **~0.50 and ~2 hours of one engineer's time per quarterly refresh |
The API costs are not enormous — this is a small deployment. On a hundred agents, the API bill scales linearly and the local approach stays flat. The real reason for local here is not the bill; it is the privacy and latency story the API cannot match at any price.
The failure modes the assistant must handle
Four failure modes with named fixes; every one appeared in staging.
The model is not sure. The prompt asks for a confidence field in the JSON. When it comes back below 0.7, the UI shows the suggestion but skips the auto-apply — the agent still sees it, does not have it applied without review.
The model refuses on a garbled input. A ticket that is a single emoji or a raw stack trace deserves a graceful "no confident classification" and a fallback to manual triage. Wrap the FastAPI endpoint with an exception handler that returns an explicit fallback rather than crashing.
Ollama went missing. The Ollama server crashed, or the service failed to start after reboot. The FastAPI service returns a 503 to the ticket tool, the tool falls back to the plain triage screen. The user experience degrades cleanly rather than the ticket tool freezing. A systemd/launchd watchdog restarts Ollama.
The model version drifted. After a pull of v4, the classification accuracy on a canary set drops two points. The Modelfile pins the version; the operator can ollama run ticket-assistant:v3 to roll back in one command. Version pinning is what turns a scary autoupdate into a reversible operation.
When to route to the big model anyway
This is the discipline the local-model advocate must have. Some tickets are worth $0.008 in GPT-4o dollars because the alternative is a wrong answer that costs a customer relationship.
The routing rule shipped in production, in three lines:
if resp["confidence"] < 0.6 or len(ticket.text) > 3000 or is_regulated(ticket):
return route_to_large_model(ticket)
return resp
Low confidence: the model itself flagged that it does not know. Escalate.
Very long inputs: past ~3 000 tokens, small-model quality falls off; the context window of the deployed model is 2k, and truncation loses information. Route the rare very long ticket to the big model.
Regulated content: for tickets flagged as complaints, legal or safety by the ticket tool's own metadata, route to the large model — the cost of misclassification is real and the volume is low. The privacy story still holds because those tickets are already reviewed by humans; the model call is a second opinion, not the primary decision.
Typical routing rate in production: 6 to 9 % of tickets go to the big model. That keeps the one-year API cost around $10–15 for the whole team and preserves the accuracy where it matters most.
Shipping habits
Three habits from modules 6, 8 and 9 make the deployment survive month two.
Rerun the benchmark on every model change. Every quarterly refresh runs bench.py (module 6) and eval.py on the 200 tickets before the new model is pushed to the fleet. A regression in latency or accuracy blocks the push.
Log the metrics locally, ship them centrally after redaction. Category, confidence, latency, model version — never the ticket text. Central dashboards need the aggregates, not the payloads.
Ship behind a rollout flag. New models roll to two "beta" agents for a week, then to the rest. The two agents are named volunteers who know they are on the beta, so their reports are useful signal rather than complaints in a shared channel.
In summary
- The full system is three local processes — a small FastAPI wrapper, the Ollama server and the fine-tuned Qwen 3B in GGUF Q4 — with no cloud dependency at inference time.
- On 200 real tickets, the fine-tuned local model matches GPT-4o-mini on the target metric and beats it on latency, cost and privacy; GPT-4o remains 3 points better on accuracy at 13× the cost.
- Four failure modes with named fixes: low confidence (no auto-apply), garbled input (graceful fallback), Ollama down (503 + watchdog), version drift (pin the tag,
run ticket-assistant:v3to roll back). - Route to the large model on low confidence, very long inputs and regulated content — typically 6–9 % of the volume, preserving accuracy where the cost of error is real.
- The three shipping habits — rerun the benchmark, log-and-redact, rollout flag — are what turns a demo into a system that survives.
Next: the recap module and the 40-question exam that turns nine modules and a project into a verifiable certificate.