Skip to main content

Module 8 — Local and offline execution

Modules 1 to 7 produced a 2 GB GGUF file that classifies tickets as well as GPT-4o-mini. This module puts it on the fifteen laptops of the actual support team, without a data centre, without a Kubernetes cluster, and without an ongoing operational cost. The point of a small local model is that it is actually deployable — this module is where that promise turns concrete.

Two runtimes, one artefact​

The GGUF file works with both.

Ollama is a friendly wrapper around llama.cpp that runs a local HTTP server, exposes a model registry, and manages updates through a pull command. On Windows and macOS it is a native installer; on Linux, one curl | sh and a systemd service. The programming interface is a small OpenAI-compatible API, which makes it a near-zero-effort integration for any Python code that already talks to OpenAI.

Bare llama.cpp is a single binary. No server, no registry — just ./main --model qwen-ticket.Q4_K_M.gguf --prompt "...". Useful when the target machine cannot run a background service, or when the integration is a subprocess spawn from inside another application.

For the ticket assistant deployment on fifteen agent laptops, we ship Ollama. The extra 30 MB is worth the update mechanism and the HTTP surface.

Registering the model with Ollama​

Ollama expects models declared through a Modelfile — a small textual descriptor that binds the GGUF to a system prompt, a chat template and default parameters.

# Modelfile
FROM ./qwen-ticket.Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

SYSTEM """You are a support-desk analyst.
Return strict JSON with keys category, summary, reply."""

PARAMETER temperature 0
PARAMETER num_ctx 2048
PARAMETER stop "<|im_end|>"

Then, on each agent laptop:

ollama create ticket-assistant -f Modelfile
ollama run ticket-assistant "..."   # smoke test

The model is now available on http://localhost:11434/v1 with the OpenAI-compatible schema.

Memory sizing​

The rule of thumb for a Q4 3B model:

  • Weights on disk: ~2 GB (the GGUF file itself).
  • Weights in RAM at runtime: ~2 GB, memory-mapped.
  • KV cache: ~50 MB per 1 000 tokens of context. At 2k context, ~100 MB.
  • Ollama server overhead: ~200 MB.
  • Total: ~2.5 GB of RAM used while the model is loaded.

That budget fits on a 16 GB agent laptop while the browser, the ticketing app and Slack are running. On 8 GB machines it does not — those need to unload the browser or step down to a 1.5B model (Qwen 2.5 1.5B fine-tuned the same way loses about 3 points of category accuracy).

The KV cache grows linearly with the conversation length. For a single-turn classification like ours, it stays trivial; if a future feature turns the assistant into a multi-turn chat, budget accordingly.

Model updates​

Small models improve every quarter. The Ollama registry lets you push a new version to the fleet without visiting fifteen laptops:

# On the build machine, after a fresh distillation + LoRA cycle
ollama create ticket-assistant:v3 -f Modelfile.v3
ollama push my-org/ticket-assistant:v3

# In the agents' start-up script (once, or as a scheduled task):
ollama pull my-org/ticket-assistant:v3

The pull downloads only the differing tensors — typically ~200 MB for a small adapter change, ~2 GB for a full retrain. Schedule the pull outside business hours, log the resulting version to a central sink so you know which laptop runs which model. Version pinning matters: a user reporting a regression must have a version to attribute it to.

Integrating with the ticket tool​

The ticket tool exists — it is a web application the agent already uses. The assistant is a small side panel that reads the current ticket text and shows the suggested category, summary and reply.

# assistant_service.py -- runs as a tiny local FastAPI service
from openai import OpenAI
from fastapi import FastAPI
from pydantic import BaseModel
import json

llm = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
app = FastAPI()

class Ticket(BaseModel):
    text: str

@app.post("/analyse")
def analyse(t: Ticket):
    rep = llm.chat.completions.create(
        model="ticket-assistant",
        messages=[{"role":"user","content": t.text}],
        response_format={"type":"json_object"},
        temperature=0)
    return json.loads(rep.choices[0].message.content)

The web tool calls POST http://localhost:8000/analyse when the agent opens a ticket. Response time on the target hardware: ~1.5 s end to end, of which ~1.3 s is the model. Perfectly usable in an interactive workflow.

What "offline" actually means​

"Offline" here means no cloud dependency during operation. It does not mean:

  • No installation step: you did download the model once from Ollama or from an internal artefact server.
  • No maintenance step: model updates require the machine to reach the update source at least occasionally.
  • No logging: the tool still logs interactions to your local database, which lives on a server. Local inference does not remove that log.

The clarity of the "offline" story matters — module 9 pushes on it. A support agent working on a train can still classify tickets they cached earlier; that is the win. Claiming "nothing ever leaves the laptop" without qualification is the mistake that turns a good privacy story into a broken one.

In summary​

  • Ollama wraps llama.cpp as a local HTTP server with a model registry — the right default for a fleet of workstations; bare llama.cpp is the option for machines that cannot run a background service.
  • A Modelfile binds the GGUF to a chat template, system prompt and default parameters — ollama create on each laptop and the model is on localhost:11434.
  • Memory budget for Q4 3B: ~2.5 GB of RAM including KV cache and server overhead — comfortable on 16 GB machines, tight on 8 GB.
  • Updates via ollama pull with version pinning — differential downloads keep the bandwidth cost low and traceable regressions require versioned deploys.
  • Integration is a small local FastAPI service the existing web tool calls — end-to-end latency ~1.5 s on the target hardware.

Next: the privacy story pushed honestly — what local execution protects, what it does not, and how to make the argument to a management team that expects a one-line answer.