Module 10 — Limits of local execution

Nine modules argued for a local assistant. This one draws the line: what a local Ollama deployment does poorly, what it cannot do at all, and the specific scenarios that eventually justify a dedicated inference server or a hosted API. Being clear-eyed here is what keeps the firm from oversizing today and undersizing tomorrow.

Quality versus hosted frontier models

A q4_K_M 14-B model on a laptop is competent. A hosted frontier model — the current top-tier assistants offered by the big labs — is measurably better on hard reasoning, multi-step math, long-context reasoning past 100k tokens, and code generation. The gap narrows every year and is negligible on many everyday tasks, but on the hardest 10 to 20 percent of the firm's queries it is real. Two coping strategies:

• Route by difficulty. Simple summaries and structured extractions run on the local model; a "hard question" toggle explicitly sends a redacted prompt to a hosted API, with the partner's approval on record.
• Raise the local ceiling. A 70-B model at q5_K_M on a workstation with 96 GB of RAM and a serious GPU closes most of the gap. It costs more up front, saves recurring API bills, and stays on-premise.

Do not pretend the gap is zero, and do not pretend it is unbridgeable. It is a decision to make per class of query.

Concurrency: several users at the same time

Ollama serves one request per loaded model at a time by default. Two associates hitting the same endpoint queue behind each other, and the second waits. Three settings shape concurrency:

Variable	Default	What it changes
OLLAMA_NUM_PARALLEL	1	Concurrent requests per loaded model
OLLAMA_MAX_LOADED_MODELS	1 on tight RAM	How many distinct models can be resident at once
OLLAMA_MAX_QUEUE	512	Requests waiting before the service returns 429

Setting OLLAMA_NUM_PARALLEL=4 lets four associates share a model, at the cost of a KV cache multiplied by four in RAM and a per-request throughput cut. Setting OLLAMA_MAX_LOADED_MODELS=2 lets a chat model and an embedding model coexist without churn. Both are the right defaults for the office server; both are wrong on a partner's MacBook, where the single seat cannot afford to lose RAM to a model no one is using.

For five to ten concurrent associates, a modest GPU server (RTX 4070 or 4080 class) with these settings is enough. Beyond that — thirty active seats, a hundred embedding calls per second from a background pipeline — a single machine is not the right answer any more.

Updates: keeping the runtime and the tags current

Ollama ships new runtime versions monthly. New tags appear on the library daily. Left alone, a deployment drifts:

• Runtime updates patch security fixes in llama.cpp and add support for new model families. Update the service quarterly at least, and read the release notes for breaking changes.
• Model updates replace the tag under the same name. qwen2.5:14b-instruct today is not the same set of bytes as six months ago. In production, pin the digest (ollama pull qwen2.5:14b-instruct@sha256:...) or a copy in a private registry (module 8).
• Modelfile updates should live in git. Every change to a SYSTEM prompt or a PARAMETER default is a commit with a reason, not a manual ollama create on the server.

The alternative — untracked updates, no digests — is the source of "the assistant answered differently yesterday, why?", and there is no support ticket to open.

Security of the host

The runtime and the models sit on the same machine as the office's files. That machine's security is the assistant's security. Three specifics beyond the usual hygiene:

• Reverse-proxy authentication and a firewall rule (module 8) are non-negotiable. An open port 11434 on the office LAN is functionally a key to the model to anyone on wifi.
• Model tags on the library are unsigned by default. A rogue tag with a poisoned Modelfile could set a SYSTEM prompt that leaks data through the answer. In practice this is a low-probability risk; in principle, curate the tags the firm uses and pull them through a private registry so the fleet cannot pull anything else.
• The RAG index (module 9) is the firm's archive in another format. Treat its folder with the same access controls as the source PDFs. A backup that copies it off-machine is a backup that leaves the office — encrypt it or exclude it.

When to move off a single machine

Three thresholds trigger a re-architecture:

• Concurrent users past what one machine serves comfortably. Add a second Ollama host behind a small load balancer (nginx upstream) and route by round-robin; the runtimes need no coordination because they are stateless.
• A model too large for any single machine the office can host. A 405-B parameter model does not fit on a $10,000 workstation. If a class of question genuinely requires it, a hosted API is the honest choice for that class only.
• A latency or throughput budget the hardware cannot meet. Ten tokens per second is fine for a partner reading answers; three hundred for a batch pipeline is not. The batch pipeline moves to a dedicated inference server (vLLM, TGI, TensorRT-LLM) that squeezes more from the same GPUs, or to a hosted API billed per token.

When to move to a hosted API — and how

Two use cases favour the hosted API even for a confidentiality-sensitive firm:

• A one-off hard question, redacted of names and numbers before sending. The bill is a few cents, the answer is measurably better, and the audit log shows what left.
• A workload that dwarfs the fleet: bulk translation of a decade of contracts, a corpus-wide re-labelling. The hosted API amortises the burst; the local runtime resumes for daily use.

The mistake is a binary choice — either everything is local or everything is hosted. In practice, the firm's assistant runs local for 95 percent of queries and reaches for a hosted API for the 5 percent that genuinely benefits, under an explicit policy.

A checklist before signing off on a local deployment

Twelve boxes to tick before the assistant is production-ready: model store on a large drive, OLLAMA_HOST bound only where intended, reverse proxy with authentication, firewall rule to the office subnet, Modelfile in git with a pinned base digest, num_ctx set deliberately per caller, RAG index encrypted at rest, backup policy that does not exfiltrate the index, OLLAMA_KEEP_ALIVE chosen for the RAM budget, OLLAMA_NUM_PARALLEL set for the number of users, monthly runtime updates on the calendar, and a short written policy for the "hard question" fallback.

Summary

• On the hardest 10 to 20 percent of queries, a hosted frontier model still wins; route by difficulty or raise the local ceiling with a bigger workstation.
• Concurrency is set with OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS; a modest GPU server handles five to ten associates comfortably.
• Pin runtime versions, model digests and Modelfile diffs in git; unpinned deployments drift silently.
• Move to a dedicated inference server or a hosted API when concurrent users, model size or throughput cross what a single machine can serve.

Course finished. The next stop is the recap and the 40-question exam.