Skip to main content

Module 8 — Integrating with existing applications

The runtime is installed (module 1), the right model is pulled and tuned (modules 2 to 6), the GPU is doing its job (module 7). This module plugs Ollama into the tools the firm already uses — a Python case-management helper, LangChain for the retrieval pipeline of module 9, and Open WebUI as the chat interface — and covers the one topic no article about local models discusses seriously enough: safely opening the service to other machines on the network.

Calling from a business script​

The simplest integration is a Python function that the case-management tool already imports. It receives a piece of contract text and a question and returns a factual paragraph:

from ollama import Client

client = Client(host="http://127.0.0.1:11434")

def ask_firm_assistant(context: str, question: str) -> str:
    resp = client.chat(
        model="firm-fr",   # the derived tag from module 5
        messages=[
            {"role": "system", "content": "Answer only from the provided context. If the answer is not there, say so."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
        ],
        options={"temperature": 0, "num_ctx": 8192, "num_predict": 512},
    )
    return resp["message"]["content"]

Two decisions in these ten lines are the ones production code should never move around. The temperature is fixed at zero because a case-management tool re-runs the same question after a re-save and users demand the same answer. The system prompt narrows the model's job to "answer from the given context" — the retrieval layer (module 9) is what feeds context, and this line closes the door on the model inventing anything outside it.

Calling from LangChain​

Course 26 taught LangChain in depth. The one-line integration is:

from langchain_ollama import ChatOllama

llm = ChatOllama(
    model="firm-fr",
    base_url="http://127.0.0.1:11434",
    temperature=0,
    num_ctx=8192,
)

Every downstream primitive from that course — chains, retrievers, memory, tools, agents — works against this llm unchanged. The Runnable interface is what makes swapping providers a one-line change: a chain written in a prototype against ChatOpenAI moves to ChatOllama by replacing the constructor, and the tests keep passing.

For embeddings, the companion class:

from langchain_ollama import OllamaEmbeddings

embed = OllamaEmbeddings(model="nomic-embed-text", base_url="http://127.0.0.1:11434")

Module 9 assembles both into a full offline RAG pipeline on the firm's PDFs.

Adding Open WebUI​

Open WebUI is a self-hosted chat interface that speaks the OpenAI API and can be pointed at Ollama's compatibility endpoint. It gives associates a browser-based ChatGPT-style UI that never talks to any cloud. The recommended install is Docker:

docker run -d --name openwebui \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OPENAI_API_BASE_URL=http://host.docker.internal:11434/v1 \
  -e OPENAI_API_KEY=ollama \
  -v openwebui:/app/backend/data \
  --restart always \
  ghcr.io/open-webui/open-webui:main

The associates browse to http://server.firm.local:3000, sign in with their firm email, pick firm-fr from the model dropdown, and start chatting. History, user accounts, per-user rate limits and RAG uploads live in the volume openwebui. The Ollama runtime knows nothing about users — Open WebUI is the layer that maps a browser session to a request.

Exposing the runtime to the LAN​

By default, Ollama binds to 127.0.0.1:11434 — no other machine can reach it. To let Open WebUI on the server serve associates on their laptops, one option is to run the container on the same server as Ollama (as above, via host.docker.internal) and only expose the WebUI's port 3000. That is the recommended shape: the runtime stays private, the UI does the network work.

If for some reason another host must reach Ollama directly (a script on a different machine, a second WebUI on a workstation), open the port carefully. Three steps in order:

  1. Set OLLAMA_HOST=0.0.0.0:11434 on the service (via the environment on Linux, the tray settings on Windows), restart it.
  2. Restrict access at the firewall: only the office subnet, never the internet.
  3. Put a reverse proxy with authentication in front — Caddy, nginx or Traefik. A minimal Caddyfile:
ollama.firm.local {
    reverse_proxy 127.0.0.1:11434
    basicauth {
        associates $2a$14$...bcrypt-hash...
    }
}

Ollama itself has no built-in authentication. Exposing port 11434 to a public network without a proxy is equivalent to publishing your inference budget for anyone to spend. It is the single most common mistake in first deployments.

Timeouts, retries and keep-alive​

A production integration handles three timing realities. The first request after a model load takes seconds; subsequent ones are fast. A client should set the HTTP timeout to at least 120 seconds on the first call. OLLAMA_KEEP_ALIVE=30m avoids reloading between two spaced requests. On the client side, retry a 503 (model still loading) with a short back-off, and surface a 500 out of memory as a hard error instead of retrying — the second try will fail the same way.

No authentication in the box

Neither the native API nor the OpenAI-compatible endpoint checks credentials. Any host that can reach the port can use the model, and can list, pull and delete tags. Always front the service with a proxy that authenticates, and never leave it on 0.0.0.0 without a firewall rule limiting the subnet.

Summary​

  • A Python client with Client(host=...) covers the simplest business integration; pin temperature and num_ctx at the call site.
  • LangChain exposes ChatOllama and OllamaEmbeddings; a chain written against another provider swaps to Ollama by replacing the constructor.
  • Open WebUI in Docker gives associates a browser-based chat that speaks the OpenAI-compatible endpoint and holds user accounts.
  • Ollama has no authentication; expose the port only through a reverse proxy with basic auth and a firewall rule.

Next module: the retrieval layer — local embeddings, a local vector index, and a fully offline document Q&A over the firm's archive.