Skip to main content

Module 4 — The local API

Everything the CLI does, the HTTP service does. Anything that speaks HTTP can call Ollama — a Python script, a Node process, LangChain (module 8), a browser inside the office. This module opens that door and shows you the three endpoints that matter, plus the OpenAI-compatible shortcut that lets existing SDKs point at the local runtime without a single code change.

The service, again​

The runtime listens by default on http://127.0.0.1:11434. Loopback binding — deliberate. To open it to the office LAN, set OLLAMA_HOST=0.0.0.0:11434 before starting the service and put a reverse proxy in front (module 8). For now, everything runs on the same machine.

The three native endpoints​

/api/generate is the single-shot completion. Given a prompt and a model, it returns text.

/api/chat is the multi-turn version. Given a list of messages with role and content, it returns the next assistant message. This is what the firm's assistant uses because a conversation with a partner has history.

/api/embeddings is not a language model at all. It turns a piece of text into a dense vector, ready for retrieval (module 9). It uses an embedding model — nomic-embed-text, mxbai-embed-large — not the chat model. Two different tags, two different jobs.

The Python client​

pip install ollama

The official client wraps HTTP for you and mirrors the CLI:

import ollama

response = ollama.chat(
    model="qwen2.5:14b-instruct-q4_K_M",
    messages=[
        {"role": "system", "content": "You are the internal assistant of a French-speaking law firm. Answer factually, cite the clause number when relevant."},
        {"role": "user", "content": "Summarize clause 8 of the attached NDA in three bullet points."},
    ],
    options={"temperature": 0, "num_ctx": 8192, "num_predict": 512},
)

print(response["message"]["content"])

Three things to notice. Generation parameters live under options, not at the top level — mirroring the PARAMETER lines of a Modelfile (module 5). The response is a dict with message.content, plus timing fields (eval_count, eval_duration) identical to the ones --verbose printed. And the model tag is required on every call — the client is stateless, the service holds the loaded weights.

Streaming tokens​

The chat above returns only when generation finishes. For a chat UI, you want the user to see tokens as they arrive:

stream = ollama.chat(
    model="qwen2.5:14b-instruct-q4_K_M",
    messages=[
        {"role": "user", "content": "Draft a two-line response acknowledging receipt of a subpoena."},
    ],
    stream=True,
)

for chunk in stream:
    print(chunk["message"]["content"], end="", flush=True)

Streaming does not change generation speed; it changes perceived speed. The first token appears in tens of milliseconds instead of after the full reply. For the law firm's dashboard, this is what makes a 15-token-per-second local model feel responsive.

Local embeddings​

The same client handles the embedding endpoint:

vec = ollama.embeddings(
    model="nomic-embed-text",
    prompt="Confidentiality clause — parties agree not to disclose...",
)["embedding"]

len(vec)   # 768 for nomic-embed-text

One vector per call. For batches, loop or call /api/embed (the newer multi-input variant) directly. Module 9 wires this into a Chroma index for the firm's PDF archive.

The OpenAI-compatible route​

The most useful integration is the one that requires no code change. Ollama exposes a subset of the OpenAI REST API at http://127.0.0.1:11434/v1/, and it accepts the OpenAI Python client with the local base URL:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:11434/v1",
    api_key="ollama",     # required by the SDK, ignored by Ollama
)

response = client.chat.completions.create(
    model="qwen2.5:14b-instruct-q4_K_M",
    messages=[
        {"role": "user", "content": "List three risks of a fixed-price contract clause."},
    ],
    temperature=0,
    stream=True,
)

for chunk in response:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

This matters. Any code written against openai — a client script, a LangChain integration, a legacy prototype — points at Ollama by changing two lines. The compatibility layer covers chat.completions, completions and embeddings; it does not cover Assistants, files or fine-tuning endpoints, which have no local equivalent.

Errors the API returns​

Three status codes are worth reading, not guessing. 404 on a model name means the tag is not in the local store — pull it first. 500 with out of memory means the model does not fit at the requested num_ctx; drop num_ctx or pick a smaller quantization (module 6). A connection refusal means the service is not running or OLLAMA_HOST binds elsewhere than the client expects.

When to pick the native API over the OpenAI shim

The native endpoint exposes fields the OpenAI shim hides — keep_alive per request, context for stateful continuation, precise timing counters. Reach for the native client when you write a bespoke integration, and reach for the OpenAI shim when you plug into an existing ecosystem.

Summary​

  • The runtime speaks HTTP on 127.0.0.1:11434; any language with an HTTP library can drive it.
  • The three native endpoints are /api/generate, /api/chat, /api/embeddings; the Python ollama client is a thin wrapper.
  • Streaming improves perceived latency without changing generation speed — essential for interactive UIs.
  • The OpenAI-compatible endpoint at /v1/ lets the standard openai SDK point at Ollama by changing only the base URL.

Next module: turning a system prompt, a set of parameters and a base tag into a reproducible Modelfile that carries the firm's house style.