Module 9 — Local document question answering

The firm has fifteen years of contracts, memos and court decisions in a shared drive. Any real assistant has to reach into that archive — not by uploading it somewhere, but by indexing it on the same machine that runs the model. This module assembles the retrieval layer from local pieces only, on the pattern of the retrieval-augmented generation course (course 18), and shows what changes when every step of the pipeline is offline.

The four moving parts, all local

A local RAG chain has four components. Each is a choice; each has a local option.

• Loader — reads a PDF or a Word file into text. pypdf, pdfplumber, or docling for higher-fidelity layout. All install as Python packages, none call an external service.
• Splitter — breaks the text into chunks small enough to embed and large enough to carry meaning. LangChain's RecursiveCharacterTextSplitter on paragraph then sentence boundaries is the working default.
• Embedding model — turns each chunk into a vector. nomic-embed-text (768 dimensions, 137 MB, strong on European languages) is the default; mxbai-embed-large (1024 dimensions, larger and slower) is a step up when precision matters. Both are pulled through Ollama and served on the same endpoint as the chat model.
• Vector store — indexes the vectors and returns nearest neighbours for a query vector. Chroma is a Python-native store that lives in a folder on disk. FAISS is faster on very large corpora but has no built-in persistence. Chroma is the recommended starting point for the firm's fifteen years of files.

Indexing the archive

The one-time script that turns data/archive/*.pdf into a queryable index:

from pathlib import Path
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings
from langchain_chroma import Chroma

# 1. Load — PDF by PDF, preserving page and source metadata.
docs = []
for pdf in Path("data/archive").rglob("*.pdf"):
    for page in PyPDFLoader(str(pdf)).load():
        page.metadata["source_name"] = pdf.name
        docs.append(page)

# 2. Split — 800-character chunks, 100-character overlap.
splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
chunks = splitter.split_documents(docs)

# 3. Embed — locally, through Ollama's endpoint.
embed = OllamaEmbeddings(model="nomic-embed-text", base_url="http://127.0.0.1:11434")

# 4. Persist — Chroma writes to disk, ready to reload without reindexing.
store = Chroma.from_documents(
    chunks,
    embedding=embed,
    persist_directory="data/index-firm",
)

On a modest machine, embedding 5,000 chunks with nomic-embed-text takes ten to twenty minutes. This is a one-time cost — subsequent runs open the folder and query it directly:

store = Chroma(persist_directory="data/index-firm", embedding_function=embed)

Two habits pay back. Store source_name and page in the chunk metadata at load time (they are used for citations). And reindex incrementally by tracking the hash of each PDF — reprocessing the whole archive because one file changed is a waste every night.

The retrieval chain

The chain is exactly the shape of the RAG course, with ChatOllama in place of a hosted LLM:

from langchain_ollama import ChatOllama
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

llm = ChatOllama(model="firm-fr", num_ctx=8192, temperature=0)

retriever = store.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={"k": 4, "score_threshold": 0.35},
)

prompt = ChatPromptTemplate.from_template(
    """Answer only from the following context. Cite the source name and page number.
If the answer is not in the context, reply: "The archive does not cover this question."

Context:
{context}

Question: {question}
"""
)

def format_docs(docs):
    return "\n\n".join(f"[{d.metadata['source_name']} p.{d.metadata.get('page', '?')}] {d.page_content}" for d in docs)

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

print(chain.invoke("What is the standard confidentiality window for a mutual NDA?"))

The score_threshold is the guardrail against confident hallucinations on out-of-corpus questions — a lesson from the RAG course transferred verbatim. The format_docs function injects the source marker into the context, so the model cites tokens it actually sees rather than fabricating page numbers from thin air.

Quality in the front-end language

Two facts govern quality in French, the firm's main working language.

First, the embedding model matters as much as the chat model. A retriever that picks the wrong four chunks cannot be saved by a smarter generator downstream. nomic-embed-text is trained on a corpus that includes French; all-MiniLM-L6-v2 (an older common default) is much weaker on it. Test with ten questions taken from real cases and count the correct top-4 hits per retriever.

Second, the chat model must be strong in the front-end language too. qwen2.5 and llama3.1 handle French competently at 7-B and above; phi3-mini handles it noticeably worse. This is the same rule as module 2 — match the tag to the audience, measure on the audience's questions.

Everything, on one machine

Compare the local pipeline to a hosted equivalent. On a hosted RAG service, chunks travel to an embedding API, vectors travel to a hosted store, queries travel to a hosted LLM. Three hops, three counterparties, three logs that record the firm's clauses. On the local pipeline, PDFs, vectors, queries and answers stay on the machine that runs Ollama. This is what makes the assistant deployable for the firm without a data-processing agreement, and without a spreadsheet of "what leaves the office".

The precision hit versus a hosted setup

A local q4_K_M model with a small embedder is meaningfully behind the state of the art on hard reasoning. On a corpus where precision beats confidentiality — say, a public research assistant — a hosted stack wins. On the firm's archive, where confidentiality is non-negotiable, the local trade-off is the right one. Course 24 (ethics) frames how to state that choice to clients.

Summary

• All four RAG components — loader, splitter, embedder, vector store — have local options; nothing leaves the machine.
• OllamaEmbeddings on nomic-embed-text plus Chroma on disk is the working default; index once, reuse forever.
• The chain shape is identical to a hosted RAG chain (course 18) with ChatOllama in place of a hosted model.
• Quality in the front-end language depends on both the embedder and the chat model; test on real questions from the audience.

Next module: an honest look at what a local model still cannot do, and where the hosted API remains the right answer.