Module 9 — Caching and cost control

The pipeline of modules 4 to 8 works. It also bills every question at full price. This module explains where the money actually goes, how caching removes 40 to 80 % of it without changing quality, and what logging is worth keeping to spot regressions before they show up in an invoice.

Where the money and the milliseconds go

A single answered question sends work through five components. Attaching costs is the first act of engineering; the numbers below use round figures for a mid-sized open stack, but the shares are stable across setups.

Component	Typical share of latency	Typical share of cost
Query embedding	5 %	1 %
Vector search	10 %	~0 %
BM25 search	5 %	~0 %
Cross-encoder rerank	25 %	5 %
Language model generation	55 %	94 %

The language model dominates cost by an order of magnitude and dominates latency by a large margin. Almost everything worth caching is on the LLM side, and almost everything worth measuring first is upstream, because the upstream mistakes are what make the LLM read too many tokens.

Breaking down the cost of one question

Model billing is per input token and per output token, at different rates. A defensible formula for one answered question:

$$\text{cost}(q) = c_{\text{in}} \cdot n_{\text{prompt}}(q) + c_{\text{out}} \cdot n_{\text{answer}}(q) + c_{\text{embed}} \cdot n_{\text{query tokens}}(q) + c_{\text{rerank}}$$

Track each term separately in your logs; the ratios tell you which lever to pull. A pipeline whose prompts have crept from 1500 to 3500 tokens without anyone noticing is a real thing, and it doubles the bill overnight.

def measure_cost(run: dict, prices: dict) -> dict:
    return {
        "prompt_tokens":  run["prompt_tokens"],
        "answer_tokens":  run["answer_tokens"],
        "query_tokens":   run["query_tokens"],
        "cost_llm_in":    prices["llm_in"]  * run["prompt_tokens"] / 1000,
        "cost_llm_out":   prices["llm_out"] * run["answer_tokens"] / 1000,
        "cost_embed":     prices["embed"]   * run["query_tokens"]  / 1000,
        "cost_rerank":    prices["rerank"],
    }

Store this dictionary alongside every log entry from module 8's log_run. Once a week, plot each component over time. Regressions appear as steps in the plot, and their causes are always upstream — a chunker change that inflated passages, a reranker that started keeping more candidates, a prompt that grew instructions.

Cache the embedding of every unique chunk

The embedding of a chunk is a pure function of its text and the model. If neither changes, the result is the same, and computing it twice is waste. During ingestion of a 300-document corpus, a small change (fixing a typo in the extractor) can reprocess thousands of chunks whose text is identical to yesterday's. Cache them.

import hashlib
import sqlite3

def cache_key(text: str, model_name: str) -> str:
    h = hashlib.sha256(f"{model_name}||{text}".encode("utf-8")).hexdigest()
    return h

class EmbeddingCache:
    def __init__(self, path: str = "cache/embeddings.sqlite"):
        self.conn = sqlite3.connect(path)
        self.conn.execute(
            "CREATE TABLE IF NOT EXISTS emb (k TEXT PRIMARY KEY, v BLOB)"
        )

    def get(self, key: str) -> bytes | None:
        row = self.conn.execute("SELECT v FROM emb WHERE k = ?", (key,)).fetchone()
        return row[0] if row else None

    def put(self, key: str, value: bytes) -> None:
        self.conn.execute("INSERT OR REPLACE INTO emb VALUES (?, ?)", (key, value))
        self.conn.commit()

On a corpus that grows by 5 to 10 % per week, this cache brings reindex cost down by 90 %. The key includes the model name — if you swap the embedding model, cache entries invalidate themselves without any manual invalidation step.

Cache the answer to frequent questions

Users repeat themselves. In a company assistant, the top 20 questions typically account for 30 to 50 % of the traffic. Serving them from a cache hits them at zero LLM cost and near-zero latency, which is the biggest win in this module.

The cache key is not just the question text: two users asking the same question in different departments must get different answers if their permissions differ. Include the user context that affects retrieval in the key.

def answer_cache_key(question: str, user_context: dict) -> str:
    ctx = "|".join(f"{k}={user_context[k]}" for k in sorted(user_context))
    return hashlib.sha256((question.strip().lower() + "||" + ctx).encode()).hexdigest()

def answer_cached(question: str, user_context: dict, ttl_seconds: int = 3600) -> dict:
    key = answer_cache_key(question, user_context)
    hit = answer_cache.get(key, max_age=ttl_seconds)
    if hit is not None:
        return {**hit, "from_cache": True}
    fresh = answer(question, user_context)
    answer_cache.put(key, fresh)
    return {**fresh, "from_cache": False}

The TTL is the safety valve. A source document was revised at 10:00; a cache with a one-day TTL will serve the old answer until tomorrow. For a corpus that changes weekly, 1 to 6 hours is a good range. Also plumb a manual invalidate-by-source call: when a document is reindexed, drop every cached answer whose citations include it.

Prompt caching: the underused free win

Modern LLM APIs offer prompt caching: the provider stores the prefix of prompts you send repeatedly (a system prompt, a set of few-shot examples) and charges input tokens at a fraction of the normal rate on cache hits. For a RAG assistant, the system prompt of module 7 is identical on every call — that is 200 to 400 tokens that should be paid once per hour, not once per question.

The engineering is small: put the fixed instructions at the very start of the prompt, before any per-question content, and configure the client to enable caching where the provider supports it. Savings: 20 to 40 % of input cost, with zero quality change.

Incremental reindexing

The naive reindex — drop the collection, re-embed every chunk, rebuild the HNSW graph — is 2 to 3 hours on the red thread and grows linearly with the corpus. It is also unnecessary when only a handful of documents changed.

Incremental reindexing was introduced at the end of module 4 with upsert and delete. Its full workflow:

def incremental_reindex(current_docs: list[dict], previous_hashes: dict) -> dict:
    changes = {"added": 0, "updated": 0, "deleted": 0, "unchanged": 0}
    seen = set()

    for doc in current_docs:
        chunks = chunk_document(extract(doc["path"]), size=300, overlap=45)
        for c in chunks:
            cid = stable_chunk_id(doc, c)
            seen.add(cid)
            new_hash = hashlib.sha256(c["text"].encode()).hexdigest()
            if previous_hashes.get(cid) == new_hash:
                changes["unchanged"] += 1
                continue
            if cid in previous_hashes:
                changes["updated"] += 1
            else:
                changes["added"] += 1
            coll.upsert(ids=[cid], documents=[c["text"]],
                        embeddings=embed([c["text"]]), metadatas=[metadata(doc, c)])
            previous_hashes[cid] = new_hash

    stale = [cid for cid in previous_hashes if cid not in seen]
    for cid in stale:
        coll.delete(ids=[cid])
        del previous_hashes[cid]
        changes["deleted"] += 1

    return changes

The previous_hashes dictionary can live in the same SQLite as the embedding cache. Combined, the two caches make a nightly reindex of a slowly changing corpus take minutes instead of hours.

Logging: what to keep, what to drop

Log everything at first, prune later. For each answered question, write a single JSON line to a rotated file:

LOG_FIELDS = [
    "timestamp", "user_context_hash", "question_hash",
    "retrieved_ids", "reranked_ids", "prompt_tokens", "answer_tokens",
    "cost_llm_in", "cost_llm_out", "cost_embed", "cost_rerank",
    "faithfulness_score", "abstained", "from_cache",
]

Note the hashes rather than the raw text. On a system that handles sensitive procedures, logging the exact question and answer is often a compliance problem. Hash them for correlation, keep the raw text in a separate, access-controlled store, and truncate that store after a short retention window.

A cost dashboard read once a month is a cost dashboard read too late

Plot cost per question, cache hit rate and mean prompt length daily, and page an on-call engineer when any of them moves more than 30 % week-over-week. LLM regressions rarely announce themselves: they show up as an invoice that doubled without any commit that looks suspicious.

In summary

• LLM generation accounts for over 90 % of cost and about half of latency; measure the cost per question as a sum of terms and track each separately.
• Cache embeddings by (text, model) with a hash key, and cache answers by (question, user context) with a short TTL and a manual invalidate-by-source hook.
• Use provider-level prompt caching on the fixed system prompt, and reindex incrementally by comparing chunk hashes rather than rebuilding the whole collection.
• Log every question as a JSON line with hashed text plus costs and metrics; plot cost per question daily and page on 30 %+ weekly moves.

Next module: wiring modules 2 through 9 together into a full document assistant, with document-level permissions and a minimal interface, on the 300 internal procedures of the red thread.