Module 10 — Costs, latency and architecture choices

The customer-support assistant is ready to leave the notebook. The last decision — the one that outlasts every other design choice — is the serving architecture. Pick the wrong one and the bill dominates every conversation for the next two years. Pick the right one and the project pays for itself in the first quarter.

This module closes the running project with a defensible answer to two questions: what does one request actually cost, and API or self-hosting?

Cost per request, honestly

Two families of costs, both denominated per request.

API pricing is quoted per million tokens, split between input and output. Typical 2026 numbers for a strong 7B-class model behind an API: about $0.20 per million input tokens and $0.60 per million output tokens. A customer-support call with 800 input tokens and 200 output tokens costs:

$C_{\text{API}} = \frac{800}{10^6} \times 0.20 + \frac{200}{10^6} \times 0.60 = 0.000\,28 \text{ USD}$

Roughly $2.80 per 10 000 requests. Predictable, no infrastructure to run, no MLOps burden.

Self-hosting replaces the per-token price with a fixed hourly rate. A single L4 GPU on a cloud provider runs about $0.60 per hour and serves a 7B AWQ model at ~50 requests per second, so at full load:

$C_{\text{self}} = \frac{0.60}{3600 \times 50} = 3.3 \cdot 10^{-6} \text{ USD per request}$

Two orders of magnitude cheaper — but only at full load. Utilisation is what decides. At 10 % load the effective cost jumps to $3.3 \cdot 10^{-5}$ per request, still cheaper than the API but the margin narrows. At 1 % load, the API wins.

The rough decision curve for a 7B-class model:

Sustained traffic	Cheaper option
< 100 requests / hour	API
100 to 10 000 / hour	Depends on GPU utilisation
> 10 000 / hour	Self-hosting

The routing pattern: small model first, large model on demand

Most support requests are easy. A greeting is not the same problem as a policy dispute. Router architectures exploit this: a small, cheap model handles most traffic, and only escalates to a large model when it detects uncertainty.

def route(question, small_model, big_model, threshold=0.7):
    small_answer = small_model.generate(question, temperature=0.3)
    confidence = small_model.self_score(question, small_answer)
    if confidence >= threshold:
        return small_answer, "small"
    return big_model.generate(question, temperature=0.3), "big"

A router that sends 80 % of traffic to a 3B model and 20 % to a 70B one costs roughly:

$C_{\text{router}} = 0.8 \times C_{3B} + 0.2 \times C_{70B}$

For 3B at $0.05 / 1M tokens and 70B at$0.90 / 1M tokens on the same profile as above, that is $0.000\,04 + 0.000\,18 \approx 0.000\,22$ per request, saving 20 to 40 % over the mid-size model on all traffic, with almost no quality loss on the routed subset.

The routing signal itself matters. Cheap options: keyword match on user question, request length, presence of an order number. Better options: a small classifier trained on your own past traffic, or a self-consistency score from the small model.

Prompt cache: the single biggest quick win

Most support conversations start with the same system prompt: role description, tone, policies, escalation rules. This block is often 1 000 to 3 000 tokens. Without a cache, it is retokenised and re-processed on every request.

Modern serving stacks — vLLM, OpenAI, Anthropic — implement prompt caching: the KV-cache of a repeated prefix is reused across requests. First call, full cost. Subsequent calls sharing the same prefix, near-zero prefill cost. Typical savings on a support workload: 40 to 70 % of the input token cost.

Two conditions must hold:

1. The cached prefix must be byte-identical. Adding the current date breaks the cache.
2. The cache lifetime is typically 5 to 15 minutes. High-traffic prefixes stay warm; low-traffic ones re-warm at each call.

# System prompt kept identical across calls
SYSTEM_PROMPT = "You are the customer-support assistant for Acme Corp..."

def answer(question, model):
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},  # cacheable
        {"role": "user", "content": question},         # variable
    ]
    return model.chat(messages, temperature=0.3)

Putting the variable part last is what makes the cache useful. A variable field embedded in the middle of the prompt breaks the cache after that field.

Latency budget: know your users' tolerance

Human tolerance for a chat interface is well studied. Rough thresholds:

• First token in under 500 ms: feels instant.
• First token in 500 ms to 2 s: feels responsive, users wait.
• First token above 2 s: users think it is broken and re-send.

A 7B AWQ model on an L4 delivers first token in 100 to 300 ms with a warm cache, 500 ms to 2 s with a cold one. A 70B model on the same setup delivers first token in 1 to 3 s cold. Route accordingly: the 70B model is fine on questions where the user is already in a text-writing mindset, painful on quick clarifications.

Streaming is not optional

Return tokens to the user as soon as they are generated (server-sent events, WebSocket). A 300-token answer generated at 100 tokens/second takes 3 seconds to complete, but with streaming the user starts reading at 200 ms. Without streaming, they wait 3 seconds staring at a spinner. Same total latency, dramatically different perceived experience.

The decision for the running project

Pulling the ten modules together, the defensible architecture for the mid-sized company's support assistant, as of 2026:

• Base model: 7B or 8B open-weight instructed model with strong support for the target language (module 1, 2).
• Fine-tuning: LoRA on 500 to 2 000 curated support examples (module 3), then DPO on 1 000 preference pairs (module 4).
• Decoding: $T=0.3$, top-p 0.9, no repetition penalty (module 5).
• Context: system prompt plus RAG for policy documents, sliding-window summary beyond 10 turns (module 6, and course 18).
• Hallucinations: grounded prompting with citations, verifier on high-stakes answers (module 7).
• Serving: AWQ 4-bit with vLLM on one L4 or a small share of one A10 (module 8).
• Evaluation: 300-question business set, LLM-as-a-judge on every deploy, human review on the top 20 disagreements (module 9).
• Architecture: router with a 3B model for easy cases, prompt cache on the system block, streaming to the UI.

Total cost target: under $0.001 per request at production volume, first token under 500 ms, hallucination rate under 2 % on the business evaluation set.

Instrument cost, latency and error rate per request from day one

The three numbers that decide the project's future are all missing from most support deployments. Log them per request, dashboard them, alert on regressions. A dashboard set up the day you go live saves you the archaeology work later.

In summary

• API pricing is per token and predictable; self-hosting is per hour and cheaper at high utilisation — the crossover is around a few thousand requests per hour for a 7B-class model.
• Router architectures send easy traffic to a cheap small model and reserve the expensive one for hard cases, saving 20 to 40 % with almost no quality loss.
• Prompt caching on a shared system prefix is the single biggest cost win on support workloads; put variable content last so the cache actually hits.
• Streaming is not optional in a chat UI; the perceived latency floor is the time to first token, not the total generation time.

The course closes here. The next stop is the recap and the final exam.