Module 9 — Tracing, evaluation and debugging

The assistant now runs end to end. Modules 5 through 8 assembled retrieval, memory, tools and an agent loop. This module answers the question every team meets once the pieces work: how do you know it still works next week? — after a prompt tweak, an embedding-model upgrade, a switch from Qwen to Llama, or a corpus refresh.

Why a trace is not a log

A log line records "model returned 42.30". A trace records "for this input, the retriever ran with these parameters and returned these five chunks; the model was called with these three messages; it produced these two tool calls; the tools returned this and this; the model was called again with this observation; it returned this final answer, in 3.2 seconds and 1 240 tokens".

The difference is what you need to debug a wrong answer. A log tells you what; a trace tells you why. Every step of a LangChain runnable emits trace events by default; you just need a tracer that persists them.

The two options in 2026:

LangSmith is the managed tracer from LangChain, and the path of least resistance: set two environment variables and every chain becomes traceable in a web UI.

export LANGSMITH_API_KEY=...
export LANGSMITH_TRACING=true
export LANGSMITH_PROJECT=expense-assistant

Open tracers — OpenTelemetry with a tracing backend (Phoenix, Langfuse, Datadog) — offer the same visibility with more setup and no vendor lock-in.

Which you pick matters less than turning tracing on before you need it. A trace collected only after the first user complaint is a trace of the wrong week.

What to read in a trace

Three things, in order.

Latency by node. If the chain takes four seconds, is it the retriever (200 ms), the model (3 200 ms) or the tool call to the currency API (600 ms)? A stacked latency view answers this in one glance. This is where you discover that switching the embedding model saved you 15 % on retrieval and nothing on end-to-end because the model dominated.

Token counts per call. Prompt tokens and completion tokens per model invocation, summed per request. A regression in cost per conversation is almost always a change in prompt shape: someone added a "Include the full document verbatim in your reasoning" line to the system prompt and the token bill quintupled overnight.

The exact messages sent to the model. Not "a system prompt then the user question" — the actual text, in the order the model saw it. Nine times out of ten a broken chain shows a clear cause here: an empty retrieval, a placeholder that did not substitute, a memory summary that swallowed a fact.

Building an evaluation dataset

A trace tells you what the chain did on one input. An evaluation dataset tells you what it does on many, systematically.

Start with 20 real user questions from the earliest traces, kept as a frozen list of (input, expected_answer). Not synthetic questions — the ones your users actually asked, warts and all. Grow it to 100 to 200 over time, with a stable split by category (policy question, receipt extraction, ambiguous, out-of-scope).

from langsmith import Client

client = Client()
dataset = client.create_dataset("expense-assistant-eval-v1")
client.create_examples(
    inputs=[{"question": q} for q in questions],
    outputs=[{"answer": a} for a in answers],
    dataset_id=dataset.id,
)

The frozen part is what makes it a benchmark. A dataset that changes between runs cannot detect a regression. Version it — -v2 when you add a batch, -v1 stays around for comparison.

Evaluators: measuring what matters

An evaluator scores one prediction against one expected output. Three families cover the daily work.

Exact-match / regex evaluators for structured outputs: the extracted amount equals 42.30, the currency is "EUR". Cheap, deterministic, ruthless.

Reference-based evaluators: is the prediction semantically equivalent to the reference? Implemented with a cheap model as a judge, they cover free-form answers that admit many phrasings.

Reference-free evaluators: is the answer grounded in the retrieved context (faithfulness), does it cite the passages it used, does it abstain when the corpus does not cover the question? These are the ones that catch hallucinations, and they matter more than exact-match on a QA assistant. Course 18, module 8, is the reference.

from langsmith.evaluation import evaluate

def is_grounded(run, example) -> dict:
    ...   # LLM-as-a-judge: is the answer supported by the context?
    return {"key": "faithfulness", "score": 1 if supported else 0}

evaluate(
    lambda inputs: policy_chain.invoke(inputs["question"]),
    data="expense-assistant-eval-v1",
    evaluators=[is_grounded, cites_a_page, does_not_hallucinate_ceiling],
)

The output is a per-example table plus aggregates: faithfulness 92 %, citations 88 %, abstention 96 %. Those three numbers, plotted across runs, are what turns "did my prompt change help?" from a hunch into a decision.

Regression testing between versions

The final habit is the one that pays for itself the fastest. Rerun the evaluation on every non-trivial change — new prompt, new embedding model, new base model, new retriever configuration — and compare the aggregates.

The workflow becomes:

1. A pull request changes the system prompt.
2. CI runs the evaluation set against the old and new chain.
3. The report shows faithfulness went from 92 % to 89 %, citations from 88 % to 85 %.
4. The PR does not merge until the regression is understood.

That is the exact same discipline a software team applies to unit tests. Nothing about LLMs changes it — only the fact that the tests are noisy, so a small drop is not necessarily a regression. Repeat runs three times and treat the median as the signal.

Evaluating with the same model you use in production

LLM-as-a-judge biases toward its own family: GPT judges GPT favourably, Claude judges Claude favourably. When possible, judge with a different model family — or, better, pin the judge to a version and never upgrade it, so at least your comparisons stay comparable.

In summary

• A trace captures every step of a chain — inputs, outputs, latency, tokens — and is what turns "sometimes it hallucinates" into a locatable bug; LangSmith or an open OpenTelemetry backend both work.
• Read a trace in three passes: latency by node, token counts per call, actual messages sent to the model.
• Build an evaluation set from 20 real user questions, freeze it, version it, and grow to 100–200 examples split by category.
• Use exact-match, reference-based and reference-free evaluators; the reference-free ones (faithfulness, citations, abstention) are what catch hallucinations.
• Rerun evaluation on every change and gate merges on regression; judge with a different model family, and repeat runs to smooth the noise.

Next module: putting all ten modules into one working project — the expense assistant, wired end to end, with tests, cost budget and a note on production.