Skip to main content

Module 9 — Observability and decision logging

An agent is deterministic in code and non-deterministic in behaviour. That combination breaks the debugging habits inherited from ordinary services: a stack trace tells you where the code failed, not why the model chose that action. Observability, for agents, means logging the decisions — thought, action, observation, cost — with enough structure that a run can be understood, compared and replayed.

What a decision log actually contains​

One JSON line per iteration, appended to a file or a database.

{
  "run_id": "r-2026-09-06-4a7b",
  "user": "u-142",
  "step": 3,
  "started_at": "2026-09-06T09:12:14.412Z",
  "thought": "The postgresql.org page confirms logical replication...",
  "action": {"name": "read_page", "args": {"url": "https://www.postgresql.org/docs/17/logical-replication.html"}},
  "observation_hash": "sha256:9e...",
  "observation_bytes": 3721,
  "tokens_in": 1842,
  "tokens_out": 76,
  "cost_usd": 0.0193,
  "latency_ms": 2140,
  "model": "openai/gpt-4o",
  "budget_state": {"tokens_used": 5231, "steps_used": 3, "steps_left": 3}
}

Six ideas concentrated in this record. The run_id links every step of the same run. The observation_hash is what allows replay without storing the observation itself (potentially large, potentially private) in the log. tokens_in and tokens_out are the honest cost breakdown — output tokens are three to five times more expensive on most providers. budget_state makes it obvious when a run is about to exit through a guardrail. And the whole record is one line, one JSON object, greppable and pipeable.

Aggregating across runs​

The per-step log becomes a table. Two queries return most of what a team needs to run an agent in production.

Cost distribution by question type. Group by an inferred question category and by run, sum tokens and dollars. Long-tail expensive runs — one question at 40 000 tokens while the median is 8 000 — reveal either an evaluation-set gap or a genuinely hard question worth an operational note.

Failure-mode distribution. Join the log with the evaluation set (module 10). For each failed run, look at the step where the answer went wrong. This is what feeds the failure catalogue of module 8 quantitatively — not "we saw a loop once", but "loops account for 40% of last week's failures".

Latency percentiles matter more than averages. A p50 of 12 seconds and a p95 of 90 seconds says one thing about user experience; a p50 of 40 seconds says something very different.

Replaying a run​

Given a log, you can re-execute a run in three modes.

Read-only replay. Feed the same question, the same tools, the same random seed to the same model — and hope the model returns the same output. In practice, providers reserve the right to change model versions, so bit-for-bit replay is not guaranteed. Log the exact model string, and expect drift over months.

Deterministic replay. Do not re-call the model. Reuse the stored assistant turns from the log, and only re-run the tools. This tests changes to tool implementations without paying for model calls. It is the workhorse of the module-10 evaluation harness.

Counterfactual replay. Re-run the loop but with one change — a different tool description, a stricter system prompt — and compare the trace. The eye reads only the diff, not the whole run. This is where descriptions get tuned in module 3, not from a whiteboard.

def replay(log_path: str, mode: str = "deterministic") -> dict:
    log = [json.loads(line) for line in open(log_path)]
    if mode == "deterministic":
        return re_execute_with_stored_actions(log)
    if mode == "counterfactual":
        return re_execute_with_new_prompt(log, PROMPT_V2)
    return re_call_model_and_tools(log)

The same agent in LangGraph​

Every module so far built the loop by hand. LangGraph — from the LangChain family, introduced in course 26 — offers a graph-based construction: nodes for each phase, edges for the transitions, and a checkpointer that automatically persists the state between iterations. For the observability chapter it is worth the comparison because LangGraph gives you the log almost for free.

from langgraph.graph import StateGraph, END
from typing_extensions import TypedDict

class State(TypedDict):
    question: str
    transcript: list
    answer: str | None

def think_and_act(state):
    reply = call_model(state["transcript"], tools_schema=SCHEMA)
    return {"transcript": state["transcript"] + [reply.message]}

def maybe_finish(state):
    last = state["transcript"][-1]
    return "finish" if last.get("tool_calls", [{}])[0].get("name") == "finish" else "act"

def act(state):
    tool_call = state["transcript"][-1]["tool_calls"][0]
    obs = run_tool(tool_call["name"], tool_call["args"])
    return {"transcript": state["transcript"] + [{"role": "tool", "content": obs}]}

graph = StateGraph(State)
graph.add_node("think", think_and_act)
graph.add_node("act", act)
graph.add_conditional_edges("think", maybe_finish, {"finish": END, "act": "act"})
graph.add_edge("act", "think")
graph.set_entry_point("think")

app = graph.compile(checkpointer=SqliteSaver.from_conn_string(":memory:"))

Two properties differ from our hand-written loop. The state is explicit — a TypedDict — and every transition is a pure function of it, which makes replay a natural operation the framework supports. And the checkpointer logs the state after every node, so restarting from step 4 after a crash is a single call, not a rewrite.

The trade-offs are also visible. The graph is more code for the simple case. Debugging a node error means learning where LangGraph inserts its own retries. And upgrading LangGraph across a major version is not free — the abstractions move, sometimes twice a year.

Our recommendation: write the first version by hand to build the mental model, then move to LangGraph once the loop has more than four branches or once multiple team members need to touch it. Both patterns produce logs that look the same to a dashboard.

A minimal dashboard​

Four charts and a table cover most operational needs.

  • Runs per hour with a stacked breakdown by outcome (finished, capped, error).
  • Cost per run, p50 and p95, on a rolling 7-day window.
  • Iterations per run, distribution histogram — a bimodal shape is a signal.
  • Failure rate by question category — from the evaluation-joined table above.

The table is a live tail of the last thirty runs, with a click-through to the full step-by-step trace. If the on-call engineer cannot see, in one page, whether the agent is healthy right now, the dashboard is not doing its job.

Instrument before you scale

Every hour spent on instrumentation before opening the agent to more users is repaid in days spent chasing "the agent behaves weirdly sometimes". The dashboard is not overhead; it is the API through which the team keeps a non-deterministic system understandable.

Summary​

  • The decision log is one JSON line per iteration, with thought, action, observation hash, tokens, cost, latency and budget state.
  • Aggregation reveals cost distributions and failure-mode distributions that ad hoc reading of traces never surfaces.
  • Three replay modes — read-only, deterministic, counterfactual — power both regression tests and prompt tuning.
  • LangGraph provides state and checkpointing for free; adopt it once branching or team size justifies the abstraction cost.

Next module: the supervised project — assembling everything into a research agent with an evaluation set, a cost report and an incident file.