Module 10 — Project: a supervised research agent

Nine modules of pieces. This module assembles them into a single runnable project — the documentary watch agent introduced in module 1 — and specifies the evaluation, the reporting and the production checklist that a team would demand before switching it on for users.

The assembled skeleton

The full loop is now a hundred lines of Python. Every decision points back to an earlier module.

def run(question: str, user: str, opts: RunOpts) -> RunResult:
    budget = Budget(**opts.budget)                        # module 7
    memories = recall(question, user=user, top_k=5)       # module 4
    plan_steps = plan(question, TOOLS)                    # module 5
    transcript = build_initial_transcript(
        question=question,
        memories=memories,
        plan=plan_steps,
    )

    log = DecisionLog(run_id=fresh_id())                  # module 9
    draft = None
    for step in range(budget.max_steps):
        reply = call_model(transcript, tools_schema=describe(TOOLS))
        log.record(step, reply, budget)                   # module 9

        if reply.wants_replan and log.replans < 2:        # module 5
            plan_steps = replan(question, transcript)
            transcript = re_inject_plan(transcript, plan_steps)
            continue

        action = reply.tool_call
        if action.name == "finish":
            draft = action.args["answer"]
            break

        obs = call_tool_with_permissions(action, user, budget)  # modules 3, 7
        transcript = append_observation(transcript, action, obs)
        transcript = maybe_summarise(transcript, budget)         # module 4

    verdict = verify_all(question, draft, transcript)     # module 6
    if verdict.needs_revision and log.reviews < 1:
        draft = revise(question, draft, verdict, transcript)

    persist_new_memories(user, question, draft)           # module 4
    log.finalize(draft=draft, budget=budget)
    return RunResult(answer=draft, log=log, capped=(draft is None))

Ten function calls, each named after a module concept. New reader of the codebase can find any piece of behaviour in under a minute — the same organizational discipline that made module 8 fixes cheap.

The evaluation set

Thirty questions, hand-curated by the product team, each labelled with:

• The question itself, in one sentence.
• A gold answer or a constraint the answer must satisfy.
• The expected sources — URLs or internal document IDs — that a correct answer should cite.
• A category label — recent-external, historical-external, internal-decision, cross-source-conflict.

Thirty is not much statistically, and it is deliberate: the set exists so that every question can be read by a human when the agent fails, not so that a p-value can be published. Larger evaluation sets are for larger deployments; at this stage, quality of judgement beats sample size.

Running the evaluation

The harness runs each question through the full agent, and for each, records three numbers.

Correctness (0–1). Automatic when a gold answer exists, judged by a separate LLM against a rubric otherwise. Every judged score is spot-checked on 20% of runs by a human — the LLM judge is trusted after that calibration, not before.

Cost (USD). The sum from the decision log, no adjustments. A run that solves the question with two iterations is cheaper than one that solves it with six; both count.

Latency (seconds). End to end, from question in to answer out. Users notice p95 more than p50.

def evaluate(agent, dataset: list[Question]) -> EvalReport:
    results = []
    for q in dataset:
        result = agent.run(q.text, user="eval", opts=RunOpts())
        results.append({
            "id": q.id,
            "correct": judge(result.answer, q.gold_answer or q.constraint),
            "cost_usd": result.log.total_cost_usd(),
            "latency_s": result.log.total_latency_s(),
            "capped": result.capped,
        })
    return EvalReport.from_rows(results)

Reading the numbers

Baseline results after a two-week iteration on the running example:

Category	Questions	Correct	Median cost	p95 latency
Recent external	9	89 %	USD 0.019	24 s
Historical external	8	100 %	USD 0.011	12 s
Internal decision	8	88 %	USD 0.014	15 s
Cross-source conflict	5	60 %	USD 0.028	41 s

Read the last row. The 60 % on cross-source conflict is not "close enough" — it is a class where the agent is worse than a coin flip about which source to trust. That is the class where verification (module 6) helps most and where a human confirmation gate (module 7) is most warranted for external-facing use.

Read the cost column. Cross-source conflicts cost 2.5× more than historical questions. That is fine when they are 17 % of traffic; it becomes a budget issue if they climb to 40 %. Track the mix.

The incident report

Every failed run — six failures on this baseline — is documented in an incident file with the same structure.

• Run ID, question text, timestamp.
• What the agent answered, and what the correct answer was.
• The first step where the trace went wrong (from module 8's methodology).
• Root cause (prompt, tool description, code, model limitation).
• The fix, and the change it produced when the failing question was re-run.

Six lines per incident, kept in a Markdown file the team reads on Monday mornings. This is not the runtime log — it is the learning log, curated by humans. Its purpose is that the same failure does not recur three months later because the person who fixed it left.

The production checklist

The go-live gate. Ten items, all binary, no exceptions.

1. Budgets enforced at the runtime layer (iteration, tokens, dollars).
2. Concurrent-runs quota per user.
3. Tool permissions encoded outside the description string, with an external tier requiring human confirmation.
4. Sandbox for observations: fenced isolation and system-prompt reminders.
5. Decision log written on every run, retained per your policy (90 days is a common default).
6. Evaluation set run automatically on every prompt or tool change.
7. Cost dashboard with a hard-stop at the daily budget.
8. User-visible cancel on any run longer than five seconds.
9. Incident report template and a review cadence.
10. Fallback answer when the agent gives up — a graceful message with a link to human help, never a fake answer.

Every item is small on its own. Skipping any one is what turns a demo into a headline.

Ship it small, then let it grow

Open the agent to a small internal group first, with the checklist above satisfied. Read every failed run for the first two weeks. Add exactly one capability per iteration. Any agent that grew ten capabilities before satisfying the checklist is an agent that will be turned off within six months.

Summary

• The full loop assembles memory, plan, ReAct, tools, verification, budgets and logging into a hundred well-named lines.
• The evaluation set is small, hand-curated and read by humans — quality of judgement over sample size at this stage.
• The report tracks correctness, cost and latency by question category, and the interesting rows are always the worst ones.
• The production checklist has ten binary items; skipping any one is what turns a good agent into a public incident.

Next: the recap and the 40-question exam.