Module 4 — Working memory and long-term memory

The transcript we built in module 2 is the agent's working memory. It grows every iteration and disappears when the loop returns. This module explains how to keep working memory readable inside a single run, and how to give the agent a long-term memory that survives across runs — using embeddings, borrowed from the retrieval-augmented generation course.

The context window as working memory

Every iteration re-sends the entire transcript. The math becomes brutal quickly.

On the running example, one iteration of the watch agent adds roughly: one thought (30 tokens), one action (40 tokens), one observation from a read_page capped at 4 000 characters (about 1 200 tokens). Ten iterations therefore add about 12 700 tokens on the input side alone. Add the fixed cost of the system prompt and the tools schema — 1 500 tokens together — and iteration ten sends the model a 14 200-token prompt to reason about.

Two consequences follow. The cost curve is quadratic, because the tenth iteration pays for the observations of iterations one to nine. The quality curve is not flat: models degrade on very long prompts, missing information located near the middle — the lost in the middle effect measured on GPT-4 and Claude 2 in 2023 and still measurable on their successors in 2026.

Summarise, cap, or drop

Three well-defined moves keep working memory small.

Cap raw observations. Every tool return is truncated to a fixed number of characters before it goes into the transcript — 4 000 in the code from module 2. The model reads the first paragraphs, which usually contain the fact it needs. If not, the same tool can be re-called with a narrower query.

Summarise the past when it exceeds a threshold. When the transcript crosses, say, 6 000 tokens, replace the oldest iterations by a paragraph produced by a cheaper model: "You searched for X, read Y, and learned Z." Cost: one extra call every few iterations. Benefit: the transcript stops growing linearly.

def maybe_summarise(transcript: list, budget_tokens: int = 6000) -> list:
    total = count_tokens(transcript)
    if total < budget_tokens:
        return transcript
    head = transcript[:2]                    # system prompt + question
    old, recent = transcript[2:-4], transcript[-4:]
    summary = summarise(old, model="cheap")  # returns one paragraph
    return head + [{"role": "system", "content": f"Earlier: {summary}"}] + recent

Drop what is safe to lose. Failed tool calls, invalid-argument observations, redundant search results with the same URL. Every dropped item is fewer tokens on every subsequent iteration.

Long-term memory: what should survive the loop

Working memory disappears when the loop ends. That is a feature — running the agent again should not be biased by the last user's question. But some knowledge should persist. Three categories are worth distinguishing.

Facts specific to a user. A preferred timezone, a document repository the user always wants searched, a spelling preference for names.

Facts specific to the domain. The names of internal products, of teams, of tools. These do not change per user but must not sit in the system prompt indefinitely — the system prompt is bounded, the domain vocabulary is not.

Feedback on past runs. "Last week, when asked about Postgres CDC, you cited a wrong URL. Trust postgresql.org over blog aggregators." A distilled lesson learned, not a full transcript.

Storing memories as embeddings

The mechanism is the retrieval-augmented generation stack from course 18. Each memory is a short text — one to three sentences — and its embedding is stored in a vector database.

def remember(text: str, kind: str, user: str | None = None):
    embedding = embed(text)
    store.insert({
        "text": text,
        "embedding": embedding,
        "kind": kind,
        "user": user,
        "created_at": time.time(),
    })

def recall(question: str, user: str | None = None, top_k: int = 5) -> list[str]:
    hits = store.search(embed(question), top_k=top_k, filter={"user": user})
    return [hit["text"] for hit in hits if hit["distance"] < 0.35]

The recall step runs once, before the loop starts, and injects its results into the system prompt: "Relevant memories: bullet, bullet, bullet." Running it once per iteration is tempting and almost always wrong — it doubles cost while adding little, because the relevant memories usually surface for the question, not for every intermediate thought.

What the agent must be told to forget

An agent that remembers everything becomes an agent that carries every past mistake and every stale fact into every new run. Three rules bound the store.

Time-to-live on volatile facts. A memory such as "the ops team is currently on-call" is worthless a week later, and misleading. Set a TTL and let the store prune.

Explicit user deletion. If a user asks the agent to forget a name, an address or a document, that request must succeed. This is not politeness — under the GDPR and equivalents, it is required. A forget(user, pattern) administrative tool that deletes matching rows is part of the design, not an afterthought.

Deduplication at write time. Storing the same fact twenty times because twenty runs stumbled on it does not make it more true. Before inserting, run a similarity search on the new text: if a match above 0.9 exists, keep the older row and update its timestamp.

A memory store is a distinct component from the retrieval corpus

Do not mix the two in the same index. The corpus is authoritative content, updated by content owners; the memory store is agent-generated notes, updated by the agent itself. Mixing them makes it impossible to trust a citation — the agent starts citing itself. Keep them in separate collections and pass them through separate retrieval steps.

The running example, with memory

The watch agent now recalls at start-up: "This user prefers postgresql.org over blog aggregators for Postgres questions." That single injected line reduces bad-source citations on our evaluation set from 18% to 4% of runs, at a cost of one extra embedding lookup and roughly forty tokens per run. Working memory management (cap + summarise) removes another 30% from the average token bill on runs that reach eight or more iterations.

Summary

• Working memory is the transcript; it grows monotonically and is why cost is quadratic in iterations.
• Three moves keep it small: cap observations, summarise old iterations, drop failed calls.
• Long-term memory is a vector store of one- to three-sentence notes, recalled once before the loop, injected in the system prompt.
• The store needs TTLs, explicit user deletion and deduplication; it is a distinct component from the retrieval corpus.

Next module: planning — writing an explicit plan before executing it, and knowing when to replan.