Module 6 — Conversation memory

Modules 4 and 5 gave the assistant knowledge. This module gives it memory — the ability to remember, across turns, that the user is Ahmed, that his trip was to Berlin, and that the previous receipt was a 42 EUR dinner. Everything hinges on keeping that memory useful without letting it eat the context window.

Why the naïve version fails

The first instinct is to keep the whole history and paste it into every prompt.

history = []
def ask(question: str) -> str:
    history.append(("human", question))
    prompt = ChatPromptTemplate.from_messages(
        [("system", "You are the expense assistant."), *history]
    )
    reply = (prompt | model | StrOutputParser()).invoke({})
    history.append(("ai", reply))
    return reply

This works for three turns. By turn thirty the prompt is thousands of tokens, every call pays the full history in and out, latency triples, and quality drops because attention smears over irrelevant early turns — the lost-in-the-middle effect course 16 named in module 6.

Memory in production is not "keep everything". It is a compression policy under a token budget.

The clean interface: RunnableWithMessageHistory

LangChain's modern approach separates two concerns: the chain does not know about history, and a wrapper injects it per session.

from langchain_core.chat_history import BaseChatMessageHistory, InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory

store: dict[str, BaseChatMessageHistory] = {}

def get_history(session_id: str) -> BaseChatMessageHistory:
    if session_id not in store:
        store[session_id] = InMemoryChatMessageHistory()
    return store[session_id]

assistant_prompt = ChatPromptTemplate.from_messages([
    ("system", "You help the user manage their expense reports."),
    MessagesPlaceholder("history"),
    ("human", "{question}"),
])

base_chain = assistant_prompt | model | StrOutputParser()

with_memory = RunnableWithMessageHistory(
    base_chain,
    get_history,
    input_messages_key="question",
    history_messages_key="history",
)

with_memory.invoke(
    {"question": "The Berlin trip was on the 4th."},
    config={"configurable": {"session_id": "ahmed"}},
)

The wrapper reads the history from get_history(session_id), splices it into the prompt at the MessagesPlaceholder, invokes the chain and appends the new human and AI messages back to the history. Per-session isolation is enforced by the session_id: Ahmed's history never leaks into Sara's session, because they look up different keys in the store.

InMemoryChatMessageHistory is fine for a demo. In production, swap it for a persistent implementation — RedisChatMessageHistory, SQLChatMessageHistory on Postgres or SQLite, or a homegrown class over your existing database. The interface is three methods (add_messages, messages, clear); implementing it against your store is an afternoon of work.

Windowed memory: keep the last N turns

The simplest compression policy is: keep only the last N messages. Everything older is dropped.

def get_history(session_id: str) -> BaseChatMessageHistory:
    history = _store.setdefault(session_id, InMemoryChatMessageHistory())
    # Trim to the last 10 messages before returning
    history.messages = history.messages[-10:]
    return history

This keeps the prompt bounded and costs nothing to compute. It fails on long-running conversations where facts from turn 3 still matter at turn 40 — the trip destination, the traveller's name, the budget code. The right N depends on the task; 8 to 12 messages is a good starting point for a Q&A assistant, higher for a coding pair.

Summary memory: compress into a rolling paragraph

The complement is a summary of the older turns, kept up to date by a cheap model, injected into the system prompt.

system: ... previous turns summary: "Ahmed is preparing his September trip
to Berlin. So far he has submitted a 42 EUR dinner and a 68 EUR taxi ride.
He asked about the meal ceiling and got the answer 25 EUR."

Two prompts, three lines of code, and the model now knows facts from turn 3 while the visible history keeps only the last 6 messages. The trick is the update policy: summarise only when the raw history exceeds a threshold, not on every turn — otherwise you pay one model call per user message for nothing.

The hybrid — window of the last N messages + rolling summary of the rest — is what most production assistants ship. Windowing gives recency, the summary gives coverage.

Persistence and privacy

Two decisions to make explicitly.

Persistence. The moment history lives longer than one process, decide where. Redis is cheap and fast but volatile if not configured. Postgres via SQLChatMessageHistory is the safe default for anything that must survive a deploy. Add a TTL on old sessions unless you have a legal reason to keep them forever — a chat log is personal data.

Privacy. Conversations often contain names, amounts and dates. If your provider is external, log carefully what you send: full history in a trace is data exfiltration by another name. Course 9 will return to redaction; the reflex to install now is a redact_pii step before persistence, not after.

A drifting session_id is a data leak

Ahmed's session_id is not "user_ahmed". It is a random UUID stored in his session cookie. Deriving it from the username lets a coworker with knowledge of a colleague's login guess the id and retrieve the history. Random, opaque, rotated on logout.

Wiring the assistant

The assistant of the course now has three moving parts: the retrieval chain of module 5, the extraction chain of module 2, and the memory wrapper of this module. The session_id runs across all three so that "the previous receipt" refers to the same object throughout the conversation.

# The full assistant, seen from the caller
with_memory.invoke(
    {"question": "How much of the Berlin dinner will be reimbursed?"},
    config={"configurable": {"session_id": ahmed_session_id}},
)

In summary

• Naïve "keep everything" memory blows up the context window and triggers lost-in-the-middle; production memory is a compression policy under a token budget.
• RunnableWithMessageHistory with a per-session_id history keeps history out of the chain code and enforces user isolation.
• Windowed memory (last N messages) is bounded and cheap; summary memory covers older facts; the hybrid is what most assistants ship.
• Persist in Redis or Postgres, add a TTL, redact PII before storage, and never derive session_id from a guessable username.

Next module: giving the assistant tools — a currency converter, a policy-ceiling lookup — that it calls itself, rather than answering by paraphrasing text.