Module 7 — Shared memory and context passing

By module 3 the crew already passes structured outputs from one task to the next via context=[…]. So why bother with memory at all? Because context=[…] only carries the outputs of listed tasks, not the smaller facts that a Writer remembers halfway through a section, and not the entities (product names, personas, version numbers) that the Analyst mentioned once and the Reviewer will need three tasks later. This module wires the three memories CrewAI provides and, more importantly, decides which ones the running project actually needs.

Turning memory on

Memory is opt-in at the crew level:

crew = Crew(
    agents=[analyst, writer, reviewer, manager],
    tasks=[extract_features, draft_sections, review_draft, arbitrate, finalise],
    process=Process.sequential,
    memory=True,
    embedder={"provider": "openai", "config": {"model": "text-embedding-3-small"}},
)

A single flag activates three subsystems. Each has a distinct scope, a distinct storage backend, and a distinct failure mode.

Short-term memory: within one run

Short-term memory stores facts produced during the current crew run, indexed for semantic retrieval. When an agent starts a task, the top-k relevant snippets from earlier steps are prepended to its prompt automatically — without the developer listing them in context=[…].

Concretely: the Analyst mentions "the audience is L2 support engineers, not managers" as a side note in its extraction. The Writer, drafting section 4, retrieves that snippet through short-term memory and adjusts the tone. Without short-term memory, that side note would only be visible if the Analyst had put it in the structured output.

Storage backend: an in-memory vector store, wiped at the end of kickoff(). Cost: one embedding call per stored snippet, one search per task start. For a five-task crew that produces twenty snippets, that is under a cent — cheap.

Long-term memory: across runs

Long-term memory persists across crew runs. It is what lets the crew remember, on Monday, that on Friday it approved a specific stylistic decision (say, "use 'we' rather than 'the platform'"). Storage is on disk (~/.crewai/long_term_memory.db by default), and the memory is scoped by the crew's identifier so two different crews do not pollute each other.

For the running project, long-term memory is a mixed blessing. On the positive side, style decisions the team accepted last week can influence this week's draft without a system-prompt update. On the negative side, a wrong decision made last week can silently steer this week's draft in the wrong direction — a very hard bug to notice when you did not sign up for a persistent state.

Rule of thumb: turn long-term memory on for crews that face a stable, recurring task with a slowly evolving preferences ("customer support tone"); leave it off for one-shot creative tasks where each run should stand on its own.

Entity memory: names, dates, versions

Entity memory is specialised for the kinds of facts that get lost otherwise: proper nouns (product names, people, teams), numbers (versions, quotas, thresholds), and dates. The mechanism is different from short-term — CrewAI extracts entity mentions from each step and stores them in a small structured index keyed by entity, not by embedding similarity.

The benefit is exact recall on the facts that matter. When the Analyst extracts "version 4.2 is the first to support SSO", the Reviewer three tasks later can retrieve "SSO → version 4.2" cheaply and check whether the Writer said "since version 4.1" (a fabrication). This is where hallucinations of the "wrong version number" kind get caught.

Entity memory is the single memory subsystem that pays for itself on almost every crew that touches structured facts. Turn it on by default; the storage overhead is negligible.

What actually crosses between tasks

By the end of module 7, the Writer's prompt on a mid-run task contains, in order:

1. The Writer's role, goal and backstory (constant, from the agent).
2. The Writer's task description and expected output (from the task).
3. The outputs of tasks named in context=[…] (module 3).
4. The top-k snippets from short-term memory (this run's earlier steps).
5. Entity memory hits relevant to the current task.
6. Optionally, hits from long-term memory (across-run).

That is a lot of tokens on a bad day. Which brings us to the actual constraint.

Keeping the cumulative context under control

The trap of "just turn memory on" is context bloat. A ten-task run with generous memory can push the Writer's prompt from 2 000 to 12 000 tokens by task 8, at which point the model both slows down and starts ignoring parts of its own instructions.

Three habits keep the bill and the confusion down.

• Cap the retrieval k on short-term memory (typically 3 to 5 snippets). More is not more — it is noise.
• Prefer context=[…] for outputs the next task strictly needs. Memory is for background facts, not for the primary input.
• Purge long-term memory deliberately when the crew's goal changes. crew.reset_memory() wipes short-term automatically at the end of a run; long-term needs a manual reset.

For the running project: entity on, short-term on, long-term off

The documentation crew benefits from short-term (Analyst side-notes reach the Writer) and from entity (version numbers stay correct through five tasks). It does not benefit from long-term — each product documentation is a fresh brief, and last week's stylistic decisions belong in the style guide file (which the Reviewer already has as a tool) rather than in a hidden database that no one reviews.

crew = Crew(
    ...,
    memory=True,
    # short-term and entity default to on when memory=True.
    # Disable long-term explicitly if not wanted (via config, or a fresh embedder path per run).
)

Read the memory hit before trusting it

When a memory hit lands in a task's context, the trace prints "Retrieved from memory: …". Read those lines. A memory that keeps retrieving the same irrelevant snippet is a signal that the snippet is too broad or the current task's description is too vague. Fix the description first, the memory second.

Summary

• Memory in CrewAI has three subsystems: short-term (this run), long-term (across runs), entity (names, dates, versions).
• Short-term is cheap and useful for side facts the primary context=[…] does not carry; entity catches hallucinated version numbers and dates almost for free.
• Long-term is powerful and risky: it persists a state you did not sign up for; enable it deliberately for recurring tasks with slowly evolving preferences.
• Keep cumulative context under control with capped k, memory for background only, and explicit resets when the crew's goal changes.

Next module: measuring the actual cost, the latency and the variability of a run, and comparing them to the single-agent baseline.