Module 6 — Context window and memory management

Every model comes with a finite context window: the maximum number of tokens it can attend to in one call. In 2020 it was 2 048. In 2026 it ranges from 8 000 for a plain open model up to a million for a few flagship APIs. That growth solved some problems and unveiled new ones — the topic of this module.

For the customer-support assistant of the running project, the practical questions are: how many past turns should the model see, what does that cost, and when does adding more context stop paying off? All three have answers that are more counter-intuitive than the marketing suggests.

Context has a linear cost, then a quadratic one

Self-attention costs $O(n^2)$ in the sequence length $n$. Reading the whole context to answer one question multiplies compute and memory by a factor that grows with the square of the length. Modern implementations (FlashAttention, paged attention) keep the memory footprint linear in $n$ during generation, but the compute stays quadratic during the initial prompt processing (the prefill phase).

A concrete example: a 7B model with $d=4096$ hidden dimension processing an 8k prompt does roughly 200 GFLOPs of attention; the same model on an 128k prompt does 50 TFLOPs — a 250x increase for a 16x longer prompt. Latency and cost follow.

Prompt length	Prefill time (H100, 7B model)	KV-cache size
2 000 tokens	60 ms	128 MB
8 000 tokens	240 ms	512 MB
32 000 tokens	1.5 s	2 GB
128 000 tokens	12 s	8 GB

The KV-cache stores the keys and values for every token seen so far, so the model does not recompute them. It grows linearly in $n$ per request; multiply by the number of concurrent users and you see why long context is the single biggest driver of serving cost.

Lost in the middle

More context is not automatically better. The 2023 "Lost in the Middle" paper by Liu et al. measured how well a model recovers information depending on where it is placed in a long prompt. The finding was consistent across models: retrieval accuracy is high at the beginning and end of the context, and drops sharply in the middle.

For a customer-support ticket where the useful information sits three-quarters of the way through a long conversation, the model may miss it entirely, while confidently answering as if it had read the full history.

Two consequences for prompt design:

1. Put the important stuff at the start or the end. System prompt at the very beginning, current user question at the very end.
2. Order retrieved passages by relevance, best last. If you concatenate ten retrieved chunks, put the most relevant one closest to the question, not first in a "top-ten" list.

Sliding-window summary

A common pattern for long conversations: keep the last $k$ turns verbatim, and replace older turns with a model-generated summary that fits in a fraction of the space.

from transformers import pipeline

summariser = pipeline("summarization", model="facebook/bart-large-cnn")

def rolling_context(history, max_turns=10, summary_budget=400):
    if len(history) <= max_turns:
        return history
    old, recent = history[:-max_turns], history[-max_turns:]
    old_text = "\n".join(f"{m['role']}: {m['content']}" for m in old)
    summary = summariser(old_text, max_length=summary_budget, do_sample=False)[0]["summary_text"]
    return [{"role": "system", "content": f"Earlier conversation summary: {summary}"}] + recent

The trade-off is explicit: newer turns are lossless, older turns lose detail. For a support conversation this is usually acceptable — the customer's name, order number and current problem are in the last five messages; the exact wording of an earlier message rarely matters.

A summariser is a lossy compressor

The summary you feed back is itself generated by a model. Any hallucination it introduces will be treated as gospel by the next turn. Two safeguards: constrain the summariser to extractive style, and periodically show a full-history transcript to your evaluation pipeline (module 9) to catch drift.

External memory: the persistent option

Some information is too long for any window and too structured for a summary: an entire product catalogue, a company knowledge base, three years of support tickets. The standard solution is external memory, retrieved on demand.

The pattern is:

1. Chunk documents into 200 to 500 token passages.
2. Embed each with a sentence-encoder.
3. Store embeddings in a vector database (FAISS, Qdrant, pgvector).
4. At query time, embed the user question, retrieve the top-$k$ passages, and prepend them to the prompt.

from sentence_transformers import SentenceTransformer
import faiss

encoder = SentenceTransformer("intfloat/multilingual-e5-base")
passages = ["Return policy: 30 days...", "Delivery times: 3-5 business days...", "..."]
index = faiss.IndexFlatIP(768)
index.add(encoder.encode(passages, normalize_embeddings=True))

def retrieve(query, k=3):
    q = encoder.encode([query], normalize_embeddings=True)
    _, idx = index.search(q, k)
    return [passages[i] for i in idx[0]]

This is the seed of retrieval-augmented generation, the topic of course 18. This module only introduces it as one of the answers to context limits; course 18 covers the full pipeline — chunking strategies, reranking, hybrid search, evaluation.

When to reach for RAG rather than a longer window

A rough decision rule for the customer-support assistant:

• Static knowledge that never changes and fits in 4 000 tokens: put it in the system prompt.
• Frequently-changing knowledge (product prices, current promotions): fetch it just-in-time as structured data, not free text.
• Large corpus, changing weekly, one relevant chunk per query: RAG.
• The whole conversation is the useful context: long-context model, plus a summariser once you exceed 32k tokens.

Long context is not free, so paying for a 200k window to store a knowledge base that a 2k retrieval could locate is one of the most common cost-multiplication mistakes in this field.

Measure prompt tokens per request in production

Instrument your prompt-building code to log the total token count on every call. In most support pipelines, the average creeps up over months as engineers add instructions. Once you see the p99 crossing your model's window, you have exactly one warning before requests start silently failing.

In summary

• Self-attention is quadratic in context length for prefill; long-context serving cost scales accordingly and is dominated by the KV-cache.
• Lost in the middle is real: put critical information at the start and end of the prompt, order retrieved passages best-last.
• Sliding-window summarisation is a cheap way to keep long conversations manageable, at the cost of a lossy compressor whose drift needs to be monitored.
• External memory (RAG) replaces "bigger window" with "targeted retrieval"; course 18 covers the full pipeline, this module places the decision in the design flow.

Next module: what to do when the model, no matter how good the context, still invents facts.