Skip to main content

Module 7 — Building the final prompt and citing sources

Modules 4 to 6 gave us a handful of reranked, deduplicated passages. This module turns them into a prompt that produces a grounded answer with verifiable citations, and refuses to answer when the passages do not support one. Everything the pipeline has built so far can be undone by a careless prompt at this step.

The three-block anatomy of a RAG prompt​

Every effective RAG prompt has the same three sections, in the same order. The vocabulary changes; the structure does not.

  1. System instructions — role, tone, abstention rule, citation format
  2. Context — the retrieved passages, each with a stable label
  3. Question — the user's actual query, plus any relevant user context

Splitting these blocks explicitly matters. Long-standing behaviour of language models: they weight tokens near the end of the prompt more heavily than tokens in the middle, and the beginning is where they "install" the instructions. Bury the abstention rule at the bottom under two thousand tokens of context and the model quietly ignores it.

SYSTEM = """You are a document-based assistant for internal company procedures.
Answer the user's question using ONLY the passages provided in the context.

Rules:
- If the passages do not contain the answer, reply exactly:
  "I do not have enough information in the provided documents to answer."
- After each fact, add a citation in the form [S<n>] referring to the passage.
- Do not use knowledge from your training if it is not confirmed by a passage.
- Answer in the language of the question.
"""

CONTEXT_TEMPLATE = """[S{n}] source: {source_name} - section: {section} - page: {page}
{text}
---"""

USER_TEMPLATE = """Context:
{context}

Question: {question}
Answer:"""

Labelling passages: the citation contract​

The [S1], [S2], [S3] labels are not decoration. They are a contract between the retriever and the model: the retriever gave the passages numbers, the prompt tells the model to reuse those numbers when it makes claims, and the caller can then check every [S<n>] in the answer against the passage that carries that label.

def build_prompt(question: str, passages: list[dict]) -> str:
    context = "\n".join(
        CONTEXT_TEMPLATE.format(
            n=i + 1,
            source_name=p["source_name"],
            section=p.get("section") or "-",
            page=p.get("page") or "-",
            text=p["text"],
        )
        for i, p in enumerate(passages)
    )
    return USER_TEMPLATE.format(context=context, question=question)

The critical property: the answer's [S<n>] markers must map to the passages that were actually in the prompt. If your citation post-processing reads "[S3]" and looks up the third document in some other list, you have decorative citations, not verifiable ones. The audit function is short:

import re

def audit_citations(answer: str, passages: list[dict]) -> dict:
    cited = set(int(m) for m in re.findall(r"\[S(\d+)\]", answer))
    max_n = len(passages)
    invalid = [n for n in cited if not (1 <= n <= max_n)]
    return {
        "cited_indices": sorted(cited),
        "invalid_indices": invalid,
        "coverage": len(cited & set(range(1, max_n + 1))) / max_n if max_n else 0,
    }

An invalid_indices that is not empty means the model invented a citation. That is a bug, and it should be caught and either regenerated or surfaced.

Ordering the context: the middle-of-prompt trap​

Language models attend better to the beginning and the end of the context than to the middle. This is the lost-in-the-middle effect, documented on almost every model. The consequence for RAG is that the passage most likely to contain the answer should be either first or last, not fourth of six.

Two policies work in practice:

Reverse rerank order: put the top-ranked passage last, closest to the question. This is often the best when the model tends to overweight the final tokens.

Bracket the strong ones: rank 1 first, rank 2 last, the rest in the middle. This makes both ends carry a strong signal.

def order_by_bracketing(passages: list[dict]) -> list[dict]:
    if len(passages) < 2:
        return passages
    strongest, second, *rest = passages
    return [strongest] + rest + [second]

Measure on your evaluation set (module 8) which policy your specific model prefers — the answer varies by a few points depending on the model family.

The abstention instruction​

The single most important sentence in the whole prompt is the one that says do not answer if the passages do not support one. Models are trained to be helpful, and helpful behaviour under low information looks like fabrication. Explicit abstention overrides that reflex — imperfectly, but noticeably.

Three formulations, from weakest to strongest:

  1. "If you are unsure, say so." — models comply about 40 % of the time.
  2. "If the passages do not contain the answer, say you do not know." — about 65 %.
  3. "If the passages do not contain the answer, reply exactly: I do not have enough information in the provided documents to answer." — about 85 %.

Giving the model the exact string to output raises abstention compliance dramatically, because it removes the "am I supposed to say I do not know, or explain what I do know?" ambiguity. It also makes downstream detection trivial: a caller can pattern-match the abstention string and route the question to a human.

A model with no abstention rule is a liability

Ship a RAG system without a hard abstention instruction and, on questions outside the corpus, it will confidently invent policies, deadlines and signatories. The most sensitive users of such a system are the ones who will spot the fabrication first, and their trust in the tool never recovers. Explicit abstention is the single largest per-line improvement to a RAG prompt.

Contradictions between passages​

The retriever will bring back contradictory passages. Two revisions of the same procedure, both indexed. A quality policy that overrides an operational note. A general rule and its documented exception. If the prompt does not tell the model what to do, it will either pick one at random or blend them into an incoherent middle.

Two clauses handle this. First, teach the model to surface contradictions rather than resolve them silently:

- If passages disagree, quote both and explain that the sources conflict,
  giving priority to the passage with the most recent effective date visible
  in its metadata.

Second, if your corpus has a hierarchy (a policy overrides a note, an in-force procedure overrides a draft), encode it in the metadata (status: in_force, effective_date, priority_class) and mention it explicitly in the system prompt. The model will then break ties using the same rule the human editors do.

Handling long contexts and window limits​

A modern language model window comfortably fits 5 to 8 passages of 300 tokens each, plus instructions, plus a question, plus an answer — that is around 3 to 4 k tokens total. On models with smaller windows or on rare very long passages, you need a plan:

MAX_CONTEXT_TOKENS = 3000

def fit_passages(passages: list[dict], max_tokens: int) -> list[dict]:
    kept, total = [], 0
    for p in passages:
        n = n_tokens(p["text"])
        if total + n > max_tokens:
            break
        kept.append(p)
        total += n
    return kept

Cut from the bottom of the ranked list (the weakest passages), never from the middle of a single passage — a truncated passage produces an unsourced half-fact.

Putting it together​

def answer(question: str, user_context: dict) -> dict:
    passages = retrieve_and_rerank(question, user_context, rerank_keep=5)
    passages = order_by_bracketing(passages)
    passages = fit_passages(passages, MAX_CONTEXT_TOKENS)

    prompt = build_prompt(question, passages)
    reply = llm.generate(SYSTEM + "\n\n" + prompt, temperature=0.1)

    audit = audit_citations(reply, passages)
    return {
        "answer": reply,
        "citations": [
            {"index": i + 1, "source_name": p["source_name"],
             "section": p.get("section"), "page": p.get("page")}
            for i, p in enumerate(passages)
        ],
        "audit": audit,
    }

Low temperature (0.0 to 0.2) is almost always the right setting for RAG. The task is faithful summarisation of a context, not creative writing. Higher temperature adds no value here, and adds variance to the citations users compare against the sources.

In summary​

  • A RAG prompt has three blocks — system, context, question — and abstention rules that end up in the middle are ignored.
  • [S<n>] labels turn citations into a verifiable contract, and an audit function flags invented indices before they reach the user.
  • Handle the lost-in-the-middle effect by placing the strongest passages at the ends of the context, and give the model an exact abstention string to raise compliance to 85 %.
  • Handle contradictions by asking the model to surface them and by encoding recency and status in the metadata; cut long contexts from the bottom of the ranked list, never inside a passage.

Next module: measuring whether all of this actually works — recall of the retriever, faithfulness of the answer to the context, and coverage of the annotated questions.