Skip to main content

Module 7 — Hallucinations: causes and remedies

A hallucination, in the LLM sense, is a confidently stated output that is not supported by any source and turns out to be wrong. The customer-support assistant of the running project cannot ship without a serious answer to this problem: a made-up refund policy quoted in a polite tone is worse than an outright error message, because it costs money and trust.

This module explains why hallucinations happen and, more importantly, which countermeasures actually reduce them versus which ones only feel like they do.

Why a model invents in the first place​

Three mechanisms explain most hallucinations, and they compound.

Cause 1 — Training objective. The model was trained on next-token prediction, which rewards producing a plausible continuation, not a true one. When the correct answer is not in the model's parameters, the objective still pushes it to emit something, and the most-probable something is often statistically plausible while being factually wrong.

Cause 2 — Compression of knowledge. A 7B model stores its knowledge in ∼1010\sim 10^{10} floating-point weights, compressed billions of pretraining tokens down to what those weights can hold. Lossy compression drops details, particularly for rare facts — names, dates, numeric values, proper nouns. The model retains the shape of the fact ("the CEO of company X is a person") and confabulates the content ("their name is Y").

Cause 3 — Alignment amplifies confidence. Instruction tuning and RLHF (modules 3 and 4) push the model towards helpful, declarative outputs. Hedged answers score lower in preference data than confident ones. As a side effect, alignment often makes hallucinations more likely, not less, because it trains the model to answer even when it should say "I do not know".

The important consequence: you cannot fine-tune your way out of hallucinations at scale. Fine-tuning changes style, not what the model actually knows.

Grounding: the only structural fix​

The single most effective countermeasure is to give the model the answer as input and constrain it to work only from what is given. This is the essence of retrieval-augmented generation (module 6, and the whole of course 18).

prompt_template = """Answer the question using ONLY the information in the sources below.
If the sources do not contain enough information, reply exactly: I do not know.

Sources:
{sources}

Question: {question}
Answer:"""

def grounded_answer(question, sources, model):
    prompt = prompt_template.format(
        sources="\n---\n".join(f"[{i+1}] {s}" for i, s in enumerate(sources)),
        question=question,
    )
    return model.generate(prompt, temperature=0.1, max_new_tokens=300)

Two properties make this work: the model gets the actual answer text as input, and the prompt gives it a licence to say "I do not know". Both are necessary. Without the sources it will guess; without the abstention licence it will still try to answer.

Reported reductions on public benchmarks (TriviaQA, NaturalQuestions with retrieval) are dramatic — often 60 to 80 % fewer factual errors — provided the retrieval itself is good. Bad retrieval hurts as much as no retrieval: the model treats irrelevant passages as evidence.

Citations: shift the burden of proof​

A step further: ask the model to cite the source of every non-trivial claim, as a bracketed index into the sources block.

Answer: Our current return window is 30 days [1], during which the item must
be unused and in its original packaging [1][3]. Refunds are issued to the
original payment method within 5 business days [2].

Two benefits. First, unsupported claims stand out as uncited sentences, easy to catch automatically and easy for a human reviewer to spot. Second, the model self-regulates: a model asked to cite tends to say less, because it will not fabricate a citation as readily as it will fabricate a fact.

The check is cheap:

import re

def unsupported_sentences(answer):
    return [s for s in re.split(r"(?<=[.!?])\s+", answer) if not re.search(r"\[\d+\]", s)]

Any sentence returned by this function needs a human look before shipping.

Abstention: the answer that is not there​

The model must have an explicit permission to answer "I do not know". Many production prompts forget this and get a confidently wrong answer where they wanted silence. A minimal upgrade to any system prompt:

"If you are not certain from the information provided, say exactly I do not know. Do not attempt to answer from general knowledge."

This works on aligned models; on base models it does not. The alignment stage (module 4) is what makes them capable of stopping at that instruction. Test it before shipping: send ten questions whose answer is not in the sources, and count how many produce "I do not know" versus a guess.

Verification by a second model​

The judge pattern: after the first model produces an answer, a second model (often the same weights, sometimes a cheaper one) checks it against the sources.

verifier_prompt = """Given the sources below and the answer, list any claim in
the answer that is not directly supported by the sources. If everything is
supported, respond exactly: OK.

Sources: {sources}
Answer: {answer}

Unsupported claims:"""

If the verifier returns anything other than "OK", the answer is rejected, revised, or flagged for human review. This roughly halves the residual hallucination rate on top of grounding and citations, at a doubled inference cost.

A verifier is not a fact-checker

The verifier only checks that claims are supported by the given sources. If the sources themselves are wrong, so is the verifier. And a verifier prone to hallucinating will happily invent that a claim is "supported". Use verification on top of grounding, never as a replacement.

What does not work, and why it feels like it should​

Three intuitive fixes fail in measurable ways:

  • "Just tell the model to be honest." Adding "Do not lie" to the system prompt has no measurable effect on hallucination rate. The model is not lying; it does not know the difference between a memorised fact and a plausible confabulation.
  • "Fine-tune on the correct answers." Fine-tuning teaches style. It does not repair the compression losses of pretraining. Feed 10 000 correct product descriptions and the model will still invent the 10 001st.
  • "Increase temperature so it explores less." Lower temperature (down to 0) reduces creative errors but increases confident wrong answers, because the same wrong-but-plausible token is picked deterministically every time.
Track hallucination rate as a first-class metric

Build an evaluation set of 200 questions with known correct answers (or known "no answer available") and measure the raw hallucination rate every time you change the model, the retriever or the prompt. Anything you do not measure will drift.

In summary​

  • Hallucinations stem from three compounding causes: the next-token objective, lossy compression of knowledge, and alignment that rewards confident answers.
  • Grounding on retrieved sources plus licence to abstain is the only structural fix; both are necessary.
  • Citations turn unsupported claims into detectable objects and make the model self-regulate; a verifier on top halves what remains at double cost.
  • Telling the model to be honest, fine-tuning on correct answers, and lowering temperature do not solve hallucinations, however intuitive they sound.

Next module: quantization and cost-efficient serving. Once the model behaves correctly, the next question is how to run it at a price the project can absorb.