Module 8 — Summarization and question answering
Two tasks that look similar and behave very differently: summarising a long review into a headline, and answering a targeted question from a passage. Both are dominated by pretrained sequence-to-sequence models today. Both fool their own metrics in ways worth knowing before you claim success.
Two families of summarizers
Extractive summarization picks sentences from the source and stitches them together. Nothing is invented; every word in the output comes from the input. TextRank (a graph-based algorithm), classical BERT-based extractors, and the sumy library all belong here.
Abstractive summarization generates a new sentence, potentially with words that never appear in the source. Models like BART, T5, Pegasus and any decoder-based LLM work this way.
The trade-off is sharp:
| Property | Extractive | Abstractive |
|---|---|---|
| Fluency | limited by source sentences | high |
| Factuality | guaranteed by construction | not guaranteed |
| Length compression | modest | strong |
| Domain adaptation | robust | can hallucinate outside its training domain |
On customer reviews, an extractive summary of The battery lasts twelve hours and the screen is bright, but the charger overheats after twenty minutes might be The charger overheats after twenty minutes — factual, but not the whole picture. An abstractive summary might be Great screen and battery, defective charger — more useful, at the risk of inventing defective where the review only said overheats.
An extractive baseline in ten lines
from sumy.parsers.plaintext import PlaintextParser
from sumy.nlp.tokenizers import Tokenizer
from sumy.summarizers.text_rank import TextRankSummarizer
def summarise_textrank(text: str, n_sentences: int = 2) -> str:
parser = PlaintextParser.from_string(text, Tokenizer("english"))
summariser = TextRankSummarizer()
sentences = summariser(parser.document, n_sentences)
return " ".join(str(s) for s in sentences)
Run it on a batch of long reviews and read a few. Two limitations show up immediately. Short reviews (under 3 sentences) get returned as-is. And the algorithm has no notion of what is a topic sentence versus a supporting detail — a review that opens with As I was saying to my brother last weekend will place that sentence in the summary.
An abstractive model in a pipeline
from transformers import pipeline
summariser = pipeline("summarization", model="sshleifer/distilbart-cnn-12-6")
def summarise_bart(text: str, max_length: int = 60) -> str:
if len(text.split()) < 30:
return text # nothing to summarise
out = summariser(text, max_length=max_length, min_length=15, do_sample=False)
return out[0]["summary_text"]
distilbart-cnn-12-6 is a distilled BART trained on CNN/DailyMail news. On news it produces solid summaries; on customer reviews it works but shows its origin — it sometimes starts summaries with The article or reaches for a formality reviews rarely have. This is a common trap: a summariser trained on news is not a summariser trained on your text. For a production system on reviews, fine-tune on a small labelled set (a few hundred examples is enough) rather than accept the news style.
ROUGE, and why it lies
The standard metric for summarization is ROUGE — a family of recall-based scores measuring overlap between the model's summary and a human reference.
- ROUGE-1 counts overlapping unigrams (words).
- ROUGE-2 counts overlapping bigrams (two-word sequences).
- ROUGE-L counts the longest common subsequence, allowing gaps.
from rouge_score import rouge_scorer
scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
scores = scorer.score(reference, hypothesis)
The problem: ROUGE is a surface-word overlap metric. It cannot tell the difference between the charger is defective and the charger is not defective — they share four words out of five, and ROUGE gives a high score. A summariser that reliably drops the word not will score well on ROUGE and be catastrophic in production. Reviews with sarcasm (amazing, my phone died in two days) receive extractive summaries that keep the word amazing and score well; they are wrong summaries.
Two workarounds are worth knowing. BERTScore replaces word overlap with cosine similarity between contextual embeddings; it catches paraphrases better and still misses negation flips. Human review on a sample remains the honest evaluation for anything deployed to customers. Publish both ROUGE and BERTScore, and read 30 summaries yourself before shipping.
Extractive question answering: pull the span
QA has two main flavours. In extractive QA, the answer is a contiguous span in a given passage; the model returns the start and end indices. This is what SQuAD introduced and what most Hugging Face question-answering pipelines do.
qa = pipeline("question-answering", model="distilbert-base-cased-distilled-squad")
context = """The battery of the phone lasts twelve hours in normal use.
It charges to 80 % in 45 minutes with the included charger,
but the MagSafe charger overheats after twenty minutes."""
for q in ["How long does the battery last?",
"What happens with the MagSafe charger?"]:
a = qa(question=q, context=context)
print(f"{q}\n -> {a['answer']} (score={a['score']:.2f})")
Because the answer is a span from the source, extractive QA cannot hallucinate: the worst it can do is return the wrong span, and a low score flags that case. For enterprise applications where a wrong answer must never be invented, extractive QA is the safe choice.
Abstractive QA and generative models
Abstractive QA generates the answer as free text, possibly rephrasing the source. It reads better, and it can compose an answer that spans multiple passages. It is also where hallucination enters: the model can produce a fluent, confident sentence that has no basis in the source. Retrieval-augmented generation (RAG) mitigates this by grounding the model on retrieved passages (module 5 gave you the retrieval part).
Two failure modes are worth naming. Context-free confabulation: the model answers from what it saw during training rather than from the provided passage, and confidently so. Attribution loss: even when the answer is correct, the user cannot verify which sentence in the source supports it. A production system that generates answers should always show the source spans that fed it — this is a UI decision as much as an ML one.
Fluency and factuality are separate axes. Modern models optimise the first; only careful evaluation ties them to the second. Deploy an abstractive summariser only after a human read of at least 30 outputs from your own domain.
For any first version on a customer-facing product — support ticket summarisation, review digest, question answering on documentation — extractive methods carry zero hallucination risk and deliver 80 % of the value. Move to abstractive only when the extractive summary is measurably too long or too disjointed for the use case.
In summary
- Extractive summarization guarantees factuality by construction but caps compression and fluency; abstractive models write better and can hallucinate details that are not in the source.
- ROUGE measures word overlap and misses the two errors that matter most: negation flips and paraphrases; combine it with BERTScore and a human read of a sample.
- Extractive QA returns a span from the source, cannot invent an answer, and comes with a confidence score; the safe default for enterprise QA.
- Abstractive QA and generative models hallucinate; grounding them on retrieved passages (RAG) and showing the source spans are the two mitigations you owe your users.
Next module: the Transformers library that ties everything you have seen since module 5 into one API.