Lesson 5 — Where it still fails
The previous lesson listed what works. This one covers what does not, because the failures are systematic rather than random, and knowing their shape lets you design around them instead of discovering them in production.
Facts are not represented
A model trained to predict likely text has no mechanism for distinguishing true from plausible. When it lacks information it does not stop; the most likely continuation of a question is an answer, so it produces one.
This is worth stating carefully because it is often described as a defect awaiting a patch. It is a property of the objective. A model optimised for likely text will produce likely text, and a fluent sentence containing an invented citation is more likely than an admission of ignorance, because admissions of ignorance are rare in the training corpus.
What this rules out:
- Anything where an invented fact is expensive and nobody verifies it
- Precise figures, dates, citations and quotations, unless retrieved from a source
- Legal, medical or financial statements presented without checking
What reduces it substantially without eliminating it: retrieval, so the answer comes from a supplied document; citations, so a human can verify; and extractive methods, which quote rather than compose.
Bias is inherited faithfully
A model trained on human text absorbs the patterns in that text, including the ones we would rather it did not.
This is measurable, not speculative. Embedding spaces reliably reproduce occupational stereotypes, associating some professions with one gender and others with another, in proportion to how the training text used them. Translation systems resolve gender-ambiguous pronouns according to the stereotype associated with the profession mentioned. Toxicity classifiers have been shown to flag text in some dialects at higher rates than equivalent text in standard usage, effectively penalising the way certain communities write.
The mechanism is not malice or carelessness in the algorithm. The model is being accurate about a biased corpus, which is harder to fix than a bug, because the bias is in the evidence rather than in the code.
Mitigations exist — curating data, adjusting the objective, filtering outputs, auditing per subgroup — and none of them resolves it. Deploying an NLP system into a decision that affects people therefore requires measuring performance per group, not only overall, because an aggregate metric hides exactly the failures that matter. The ethics course covers the governance side.
Most languages are poorly served
The field's progress has been extremely uneven.
Around twenty languages have enough digitised text, annotated datasets and commercial interest to be well supported. Several thousand others have little or none, and the consequences compound: worse translation, worse transcription, worse tokenisation efficiency — and, per lesson 2, higher cost per sentence.
The distribution of available text tracks economic and historical power, so the technology serves best those who were already best served. That this is not anyone's deliberate policy does not make it less real for the people affected.
Robustness is thin
Models are far more brittle than their benchmark scores suggest.
Small perturbations change outputs. A rephrasing that preserves meaning can flip a classification. Inserting an irrelevant sentence can change an answer. This has been demonstrated repeatedly across models and tasks.
Domain shift hurts more than expected. A model trained on news text degrades on clinical notes, legal filings or customer messages — different vocabulary, different structure, different conventions.
Adversarial inputs work. Text can be crafted to produce a chosen output, which matters for any classifier used in moderation or security, where an adversary is actively trying.
Benchmarks overstate performance
The scores you read need discounting, for three specific reasons.
Contamination. Models are trained on enormous web scrapes that frequently include the benchmark data itself. A high score may partly reflect memorisation of the test set, which is the leakage problem at internet scale, and it is genuinely difficult to rule out.
Benchmarks are cleaner than reality. Curated, well-formed, grammatical text. Your production input has typos, fragments, mixed languages, copied signatures and formatting debris.
The metric may not measure what you need. Translation quality scores compare against a reference translation, and a good translation that differs from the reference scores badly while a poor one echoing its wording scores well. Automated metrics are convenient and only loosely correlated with usefulness.
Never accept a benchmark figure as an estimate of performance on your problem. Build a small evaluation set from your own data, representative of the mess you actually receive, and measure on that. A hundred carefully chosen real examples tell you more than any public leaderboard.
What has not been solved at all
Being explicit about the frontier:
Reasoning over long documents. Retrieving a fact from a hundred pages works. Genuinely following an argument across them, noticing that page 12 contradicts page 80, remains unreliable.
Knowing what it does not know. Calibrated uncertainty — saying "I am unsure" accurately and consistently — is an open problem, and it is the one that would matter most commercially.
Common-sense physical reasoning. Questions about how objects behave, which humans answer without thought, are answered inconsistently.
Genuine compositional generalisation. Applying understood rules reliably to combinations never seen. Performance degrades in ways a system with real rules would not.
In three sentences
Models fabricate facts because they are optimised for likely text rather than true text, which is a property of the objective rather than a bug, and retrieval with citations reduces it without removing it. Bias is inherited faithfully from the training corpus and is measurable in embeddings, translation and moderation, which is why any system affecting people must be evaluated per group rather than on an aggregate metric. Benchmark scores overstate real performance because of test-set contamination, unrealistically clean data and metrics only loosely tied to usefulness, so the only trustworthy evaluation is one built from your own messy examples.
Next — Lesson 6: recap and FAQ →