Lesson 1 — Why language is hard
Computers handled arithmetic before they handled language, by a margin of decades. Not because language involves harder computation, but because it violates almost every property that makes something computable. Five properties, specifically.
1. Ambiguity at every level
The same sequence of characters routinely means several things, and every layer of language contributes its own ambiguity.
Words: "bank" is a financial institution or the side of a river. "Set" has over four hundred recorded senses.
Structure: "I saw the man with the telescope." Did you use the telescope, or did he have one? Both readings are grammatically valid, and the sentence does not resolve which.
Reference: "The trophy would not fit in the suitcase because it was too big." What was too big? Every human says the trophy instantly, and nothing in the grammar says so — you resolved it using knowledge about how objects and containers work.
Scope: "Every student read a book." One book between them, or one each? Both are legitimate.
Human listeners resolve these so fast that the ambiguity is invisible, which is precisely why they are hard to program around: we cannot state the rule we used.
2. Meaning depends on context
The same words mean different things depending on who says them, to whom, when, and after what.
"That's just great."
Delight or exasperation, decided entirely by circumstance. And context is not only the surrounding sentence: it is the preceding conversation, the relationship between speakers, the culture, the physical situation, and what both parties already know.
This is why systems that process a sentence in isolation hit a ceiling, and why the arrival of models that consider long stretches of surrounding text was such a step change.
3. Most of the meaning is not stated
"Can you pass the salt?"
Literally a question about ability. Answering "yes" and doing nothing would be technically correct and socially absurd. This is pragmatics: what is meant beyond what is said.
"Some of the students passed."
Strictly this is compatible with all of them passing. Every listener infers that some did not, because we assume a speaker who knew all had passed would have said so.
Human communication rests on shared assumptions that are never stated because stating them would be tedious. A system with no model of speaker intent misses a substantial layer of meaning while parsing every word correctly.
4. Endless variation for the same content
There is no canonical form. All of these convey roughly one thing:
- "The delivery arrived late."
- "My package showed up after the promised date."
- "parcel came 3 days late, furious"
- "Order #4471 — delivery SLA missed."
A system that must recognise "late delivery" has to handle formal prose, casual writing, abbreviations, typing errors, domain jargon and internal reference codes. Real text is messier than any dataset suggests, and the gap between clean benchmark text and actual customer messages is where many projects discover their model is fragile.
5. The vocabulary never closes
New words appear constantly: brand names, technical terms, slang, proper nouns, deliberate misspellings. No fixed dictionary stays complete, which means any system built around one degrades from the day it ships. Lesson 2 shows how modern tokenisation works around this.
Why grammar rules failed
For decades the reasonable approach was to encode linguistic rules: a grammar, a lexicon, a parser. Linguists had described language in enormous detail, so transcribing that description looked like engineering work.
It did not scale, for reasons that now look inevitable:
Exceptions outnumbered rules. Every rule attracted a list of exceptions, and the exceptions had exceptions.
Ambiguity multiplied. A grammar capable of parsing real sentences produces many valid parses for each, and choosing between them requires exactly the world knowledge the grammar does not contain.
Real text is not grammatical. People write fragments, run-ons and typos. A parser demanding well-formed input fails on most of what exists.
Each language needed rebuilding. No effort transferred between them.
The statistical turn reframed the problem: rather than deciding what a sentence means, predict which interpretation is most likely given enormous quantities of observed text. Ambiguity stops being an obstacle to eliminate and becomes a probability to estimate.
The winning insight was not a better theory of language. It was accepting that the regularities of language can be estimated from data even when they cannot be stated as rules — the same move that made vision work, applied to text. Everything since has been about representing context better and training on more of it.
In three sentences
Language resists computation because it is ambiguous at every level, because meaning depends on context extending far beyond the sentence, and because most of what is communicated is inferred rather than stated. It also varies endlessly in form and never closes its vocabulary, so any system built on a fixed dictionary decays from launch. Rule-based grammars failed because exceptions outnumbered rules and real text is not grammatical, and the statistical turn succeeded by estimating likely interpretations from data instead of deducing correct ones from theory.
Next — Lesson 2: tokenisation →