Lesson 2 — Tokenisation
Before a model sees any text, the text is cut into pieces. This step looks like plumbing and it has consequences that surface constantly: it explains why you are billed in tokens, why models miscount letters, and why the same sentence costs twice as much in one language as another.
The obvious approach, and why it fails
Split on spaces. Each word becomes a unit.
It falls apart immediately:
The vocabulary explodes. English has hundreds of thousands of word forms, and every one needs its own entry and its own learned vector. Most appear too rarely to learn anything reliable about them.
Unknown words are fatal. "Tokenisation" not in your dictionary? The model sees nothing at all. And as lesson 1 noted, the vocabulary never closes: new names, products and slang arrive continuously.
Related forms are unrelated. "Run", "runs", "running" and "ran" become four independent entries with no connection, so anything learned about one teaches nothing about the others.
Many languages have no word spaces. Chinese and Japanese do not separate words. Splitting on spaces gives you one enormous token per sentence.
Compounding languages break it entirely. German forms arbitrarily long compounds, so the set of possible words is effectively unbounded.
Subword tokenisation: the compromise that works
The solution now used everywhere is to work with subwords: keep frequent words whole and split rare ones into meaningful fragments.
So a sentence might become:
"Tokenisation is straightforward"
-> ["Token", "isation", " is", " straight", "forward"]
"is" is common enough to be one token. "Tokenisation" is rare, so it is built from pieces that recur across many words. Nothing is ever unknown, because in the worst case a word decomposes into individual characters.
This gives three properties at once:
- A fixed vocabulary, typically 30,000 to 100,000 tokens
- No unknown words ever, since any string can be assembled from pieces
- Shared structure between related forms, because "run" and "running" share the "run" fragment
The algorithms you will see named — byte-pair encoding, WordPiece, SentencePiece — differ in how they choose which fragments to keep. They all begin from characters and repeatedly merge the most frequent adjacent pair until the vocabulary reaches the target size. The merges are learned from a training corpus, which matters more than it sounds: the tokeniser carries the biases of the text it was built from.
Why you are billed in tokens
This is where tokenisation becomes commercially visible. Model providers charge per token because tokens are the unit the model processes, and the rough conversion for English is that one token is about three-quarters of a word, or roughly four characters.
A useful rule: 1,000 tokens is around 750 English words, or a page and a half.
Two practical consequences.
Costs and limits are in tokens, not words. A context window of 128,000 tokens is around 96,000 English words. Your estimate in pages needs converting.
The exchange rate varies by language, considerably. Tokenisers are trained mostly on English text, so English gets efficient tokens. Languages written in other scripts, or with rich morphology, get split into many more pieces for the same meaning.
| Language | Approximate tokens for the same content |
|---|---|
| English | baseline |
| French, Spanish, German | 1.2 to 1.5 times more |
| Arabic, Russian, Hindi | 2 to 3 times more |
| Chinese, Japanese | 1.5 to 2.5 times more |
The same question costs two or three times more to ask in Arabic or Hindi than in English, and fits in a proportionally smaller context window. This is not a policy decision by anyone; it falls out of tokenisers trained predominantly on English text. It is nonetheless a real disadvantage for non-English users, and it is one of the quieter ways technical choices distribute cost unevenly.
Why models are bad at counting letters
A recurring surprise: ask a language model how many times a letter appears in a word and it often gets it wrong, despite being able to write a passable essay.
Tokenisation is the explanation. The model does not see letters. It sees token identifiers — numbers standing for fragments. If a word arrives as two tokens, the model has no direct access to its individual characters, only to a representation of the chunks.
The same reason explains difficulty with rhymes, spelling puzzles, reversing strings and character-level arithmetic. These are not reasoning failures; they are consequences of the input representation, which is why prompting tricks rarely fix them and inserting spaces between letters sometimes does.
The pipeline
The step from token IDs to embeddings is lesson 3, and it is where meaning enters. Until then, a token ID is an arbitrary number: token 3797 is not more similar to 3798 than to 15. All the semantics live in the vectors.
Tokenisation explains a cluster of behaviours that otherwise look arbitrary: token-based pricing, context limits that do not match your page count, worse economics for non-English languages, and unreliable character-level tasks. When a model behaves strangely on something involving spelling, tokenisation is the first place to look.
In three sentences
Splitting text on spaces fails because vocabularies explode, unknown words are fatal, related forms lose their connection, and many languages have no word spaces. Subword tokenisation keeps frequent words whole and splits rare ones into recurring fragments, which gives a fixed vocabulary that can represent any string including words never seen. It also explains token-based pricing, why the same content costs two or three times more in Arabic or Hindi than in English, and why models are unreliable at counting letters.