#tokenization — LLMs and transformers

Tokenization and BPE: how an LLM chops up text.

What you'll play with

Welcome to #tokenization. On screen, the sentence "The neural networks learn by adjusting their weights." split by a BPE tokenizer trained on a mini English corpus, after 30 merges: each tile is a token, the unit the model actually reads. On the right, the counter. An LLM never sees letters or words, only these numbered chunks in a vocabulary, and it is their count that sets the price of a request and what fits inside its context window.
Let's start with the most naive split: one token per character. Type /method chars.
The opposite: one token per word. /method words.
BPE starts from characters and learns to glue the corpus's most frequent pairs. Reset the merge counter to zero: /merges 0.
Apply the first merge learned on the corpus: /merge.
Repeat forty times and you get a real tokenizer: /merges 40.
Let's stress it on a sentence full of accented loanwords: /example accents.
Turn that into a bill: /count.
Your turn: /text … with your own sentence, /example code (Python facing an English tokenizer), /language fr (French corpus: watch English words become badly split), /vocabulary (the 20 most frequent tokens), /merges 60 to push BPE all the way, /reset to start over. Next stop: the #attention channel, where these tokens look at each other to give each other meaning.

Channel commands

/text <word> — Replace the text to tokenize (the whole line, 120 characters at most).
/method <chars|words|bpe> — Change how text is split: by character, by word or by BPE sub-words.
/merges <0..60> — Number of BPE merges applied to the text (switches to bpe method).
/merge — Apply one more BPE merge and highlight the merged pair.
/language <fr|en> — Corpus BPE is learned on: English (40 sentences) or French (42 sentences).
/vocabulary — Size of the current vocabulary and its 20 most frequent tokens in the corpus.
/count — Characters, tokens, ratio and estimated cost if 1 token = 1 billing unit.
/example <short|long|accents|code> — Load a preset text: short, long, full of accents, or code.
/reset — Return to the initial sentence, BPE, 30 merges, English corpus.

Glossary

Token: The unit a language model reads and produces: a chunk of text (letter, word fragment, whole word, sign) numbered in a vocabulary. An LLM never sees letters or words, only a sequence of token numbers. The price of a request and the size of the context window are counted in tokens.
Tokenization: The step that turns a text into a sequence of tokens before entering the model, and glues the produced tokens back into text at the output. Per character: very long sequences; per word: open vocabulary and unknown words; per sub-word (BPE, WordPiece): the compromise adopted by every LLM.
BPE (byte-pair encoding): A sub-word tokenization algorithm: start from characters (or bytes), count every pair of adjacent symbols in the corpus, merge the most frequent one, and repeat a fixed number of times. The learned merges are then applied in the same order to any new text. Used by GPT, Llama, Mistral.
Vocabulary: The table of all tokens a model knows, each with its number. Its size is a design choice (32,000 to 200,000 entries for current LLMs): the larger it is, the shorter texts become in tokens, but the heavier the embedding table and output layer.
Sub-word: A token smaller than a word: a root, a prefix, an ending ("re", "er", "ing·"). A frequent word is a single sub-word, a rare or invented one is split into several: the model can then read any word without ever hitting an unknown.
Unknown token (UNK): A placeholder symbol that a word-level tokenizer emits for any word missing from its vocabulary: the information is lost. Sub-word tokenizers avoid it by falling back to characters, and byte-level tokenizers remove it altogether (256 bytes are enough to write anything).
Context window: The maximum number of tokens a model can take into account at once (question, provided documents and answer included): 8,000, 128,000, sometimes over a million. It is measured in tokens, never in words: a poorly tokenized text (rare language, code, accents) consumes more of it.
End-of-word marker: A special symbol appended to the last character of each word before learning BPE ("" in Sennrich, shown "·" here). It distinguishes "es" in the middle of a word from "es" at the end, and lets us glue the text back without storing spaces. SentencePiece does the opposite with a start-of-word marker "▁".
WordPiece / SentencePiece: Two cousins of BPE. WordPiece (BERT) picks the merge that most increases the corpus likelihood rather than the most frequent one, and prefixes internal sub-words with "##". SentencePiece (T5, Llama) treats raw text, spaces included, as a sequence of symbols: it does not need to pre-split into words, which suits languages without spaces.
Token cost: What a text consumes, measured in tokens: the billing unit of LLM APIs and what fills the context window. The chars-per-token ratio (≈ 4 in English, less in French or in code) tells how much a text costs for the same length: a tokenizer trained mostly on English makes other languages more expensive.

Other channels in LLMs and transformers

#tokenization — Tokenization and BPE: how an LLM chops up text.
#attention — Self-attention: each word looks at the others.
#transformer-block — The transformer block: attention, residuals, normalization, FFN.
#generation-temperature — Generation: temperature, top-k and top-p.
#rag — RAG: answering from your documents.
#alignment-rlhf — Alignment and RLHF: learning what humans prefer.