Module 2 — Tokenization: words, subwords, BPE, SentencePiece

Module 1 cleaned the reviews. This module cuts them into pieces. The choice of tokenizer decides three things at once: the size of your vocabulary, the number of tokens per document (which is what you pay when you call a model), and the model's ability to read a word it has never seen before.

Word tokenization and the vocabulary trap

The obvious approach splits on whitespace and punctuation. The battery lasts 12 hours! becomes six tokens: [The, battery, lasts, 12, hours, !]. Simple, fast, works fine on a TF-IDF baseline.

The problem is the vocabulary. On the ten thousand English reviews of the course corpus, a whitespace tokenizer produces roughly 25 000 distinct tokens. Half of them appear only once: brand names, misspellings, plurals of technical terms. Any word not in this vocabulary at inference time becomes an <UNK> token, and the classifier throws away its signal.

The blunt fix — enlarge the vocabulary to a million entries — is worse. Embedding matrices are sized vocabulary × dimension; a million-word vocabulary with 300-dimensional embeddings costs 1.2 GB, and 90 % of that memory holds words the model sees once or never.

Subwords cut the problem in half

The idea, introduced with BPE (Byte Pair Encoding) and refined by WordPiece and SentencePiece, is to keep a small vocabulary of frequent pieces. Common words stay whole. Rare words are decomposed into their frequent subparts. Nothing is ever unknown, because at worst a word is spelled out one character at a time.

On the same corpus, a 30 000-entry BPE tokenizer represents every word, including brand names invented tomorrow, and the average review costs about 35 tokens instead of 25 whole words.

BPE step by step on twenty words

Here is BPE applied by hand to a toy vocabulary. It is worth doing once with paper, because it removes all mystery from what the tokeniser actually stores.

from collections import Counter

corpus = "low low low low low lower lower newest newest newest newest newest widest widest widest"
counts = Counter(corpus.split())
# {"low": 5, "lower": 2, "newest": 5, "widest": 3}

def split_to_chars(word):
    return list(word) + ["</w>"]           # </w> marks a word boundary

Start with a vocabulary of single characters. At each step, count every adjacent character pair across all words, weighted by word frequency, and merge the most common pair into a new token.

Iteration 1: the pair (e, s) appears 8 times (5 in newest, 3 in widest). Merge to es. Vocabulary gains one entry.

Iteration 2: the pair (es, t) appears 8 times. Merge to est.

Iteration 3: the pair (est, </w>) appears 8 times. Merge to est</w>.

Continue for a fixed number of merges — say, 30 000 in a real tokenizer. What you store on disk is the ordered list of merges, nothing else. To tokenize newest, apply the merges in order to n e w e s t </w>: es, then est, then est</w>, and stop at three tokens n, ew, est</w>.

The frequent pieces of English — endings like ing, tion, est, prefixes like un, re, pre — emerge on their own from the statistics of the training corpus. No linguist wrote them down.

WordPiece and SentencePiece: two variations

WordPiece (used by BERT) makes one change to BPE: instead of picking the most frequent pair at each step, it picks the pair that most improves the likelihood of a language model over the training corpus. Same output format, slightly different merges. Its tokens keep the ## prefix to mark a piece that continues a word: tokenization → token, ##ization.

SentencePiece (used by T5, XLM-RoBERTa, most multilingual models) works directly on raw text as a stream of bytes, without treating spaces as a special separator. The space becomes a normal character encoded as ▁ (U+2581) at the start of each word. This has one decisive consequence: SentencePiece works the same way on Chinese, Thai and Japanese (which don't use spaces) as on English. It is the tokenizer of choice for multilingual models.

Token cost varies wildly across languages

This is the point that surprises most teams and dominates any budget calculation on a paid API. The same review does not cost the same in English, French and Arabic. On a standard BPE tokenizer trained mostly on English (GPT-style):

Language	Text	Tokens
English	the battery lasts twelve hours	6
French	la batterie tient douze heures	8-9
Arabic	البطارية تدوم اثنتي عشرة ساعة	20-25

The tokenizer breaks Arabic script into two- or three-byte pieces because it has seen very little of it during training. A monolingual Arabic tokenizer would cut the same phrase into 5 or 6 tokens. This is why the same conversation with an English-first model costs three or four times more in Arabic than in English, both in latency and in dollars.

Never train a downstream model on tokens from a different tokenizer than the one you'll serve with

Fine-tuning on bert-base-uncased tokens and serving on bert-base-cased produces silent nonsense: the vocabulary indices point to different pieces. The AutoTokenizer and AutoModel pairs of module 9 are named after the model precisely to prevent this mistake — but you can still cross them by hand.

Measuring tokenizer cost on your corpus

Before you commit to a model, run its tokenizer on a sample and count. It takes ten lines and it can change your architecture.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
sample = load_reviews(n=1000)                # your own loader

lengths = [len(tok(r).input_ids) for r in sample]
print(f"median tokens per review: {sorted(lengths)[len(lengths)//2]}")
print(f"max: {max(lengths)}")

The result tells you two things. The median decides how many tokens you pay for at inference; the max decides whether you have to truncate. BERT-base has a limit of 512 tokens; a T5 or Longformer variant reaches thousands, at a price. If 5 % of your reviews exceed 512 tokens, you must either truncate them (losing the end, which is often the punchline of a review) or move to a model that reads longer sequences.

Fixed vocabulary, learned indices

Two different tokenizers with the same size (30 522 entries in BERT versus 32 000 in T5) index completely unrelated pieces. The vocabulary size is a shape parameter, not a semantic one; only pairs are safe.

In summary

• Word tokenization is simple but produces a huge vocabulary of which most entries appear once; the tail becomes <UNK> and its signal is lost.
• Subword tokenization (BPE, WordPiece, SentencePiece) keeps a small vocabulary of frequent pieces; any word, including neologisms, becomes a sequence of known tokens.
• Token cost is language-dependent: an English-first tokenizer breaks Arabic script into three or four times more tokens than a monolingual Arabic tokenizer, and the invoice follows.
• Always pair the tokenizer with its model; measure the median and max tokens per document on your own corpus before deciding on a maximum length or a budget.

Next module: TF-IDF and the logistic-regression baseline you have to beat before considering anything more expensive.