Module 2 — Pretraining: data, tokens and scaling laws

Module 1 gave the shape of the field. This one goes inside the object itself: what a pretrained large language model is actually made of, and how the two knobs — how much data, how many parameters — trade off. For the customer-support assistant of the running project, this is what tells you whether the 7 to 8 billion parameter base model you plan to fine-tune has been trained on enough of your language.

A pretraining corpus, in one honest sentence

A modern LLM is trained on a filtered mixture of web pages, books, code and dialogue transcripts, adding up to trillions of tokens. The mixture matters more than the total. Common Crawl alone — the raw dump of the public web — is noisy enough to lower the score of a model trained on it directly. Every serious pretraining pipeline spends most of its engineering budget on filtering that dump.

Four filtering steps are near-universal:

1. Language identification, to keep or split by language.
2. Quality classifiers, often trained on Wikipedia versus a random sample of the web, to drop the bottom decile of pages.
3. Deduplication, exact and fuzzy, across documents and across sequences within them.
4. Toxicity and personal information filters, usually rule-based plus a small classifier.

The recipe that shipped Llama 3 devotes tens of thousands of GPU-hours to filtering alone, before a single pretraining step. This is not decorative work: a corpus that repeats the same paragraph a hundred times teaches the model to memorise, not to generalise.

Deduplication is what protects generalisation

Duplicates in the training set look harmless but are a leading cause of memorisation. The classic experiment: train two models on the same corpus, one deduplicated, one not. The deduplicated model reaches the same validation loss with fewer parameters and produces less verbatim output at generation time.

The standard technique is MinHash on n-grams, cheap enough to run on the whole corpus:

from datasketch import MinHash, MinHashLSH

def signature(text, num_perm=128):
    m = MinHash(num_perm=num_perm)
    for gram in {text[i:i + 5] for i in range(len(text) - 4)}:
        m.update(gram.encode("utf-8"))
    return m

lsh = MinHashLSH(threshold=0.8, num_perm=128)
for doc_id, text in enumerate(corpus):
    lsh.insert(doc_id, signature(text))

duplicates = {d for d, _ in lsh.query_pairs()}

Two documents whose 5-gram sets overlap by more than 80 % are treated as duplicates. On Common Crawl-scale corpora this removes 30 to 50 % of raw tokens.

The Chinchilla scaling laws in one paragraph

Before 2022, the received wisdom was: given a compute budget, spend it on parameters. The 2022 Chinchilla paper by Hoffmann et al. showed the reverse. For a fixed compute budget $C$, expressed in floating-point operations, the loss-optimal split is:

$C \approx 6 \, N \, D$

where $N$ is the number of parameters and $D$ is the number of training tokens, with roughly $D \approx 20\,N$ at the optimum. In words: for every parameter, you want about twenty training tokens.

GPT-3, at 175B parameters, was trained on only about 300B tokens — a ratio of 1.7 tokens per parameter, well below the Chinchilla optimum. A model of the same compute cost, but smaller and trained on more data, would score higher.

Chinchilla is loss-optimal, not deployment-optimal

The Chinchilla law minimises training loss for a given training cost. It says nothing about inference cost. A model that is twice as small and trained on twice as many tokens is cheaper to train and cheaper to serve. This is why every open model released since 2023 sits at or well past the Chinchilla ratio — Llama 3 at 8B was trained on 15T tokens, a ratio of nearly 2000.

Compute, in numbers you can quote

The compute cost of a pretraining run is dominated by the forward-and-backward pass over the training tokens. A useful order of magnitude:

• Total FLOPs $\approx 6 \, N \, D$.
• Wall-clock hours on an H100 (peak ~1 PFLOP/s, ~40 % utilisation) $\approx \frac{6 \, N \, D}{0.4 \cdot 10^{15} \cdot 3600}$.
• Electricity at 700 W per H100.

For a 7B model trained on 2T tokens: $6 \times 7 \cdot 10^9 \times 2 \cdot 10^{12} = 8.4 \cdot 10^{22}$ FLOPs. That is roughly 58,000 H100-hours at 40 % utilisation, or about 20 GPUs for four months, or a two-week job on a thousand-GPU cluster. Electricity alone runs into hundreds of megawatt-hours. This is why almost no one pretrains from scratch; almost everyone fine-tunes.

The language share problem

Every corpus is dominated by English. Even the best open pretraining mixtures allocate 85 to 95 % of tokens to English, with the remainder split across dozens of languages. Two consequences follow.

First, a base model's fluency in French, Arabic or Vietnamese is roughly proportional to the number of tokens seen in that language — often two to three orders of magnitude less than English. Second, benchmarks tell you almost nothing about that fluency: MMLU is English, HumanEval is English, MT-Bench used to be English-only.

For the customer-support assistant of the running project, this means the shortlist changes as soon as you require French or Arabic output at production quality. Models that never saw much of your language during pretraining cannot be fixed by fine-tuning on a few thousand support tickets — that is the topic of module 3, and one of the harder lessons of it.

Estimate your language share before shortlisting

Most open-model cards publish the language mix. If yours is below 1 % of pretraining tokens, expect awkward output, especially on rare morphological forms. Prefer a model whose card mentions the target language explicitly, even at a smaller parameter count.

In summary

• Pretraining data is a filtered mixture of web, books, code and dialogue at trillion-token scale; the filtering budget rivals the training budget itself.
• Deduplication on n-gram MinHash removes 30 to 50 % of raw web tokens and is what prevents memorisation from dominating generation.
• The Chinchilla law ($C \approx 6\,N\,D$, with $D \approx 20\,N$ at the training optimum) says compute is best spent on more tokens, not more parameters — and inference cost pushes the ratio further towards data.
• The language share of pretraining decides how far fine-tuning can take you; check the model card before shortlisting for a non-English project.

Next module: how to turn one of these base models into an assistant that actually follows instructions.