Lesson 2 — The three training stages
Almost every confusing behaviour you have seen from a language model traces back to one of three training stages. Knowing which stage produced which behaviour is the fastest route to predicting what a model will do.
Stage 1 — Pretraining: where the capability comes from
Take trillions of tokens of text — web pages, books, code repositories, reference works, forums — and train the model to predict the next token. Nothing else. No labels, no human instruction, no notion of a task.
This stage is the expensive one, costing millions of dollars and weeks of time on thousands of accelerators, and it produces almost everything the model can do. Grammar, factual associations, reasoning patterns, programming ability, translation, style: all of it comes from here.
What comes out is called a base model, and it is genuinely strange to use. Ask a base model "What is the capital of France?" and a plausible continuation is:
"What is the capital of Germany? What is the capital of Italy?"
Because that is what such a sentence looks like in a list of quiz questions. The model is not being unhelpful; it has no concept of helpfulness. It continues text.
Two facts about this stage with lasting consequences:
The knowledge cutoff is set here. Everything the model knows about the world was in this data, which ends at a specific date.
Data quality decides quality. Later work established that filtered, deduplicated, higher-quality text produces better models at the same size than more text. It also means whatever biases and errors that text carried are now in the weights.
Stage 2 — Instruction tuning: making it usable
Now train the same model further, on a much smaller dataset of pairs: a request, and a good response to it. Tens of thousands to a few hundred thousand examples, many written by hired specialists.
This teaches the model the shape of the interaction. It learns that a question should be answered, that a request to summarise means produce a summary, that an instruction is addressed to it. The transformation in usability is dramatic and the change in underlying capability is small — this stage mostly surfaces what pretraining already built.
The result is an instruct or chat model, which is what every consumer-facing product uses.
This is also where a specific and much-misunderstood behaviour originates. The model learned that questions get confident answers, because the training examples were confident answers. It did not learn "answer confidently when you know", because the examples carry no signal about what the model knows. A large part of what looks like overconfidence was taught here.
Stage 3 — Preference alignment: shaping behaviour
The final stage uses human judgement about which of two responses is better.
The original method, reinforcement learning from human feedback, works in three steps: collect human comparisons between candidate responses, train a reward model to predict those preferences, then optimise the language model against that reward. Newer methods such as direct preference optimisation achieve similar results by learning from the comparisons directly, without a separate reward model, which is simpler and cheaper and has largely displaced the original approach in practice.
What this stage shapes:
- Tone, helpfulness, and how much explanation to give
- Refusing harmful requests
- Formatting habits, including the fondness for bulleted lists
- Admitting uncertainty, to the limited extent models do
Where each behaviour comes from
This table is the practical payoff of the lesson:
| Behaviour you observe | Which stage produced it |
|---|---|
| Knows a fact, or does not | Pretraining |
| Writes working code | Pretraining |
| Reproduces a bias | Pretraining, occasionally sharpened by alignment |
| Answers your question rather than continuing it | Instruction tuning |
| States falsehoods confidently | Instruction tuning, taught by confident examples |
| Refuses a request | Preference alignment |
| Over-refuses something harmless | Preference alignment, tuned cautiously |
| Prefers bulleted lists and hedged phrasing | Preference alignment |
| Follows "you are a helpful assistant that…" | System prompt, at request time |
What a system prompt actually is
A system prompt is text placed at the front of the context, marked as higher priority by conventions the model learned during training. It is not a configuration setting and not a security boundary — it is text in the same context window as everything else.
Two consequences worth being clear about:
It consumes your context budget. A long system prompt costs tokens on every single request.
It can be overridden by other text. Since it is text competing with other text, sufficiently insistent instructions elsewhere in the context can win. This is the basis of prompt injection, in lesson 5, and it is why treating a system prompt as an access control mechanism is a mistake with real consequences.
Why jailbreaks keep working
Alignment shapes the model's tendencies. It does not remove capabilities.
A model that has been trained to refuse a request still contains whatever knowledge would answer it, because that knowledge is in the pretrained weights and stage 3 only adjusted the disposition to use it. Refusal is a learned behaviour over a distribution of phrasings, and phrasings outside that distribution — hypothetical framings, role-play, another language, an encoding, an unusual context — can land outside where the refusal was trained.
This is why jailbreaks are patched individually and keep reappearing. The refusal is a probabilistic tendency layered on top of an unchanged capability, not a filter, and no amount of alignment work changes that structure.
Which stage do you interact with?
Almost always the aligned model, through a system prompt, and it is worth knowing your options:
Base models are available for some open families. Useful for research and for fine-tuning from scratch, unpleasant to use directly.
Instruct models are the standard. Everything hosted is one of these.
Your own fine-tune starts from an instruct model and continues training on your examples. This adjusts style, format and domain vocabulary — and, per lesson 3, is a poor way to add knowledge.
Capability comes from pretraining, usability from instruction tuning, and manner from alignment. So when a model fails, ask which stage the failure belongs to: missing knowledge is a pretraining limit that retrieval fixes, wrong format is an instruction problem that prompting or fine-tuning fixes, and unhelpful refusal is an alignment artefact that a different model may not share. A bigger model addresses only the first, and only sometimes.
In three sentences
Pretraining on trillions of tokens produces nearly all of a model's capability and sets its knowledge cutoff, but yields a base model that merely continues text and is unusable directly. Instruction tuning on perhaps a hundred thousand curated request-and-response pairs makes it answer rather than continue, and incidentally teaches the confident tone that gets mistaken for knowledge, while preference alignment from human comparisons shapes helpfulness, refusals and formatting habits. Because alignment adjusts disposition rather than removing capability, jailbreaks keep working, and because a system prompt is just prioritised text in the same context window, it is not a security boundary — which is why diagnosing a failure by asking which stage it belongs to is more productive than reaching for a bigger model.
Next — Lesson 3: retrieval →