Lesson 1 — What makes a model large
A language model predicts the next token. Language models have done this since the 1990s, using word counts and simple statistics, and they were useful for spelling correction and not much else.
The architecture changed in 2017 with the transformer. Then, more consequentially, the scale changed.
Parameters, and what the numbers mean
A parameter is a single number inside the network, adjusted during training. It is the unit in which these models are measured.
| Model era | Parameters | What it could do |
|---|---|---|
| 2018, first transformers | ~100 million | Fluent short text, weak coherence |
| 2019 | ~1.5 billion | Paragraph coherence, simple tasks |
| 2020 | ~175 billion | Follows examples in the prompt, broad tasks |
| 2023 onwards | Hundreds of billions to trillions | Multi-step reasoning, code, tool use |
| Small open models today | 3 to 30 billion | Much of the above, on a laptop |
Note the last row. Parameter count stopped being a clean proxy for capability once training data quality, training duration and post-training methods improved. A well-trained 8-billion-parameter model from 2026 outperforms a 175-billion-parameter model from 2020 on most tasks, which is why quoting parameter counts as a specification has become mostly marketing.
Three things scaled together, and all three mattered:
Parameters — capacity to store patterns. Data — trillions of tokens of text, roughly the readable public internet plus books and code. Compute — the training runs cost millions of dollars in electricity and hardware time.
The scaling laws discovered around 2020 showed that model performance improved predictably as these three increased, with a specific relationship between them. This turned scaling from a gamble into a plan, and much of the subsequent investment followed directly from that predictability.
What emerged
Here is the genuinely surprising part. Certain capabilities were absent at smaller scales, then present at larger ones, without anyone training for them.
Following instructions. A model trained only to continue text became able to respond to "Translate this into French" as an instruction rather than continuing it as a sentence.
In-context learning. Give two or three examples in the prompt and the model performs the task on a new input — with no weight updates, no training. It is the most practically useful capability and the least expected: the model learned to recognise a pattern being demonstrated and apply it, purely from having predicted text that contained many such demonstrations.
Multi-step reasoning. Asked to work through a problem step by step, larger models became able to chain intermediate conclusions in a way smaller ones could not.
Code. Trained on public repositories alongside prose, models became able to write working programs, which was not the objective.
What "emergent" should and should not mean
The word invites overinterpretation, so it is worth being precise.
A defensible reading: capabilities appear at scale that were not explicitly trained for, because predicting text well at that scale requires them.
An indefensible reading: something mysterious happens beyond a threshold, so further scaling will produce anything you can name.
Later analysis complicated the picture usefully. Several apparently sharp emergences turned out to be artefacts of how they were measured — a task scored as all-or-nothing looks like a sudden jump, while the same task scored on partial credit shows smooth improvement. Real capability gains happened; the discontinuity was often in the metric.
The practical implication: treat "the next model will solve this" as a hypothesis to test rather than a plan. Some things have improved dramatically with scale and others have barely moved.
What has barely moved
Worth listing, because it defines where the surrounding architecture is needed:
Reliable arithmetic and counting. Prediction is not computation, and tokenisation fragments numbers. The fix is calling a calculator, which is lesson 4.
Knowing what it does not know. Confidence is poorly correlated with correctness. This has not improved much with scale.
Genuinely current information. Training has a cutoff. Retrieval fixes this, which is lesson 3.
Consistency. The same question phrased two ways can yield different answers, including different factual claims.
Long-horizon reliability. Each step in a chain has an error rate, and they compound. Lesson 4 covers the arithmetic of this, which is unforgiving.
Where the frontier has actually moved recently
Since roughly 2024, gains have come less from raw size and more from four other directions, which is useful to know because it changes what you should expect:
Better data rather than more data. Filtered, deduplicated, higher-quality training text produces better models at the same size. High-quality text is also finite, which is a real constraint.
Post-training. Instruction tuning and preference alignment, covered in lesson 2, changed usability far more than the last doubling of parameters did.
Reasoning at inference time. Models trained to generate long internal working before answering perform substantially better on hard problems. They also cost more per answer, which makes the trade-off explicit rather than hidden.
Tool use. Letting the model call a calculator, a search engine or a database removes whole classes of failure rather than mitigating them.
An LLM is an extremely capable pattern completer with a vast compressed store of textual regularity, no access to truth, no memory between conversations, and no ability to compute. Everything useful built on top of one — retrieval, tools, validation, agents — exists to supply what that description lacks. The rest of this course is those four things.
In three sentences
A large language model is a transformer trained to predict the next token, where parameters, training data and compute all scaled together by orders of magnitude, and the scaling laws made that improvement predictable enough to justify the investment. Scale produced capabilities nobody trained for — instruction following, in-context learning from a few examples, multi-step reasoning, programming — because predicting text well at that scale requires them, though several apparently sudden emergences turned out to be artefacts of all-or-nothing scoring rather than genuine discontinuities. What has barely improved defines what you must build around the model: reliable arithmetic, knowing what it does not know, current information, consistency and long-horizon reliability, and recent progress has come more from better data, post-training, inference-time reasoning and tool use than from raw size.