Module 1 — What scaling really changes
Course 12 explained how a Transformer computes attention. Course 13 explained what natural language processing tries to solve. This one explains what happened when the two met at very large scale, and why that changed the shape of the field.
The running project across the ten modules is the deployment of a customer-support assistant for a mid-sized company. Every module tightens one screw of that project. This first module answers the very first question: does size alone justify picking a large model over a smaller, cheaper one?
From narrow classifiers to general-purpose text engines
Before 2020, the standard workflow in NLP was: pick a task, curate a dataset, train a model on it. Sentiment classification had its model. Named entity recognition had its model. Machine translation had its model. Every new task started from a labelled corpus and finished with a bespoke artifact.
The GPT-2 to GPT-3 jump replaced that pipeline with a single object: one autoregressive Transformer, trained once on a very large text corpus, then asked to perform arbitrary tasks by describing them in natural language. Sentiment classification became "Review: [text]. Sentiment (positive or negative):". The same model answered translation, summarisation and question answering the same way.
The technical change was small. The behavioural change was enormous, because it moved the interface from code to prompt, and it made a single artifact reusable across teams that previously trained ten.
In-context learning is a strong claim
The behaviour that got everyone's attention was in-context learning: showing a model a few worked examples in the prompt and having it generalise to a new one, with no gradient step.
from transformers import pipeline
generator = pipeline("text-generation", model="Qwen/Qwen2.5-7B-Instruct")
prompt = """Translate English to French.
sea otter => loutre de mer
peppermint => menthe poivrée
cheese =>"""
print(generator(prompt, max_new_tokens=8, do_sample=False)[0]["generated_text"])
The model has not been trained on that pair. It infers the pattern from three examples and produces "fromage". This is a real phenomenon and it scales with model size: small models fail at it, large ones succeed.
No weight is updated. The pattern lives only in the prompt, and it vanishes at the next call. Treating in-context learning as a substitute for fine-tuning gives you a system that "works in the demo" and quietly regresses when the prompt changes shape.
Emergent abilities: real, or measurement artifact?
Several papers reported emergent abilities: capabilities that appear abruptly when the model crosses a size threshold, invisible before it, present after. Three-digit multiplication is the canonical example.
The claim was later challenged. A 2023 paper by Schaeffer, Miranda and Koyejo showed that many "emergences" disappear when you change the metric from exact-match to a smoother one, such as edit distance. The transition then becomes gradual, and the threshold effect vanishes.
The honest summary is: some capabilities do improve non-linearly with scale, but the shape of the curve depends on how you score, and dramatic step changes are often as much about the metric as about the model. Building a business on "the next size up will fix it" is a bet, not a plan.
The open versus proprietary split
Since 2023, the field has bifurcated. Two families of models coexist:
| Family | Access | Typical fit | Cost driver |
|---|---|---|---|
| Proprietary API (GPT, Claude, Gemini) | HTTP call, per token | Rapid prototyping, low volume | Per-token price |
| Open-weight (Llama, Mistral, Qwen, DeepSeek) | Weights on Hugging Face | Fine-tuning, high volume, on-premise | GPU rental or purchase |
The gap between the best proprietary model and the best open model at a given moment is real but narrower than the marketing suggests, and it closes every quarter. For the customer-support assistant of the running project, the decisive criterion is not raw benchmark score but data control, cost at target volume and ability to fine-tune — three points we return to in modules 3, 8 and 10.
A brief timeline
- 2019 — GPT-2, 1.5B parameters. First convincing zero-shot generation.
- 2020 — GPT-3, 175B. In-context learning becomes a headline.
- 2022 — InstructGPT and ChatGPT. Alignment turns a base model into a usable product.
- 2023 — Llama 2 open release; RAG becomes standard. LLaMA reproductions democratise the recipe.
- 2024 — Mixture-of-Experts models (Mixtral, DeepSeek-V2); DPO overtakes PPO for most alignment jobs.
- 2025 — Reasoning models (o1, DeepSeek-R1) trained with reinforcement learning against verifiable rewards.
- 2026 — Small-and-strong open models under 10B parameters compete with much larger APIs on many tasks.
Each of those milestones adds one option to the shortlist you consider at the start of a project. It does not delete the previous ones.
In summary
- Scale replaced the one-model-per-task pipeline with a single general-purpose text engine driven by prompts, which changed the interface more than the algorithm.
- In-context learning is a real phenomenon that grows with size, but it is not learning: weights do not change and behaviour is fragile.
- Emergent abilities are partly a metric artifact; some capabilities do improve non-linearly, but planning on the next threshold is a bet, not engineering.
- The open versus proprietary split is now stable; the choice for a given project turns on data control, cost at volume and fine-tuning need, not on raw score.
Next module: pretraining itself — the data, the tokens and the scaling laws that decide how much compute to spend.