Module 1 — Prompting, RAG or fine-tuning: choosing the right answer
Fine-tuning is the loudest of the three levers you have on a language model. It also happens to be the most expensive to run and the easiest to misuse. This first module is deliberately not about training code — it is about deciding whether training is what your problem actually needs.
What a fine-tune teaches, and what it does not
The instinct new teams share when a general model disappoints them is to reach for training. That instinct is usually wrong. A fine-tune changes the weights of the model, which means it changes the behaviors that get amortized across every response: tone, format, refusal style, choice of section headings, whether a JSON output is closed with a trailing comma. Behavior is exactly what the base checkpoint has learned poorly for your niche, and it is what a few thousand well-chosen examples can teach quickly.
A fine-tune does not teach fresh facts. If your product's pricing sheet was updated last Monday, no amount of training will make yesterday's checkpoint quote the new numbers — and even a fresh training run will only remember them until the next update. Facts that move belong in a retrieval index, not in the weights. This is not an opinion, it is the same reason nobody bakes today's date into a compiled binary: the storage medium is wrong for the update frequency.
The third lever, prompting, sits below both of these. It reshapes behavior for the duration of a single request, at zero training cost. It is the right first tool when a general model already knows how to do the job but needs to be reminded which shape the output must take.
A decision that fits on one screen
The red thread of the course is a concrete task: turning raw meeting transcripts into structured minutes with attendees, decisions and action items in a fixed schema. Faced with that problem, ask the three questions in this order.
| Symptom | The lever that fits |
|---|---|
| The model can do the task on one example but forgets the schema on the next | Prompt — a few-shot template will pin the format |
| The model needs facts that live in a document your team maintains | RAG — retrieve the passage, cite it, keep the corpus fresh |
| The behavior is right but only after 800 tokens of instructions per request | Fine-tune — amortize those instructions into the weights |
| The format has to be strictly one JSON shape across a million calls | Fine-tune — a system prompt is a soft suggestion, weights are a hard prior |
| The domain vocabulary is unusual (legal, medical, one company's slang) | Fine-tune — nothing else fixes a base model's blind spots |
Notice what is not on that table: "the model got a hard question wrong once." One counter-example is not a signal, it is noise. Training a 7-billion-parameter model to fix one anecdote is the software-engineering equivalent of a heart transplant to cure a paper cut.
The total cost, not the sticker price
The three levers are usually compared by their compute cost. That is the wrong axis, because compute is the smallest bill of the three. The comparison that matters is the operational cost — writing, evaluating, updating, versioning — over the two years the model will be in production.
| Lever | Setup effort | Cost to update | Cost per request |
|---|---|---|---|
| Prompt | Minutes | Edit a string | Extra tokens on every call |
| RAG | Days: extraction, chunking, index, eval | Reindex the new document | Retrieval + generation |
| Fine-tune | Weeks: dataset, training, eval, deploy | New training run | Cheaper per call: shorter prompts |
The fine-tuning line is the one that gets underestimated. The training itself is a handful of hours on a rented 24 GB card — modules 5 to 8 will show that this bill lives well under fifty dollars. The setup cost is your dataset: a couple of thousand (transcript, minutes) pairs, written and reviewed by someone who understands what "good minutes" look like. That work is often two person-weeks. Skipping it is the mistake that produces every failed fine-tune I have ever debugged.
Bad reasons to fine-tune
Three motives should stop the discussion cold.
The first is "the vendor released a new checkpoint, our fine-tune is out of date." If a monthly release cycle from an upstream provider forces you to re-train, you have picked the wrong lever for a moving target — that is a RAG or a prompt problem.
The second is "we want the model to know our company's data." No, you don't. You want the model to retrieve your company's data on demand. Baking documents into weights is expensive, opaque, hard to audit and impossible to redact — a leak in the training set is a leak in every future response.
The third is "our metrics are 2 points below the API model." Sometimes that gap is real and worth closing; more often it is a benchmark artifact that vanishes on the requests you actually get in production. Measure the gap on your traffic before spending three weeks on training.
Fine-tuning is a behavior amortization technique: it moves an instruction that would otherwise be repeated in every prompt into the weights, and it locks the model on the shape of your task. Facts, freshness and citations are not part of what it fixes — those belong to RAG or to the prompt. The next module builds the dataset that makes this amortization possible; get that step wrong and no training trick can save the run.
Summary
- Fine-tuning changes behavior (tone, format, refusal style), not facts — moving facts is the job of RAG, not of training.
- The right lever is the cheapest one that solves the problem: try the prompt first, then RAG, then a fine-tune when neither is enough.
- The dominant cost of a fine-tune is not the GPU bill, it is the dataset: two person-weeks of curated examples before you touch any training script.
- Bad reasons to fine-tune include chasing upstream releases, memorizing corporate documents and closing a benchmark gap that does not appear on real traffic.
Next module: turning the "two thousand (transcript, minutes) pairs" line above into an actual dataset that a training loop can consume.