Skip to main content

Module 2 — Preparing an instruction dataset

Module 1 landed on a firm statement: the dominant cost of a fine-tune is the dataset, not the GPU. This module is the concrete version of that statement. Every hour spent making the two thousand (transcript, minutes) pairs cleaner, more diverse and better aligned with the target format is an hour saved on training, evaluation and post-release firefighting.

The conversational format​

Modern instruction-tuned models expect examples that look like a conversation. The minimum unit is a turn: a user message and an assistant response. For our meeting-minutes task, one training example is one turn — the transcript goes in the user slot, the structured minutes go in the assistant slot, and a system message at the start of the sequence carries the schema.

example = {
    "messages": [
        {"role": "system", "content": "You write meeting minutes in the fixed schema. Return JSON only."},
        {"role": "user",   "content": "Meeting transcript:\n\n[Alice] We should ship the mobile app on...\n..."},
        {"role": "assistant", "content": '{"attendees": ["Alice", "Bob"], "decisions": [...], "action_items": [...]}'},
    ]
}

A few teams still write examples in the older {"prompt": ..., "completion": ...} schema. It works, but it will bite you the moment you want to add a second user turn or a follow-up correction — the messages list is the format the whole ecosystem is standardizing on, so start there.

The chat template the model expects​

Every instruction-tuned checkpoint ships with a chat template, a small piece of Jinja that turns the messages above into the exact string the model was trained to read. Different families use different templates — Llama 3 uses <|start_header_id|> and <|end_header_id|> markers, Mistral uses [INST] and [/INST], Qwen uses ChatML. Applying the wrong template is one of the most common causes of a fine-tune that produces garbage tokens on inference, and it is silent: nothing errors at training time, the loss just goes down more slowly and the outputs look off.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
prompt = tokenizer.apply_chat_template(
    example["messages"],
    tokenize=False,
    add_generation_prompt=False,   # False at training time, True at inference
)

Rule of thumb: never write the template by hand. Always let apply_chat_template do it, and let the same tokenizer that will run inference produce the training strings. That guarantees one canonical rendering across the two phases.

Quality first, quantity second​

Two thousand well-curated examples beat twenty thousand noisy ones. Every mislabeled pair teaches the model to reproduce the mistake with high confidence. Three checks catch most of the rot before training starts.

  • Format compliance. Every assistant response must parse as the target JSON schema without exceptions. Write a validator; drop examples that fail it. The rate at which examples fail this test is a proxy for how bad your source data is.
  • Length distribution. Look at the histogram of transcript lengths and of minutes lengths. A long tail of five-thousand-token transcripts will silently dominate the loss because those examples produce more tokens per batch. Cap or bucket them.
  • Diversity. A dataset made of ninety-eight standup meetings and two board meetings will produce a model that panics on a board meeting. Sample the dataset by meeting type and check that the distribution matches production traffic.

Deduplication is not optional​

Public transcripts leak. Internal meeting notes get copied. A dataset assembled from multiple sources will contain near-duplicates that split themselves across the train/validation boundary at random, and the validation loss will look artificially low for reasons that have nothing to do with generalization. That failure mode has a name — dataset contamination — and it is the single biggest reason a fine-tune looks great in the notebook and disappoints in production.

Two levels of deduplication are cheap and effective. A first pass on exact strings removes trivial copies. A second pass on near-duplicates — MinHash on shingles of five words, similarity threshold around 0.8 — catches the paraphrased ones.

from datasketch import MinHash, MinHashLSH

lsh = MinHashLSH(threshold=0.8, num_perm=128)
for i, ex in enumerate(dataset):
    m = MinHash(num_perm=128)
    for shingle in shingles(ex["messages"][1]["content"], k=5):
        m.update(shingle.encode())
    lsh.insert(str(i), m)

Splitting for validation, and doing it once​

Split the deduplicated dataset before you look at any example a second time. A common split is 90/10 for train and validation, with a third held-out test set of a few hundred pairs frozen from the very start and never touched until module 10.

The temptation to shuffle the split "just to try one more configuration" is what turns a validation set into a second training set. Discipline here pays back tenfold at evaluation time.

Synthetic data, and the trap that comes with it​

You can generate (transcript, minutes) pairs with a larger model — GPT-class or Claude-class — and this is often the fastest way to get from zero examples to two thousand. It is also a technique with two well-known failure modes.

The first is stylistic collapse: the fine-tuned model learns to imitate the generator's tone, and it starts producing minutes that sound like the generator's marketing prose rather than your team's terse voice. Mitigation: keep at least a third of the dataset human-written, and make it the harder third.

The second is error amplification: a small mistake the generator systematically makes (misspelling an internal term, mangling a currency format) becomes a hard-baked behavior in the fine-tune. Mitigation: run the same validator you use on the human data, and spot-check a random 5 % of the synthetic pairs by hand.

A dataset is a contract

Everything the model does after training is a consequence of what is in the dataset. If your minutes reference confidential names, the model will reproduce them; if the JSON schema is inconsistent across examples, the model will produce inconsistent JSON. Treat the dataset like production code, not like a scratchpad — versioned, reviewed, with tests.

Summary​

  • One training example is a conversation (system, user, assistant), rendered through the model's chat template — never write the special tokens by hand.
  • Quality checks (format validator, length distribution, diversity by meeting type) matter more than raw volume; two thousand curated pairs beat twenty thousand noisy ones.
  • Deduplicate at both exact-string and near-duplicate levels, then split into train, validation and a held-out test set before any hyperparameter tuning.
  • Synthetic data helps you bootstrap but risks stylistic collapse and error amplification; keep at least a third of the dataset human-written and audited.

Next module: what training this dataset would actually cost if you tried to update every parameter of a 7-billion-parameter model — the number that motivates the whole rest of the course.