Module 3 — Supervised instruction tuning

Module 2 left us with a base model: it predicts the next token, and nothing else. Ask it "Write a polite refusal for a refund request" and it will happily continue the sentence — "…was drafted by the customer service team last week, and it read as follows…". It has no notion that a question deserves an answer rather than a continuation.

This module turns that base model into an instruction-following assistant, the first step of every customer-support project of the running example. It is also the cheapest and most controllable step of the alignment pipeline, and the one where most teams get the biggest single quality jump.

Base model versus instructed model

The two are the same architecture and, initially, the same weights. Fine-tuning changes only the last few percent of the weights, on a much smaller corpus, with a much lower learning rate.

Property	Base model	Instructed model
Objective	next-token prediction on generic text	next-token prediction on <instruction, answer> pairs
Corpus size	trillions of tokens	tens of thousands to a few million examples
Behaviour on a question	continues as if it were text	answers as if in a dialogue
Suffix on Hugging Face	-base or nothing	-Instruct, -Chat, -it

For 90 % of production use, you want the instructed variant. The base model is useful only when you intend to do the instruction tuning yourself, either to inject domain data or to control the alignment style.

The shape of an instruction dataset

An instruction dataset is a list of pairs. The simplest schema, popularised by Alpaca in 2023, has three fields:

{
  "instruction": "Draft a polite reply to a customer whose parcel is late.",
  "input": "",
  "output": "Dear customer, thank you for reaching out about your order..."
}

More recent datasets — UltraChat, OpenHermes, Tulu — extend this to multi-turn conversations, where the model sees the full history and has to answer only the last turn.

Three sources are used in practice, and their trade-offs matter:

1. Human-written (like Dolly): highest quality per example, expensive, small.
2. Model-generated (Self-Instruct, WizardLM): cheap, large, but inherits the biases and mistakes of the generator model.
3. Task-transformed (FLAN, T0): existing labelled NLP datasets reformulated as instructions; middle ground on cost and quality.

For a customer-support project, the sweet spot is a hybrid: a public instruction dataset for general fluency, plus a few thousand human-written examples that carry your voice and your business rules.

The chat template is not decoration

Every instructed model expects its input in a specific format, the chat template, which encodes turn boundaries as special tokens. Get it wrong and quality collapses silently — no error, just noticeably worse answers.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

messages = [
    {"role": "system", "content": "You are a customer-support assistant."},
    {"role": "user", "content": "My order has not arrived. What should I do?"},
]

prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
print(prompt)

For Qwen this expands to <|im_start|>system\n...<|im_end|>\n<|im_start|>user\n...<|im_end|>\n<|im_start|>assistant\n. For Llama 3 it is different. For Mistral it is different again. Never handcraft the template yourself; always let the tokenizer do it, so a model switch does not require a code change.

A wrong chat template does not raise an exception

The model will keep generating. It will just be markedly worse — less coherent, more repetitive, prone to hallucinating the wrong role. If a new model plugs in with a big quality drop, the chat template is the very first place to look.

Quality against quantity, and the LIMA lesson

The 2023 LIMA paper made one point with unusual force: a thousand carefully written instruction examples can beat a hundred thousand mediocre ones, at a fraction of the compute. The reason is that instruction tuning does not teach the model new knowledge; it teaches it to use what it already learned in pretraining. Sloppy examples teach sloppy formatting.

The practical rule is: a hundred hand-written examples that a subject-matter expert would sign off on are worth more than ten thousand mechanically generated ones. Two hundred examples per business capability — refund, delivery, warranty, cancellation — is a realistic first target.

A minimal LoRA fine-tuning loop

Full fine-tuning of a 7B model requires updating all 7 billion weights and, in practice, several tens of GB of GPU memory. LoRA (Low-Rank Adaptation) freezes the base weights and inserts small trainable matrices, cutting memory by an order of magnitude with almost no quality loss.

from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer

model_id = "Qwen/Qwen2.5-7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16")

dataset = load_dataset("json", data_files="support_tickets.jsonl", split="train")

lora = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    args=SFTConfig(
        output_dir="./support-lora",
        num_train_epochs=3,
        per_device_train_batch_size=2,
        learning_rate=2e-4,
        bf16=True,
    ),
    train_dataset=dataset,
    peft_config=lora,
)
trainer.train()

This runs on a single 24 GB consumer GPU for a few thousand examples. The output is a small adapter file (tens of MB) that plugs on top of the frozen base model at inference time.

Split the eval set before the first training call

Reserve 5 to 10 % of your instructions before you touch the training loop, and hand-score its outputs after each run. This is what tells you whether the model is learning your business or just learning to imitate a formatting style.

In summary

• Instruction tuning turns a base model into one that answers questions instead of continuing them, using a small labelled dataset and a low learning rate.
• The chat template is model-specific and silent when wrong; always apply it with the tokenizer's own function.
• Quality trumps quantity: a few hundred expert-written examples per business capability beat tens of thousands of noisy ones.
• LoRA makes fine-tuning of a 7B model fit on one consumer GPU and produces a small adapter that plugs on top of the frozen base — the practical default for the customer-support assistant.

Next module: alignment, where preferences enter the loop and the model learns not just to answer but to answer the way you want.