Module 7 — Cheap fine-tuning of a small model

Module 3 produced 5 000 tickets labelled by the large teacher. Module 4 shrank the base to 2 GB. This module puts them together: fine-tune Qwen 2.5 3B on the distilled ticket dataset with LoRA, on modest hardware, in under an hour, for less than a dollar of GPU rental.

The techniques come from course 19 (fine-tuning) — this module assumes you have read it. What is specific to SLMs is that the numbers are small enough that the whole loop runs on a laptop with an RTX 4060 or on a Colab free instance, without any research infrastructure.

Why LoRA and not a full fine-tune

A full fine-tune of a 3B model updates 3 billion parameters. In fp16 that is ~12 GB of gradient state plus optimizer state, i.e. a GPU with at least 24 GB of VRAM and half a day of training. That is not cheap.

LoRA (Low-Rank Adaptation) freezes the base weights and injects two small matrices per attention projection. At rank 16, the trainable parameter count on a 3B model drops to about 20 million — 0.7 % of the base. Training fits in 8 GB of VRAM and finishes in an hour on entry hardware.

The trade-off: LoRA is slightly less expressive than a full fine-tune. On narrow tasks — including ticket classification — that difference is invisible. On tasks that require rewriting the model's world model, it shows. Ours does not, so LoRA is the answer.

The training script, end to end

# train.py -- LoRA fine-tune Qwen 2.5 3B on 5 000 tickets, ~50 min on RTX 4060
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer

BASE = "Qwen/Qwen2.5-3B-Instruct"
tok  = AutoTokenizer.from_pretrained(BASE)
base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="bfloat16")

lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                  target_modules=["q_proj","k_proj","v_proj","o_proj"],
                  task_type="CAUSAL_LM")
model = get_peft_model(base, lora)          # 20M trainable / 3B total

def to_chat(ex):
    """Turn a labelled ticket into a chat sample the trainer consumes."""
    msgs = [
        {"role":"system","content":"You are a support-desk analyst. Reply strict JSON."},
        {"role":"user","content": ex["text"]},
        {"role":"assistant","content": ex["label_json"]},
    ]
    return {"text": tok.apply_chat_template(msgs, tokenize=False)}

ds = load_dataset("json", data_files="labelled.jsonl")["train"].map(to_chat)
ds = ds.train_test_split(test_size=0.1, seed=42)

args = TrainingArguments(
    output_dir="qwen-ticket-lora",
    per_device_train_batch_size=2, gradient_accumulation_steps=8,
    learning_rate=2e-4, num_train_epochs=2, bf16=True,
    logging_steps=25, eval_strategy="steps", eval_steps=100, save_steps=200,
)
trainer = SFTTrainer(model=model, args=args,
                     train_dataset=ds["train"], eval_dataset=ds["test"],
                     tokenizer=tok, dataset_text_field="text",
                     max_seq_length=1024)
trainer.train()
model.save_pretrained("qwen-ticket-lora")

The knobs to remember: r=16 (rank of the adapter, sweet spot for SLMs), lora_alpha=32 (scaling; roughly 2*r), learning_rate=2e-4 (LoRA tolerates higher LRs than full fine-tunes), two epochs (more overfits on 5 000 examples).

Training log at end of run:

epoch 2.00 | train_loss 0.31 | eval_loss 0.34 | 47 min | RTX 4060 8 GB

The trainable-parameter line at the top of the run confirms the LoRA is doing what you think: trainable params: 20,971,520 || all params: 3,105,714,688 || trainable%: 0.68.

The GPU-minute budget

The whole loop on an RTX 4060 (rented at ~$0.30/hour on Runpod in 2026):

Step	Time	Cost
Distillation of 5 000 tickets (module 3, gpt-4o-mini)	~90 min	~$0.20
LoRA training (this module, RTX 4060 8 GB)	~50 min	~$0.25
Merge + convert to GGUF + quantize	~10 min	negligible
Total from raw tickets to shippable Q4 model	~2h 30m	~$0.50

Under a dollar and under three hours. That is the number to internalise — it changes what a small team can plausibly attempt. Custom SLMs stop being a research project and become a normal engineering task.

Merging the adapter and exporting

After training, the LoRA adapter is a small extra weight tensor. Deployment prefers a merged model — a single set of weights that the runtime does not have to combine on the fly.

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B-Instruct",
                                             torch_dtype="bfloat16")
merged = PeftModel.from_pretrained(base, "qwen-ticket-lora").merge_and_unload()
merged.save_pretrained("qwen-ticket-merged")

Then convert to GGUF and quantize as in modules 4 and 5. The resulting qwen-ticket.Q4_K_M.gguf is what modules 8 and 10 deploy.

The lift, measured

The reason we do this at all — on the 200-ticket held-out evaluation from module 1:

Configuration	Category accuracy	Summary score	Reply score
Qwen 2.5 3B base, Q4_K_M	88 %	3.9	3.5
+ LoRA on 5 000 distilled tickets, Q4_K_M	93 %	4.2	4.0
GPT-4o-mini via API (module 1 baseline)	93 %	4.3	4.1

The fine-tuned local model matches the large API model on category accuracy and comes within 0.1 on summary and reply quality. For a task that pays a real cost per API call and cares about privacy, this is a rational deployment.

What can go wrong

Overfitting on small data. 5 000 examples is a lot for LoRA at r=16, but three epochs already overfits: eval loss stops improving while train loss keeps falling. Two epochs is the sweet spot; watch eval_loss, not train_loss.

Chat template drift. The training samples must use the base model's own chat template (tok.apply_chat_template). A hand-rolled <|system|> template that does not match the tokenizer's expected format produces a model that trained on a language slightly different from the one it deploys in. Silent 5-point drop.

Evaluating on the training set. The held-out 200 tickets must not overlap with the 5 000 distilled. Split before distillation, not after.

In summary

• LoRA on a 3B model updates 20M parameters (0.7 %) — fits in 8 GB of VRAM, ~50 minutes of RTX 4060 time, under $0.25 in rental.
• The full custom-SLM loop from raw tickets to shippable Q4 GGUF is ~2h 30m and under $0.50 in 2026 — cheap enough for a normal engineering budget.
• Key knobs: r=16, lora_alpha=32, lr=2e-4, 2 epochs; watch eval_loss to catch overfitting.
• On the ticket assistant, LoRA closes the gap of module 1 (88 % → 93 %) and the fine-tuned local model matches the API baseline on category accuracy.
• Ship the merged model as GGUF Q4 — the adapter is training scaffolding, not the deployment artefact.

Next: taking that artefact to the actual workstations — Ollama, memory budgets, model updates and integration into the ticket tool.