Skip to main content

Module 7 — Fine-tuning hyperparameters

By now the architecture is set: Mistral-7B in NF4, LoRA on the four attention projections, rank 16, alpha 32. What remains is the training regime — learning rate, batch size, epochs, sequence length, warmup. These five numbers do more to decide the outcome of a training run than any other single choice, and the good news is that the reasonable ranges are narrow. This module is the shortest of the course, on purpose: it gives you starting values that work, and the tools to move them.

Learning rate: the parameter that dominates everything​

For LoRA and QLoRA, the learning rate that works consistently sits in the range 1e-4 to 3e-4. This is much higher than a full fine-tune of the same base (typically 1e-5 to 5e-5), and there is a reason: the LoRA branch is randomly initialized and needs to move a lot to become useful, whereas the base weights of a full fine-tune are already well-placed and only need small nudges.

Two failure modes bracket the range.

Too low (below 5e-5) leaves the loss flat or barely-descending. Twenty epochs later, the model has learned nothing measurable. The temptation is to blame the data. The right move is to double the learning rate and try again.

Too high (above 1e-3) causes the loss to spike upward mid-training, sometimes to NaN. The model has fallen off the low-loss basin and cannot recover, because the LoRA branch has moved too far to be corrected by subsequent small updates. Halve the learning rate and try again.

A safe starting value for our meeting-minutes task on Mistral-7B: 2e-4.

Warmup: giving the optimizer a chance​

Adam-family optimizers estimate the first and second moments of the gradient across the first few hundred steps. With no data yet, those estimates are noisy, and applying the full learning rate on the very first step tends to push the model in a random direction that costs several hundred steps to recover from.

The fix is a linear warmup: start the learning rate at zero, ramp up linearly to the target value over the first few percent of the run, then apply the main scheduler (cosine or linear decay).

The rule of thumb is 3 % of total training steps as warmup, capped at a couple of hundred steps for short runs. For our two-thousand-example dataset trained for three epochs with an effective batch size of 32, the run is about 190 steps long, so warmup of 6 steps is enough.

from transformers import get_cosine_schedule_with_warmup

total_steps = len(train_loader) * num_epochs
scheduler = get_cosine_schedule_with_warmup(
    optimizer,
    num_warmup_steps=int(0.03 * total_steps),
    num_training_steps=total_steps,
)

Effective batch size: the number that hides gradient accumulation​

The GPU can only physically process a small batch — for our QLoRA setup, four sequences of 1024 tokens at a time. But the number that actually shapes training dynamics is the effective batch size: the number of examples that contribute to each optimizer step.

Gradient accumulation lets you decouple the two. If the physical batch is 4 and you accumulate over 8 forward passes before calling optimizer.step(), the effective batch is 32.

training_args = TrainingArguments(
    per_device_train_batch_size=4,      # what fits in memory
    gradient_accumulation_steps=8,      # effective batch = 32
    ...
)

For LoRA fine-tunes on instruction data, an effective batch size of 16 to 64 is the working range. Larger batches (128 and above) produce smoother gradient estimates but require more warmup steps and a slightly higher learning rate to compensate. Smaller batches (below 8) are noisy and often unstable.

Sequence length: the memory-cost lever​

Attention memory grows quadratically with sequence length. Doubling the sequence length quadruples the activation memory, and it is the single most efficient way to trigger OOM on a 24 GB card.

For meeting minutes, most transcripts fit under 2 000 tokens. Set max_seq_length = 1024 for the first training run and look at the truncation rate. If more than 5 % of examples are being truncated, increase to 2048 and drop the batch size to compensate. Do not set the sequence length higher than the P95 of your data — you are paying for capacity you do not use.

Epochs: the "how long" question​

Instruction fine-tuning is famously sample-inefficient: the base model already knows what most of the words mean, and your dataset is teaching a narrow additional behavior. Two to four epochs is the usual range, and one is often already enough on a well-curated dataset.

Do not train for ten epochs "to be safe" — you will overfit. Module 8 is the diagnostic that tells you when to stop, but the rule to internalize is: more epochs is not more quality, past a threshold that lives around three passes over the training set.

The complete starting configuration​

Putting the numbers together, here is a Trainer configuration that will not surprise you on our red-thread task.

from transformers import TrainingArguments

args = TrainingArguments(
    output_dir="./out",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,             # effective batch = 32
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    optim="paged_adamw_8bit",
    max_grad_norm=1.0,
    logging_steps=10,
    eval_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=2,
    bf16=True,
    gradient_checkpointing=True,
    report_to="none",
)

Two lines deserve a comment. max_grad_norm=1.0 clips gradients before the optimizer step; without it, a single anomalous batch can spike the gradients and destabilize the run. gradient_checkpointing=True recomputes activations on the backward pass instead of storing them, trading time for memory — essential on a 24 GB card.

What to move first when the run looks bad​

You will not get the numbers right the first time. The order in which to touch them, by empirical payoff:

  1. Learning rate — double it or halve it based on the shape of the training curve.
  2. Effective batch size — increase it if the loss is jagged, decrease it if you want more frequent updates.
  3. Rank — increase it if the loss plateaus above your target, decrease it if training overfits.
  4. Warmup — increase it if the loss spikes in the first ten steps.
  5. Everything else — small changes that matter only after the above are tuned.
Do not sweep more than one hyperparameter at a time

It is tempting, when a run underperforms, to change three numbers at once "to save time". Do not. You will not know which change helped, and the next run will inherit two bad guesses out of three. Change one number, run, observe, then change the next.

Summary​

  • Learning rate for LoRA and QLoRA lives in 1e-4 to 3e-4; higher than full fine-tuning because the LoRA branch starts from a random initialization.
  • Warmup of about 3 % of total steps prevents the first-step overshoot; combine with a cosine decay for the rest of training.
  • Effective batch size (physical batch times gradient accumulation) belongs in the 16 to 64 range; larger batches need higher warmup and learning rate.
  • Cap sequence length at the P95 of your data, cap epochs at three, and clip gradients at norm 1 to make the run robust to occasional bad batches.

Next module: watching the two loss curves as the run proceeds, and knowing when to stop.