Skip to main content

Module 6 — QLoRA and 4-bit quantization

Module 5 dropped the trainable parameter count to 0.12 % of the model. The remaining 99.88 % — the frozen base — still takes 14 GB in bfloat16 for a 7B model, which does not fit on a 24 GB card once you add optimizer state, activations and a decent batch size. QLoRA closes the remaining gap by compressing the frozen base to 4 bits per weight, without training a single one of them. The result is a fine-tune that fits on a rented $0.40-per-hour card and loses almost nothing in quality.

The idea, in one sentence​

The frozen base weights participate in the forward and backward passes, but they never receive a gradient update. If we can encode them in fewer bits while keeping the forward-pass values close to the original bfloat16 values, the training loss will barely notice. QLoRA does exactly that with a 4-bit representation — a factor of four compression compared to bfloat16.

Three engineering pieces make that idea practical: NF4 quantization, double quantization, and the paged optimizer. Each solves a specific pain point that a naive 4-bit encoding runs into.

NF4: quantizing to a distribution, not to a range​

Naive 4-bit quantization takes the range of a tensor and slices it into 16 equal buckets. That works when weights are uniformly distributed, but neural network weights are not uniformly distributed — they cluster around zero and taper off symmetrically, approximately following a normal distribution.

NF4 — 4-bit NormalFloat — chooses its 16 quantization levels so that they are the quantiles of a standard normal distribution. Each level covers the same probability mass, not the same linear range, so more resolution is spent where the weights actually live (near zero) and less on the tails (where few weights sit). Empirically, NF4 keeps model quality closer to the original than a naive 4-bit encoding does, at exactly the same bit budget.

from transformers import BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",              # NormalFloat 4 rather than fp4
    bnb_4bit_use_double_quant=True,         # see below
    bnb_4bit_compute_dtype=torch.bfloat16,  # dequantize back to bf16 on the fly
)

The bnb_4bit_compute_dtype is worth pausing on. The weights sit in memory in 4 bits, but every matrix multiplication is executed by dequantizing them back to bfloat16 on the fly. Storage is cheap, math is precise; the trick is doing the dequantization fast enough that the training step is not bottlenecked on it, which the CUDA kernels in bitsandbytes handle.

Double quantization: quantizing the quantization constants​

Every quantization scheme carries scale factors: a small number of extra values, in higher precision, that tell you how to map the 4-bit levels back to the original range. In naive 4-bit, those scales are stored per group of 64 or 128 weights and take 32 bits each. Across a 7B model, that adds a couple of gigabytes on top of the 4-bit weights themselves — small, but not free.

Double quantization quantizes those scale factors too. The scales are compressed from 32 bits to 8 bits, with a second, coarser set of meta-scales at 32 bits. The math is only lightly lossy on the scales themselves, and the effect on the underlying model quality is negligible — but the memory saving is around 0.4 GB on a 7B model. On a 24 GB card, that half a gigabyte is the difference between "fits with batch size 4" and "OOM at batch size 4".

The paged optimizer: absorbing the batch spikes​

Even with the base compressed to 4 bits and the LoRA optimizer state under 100 MB, training runs occasionally spike in memory when a long sequence goes through gradient checkpointing at an inopportune time. Those spikes push the process over the card's limit and kill the run.

The paged optimizer, also from bitsandbytes, wraps the AdamW optimizer state in a paging mechanism that spills to CPU memory when the GPU is under pressure and pages back when it is not. It is slower than pure GPU memory in the worst case, but it prevents crashes on transient spikes, which is what you actually want on a card you cannot upgrade mid-run.

A complete configuration for a single 24 GB card​

Putting the pieces together, here is the training preparation for our meeting-minutes fine-tune on Mistral-7B, on a rented RTX 4090 with 24 GB of VRAM.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-v0.3",
    quantization_config=bnb_config,
    device_map="auto",
)
model = prepare_model_for_kbit_training(model)   # casts, sets grad flags

lora_config = LoraConfig(
    r=16, lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)

Memory footprint of this configuration:

  • Base weights in NF4: about 4.2 GB (down from 14 GB in bfloat16).
  • Meta-scales after double quantization: 0.4 GB.
  • LoRA parameters (bfloat16): 17 MB.
  • Optimizer state (paged AdamW on LoRA only): 70 MB.
  • Activations with gradient checkpointing, batch 4, sequence 1024: 8 to 10 GB.

Total: around 14 GB, which leaves a healthy 10 GB of headroom on a 24 GB card. Batch size and sequence length can be pushed further from there, and module 7 gives sensible starting values.

The quality gap, measured​

The natural question is how much quality QLoRA sacrifices compared to full-precision LoRA. The QLoRA paper measured this across dozens of tasks, and the answer is remarkably good: the gap is within one point of MMLU on average, and often unmeasurable on downstream tasks. On format-heavy tasks like the meeting-minutes generation of this course, we have never seen a measurable regression from moving to QLoRA.

The one exception is when the base model is small (below 3B parameters), where NF4 loses enough per-weight precision to matter. For 7B and above, QLoRA is essentially free.

QLoRA is not slower to train

The compute-time cost of dequantizing on every matrix multiplication is often assumed to make training slower. In practice, the dequantization kernels overlap with memory movement, and the reduction in weight-tensor size reduces the memory bandwidth pressure. On the same 24 GB card, a QLoRA run typically runs at 90 to 100 % of the LoRA speed, and it fits configurations that plain LoRA does not.

Summary​

  • QLoRA quantizes the frozen base to 4 bits per weight and trains LoRA adapters on top, cutting the base's memory footprint by a factor of four.
  • NF4 places its 16 quantization levels at the quantiles of a normal distribution, which is where the weights actually live; double quantization compresses the scale factors themselves.
  • The paged optimizer spills state to CPU on transient spikes and prevents crashes on cards without margin — a critical detail on a 24 GB card.
  • A complete Mistral-7B QLoRA fits in about 14 GB, and the measured quality gap versus full-precision LoRA is under one point on standard benchmarks.

Next module: choosing the learning rate, batch size and epoch count that will make this configuration actually converge.