Module 5 — LoRA: low-rank adapters

Module 4 introduced LoRA as the winner among parameter-efficient methods, but glossed over the mathematics. This module fills that gap. Understanding how a rank-16 factorization becomes a fine-tune of a seven-billion-parameter model — with two hyperparameters you actually have to choose — is what separates a working recipe from a working intuition.

The decomposition, in one line

Consider a single linear layer inside a transformer block, with a weight matrix $W \in \mathbb{R}^{d \times k}$. During training, the standard gradient step writes

$$W \leftarrow W - \eta \nabla W$$

LoRA's proposal is to keep $W$ frozen and to represent the update as the product of two matrices with a very small shared dimension $r$:

$$W' = W + \Delta W, \quad \Delta W = B A, \quad A \in \mathbb{R}^{r \times k}, \quad B \in \mathbb{R}^{d \times r}$$

Only $A$ and $B$ are trained. Because $r$ is chosen far smaller than either $d$ or $k$, the update $\Delta W$ is a rank-$r$ matrix, and the total number of trainable parameters is $r \cdot (d + k)$ instead of $d \cdot k$.

Numerical example. In a Llama-3 attention layer, $d = k = 4096$. A full update trains $4096^2 \approx 16.8$ million parameters per layer. With rank $r = 16$, LoRA trains $16 \cdot (4096 + 4096) \approx 131 000$ parameters per layer — a 128-fold reduction. Multiplied across all thirty-two attention layers and their query and value projections, this is what pulls the trainable count from seven billion down to a few million.

Why low rank makes sense at all

The empirical premise behind LoRA is that the update needed for a fine-tune has very low intrinsic rank. When you compute the SVD of the update produced by a full fine-tune, a handful of singular values dominate, and truncating to rank 8 or 16 loses almost nothing. LoRA short-circuits the full computation by imposing that low rank from the start.

This is not magic — it fails on tasks that are far from the base model — but on instruction tuning, style adjustment and format specialization, it holds up remarkably well. The intuition is that the fine-tune is nudging an already-competent model, not rebuilding it.

Initialization: why the model behaves like the base at step 0

LoRA initializes $A$ with a small Gaussian and $B$ with zeros. That choice matters. At step 0, $\Delta W = B A = 0$, and the augmented model produces exactly the same outputs as the base model. Training then moves $B$ away from zero and away from the frozen behavior in a controlled, gradient-driven way.

If you initialized both $A$ and $B$ randomly, the model at step 0 would produce garbage, the first optimizer step would take an enormous gradient, and the loss would either explode or drag the model far from the base before any useful signal was learned. The zero-init of $B$ is one of those small design choices that quietly makes the whole method work.

Rank and alpha, in that order

LoRA has two knobs. The rank $r$ controls the capacity of the update, and the alpha $\alpha$ controls the effective learning rate of the LoRA branch. In code, the update is scaled by $\alpha / r$:

$$W' = W + \frac{\alpha}{r} B A$$

Two practical rules for choosing them.

Choose the rank based on the task's distance from the base. For a task very close to the base — light style tuning, format enforcement — ranks of 4 to 8 suffice. For a moderate specialization — our meeting-minutes task — ranks of 16 to 32 hit the sweet spot. For a large behavior shift — a new domain, a new language — ranks of 64 or above start to be worth the extra memory. Above rank 128, you rarely see further quality gains, and you start to compete with full fine-tuning on memory.

Set alpha to twice the rank as a default. The $\alpha/r$ scaling means that if you sweep the rank at fixed $\alpha$, the effective learning rate changes with it. Fixing $\alpha = 2r$ decouples the two knobs: the update magnitude stays constant, and the rank purely controls capacity. This is the convention the peft library uses in most tutorials, and it is the one you should adopt unless a reason tells you otherwise.

Which modules to target

LoRA can be inserted on any linear layer in the model. In practice, the choice matters and follows a rough hierarchy.

• q_proj and v_proj (query and value in attention) are the classical minimum, from the original LoRA paper. This is the cheapest option and often sufficient.
• All attention projections (q_proj, k_proj, v_proj, o_proj) adds a modest cost and improves quality noticeably on harder tasks.
• Attention plus MLP projections (gate_proj, up_proj, down_proj on Llama-style architectures) is the maximal common configuration. It roughly doubles the trainable parameter count and gives the last few points of quality on tasks that need it.

Do not target the embedding layer or the language-modeling head unless you know why: they are large, they interact with tokenization, and updating them often causes more regressions than gains.

Stacking adapters, swapping tasks

The property that changed how teams ship fine-tunes: the base model is untouched, so multiple LoRA adapters can be trained independently and swapped in at inference. One base model, one adapter per team, one adapter per output format. Loading a new adapter is a memory copy of a few hundred megabytes, not a full model reload.

model.load_adapter("path/to/minutes-adapter", adapter_name="minutes")
model.load_adapter("path/to/summary-adapter", adapter_name="summary")

model.set_adapter("minutes")
# ... run inference for the minutes task ...
model.set_adapter("summary")
# ... run inference for the summary task ...

You can even combine adapters at inference (weighted merges) for zero-shot combinations of behaviors, though quality drops off as you move away from configurations you actually trained.

The trainable-parameter number, on our task

For our meeting-minutes fine-tune on Mistral-7B, targeting q_proj and v_proj at rank 16 with alpha 32:

• 32 transformer layers.
• 2 matrices per layer, each $4096 \times 4096$.
• LoRA parameters per matrix: $16 \cdot (4096 + 4096) = 131\,072$.
• Total: $32 \cdot 2 \cdot 131\,072 \approx 8.4$ million trainable parameters.

That is 0.12 % of the 7 billion base parameters. All the training-memory math from module 3 now applies to those 8.4 million parameters, not to the whole model. The optimizer state alone drops from 56 GB to under 70 MB.

Alpha is not a learning rate replacement

It is tempting to try increasing $\alpha$ instead of the learning rate when training feels slow. Resist: the learning rate scheduler and the optimizer are designed to shape the gradient magnitude across training, while $\alpha$ is a fixed constant baked into the forward pass. Change the learning rate first (module 7), and only touch $\alpha$ if the rank changed.

Summary

• LoRA writes the update to a weight matrix as a product of two rank-$r$ matrices $B A$, with $B$ initialized to zero so the augmented model is identical to the base at step 0.
• Two hyperparameters: rank $r$ (capacity, choose 8 to 64 depending on task distance) and alpha $\alpha$ (default to $2r$ to decouple rank from effective learning rate).
• The target modules determine coverage: q_proj and v_proj is the cheap default, all four attention projections is a solid middle, and adding the MLP is the quality-oriented maximum.
• Because the base is frozen, multiple adapters can be trained and swapped at inference — one base model, many specialized behaviors, a few hundred megabytes per behavior.

Next module: pushing this method one step further by quantizing the frozen base to 4 bits, so the whole training fits on a single 24 GB card.