Module 3 — Full fine-tuning and its real cost

The name fine-tuning was coined in an era when models had ten million parameters. Updating all ten million of them once was inexpensive. The same word is still used for models with seven billion parameters — and the arithmetic is no longer the same. This module runs the arithmetic in full, because everything that comes after (PEFT, LoRA, QLoRA) is a response to what you are about to compute.

The memory formula, term by term

Training a neural network keeps four things in GPU memory at once: the weights, the gradients, the optimizer states, and the activations produced during the forward pass. For a model with $P$ parameters trained in mixed precision with the Adam family of optimizers, the first three add up to roughly

$$\text{memory} \approx P \times (2 + 2 + 8) \text{ bytes} = 12 P \text{ bytes}$$

The 2 bytes are for the weights in bfloat16, the 2 bytes for the gradients in bfloat16, and the 8 bytes for the AdamW optimizer state — two moments per parameter, each stored in float32 for numerical stability. That is before you have even opened a file.

For a 7-billion-parameter model:

$$12 \times 7 \times 10^9 = 84 \text{ GB}$$

Eighty-four gigabytes of GPU memory, just for the training state. The consumer card people actually rent — an RTX 3090 or 4090 — has 24 GB. The professional card in a Colab Pro instance — an A100 — has 40 or 80 GB. Neither of them fits a full fine-tune of a 7B model. You would need at least an H100 with 80 GB or a small cluster with tensor parallelism.

Activations, the term that gets forgotten

The 84 GB above is the static cost. On top of that, the forward pass stores an activation at every layer, and the memory that costs depends on the batch size $B$, the sequence length $L$, the hidden dimension $H$, and the number of layers $N$:

$$\text{activations} \approx c \times B \times L \times H \times N$$

with $c$ a small constant that depends on the exact architecture. For a Llama-3-8B forward pass on a batch of 4 sequences of 4096 tokens, activations add another 10 to 20 GB. Gradient checkpointing brings that back down at the price of a slower training step, but it does not touch the 84 GB baseline.

The dollar cost of one epoch

Rented on a market like Vast.ai or RunPod, an eight-GPU H100 node runs at roughly $25 per hour in 2026 prices. Training our two-thousand-pair dataset for three epochs at a batch size that fits in that memory takes on the order of four to six hours. That is a hundred to a hundred and fifty dollars per training run — and you rarely get the hyperparameters right on the first try.

Multiply by three or four attempts, and one experiment on one dataset costs between $400 and $600. That is not a catastrophic number, but it is the wrong number to defend when the alternative in module 5 fits on a single $0.40-per-hour card.

Catastrophic forgetting

Money is not the only tax. Full fine-tuning updates every weight in the model, including the ones that encode general skills — arithmetic, language identification, common-sense knowledge, refusal behaviors. A dataset of two thousand meeting-minutes pairs is narrow: the training signal pulls every weight toward the narrow distribution, and the general capabilities degrade. The model becomes excellent at writing minutes and worse at everything else. This is called catastrophic forgetting, and it is not a small effect — a general-knowledge benchmark can drop ten to twenty points after a naive full fine-tune on a narrow task.

The usual counter-measures — mixing in a small fraction of general instruction data, using a very low learning rate, freezing the embedding layer — all reduce the effect without eliminating it. Parameter-efficient methods, which we get to in module 4, sidestep the problem entirely because they never touch the base weights.

The other, sneakier tax

There is a fourth cost that only shows up at deployment: storage and serving. A full fine-tune produces a complete new set of weights. For a 7B model, that is 14 GB in bfloat16 for each fine-tune. If you serve ten specialized variants — one per team, one per language, one per document type — you now store 140 GB of near-identical weights and reload them on a switch. Module 5 will show that LoRA replaces those 14 GB per variant with 20 to 200 MB, which is a factor of a hundred and matters more in production than any training-time optimization.

When full fine-tuning is still the right answer

I want to be careful not to leave you with the impression that full fine-tuning is a mistake. There are three cases in which it is still the right tool.

The first is when the target behavior is very far from what the base model does: writing in a low-resource language it barely knows, producing outputs in a completely novel format, replicating a style with no adjacency in the pretraining data. In these cases, low-rank adapters do not have enough capacity, and only updating every weight does the job.

The second is when the resulting model will be served at massive scale. If you serve ten million requests a day, the extra latency and memory of managing LoRA adapters at inference time may outweigh the training savings, and merging into a single deployable checkpoint is cleaner.

The third is when you have a lot of data — hundreds of thousands to millions of examples — and the low-rank bottleneck of LoRA becomes the limiting factor. This is closer to continued pretraining than to instruction tuning, and it does not describe the two-thousand-pair scenario we care about here.

Read the numbers once, then never train full again

The reason this course spends a full module on numbers you will not use is that you need to have run them at least once. Fine-tuning courses that skip module 3 leave their readers unable to say, in a design review, why they picked LoRA. Being able to say "84 GB versus 15 GB, and I do not have an H100" is what turns the choice from a fashion into an engineering decision.

Summary

• Full fine-tuning of a 7-billion-parameter model requires around 84 GB of GPU memory just for weights, gradients and optimizer states, before activations — well beyond a consumer card.
• Training runs cost between $100 and $600 on rented hardware, and each variant produces a full 14 GB checkpoint to store and serve.
• Updating every weight causes catastrophic forgetting of general skills, which parameter-efficient methods avoid by construction.
• Full fine-tuning is still right when the target behavior is far from the base, at massive serving scale, or with hundreds of thousands of examples — none of which describe our red-thread task.

Next module: the family of techniques that keep the base weights frozen and train a small set of extra parameters instead.