Lesson 5 — What it really costs, and what to do instead
The previous lessons explained how deep learning works. This one is about whether you can afford it, and the answer reframes everything: you almost never train from scratch, and understanding why is the most practically valuable thing in this course.
The three costs
Data
Deep networks have many parameters and each parameter needs evidence. Rough orders of magnitude for training from nothing:
| Task | Realistic requirement to train from scratch |
|---|---|
| Image classification | thousands of images per class |
| Object detection | tens of thousands of annotated images with boxes |
| Speech recognition | thousands of hours of transcribed audio |
| Machine translation | millions of aligned sentence pairs |
| A language model | trillions of words |
If you have three hundred examples and someone proposes training a network from scratch, the number of examples is the problem, and no architecture choice will rescue it.
Compute
Training means performing the forward and backward passes over your entire dataset, repeatedly. On a modest problem that is minutes on a single GPU. At the other end, training a frontier language model occupies thousands of accelerators for weeks and costs a sum that only a handful of organisations can commit.
The number worth internalising is the ratio: a GPU is often 10 to 50 times faster than a CPU for this work. A run that takes a week on a processor takes hours on a graphics card, which is why the hardware is not optional past a certain scale.
Time and attention
The cost people underestimate. Deep learning involves more failed runs than successful ones: a learning rate that diverges, an input pipeline with a subtle bug, a model that trains for six hours and produces nonsense because labels were misaligned. Each iteration is slow enough that mistakes are expensive, which makes careful setup worth more here than in classical machine learning.
Transfer learning: the answer
Here is the observation that changes the economics.
A network trained on millions of general images has learned, in its early layers, to detect edges, textures, curves and shapes. None of that is specific to the categories it was trained on. Edges are edges whether you are classifying dog breeds, inspecting solder joints or reading X-rays.
So instead of learning those features again from your small dataset, you reuse them.
The scale of the improvement is worth stating plainly. A task needing tens of thousands of images from scratch is frequently solvable with a few hundred per class by fine-tuning, in minutes on a free hosted GPU. This is the single largest practical shift in applied deep learning of the last decade, and it is why small teams can now attempt things that required a research lab in 2015.
The variants you will meet
| Approach | What you train | Use when |
|---|---|---|
| Feature extraction | only a new final layer | very little data, or your task closely resembles the original |
| Fine-tuning | the last few layers, or all of them gently | a moderate amount of data, a somewhat different task |
| Full fine-tuning | every parameter, at a small learning rate | plenty of data and compute, a meaningfully different domain |
| Parameter-efficient tuning (LoRA) | a small set of added parameters | large language models, where full tuning is unaffordable |
LoRA deserves a note because it is what makes adapting large language models practical. Rather than updating billions of parameters, it inserts a small number of new ones and trains only those. The result approaches full fine-tuning quality at a small fraction of the memory and cost, which is why a laptop-class setup can now adapt a substantial model. The premium fine-tuning course covers it properly.
Hugging Face hosts hundreds of thousands of pre-trained models for text, vision and audio, with a uniform interface. The genuinely valuable skill in applied deep learning is not implementing architectures — it is choosing an appropriate pre-trained model and adapting it well.
When deep learning is the wrong tool
Being explicit, because the reflex to use it is strong and often expensive:
Your data is a table. Gradient-boosted trees usually win, train in seconds and explain themselves. This is not a close call on most tabular problems.
You have very little data. Even fine-tuning has a floor. Below roughly a hundred examples per class, consider classical methods, data augmentation, or gathering more data.
You must explain every decision. A deep network's reasoning is not readable. In regulated lending or hiring, that can be a legal blocker regardless of accuracy.
You need guaranteed behaviour. A model gives probabilities. Where a specific input must always produce a specific output, write the rule.
Latency or hardware is tightly constrained. A large model may not fit the budget of an embedded device. Compression helps — quantisation, distillation, and formats such as ONNX and TensorFlow Lite, all covered in the premium catalogue — but it is a constraint to design around from the start rather than discover at the end.
The honest summary
Deep learning is expensive to build from nothing and cheap to adapt. The gap between those two facts is where nearly all practical value sits. A team that understands transfer learning can deliver, in a fortnight and on a free GPU, results that would have required a research budget a decade ago — and a team that insists on training from scratch will spend months rediscovering edge detectors.
In three sentences
Training from scratch needs thousands of examples per class and serious compute, which puts it out of reach for most projects and makes it the wrong default. A pre-trained model has already learned general-purpose features that transfer across tasks, so replacing its final layer and fine-tuning on a few hundred of your own examples usually beats anything you could train yourself, in minutes rather than weeks. Deep learning remains the wrong tool on tables, on very small datasets, where every decision must be explained, and where behaviour must be guaranteed.
Next — Lesson 6: recap and FAQ →