Lesson 6 — Recap and FAQ
The five lessons in one page
Lesson 1 — the neuron. It multiplies inputs by learned weights, adds a bias, and applies a simple non-linear function. Those weights and biases are everything the network knows. The activation function is essential rather than decorative: without it, a hundred layers collapse into one.
Lesson 2 — depth. Each layer builds on the abstractions below, so a reusable hierarchy emerges — edges, textures, parts, objects — that nobody programmed. That is why deep learning works where features cannot be hand-designed, and why ReLU and residual connections, which let gradients travel back through many layers, were what made depth usable.
Lesson 3 — CNNs. A plain network needs a hundred million weights for one image layer and ignores that pixels are adjacent. Convolution slides small shared filters across the image, giving few parameters and position independence, while pooling shrinks maps so deeper filters see more.
Lesson 4 — transformers. Recurrent networks read step by step and lose early information; transformers let every position attend to every other through dot products. Parallel processing and constant path length are what made internet-scale training possible.
Lesson 5 — the cost. Training from scratch needs thousands of examples per class and serious compute. Fine-tuning a pre-trained model on a few hundred examples usually beats it, in minutes, which is the largest practical shift of the last decade.
Which architecture for which data
| Your data | Architecture | Note |
|---|---|---|
| Table of columns | gradient-boosted trees, not deep learning | trees usually win on tabular data |
| Images, classification | CNN, pre-trained | fine-tune; do not start from scratch |
| Images, locate objects | detection network (YOLO family, DETR) | needs boxes, not just labels |
| Images, per-pixel labels | segmentation network (U-Net family) | medical imaging, satellite |
| Text, classification | pre-trained transformer (BERT family) | fine-tuning is cheap and effective |
| Text, generation | decoder transformer (GPT family) | this is what an LLM is |
| Audio, speech | transformer (Whisper family) | replaced older recurrent pipelines |
| Time series | gradient boosting first, then transformers | classical methods are strong here |
| Anything, very little data | fine-tune, or classical methods | depth needs evidence |
Glossary
| Term | What it means |
|---|---|
| Weight / parameter | one learned number inside the model |
| Bias | the learned offset added in each neuron |
| Activation | the non-linear function that makes depth meaningful |
| Layer | a group of neurons receiving the same inputs |
| Epoch | one complete pass through the training data |
| Batch | the group of examples processed before each update |
| Learning rate | how large a step each update takes |
| Dropout | randomly disabling neurons during training, to force redundancy |
| Batch normalisation | rescaling values between layers to keep training stable |
| Residual connection | a shortcut letting gradients skip past a block |
| Convolution | sliding a small shared filter across the input |
| Pooling | shrinking a map, usually by taking the maximum |
| Attention | weighting every position against every other by dot product |
| Embedding | a learned vector representation of an item |
| Fine-tuning | adapting a pre-trained model to your task |
| Inference | using the trained model to predict |
The 12 questions people actually ask
1. How do I choose the number of layers and neurons?
In practice you do not design this from scratch. You take an architecture known to work for your data type, almost always pre-trained, and adapt it. Designing depth and width from first principles is a research activity; treating it as a required beginner decision leads to months of pointless tuning.
2. What is a batch size and does it matter?
The batch is how many examples are processed before each parameter update. Larger batches give smoother, more reliable gradient estimates and use more memory; smaller batches add noise that sometimes helps generalisation. Start with whatever fits comfortably in memory, commonly 32 or 64, and treat it as a secondary concern behind the learning rate.
3. How many epochs should I train for?
Not a number you pick in advance. Watch the validation loss: while it falls, keep going; once it starts rising while training loss keeps falling, you have begun overfitting and should stop. Early stopping automates exactly this, and it is one of the highest-value few lines you will add.
4. My network is not learning. Where do I look?
In this order: is the learning rate far too high, which shows as a loss that explodes or becomes not-a-number; are the inputs scaled sensibly; are labels aligned with the right examples; can the model overfit a tiny sample of ten examples, which proves the training loop works at all. That last check is the fastest way to separate a broken pipeline from a hard problem.
5. Why is my model excellent in training and poor in validation?
Overfitting. The remedies, roughly in order of effectiveness: more data, then data augmentation, then a simpler or smaller model, then dropout and weight decay, then early stopping. With a small dataset, fine-tuning a pre-trained model addresses the cause rather than the symptom.
6. What is data augmentation?
Creating additional training examples by transforming the ones you have: flipping, rotating, cropping and adjusting the brightness of images, or paraphrasing text. It works because the transformed version is still a valid example of the same class, and it is one of the most cost-effective ways to reduce overfitting when data is scarce. The transformations must preserve the label, which is why flipping a photograph is fine and flipping a handwritten digit is not.
7. Can I run deep learning without a GPU?
For inference on a small model, yes, comfortably. For fine-tuning, it is painful but possible on a small dataset. For training from scratch, no. Since Google Colab lends you a GPU at no cost, this is rarely the binding constraint while learning.
8. What is the difference between a parameter and a hyperparameter?
A parameter is learned by training: the weights and biases. A hyperparameter is chosen by you: learning rate, batch size, number of layers, dropout rate. Training finds parameters; you search for hyperparameters, on the validation set.
9. Why are language models so much larger than image models?
Because language is vastly more open-ended. Recognising a thousand object categories is a bounded problem; producing coherent text about anything requires an enormous amount of world knowledge stored in the parameters. Capacity is what allows that storage, so the parameter counts diverge by orders of magnitude.
10. Are CNNs obsolete now that transformers exist?
No. Vision transformers match or beat CNNs given large datasets, and typically need more data to get there. CNNs remain the pragmatic choice on modest datasets and constrained hardware, and both are actively used. Announcements of one architecture retiring another are usually premature.
11. What does it mean that a model has 7 billion parameters?
Seven billion individual learned numbers. At two bytes each that is roughly 14 GB just to hold the model in memory, before any computation. This is why model size determines what hardware you need, and why quantisation — storing parameters at lower precision — is such a practically important technique.
12. Where should I go after this course?
Natural Language Processing or Computer Vision depending on your data, then Large Language Models where transformers reach their current form. MLOps covers what happens after a model works.
The premium catalogue has you training real models in PyTorch, with datasets, projects and a verifiable certificate after a 40-question examination. Included in every paid plan.
Last step — Take the quiz and see your attestation →