#transfer-learning — Deep learning

Transfer learning: start from an already-trained network.

What you'll play with

Welcome to #transfer-learning. On screen, two identical 2-12-12-1 networks facing the same task: classifying 60 target points, blue and pink, from slightly shifted moons. On the left, transfer: a network already trained on 400 moons (the source task) whose first two layers are frozen (icy blue) — only the pink head will learn. On the right, from scratch: the same network, random weights, everything to learn. Both decision surfaces are still grey: neither has looked at the target. This is what a hospital does when it reuses a network trained on millions of photos to read 200 X-rays: when you have few examples, you never start from zero.
Let's make the target really scarce: type /data 20. Twenty examples, that is all the two networks will see. Along the way, the source network is pre-trained once and for all (400 points, 8 epochs) and copied to the left; on the right, random weights.
Same budget for both: /train 20. Twenty epochs on twenty points, and the two curves get traced in the middle, epoch by epoch.
What if we unfreeze everything? /freeze 0: no layer is frozen anymore, all 205 parameters of the transfer network are retrained on the 20 points, with the same 20-epoch budget (replayed).
Refreeze the first two layers: /freeze 2. The features learned on the source become untouchable again; only the head (12 weights + 1 bias) adapts.
Let's push the target away from the source: /rotation 90. The target moons rotate a quarter turn — the cloud spins before your eyes — and the 20 epochs are replayed.
Put numbers on all this: /compare. The tutor sums up validation accuracy, gap, trained parameters and budget, and the band between the two curves appears.
Your turn: /rotation 0 then /data 200 (with plenty of data, the fresh network nearly catches up), /freeze 1 (only freeze the first layer: the classic fine-tuning trade-off), /lr 0.1, /target circles (a different family of shapes: moon features are useless), /source circles (circles are rotation-invariant: rotate them and transfer holds), /seed 7, /reset. Next up: #gan, where two networks no longer cooperate but compete.

Channel commands

/source <moons|circles> — Shape of the source task (400 points) on which the network is pre-trained.
/target <moons|circles> — Shape of the target task, data-poor.
/rotation <degrees=0..180> — Rotates the target: the more it rotates, the further it drifts from the source.
/data <points=10..200> — Number of target training points.
/freeze <0|1|2> — Number of first layers frozen in the transfer network (0 = retrain everything).
/train <epochs=5..60> — Continues fine-tuning both networks, same budget for each.
/lr <0.001..0.5> — Fine-tuning learning rate (both networks).
/compare — Numeric summary of transfer vs from scratch; shows or hides the gap band.
/seed <1..999> — Different target-point sampling and different initialisation for the fresh network.
/reset — Back to moons → moons, 60 points, 2 frozen layers, 0 epochs.

Glossary

Transfer learning: Reusing a network trained on a data-rich source task as the starting point for a target task with little data. The lower layers, which detect generic patterns, are taken as-is; only what is target-specific gets learned. It is the norm in vision and language: almost nobody starts from zero.
Pre-training: First training phase, on the source task, which produces the starting weights. Expensive but done once and then shared: ImageNet for images, billions of words for language models. Here: 400 points, 8 epochs, cached.
Fine-tuning: Continuing to train the pre-trained network on the target task, with a small learning rate. You can free every layer or only the last ones: the more you free, the more the network adapts — and the more it risks overfitting when data is scarce.
Layer freezing: Preventing weight updates on some layers during fine-tuning: their gradients are ignored. Freeze the lower, most generic layers first. Fewer free parameters means less overfitting and faster training.
Classification head: The last layers of the network, which turn the extracted features into a decision. This is the part you always replace or retrain when transferring: 12 weights and 1 bias here, out of 205 total parameters.
Feature extraction: Using the pre-trained network, frozen, as a fixed function that turns the input into a feature vector, then training a simple classifier on top. That is the /freeze 2 case: the source "sees", the head decides.
Source / target domain: The domain is the data distribution. The source is that of pre-training (abundant), the target that of the actual task (scarce). Transfer works better the more the two resemble each other: same moons slightly shifted, yes; moons then circles, much less.
Domain shift: Gap between the source and target distributions: here a rotation and a translation. Small shift, transfer holds; large shift, the frozen features become unfit and transfer can do worse than chance (negative transfer).
Catastrophic forgetting: When a network retrained on a new task overwrites what it knew of the old one. Freeing every layer with a large learning rate invites it; freezing lower layers or reducing the step size limits it.
Few-shot: Regime where the target task only provides a few examples per class (10 here). Learning 205 parameters from scratch is nearly impossible; reusing 192 already-learned ones and tuning only 13 becomes reasonable. Transfer is the first answer to a data shortage.

Other channels in Deep learning

#optimizers — SGD, Momentum and Adam: the race to the minimum.
#batch-normalization — Batch normalization: keeping activations in the right range.
#rnn-lstm — RNN and LSTM: remembering a sequence.
#autoencoder — Autoencoder: compress then reconstruct.
#transfer-learning — Transfer learning: start from an already-trained network.
#gan — GAN: a forger against an inspector