#optimizers — Deep learning

SGD, Momentum and Adam: the race to the minimum.

What you'll play with

Welcome to #optimizers. On screen, the loss surface L(x, y) of a two-weight model — a bowl ten times steeper along y than along x — and three balls launched from the same point: SGD (blue), Momentum (yellow), Adam (pink). Same slope under their feet, three ways to use it. Watch SGD: it plunges along y then crawls along x, while the other two are already at the bottom after 60 steps. An optimizer is the rule that turns the gradient into weight moves; it is what makes a training take ten minutes… or ten hours.
Onto the real terrain: a curved valley (softened Rosenbrock), the classic nightmare of optimizers. Type /surface valley. All three restart from the same point, high on a slope.
Isolate the first one: /optimizer sgd. Pure gradient descent, p ← p − lr·∇L, nothing else: at each step you move lr times the slope, in the direction of descent.
Triple the learning rate: /lr 0.3. Look at the beginning of the trace.
Momentum keeps a velocity: v ← β·v + ∇L then p ← p − lr·v. Zigzags, which flip sign at each step, cancel out in v; coherent steps along the valley add up. Show it: /optimizer momentum.
Adam adds a second memory: the mean of the squares of the gradient, coordinate by coordinate, and divides the step by its square root. A direction with a huge gradient gets a small step, a flat direction a large one. Show it: /optimizer adam.
Put all three side by side: /compare. The table gives, for each, the final loss and the number of steps to bring the loss below 1% of its initial value.
Your turn: /surface hills then /compare — from the same ridge, all three can end up in three different basins; /surface saddle to see how each escapes the saddle point; /noise 0.5 for noisy gradients like mini-batches; /momentum 0.5 to feel the effect of β; /start -2 -2 to change the initialization; /reset to start over. Next: #batch-normalization, or how to make the landscape itself easier to descend.

Channel commands

/surface <bowl|valley|saddle|hills> — Change the loss landscape (and reset the recommended start).
/optimizer <sgd|momentum|adam|all> — Show a single optimizer, or all three.
/lr <0.001..1> — Set the learning rate for all three optimizers and recompute.
/steps <1..300> — Number of steps played by each optimizer.
/momentum <0..0.99> — Coefficient β: Momentum's momentum and Adam's β1.
/start <x=-3..3> <y=-3..3> — Starting point shared by the three optimizers.
/noise <0..1> — Standard deviation of the noise added to the gradient (simulated mini-batches).
/seed <1..9999> — Noise seed: another reproducible draw.
/compare — Show the three traces and the table of final losses.
/reset — Back to the bowl, lr = 0.1, β = 0.9, 60 steps, no noise, all three displayed.

Glossary

Optimizer: Rule that turns the gradient of the loss into a weight update. SGD, Momentum, RMSProp and Adam are optimizers: same slope in, different moves out. It is one of the choices that most affect training speed.
Learning rate: Factor lr that sets the length of each step: p ← p − lr·∇L. Too small, descent crawls; too large, it zigzags in steep directions and then diverges. On a surface of curvature λ, pure descent is stable only if lr < 2/λ.
Momentum: Adds a velocity to the weights: v ← β·v + ∇L, p ← p − lr·v. Coherent gradients add up, zigzags cancel out. At steady state the effective step is lr / (1 − β) — ten times lr for β = 0.9 — hence more speed in valleys, but a risk of overshoot.
Adam: Optimizer that combines momentum (moving average of the gradient, β1) and RMSProp (moving average of the squared gradient, β2): p ← p − lr·m̂ / (√v̂ + ε), with bias correction of the averages at startup. The step is normalized coordinate by coordinate, which makes it robust to the choice of lr. AdamW is the variant with decoupled regularization, standard for transformers.
RMSProp: Divides each coordinate's step by the square root of a moving average of its squared gradient: directions with a strong gradient get a small step, flat directions a large one. This is the 'adaptive' half of Adam, inherited from Adagrad which accumulated without forgetting and eventually stalled.
Conditioning: Ratio between the largest and smallest curvatures of the loss (eigenvalues of the Hessian). A bowl ten times steeper in one direction has a condition number of 10: the stable lr for the steep direction is ten times too small for the flat one, and pure descent crawls or zigzags. This is the primary problem Momentum and Adam solve.
Saddle point: Point where the gradient is zero without being a minimum: the loss rises in some directions and falls in others. Descent slows dramatically there (the slope is nearly flat) before escaping through the descending direction. In high dimension, saddle points are far more common than local minima.
Local minimum: A dip in the loss lower than its immediate neighbourhood, but not necessarily the deepest of all (the global minimum). Gradient descent stops in whichever basin it reaches first: the starting point (initialization) and the optimizer's momentum decide which basin.
Learning rate schedule: Vary lr over the course of training: large at the start to make progress fast, smaller later to settle at the bottom without zigzagging or shaking under the noise of mini-batches. Step schedules, cosine decay, warmup: almost every modern training uses one.
Noisy gradient: The gradient computed on a mini-batch is only an estimate of the full-set gradient: it is noisy. SGD follows it literally and jitters around the minimum; Momentum averages several and smooths; Adam normalizes its amplitude. This is the 'S' (stochastic) in SGD, simulated here by /noise.

Other channels in Deep learning

#optimizers — SGD, Momentum and Adam: the race to the minimum.
#batch-normalization — Batch normalization: keeping activations in the right range.
#rnn-lstm — RNN and LSTM: remembering a sequence.
#autoencoder — Autoencoder: compress then reconstruct.
#transfer-learning — Transfer learning: start from an already-trained network.
#gan — GAN: a forger against an inspector