#gradient-descent — Neural networks

Loss is a landscape. The gradient says which way is up, so we step the other way.

What you'll play with

Welcome to #gradient-descent. The surface you see is the loss L(x, y) of a model with only two weights, x and y: blue = low, pink = high. The ball is our current weights. The goal: reach the lowest point. Except no one has the map — we only feel the slope under our feet. That slope is the gradient ∇L.
Roll the ball: /step 10. The red arrow shows -∇L, the direction of steepest descent. At every step, we move by lr × ∇L that way, and we recompute the slope.
The learning rate (lr) sets the length of each step. Let us push it up: /lr 0.9.
Now, /step 10. With a step that is too large, the ball jumps over the bottom, lands higher on the other side… and eventually flies off: the loss diverges.
The other way around, a tiny step is safe… but slow. Set /lr 0.05, then chain /step 50 freely: the loss drops at every step, and yet you are still far from the bottom.
Momentum gives the ball inertia: v ← μ·v − lr·∇L, then p ← p + v. Zigzags (which flip sign at every step) cancel, consistent steps add up. Turn it on: /momentum 0.9, then /start -2.5 2 and /step 40 to compare.
Real losses are not bowls. Switch to /surface hills: two dips, only one is the lowest. Run /momentum 0 then /step 80, then change the start with /start -2 2 and rerun /step 80.
Two last experiments. /surface saddle: at the center the gradient is zero… and yet this is not a minimum — run /step 10 a few times and watch the ball escape. /surface valley with /momentum 0.7 then /step 30: this is where momentum shines, in a curved valley where pure descent crawls. One question remains: how do we get ∇L for millions of weights? Answer in #backpropagation.

Channel commands

/step <n=1..200> — Run n iterations of descent (10 by default).
/lr <0.001..2> — Set the learning rate.
/momentum <0..0.99> — Give the ball inertia (0 = pure gradient descent).
/surface <bowl|valley|saddle|hills> — Change the loss landscape and put the ball back at the start.
/start <x=-3..3> <y=-3..3> — Move the ball and erase the trajectory.
/reset — Back to bowl, lr 0.1, no momentum.

Glossary

Gradient descent: An optimization algorithm: at every step we move the parameters opposite to the loss gradient, θ ← θ − η·∇L. It is the "marble rolling downhill".
Gradient: Vector of partial derivatives of the loss with respect to each parameter. It points to the steepest ascent; we follow its opposite.
Learning rate: The step size η (eta). Too small: thousands of iterations. Too large: you overshoot the minimum and diverge. The single most important knob in practice.
Loss function: Measures the gap between predictions and targets (MSE, cross-entropy…). Learning is minimizing it; its surface is the landscape the marble rolls down.
Local minimum: A pit where every direction goes back up, without being the lowest point of the surface. Gradient descent can get stuck there.
Saddle point: A point where the gradient is zero but the surface goes up in one direction and down in another (horse-saddle shape). In high dimensions, more common than true local minima.
Momentum: We keep a "velocity": each update accumulates past gradients. The marble skips over small dips and accelerates on long slopes.
Stochastic gradient descent: We compute the gradient on a small batch of examples (mini-batch) instead of the whole dataset: cheaper, and the noise helps escape local minima.
Convergence: The moment when steps become negligible: the loss no longer drops. It does not guarantee the global minimum.

Other channels in Neural networks

#neuron — Let us break a neuron apart: inputs, weights, sum, activation.
#dropout — Dropout regularization: the best friend of deep networks.
#activation — Why an activation? ReLU, sigmoid, tanh, Leaky ReLU, GELU and the vanishing gradient.
#gradient-descent — Loss is a landscape. The gradient says which way is up, so we step the other way.
#backpropagation — The computation graph replayed backwards: each node receives ∂L/∂(itself) and the chain rule does the rest.
#overfitting — A big model on few data points: the decision boundary twists until it memorizes the noise.
#cnn-filters — An image is a grid of numbers. A 3×3 filter slides over it, multiplies, adds up: that is a convolution.
#is-it-einstein — Is it Einstein? Two faces go through the scanner of a network that learned only Einstein: Haythem → NO, Einstein → YES. Watching is free; touching is Premium.
#embeddings-3d — A word becomes a vector: close in space = close in meaning, and you can do math on them.