#rnn-lstm — Deep learning

RNN and LSTM: remembering a sequence.

What you'll play with

Welcome to #rnn-lstm. On screen, a recurrent network unrolled in time: one box per symbol of the sequence 10110, read from left to right. Below each box, the input x_t; above, the hidden state h_t — six bars, blue positive, red negative — and an arrow that hands it to the next step: h_t = tanh(W·x_t + U·h_{t−1} + b), with h_0 = 0. It is the same cell, the same weights, copied at each step: its only memory is h. The current step is highlighted, later ones are dimmed; /advance moves one step. The channel's question: what remains of the first symbol once we reach the end?
Give it a trap sequence — one 1, seven 0s, then one 1: /sequence 100000001. The task is 'memory': at the end, the network has to remember the first symbol.
Jump straight to the end: /step 9. Compare h_9 to what it would be if the sequence had started with a 0 — that is the 'gap to the first symbol' given in the reply.
Why wouldn't training fix this? Because the error signal follows the same path, backwards. Show the norm of the gradient backpropagated through time, step by step: /gradient.
Change the cell: /cell lstm. The LSTM adds a second memory, the cell state c_t — the green ribbon along the top — and three sigmoid gates, drawn as valves inside each box.
Highlight the forget gate: /gate forget. It decides, at each step, how much of c_{t−1} survives.
And the gradient? Ask for the curve again: /gradient.
Your turn: /task counter (count the 1s) then /length 12 for a long random sequence, /gate input or /gate output to read the other valves, /advance to follow h_t and c_t step by step, /sequence abcabc for letters (vowel = 1, consonant = 0), /seed 42 for different weights, /cell rnn and /gradient to compare again, /reset to start over. Next: channel #autoencoder, where a network learns to compress and then reconstruct its inputs.

Channel commands

/cell <rnn|lstm> — Change the recurrent cell (same seed weights); hides the gradient curve and the highlighted gate.
/sequence <word> — New sequence of 3 to 12 symbols: 0/1 or letters (vowel = 1, consonant = 0). Returns to step 1.
/length <3..12> — Draws a random sequence of this length (same alphabet as the current sequence). Returns to step 1.
/step <1..12> — Move to step t (the sequence is already fully unrolled: we just choose what to look at).
/advance — Move to the next step (t + 1); after the last one, wraps back to step 1.
/gate <forget|input|output|none> — Highlight one of the LSTM gates in every cell (valves = sigmoids) and detail its value at the current step.
/gradient — Show or hide the curve (log scale) of the norm of the gradient backpropagated through time, from step T to step 1.
/task <memory|counter|parity> — Change the toy task read from h_t: memory of the first symbol, counter of 1s, parity. Refits the linear readout.
/seed <1..9999> — Re-draw the cell's fixed weights (W, U, biases) and the trial sequences from another seed.
/reset — Back to the simple RNN, sequence 10110, memory task, step 1, seed 7, no gradient curve.

Glossary

recurrent neural network (RNN): Network that reads a sequence one element at a time by reusing the same cell (same weights) at every step, and that passes a state from one step to the next: h_t = tanh(W·x_t + U·h_{t−1} + b). Text, audio, time series: anything with an order.
hidden state: The vector h_t that the cell recomputes at each step: its working memory, everything it 'knows' about the sequence read so far. In a simple RNN, it is entirely rewritten at each step, hence its short memory.
unrolling in time: Representing the recurrent network as a chain of copies of the cell, one per time step, linked by the hidden state. This is what the scene shows: an RNN of T steps is a deep network of T layers that share their weights.
backpropagation through time (BPTT): Backpropagation applied to the unrolled network: the loss gradient flows back from h_T to h_1 by multiplying, at each step, by the Jacobian ∂h_{t+1}/∂h_t. The corrections from every step add up on the same shared weights.
vanishing gradient: When successive Jacobians have norm < 1, their product tends to zero exponentially: the gradient reaching the earliest steps is tiny and the network can no longer learn long dependencies. This is the curve that collapses for the RNN.
exploding gradient: The symmetric problem: Jacobians of norm > 1 make the gradient grow exponentially along the sequence, up to absurd updates. Common remedy: gradient clipping, which caps its norm before the update.
LSTM: Long Short-Term Memory (Hochreiter & Schmidhuber, 1997): a recurrent cell equipped with an additive cell state c_t and three learned gates (forget, input, output). The gradient flows along c without going through any tanh, which fixes vanishing in practice.
forget, input and output gates: Three sigmoid vectors, between 0 (closed) and 1 (open), computed from x_t and h_{t−1}. f decides how much of c_{t−1} survives, i how much of the candidate g_t is added, o how much of tanh(c_t) is exposed in h_t. These are the valves in the scene.
cell state: The ribbon c_t = f_t ⊙ c_{t−1} + i_t ⊙ g_t of the LSTM: a long memory updated by addition, not rewriting. With a forget gate close to 1, information written at step 1 is still there at step 12 — and the gradient makes the reverse trip just as well.
GRU: Gated Recurrent Unit (Cho et al., 2014): a lightweight variant of the LSTM with two gates (update and reset), no separate cell state. Fewer parameters, often comparable results; both have largely been replaced by transformers for text, but remain in use on time series and embedded devices.

Other channels in Deep learning

#optimizers — SGD, Momentum and Adam: the race to the minimum.
#batch-normalization — Batch normalization: keeping activations in the right range.
#rnn-lstm — RNN and LSTM: remembering a sequence.
#autoencoder — Autoencoder: compress then reconstruct.
#transfer-learning — Transfer learning: start from an already-trained network.
#gan — GAN: a forger against an inspector