Module 1 — The limits of recurrent networks that attention removes

Course 11 ended on a working LSTM and a clear worry: whatever you do, one machine translation still runs one word at a time and forgets the beginning of a long sentence by the end. This module names those two ceilings — sequentiality and the context bottleneck — and shows the exact idea that removed them. Everything the rest of the course builds is a variation on that idea.

The two problems, stated cleanly

A recurrent network reads a sequence token by token. To compute the hidden state $h_t$, it needs $h_{t-1}$. To compute $h_{t-1}$, it needs $h_{t-2}$. The chain has to be walked, step by step, in order.

Two costs follow from that ordering.

• Sequentiality kills parallelism. A modern accelerator holds thousands of cores that can multiply matrices in one shot. A recurrent network hands them one token at a time. Training time grows linearly with the length of the sentence, and no amount of hardware shortens it. In practice this capped useful sequence lengths in 2016 at a few hundred tokens, well below the length of a chapter or a legal contract.
• The context bottleneck is worse still. In a classic sequence-to-sequence architecture, the encoder compresses the whole input sentence into one fixed-size vector, typically a few hundred numbers. The decoder then produces the output from that vector alone. Long sentences push more information than the vector can hold; the beginning is quietly overwritten by the end, and translation quality collapses beyond roughly forty tokens.

Both problems are structural. Bigger LSTMs, deeper stacks, careful gradient clipping all help, but they treat symptoms.

Bahdanau attention: the bridge idea

The unlock arrived in 2014, in a paper by Bahdanau, Cho and Bengio titled "Neural Machine Translation by Jointly Learning to Align and Translate". The idea is simple to state and revolutionary in consequences: the decoder no longer reads a single context vector, it reads all encoder states with weights that depend on what it is producing right now.

Concretely, at each output step the decoder computes an alignment score between its current state and each of the encoder's hidden states. Those scores go through a softmax and become weights that sum to one. The decoder then reads a weighted average of the encoder states, and that average is its context at this step.

Two consequences make this a bridge, not a fix:

• The bottleneck vanishes because no fixed-size vector separates encoder from decoder. Every encoder position stays reachable, at every output step.
• The weights are interpretable: plotted as a heatmap, they show which input words the decoder is looking at when it produces each output word. For French-English translation, the diagonal is bright with expected off-diagonal spikes — attention discovers alignments no one hand-coded.

Bahdanau attention still runs on top of a recurrent encoder and decoder, so the sequentiality cost is unchanged. But it planted the question that "Attention Is All You Need" would answer three years later: what if the recurrent backbone is not needed either?

A short chronology, 2014 to 2017

Understanding the sequence of ideas is worth more than any single paper, because each step responds to the previous ceiling.

Year	Contribution	What it changed
2014	Bahdanau attention, "align and translate"	encoder outputs stay reachable, alignments become visible
2015	Luong attention (dot-product form)	simpler scoring, sets the template used everywhere since
2016	ByteNet, ConvS2S (Facebook)	proof that convolutions can replace recurrence on translation
2017	"Attention Is All You Need" (Vaswani et al.)	removes both recurrence and convolution, keeps only attention

The 2017 paper is short and technical. Its title is a promise: no LSTM, no convolution, only attention and feed-forward layers. The architecture it proposes is the Transformer, and the rest of this course is a slow, code-first tour of it.

Why parallelism, precisely

The Transformer replaces the recurrent chain with an operation that is applied to all positions at once. Where an LSTM computes $h_t$ from $h_{t-1}$, the Transformer computes every $h_t$ in one matrix multiplication over the whole sequence. The result is that training time on a batch of a hundred sentences no longer depends on the sentences being long: it depends on total FLOPs, which the hardware can distribute across cores.

That is not a small gain. It is the difference between training on ten million sentence pairs in a week (LSTM territory in 2016) and doing so overnight (Transformer territory in 2018). The scaling laws that gave us GPT-3 and beyond rely entirely on this parallelism: without it, no one would train on a trillion tokens.

Parallelism is not free at inference

Training a Transformer is embarrassingly parallel, but generating text one token at a time is not. Module 7 shows why decoder inference is closer to an LSTM in structure than to a Transformer, and why the "KV cache" was invented to keep it usable in production.

What this course inherits from Bahdanau

The variables you will meet next module — query, key, value — are direct descendants of Bahdanau's alignment score. The score becomes a dot product, the softmax stays, the weighted average of encoder states becomes a weighted average of "values". Read the 2014 paper if you want the historical framing; read the 2017 paper if you want the exact object we are about to build.

Keep two open tabs

Have the Transformer paper (arxiv 1706.03762) and the original attention paper (arxiv 1409.0473) side by side while you read the next four modules. Every equation in the newer paper has a counterpart in the older one. The counterpart is instructive: it names what changed and what stayed.

In summary

• Two ceilings stopped recurrent networks: sequentiality (one token at a time) and the context bottleneck (a single vector between encoder and decoder).
• Bahdanau's 2014 attention removed the bottleneck by letting the decoder read a weighted average of all encoder states, with weights that depend on what it is producing.
• The 2017 Transformer removed the recurrent backbone too, keeping only attention and feed-forward layers, which is what unlocked parallel training on massive corpora.
• The trio query, key, value in the next module is a direct descendant of Bahdanau's alignment score, dressed as matrix multiplications.

Next module: the exact arithmetic of attention, computed by hand on three tokens before we ever call a framework.