Lesson 4 — Sequences and transformers
CNNs handle data laid out in space. A different problem is data laid out in time or order: a sentence, a share price, a patient's history, an audio signal. Here what matters is not just each element but the relationships between elements that may be far apart.
The problem with order
Two sentences with identical words:
The dog bit the man. The man bit the dog.
Any model that treats input as an unordered bag of words cannot distinguish them, and the distinction is the entire meaning. Order carries information, and a sequence model must use it.
There is a second, harder requirement: long-range dependency.
The keys that I left on the kitchen table yesterday evening, next to the letters that arrived while we were away, are missing.
To choose "are" rather than "is", a model must connect it to "keys", eighteen words earlier, past two intervening clauses containing singular nouns. Handling that reliably is the central difficulty of sequence modelling.
Recurrent networks: reading one step at a time
The first approach was the recurrent neural network. It reads the sequence element by element, maintaining a hidden state — a vector summarising everything seen so far. At each step it combines the new element with the current state to produce an updated state.
The idea is sound and it has a structural flaw. The state is a fixed-size vector, so everything about a long sequence must be squeezed into it, and information from early on is progressively overwritten by what follows. In practice a plain RNN loses the thread after a handful of steps, for the same vanishing gradient reason we met in lesson 2: the signal has to travel back through as many multiplications as there are time steps.
LSTM and GRU were designed to fix this, and largely did. They add learned gates that decide what to keep, what to discard, and what to output, giving the network explicit control over its memory. LSTMs powered machine translation and speech recognition for years, and they are a genuine engineering success.
They retain one limitation that no amount of gating solves: they are inherently sequential. Step 50 cannot be computed before step 49. On modern hardware with thousands of parallel cores, that is a crippling constraint, because you cannot parallelise across the length of the sequence. Training on very large corpora becomes impractical, and very long dependencies remain difficult.
Attention: look at everything at once
The 2017 paper "Attention Is All You Need" made a radical proposal: abandon recurrence entirely.
Instead of reading step by step and carrying a summary, let every position in the sequence look directly at every other position, and learn how much to weigh each one.
That is attention. For each word, the model asks: which other words in this sentence matter for understanding me, and how much? Then it takes a weighted average of their representations.
Processing "are" in the sentence above, attention can put most of its weight directly on "keys", eighteen positions back. No hidden state to preserve, no chain of steps to traverse — a direct connection, of the same length regardless of distance.
How attention is computed
Mechanically, it is the dot product from the maths course, applied at scale. Each position produces three vectors:
- a query: what am I looking for
- a key: what do I offer
- a value: what I actually contribute
Each query is compared against every key by dot product, which measures alignment. The resulting scores are turned into weights that sum to one, and the output is that weighted combination of the values.
So the whole mechanism is: dot products to decide relevance, weighted sums to combine. Nothing more exotic than that, applied across every pair of positions simultaneously.
Multi-head attention runs several of these in parallel, each free to attend to different kinds of relationship — one head tracking grammatical agreement, another resolving what a pronoun refers to, another following topical connections.
The two properties that won
Parallelism. Every position is processed simultaneously, which fits GPUs perfectly. This is what made training on internet-scale text feasible, and therefore what made large language models possible at all.
Constant path length. Connecting two positions takes one step regardless of how far apart they are. A dependency across a thousand tokens is no harder than one across three.
Since order is no longer implicit in the processing sequence, transformers add positional encoding: information about each token's position, injected into its representation. Order becomes data rather than a consequence of the architecture.
The cost of attention
Attention compares every position with every other, so the work grows with the square of the sequence length. Double the length and you quadruple the cost.
This is the reason context windows were historically limited, and why extending them was a significant engineering achievement rather than a configuration change. A large family of techniques — sparse attention, sliding windows, linear approximations and better memory handling such as FlashAttention — exists to soften this quadratic wall. The premium transformers course covers them.
Where transformers ended up
The architecture escaped language almost immediately, and the pattern of its spread is the interesting part:
- Text: translation, summarisation, and every large language model
- Images: vision transformers, treating patches like words
- Audio: speech recognition and generation
- Biology: protein structure prediction
- Multimodal: single models handling text, images and audio together
The remarkable fact is that one architecture, designed for translation, turned out to work on nearly any sequence of anything. Attention over a set of elements is apparently a general enough mechanism that specialising per data type matters less than anyone expected in 2017.
In three sentences
Sequence models must use order and connect elements that are far apart, and recurrent networks did this by carrying a fixed-size summary that inevitably loses early information, with LSTM gates improving matters without removing the sequential bottleneck. Transformers dropped recurrence entirely: every position looks at every other through dot-product attention, giving parallel processing and a constant path length between any two positions. Those two properties made internet-scale training possible and are why one architecture built for translation now handles text, images, audio and proteins.