Skip to main content

Transformers and the attention mechanism

Build a complete Transformer in PyTorch, block by block, and understand why this architecture replaced recurrent networks on every language and vision benchmark since 2017.

Course duration: 8h​

What you will learn​

  • Explain the two limits of recurrent networks that attention removes
  • Compute attention by hand on three tokens, then vectorise it
  • Assemble multi-head attention, positional encoding and residual blocks
  • Distinguish an encoder (BERT), a decoder (GPT) and an encoder-decoder (T5)
  • Reason about the quadratic cost of attention and the tricks that tame it
  • Train a full Transformer from scratch on a toy task and read its attention maps

Prerequisites​

  • Course 11 (recurrent networks and LSTM): what a sequence model does
  • A framework: course 08 (TensorFlow) or course 09 (PyTorch)
  • Solid Python, NumPy and basic linear algebra: matrix multiplication, softmax

Course modules​

  1. The limits of recurrent networks that attention removes
  2. Query, key, value: attention step by step
  3. Multi-head attention
  4. Positional encoding: absolute, then rotary
  5. Residual connections and layer normalization
  6. The encoder: BERT and understanding models
  7. The decoder: GPT and generation
  8. Encoder-decoder: T5 and translation
  9. Quadratic cost and efficient attention
  10. Implementing a complete Transformer block

The red thread​

Every module extends the same object: a Transformer written by hand in PyTorch, about two hundred lines, built one block at a time. Single-head attention appears in module 2, multi-head in 3, positional encoding in 4, residuals and normalization in 5. The blocks are assembled into an encoder in module 6, a decoder in module 7, an encoder-decoder in module 8. Module 10 trains the finished object on a toy task — first reverse-copy, then date translation, where "3 March 2026" must come out as "2026-03-03". The task is small enough to run on a laptop, and its attention maps are readable, which is why we chose it. Real-world models (BERT, GPT, T5) appear as points of comparison, loaded from the Hugging Face Transformers library.

Assessment and certificate​

The course ends with a 40-question exam covering every module: the arithmetic of attention on three tokens, sizing multiple heads, matching a family (encoder, decoder, encoder-decoder) to a task, catching a missing causal mask, and estimating memory as sequence length grows. On success, a certificate of completion is issued; its number can be verified by anyone on the platform.

Free courses, by contrast, end with a 5-question quiz and a certificate preview, without certification.