Module 8 — Encoder-decoder: T5 and translation

Modules 6 and 7 built the two halves separately. The original 2017 Transformer is neither: it is the two halves connected. The connector is a new attention operation called cross-attention, and the resulting architecture is what the paper actually proposed for machine translation. This module wires the encoder and the decoder together, introduces T5 and its "everything is text-to-text" framing, and helps you pick a family for your problem.

Cross-attention: queries from the decoder, keys and values from the encoder

Inside a decoder layer, self-attention lets the current position attend to previous positions of the decoder's own sequence. But the decoder also needs to look at the source sentence — the input the encoder just processed. That is the job of cross-attention.

The mechanism reuses the module 2 formula with a twist:

• Queries come from the decoder's hidden states: what the decoder is currently trying to produce.
• Keys and values come from the encoder's final output: the contextualised representation of the source.

$\text{CrossAttn}(H^\text{dec}, H^\text{enc}) = \text{softmax}\!\left(\frac{H^\text{dec} W_Q \, (H^\text{enc} W_K)^\top}{\sqrt{d_k}}\right) H^\text{enc} W_V.$

A decoder layer therefore contains three sub-layers instead of two:

1. Masked self-attention on the decoder sequence (causal mask).
2. Cross-attention: decoder queries attending to encoder keys and values.
3. Feed-forward network.

Each is wrapped in a pre-norm residual, exactly as in module 5.

import torch.nn as nn
from mha import MultiHeadAttention
from encoder_layer import FeedForward, SubLayerConnection

class DecoderLayer(nn.Module):
    def __init__(self, d_model=512, num_heads=8, dropout=0.1):
        super().__init__()
        self.self_mha = MultiHeadAttention(d_model, num_heads)
        self.cross_mha = MultiHeadAttention(d_model, num_heads)
        self.ff = FeedForward(d_model, dropout=dropout)
        self.sub1 = SubLayerConnection(d_model, dropout)
        self.sub2 = SubLayerConnection(d_model, dropout)
        self.sub3 = SubLayerConnection(d_model, dropout)

    def forward(self, x, memory, tgt_mask=None, src_mask=None):
        x = self.sub1(x, lambda y: self.self_mha(y, tgt_mask)[0])
        x = self.sub2(x, lambda y: self.cross_mha_call(y, memory, src_mask))
        x = self.sub3(x, self.ff)
        return x

    def cross_mha_call(self, q_input, kv_input, mask):
        # Small adapter: multi-head attention where q comes from decoder
        # and k, v come from the encoder memory. The module 3 signature
        # takes a single x; in the red-thread code (module 10) we adjust
        # the MultiHeadAttention module to accept separate q and kv inputs.
        return self.cross_mha(kv_input, mask)[0]  # placeholder wiring

We leave the exact plumbing to module 10, where the full assembly happens. The pattern is what matters here: the decoder now reads two things at every layer, its own past and the encoder's output.

What T5 changed

T5 (Raffel et al., 2019) is an encoder-decoder Transformer. The architecture is the 2017 paper's, tuned. What makes T5 famous is not the layers, it is the framing of every task as text-to-text.

• Translation: input "translate English to French: The house is blue.", target "La maison est bleue.".
• Classification: input "cola sentence: The car drove home.", target "acceptable".
• Summarisation: input "summarize: <article>", target "<summary>".
• Regression: input "stsb sentence1: ... sentence2: ...", target "3.6" — yes, the score is generated as text.

Every task shares the same architecture and the same loss (cross-entropy on tokens). The only thing that changes is the format of the training data. That framing had two consequences:

• Multi-task pre-training is trivial: mix training pairs from every task in the same batch, and the model handles them all.
• The distinction between "backbone" and "task head" disappears: there is no head. The model always predicts tokens.

BART (Facebook, 2019) is a close cousin: same encoder-decoder skeleton, different pre-training (denoising: corrupt the input, reconstruct it). Both are strong choices for generation with a source, that is, tasks where the output depends explicitly on an input passage.

When to prefer each family

The three families you now know cover almost every text task. The choice depends on whether the output is conditioned on an input passage and, if so, how tightly.

Task	Family	Why
Sentiment, spam, intent	Encoder (BERT / RoBERTa)	one label per input, no generation
Named entities, span extraction	Encoder	one label per token, over the same input
Free-form chat, code completion	Decoder (GPT / LLaMA)	continue an open-ended prompt
Translation, summarisation	Encoder-decoder (T5 / BART)	output derives from a specific input passage
Question answering (extractive)	Encoder	answer is a span of the passage
Question answering (generative, RAG)	Decoder or encoder-decoder	answer is written given retrieved context

The last row deserves a note. Modern LLMs are decoder-only, and they answer questions given retrieved passages by placing those passages in the prompt. That works, but it wastes the entire self-attention over the passage on tokens that the model does not need to generate — an encoder would compress them more efficiently. It is a live design debate: encoder-decoder models remain quietly competitive for long-context input tasks, even though decoder-only ones dominate the headlines.

Training an encoder-decoder for translation

The classic setup is teacher forcing. During training, the decoder is fed the ground-truth target sequence shifted by one, and must predict the next token at every position. The causal mask ensures no future leakage.

# Shapes: src (B, S), tgt (B, T)
tgt_in = tgt[:, :-1]          # start-of-sentence + real tokens
tgt_out = tgt[:, 1:]          # what the model must predict
memory = encoder(src, src_mask)
logits = decoder(tgt_in, memory, tgt_mask, src_mask)
loss = F.cross_entropy(logits.reshape(-1, vocab), tgt_out.reshape(-1))

Two subtleties trip up first implementations. The shift by one is what turns a sequence into (input, target) pairs — forget it and the model learns identity. Padding tokens in the target must be masked out of the loss, otherwise their gradient dominates on short sentences padded to full length.

The red thread lands here

Our own Transformer is now an encoder-decoder. The date translation task from module 10 fits this family exactly: the encoder reads "3 March 2026", the decoder writes "2026-03-03". Attention weights become readable maps — the decoder head that produces "2026" should attend to the year token in the source, the one producing "03" for the day should attend to "3". Watching those maps light up on the correct positions is a good sanity check that our full stack of blocks, from module 2 to module 8, is wired properly.

Choose the family before scale

Doubling the size of the wrong family is a waste. If your task is classification, an encoder ten times smaller than the LLM you were considering will match or beat it, at a fraction of the cost. If your task is generation on an input passage, an encoder-decoder in the 250M-parameter range can outperform a 7B decoder-only model that has to hold both passage and generation in its context window. The family is the first choice; scale is the second.

Read the T5 paper's ablation section

Raffel et al. ran hundreds of small experiments comparing encoder-decoder, decoder-only and prefix-LM variants at fixed compute. Their table is the closest thing to a decision matrix the field has. It also predates the LLM boom, so pair it with more recent studies before deciding for a new project.

In summary

• Cross-attention connects the decoder to the encoder: queries from the decoder, keys and values from the encoder's output, everything else unchanged from module 2.
• A decoder layer with cross-attention has three sub-layers — self-attention, cross-attention, feed-forward — each wrapped in a residual.
• T5 proved that every text task can be recast as text-to-text; BART is a close cousin with denoising pre-training. Both remain strong on tasks with a defined input passage.
• Family choice comes first: encoder for understanding, decoder for open-ended generation, encoder-decoder for conditioned generation — then scale to fit the target metric.

Next module: the elephant in the room. All three families we have built pay a quadratic cost in sequence length. We look at what breaks first and what the community does about it.