Module 7 — The decoder: GPT and generation

The encoder in module 6 reads a sentence in both directions and produces representations. That works for classification and extraction, and stops there — you cannot ask an encoder to write. Generation requires a decoder: the same building blocks with one crucial change, a mask that stops each position from cheating by looking ahead. This module builds the decoder, trains it with a causal objective, and introduces the KV cache that makes real inference possible.

The causal mask

A decoder is an encoder plus a causal mask on attention. The mask forbids position $t$ from attending to positions $t+1, t+2, \dots$. Concretely, before applying softmax, entries in the score matrix at those positions are set to $-\infty$ so their softmax weights become zero.

$M_{t, s} = \begin{cases} 0 & \text{if } s \le t, \\ -\infty & \text{if } s > t. \end{cases}$

In PyTorch, that is a lower-triangular matrix used with masked_fill:

import torch

def causal_mask(n):
    return torch.tril(torch.ones(n, n)).unsqueeze(0).unsqueeze(0)  # (1, 1, n, n)

mask = causal_mask(5)   # ready to pass as `mask` in MultiHeadAttention

The multi-head module from module 3 already accepts a mask argument. Reusing it means the decoder layer is almost identical to the encoder layer — the only difference is the shape of the mask.

Why the mask is not optional

Removing the causal mask by mistake gives a model that trains beautifully and then produces nonsense at inference. Here is why.

During training, the model receives the full target sequence and predicts each token given all previous ones — with the mask, "previous" is enforced by structure. Without the mask, position $t$ sees position $t+1$, which is the next token. Predicting it becomes trivial: copy the input. Training loss drops to near zero.

At inference, positions $t+1$ onward do not exist yet — you are generating them. The mechanism the model relied on is gone, and its outputs become effectively random. This bug is silent during training and catastrophic afterwards. Every real implementation asserts the mask is present.

The mask must be applied inside every layer

It is tempting, and wrong, to apply the mask only at the last layer. Attention layers mix positions each time they are called, so a leak at layer 3 propagates through layers 4 to 12 regardless of what layer 12 does. The mask has to be a structural property of every attention call in the decoder.

Training: next-token prediction

The decoder's training objective is simple. Given a sequence of tokens $(x_1, \dots, x_n)$, the model predicts, at every position $t$, the distribution over the next token $x_{t+1}$.

Cross-entropy loss is averaged over all positions. The training set is the whole internet. For a corpus of $10^{12}$ tokens, the model sees $10^{12}$ (input, target) pairs — enormous scale, no annotation needed. This is why GPT-family models can be pre-trained on such quantities: the labels are the text itself.

The relationship to BERT's masked objective is instructive:

Aspect	BERT (encoder)	GPT (decoder)
Attention	bidirectional	causal (unidirectional)
Signal per input	15 % of tokens	every token
Best for	understanding	generation
Adapting to a task	fine-tune with a head	prompt or fine-tune

The "signal per input" row is often overlooked: GPT-style training uses every token as a supervised label, while BERT throws away 85 % of the sentence's tokens per pass. That is a large factor in why decoder-only models scale so well.

Generation: from logits to text

At inference, the decoder is fed a prompt and asked to continue, one token at a time.

def generate(model, tokenizer, prompt, max_new=50):
    ids = tokenizer(prompt, return_tensors="pt").input_ids
    for _ in range(max_new):
        logits = model(ids).logits[:, -1, :]           # last position
        next_id = logits.argmax(dim=-1, keepdim=True)  # greedy
        ids = torch.cat([ids, next_id], dim=1)
    return tokenizer.decode(ids[0])

That naive loop hides two decisions.

How to pick the next token. Four options, each with a trade-off:

Strategy	How	Behaviour
Greedy	argmax on the logits	deterministic, often loops or repeats
Beam search	keep the $k$ best partial sequences	fluent, but bland and expensive
Sampling with temperature	sample from softmax at temperature $T$	creative, quality drops beyond $T=1$
Top-$k$ or nucleus sampling	restrict sampling to the most probable tokens	current default for chatbots

Temperature $T$ acts on the softmax: dividing logits by $T$ before softmax. $T = 0$ collapses to greedy, $T = 1$ is unbiased sampling, $T > 1$ flattens the distribution and eventually produces noise. Real chat interfaces set $T$ between 0.7 and 1.0.

When to stop. The model does not know when to stop by itself. Two mechanisms are common: an end-of-text token in the vocabulary (<|endoftext|>) that terminates the loop when sampled, and a maximum length imposed from outside. Without either, the model runs until it hits your budget, and often into repetition.

The quadratic problem, seen from generation

Every new token means running the entire model over the whole sequence generated so far. For a 4000-token generation, that is roughly 4000 forward passes, each on a sequence that grows by one. The naive cost is quadratic in the output length, on top of the quadratic memory of attention itself.

KV cache is the standard fix. Look at the attention operation: for a new position $t+1$, we only need $Q_{t+1}$, but we still compute against all past $K$ and $V$. Those past $K$ and $V$ do not change from step to step — they depend only on positions $\le t$, which we have already seen. So we cache them.

• At the first pass, compute $K$ and $V$ for every position and store them.
• At every next step, compute $K_{t+1}$ and $V_{t+1}$ only, append them to the cache, and attend $Q_{t+1}$ against the full cached $K$ and $V$.

Cost drops from quadratic to linear in the output length. Every serious inference server implements the KV cache. It is also the main memory hog at inference: for a large model with a long context, the cache can outgrow the model weights themselves.

Read the memory of a running LLM

On an inference server, run nvidia-smi while a generation is happening. What grows over time is the KV cache. If you see memory growing past what the model weights alone would require, that is the cache. Cutting the max context length is the direct way to reduce it, at the cost of losing long history.

The GPT family

The decoder-only architecture went from GPT (2018) to GPT-2 (2019), GPT-3 (2020), GPT-4 (2023) with the same core skeleton — bigger, with more data, longer context, and small refinements: pre-norm becomes standard, RoPE replaces sinusoidal encoding, activation swaps to SwiGLU. Open-weight cousins (LLaMA, Mistral, Falcon, Qwen, DeepSeek) rediscover the same skeleton with variations.

The lesson is that decoder-only Transformers scale unreasonably well. Modules 8 and 9 discuss when they are the right family for the job and what limits their growth.

In summary

• A decoder is an encoder with a causal mask: position $t$ attends only to positions $\le t$, enforced by $-\infty$ scores upstream of the softmax.
• Training is next-token prediction on unlabelled text: every token in the corpus is a supervision signal.
• Decoding turns logits into text via greedy, beam, temperature or nucleus sampling; each trades off determinism, diversity and cost.
• The KV cache drops inference cost from quadratic to linear in the output length and is the main memory hog on production servers.

Next module: encoder-decoder Transformers, the T5 family, and cross-attention — the block that joins the two halves.