Module 4 — Positional encoding

Modules 2 and 3 built an operation that mixes tokens by content alone. Nothing in it depends on the order in which the tokens arrive. Shuffle the input rows and, up to a matching permutation of the output rows, the result is unchanged. For language that is a disaster: "the dog bit the man" and "the man bit the dog" would receive the same representation. This module puts order back in.

The problem, stated cleanly

Take a sequence $X = (x_1, \dots, x_n)$ and a permutation $\pi$. Apply multi-head attention to $X$ and to $\pi(X)$. You get $Y$ and $\pi(Y)$: same rows, reordered. The mechanism is permutation-equivariant with no notion of "before" or "after".

A recurrent network never has this problem because it walks the tokens in order. A convolution has locality baked in through its receptive field. Attention, by design, sees everything at once and treats positions as interchangeable. Something has to add position information back, and it has to do so before the first attention layer.

The Transformer's answer is deceptively simple: add a position-dependent vector to each token embedding. The sum lives in the same space as the embedding, so the rest of the network needs no change. All the design choice is packed into how that vector is built.

Sinusoidal encoding: the 2017 default

The original Transformer paper defines a fixed vector $\text{PE}_t \in \mathbb{R}^d$ for each position $t$:

$$\text{PE}_{t, 2i} = \sin\!\left(\frac{t}{10000^{2i/d}}\right), \qquad \text{PE}_{t, 2i+1} = \cos\!\left(\frac{t}{10000^{2i/d}}\right).$$

Read the formula slowly. Even and odd coordinates come in $\sin$ / $\cos$ pairs. Each pair oscillates at its own frequency, from very slow (short wavelengths on the last coordinates would be misleading — it is the low-index pairs that oscillate fastest) to very slow at the other end. Every position $t$ gets a unique vector, and neighbouring positions produce vectors that are close by construction.

Two properties make this choice more than a curiosity.

• Relative position is a linear transformation. For a fixed offset $k$, the vector $\text{PE}_{t+k}$ can be written as a fixed rotation applied to $\text{PE}_t$. A subsequent linear layer can therefore learn to attend to "the token five positions to my left" independently of $t$, using a fixed weight.
• Extrapolation past training length, at least in theory. Because the formula is defined for any $t$, positions beyond the ones seen during training receive valid vectors. In practice extrapolation degrades quickly, which is what motivated rotary encoding later.

import torch
import math

def sinusoidal_pe(max_len, d_model):
    pe = torch.zeros(max_len, d_model)
    pos = torch.arange(max_len).unsqueeze(1).float()
    div = torch.exp(torch.arange(0, d_model, 2).float() * -(math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(pos * div)
    pe[:, 1::2] = torch.cos(pos * div)
    return pe

pe = sinusoidal_pe(max_len=512, d_model=64)

We add this to the token embedding at the entry of the encoder:

class TokenAndPositional(torch.nn.Module):
    def __init__(self, vocab_size, d_model, max_len=1024):
        super().__init__()
        self.tok = torch.nn.Embedding(vocab_size, d_model)
        self.register_buffer("pe", sinusoidal_pe(max_len, d_model))

    def forward(self, ids):
        return self.tok(ids) + self.pe[: ids.size(1)]

Save this block under embedding.py. It is the input layer of the Transformer we assemble at module 10.

Learned positional embeddings

BERT and most GPT variants use a learned embedding table indexed by position, exactly like the token embedding but keyed on $t$ rather than on token id. The advantage is expressivity: the model finds its own encoding scheme. The disadvantage is that no vector exists for a position larger than what was seen during training. Beyond the training context length, the model cannot even embed the input — the failure is total, not gradual.

The design choice reduces to a trade-off:

Scheme	Length extrapolation	Learned freedom	Where used
Sinusoidal	mathematically defined, empirically degrades	none, values are fixed	original Transformer, T5, some seq2seq
Learned absolute	none past training context	full	BERT, RoBERTa, GPT-2, GPT-3
Rotary (RoPE)	good, especially with tricks like YaRN	none, values are fixed	LLaMA, PaLM, most modern LLMs
ALiBi	good, biases attention scores directly	fixed slope per head	some efficient models

The last two rows are what today's models actually use, so they deserve their own section.

RoPE: rotate queries and keys instead of adding

Rotary positional encoding, introduced in the RoFormer paper (2021) and popularised by LLaMA, throws away the "add to the embedding" idea entirely. Instead, it rotates each pair of coordinates in $Q$ and $K$ by an angle that depends on the position:

$$q_t \leftarrow R(t) \, q_t, \quad k_t \leftarrow R(t) \, k_t,$$

where $R(t)$ is a block-diagonal matrix of $2 \times 2$ rotations at frequencies chosen exactly like the sinusoidal ones. The trick is that the dot product between a rotated query at position $t_q$ and a rotated key at position $t_k$ depends only on their difference $t_k - t_q$:

$(R(t_q) q_t)^\top (R(t_k) k_t) = q_t^\top R(t_k - t_q) k_t.$

Attention scores therefore become a function of relative position, without any explicit relative-position embedding table. Two benefits follow. First, extrapolating to longer sequences is well-behaved: at inference, the model applies rotations at positions it has never seen, but they lie on the same continuous curve. Second, position lives in the attention operation itself, not in the input embedding, which frees the embedding layer for content only.

Length extrapolation is the hard problem

Everyone wants a model trained on 4k tokens to work on 128k tokens. Naive absolute encodings do not survive that jump; sinusoidal is better on paper but still degrades; RoPE with the right frequency scaling (YaRN, NTK-aware scaling) is what current long-context LLMs actually use.

Extending the context length is not free

Doubling the context length quadruples attention cost — see module 9. Even with FlashAttention, memory grows fast. A model trained on 8k tokens does not "just work" on 200k tokens: the extension requires continued training on long documents with a rescaled positional encoding, plus attention tricks. Vendors sometimes announce a headline context length that only their premium tier actually reaches.

Which one for our red-thread Transformer

For the date-translation task in module 10 we use sinusoidal encoding. The task is short (input length up to twenty characters), the sinusoidal formula makes the position of each digit explicit, and the built-in vector is easy to plot. We swap to RoPE in the "going further" section of module 10, once the baseline works.

Debug positional encoding by ablation

When a Transformer misbehaves on structural tasks — copy, reverse, sort — remove the positional encoding entirely and rerun. If loss curves are identical, the model is not using position at all, which points to a bug in how the encoding is added (wrong shape, broadcasting error, buffer not registered). Attention is order-blind by design, so no positional signal means no positional behaviour.

In summary

• Attention is permutation-equivariant: it needs an external signal to know that "the dog bit the man" is not the same sentence as "the man bit the dog".
• Sinusoidal encoding adds a fixed vector per position, built from $\sin$ / $\cos$ pairs at geometrically decreasing frequencies. Learned encodings offer more freedom but no extrapolation.
• RoPE rotates queries and keys by a position-dependent angle, turning absolute positions into relative ones inside the attention operation itself. It powers most current long-context LLMs.
• Length extrapolation is the hard follow-up: it requires the right encoding and continued training on long sequences with the right scaling.

Next module: residual connections and layer normalization, which keep gradients alive across the many layers we are about to stack.