#transformer-block — LLMs and transformers

The transformer block: attention, residuals, normalization, FFN.

What you'll play with

Welcome to #transformer-block. On screen, a toy transformer seen in cross-section: at the bottom, the yellow box of embeddings + positions; above it, L = 2 stacked blocks, each made of an attention sub-layer (blue) and a FFN (pink), framed by a grey normalization band and, on the left, a yellow arc: the residual connection. At the top, the green output. The 5 tokens of "the cat watches the dog" are beads placed at the input: their size follows the norm of their vector (dimension d = 8), their color its direction. GPT-2 is exactly this cross-section repeated 12 times at d = 768; GPT-3, 96 times at d = 12,288. Everything fits inside this block.
Push the sentence through the stack: /forward. The beads climb sub-layer by sub-layer; on the right, the norm profile appears as they pass.
Cut the residual connection: /residual off. Each sub-layer now replaces the vector instead of correcting it.
Remove normalization too: /norm none. No more grey bands: nothing pins the vector scale.
Wire the residual back in, without restoring the norm: /residual on.
Bring pre-norm back: /norm pre. The grey band returns below each sub-layer: it is the normalized vector that enters attention and the FFN, the residual flow stays intact.
Drop the positional encoding: /position none. The embeddings enter alone, without knowing where they sit.
Stack them up: /layers 6. Twelve sub-layers, and the parameter counter at the top triples: the count is linear in L (and quadratic in d).
Your turn: /params for the parameter breakdown (formula and terms), /norm post for the 2017 recipe, /position learned for a learned position table as in GPT-2, /ffn 64 to widen the FFN, /dimension 16 to double d (attention, quadratic in d, is × 4), /sentence the dog watches the cat to change the text, /step 2 to stop the beads after the FFN of block 1, /reset to start over. Next stop: the #generation-temperature channel, where the output head becomes a probability distribution over the vocabulary and one chooses the next word.

Channel commands

/forward — Re-seed the weights and replay the sentence sub-layer by sub-layer.
/layers <1..6> — Number L of stacked blocks.
/position <sine|learned|none> — Positional encoding added to the embeddings: sine, learned table or none.
/residual <on|off> — Turn the residual connection around each sub-layer on or off.
/norm <pre|post|none> — Place layer normalization before (pre), after (post) or nowhere.
/ffn <8..64> — Width w of the feed-forward hidden layer.
/dimension <8|16> — Dimension d of the vectors (8 or 16).
/sentence <text> — Replace the propagated sentence (the whole line, 2 to 8 words).
/step <n=0..13> — Stop the beads at level n: 0 = input, 1..2L = sub-layers, 2L+1 = output.
/params — Break down the parameter count.
/reset — Back to L = 2, d = 8, w = 32, sine positions, residual on, pre-norm.

Glossary

transformer block: The unit repeated L times inside a transformer: a multi-head attention sub-layer then a feed-forward network, each wrapped by a residual connection and a layer normalization. The embeddings enter at the bottom, the output head reads the last block.
positional encoding: A vector added to the embedding of each token to tell it which place it sits at: sines and cosines of decreasing frequencies (2017) or a learned table (GPT-2). Without it, attention only sees a bag of words and two identical tokens share the same representation.
residual connection: A shortcut that adds the input of a sub-layer to its output: x + f(x). The flow keeps the original information, each sub-layer only learns a correction, and the gradient flows back without shrinking. Without it, the token representations collapse onto each other.
layer normalization: Recenters and rescales each token vector on its d components (mean 0, variance 1, so norm sqrt(d)), with a learned gain and bias. It pins the signal's scale at any depth and stabilizes training.
pre-norm vs post-norm: Where to place the normalization: pre-norm x + f(LN(x)) normalizes the input of the sub-layer and leaves the residual flow untouched (GPT-2, LLaMA); post-norm LN(x + f(x)) normalizes after the sum (2017 transformer, BERT), trickier to train at depth.
feed-forward network: A small perceptron applied to each token separately inside every block: W2 · GELU(W1 · x), with a hidden layer of width w (often 4d). This is where the model stores most of its knowledge, and two thirds of its parameters per block.
multi-head attention: Attention computed h times in parallel on subspaces of dimension d/h, each head with its own Q, K, V matrices, then the results concatenated and projected through an output matrix. Each head can track a different relation (syntax, coreference, position).
encoder vs decoder: Two ways of stacking blocks: the encoder (BERT, this channel) lets every token look at the whole sentence; the decoder (GPT) adds a causal mask — a token only sees the ones before it — to predict the next word. The 2017 transformer combined both for translation.
depth: The number L of stacked blocks: 12 for GPT-2 small, 96 for GPT-3, 80 for LLaMA-2 70B. Residuals and normalization are what makes it possible to go this deep without the signal exploding or fading from one block to the next.
parameter count: Size of the model: per block, roughly 4d² + 2dw (attention + FFN, i.e. 12d² if w = 4d), multiplied by L, plus V×d for the embeddings. Linear in L, quadratic in d: doubling the width costs four times more than doubling the depth.

Other channels in LLMs and transformers

#tokenization — Tokenization and BPE: how an LLM chops up text.
#attention — Self-attention: each word looks at the others.
#transformer-block — The transformer block: attention, residuals, normalization, FFN.
#generation-temperature — Generation: temperature, top-k and top-p.
#rag — RAG: answering from your documents.
#alignment-rlhf — Alignment and RLHF: learning what humans prefer.