#attention — LLMs and transformers

Self-attention: each word looks at the others.

What you'll play with

Welcome to #attention. At the bottom, the twelve tokens of "the small black cat sleeps on the couch because it is tired". The raised token, black, is the query: the pink arcs leaving it go to every word, and their thickness is the attention weight — the thickest arc heads to "cat", the noun the adjective modifies. This is the mechanism at the heart of transformers: each word builds a query Q, compares it (dot product) to the key K of every word, a softmax turns these scores into weights that sum to 1, and the word's output is the weighted average of the values V. Concrete question: how does a model know that "it" refers to the cat and not the couch? It looks.
Let the pronoun look: type /query it.
The softmax has a temperature T: scores are divided by T before the exponential. Lower it: /temperature 0.3.
Now the opposite: /temperature 3.
Every token is a query in turn: that makes an n × n matrix. First reset /temperature 1, then switch to matrix view: /view matrix.
A language model generates left to right: when it computes word #5, words 6 to 12 do not exist yet. We enforce this with the causal mask: /mask on.
A single head = a single relation. Transformers stack several in parallel: /head all.
Your turn: /head 2 then /query sleeps (the verb looks for its subject), /example agreement ("she" looks at "woman", "him" would look at "man"), /view arcs to return to arcs, /detail cat (the step-by-step calculation: q·k, ÷√d, ÷T, exp, normalization), /seed 7 (numbers move, relations stay), /reset to start over. Next stop: the #transformer-block channel, where this attention is stacked with dense layers and residual connections.

Channel commands

/sentence <word> — Replace the analyzed sentence (the whole line; 2 to 12 tokens).
/query <word> — Pick the token that "looks at" the others (a word from the sentence, or its number).
/head <1|2|3|4|all> — Show a single attention head, or all four stacked.
/temperature <T=0.1..5> — Softmax temperature: low = sharp attention, high = diluted.
/mask <on|off> — Causal mask: each token only sees the tokens that precede it (and itself).
/view <arcs|matrix> — Arcs from the query token, or full attention matrix.
/example <agreement|pronoun|long> — Load a prepared sentence with its most telling query.
/seed <n=1..999> — Change the deterministic noise of embeddings and Q/K/V projections.
/detail <word> — Decompose the weight between the query and this token: q·k, ÷√d, ÷T, exp, normalization.
/reset — Return to the default sentence, query "black", head 1, T = 1, mask off, arcs view.

Glossary

Attention: Operation that lets each element of a sequence weight the others by relevance, then combine their information. In the scene: the arcs leaving the query token, thicker toward the words that matter to it.
Query, key, value (Q, K, V): Three linear projections of the same input vector. The query says what the word is looking for, the key what each word offers, the value what it transmits if it is looked at. The score q·k measures the agreement between what is sought and what is offered.
Softmax and temperature: Turns arbitrary scores into positive weights that sum to 1: exp(s_j / T) / Σ exp(s_k / T). A low temperature T amplifies gaps (sharp attention), a high one squashes them (diluted attention, close to uniform).
Scaled dot-product attention: softmax(Q·Kᵀ / √d_k)·V: the scores are divided by the square root of the key dimension (√4 = 2 here). Without that scaling, in high dimension the dot products get huge and the softmax saturates, killing the gradient.
Multi-head attention: Several attentions computed in parallel with different Q, K, V projections, then concatenated and projected. Each head can specialize in a relation: agreement, subject-verb, meaning, position… /head all stacks them together.
Causal mask: Sets scores toward future tokens to −∞ before the softmax: their weight becomes 0 and the matrix becomes lower triangular. This is what lets a decoder (GPT) generate word by word without cheating by reading ahead.
Self-attention vs cross-attention: In self-attention, queries, keys and values come from the same sequence: the sentence looks at itself (this channel). In cross-attention, queries come from one sequence (the translation being generated) and keys/values from another (the source sentence).
Attention matrix: An n × n table of weights: row i gives how the attention of token i is spread across all tokens, and it sums to 1. /view matrix lays it on the floor, one cube per cell.
Contextualized vector: The attention output for a token: the average of the values V weighted by its weights. The vector for "it" now carries a bit of "cat": this is how a word acquires a sentence-dependent meaning, whereas a static embedding only has one.
Attention entropy: The Shannon entropy of a row of the matrix, in bits: 0 if the token only looks at one word, log₂(n) if it looks at every word equally. It measures whether attention is sharp or diluted; temperature dials it up or down.

Other channels in LLMs and transformers

#tokenization — Tokenization and BPE: how an LLM chops up text.
#attention — Self-attention: each word looks at the others.
#transformer-block — The transformer block: attention, residuals, normalization, FFN.
#generation-temperature — Generation: temperature, top-k and top-p.
#rag — RAG: answering from your documents.
#alignment-rlhf — Alignment and RLHF: learning what humans prefer.