#generation-temperature — LLMs and transformers

Generation: temperature, top-k and top-p.

What you'll play with

Welcome to #generation-temperature. At the top, the prompt the model in blue tiles. In the center, twelve bars: the next-word distribution computed by a tiny language model trained on 63 English sentences (weather, school, AI, cooking — 142-word vocabulary). An LLM does not "know" which word comes next: it assigns a probability to every word in its vocabulary, then samples from that distribution. Temperature, top-k and top-p tune that sampling — that is the difference between a flat, repetitive text and one that is inventive… or incoherent. On the right, the temperature gauge (0.8 to start).
Type /generate 5: the model samples five words one after another. At each step it looks at the last two words, computes the next-word distribution, samples a word, appends it to the sentence and starts over. The sampled bar lights up, the word joins the sentence, then the bars slide to the distribution of the following word.
Cool it down: /temperature 0.2. Temperature divides the logits (log-probabilities) before the softmax: with T < 1 the gaps grow and the distribution becomes peaked. The text is regenerated with the same seed, only the distribution has changed.
Heat it up: /temperature 1.8. With T > 1 the gaps flatten: the distribution becomes flat and rare words become almost as likely as good ones. Randomness takes over what the model has learned.
Go back to /temperature 0.8, the usual setting of assistants: enough randomness to vary the phrasing, enough structure to stay coherent. A compromise, not a truth.
Another lever: sample only from the top k words. Type /top-k 3. Rare words can no longer come out at all, whatever the temperature.
Top-k is rigid: three candidates whether the distribution is flat or peaked. Reset with /top-k 0, then try /top-p 0.9 (nucleus sampling): keep the smallest set of words whose cumulative mass reaches 90%, so the number of candidates adapts to each step.
What if we stopped sampling altogether? /greedy always picks the most likely word (blue bar): that is greedy decoding, the equivalent of T → 0. The text is regenerated.
Your turn to play: /prompt the soup or /prompt tomorrow it to change subject, /generate 20 for a longer continuation, /word followed by one of the twelve shown words to force the next word (autocomplete suggests them), /temperature 0.05 for near-greedy, /top-p 0.5 for cautious sampling, /clear to start over from the prompt, /reset to restore everything. Next stop: the #rag channel, where we hand the model documents to read before it generates.

Channel commands

/prompt <words> — Changes the prompt (the whole line) and clears the generated continuation.
/temperature <T=0.05..2> — Divides the logits by T: < 1 sharpens the distribution, > 1 flattens it. Regenerates the text.
/top-k <k=0..50> — Samples only from the k most likely words (0 = disabled). Regenerates the text.
/top-p <p=0..1> — Nucleus sampling: keeps words up to a cumulative mass p (1 = disabled). Regenerates the text.
/generate <n=1..30> — Samples n words in a row (5 by default) and appends them to the text.
/greedy — Toggles between random sampling and greedy decoding (always the most likely word), then regenerates.
/seed <n=0..9999> — Changes the sampling seed and regenerates the text (forced words are kept).
/word <word> — Forces the next word from the 12 words shown.
/clear — Empties the generated continuation and keeps the prompt and the settings.
/reset — Restores the prompt "the model", T = 0.8, no filter, seed 7, random sampling.

Glossary

temperature: A number that divides the logits before the softmax: p_i ∝ exp(z_i / T). T < 1 amplifies the gaps (peaked distribution, safe and repetitive text), T > 1 flattens them (flat distribution, inventive then incoherent text). T → 0 reduces to greedy decoding.
top-k: Filter that keeps only the k most likely words and renormalizes their mass to 100% before sampling. Simple and effective against rare words, but rigid: k stays the same whether the distribution is flat or peaked.
top-p: Also called nucleus sampling: keep, by decreasing probability, the smallest set of words whose cumulative mass reaches p (often 0.9), then renormalize. The number of candidates adapts at every step: one when the model is sure, many when it hesitates.
sampling and greedy decoding: Two ways to pick the next word from the same distribution: sampling draws at random according to the probabilities (varied, non-deterministic without a seed); greedy decoding always takes the most likely one (deterministic, often repetitive). Temperature, top-k and top-p only concern sampling.
logits and softmax: The logits are the raw scores z_i the network produces for each word in the vocabulary; the softmax turns them into probabilities: p_i = exp(z_i) / Σ exp(z_j). Temperature applies between the two: softmax(z / T).
entropy: Measure of a distribution's uncertainty: H = −Σ p_i log₂ p_i, in bits. 0 bit when a single word is possible, log₂(V) bits when the whole vocabulary is equiprobable. Temperature drives it up, top-k and top-p drive it down.
perplexity: Exponential of the negative mean log-probability of a text under the model: PP = exp(−(1/N) Σ log p(w_i | context)). It reads as the number of words the model "hesitates" between on average: 1 = perfectly predictable text, V = as surprising as a uniform draw. Used to evaluate language models.
beam search: Instead of keeping a single text (greedy), keep at every step the B best partial sequences and pick the most likely one at the end. Good for translation or summarization, where a "right" answer exists; rarely used for conversation, where it produces flat and repetitive text.
hallucination: A fluent but false or invented statement. High temperature produces them mechanically: unlikely words come out and the model then rides them as if they were true. Lowering temperature or tightening top-p reduces the risk, without removing it: the model can be confident… and wrong.
n-gram: A language model that predicts the next word from the n − 1 previous words by counting sequences seen in a corpus (bigrams: 1 word of context, trigrams: 2). This is the model in this channel, with backoff to lower order and smoothing when a sequence has never been seen. LLMs do the same with a context of thousands of words and a network instead of counts.

Other channels in LLMs and transformers

#tokenization — Tokenization and BPE: how an LLM chops up text.
#attention — Self-attention: each word looks at the others.
#transformer-block — The transformer block: attention, residuals, normalization, FFN.
#generation-temperature — Generation: temperature, top-k and top-p.
#rag — RAG: answering from your documents.
#alignment-rlhf — Alignment and RLHF: learning what humans prefer.