Module 5 — Decoding: temperature, top-k, top-p

The model from module 4 does not directly emit text. At every step it emits a probability distribution over the whole vocabulary — typically 50 000 to 150 000 tokens — and something else has to pick one. That something else is the decoding strategy, and its parameters (temperature, top-k, top-p, repetition penalty) change output quality more than most people realise.

For the customer-support assistant of the running project, decoding is the last cheap knob before you commit to a specific model. Two teams using the same model with two different settings will report very different quality.

From logits to a token

The output of the final Transformer block is a vector of logits: one real number per vocabulary token, no bound. A softmax turns those into a probability distribution:

$p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}$

where $z_i$ is the logit of token $i$ and $T$ is the temperature. The decoding strategy is what samples from this distribution.

The two extremes are trivial:

• Greedy (temperature=0): always pick the highest-probability token. Fully deterministic. Prone to repetition on longer outputs.
• Pure sampling (temperature=1, no truncation): sample directly from the distribution. Diverse. Occasionally chooses very-low-probability tokens that produce nonsense.

Real deployments live between the two, using temperature plus one truncation method.

Temperature warps the distribution

Temperature $T$ acts on the exponents inside the softmax. Two intuitions:

• $T \to 0$: the distribution collapses onto the single highest logit. Greedy.
• $T \to \infty$: the distribution flattens towards uniform. Every token becomes almost equally likely.

Values above 1 are rarely useful in production. A useful range on a decent model is $0.2 \le T \le 0.8$: 0.2 for extractive tasks (structured output, tool-calling), 0.7 for open-ended writing, 0 for reproducible tests.

import torch
import torch.nn.functional as F

def apply_temperature(logits, temperature):
    return logits / temperature

logits = torch.tensor([2.0, 1.0, 0.5, 0.1])
for t in [0.2, 1.0, 2.0]:
    probs = F.softmax(apply_temperature(logits, t), dim=-1)
    print(f"T={t}: {probs.round(decimals=3).tolist()}")

At $T=0.2$ the top token gets ~99 % of the mass; at $T=2.0$ it gets ~40 %. Same logits, four different distributions.

Top-k: keep the k most probable tokens

Top-k sampling replaces the distribution by its top $k$ tokens, renormalised. Everything else gets probability zero.

def top_k(logits, k):
    values, _ = torch.topk(logits, k)
    threshold = values[..., -1, None]
    return torch.where(logits < threshold, torch.full_like(logits, -float("inf")), logits)

Typical values: $k = 20$ to $k = 100$. The problem with top-k is that the right $k$ depends on the distribution's sharpness at that particular step. When the model is confident, $k=50$ dilutes the top token with 49 near-zero-probability alternatives; when the model is uncertain, $k=50$ cuts off legitimate options.

Top-p (nucleus) adapts to each step

Nucleus sampling, introduced by Holtzman et al. in 2020, keeps the smallest set of tokens whose cumulative probability exceeds a threshold $p$.

Sort the probabilities in decreasing order, compute the cumulative sum, keep tokens while the cumulative sum is below $p$, drop the rest, renormalise. When the distribution is peaked, this keeps 2 or 3 tokens; when it is flat, it keeps 50.

This is why top-p has largely replaced top-k in practice. A single value ($p = 0.9$ is a good default, $p = 0.95$ slightly more diverse) works across all steps.

def top_p(logits, p):
    sorted_logits, sorted_idx = torch.sort(logits, descending=True)
    cumulative = torch.softmax(sorted_logits, dim=-1).cumsum(dim=-1)
    mask = cumulative - torch.softmax(sorted_logits, dim=-1) > p
    sorted_logits = sorted_logits.masked_fill(mask, -float("inf"))
    return sorted_logits.scatter(0, sorted_idx, sorted_logits)

Repetition penalties: the last line of defence

Even good decoding can loop. The model gets stuck on "Thank you for reaching out. Thank you for reaching out. Thank you for reaching out." Two mechanisms fight this.

• Repetition penalty (a multiplier, typically 1.1 to 1.3) divides the logit of any token already present in the context, making it less likely to reappear.
• Frequency penalty and presence penalty (from the OpenAI API vocabulary) do the same, additively rather than multiplicatively.

These are Band-Aids. A well-trained instructed model with $T=0.7$ and $p=0.9$ almost never needs them. If yours does, the diagnosis is usually earlier in the pipeline: wrong chat template (module 3), broken alignment, or a truncated prompt.

Repetition penalty leaks into structured output

Applying a repetition penalty to a task where the output is JSON, code or a table breaks things silently: the model avoids repeating the token for " or } or a variable name that must appear multiple times. For structured tasks, set repetition penalty to 1.0 and rely on temperature and top-p only.

Determinism, or lack of it

Setting temperature=0 makes the sampling deterministic, but not the output. On GPUs, the order of floating-point reductions inside attention differs from run to run, so logits differ at the ~$10^{-6}$ level, and if two tokens have very close logits, a different one may win. Add batching, and the same prompt in a different batch position may produce different logits again.

Two common misconceptions to unlearn:

• "Setting temperature=0 guarantees reproducibility." No — it removes one source of randomness but not the others.
• "Setting a random seed guarantees reproducibility." No — the seed controls PyTorch's PRNG but not CUDA reductions.

For true reproducibility, batch size 1, deterministic kernels, single-GPU inference, plus temperature=0. Expect a 2 to 5x slowdown.

Recommended defaults by task

• Structured output (JSON, function calls): $T=0$, greedy, no penalties.
• Customer-support drafts: $T=0.3$, $p=0.9$, repetition penalty 1.05.
• Creative writing: $T=0.8$, $p=0.95$, no penalty.
• Code completion: $T=0.2$, $p=0.95$, no penalty. Publish these somewhere everyone on the team can find; inconsistent decoding across services is a major source of unreproducible bug reports.

In summary

• Decoding samples the next token from a probability distribution; temperature sharpens or flattens it, top-k and top-p truncate it.
• Top-p (nucleus) adapts to each step and is the modern default, with $p \in [0.9, 0.95]$ across most tasks.
• Repetition penalty helps a badly aligned or badly prompted model and breaks structured output; treat it as a symptom to investigate, not a permanent setting.
• $T=0$ removes one randomness source but not all; true reproducibility on GPUs requires batch size 1, deterministic kernels and single-device inference.

Next module: context window and memory. Once the decoding is sane, the next scaling question is what the model can remember across a long conversation.