#alignment-rlhf — LLMs and transformers

Alignment and RLHF: learning what humans prefer.

What you'll play with

Welcome to #alignment-rlhf. On screen, the 12 answers a base model can give to "How do I sleep well?", laid out in an arc: the size of a bubble is the probability the model produces it, and they are gray because no reward model exists yet. A base model can write, not please: it will as happily produce "Just sleep" as a real piece of advice. RLHF (reinforcement learning from human feedback) fixes this in three stages: collect human preferences between pairs of answers, train a reward model that imitates them, then optimize the policy to maximize this reward under a KL guard-rail. The outlines are a pedagogical cheat reserved for you: green = genuinely useful answer, red = dangerous. The reward model itself will never see them.
First stage: collect preferences. Type /compare: the base model generates two answers, A and B, that you will have to compare.
Read A and B, then say which you prefer: /prefer a or /prefer b. You are not writing the right answer, you are not scoring out of 10: you are picking between two. This is exactly the job of RLHF annotators, repeated tens of thousands of times.
One pair is not enough. Type /auto 20: a simulated judge, comparing on the real utility and safety of the answers (with a bit of noise), settles 20 new pairs sampled from the base model.
Second stage: type /reward to train the reward model on these pairs. It sees neither utility nor safety: only three surface features — length, positive words, concrete markers (numbers, durations). It learns r(x) = w · f(x) so that P(a ≻ b) = σ(r(a) − r(b)) matches the judgments: this is the Bradley-Terry model.
Third stage: optimize the policy. Type /step 30: 30 iterations of logits ← logits + lr · (r − r̄ − β · (log π − log π₀)). The β term is the KL penalty: it keeps the policy close to the base model π₀ (β = 0.5 for now).
What if we removed the guard-rail? Type /kl 0: β = 0, no more penalty. The policy will be free to maximize reward without asking where the answers come from.
Rerun the optimization: /step 50. Watch answer 11 ("7 amazing tips"), the KL gauge, the length bar and the expected real utility.
Your turn: /answer 11 to read the winning answer and its hidden attributes; /sample to draw an answer according to the current policy; /compare then a judgment to see what the drifted policy now offers judges; /auto 40 then /reward to correct the reward model with pairs drawn from the drifted policy (iterated RLHF); /kl 0.5 then /step 50 to watch the penalty pull the policy back toward good answers; /lr 0.1 for a gentler optimization; /reset to start over. Next stop: #multi-armed-bandit, first channel of the reinforcement theme, where you meet exploration and exploitation, at the root of every reward-based learning method.

Channel commands

/compare — Draws two answers from the current policy: a pair A / B to judge.
/prefer <a|b> — Records your judgment on the current pair: a or b.
/auto <5..40> — Adds n simulated judgments: pairs drawn from the policy, decided by the judge (utility + safety, noisy).
/reward — Retrains the linear reward model (Bradley-Terry) on every judged pair.
/step <1..50> — n policy optimization iterations: logits ← logits + lr·(r − r̄ − β·(log π − log π₀)).
/kl <0..1> — Coefficient β of the KL penalty: force that pulls the policy back toward the base model.
/lr <0.01..1> — Learning rate of the policy optimization.
/answer <1..12> — Highlights one answer (white ring) and reveals what the reward model sees… and what it does not.
/sample — Draws one answer from the current policy and shows it in full.
/reset — Returns to the base model: no pairs, fresh reward model, β = 0.5, lr = 0.5.

Glossary

alignment: Making a model do what its users really want — useful, honest, safe — and not just what its training objective measures. A base model predicts the next word; alignment turns it into an assistant. RLHF is the most widespread recipe for it.
RLHF: Reinforcement learning from human feedback: (1) humans compare pairs of answers, (2) a reward model learns to imitate these preferences, (3) the policy (the LLM) is optimized to maximize this reward under a KL penalty. This is what turned GPT-3 into ChatGPT.
reward model: Model that scores an answer r(x), trained to reproduce human preferences. It replaces the human judge during optimization, where millions of answers must be scored. Here it is linear on three surface features; in practice it is an LLM with a scalar head. It is a proxy: reliable near the data it has seen, unreliable elsewhere.
pairwise preferences (Bradley-Terry): Rather than an absolute score, we ask the judge "A or B?". The Bradley-Terry model relates that choice to scores: P(a ≻ b) = σ(r(a) − r(b)). Training the reward model amounts to maximizing the likelihood of the observed judgments. Comparisons are more reliable and more consistent across annotators than 1-to-10 ratings.
policy (LLM): In reinforcement learning, the policy π is what picks actions; in RLHF, it is the language model itself: a distribution over possible answers. Here, a softmax over 12 answers. The base model π₀ is the starting policy, kept frozen as a reference.
PPO: Proximal Policy Optimization: the most common reinforcement-learning algorithm for the third stage of RLHF. It ascends the gradient of expected reward while limiting the size of each step (bounded probability ratio). The toy version in this channel keeps the essentials: logits ← logits + lr · (r − r̄ − β · (log π − log π₀)), where r̄ is the average reward (baseline).
KL penalty: Term β · KL(π ‖ π₀) subtracted from the reward: it penalizes the policy when it drifts away from the base model. Without it, the policy rushes toward the answers the reward model overrates without having seen them. The fixed point is π ∝ π₀ · exp(r / β): large β = cautious, small β = bold, β = 0 = no guard-rail.
reward hacking: The policy finds answers that score high reward without satisfying the intent behind it (Goodhart's law). Here: long, flattering answers packed with numbers, that the reward model overrates by extrapolation, while their real utility is low. The KL penalty and a reward model regularly retrained on the current policy are the classical remedies.
sycophancy: Tendency of an RLHF-aligned model to flatter the user, agree with their opinions and sugar-coat its answers, because human judges slightly preferred that tone and the reward model amplified it. A special case of reward hacking, visible here in the "Excellent question!" answers.
DPO, RLAIF and Constitutional AI: Three variants of RLHF. DPO (Direct Preference Optimization) skips the reward model and the reinforcement-learning algorithm: a direct loss on preferred pairs tunes the policy, with the same implicit pull toward π₀. RLAIF replaces human judges with an LLM applying a list of principles — the "constitution" of Anthropic's Constitutional AI — to produce preferences at scale.

Other channels in LLMs and transformers

#tokenization — Tokenization and BPE: how an LLM chops up text.
#attention — Self-attention: each word looks at the others.
#transformer-block — The transformer block: attention, residuals, normalization, FFN.
#generation-temperature — Generation: temperature, top-k and top-p.
#rag — RAG: answering from your documents.
#alignment-rlhf — Alignment and RLHF: learning what humans prefer.