Module 4 — Alignment: RLHF and preference methods

Module 3 gave us an assistant that follows instructions. It will now answer "How do I refund a customer?" instead of continuing the sentence. But it may still answer badly — too curt, too verbose, wrong tone for the brand, willing to invent a policy. Alignment is the step that closes the gap between "answers" and "answers the way you would want", and it is the second half of the recipe that turned InstructGPT into ChatGPT.

For the customer-support assistant of the running project, this is the module that decides refusal behaviour, tone consistency and whether the model will make up a refund policy that does not exist.

Why supervised fine-tuning is not enough

Supervised instruction tuning has one hard limit: it only knows how to imitate good examples. It has no way to learn from bad ones. Yet the useful signal from a human reviewer is often "answer A is better than answer B" — a comparative judgement, not a canonical answer.

Preference learning turns that comparison into a training signal. Two paradigms compete: RLHF (Reinforcement Learning from Human Feedback), the original method that shipped ChatGPT, and DPO (Direct Preference Optimization), the 2023 alternative that replaced it in most open-source pipelines.

RLHF in three stages

RLHF is a three-stage pipeline, not a single algorithm.

Stage 1 — Supervised fine-tuning. This is exactly module 3, on an instruction dataset. Output: an SFT model.

Stage 2 — Reward model training. Collect a preference dataset of triples (prompt, chosen_response, rejected_response) — humans see two SFT outputs and pick the better one. Train a small network on top of the SFT model to score any response, with the objective that the score of the chosen one exceeds the score of the rejected one.

$\mathcal{L}_{\text{RM}} = - \log \sigma \bigl( r_\theta(\text{chosen}) - r_\theta(\text{rejected}) \bigr)$

Output: a reward model $r_\theta$ that returns a scalar for any (prompt, response).

Stage 3 — Reinforcement learning with PPO. The SFT model is now the policy. It generates responses, the reward model scores them, PPO updates the policy to increase expected reward, and a KL penalty keeps the policy from drifting too far from the SFT starting point:

$\mathcal{L}_{\text{PPO}} = \mathbb{E}\bigl[ r_\theta(\text{response}) - \beta \, \text{KL}(\pi \, \|\, \pi_{\text{SFT}}) \bigr]$

PPO works. It is also notoriously unstable: the reward model has to be recalibrated, the KL coefficient has to be scheduled, generations have to be sampled at just the right temperature, and any of these can collapse into repetitive or degenerate outputs.

DPO: skip the reward model altogether

The 2023 DPO paper by Rafailov et al. made a clean observation. The optimal policy under an RLHF objective has a closed-form relationship with the reward model. You can invert that relationship, substitute it into the reward-model loss, and end up with a single loss that is trained directly on preference pairs, with no reinforcement learning and no reward model.

$\mathcal{L}_{\text{DPO}} = - \log \sigma \left( \beta \log \frac{\pi(\text{chosen})}{\pi_{\text{ref}}(\text{chosen})} - \beta \log \frac{\pi(\text{rejected})}{\pi_{\text{ref}}(\text{rejected})} \right)$

Two model calls per training step (the current policy and a frozen reference copy), a standard cross-entropy-style loss, no policy rollout, no RL machinery. In practice DPO reaches within a few points of PPO on standard benchmarks, at a small fraction of the engineering cost.

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOConfig, DPOTrainer

model_id = "Qwen/Qwen2.5-7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16")
reference = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16")

preferences = load_dataset("json", data_files="support_prefs.jsonl", split="train")
# Each row has keys: prompt, chosen, rejected

trainer = DPOTrainer(
    model=model,
    ref_model=reference,
    tokenizer=tokenizer,
    args=DPOConfig(
        output_dir="./support-dpo",
        beta=0.1,
        num_train_epochs=1,
        per_device_train_batch_size=1,
        learning_rate=5e-7,
    ),
    train_dataset=preferences,
)
trainer.train()

For the customer-support assistant, this is the practical recommendation as of 2026: collect a few thousand preference pairs from your reviewers, run DPO on top of the SFT LoRA, ship.

Refusal, and the over-refusal trap

Alignment trains the model to refuse certain requests: illegal content, self-harm, medical or legal advice outside its scope. This is done by adding refusal examples to the SFT set and refusal-preferring pairs to the preference set. Done well, it works.

Done poorly, you get over-refusal: the model refuses "How do I kill a Python process on Linux?" because it sees the word "kill". Or it refuses to summarise a news article about a war. Or it hedges every answer into uselessness. The XSTest and OR-Bench benchmarks measure exactly this.

Over-refusal is invisible in standard benchmarks

Truthfulness scores go up when the model refuses more. Helpfulness scores go down, but slower. A model that refuses one legitimate request in five will still look competitive on MMLU. Add an over-refusal test to your evaluation set (module 9) or you will ship the problem.

What alignment does not guarantee

Alignment aligns the style and the stated policy of the outputs. It does not guarantee factuality, it does not eliminate hallucinations, and it does not give the model access to information it did not learn during pretraining. A well-aligned model will confidently state a wrong refund policy in the same polite voice.

Three closely related module numbers to remember:

• Hallucinations are their own problem, addressed in module 7.
• Grounding in real sources is the topic of course 18 on RAG.
• Whether alignment holds under adversarial prompts is a whole other field (jailbreaks), covered in the security-focused course of the track.

Preference pairs are cheaper to collect than gold answers

Two reviewers agreeing that response A is better than B agree faster than one reviewer writing a perfect answer from scratch. Structure your data-collection tool around comparisons, not authoring, and you will get more useful signal per hour of reviewer time.

In summary

• Alignment teaches the model not just to answer, but to answer the way you want; it turns preference pairs into a training signal.
• RLHF works in three stages (SFT, reward model, PPO) and is powerful but unstable; DPO collapses the same objective into a single supervised loss with a frozen reference model.
• Refusal is desirable, over-refusal is a real regression that standard benchmarks do not catch — evaluate it explicitly.
• Alignment aligns style and stated policy; it does not eliminate hallucinations or grant access to knowledge the model does not have, which is why modules 7 and course 18 exist.

Next module: decoding. Once alignment is done, the sampling parameters at generation time make an enormous difference to what your users actually see.