Module 7 — Text conditioning: guiding generation with a prompt

The diffusion model of module 6 samples random faces from noise. To make it obey a prompt like "a small dog wearing sunglasses", two ingredients are needed: a way to turn text into vectors the U-Net can consume, and a mechanism for the U-Net to attend to those vectors while denoising. This module covers both, then explains classifier-free guidance — the trick that made text-to-image generation actually usable.

A text encoder produces vectors the U-Net can attend to

Text is a variable-length sequence of tokens. The U-Net expects tensors of a fixed spatial shape. Bridging the two requires a text encoder that turns a prompt into a sequence of token embeddings, each of a fixed dimension, that the U-Net will read via cross-attention.

Stable Diffusion uses the CLIP text encoder — the text half of the CLIP model trained by OpenAI (Radford 2021) on 400 million image-caption pairs. CLIP was trained so that matching image-caption pairs have similar embeddings and non-matching pairs have dissimilar ones. This alignment property is what makes its text embeddings so useful for conditioning generation: they already sit in a space that has been aligned with image concepts.

from transformers import CLIPTokenizer, CLIPTextModel

tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-base-patch32")
text_encoder = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32").to(device).eval()

with torch.no_grad():
    tokens = tokenizer(
        ["a small dog wearing sunglasses"],
        padding="max_length", max_length=77, truncation=True, return_tensors="pt",
    ).to(device)
    text_emb = text_encoder(**tokens).last_hidden_state  # (1, 77, 512)

The output is a $(B, 77, 512)$ tensor: 77 token positions of 512-dimensional embeddings. The U-Net will use every one of these positions during cross-attention, not just a pooled summary.

Cross-attention: how the U-Net reads the text

Inside each block of a conditioned U-Net, after the usual self-attention over spatial positions, a cross-attention layer lets the image features attend to the text embeddings. The mechanism is the same attention as in a transformer, but the queries come from the image and the keys and values come from the text.

$\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{Q K^\top}{\sqrt{d_k}} \right) V$

with $Q$ derived from the image feature map (shape $B \times HW \times d$), and $K$, $V$ from the text embeddings ($B \times 77 \times d$). Each spatial position of the image asks each word: "how relevant are you to me?" and the answer, weighted, is added to the image features.

class CrossAttention(nn.Module):
    def __init__(self, d_model=256, d_text=512, heads=8):
        super().__init__()
        self.heads = heads
        self.q = nn.Linear(d_model, d_model)
        self.k = nn.Linear(d_text, d_model)
        self.v = nn.Linear(d_text, d_model)
        self.out = nn.Linear(d_model, d_model)

    def forward(self, x, text):
        B, N, D = x.shape
        Q = self.q(x).view(B, N, self.heads, D // self.heads).transpose(1, 2)
        K = self.k(text).view(B, -1, self.heads, D // self.heads).transpose(1, 2)
        V = self.v(text).view(B, -1, self.heads, D // self.heads).transpose(1, 2)
        attn = (Q @ K.transpose(-1, -2)) / (D // self.heads) ** 0.5
        attn = attn.softmax(dim=-1)
        return self.out((attn @ V).transpose(1, 2).reshape(B, N, D))

This is the same computation as transformer self-attention with a source-target twist. The output is added to the image feature map with a residual connection, and life carries on as in module 6.

Training with paired data

Training a conditional diffusion model needs image-text pairs. Large-scale open datasets like LAION-5B collect billions of them from web pages. For each batch, the pipeline is:

1. Sample a real image $x_0$ and its caption $c$.
2. Encode the caption to get $\tau = \text{TextEncoder}(c)$.
3. Sample a random $t$ and noise the image: $x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \varepsilon$.
4. Predict the noise conditioned on $\tau$: $\hat{\varepsilon} = \varepsilon_\theta(x_t, t, \tau)$.
5. Optimise the squared error $\| \varepsilon - \hat{\varepsilon} \|^2$.

The text encoder is usually frozen during this training. Only the U-Net learns; CLIP already knows how to embed text.

Classifier-free guidance, the trick that made prompts work

If you train the model just as described and sample from it, the prompts are only weakly followed. The generator produces something loosely related to the caption, or drifts into generic images. To make prompts crisp, we need classifier-free guidance (Ho and Salimans 2022).

During training, drop the caption with some probability — typically 10% — and replace it with an empty embedding. The U-Net thus learns two things at once: to denoise conditionally on the text, and to denoise unconditionally.

At sampling time, run the U-Net twice at each step, once with the prompt and once with the empty embedding, and combine:

$\hat{\varepsilon} = \varepsilon_\theta(x_t, t, \emptyset) + w \cdot \big( \varepsilon_\theta(x_t, t, \tau) - \varepsilon_\theta(x_t, t, \emptyset) \big)$

$w$ is the guidance scale. When $w = 1$, sampling is purely conditional. When $w > 1$, the model is pushed further in the direction the prompt suggests than the plain conditional would go. Typical values are $w \in [5, 15]$ for Stable Diffusion.

The guidance scale is a real dial

The guidance scale is not a free parameter to leave at the default; it materially changes the samples.

Guidance scale	Behaviour
$w = 1$	prompt loosely followed, images look generic
$w = 3$	reasonable prompt adherence, natural-looking images
$w = 7.5$	Stable Diffusion default, strong prompt adherence
$w = 15$	very strong prompt adherence, images start looking oversaturated
$w \geq 30$	artifacts, cartoonish colours, loss of detail

A higher guidance scale is not "more accurate"

Users often assume that pushing $w$ higher will "listen more closely" to the prompt. Past a point, it produces oversaturated, mannered images that follow the prompt in the aesthetic sense but destroy the natural image distribution. Guidance scale is a quality-versus-fidelity trade-off, not a fidelity dial.

Sampling in practice

@torch.no_grad()
def sample_with_guidance(unet, tokenizer, text_encoder, prompts, w=7.5, n_steps=50):
    text_emb = encode(tokenizer, text_encoder, prompts)
    empty_emb = encode(tokenizer, text_encoder, [""] * len(prompts))
    x = torch.randn((len(prompts), 4, 64, 64), device=device)  # latent shape for SD-like model

    for t in ddim_timesteps(n_steps):
        eps_cond = unet(x, t, text_emb)
        eps_uncond = unet(x, t, empty_emb)
        eps = eps_uncond + w * (eps_cond - eps_uncond)
        x = ddim_step(x, eps, t)
    return decode_latent(x)

At every sampling step, the U-Net runs twice. That is why generating with guidance is roughly twice as slow as without it, and why the CFG-free variants that have appeared since 2024 are an active research direction.

Stable Diffusion in one paragraph

Stable Diffusion combines every ingredient of modules 6 and 7. A latent autoencoder compresses $512 \times 512$ RGB images to $64 \times 64 \times 4$ latents. A U-Net with self-attention, cross-attention and time embeddings denoises those latents, conditioned on CLIP text embeddings. Classifier-free guidance sharpens the prompt adherence at sample time. DDIM sampling in 20 to 50 steps produces the final latent, which the decoder maps back to a full-resolution image. Course 27 dedicates ten modules to that stack; this module is what you need to make sense of it.

Prompt engineering is a debug tool, not a science

Prompts that mention specific artists, camera lenses, lighting terms and negative prompts work not because the model understands them like a human would, but because the training data used those phrases in captions that accompanied the desired visual style. When a prompt fails, the fix is usually to look at what worked for others rather than reason from first principles.

In summary

• A text encoder — CLIP in practice — turns a prompt into a sequence of embeddings that the U-Net attends to via cross-attention.
• The text encoder is usually frozen during diffusion training; only the U-Net's denoiser and its attention layers learn.
• Classifier-free guidance trains the model both conditionally and unconditionally by dropping the prompt with probability around 10%, then extrapolates at sampling time with a guidance scale $w$.
• A high guidance scale sharpens prompt adherence but distorts the image distribution; typical $w \in [5, 15]$, and pushing beyond that produces oversaturated artifacts.

Next module: metrics — how do we measure that these images are "good" when there is no ground truth to compare each sample to, and when likelihood is undefined?