Module 1 — Latent diffusion: autoencoder, U-Net, scheduler
Course 15 explained diffusion as a general idea: corrupt an image with noise, then learn to reverse the corruption. Stable Diffusion is the version of that idea that made image generation cheap enough to run on a single GPU, and it did so by moving the whole process out of pixel space and into a compressed latent space. This module opens the box on that move — three components, one text encoder, one first generation — and sets up the vocabulary the whole course reuses.
The running example, module by module, is the visual identity of a small furniture brand called BoisClair. We start here with a single generation: a reference oak chair on a neutral studio background, the exact image that every other module will remix.
Why "latent" changes everything
Generating a 1024 by 1024 pixel image directly means running a large neural network hundreds of times on a million-dimensional tensor. That is why the first diffusion models needed a data center. Stable Diffusion inserts a variational autoencoder (VAE) that compresses a 1024 by 1024 image into a 128 by 128 latent tensor with 4 channels — roughly 48 times smaller. The heavy denoising work happens on that small tensor, and the VAE decodes back to pixels only at the end.
The savings are not marginal. A 1024 image at 30 denoising steps is a few seconds on a mid-range GPU. In pixel space at that resolution it would be minutes and 40 GB of memory. Everything that follows in this course — ControlNet, LoRA, inpainting, upscaling — is a variation on the same trick.
The three components you must know by name
The VAE (autoencoder) is a pair: an encoder that maps images to latents, and a decoder that maps latents back to images. During generation, only the decoder runs (you start from noise, not from an image). During image-to-image and inpainting — modules 4 and 5 — the encoder runs first to lift the input image into the latent space. A subtly broken VAE is the cause of the classic "waxy skin, muddy colors" look; SDXL ships with a fixed VAE that avoids that pitfall.
The U-Net (denoiser) is the workhorse. Given a noisy latent and a timestep (how much noise is present), it predicts the noise to remove. Iterating that prediction over 20 to 50 steps walks a pure-noise latent back to a clean latent that decodes into a plausible image. The U-Net is where the model's knowledge lives: styles, objects, compositions, the fact that a chair has four legs. SDXL uses a U-Net that is roughly three times bigger than SD 1.5.
The scheduler (also called sampler) is the algorithm that decides how much noise to remove at each step and how the steps relate to the underlying noise schedule. Familiar names: Euler, Euler ancestral, DPM++ 2M, DPM++ SDE, UniPC. Different schedulers converge at different speeds and produce slightly different textures at low step counts. Module 3 comes back to this trade-off.
The text encoder: where the prompt turns into geometry
The three components above do not read English. A separate text encoder — a frozen CLIP for SD 1.5, and two encoders (CLIP-L and OpenCLIP-G) concatenated for SDXL — turns your prompt into a sequence of embedding vectors. The U-Net receives those vectors through cross-attention layers at every scale, which is how "mid-century oak dining chair, studio lighting, neutral gray backdrop" becomes geometry rather than a caption typed on top.
Two consequences you will feel in module 2. First, the text encoder has a context limit (77 tokens for CLIP, doubled by concatenation in SDXL). Very long prompts get truncated silently. Second, the encoder does not understand negations well — "not blurry" is not what a negative prompt does; module 2 explains what it actually does.
Latent sizes and what SDXL changes
For SD 1.5, the native training resolution is 512 by 512 pixels, i.e. a 64 by 64 latent with 4 channels. Generating at that size is fastest and highest quality; generating at 1024 requires an upscaling pipeline (module 8). Generating at 768 by 768 works, but you drift out of the training distribution and duplicated body parts start appearing.
SDXL was retrained at 1024 by 1024 as the native resolution. That single change is why SDXL images are sharper, better composed and more physically consistent than SD 1.5 at the same prompt. The latent is 128 by 128 with 4 channels; the U-Net is bigger; the text encoding is richer. The price is memory — SDXL needs 8 GB of VRAM in half precision, 12 GB comfortably — and a second or two more per image. This course uses SDXL as the reference. SD 1.5 is mentioned when a modest machine (6 GB VRAM or CPU-only) is the constraint.
A first generation with diffusers
import torch
from diffusers import StableDiffusionXLPipeline
pipe = StableDiffusionXLPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16,
variant="fp16",
).to("cuda")
prompt = (
"mid-century oak dining chair, four splayed legs, "
"curved backrest, product photography, studio lighting, "
"neutral gray backdrop, soft shadow, sharp focus"
)
image = pipe(
prompt,
num_inference_steps=30,
guidance_scale=7.0,
generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("boisclair_reference_chair.png")
Five knobs are visible here, and each has a full module later. prompt (module 2), num_inference_steps and guidance_scale (module 3), the seed via torch.Generator (module 3 again for reproducibility), and the implicit scheduler that the pipeline picked for you (also module 3). The variant="fp16" and torch_dtype=torch.float16 are the two lines that fit SDXL on 8 GB — module 9 goes further with attention slicing and CPU offload.
In a graphical UI, the same generation is a graph with three nodes: Load Checkpoint (SDXL base), CLIP Text Encode (your prompt), KSampler (30 steps, guidance 7, seed 42, scheduler DPM++ 2M Karras), plus VAE Decode and Save Image. The code and the graph do the same thing; the code is auditable, the graph is easier to iterate on visually.
In summary
- Stable Diffusion runs diffusion in a compressed latent space thanks to a VAE, making 1024 image generation feasible on a single GPU.
- The three components are the VAE (encode/decode), the U-Net (predict the noise), and the scheduler (walk the denoising steps); a separate text encoder conditions the U-Net through cross-attention.
- SDXL is trained natively at 1024 and uses two concatenated text encoders and a larger U-Net; it is the reference model for this course, SD 1.5 is the fallback for modest hardware.
- A first
diffusersgeneration exposes five knobs — prompt, steps, guidance, seed, scheduler — each of which has its own module later.
Next module: writing prompts that actually change the image, positive and negative, and understanding what a negative prompt can and cannot fix.