Lesson 2 — How images are generated
Diffusion sounds exotic and rests on one genuinely clever observation: destroying an image is easy, so learn to undo the destruction.
The training idea
Take a photograph. Add a small amount of random noise. Add a little more. Keep going for perhaps a thousand steps until nothing remains but static.
Every step of that process is trivially easy to perform and, crucially, you know exactly what you did. So you have a thousand pairs of "slightly noisier" and "slightly less noisy" images, for free, for every photograph in your dataset.
Now train a network on one task: given a noisy image, predict the noise that was added. Subtract the prediction and you have a slightly cleaner image.
This is the whole trick, and notice what it bought. The evaluation problem from lesson 1 — nobody can write down an objective for "this image looks good" — disappears entirely. Predicting noise has an exact answer that can be measured precisely, so training is stable and scales cleanly. No adversarial critic, no collapse.
Generating: run it backwards from nothing
To create an image, start from pure random noise — not a noisy photograph, just static — and apply the trained network repeatedly. Each pass removes a little of what it believes is noise. After twenty to fifty steps, an image is there.
It is reasonable to find this suspicious. There was nothing in the noise. Where did the image come from?
From the network's weights. Trained on hundreds of millions of images, the network has learned what natural images look like — that skies are above ground, that faces have a particular arrangement, that shadows fall consistently. Asked to remove noise from static, it produces the most plausible image consistent with what it knows. The noise supplies randomness; the weights supply everything else. Different noise, different image.
How the prompt steers it
Unguided, the process makes some plausible image. Text guidance makes it make the image you asked for.
The mechanism: your prompt is encoded into a vector by a text model trained to place images and text in a shared space — the CLIP idea from the vision course. That vector is fed into the denoising network at every step, so the network's prediction of "what noise is present" becomes conditional on your description. Effectively you have changed the question from "what plausible image is hiding in this noise?" to "what plausible image matching this description is hiding in this noise?".
Two controls follow directly from this, and understanding them explains most of the settings you will meet:
Guidance scale (often labelled CFG) sets how strongly the prompt pulls the process. Low values produce images that ignore parts of your prompt but look natural. High values follow the prompt closely and start to look oversaturated and strange, because the model is being pushed away from what natural images look like. The useful range for most models is around 5 to 9, and the trade-off is prompt adherence against naturalness.
Steps sets how many denoising passes to run. Too few leaves the image unfinished; beyond a point, more steps change almost nothing while costing time and money linearly. Twenty to forty is typical.
The seed is the number that determines the initial noise. The same seed with the same prompt and settings reproduces the same image exactly, which is what makes iteration possible: fix the seed, change one word, and you see what that word did rather than getting an unrelated picture.
Working in latent space
Running a thousand denoising steps on three million pixels would be prohibitive. Latent diffusion — the design behind Stable Diffusion and its relatives — runs the whole process in a compressed space perhaps forty-eight times smaller, then decodes the result to full resolution once at the end.
This single change is why image generation runs on a consumer graphics card instead of a rack of accelerators, and therefore why an open ecosystem of local tools exists at all. Almost every practical consequence in this lesson follows from it.
Control beyond the prompt
Prompts are a blunt instrument for anything compositional. "A person standing on the left facing away" is not reliably achievable by description. Three additions solve different parts of this:
Image-to-image starts from your image with partial noise added rather than from pure noise, so the output retains its composition while changing style or detail. The strength setting decides how far it can drift.
Inpainting regenerates only a masked region and leaves the rest untouched. This is the workhorse of practical editing: remove an object, change a shirt colour, fix a hand.
ControlNet conditions generation on a structural input — a pose skeleton, an edge map, a depth map — so you specify the geometry directly and let the prompt handle style. This is the difference between hoping for a composition and specifying one.
LoRA adapters are small trained files, often a few dozen megabytes, that adjust a base model towards a specific style, character or object. Training one takes a handful of images and a few minutes on a rented GPU, which is why a large library of them exists. This is how consistent characters and house styles are achieved in practice.
| You need to | Use |
|---|---|
| Something new from a description | Text-to-image |
| Restyle an existing image | Image-to-image |
| Change one region only | Inpainting |
| Control pose or composition exactly | ControlNet |
| A consistent style or character | A LoRA adapter |
| Higher resolution | Generate then upscale |
What still goes wrong
Recognisable and worth expecting:
Hands and counting. Improving steadily, still the classic tell. The model learned what hands look like statistically, not that they have exactly five fingers.
Text in images. Increasingly workable in recent models and historically poor, because letters are structure rather than texture.
Compositional relationships. "A red cube on top of a blue sphere" often returns the colours swapped or the arrangement inverted. Prompts describe content well and relations badly.
Consistency across images. Producing the same character in ten scenes needs a LoRA or a reference-conditioned method. It is not achievable by prompt alone.
Generate four images at low step count to find a composition you like. Note its seed. Then raise the steps and refine the prompt with that seed fixed, changing one thing at a time. This costs a fraction of what randomly regenerating does and it converges, because you are actually learning what each change does.
In three sentences
Diffusion works by adding noise to real images step by step, which is easy and gives exact training targets, then learning to predict and remove that noise — sidestepping the problem that nobody can write down an objective for "this looks good". Generation runs the process from pure random noise, with the image coming entirely from the network's learned sense of what natural images look like, steered at every step by a vector encoding your prompt, where guidance scale trades prompt adherence against naturalness and the seed makes results reproducible. Running all of this in a compressed latent space is what puts image generation on consumer hardware, and precise control comes from inpainting, ControlNet and LoRA adapters rather than from longer prompts.