Module 9 — Compute cost and memory optimization

The BoisClair pipeline of modules 1 to 8 runs comfortably on an RTX 4070 with 12 GB of VRAM. It does not run out of the box on a 6 GB laptop GPU, and it does not scale to a hundred images per minute on the same machine. This module is about the levers that decide how much memory a generation uses, how fast it runs, and when to reach for each lever. It also gives an honest table of what you can expect on the hardware most learners actually have.

What a generation actually costs

Three pieces of a diffusion generation dominate the memory bill.

The U-Net weights. SDXL base in float32 is about 10 GB. In float16 (half precision) it drops to about 5 GB. That halving is essentially free on any GPU newer than 2018 — quality loss is imperceptible.

The activations during a step. Cross-attention layers hold large tensors for the duration of a step. At 1024 by 1024, activations for SDXL are roughly 3 to 5 GB in float16. This is the number that pushes an 8 GB card over the edge on complex prompts.

The VAE at decode time. The final decode from latent to pixel space is memory-heavy: for a 1024 image the VAE briefly needs about 2 GB of activations. On tight cards this is the last moment of the generation that OOMs — with the image almost done.

Two other pieces matter for wall-clock time. The attention operator itself dominates the per-step time; a faster attention implementation is often a bigger win than fewer steps. And the scheduler: modern deterministic schedulers converge in 25 to 30 steps where ancestral ones need 40 to 50.

The five levers you actually use

Half precision. torch_dtype=torch.float16, variant="fp16" at pipeline load. Non-negotiable for any consumer GPU. Cuts weights and activations roughly in half; quality identical for the human eye.

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16",
).to("cuda")

Attention slicing. pipe.enable_attention_slicing(). Splits the attention matmul into smaller chunks that fit in memory. Modest speed cost (a few percent), meaningful memory savings on activations. Turn on at 8 GB VRAM; leave off above 12 GB.

VAE tiling and slicing. pipe.enable_vae_tiling() decodes the VAE in tiles instead of the full latent. Essential for 1536 and 2048 outputs on 8–12 GB cards; unnecessary at 1024 with more than 12 GB.

CPU offload. pipe.enable_model_cpu_offload() moves each sub-module (text encoder, U-Net, VAE) between CPU and GPU as needed, so only one lives on the GPU at a time. Turns an OOM into a working generation on a 6 GB card, at a two-to-three times wall-clock cost.

Sequential model offload — pipe.enable_sequential_cpu_offload(). Even more aggressive; offloads sub-parts of the U-Net. Enables SDXL on 4 GB VRAM at ten-to-twenty times the wall-clock cost. Only for hardware-constrained cases.

A minimal frugal setup on 8 GB VRAM

import torch
from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16",
).to("cuda")

pipe.enable_attention_slicing()
pipe.enable_vae_tiling()
# do NOT enable model_cpu_offload on 8 GB — it works but is three times slower

That configuration produces 1024 SDXL images in about 8 to 12 seconds per image on an RTX 3060 8 GB. Without slicing and tiling, the same card OOMs on a fresh prompt after two or three generations because of Python's fragmentation of the VRAM allocator.

A minimal fast setup on 24 GB VRAM

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16",
).to("cuda")

# torch>=2.0 uses efficient scaled_dot_product_attention out of the box
# nothing else needed at 24 GB VRAM

Same 1024 image: 2 to 3 seconds on an RTX 4090. Batching four images in one call is 5 to 6 seconds — a batch is cheaper per image than four sequential calls because the U-Net weights load only once.

Resolution scales cost non-linearly

Doubling the resolution roughly quadruples the compute and quadruples the memory — attention is quadratic in sequence length, and the latent has four times more elements. That fact drives the ladder from module 8: generate at 1024, upscale by dedicated networks or by a low-strength image-to-image at 2048, do not generate at 2048 from scratch.

Time-per-image on the hardware you might have

The numbers below are order-of-magnitude, SDXL base, 1024 image, 30 steps, DPM++ 2M Karras, PyTorch 2.x, diffusers at a recent version.

GPU	VRAM	Settings	Seconds per image
RTX 4090	24 GB	fp16 only	2–3
RTX 4070	12 GB	fp16	4–6
RTX 3060	8 GB	fp16 + slicing + VAE tiling	8–12
RTX 2060	6 GB	fp16 + model_cpu_offload	25–45
M2 Pro (Mac)	shared 16–32 GB	fp16 via MPS	15–25
CPU only	—	avoid — falls back to SD 1.5 for sanity	5–10 min

Two lessons in this table.

The 8 GB step is where you meet the levers. Above it the defaults just work; below it, without attention slicing and VAE tiling, you spend more time debugging OOMs than generating.

Apple silicon is real but slower. The MPS backend produces correct results and is usable for iteration; production batches still want CUDA.

Cost-per-image on a rented GPU

For a project that outstrips a home machine, renting matters. In 2026 numbers, a mid-tier cloud A100 40 GB is roughly 1.20 to 1.80 USD per hour on the spot market. At 3 seconds per SDXL image, that is about 0.001 USD per image — one-tenth of a cent. A LoRA training run of two hours costs 2 to 4 USD. Nothing in this course is bottlenecked by cloud budget; it is bottlenecked by wall-clock time on a home machine, and the levers above are what solve that.

Torch compile is a trap on iteration

torch.compile(pipe.unet) cuts per-step time by 20 to 30 % on modern GPUs, at the cost of a 30–90 second warm-up that also runs the first time you change any input shape. For batches at fixed resolution it is a good deal; for interactive iteration where you tweak size and prompt, the warm-up dominates and the compile is a net loss.

In summary

• Three costs dominate a generation: U-Net weights (halved by fp16), attention activations (cut by attention slicing), and VAE decode (cut by VAE tiling); learn them in that order.
• On 8 GB VRAM, always enable attention_slicing and vae_tiling; on 6 GB or less, add model_cpu_offload for two to three times slower generation that at least works.
• Doubling resolution quadruples cost; use the 1024-generate then low-strength-upscale ladder from module 8 rather than generating at 2048 from scratch.
• Rented GPU cost is negligible (fractions of a cent per image at 2026 prices); the real bottleneck is home wall-clock time, which the five levers above are built to fix.

Next module: the non-technical constraints that decide whether the images can actually be used — model licenses, brand and personality rights, provenance metadata and acceptable-use policy.