Module 6 — ControlNet: pose, edges and depth
Prompts describe. Denoising strength preserves. Neither can say "put the chair here, at this angle, with a hand touching the backrest". That is a job for ControlNet — a companion network that reads a structural signal extracted from a reference image (edges, depth, human pose) and steers the U-Net toward images consistent with that structure. This module wires the three preprocessors that matter for a product-photography workflow, tunes the control weight, and shows how to combine two controls without overriding the model.
The BoisClair goal for this module: a staged shot of the walnut chair in a living-room, with a stylized human figure sitting on it in a specific pose, without hiring a model or renting a set.
The mechanic in three sentences
ControlNet is a copy of the U-Net's encoder plus a small "hint block", trained to inject an extra signal at every scale during denoising. You feed it a control image (Canny edges, a depth map, an OpenPose skeleton, etc.) and it whispers geometry into the U-Net at each step. The base U-Net still handles style, color and texture from your prompt; ControlNet only fixes the geometry.
Two knobs. controlnet_conditioning_scale — how loud the whisper is. And control_guidance_start / _end — the fraction of the trajectory during which the whisper is applied.
The three preprocessors that matter
Canny edges — a classical edge-detection filter that returns a black-and-white line drawing of the reference. Best when you have a clean line sketch or you want the output to follow the exact contours of a photograph.
Depth — a monocular depth estimator (MiDaS, DPT) returns a grayscale map where near is bright and far is dark. Best when what you care about is the layout of a scene in 3D — a chair on a rug, a lamp behind it — without pinning every contour.
OpenPose — extracts a skeleton of any person in the reference image and returns a colored stick figure. Best when a human pose must be reproduced exactly and the rest of the frame is free to be reinvented.
For BoisClair's living-room shot, we combine two: depth to lock the room and the chair's position, OpenPose to place a figure sitting on it.
A first ControlNet generation with depth
import torch
from PIL import Image
from diffusers import (
StableDiffusionXLControlNetPipeline,
ControlNetModel,
DPMSolverMultistepScheduler,
)
from transformers import DPTImageProcessor, DPTForDepthEstimation
# 1. Extract a depth map from a reference photograph of a living-room
proc = DPTImageProcessor.from_pretrained("Intel/dpt-large")
depth = DPTForDepthEstimation.from_pretrained("Intel/dpt-large").to("cuda")
ref = Image.open("living_room_reference.jpg").resize((1024, 1024))
inputs = proc(images=ref, return_tensors="pt").to("cuda")
with torch.no_grad():
predicted = depth(**inputs).predicted_depth
depth_map = predicted.squeeze().cpu().numpy()
# ... normalise to 0..255, tile to 3 channels, save as PIL image `depth_pil`
# 2. Build a ControlNet SDXL pipeline
controlnet = ControlNetModel.from_pretrained(
"diffusers/controlnet-depth-sdxl-1.0",
torch_dtype=torch.float16,
)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
controlnet=controlnet,
torch_dtype=torch.float16,
variant="fp16",
).to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
pipe.scheduler.config, use_karras_sigmas=True,
)
# 3. Generate
prompt = (
"walnut mid-century dining chair on a beige linen rug, "
"sunlit living-room, large window on the left, hardwood floor, "
"editorial interior photography, sharp focus"
)
image = pipe(
prompt=prompt,
image=depth_pil,
controlnet_conditioning_scale=0.7,
control_guidance_start=0.0,
control_guidance_end=0.8,
num_inference_steps=30,
guidance_scale=6.5,
generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("boisclair_livingroom_depth.png")
Control weight: the loudness dial
controlnet_conditioning_scale runs from 0.0 (ignore the control) to about 2.0 (dominate the prompt). Four regions.
- 0.0–0.3: the control is a suggestion. Composition drifts freely.
- 0.4–0.7: the working range for most product shots. Layout is respected, the prompt still decides materials and mood.
- 0.8–1.0: strong lock. Composition is nearly identical to the reference; the model has little freedom.
- Above 1.2: the control overwhelms the model. Colors flatten, textures crack, the image looks like it is painted on the control map.
Two failure modes have a specific diagnostic. If the composition matches the reference but the chair looks like plastic, lower the control weight. If the composition drifts and the chair is fine, raise the control weight.
Start and end: when the whisper speaks
control_guidance_start=0.0, control_guidance_end=1.0 applies the control throughout the trajectory. Two adjustments matter.
End before 1.0 (typically 0.7–0.85). The last steps of denoising decide fine texture; releasing the control there lets the model add material detail that the control map does not know about. This single trick is what avoids the "ControlNet look" where images feel over-constrained.
Start after 0.0 (rarely useful, but valid). If you want the model to establish its own composition before the control snaps in, start at 0.1–0.2. Uncommon for product shots, useful for artistic remixes.
Combining two controls
The multi-ControlNet pipeline accepts a list of ControlNets and a list of control images, one per net, plus a matching list of weights. The rule of thumb: individual weights that would each be 0.7 alone should each drop to 0.5 when combined, because their effects add. Otherwise the U-Net is starved of freedom.
from diffusers import MultiControlNetModel
controlnets = MultiControlNetModel([
ControlNetModel.from_pretrained("diffusers/controlnet-depth-sdxl-1.0", torch_dtype=torch.float16),
ControlNetModel.from_pretrained("thibaud/controlnet-openpose-sdxl-1.0", torch_dtype=torch.float16),
])
pipe.controlnet = controlnets
image = pipe(
prompt=prompt + ", a person in a sweater sitting on the chair",
image=[depth_pil, openpose_pil],
controlnet_conditioning_scale=[0.5, 0.6],
control_guidance_end=[0.8, 0.9],
num_inference_steps=30,
guidance_scale=6.5,
generator=torch.Generator("cuda").manual_seed(42),
).images[0]
The depth map keeps the room geometry; the OpenPose skeleton pins the figure's posture. The prompt takes care of "sweater" and "sitting".
When ControlNet is not the answer
Three cases where a different tool is cheaper.
Object identity across scenes. ControlNet locks geometry, not identity. Making the exact same chair appear in ten scenes is a LoRA job (module 7), not a ControlNet job.
Small local retouch. Fixing a warped armrest is faster with inpainting (module 5) than by re-generating the whole image under a Canny map.
Style transfer. ControlNet does not carry style. "In the style of illustrator X" through a control map still produces the base model's style. Use a LoRA or an IP-Adapter on top.
An SD 1.5 ControlNet plugged into an SDXL pipeline silently produces garbage. The versions are not interchangeable. Match the base model (sd-1.5, sdxl-1.0) with a ControlNet trained for it — the file name usually says so.
In summary
- ControlNet injects a structural signal — Canny, depth, OpenPose — at every scale of the U-Net; the base model still handles style, color and texture from the prompt.
- Control weight 0.4–0.7 is the working range; above 1.0 the control dominates and images look flat, below 0.3 composition drifts freely.
- End the control before the last steps (0.7–0.85) so fine texture can be added freely; this is what avoids the over-constrained "ControlNet look".
- Combine two controls by lowering each weight (0.5 + 0.6 rather than 0.7 + 0.7); match the ControlNet version (SD 1.5 vs SDXL) to the base model or generation silently degrades.
Next module: teaching the model a house style with a LoRA on 20 to 40 images, and spotting overfitting before it ships.