Module 4 — Text-to-image and image-to-image

Modules 1 to 3 built the BoisClair reference chair from noise. The catalog now needs the same chair in walnut, in charcoal painted oak, and photographed on a beige linen rug. Running the text-to-image prompt again gives a different chair each time. The right move is image-to-image: start from the reference image instead of from noise, keep most of the composition, change only what the prompt asks for.

The mechanic in one paragraph

Image-to-image reuses the same U-Net, the same scheduler and the same prompt-conditioning machinery. What changes is the starting point. Instead of a pure-noise latent, the pipeline encodes your input image with the VAE, adds a controlled amount of noise to that latent, and then denoises for the remaining steps. If you add 100 % noise you are back to text-to-image. If you add 20 % noise, only the last 20 % of the denoising trajectory runs, and the model can only nudge the image — it cannot reinvent it.

That "amount of noise" is exposed by diffusers under the name strength (also called denoising strength in some UIs). It is the single most important dial in this module.

Denoising strength: the dial

strength ranges from 0.0 (do nothing) to 1.0 (ignore the input, become text-to-image). Four regions matter.

• 0.0–0.2: barely any change. Useful for a final "polish" pass or to change a texture very slightly.
• 0.3–0.5: the working range for on-brand variants. Color swaps, wood grain changes, minor material tweaks — the chair recognizably stays the same object.
• 0.6–0.75: composition can move. The chair may re-orient, the background can be reinvented. Good for ambience shots that keep the "chair species" but let the scene breathe.
• 0.8–1.0: essentially text-to-image with a color palette hint. Rarely what you want if the point was to keep the object.

Everything else — steps, guidance, seed, scheduler — behaves as in module 3. num_inference_steps still means "the full trajectory"; the pipeline runs strength * num_inference_steps of them starting from the noised latent.

Color variants of the BoisClair chair

from diffusers import StableDiffusionXLImg2ImgPipeline
from PIL import Image
import torch

pipe = StableDiffusionXLImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16",
).to("cuda")

reference = Image.open("boisclair_reference_chair.png").resize((1024, 1024))

variants = {
    "walnut":    "warm walnut wood, deep chocolate grain",
    "charcoal":  "matte charcoal painted wood, subtle brush texture",
    "whitewash": "pale whitewashed oak, driftwood tone",
}

for name, material in variants.items():
    prompt = (
        f"mid-century dining chair, four splayed legs, curved backrest, "
        f"{material}, product photography, studio lighting, "
        f"neutral gray backdrop, sharp focus"
    )
    image = pipe(
        prompt=prompt,
        image=reference,
        strength=0.45,
        num_inference_steps=30,
        guidance_scale=6.5,
        generator=torch.Generator("cuda").manual_seed(42),
    ).images[0]
    image.save(f"boisclair_{name}.png")

Five things this script gets right.

Same seed as the reference. With strength=0.45 and the same seed, the trajectory stays close to the reference, and the three variants sit as a family instead of drifting into three unrelated chairs.

Guidance slightly lower (6.5 instead of 7.0). At high denoising strength you want a strong prompt; at low denoising strength a strong prompt fights the image and produces over-baked variants. Below 0.6 strength, drop guidance by about one point.

Prompt shifts only in the material slot. The other four slots of module 2 stay identical. Two words change; the model changes two things.

Consistent resolution. The input must match a resolution the SDXL U-Net was trained for (1024 by 1024, 1152 by 896, 896 by 1152 and a few others). Feeding an odd size wastes compute and can degrade quality.

No prompt for what should not change. "The rug stays beige" is not a spatial constraint the encoder respects. Leaving those regions out of the prompt is usually better than describing them.

When image-to-image is the wrong tool

Image-to-image is easy to over-use. Three cases where a different tool wins.

Small local retouch. Removing a scratch on the armrest, swapping the fabric of a cushion without touching anything else. Full-image image-to-image will drift the rest of the frame; inpainting (module 5) is the surgical tool.

Composition control. Reproducing the exact pose of a person or the exact perspective of a room. Denoising strength cannot pin geometry the way ControlNet can (module 6).

Style transfer to a house style. Making a photograph look like a BoisClair catalog rendered by illustrator X. That is not image-to-image at 0.7 strength — that is a LoRA (module 7) trained on the target style.

Chaining the two modes

The pattern used across BoisClair's catalog looks like this:

1. Text-to-image — produce a hero image from a fresh prompt (module 1 to 3).
2. Image-to-image, low strength — produce the color and material family (this module).
3. Inpainting — fix the local detail that annoys you.
4. Outpainting — extend the frame for a banner or a social-media format.
5. Upscale — bring the print-ready version out.

Each step reads and writes a PNG; each step logs its seed, prompt and strength alongside the file. That log is what lets a colleague pick up the project six months later and produce a new color variant that still looks like a BoisClair chair.

Color drift with high denoising strength

Even at strength=0.5, image-to-image gently shifts hues — a "warm oak" reference can drift toward orange after two passes. If you chain image-to-image on the output of image-to-image, always start again from the original reference for each variant, not from the previous variant. Chained drift is the single biggest silent quality regression in early Stable Diffusion projects.

In summary

• Image-to-image starts from an encoded input latent with added noise, denoises for the remaining steps, and reuses the same U-Net, scheduler and prompt conditioning as text-to-image.
• Denoising strength is the master dial: 0.3–0.5 for on-brand variants, 0.6–0.75 to reinvent the background, 0.8–1.0 is essentially text-to-image with a hint.
• Keep the seed constant and shift only the changing slot of the prompt for a coherent product family; drop guidance by about one point at low strengths.
• Do not chain image-to-image on image-to-image — color drifts. Restart from the original reference for each variant.

Next module: precise local retouch with a mask, and extending the frame beyond the original canvas for a banner-format shot.