Skip to main content

Module 5 — Inpainting and outpainting

Module 4 gave you clean color variants of the BoisClair chair, but two problems remained: the armrest of the walnut version has an ugly seam where wood grain should flow, and the marketing team asks for a 1600 by 900 banner format from a 1024 by 1024 hero shot. Both are jobs for masked generation — inpainting for the local retouch, outpainting for the canvas extension — and both use a dedicated pipeline that a plain img2img cannot replace.

Inpainting: the surgical tool​

Inpainting generates content only inside a mask, leaving the rest of the image untouched, and it does so with a model that was fine-tuned for the task. The dedicated model was trained on pairs of (masked image, filled image), so it knows how to make the newly generated pixels blend with the pixels around them. Using a plain img2img pipeline with a mask trick works, but the edges betray the join.

The two inputs are the image and a binary mask: white where the model should generate, black where it should keep the original. Every pixel value that is not exactly black is treated as "generate here" with a soft weight — so a mask with a slight feather is what avoids a hard seam.

import torch
from PIL import Image
from diffusers import StableDiffusionXLInpaintPipeline

pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
    "diffusers/stable-diffusion-xl-1.0-inpainting-0.1",
    torch_dtype=torch.float16,
    variant="fp16",
).to("cuda")

image = Image.open("boisclair_walnut.png").resize((1024, 1024))
mask  = Image.open("armrest_mask.png").resize((1024, 1024)).convert("L")

fixed = pipe(
    prompt=(
        "walnut wood grain, continuous straight grain across armrest, "
        "matte varnish, sharp focus, product photography"
    ),
    image=image,
    mask_image=mask,
    strength=0.9,
    num_inference_steps=35,
    guidance_scale=7.0,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]
fixed.save("boisclair_walnut_fixed.png")

Five details this script gets right.

The dedicated inpaint checkpoint. sd-xl-1.0-inpainting-0.1 is the SDXL variant fine-tuned for the task; it accepts an extra channel that describes the mask and it produces cleaner edges than the base model with a manual composite.

High strength inside the mask. You want the model to fully redraw the masked area — 0.8 to 1.0 is the useful range for inpainting. Low strength keeps the ugly seam; that is what you are trying to remove.

A prompt that describes the region, not the whole image. The prompt is applied where the mask is white. Describing the chair again wastes the token budget on things the model already sees.

A feathered mask. A binary black-and-white mask with a hard edge produces a visible border; a mask with a soft 4–8 pixel feather at the boundary blends the two regions.

Seed reused from module 4. Same seed, same overall lighting; the fill is coherent with the rest of the image.

Building the mask​

A mask can come from three places, in order of cost.

Painted by hand in an image editor. Fastest for one image; unscalable.

Painted programmatically with PIL for regular shapes. Good for known coordinates — for example a rectangle over the whole background to swap it.

Predicted by a segmentation model. SAM (Segment Anything) with a single click on the chair returns a pixel-precise chair mask; inverting it gives a background mask. This is how the BoisClair project produces "same chair, new floor" variants at scale.

Two habits.

Feather the mask by 4 to 8 pixels before feeding it to the pipeline (ImageFilter.GaussianBlur); the pipeline itself does not do this for you.

Enlarge the mask by a few pixels around the object you are removing. A tight mask leaves a halo of the removed content that the model has to work around; a slightly generous mask gives the model breathing room.

Outpainting: extending the canvas​

Outpainting is inpainting applied to newly added blank pixels around the original image. The workflow: paste the original image into a larger canvas, paint the mask on the newly added area (with the original image kept as black in the mask), and run the inpaint pipeline with a prompt that describes what should be in the new area.

def outpaint_right(pipe, image, extra_width, prompt, feather=16, seed=42):
    W, H = image.size
    canvas = Image.new("RGB", (W + extra_width, H), color=(127, 127, 127))
    canvas.paste(image, (0, 0))

    mask = Image.new("L", (W + extra_width, H), color=255)
    mask.paste(0, (0, 0, W, H))
    mask = mask.filter(ImageFilter.GaussianBlur(feather))

    return pipe(
        prompt=prompt,
        image=canvas,
        mask_image=mask,
        strength=0.95,
        num_inference_steps=35,
        guidance_scale=6.5,
        generator=torch.Generator("cuda").manual_seed(seed),
    ).images[0]

Three principles that make outpainting work.

Extend in stages. Doubling the width in one call rarely works — the model has no context on the far side. Extend by 256 or 384 pixels at a time, feeding the result back in as the new input. Three moderate extensions beat one aggressive one.

Overlap in the mask. Do not start the mask at the exact right edge of the original image. Include 32 to 64 pixels of the original in the "to generate" region, so the model can smooth the join. Otherwise a vertical seam appears where the training distribution ends.

Prompt what should be there, not what should not. Describe the extended region — "continuing neutral gray backdrop, soft ground shadow, subtle vignette" — rather than "same as the left, empty". "Empty" is not a concept the encoder maps cleanly.

For BoisClair, the 1024 hero becomes a 1600 by 900 banner in three passes: extend right 384, extend left 384, crop the top and bottom to 900. The chair stays in the same pixel range; the backdrop grows around it.

Edge coherence: the failure mode to watch​

The one recurring failure of both inpaint and outpaint is a visible transition at the mask boundary: a color break, a texture change, a floating edge. Three counter-measures.

Feather the mask as above. The single most common fix.

Match the noise level. If the source image is very clean and the model produces slightly noisy pixels in the mask, the join reads as a seam. A brief second pass at low strength (0.15) over the whole image after the fill unifies the grain — an old print-industry trick.

Color-match the mask region before generation. In outpainting, filling the blank area with the average color of the adjacent pixels (rather than pure gray) gives the model a starting point close to the target and shortens the transition.

ComfyUI equivalent

The same operations exist as nodes: VAE Encode (for Inpaint), Set Latent Noise Mask, and the same KSampler. ComfyUI's mask editor is more comfortable than PIL for hand-drawn masks; for programmatic and repeatable masks (banner sizes, segmentation-driven fills), scripts win.

In summary​

  • Inpainting uses a dedicated fine-tuned checkpoint and a binary mask (white = generate, black = keep); the prompt describes the region, strength is high (0.8–1.0), and the mask is feathered to avoid a hard seam.
  • Masks come from hand painting, programmatic PIL or segmentation models like SAM; always feather 4–8 pixels and enlarge the mask slightly around removed objects.
  • Outpainting extends the canvas in stages with mask overlap into the original image; a prompt that describes the new region beats a prompt that describes what it is not.
  • Edge coherence is the recurring failure — feather the mask, match noise levels with a light second pass, color-match the blank area before generating.

Next module: leaving prompt-only control behind, and pinning geometry with ControlNet — pose, edges, depth — for staged product shots.