Skip to main content

Module 7 — LoRA and textual inversion for a style

The BoisClair catalog now has coherent chairs and controlled compositions. The next request is harder: make every future image look like BoisClair — the same warm earth-tone palette, the same soft rim light, the same natural fibers in the props, whatever prompt the marketing team writes. That is a style problem, and neither prompt engineering nor ControlNet is enough. The right tool is a small adapter you train yourself.

Full fine-tuning is not the right answer​

Fine-tuning the whole SDXL U-Net on 30 catalog images is possible and almost always wrong. Two reasons.

Storage. An SDXL checkpoint is roughly 6.5 GB. Ten styles, ten checkpoints, 65 GB of disk. Sharing them by email is out; hosting them on a home server is a maintenance job.

Overwriting. Training the full model on a narrow dataset degrades everything the model knew before. Ask for "a cat" after full fine-tuning on chairs and you get a chair with cat fur. It is called catastrophic forgetting for a reason.

Two techniques avoid both problems by learning something much smaller. LoRA learns a low-rank adapter — a few million extra parameters that plug into the U-Net at inference time. Textual inversion learns a single new token embedding — a few kilobytes. Both are additive: the base model is untouched.

Textual inversion in one paragraph​

Textual inversion freezes the entire model and trains one new word embedding to represent a concept present in a small set of reference images. You provide five to ten images and a placeholder token like <boisclair-style>; the training loop tunes the embedding until using that token in a prompt reliably produces the concept. The trained artifact is a .pt or .safetensors file of a few kilobytes.

Strengths: tiny, easy to share, composable with anything else at inference. Weaknesses: because only one token learns, it cannot represent a full style — it works well for a specific object or a single motif (a logo shape, a piece of fabric) and less well for a broad "look". For BoisClair we will use it for the BoisClair fabric pattern, not for the whole style.

LoRA: the workhorse for a style​

LoRA (Low-Rank Adaptation) inserts small trainable matrices into the attention layers of the U-Net. Instead of retraining the full weight matrices, it learns two skinny matrices whose product is added at inference. The trained artifact is 10 to 200 MB depending on rank and coverage — small enough to email.

Four ingredients matter.

A dataset of 20 to 40 images. Below 15 you overfit hard; above 60 you dilute the style. For BoisClair: 30 catalog shots, all in the target aesthetic, at 1024 resolution.

Captions, one per image. A short, honest description with a trigger word you invented: "bcstyle a walnut dining chair on a beige rug, warm sunlight". The trigger word is the token you will use later at inference to summon the style.

A rank. LoRA's rank controls how expressive the adapter is. Rank 4 to 8 for a style, 16 to 32 if you need the model to also learn a new object. Higher rank = larger file, larger risk of overfitting.

Training steps and learning rate. For SDXL, 1500 to 3000 steps at a learning rate around 1e-4 with the Adam optimizer is the standard starting point. Overshoot and the LoRA memorizes your training images; undershoot and the style barely shows.

Training a LoRA with diffusers​

The diffusers training scripts (train_text_to_image_lora_sdxl.py) handle the boilerplate — dataset loading, mixed precision, checkpointing. A typical call from the command line:

accelerate launch train_text_to_image_lora_sdxl.py \
    --pretrained_model_name_or_path stabilityai/stable-diffusion-xl-base-1.0 \
    --pretrained_vae_model_name_or_path madebyollin/sdxl-vae-fp16-fix \
    --dataset_name /data/boisclair-catalog-30 \
    --caption_column caption \
    --resolution 1024 \
    --train_batch_size 1 \
    --gradient_accumulation_steps 4 \
    --learning_rate 1e-4 \
    --lr_scheduler cosine \
    --max_train_steps 2500 \
    --rank 8 \
    --checkpointing_steps 500 \
    --validation_prompt "bcstyle a walnut chair, product photography" \
    --output_dir /models/boisclair-lora

On a 24 GB GPU this runs in about two to three hours. On a 12 GB GPU it fits with --train_batch_size 1 --gradient_checkpointing and takes six to eight hours. On less than 8 GB, use SD 1.5 instead of SDXL and accept the quality cost.

Using a LoRA at inference​

Once trained, a LoRA is loaded into any compatible pipeline in two lines.

pipe.load_lora_weights("/models/boisclair-lora", weight_name="pytorch_lora_weights.safetensors")
pipe.set_adapters(["default"], adapter_weights=[0.8])

image = pipe(
    "bcstyle a whitewashed oak chair on a beige linen rug, sunlit interior",
    num_inference_steps=30,
    guidance_scale=6.5,
).images[0]

Two knobs at inference.

The trigger word. Include it in the prompt or the LoRA barely activates. This is one of the most common "my LoRA does nothing" bugs.

The adapter weight. 0.6 to 0.9 is the useful range. 1.0 is often too strong and produces caricatures of the style. Below 0.4 the LoRA is a garnish; above 1.2 it overwhelms the base model.

The two failure modes to watch​

Visual overfitting: memorized training images. Symptoms — the same chair angle keeps appearing regardless of prompt, watermarks or props from the training set leak into unrelated outputs, the LoRA refuses to change materials. Fixes — fewer steps, lower learning rate, more varied training captions, more images.

Style leakage into unrelated concepts. Symptoms — asking for "a cat" produces a cat draped over a walnut chair. Fixes — always include the trigger word only where you want the style, drop the adapter weight, prefer rank 4 over rank 16.

Detecting both failures is a matter of a validation prompt set: five prompts inside the style ("bcstyle a chair", "bcstyle a bookshelf") and five outside it ("a cat on a bed", "a mountain landscape"). Run all ten after each checkpoint and read the images. A LoRA that ruins the outside-style prompts is not ready to ship.

Dreambooth is a different tool

Dreambooth also learns a subject or style, but it fine-tunes more of the model, produces a full-size checkpoint (multi-GB), and is far more prone to catastrophic forgetting. For a style, LoRA is the modern default. For learning a very specific subject (a person's face, a specific product), Dreambooth-LoRA — Dreambooth's training loop combined with LoRA's adapter — is the current sweet spot.

In summary​

  • Full fine-tuning is the wrong tool for a style: too big, too destructive; LoRA learns a small adapter (10–200 MB), textual inversion learns a single token embedding (kilobytes).
  • A LoRA for a house style trains on 20 to 40 captioned images at rank 4–8, ~2500 steps, learning rate around 1e-4, with a trigger word you invent.
  • At inference, include the trigger word in the prompt and set the adapter weight between 0.6 and 0.9; higher weights caricature the style.
  • Detect overfitting with a validation set of ten prompts, half inside the style, half outside; a LoRA that ruins the outside prompts is not ready to ship.

Next module: bringing the print-ready version out of a 1024 hero shot — upscaling by diffusion or by dedicated networks, and restoring faces without turning them plastic.