LoRA Training Guide

This guide explains how to use train_person_sdxl_lora.ipynb locally or in Google Colab. The notebook trains a Stable Diffusion 1.5 DreamBooth LoRA at 512 px and is configured for NVIDIA GPUs with approximately 8 GB of VRAM.

Use only photographs of yourself or an adult who explicitly consented to training and the intended generated images. The notebook keeps its consent checks and the base model's supplied image safety component enabled.

Requirements

• 6 usable unique images minimum; 8-20 varied images are recommended
• One consenting adult per image, without other prominent faces
• JPG, JPEG, JFIF, PNG, WebP, BMP, TIFF, HEIC, HEIF, or AVIF files
• Approximately 15 GB of free disk space for dependencies, model files, class images, checkpoints, and outputs
• Internet access during initial setup and model downloads
• An NVIDIA CUDA GPU; 8 GB VRAM is sufficient for the default configuration

Animated and multi-frame images, likely group photos, near-duplicates, corrupt files, and images without a detectable face are skipped during preprocessing.

Prepare Reference Images

Place all reference images in one folder. Use sharp, unfiltered photographs with varied lighting, backgrounds, expressions, and camera angles. Include a mixture of close portraits and upper-body photographs if the adapter will generate both portraits and business scenes.

Avoid:

• Group photographs
• Repeated or nearly identical images
• Heavy beauty filters
• Watermarks and text overlays
• Sunglasses in most images
• Very small, blurred, or obstructed faces

The notebook never modifies the original files. It creates normalized 512 x 512 training copies under the configured output directory.

Run Locally

1. Install prerequisites

Install the following before opening the notebook:

• Python 3.10 or 3.11
• JupyterLab, Jupyter Notebook, or VS Code with the Python and Jupyter extensions
• A current NVIDIA driver compatible with CUDA 12
• Git, when cloning the repository

Clone the repository and enter it:

git clone <repository-url>
Set-Location image-trainer-model

Open train_person_sdxl_lora.ipynb and select the intended Python kernel.

2. Bootstrap the environment

Run the Install dependencies cell. It preserves an existing CUDA-enabled PyTorch installation. When PyTorch is absent, it installs the CUDA 12.8 build and downloads the version-pinned official Diffusers training script.

If the cell asks for a kernel restart:

1. Restart the Jupyter kernel.
2. Run the dependency cell again; it is safe to rerun.
3. Continue with the configuration cell.

3. Configure the run

Run the configuration cell and enter:

• The absolute or relative path to the reference-image folder
• The consenting adult's name

Example Windows path:

C:\Users\your-name\Pictures\person-reference

The notebook derives a unique trigger token and creates output under:

outputs/<normalized-name>_sd15_lora/

The default settings are designed for an 8 GB GPU:

• Resolution: 512 px
• Batch size: 1
• Gradient accumulation: 4
• FP16 mixed precision
• Gradient checkpointing
• Precomputed text embeddings
• 8-bit Adam
• LoRA rank 32
• Adaptive 600-1,200 training steps

4. Validate and preprocess

Answer the adult-consent and responsible-use prompts. The validation cell confirms CUDA availability and reports detected VRAM.

Run the preprocessing cell and inspect its image grid and report before training. The report lists accepted, held-out, and skipped files. If fewer than six usable unique images remain, improve the reference set and rerun preprocessing.

5. Train

Run the training-configuration cell, review the printed command, and then run the training cell. The first run also creates reusable generic class images for prior preservation.

Typical training time is approximately 1-3 hours on a Colab T4 and varies by local GPU. Training checkpoints are written every 250 steps. With RESUME_FROM_LATEST=True, rerunning the training cell resumes the latest compatible checkpoint in the same output directory.

6. Generate and evaluate

Run the memory-cleanup and inference cells. Edit the business portrait example or call:

image = generate_person_image(
    prompt="professional executive portrait in a modern office",
    width=512,
    height=768,
    steps=40,
    guidance_scale=6.5,
    seed=42,
    adapter_strength=0.9,
)

The helper automatically inserts the learned subject token. Generated files are saved under generated/. The evaluation cell reports held-out face-embedding similarity, CLIP prompt alignment, and basic image-quality indicators. These metrics are diagnostic and do not guarantee likeness or correctness.

Run in Google Colab

1. Select a GPU

In Colab, select Runtime > Change runtime type > T4 GPU or a better NVIDIA GPU. Confirm a GPU was assigned before starting:

!nvidia-smi

2. Open the notebook

Use either approach:

• Open the notebook directly from its GitHub URL using File > Open notebook > GitHub.
• Clone the repository in Colab and open the notebook from the Files panel:

!git clone <repository-url>
%cd image-trainer-model

Ensure the current working directory is the cloned project before running the dependency cell when using the clone approach.

3. Persist inputs and outputs with Google Drive

Colab's /content storage is temporary. Mount Drive before configuration if the training inputs or outputs must survive a runtime reset:

from google.colab import drive
drive.mount("/content/drive")

Use a Drive reference path when prompted, for example:

/content/drive/MyDrive/person-reference

To persist outputs, edit OUTPUT_DIR in the configuration cell before running it:

OUTPUT_DIR = Path("/content/drive/MyDrive/person-lora-output") / f"{name_slug}_sd15_lora"

Keep the subsequent PREPROCESSED_DIR, TRAIN_DIR, HOLDOUT_DIR, and GENERATED_DIR assignments unchanged so they derive from the new output location.

4. Install and restart

Run the dependency cell. Colab normally already has CUDA-enabled PyTorch, so the notebook preserves it and installs the remaining pinned packages. If prompted to restart:

1. Select Runtime > Restart session.
2. Rerun the dependency cell.
3. Remount Google Drive.
4. Continue through the remaining cells in order.

5. Train, resume, and download

Run the notebook cells in order. If Colab disconnects, reconnect, rerun setup and configuration with the same values, mount the same Drive path, and run the training cells again. RESUME_FROM_LATEST=True resumes the most recent checkpoint when the output directory was persisted.

The export cell creates:

outputs/<normalized-name>_sd15_lora/<normalized-name>_lora_bundle.zip

When outputs are stored only under /content, download the ZIP before ending the Colab session.

Output Files

The output directory can contain:

Path	Purpose
pytorch_lora_weights.safetensors	Trained LoRA adapter weights
checkpoint-*	Resumable training state
class_images/	Reusable prior-preservation images
dataset/	Preprocessed training and holdout images
generated/	Generated images and prompt metadata
evaluation/	Held-out validation samples
evaluation_results.json	Identity, prompt-alignment, and quality diagnostics
training_configuration.json	Reproducible training configuration
MODEL_CARD.md	Consent, usage, and limitation documentation
*_lora_bundle.zip	Shareable adapter package without private references or checkpoints

Do not commit private reference images, outputs/, generated class images, or checkpoints to a public repository.

Troubleshooting

CUDA is unavailable

• In Colab, confirm the runtime type is set to a GPU and rerun !nvidia-smi.
• Locally, update the NVIDIA driver and verify that the selected notebook kernel uses CUDA-enabled PyTorch.
• Restart the kernel after a new PyTorch installation.

CUDA out of memory

Apply these changes in order, restarting the runtime after an out-of-memory failure:

1. Keep TRAIN_BATCH_SIZE = 1 and LOW_VRAM_MODE = True.
2. Change LORA_RANK from 32 to 16.
3. Reduce RESOLUTION from 512 to 448.
4. Close other GPU applications before training.

Do not increase batch size on an 8 GB card.

Too few images remain after preprocessing

Read the skipped entries in preprocessing_report.json. Add sharper single-person photographs, remove near-duplicates, and ensure the face is visible and reasonably large.

Likeness is weak

• Improve image variety and sharpness before increasing training time.
• Prefer 8-20 high-quality images.
• Try adapter strengths between 0.8 and 1.0.
• Keep the generated subject token in prompts; the helper adds it automatically.

Outputs copy reference clothing or backgrounds

The adapter is likely overfitting. Lower STEPS_PER_TRAIN_IMAGE from 100 to 80, use more varied references, or lower adapter_strength during generation.

Colab files disappeared

Files under /content are temporary. Mount Google Drive and configure OUTPUT_DIR to point into Drive before preprocessing and training.

Privacy and Safety

All reference processing and inference run locally in the selected notebook environment. The notebook adds no external prompt-keyword filter or remote moderation service. It retains the base model's supplied image safety component and explicit adult-consent and responsible-use checks.

Generated images can contain identity, anatomy, text, or contextual errors. Review outputs before use and do not present generated images as authentic photographs when doing so could mislead people.