deforum support LoRAs

[scottdraves]

can we get lora support in our deforum container?

maybe we can start by just adding a handful of popular ones for testing, we can add them to our container.
but really we need to make this user uploadedable. they are not big & since they are just safetensor format, should be, uh, safe :)

this lora https://civitai.com/models/269592/fractal-geometry-style-fluxsdxl15?modelVersionId=303921 (be sure to get the SDXL version)
was used to make this video https://www.reddit.com/r/Fractalish/comments/1vvsjjt/a_picture_of_my_kitty_spangles/
the prompts and other settings:

{
    "W": 1344,
    "H": 768,
    "show_info_on_ui": true,
    "tiling": false,
    "restore_faces": false,
    "seed_resize_from_w": 0,
    "seed_resize_from_h": 0,
    "seed": 788322001,
    "sampler": "Euler a",
    "scheduler": "Automatic",
    "steps": 25,
    "batch_name": "Deforum_20260821221219",
    "seed_behavior": "iter",
    "seed_iter_N": 1,
    "use_init": false,
    "strength": 0.8,
    "strength_0_no_init": true,
    "init_image": null,
    "use_mask": false,
    "use_alpha_as_mask": false,
    "mask_file": "https://deforum.github.io/a1/M1.jpg",
    "invert_mask": false,
    "mask_contrast_adjust": 1.0,
    "mask_brightness_adjust": 1.0,
    "overlay_mask": true,
    "mask_overlay_blur": 4,
    "fill": 1,
    "full_res_mask": true,
    "full_res_mask_padding": 4,
    "reroll_blank_frames": "ignore",
    "reroll_patience": 10.0,
    "motion_preview_mode": false,
    "prompts": {
        "0": " pretty <lora:ral-frctlgmtry:1> --neg nsfw, nude",
        "600": " happy <lora:ral-frctlgmtry:1> --neg nsfw, nude",
        "1200": " mean <lora:ral-frctlgmtry:1> --neg nsfw, nude",
        "1800": " funny <lora:ral-frctlgmtry:1> --neg nsfw, nude"
    },
    "positive_prompts": "",
    "negative_prompts": "nsfw, nude",
    "animation_mode": "3D",
    "max_frames": 2400,
    "border": "replicate",
    "angle": "0: (0)",
    "zoom": "0: (1.0025+0.002*sin(1.25*3.14*t/30))",
...

[scottdraves]

another option: https://civitai.com/models/630442/fractalex
https://civitai.com/models/404093/yhnrfractalxl?modelVersionId=450556

[scottdraves]

we have some loaded directly in the runpod container and they work great.

it sucks to have to build the container to add one though, because it can take a long time.
and we want users to be able to upload their own. but we don't want to just rely on links because what if they go down?
we want reproducibility, and the only way to do that is copy them ourselves....

here's @mystixxx plan to avoid the rebuild:

my idea is to store the LoRAs and a manifest on the existing RunPod volume we use for deforum, then use the BullMQ worker to add/remove them through RunPod S3 API. deforum can load them directly from the volume, so we wont need to rebuild the container every time. we can start with a CLI or a small admin UI. shouldnt be much work, and we can keep it alpha-only for now. since there is no need for stage and alpha. its same volume they reuse

[scottdraves]

here is how it is done now: e-dream-ai/gpu-container-deforum@21fb12f

[scottdraves]

forget about interpolation for now, let's just get loading working.
there are two possible next steps here, one is to just fix the container rebuild, and the next is to make them user settable or uploadable.
you could just use a civitai url but there are security and reproducibility risks.
best to ingest and host ourselves, like a video asset in R2.
can that simplify this?

[scottdraves]

Wrote up a plan for this: docs/plans/2026-09-04-deforum-lora-r2-registry.md in metarepo.

Short version — yes, hosting them ourselves in R2 simplifies it, and it collapses both next steps into one change:

• R2 is the source of truth, no RunPod-side state. The container already builds a boto3 R2 client to upload every render, so reading a LoRA back out is a get_object. It downloads at job start into a container-local, digest-keyed cache (loras/<sha256>.safetensors). No volume, no manifest sync, no admin tooling against RunPod's S3 API. This kills the rebuild on its own — Phase 1 is container + worker only.
• Uploaded LoRAs become Dreams, with lora as a third mediaType alongside video and image — so they get an owner, a thumbnail, feed presence, votes, and the existing multipart upload flow for free. Good thumbnail option: queue a short deforum render using the LoRA, so the catalog shows what each one actually does.
  • ⚠️ Trap worth knowing before we add the enum value: the native-client and playlist queries are written as mediaType: Not(DreamMediaType.IMAGE) (client.controller.ts:229/278/334, client.util.ts:97), so a third type silently joins client playback — the screensaver would try to play a 218 MB safetensors file. Those need to become explicit In([VIDEO]) in the same PR.
• Aliases are a single nullable loraAlias column with a partial unique index. Claimed automatically on upload when the name is free, never overwritten by another upload, and writable by admins on any dream (clearing a bad name, or blocking one by pointing it at a safe LoRA). Before a LoRA has a name it's addressable by UUID with no syntax change — <lora:3f2a9c14-…:0.8> already parses against the existing _LORA_TAG regex.
• Prompt syntax doesn't change at all. LoraRegistry re-scans LORA_PATH on every render, so a file dropped in at job start is picked up with no restart and no deforum-studio change.

Per @mystixxx's volume plan: same goal, but the volume ends up being state we'd have to keep coherent, and it pins the endpoint to one datacenter. The doc has a section on why container-local won out.

Interpolation stays out of scope per the comment above — it's orthogonal, and unaffected by any of this.

Open questions at the end of the doc: collision behaviour on upload (leave unaliased vs. auto-suffix), whether LoRA dreams belong in the default feed, and whether claiming a name should require publishing the LoRA.

[scottdraves]

Split the LoRA-interpolation comment above out into its own issue: #486. This issue stays focused on LoRA storage/upload.

[mystixxx]

I’d keep the R2 storage and worker cache approach, with a separate lora table for metadata.

LoRA records

Each record would contain:

• UUID, owner, name and canonical alias
• R2 object key, SHA-256 and file size
• Base model and trigger words
• Optional thumbnail
• Visibility, NSFW and license metadata
• Ingestion status

Using a Dream subtype would reuse ownership, metadata and browsing/editing infrastructure. A separate table needs some additional API/UI wiring, but avoids adding LoRA exceptions to Dream video ingestion, playback and playlist paths. We can still reuse the R2 utilities and shared UI components.

Upload and validation

The flow would be:

1. Upload or ingest from a URL into a private staging object.
2. Validate and hash through BullMQ.
3. Store the validated file at loras/<sha256>.safetensors in R2.
4. Mark the LoRA record as ready.

Validation and access rules:

• Validate the complete file using the Safetensors parser, including structural checks and file/header/tensor-count limits.
• Verify supported LoRA tensor structure and SDXL compatibility before making it available to Deforum.
• Compute SHA-256 server-side. Clients should never write directly to the final content-addressed key.
• Keep uploads private by default. Check access for both alias and UUID lookups.
• Normalize aliases consistently, enforce case-insensitive uniqueness and reserve existing built-in aliases.
• Resolve prompt tags server-side into authorized records. Dont accept arbitrary filesystem paths or user-selected R2 keys.
• Handle external URLs only through controlled ingestion, with download limits, timeouts and checks preventing access to internal addresses, including through redirects.

Loading and reproducibility

Save the resolved LoRA UUID, digest and strength with each render. Retries and rerenders should use that saved mapping, so changing an alias doesnt silently change the model.

The GPU worker downloads approved files into its digest-keyed cache, verifies the checksum and exposes only the requested aliases for that job. Cache eviction must preserve active files, and failed jobs must clean up their loaded LoRA state before another job runs.

Deleting a LoRA record also needs to account for shared files and existing render references, deleting one upload should not remove bytes still used elsewhere.