Deforum: interpolate between LoRAs within a single render (per-frame LoRA strengths)

[scottdraves]

Split out of #479 (original comment). Orthogonal to the LoRA storage/upload work in that issue and can land independently — this only changes how strengths are parsed and applied, not where the files come from.

Code refs are e-dream-ai/deforum-studio @ 15c1d1d.

What blocks it

prepare_lora_prompts parses LoRA tags once, up front, and rejects any variation across keyframes:

ValueError: All Deforum keyframes must use the same LoRAs and strengths
  src/deforum/utils/lora.py:99
  <- pipeline_deforum_animation.py:255, pre_setup()

A render that crossfades <lora:a:0.9> <lora:b:0.1> -> <lora:a:0.1> <lora:b:0.9> across keyframes fails in ~1s, before any frame is sampled. There is a second guard at comfy_sd_generator.py:94: configure_loras() raises "LoRAs cannot change after generation has started".

So LoRA strengths are constant for an entire render. Static blends are fine — <lora:trichom-style:0.5> <lora:ral-frctlgmtry:0.5> renders correctly, and a ladder of fixed ratios from 1/24 to 23/24 produces a clean monotonic progression. But that interpolation is quantized across N separate renders; it can't happen within one.

The prompt-weighting workaround is not equivalent

You can mount both LoRAs statically and animate their trigger words instead, using the backtick math that split_weighted_subprompts (pipeline_deforum_animation.py:645) evaluates per frame with t bound to the frame index:

<lora:ral-frctlgmtry:1> <lora:trichom-style:1>
(ral-frctlgmtry, pretty:`1-t/199`) (ral-trichom, mantis priest dialog:`t/199`)

This renders, and looks good. But it is not an interpolation between the two LoRAs, for two reasons:

1. A LoRA is a weight delta (W' = W + a*dW) applied to the UNet and CLIP — lora.py sets strength_model and strength_clip to the same value. Mounting it changes the model globally regardless of the prompt. At t=0 above, trichom is still fully mounted and deforming the very text encoder that encodes "pretty".
2. The encoder is called with mean_normalization=True (comfy_sd_generator.py:293), and A1111 emphasis rescales embeddings relative to the mean. A zero-weight phrase still occupies token slots and still participates in that normalization — weight 0 is not deletion.

Net: the endpoints are unreachable. It traces an off-axis path through a fixed both-LoRAs-mounted model, never touching either pure style.

Proposed fix: per-frame LoRA strengths

The building blocks are already there.

1. Allow a backtick expression in the strength field. _LORA_TAG (lora.py:10) currently accepts only a float. Widen it to accept `expr` and evaluate it with the same parse_weight used for prompt weights, so the syntax stays consistent with the rest of deforum:

<lora:ral-frctlgmtry:`1-t/199`> <lora:trichom-style:`t/199`>

2. Relax the equality check. prepare_lora_prompts should require the same LoRA set across keyframes but permit differing strengths, returning a per-frame schedule instead of a single Tuple[LoraSpec, ...].

3. Keep pristine base weights. Today _apply_configured_loras (comfy_sd_generator.py:333) overwrites self.model/self.clip with the patched pair and latches _loras_applied = True. Re-patching per frame requires holding unpatched base_model/base_clip from checkpoint load and re-deriving from those each time strengths change — otherwise patches compound.

4. Re-patch per frame. load_lora already returns a fresh patched pair from comfy.sd.load_lora_for_models, so this is a call, not a rewrite.

Cost

Lower than it looks:

• No disk I/O. _load_lora_state (comfy_sd_generator.py:357) already memoizes state dicts in self.loaded_loras keyed by path. Re-patching reuses the cached tensors.
• load_lora_for_models works through ComfyUI's ModelPatcher, which records patches and applies them at sample time rather than eagerly copying all weights.
• Skip the work entirely when the frame's strengths are unchanged from the previous frame, so static-LoRA renders (all current dreams) take exactly the same path they do now.

The per-frame patch cost still needs measuring against 25 sampler steps at 1344x768 before committing — that's the main open question, and worth a spike before the full change.