Skip to content

fix(ltx): dev transformer + distill LoRA so camera LoRAs bind - #1

Merged
mystixxx merged 1 commit into
mainfrom
fix/ltx-camera-lora-dev-model
Jun 30, 2026
Merged

mystixxx merged 1 commit into
mainfrom
fix/ltx-camera-lora-dev-model

Conversation

@mcarlson

Copy link
Copy Markdown
Collaborator

Why

The LTX "Camera" presets produced no camera motion in any direction. Root cause: the per-direction camera-control LoRAs are trained for LTX-2 19B, but the deployed transformer was ltx-2.3-22b-distilled. ComfyUI can't bind the mismatched keys — it logs lora key not loaded and silently emits base-model output while the job reports success. Full diagnosis: docs/plans/2026-06-19-ltx-2.3-camera-lora-not-applied.md (metarepo).

The fix (3 coordinated PRs)

Switch the LTX i2v path to the full "dev" 22B transformer (which the 19B camera LoRAs do bind to), and add the distill LoRA as a model-only loader ahead of the camera LoRA so few-step LCM sampling is preserved.

Repo PR Change
gpu-container-ltx this/linked Bake the dev transformer + ltx-2-19b-distilled-lora-384; test_input.json updated to the dev chain
worker this/linked createLtxI2VWorkflow: node 1 = dev UNET, node 7 = LoraLoaderModelOnly distill @0.6, node 6 Power Lora Loader (camera LoRA) chains off node 7; both CFGGuider passes consume node 6
frontend this/linked Enable all 7 LTX Camera preset directions (static + dolly in/out/left/right + jib up/down)

Model chain: UNETLoader(dev)LoraLoaderModelOnly(distill @0.6)Power Lora Loader(camera @0.4) → both CFGGuiders.

Deploy order (important)

The worker references the dev model + distill LoRA by filename, so the image must ship them first:

  1. Merge gpu-container-ltx → GHCR build (Actions) → update the RunPod LTX endpoint to the new image.
  2. Deploy worker to Heroku (stage).
  3. Merge frontend (just enables the UI presets — harmless on its own, but moves only work once 1+2 are live).

Verification runbook (this is "done")

Stage A — container-direct (isolates the model/LoRA chain from the app):

  1. After the GHCR build, point the RunPod LTX endpoint at the new image and cold-start a worker.
  2. Submit test_input.json (runs static @0.4). Confirm it completes and the output is coherent — this validates the dev model + distill LoRA produce good few-step output.
  3. Swap node 6 lora_01.lora to ltx-2-19b-lora-camera-control-dolly-in.safetensors; resubmit; download the video.
  4. Check the ComfyUI/RunPod log for node 6: confirm there is no lora key not loaded / N keys not loaded warning — this is the definitive bind check. Also confirm no VRAM OOM (dev fp8 ≈ distilled footprint, but verify on the endpoint's GPU).
  5. Visually: static holds still; dolly-in shows a forward push. If motion is weak, raise camera strength (0.4 → 0.6/0.8). If quality degrades / looks under-denoised, adjust distill strength (0.6).

Stage B — full Studio E2E:
6. With the worker deployed and frontend merged to stage, open Studio → "LTX Camera" preset → run a batch across one image.
7. Confirm each column shows the correct motion: static still; dolly in/out push/pull; dolly left/right slide; jib up/down rise/descend.

Tuning & risks

  • Starting strengths: distill 0.6, camera 0.4. Both are starting points — tune per Stage A/B results.
  • dolly-left / dolly-right are the least-validated directions (community reports were mixed on left/right even on the dev model). Verify them specifically in Stage B; if they show no motion, bump their strength or drop those two preset actions.
  • Stale reference workflow: gpu-container-ltx/workflows/jef-ltx-2.3-i2v.json still points at the distilled model. The worker builds its own workflow so production is unaffected, but local docker-compose testing would use the old model. Update it if you test locally.
  • Image size / cold start: baking the dev model + distill LoRA + 7 camera LoRAs grows the image → slower cold starts.

🤖 Generated with Claude Code

Switch the baked diffusion model from ltx-2.3-22b-distilled to the full
(dev) 22b transformer and add ltx-2-19b-distilled-lora-384. The LTX-2 19b
camera-control LoRAs produce no key matches on the distilled model and
silently fall back to base-model output (no camera motion); they bind to
the dev model. The distill LoRA (@0.6) restores few-step LCM sampling on
the dev transformer. test_input.json updated to the dev chain (node 7
distill LoRA → node 6 camera LoRA).

Pairs with the worker createLtxI2VWorkflow change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mystixxx
mystixxx merged commit 9b9a8ad into main Jun 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants