Skip to content

feat(video): Add ComfyUI workflow presets for Wan 2.2, LTX-2.5, and HunyuanVideo #9

Description

@spelech

🎯 Objective & Overview

Integrate local Diffusion Transformer (DiT) video generation workflows into LocalLLMServerManager by orchestrating ComfyUI execution pipelines for Wan 2.2, LTX-2.5, and HunyuanVideo 1.5.


📂 Files & Components to Touch

  1. Workflow Presets (Workflows/Video/):
    • Workflows/Video/wan2.2_t2v.json (Wan 2.2 Text-to-Video API workflow)
    • Workflows/Video/wan2.2_i2v.json (Wan 2.2 Image-to-Video API workflow)
    • Workflows/Video/ltx2.5_t2v.json (LTX-2.5 Text-to-Video with synced audio)
    • Workflows/Video/hunyuanvideo1.5_t2v.json (HunyuanVideo 1.5 cinematic T2V)
  2. Endpoints (Endpoints/WorkflowEndpoints.cs):
    • Add GET /api/video/workflows listing available video workflow templates.
    • Add POST /api/video/generate queuing video generation on ComfyUI /prompt.
    • Add GET /api/video/files listing generated .mp4 / .webm outputs in wwwroot/output_video/ or configured video directory.
  3. Settings (LocalLLMServerManager.Shared/Models/AppSettings.cs):
    • Add string VideoModelsPath = "" and string VideoOutputPath = "" to AppSettings.
  4. VRAM Orchestration (Services/VramOrchestrator.cs):
    • Ensure EnsureVramForComfyUiAsync() unloads all active Ollama LLM models before video rendering to avoid CUDA OOM.

🔌 API Contract

POST /api/video/generate

{
  "workflowId": "wan2.2_t2v",
  "prompt": "Cinematic shot of a neon cyberpunk city at night, rain reflections, 4k",
  "negativePrompt": "blurry, low quality, distorted",
  "width": 832,
  "height": 480,
  "frames": 49,
  "fps": 16,
  "seed": -1
}

Response: 200 OK

{
  "promptId": "a1b2c3d4-e5f6-...",
  "status": "queued",
  "wsUrl": "ws://127.0.0.1:8188/ws"
}

🛠️ Step-by-Step Implementation Guide

  1. Export clean API-format ComfyUI workflow JSONs with prompt placeholder tokens ({{PROMPT}}, {{WIDTH}}, {{HEIGHT}}, {{FRAMES}}).
  2. Implement backend substitution and queuing logic in WorkflowEndpoints.cs.
  3. Track execution via ComfyUI WebSocket/polling and save output videos to output_video/.
  4. Add unit and integration tests in LocalLLMServerManager.Tests/WorkflowEndpointsTests.cs.

🧪 Verification

  • dotnet build
  • dotnet test --filter Category!=LiveExternal
  • npm run lint & npx tsc --noEmit

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions