🎯 Objective & Overview
Integrate local Diffusion Transformer (DiT) video generation workflows into LocalLLMServerManager by orchestrating ComfyUI execution pipelines for Wan 2.2, LTX-2.5, and HunyuanVideo 1.5.
📂 Files & Components to Touch
- Workflow Presets (
Workflows/Video/):
Workflows/Video/wan2.2_t2v.json (Wan 2.2 Text-to-Video API workflow)
Workflows/Video/wan2.2_i2v.json (Wan 2.2 Image-to-Video API workflow)
Workflows/Video/ltx2.5_t2v.json (LTX-2.5 Text-to-Video with synced audio)
Workflows/Video/hunyuanvideo1.5_t2v.json (HunyuanVideo 1.5 cinematic T2V)
- Endpoints (
Endpoints/WorkflowEndpoints.cs):
- Add
GET /api/video/workflows listing available video workflow templates.
- Add
POST /api/video/generate queuing video generation on ComfyUI /prompt.
- Add
GET /api/video/files listing generated .mp4 / .webm outputs in wwwroot/output_video/ or configured video directory.
- Settings (
LocalLLMServerManager.Shared/Models/AppSettings.cs):
- Add
string VideoModelsPath = "" and string VideoOutputPath = "" to AppSettings.
- VRAM Orchestration (
Services/VramOrchestrator.cs):
- Ensure
EnsureVramForComfyUiAsync() unloads all active Ollama LLM models before video rendering to avoid CUDA OOM.
🔌 API Contract
POST /api/video/generate
{
"workflowId": "wan2.2_t2v",
"prompt": "Cinematic shot of a neon cyberpunk city at night, rain reflections, 4k",
"negativePrompt": "blurry, low quality, distorted",
"width": 832,
"height": 480,
"frames": 49,
"fps": 16,
"seed": -1
}
Response: 200 OK
{
"promptId": "a1b2c3d4-e5f6-...",
"status": "queued",
"wsUrl": "ws://127.0.0.1:8188/ws"
}
🛠️ Step-by-Step Implementation Guide
- Export clean API-format ComfyUI workflow JSONs with prompt placeholder tokens (
{{PROMPT}}, {{WIDTH}}, {{HEIGHT}}, {{FRAMES}}).
- Implement backend substitution and queuing logic in
WorkflowEndpoints.cs.
- Track execution via ComfyUI WebSocket/polling and save output videos to
output_video/.
- Add unit and integration tests in
LocalLLMServerManager.Tests/WorkflowEndpointsTests.cs.
🧪 Verification
dotnet build
dotnet test --filter Category!=LiveExternal
npm run lint & npx tsc --noEmit
🎯 Objective & Overview
Integrate local Diffusion Transformer (DiT) video generation workflows into
LocalLLMServerManagerby orchestrating ComfyUI execution pipelines for Wan 2.2, LTX-2.5, and HunyuanVideo 1.5.📂 Files & Components to Touch
Workflows/Video/):Workflows/Video/wan2.2_t2v.json(Wan 2.2 Text-to-Video API workflow)Workflows/Video/wan2.2_i2v.json(Wan 2.2 Image-to-Video API workflow)Workflows/Video/ltx2.5_t2v.json(LTX-2.5 Text-to-Video with synced audio)Workflows/Video/hunyuanvideo1.5_t2v.json(HunyuanVideo 1.5 cinematic T2V)Endpoints/WorkflowEndpoints.cs):GET /api/video/workflowslisting available video workflow templates.POST /api/video/generatequeuing video generation on ComfyUI/prompt.GET /api/video/fileslisting generated.mp4/.webmoutputs inwwwroot/output_video/or configured video directory.LocalLLMServerManager.Shared/Models/AppSettings.cs):string VideoModelsPath = ""andstring VideoOutputPath = ""toAppSettings.Services/VramOrchestrator.cs):EnsureVramForComfyUiAsync()unloads all active Ollama LLM models before video rendering to avoid CUDA OOM.🔌 API Contract
POST /api/video/generate{ "workflowId": "wan2.2_t2v", "prompt": "Cinematic shot of a neon cyberpunk city at night, rain reflections, 4k", "negativePrompt": "blurry, low quality, distorted", "width": 832, "height": 480, "frames": 49, "fps": 16, "seed": -1 }Response:
200 OK{ "promptId": "a1b2c3d4-e5f6-...", "status": "queued", "wsUrl": "ws://127.0.0.1:8188/ws" }🛠️ Step-by-Step Implementation Guide
{{PROMPT}},{{WIDTH}},{{HEIGHT}},{{FRAMES}}).WorkflowEndpoints.cs.output_video/.LocalLLMServerManager.Tests/WorkflowEndpointsTests.cs.🧪 Verification
dotnet builddotnet test --filter Category!=LiveExternalnpm run lint&npx tsc --noEmit