Conversation
Switch disagg diffusion demo from Wan2.2-TI2V-5B to HunyuanVideo v1 (13B DiT + Llama3-8B + CLIP dual encoders). Key changes: - Auto-detect dual text encoders from model_index.json - Handle multi-element tensor lists (incompatible shapes) in IntermediateOutputStage and inject_tensors_to_req - Patch HunyuanConfig missing task_type default - Patch SGLang triton norm_infer for non-contiguous tensors - Add .contiguous() in DeviceMoveStage for ZMQ-transferred tensors - Save output as mp4 via imageio instead of raw npy - Make prompt configurable via PROMPT env var Tested: 2 concurrent requests, 4 GPUs (enc=GPU0, den=GPU1,2 TP=2, vae=GPU3), HunyuanVideo 544x960 resolution. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rewrite design doc with high-level architecture diagrams and pipeline parallelism illustration - Add benchmark results (native vs disagg, HunyuanVideo 544x960) - Update milestone tracker: mark SGLang integration as completed - Add scaling roadmap (NIXL transfer, independent stage scaling, encoder caching, sequence parallelism, heterogeneous hardware) - Update README with quick-start commands and env var reference Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace ZMQ pickle tensor transfer with NIXL RDMA for inter-stage communication. Only small metadata (~1.5 KB) now travels over ZMQ; actual tensor data (embeddings, latents) transfers GPU-to-GPU without CPU round-trips. - New nixl_transfer.py: NixlTensorSender/Receiver with persistent Connector and async-to-sync bridge for PipelineStage.forward() - NixlSendStage: registers GPU tensors as NIXL-readable, returns metadata in OutputBatch; handles dual-encoder indexed fields - NixlReceiveStage: RDMA-pulls tensors from sender GPU, reconstructs indexed fields back into lists; falls back to CPU device-move when NIXL metadata is absent - Orchestrator routes NIXL metadata between stages instead of tensors - New serve.py: FastAPI HTTP server for curl-based requests - Update design doc: remove benchmarks, reflect NIXL architecture Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rewrite README with full design, architecture diagram, data flow, implementation details, quick start, API reference, and roadmap - Remove separate design doc (all content now in README) - Improve serve.py: clear process architecture in docstring, detailed health endpoint showing GPU layout and worker count - Verified: 3 sequential HTTP requests via curl all succeed with NIXL RDMA transfer on 4 GPUs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace ZMQ orchestrator (serve.py) with Dynamo RPC workers. Each worker spawns SGLang Scheduler subprocess(es) via launch_partial_server() and bridges Dynamo RPC <-> ZMQ, enabling TP support and NIXL RDMA GPU-direct tensor transfer between stages. Key changes: - Workers use launch_partial_server() + StageClient instead of serve.py - NixlSendStage/NixlReceiveStage for GPU-direct inter-stage transfer - Dual encoder auto-detection (Llama 8B + CLIP for HunyuanVideo) - Triton norm contiguous patch for HunyuanVideo transformer - HunyuanVideo defaults (544x960, 50 steps, guidance_scale=1.0) - Removed serve.py (ZMQ orchestrator, replaced by Dynamo RPC workers) - Added Dynamo RPC endpoint wrappers in encoder/denoiser/vae_worker.py - Simplified protocol.py (removed NixlTensorSender/Receiver) - Updated run_disagg.py orchestrator for HunyuanVideo defaults Tested: HunyuanVideo 13B, 4 GPUs (Enc=1, Den TP=2, VAE=1), NIXL RDMA, 30 steps 544x960 -> high quality output verified. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- run_all.sh: one-command launch of etcd + 3 workers + orchestrator - README: remove phase terminology, add architecture/flow diagrams, roadmap Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move StageClient, patch_hunyuan_config, detect_encoder_modules, save_video to sglang_utils.py (was duplicated in run_e2e_sglang.py and vae_worker.py) - Workers now import from sglang_utils instead of run_e2e_sglang.py, avoiding module-level side effects on import - Remove unused build_config() function - Remove duplicate _save_video_frames from vae_worker.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… worker boilerplate Performance: - Replace asyncio.run() with loop.run_until_complete() in NIXL transfer to reuse existing event loop and keep _keep_alive tasks alive - Remove unnecessary __count metadata tensors and CPU/GPU tensor filtering in NixlSendStage - Use batch.logging_info directly instead of creating RequestTimings per call Code quality: - Remove dead build_server_args(), _ensure_distributed_init(), and extract_tensors_from_req() from sglang_utils - Remove backward-compat aliases in run_e2e_sglang.py - Add launch_stage_server() helper consolidating worker boilerplate - Refactor encoder/denoiser/vae workers to use launch_stage_server() - Fix sglang module paths (pipelines_core -> pipelines, etc.) for compatibility with sglang 0.5.5 - Clean up unused imports across all files Verified: E2E test passes with 3 high-quality requests (61 frames, 50 steps, TP=2 denoiser, NIXL RDMA transfers). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each pipeline stage (Encoder, Denoiser, VAE) can now run N workers. The orchestrator round-robins requests across workers per stage independently, enabling full GPU utilization (e.g. 8 GPU with 2 workers per stage). GPU spec uses `;` to separate workers: GPU_DEN="1,2;5,6" launches 2 TP=2 denoiser workers. Backward compatible — no `;` means 1 worker. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add concurrent benchmark example to the multi-worker section. Verified with 50-request run on 8×H20 (2 workers/stage): median=10.79s/req, std=0.55s, 0 failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
NIXL/UCX crashes when multiple workers in the same stage do concurrent RDMA transfers (NIXL_ERR_REMOTE_DISCONNECT). Add a per-pool asyncio.Lock so only one request per stage is in-flight at a time. Pipeline parallelism across stages is preserved — Encoder, Denoiser, and VAE each run simultaneously on different requests. Use CONCURRENCY>=3 to keep the 3-stage pipeline fully saturated. Warmup now sends N requests (one per worker) to initialise all NIXL connectors before concurrent traffic begins. Verified: 50 requests, 8×H20, concurrency=4, 0 failures. Wall time: 272s (vs 545s sequential) — 2x throughput. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove development-era directory prefixes (phase0_validate → validate,
phase1_workers → workers, phase2_orchestrator → orchestrator) and delete
the stale launch/ directory superseded by root run_all.sh.
Also update sglang imports for 0.5.8 API changes:
- pipelines.{schedule_batch,stages} → pipelines_core.*
- configs.pipelines → configs.pipeline_configs
- configs.sample.base → configs.sample.sampling_params
- scheduler_endpoint() method → property
Verified: 8-GPU multi-worker (2 enc, 2 den TP=2, 2 vae) with 3
concurrent requests, all completed successfully.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sglang 0.5.8 removed Req.logging_info and OutputBatch's logging_info parameter, and gpu_worker.execute_forward() now expects OutputBatch.timings to be set (accesses timings.total_duration_ms). Changes: - NixlSendStage: drop logging_info usage, provide RequestTimings in every OutputBatch so gpu_worker can record total_duration_ms - orchestrator: use round_robin() dispatch, dynamic per-stage semaphores based on discovered worker count Verified: 4/4 requests (1 single + 3 concurrent) produce valid mp4 videos via NIXL RDMA end-to-end. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Log tensor shape/dtype/mean/std/min/max at NIXL send and receive points for diagnosing data transfer issues. Uses logger.debug() so no overhead at default INFO level. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comprehensive design document covering the architecture, component design, and SGLang/HunyuanVideo implementation details. Sections 1-3 are model-agnostic and backend-agnostic; Section 4 isolates SGLang specifics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…mpat Replace round-robin semaphore-based orchestrator with WorkerManager that tracks per-worker busy/idle state and dispatches via client.direct(). Add dispatch_with_retry for automatic failover to a different worker. - Add ZMQ tensor fallback path (base64-encoded torch.save) for when NIXL is unavailable; controlled via DISABLE_NIXL env var - Fix sglang 0.5.8 API: runtime.endpoint() replaces ns.component(), OutputBatch(metrics=) replaces timings=, fix add_stage arg order - Add NIXL pull retry with exponential backoff for REMOTE_DISCONNECT - Fix NIXL sender buffer leak: replace broken async _keep_alive with synchronous _sweep() that polls readable.status - Add StageClient.forward() timeout (STAGE_FORWARD_TIMEOUT_S) - Add stress_test.sh for concurrent load testing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root causes and fixes: 1. PersistentConnector: replace _PersistentConnector singleton (which created a new Connection per operation) with a proper subclass that reuses one Connection/agent. Each Sender/Receiver owns its own instance, eagerly initialized at stage construction. 2. UCX transport: default UCX_TLS=all picks IB RDMA which fails across NUMA nodes. Must set UCX_TLS=cuda_ipc,tcp,self,cuda_copy,cma to force NVLink path for intra-node GPU-direct transfers. 3. Sender returns readable_op: NixlTensorSender.send() now returns (meta, readable_op). NixlSendStage holds readable_ops in _active_readables until receiver completes the RDMA pull, preventing premature descriptor deregistration. 4. TP broadcast: with TP>1, only rank 0 gets NIXL data. Added _tp_broadcast_fields to distribute tensors to all TP ranks via torch.distributed.broadcast after the NIXL pull. 5. ZMQ socket recovery: StageClient.forward() now resets the ZMQ REQ socket after timeout, preventing cascading EFSM errors. 6. Worker discovery: orchestrator now waits WORKER_SETTLE_S (default 30s) after initial discovery to pick up slow-registering workers. Verified: 20 concurrent requests, 3 TP=2 denoisers, 8 GPUs, 0 failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…N.md - Add request_validation.py with pydantic-based HTTP body validation - Orchestrator: observe all stages (encoder/denoiser/vae) during settle period instead of only denoiser - Orchestrator: structured error handling (ValidationError → 400) - protocol.py: add Field annotations - DESIGN.md: streamline and align with current implementation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Restore ASCII art architecture/flow diagrams (more portable than mermaid). Update to reflect current implementation: - PersistentConnector pattern and agent lifecycle - One-sided RDMA pull with buffer lifecycle diagram - TP broadcast for multi-GPU denoisers - UCX_TLS configuration for intra-node transfers - StageClient ZMQ socket recovery - Worker settle period for discovery - Measured performance numbers (8-GPU HunyuanVideo) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove GPU tensor debug stats logging (.float() copy was 600ms+ per request even at DEBUG level due to GPU synchronization) - Skip torch.cat when only one tensor (avoid unnecessary GPU copy) - Reuse event loop in _run_coro instead of creating new one each call - Add NIXL send timing breakdown log (flatten/descriptor/create/meta) NixlSendStage: 640ms → 3ms (213x faster) N=10 stress: 53s → 41s (23% faster) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview:
Details:
Where should the reviewer start?
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)