[Training][Rollout] Add direct NCCL delta weight updates

docs/en/advanced/delta-weight-sync.md

@@ -5,15 +5,24 @@ that changed between two syncs, instead of a full checkpoint each time. It targe
 training/inference disaggregation across clusters or datacenters, where writing the whole actor
 every sync is the dominant cost.
 
-It is **disk-transport only**. The trainer publishes each sync as a canonical HF checkpoint
+> **Status:** `--update-weight-mode delta` selects the **direct NCCL** delta implementation
+> (`--update-weight-transport nccl`): the trainer diffs against a checkpoint-coordinate
+> snapshot and ships only changed values over the existing NCCL channel into vLLM's
+> checkpoint weight patch API. See
+> [`examples/delta_weight_sync/README.md`](https://github.com/vllm-project/vime/blob/main/examples/delta_weight_sync/README.md)
+> for requirements and configuration. The **disk-based** design documented
+> below is reserved and not implemented: `--update-weight-mode delta
+> --update-weight-transport disk` raises `NotImplementedError`.
+
+The disk-based design is **disk-transport** based. The trainer publishes each sync as a canonical HF checkpoint
 directory; the engine's `/pull_weights` endpoint (shipped in vime's vllm patch) fans the
 apply out to **every host the engine spans** and verifies it, then the engine reloads the
 patched local checkpoint through the **ordinary** `update_weights_from_disk` endpoint. vime
 only ever talks to one endpoint per engine, so multi-node serving and external rollout engines
 need nothing extra on the vime side.
 
-Vime currently guards this mechanically synchronized path with a `NotImplementedError` when
-`--update-weight-mode=delta` is selected; the implementation below remains upstream reference code.
+Vime guards this path with a `NotImplementedError` when `--update-weight-transport=disk` is
+combined with delta mode; the implementation below remains upstream reference code.
 
 ## Configuration
 

docs/en/get_started/usage.md

@@ -145,7 +145,7 @@ Note:
   - By default, vLLM reads the maximum context length from the `config.json` in the Hugging Face checkpoint. You can use the `--vllm-max-model-len` parameter to override this value to support longer inference.
   - During co-located training and inference, although Megatron and vLLM will offload sequentially, they still need to leave some memory for each other. You need to adjust vLLM's total VRAM usage by reducing `--vllm-gpu-memory-utilization`.
   - vime supports passing through vllm-router parameters by adding a `router` prefix to the original parameter name. For example, vllm-router's `--balance-abs-threshold` parameter should be set as `--router-balance-abs-threshold`. Since vllm-router uses cache-aware routing by default, it may cause uneven request distribution. You can set `--router-balance-abs-threshold 0` to force balanced distribution, but this may affect prefix cache hit rate in multi-turn conversation scenarios. For multi-turn sessions that require session affinity, set `--router-policy consistent_hash` and send a stable `x-session-id` for each session.
-  - If vLLM engines are pre-launched by an external system, connect to them with `--rollout-external-engine-addrs host1:port host2:port`. When the trainer and engines cannot form an NCCL weight-update group, use `--update-weight-mode full --update-weight-transport disk --update-weight-disk-dir /shared/fs/updates`; vime writes a complete HF checkpoint and asks vLLM to hot-load it through `update_weights_from_disk`. For large models or cross-cluster deployments, use `--update-weight-mode delta --update-weight-transport disk` instead. See [External Rollout Engines Roadmap](../advanced/external-rollout-engines.md) and [Delta Weight Sync](../advanced/delta-weight-sync.md).
+  - If vLLM engines are pre-launched by an external system, connect to them with `--rollout-external-engine-addrs host1:port host2:port`. When the trainer and engines cannot form an NCCL weight-update group, use `--update-weight-mode full --update-weight-transport disk --update-weight-disk-dir /shared/fs/updates`; vime writes a complete HF checkpoint and asks vLLM to hot-load it through `update_weights_from_disk`. When the trainer and engines can form an NCCL group, `--update-weight-mode delta --update-weight-transport nccl` ships only the weights that changed between syncs (see [Delta Weight Sync](../advanced/delta-weight-sync.md)); disk-based delta sync is a reserved design and not yet implemented. See also [External Rollout Engines Roadmap](../advanced/external-rollout-engines.md).
 
 For details on some of vLLM's customizations and the principles behind how vime incorporates vLLM, please see the "How to Use vLLM" section.
 

docs/zh/advanced/delta-weight-sync.md

@@ -2,9 +2,16 @@
 
 Delta 权重同步只发送两次同步之间发生变化的字节，而不是每次都写一份完整 checkpoint，以此让非 colocate 的 rollout engine 保持最新。它面向大模型、跨集群或跨数据中心的训推解耦场景——这种场景下每次都写整份 actor 权重是主要开销。
 
-它**只支持 disk transport**。训练端把每次同步发布为一份 canonical HF checkpoint 目录；engine 的 `/pull_weights` 端点（随 vime 的 vllm patch 提供）把 apply 扇出到 **engine 覆盖的每一个 host** 并校验，随后 engine 通过**原生**的 `update_weights_from_disk` 端点 reload 打过补丁的本地 checkpoint。vime 对每个 engine 只与一个端点通信，所以多节点 serving 和外部 rollout engine 在 vime 侧都不需要任何额外支持。
+> **状态：**`--update-weight-mode delta` 现在选择的是**直连 NCCL** 的 delta 实现
+> (`--update-weight-transport nccl`)：训练端基于 checkpoint 坐标快照做 diff，只把变化的
+> 权重通过既有 NCCL 通道送进 vLLM 的 checkpoint weight patch API。环境要求和配置方式见
+> [`examples/delta_weight_sync/README.md`](https://github.com/vllm-project/vime/blob/main/examples/delta_weight_sync/README.md)。
+> 下文描述的**基于磁盘**的设计为保留设计、尚未实现：`--update-weight-mode delta
+> --update-weight-transport disk` 会抛出 `NotImplementedError`。
 
-Vime 当前在选择 `--update-weight-mode=delta` 时会通过 `NotImplementedError` guard 拒绝该路径；下文保留为机械同步的上游参考实现。
+磁盘方案基于 **disk transport**。训练端把每次同步发布为一份 canonical HF checkpoint 目录；engine 的 `/pull_weights` 端点（随 vime 的 vllm patch 提供）把 apply 扇出到 **engine 覆盖的每一个 host** 并校验，随后 engine 通过**原生**的 `update_weights_from_disk` 端点 reload 打过补丁的本地 checkpoint。vime 对每个 engine 只与一个端点通信，所以多节点 serving 和外部 rollout engine 在 vime 侧都不需要任何额外支持。
+
+Vime 在 delta 模式与 `--update-weight-transport=disk` 组合时会通过 `NotImplementedError` guard 拒绝该路径；下文保留为上游参考实现。
 
 ## 配置
 

docs/zh/get_started/usage.md

@@ -147,7 +147,7 @@ vLLM 的加载非常简单，只需要：
 - vLLM 默认会从 huggingface ckpt 中 `config.json` 读取模型的最大 context length，可以使用 `--vllm-max-model-len` 参数来对这个值进行覆盖，从而支持进行更长的推理；
 - 在训推一体的训练过程中，虽然 megatron 和 vLLM 会先后 offload，但是还是需要为对方留有一些空间，需要通过减小 `--vllm-gpu-memory-utilization` 来调整 vLLM 的显存占用总量。
 - vime 支持透传 vllm-router 的参数，方式是在原参数名前加上 `router` 前缀。例如，vllm-router 的 `--balance-abs-threshold` 参数需要设置为 `--router-balance-abs-threshold`。由于 vllm-router 默认使用 cache-aware routing，可能会导致请求分配不均衡。可以通过设置 `--router-balance-abs-threshold 0` 来强制均衡分配，但这可能会影响多轮对话场景下 prefix cache 的命中率。对于需要会话亲和的多轮会话，可以设置 `--router-policy consistent_hash`，并为每个会话发送稳定的 `x-session-id`。
-- 如果 vLLM engine 已经由外部系统预启动，可以通过 `--rollout-external-engine-addrs host1:port host2:port` 连接。此时如果训练器和 engine 无法建立 NCCL 权重同步 group，可以使用 `--update-weight-mode full --update-weight-transport disk --update-weight-disk-dir /shared/fs/updates`，vime 会写完整 HF checkpoint 并调用 vLLM 的 `update_weights_from_disk` 热加载；大模型或跨集群场景可进一步使用 `--update-weight-mode delta --update-weight-transport disk`。详见 [External Rollout Engines 配置路线图](../advanced/external-rollout-engines.md) 和 [Delta 权重同步](../advanced/delta-weight-sync.md)。
+- 如果 vLLM engine 已经由外部系统预启动，可以通过 `--rollout-external-engine-addrs host1:port host2:port` 连接。此时如果训练器和 engine 无法建立 NCCL 权重同步 group，可以使用 `--update-weight-mode full --update-weight-transport disk --update-weight-disk-dir /shared/fs/updates`，vime 会写完整 HF checkpoint 并调用 vLLM 的 `update_weights_from_disk` 热加载；当训练器与 engine 可以建立 NCCL group 时，可使用 `--update-weight-mode delta --update-weight-transport nccl` 只传输两次同步之间发生变化的权重（详见 [Delta 权重同步](../advanced/delta-weight-sync.md)）；基于磁盘的 delta 同步为保留设计、尚未实现。另见 [External Rollout Engines 配置路线图](../advanced/external-rollout-engines.md)。
 
 对于一些 vLLM 的自定义以及 vime 引入 vLLM 的原理，请见 vLLM 使用方法一节。
 

examples/delta_weight_sync/README.md

@@ -1,44 +1,110 @@
 # Delta Weight Sync
 
-Non-colocated weight sync that ships only the **changed bytes** between two syncs instead of a
-full checkpoint, for training/inference disaggregation across clusters or datacenters. The
-trainer publishes per-tensor deltas to a shared filesystem as a canonical HF checkpoint
-directory; each engine's `/pull_weights` applies them into a host-local checkpoint on every
-host it spans, and the engines reload through the ordinary `update_weights_from_disk` path —
-vime only ever talks to one endpoint per engine.
+VIME currently provides a direct delta-weight-update (DWU) MVP for a
+non-colocated Megatron trainer and VIME-launched vLLM rollout engines. The
+trainer exports canonical Hugging Face/checkpoint-coordinate tensors, keeps a
+committed CPU snapshot, and sends:
 
-Vime currently rejects `--update-weight-mode delta` with a `NotImplementedError`; this example
-is retained as mechanically synchronized upstream reference material.
+- a dense BF16 seed for version 1; then
+- absolute BF16 values plus flattened `int32` checkpoint indices for elements
+  whose bit patterns changed after a committed optimizer step.
 
-See [Delta Weight Sync](../../docs/en/advanced/delta-weight-sync.md) for the full mechanism,
-encodings, integrity checks, and shared-filesystem visibility hooks.
+The vLLM worker applies those patches through the model's native
+`load_weights()` mapping. Consequently, VIME does not need to know vLLM's
+runtime QKV/gate-up packing or tensor-parallel parameter names.
 
-## Try it
+```text
+Megatron TP/DP weights
+        -> canonical HF export
+        -> dense seed or absolute sparse checkpoint patches
+        -> NCCL
+        -> vLLM CheckpointWeightPatch
+        -> native model.load_weights()
+```
+
+## vLLM dependency
+
+Direct DWU requires a vLLM build that contains the checkpoint weight patch
+API (`CheckpointWeightPatch` and `load_checkpoint_weight_patches()` in
+`vllm.model_executor.model_loader.checkpoint_weight_patch`) from
+[vLLM PR #50723](https://github.com/vllm-project/vllm/pull/50723).
 
-`run-glm4.7-30B-A3B-delta.sh` runs the disk delta path on GLM-4.7-Flash, non-colocated across a
-2-node (16-GPU) Ray cluster. See its header for prerequisites.
+Until #50723 merges, no vLLM release contains that API; build vLLM from the
+PR branch. This path was tested at PR commit
+`fd07acd5b596c11f949fa71b5f0ee926b9e6bf17`; vime fails fast at engine startup
+if the patch API is missing.
 
-## Minimal flags
+## Enabling direct DWU
 
-Add to a non-colocated training run (the trainer and engines only need to share the filesystem
-at `--update-weight-disk-dir`):
+Add to a non-colocated Megatron training run (vime starts and owns the vLLM
+engines; do not set `--rollout-external`):
 
 ```bash
 --update-weight-mode delta \
---update-weight-transport disk \
---update-weight-disk-dir   /shared/fs/delta-updates \
---update-weight-local-checkpoint-dir /local/nvme/rollout-ckpt \
---update-weight-delta-encoding xor \
---update-weight-delta-checksum xxh3-128
+--update-weight-transport nccl
 ```
 
-- `--update-weight-disk-dir` — shared directory the trainer writes deltas to and the hosts read.
-- `--update-weight-local-checkpoint-dir` — host-local full HF checkpoint the delta patches in
-  place; materialized from the engine's model path on the first `/pull_weights`.
-- `--update-weight-delta-encoding` — `xor` (smallest/fastest) or `overwrite` (idempotent).
-- `--update-weight-delta-checksum` — `xxh3-128` (default), `blake3`, or `adler32`.
+The first sync ships a mandatory dense seed (start version 0) that aligns the
+rollout weights with the trainer checkpoint; every later sync ships only the
+weights whose BF16 bits changed.
+
+## Current MVP boundary
+
+The direct path currently requires:
+
+- `--train-backend megatron`;
+- non-colocated, VIME-launched rollout engines;
+- BF16, unquantized weights;
+- Megatron PP=1 and VPP=1;
+- vLLM PP=1 and DP=1;
+- no rollout offload, speculative decoding, MTP draft update, fault-tolerant
+  worker replacement, or fully-async rollout; and
+- version 0 startup followed by a mandatory dense seed.
+
+The source currently performs a full canonical-HF export and keeps the
+committed weights in CPU memory. Steady-state network traffic is sparse, but
+source traversal and snapshot memory are not yet sparse or sharded.
+
+Delta over disk is only a reserved interface. The current argument validator
+rejects:
+
+```bash
+--update-weight-mode delta --update-weight-transport disk
+```
+
+with `NotImplementedError`. The existing GLM disk script is retained as
+historical interface material; it is not a runnable path for the current MVP.
+
+## Verifying a run
+
+A successful direct-DWU run must show all of the following in the Ray job
+log:
+
+1. Version 1 logs `dense_seed=True` with `changed == total`.
+2. After a real optimizer step, version 2 or later logs
+   `dense_seed=False`, `0 < changed < total`, and a smaller `wire_bytes` than
+   the dense seed.
+3. Training reports a finite, nonzero `train/grad_norm`.
+4. Rollout generation succeeds after the sparse commit and no worker reports a
+   base-version, sequence, final-manifest, or failed-session error.
+
+The updater exports these step metrics after a committed update:
+
+```text
+weight_sync/is_dense_seed
+weight_sync/total_elements
+weight_sync/changed_elements
+weight_sync/delta_density
+weight_sync/wire_bytes
+weight_sync/seconds
+```
+
+Its summary line has this form:
+
+```text
+Direct DWU committed version=<N> dense_seed=<bool> changed=<M>/<T> \
+density=<ratio> wire_bytes=<bytes> seconds=<seconds>
+```
 
-For object-store-backed volumes that need an explicit commit/refresh to make writes visible
-across hosts, supply `--custom-update-weight-post-write-path` (trainer side) /
-`--vllm-custom-pull-weights-pre-read-hook` (engine side) — no vendor-specific code lives in vime
-or vllm; see the doc.
+Process launch, a dense seed alone, or static tests do not by themselves
+demonstrate a working delta path; check all four criteria above.

tests/test_empty_colocated_weight_bucket.py

@@ -112,6 +112,8 @@ def gather_object(obj, object_gather_list, dst, group):
     update_from_distributed_mod.disconnect_rollout_engines_from_distributed = lambda *args, **kwargs: None
     update_from_distributed_mod.post_process_weights = lambda *args, **kwargs: None
     update_from_distributed_mod.update_weights_from_distributed = lambda *args, **kwargs: []
+    coordinator_mod = types.ModuleType("vime.backends.megatron_utils.update_weight.coordinator")
+    coordinator_mod.WeightUpdateCoordinator = object
 
     monkeypatch.setitem(sys.modules, "vime", vime_pkg)
     monkeypatch.setitem(sys.modules, "vime.backends", vime_backends_pkg)
@@ -132,6 +134,11 @@ def gather_object(obj, object_gather_list, dst, group):
         "vime.backends.megatron_utils.update_weight.update_weight_from_distributed",
         update_from_distributed_mod,
     )
+    monkeypatch.setitem(
+        sys.modules,
+        "vime.backends.megatron_utils.update_weight.coordinator",
+        coordinator_mod,
+    )
 
     return dist_state