Skip to content

feat(ascend): add disk checkpoint NPU patches - #1

Open
wangx700 wants to merge 3 commits into
Meihan-chen:sync/ascend-mainfrom
wangx700:feat/pr413-npu-disk-patches
Open

wangx700 wants to merge 3 commits into
Meihan-chen:sync/ascend-mainfrom
wangx700:feat/pr413-npu-disk-patches

Conversation

@wangx700

@wangx700 wangx700 commented Sep 15, 2026

Copy link
Copy Markdown

Adds only the disk checkpoint serving patches required by the PR-413 NPU stack.

8-card disk validation

Validated on Ascend NPUs 8-15 with Qwen3-4B, disk transport, num_rollout=10, rollout batch size 4, two samples per prompt, and response length 256. The table excludes the final v10 update because it overlaps Ray/vLLM job teardown; it compares the initial sync and stable v1-v9 updates.

Stage Delta ratio Delta wire Delta update Delta E2E Full ratio Full checkpoint Full update Full E2E
Initial sync - - 5.64s 34.05s - - 6.13s 48.41s
v1 0.56% 0.13 GB 10.19s 28.45s 100% 7.49 GiB 6.59s 31.09s
v2 0.63% 0.14 GB 4.67s 21.66s 100% 7.49 GiB 6.41s 30.36s
v3 0.70% 0.15 GB 4.24s 20.49s 100% 7.49 GiB 7.26s 31.12s
v4 0.60% 0.13 GB 4.98s 28.05s 100% 7.49 GiB 5.83s 26.13s
v5 0.84% 0.17 GB 4.17s 20.98s 100% 7.49 GiB 6.91s 25.66s
v6 0.63% 0.14 GB 4.23s 20.89s 100% 7.49 GiB 6.12s 26.96s
v7 0.61% 0.13 GB 4.34s 21.32s 100% 7.49 GiB 6.59s 28.42s
v8 0.58% 0.13 GB 4.16s 21.11s 100% 7.49 GiB 6.79s 29.80s
v9 0.77% 0.16 GB 4.30s 21.54s 100% 7.49 GiB 6.38s 29.86s
v1-v9 mean 0.66% ~0.14 GB 5.03s - 100% 7.49 GiB 6.54s -
10 rollout steps - - - 238.55s - - - 307.81s

E2E uses perf/step_time and covers rollout, training wait, training, and the update attributed to the step. Delta transfers about 0.66% of the model on average, but vLLM still reloads a reconstructed 7.49 GiB safetensors checkpoint on this NPU path.

@wangx700
wangx700 force-pushed the feat/pr413-npu-disk-patches branch from af68281 to ea7ec14 Compare September 15, 2026 02:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant