Conversation
…lm-project#163) * update docker patch. * fix mindspeed.patch try-except formatting per review Replace malformed features_manager hunks with proper try/except/pass blocks. * add torch_npu.patch for NPU Docker build Wrap eager_connect_single_device in try/except to avoid RuntimeError on A3.
…d HCCL support - Add is_npu() conditional for NPU/GPU device selection - Import mindspeed.megatron_adaptor on NPU (triggers transfer_to_npu) - Use hccl backend without device_id on NPU, nccl with device_id on GPU - Conditional torch.npu/torch.cuda for synchronize and empty_cache
- Rewrite Dockerfile.npu for verl base image (no git clone, COPY instead) - Use patch -p1 instead of git apply for shallow clone compatibility - Add megatron-all-changes.patch (slim 2203 lines, replaces 3 separate patches) - Fix corrupt vllm-ascend.patch and mindspeed.patch - Adapt convert_hf_to_torch_dist.py with is_npu() conditional and mindspeed import
…ch robustness flags - convert_hf_to_torch_dist.py: add back is_npu() + mindspeed.megatron_adaptor (lost during PR vllm-project#230 merge), use conditional cuda/npu paths - Dockerfile.npu: add -f --no-backup-if-mismatch and || true to patch commands to handle already-modified files in base image
…on for packed sequence support MindSpeed FusionAttentionFeature is not applied in Ray workers because sys.argv lacks --use-flash-attn, causing DotProductAttention to crash with 'Packed sequence is not supported' assertion. This patch replaces DotProductAttention.forward with npu_fusion_attention BEFORE mindspeed import, ensuring packed sequences (THD format) work correctly.
…on_patch sparse_mode default changed from 2 (rightDownCausal) to 0 (defaultMask) to match MindSpeed FusionAttentionFeature defaults. next_tockens default changed from 65536 to 0 for causal attention, preventing information leakage from future tokens during training. Incorrect defaults caused high train_rollout_logprob_abs_diff (~0.05-0.12 vs expected ~0.02) because the attention mask was not properly causal.
…PU resource allocation - vllm_engine.py: set is_checkpoint_format=False to bypass vLLM 0.20.2's initialize_layerwise_reload which is incompatible with Ascend layers, causing all weight updates to fail silently. This reduces train_rollout_logprob_abs_diff from ~0.10 to ~0.02. - actor_group.py: comment out NOSET_VISIBLE_DEVICES_ENV_VARS_LIST to let Ray manage ASCEND_RT_VISIBLE_DEVICES per worker - rollout.py: explicitly set ASCEND_RT_VISIBLE_DEVICES per engine and set NPU resource to 0 (devices are managed via env var, not Ray resource)
Add HF-to-packed name remapping in vllm-ascend load_weights_direct so HCCL weight sync loads qkv_proj and gate_up_proj correctly. Restore collective_rpc weight-update hooks and torch.npu IPC handling in vime. Extend Megatron and MindSpeed docker patches for packed-sequence training compatibility.
….npu Add torch-memory-saver-npu.patch to hook aclrtMalloc for PyTorch NPU CachingAllocator. Dockerfile uses base-image sgl-kernel-npu when present, otherwise clones from SGL_KERNEL_NPU_REPO, then builds and installs the preload .so.
Restore docker/ to match main branch, removing ascend-specific NPU Dockerfiles and patches ahead of merging npu branch Docker support.
* feat(ascend): add Dockerfile.npu with npu_patch and colocate vllm-ascend patch
Introduce ascend-branch NPU Docker build aligned with npu Dockerfile.npu: clone ascend vime, apply training-stack patches, and apply vllm-ascend.patch before editable install.
* feat(ascend): extend npu_patch for colocate mode on Ascend NPU
Align docker/npu_patch with verified colocate smoke on 910B (Qwen3-4B):
training and vLLM rollout share the same NPU cards.
Dockerfile.npu:
- Apply vllm.patch when installing vLLM (before pip install)
- Skip vllm.patch / vllm-ascend.patch in the training-stack patch loop
Patch summary:
- vllm-ascend.patch
- Rewrite load_weights_direct: map QKV packed weights, use weight_loader,
and TP shard fallback for Megatron→vLLM weight sync
- Bypass startup free-memory check (if False) for colocate shared memory
- Comment out memory-profiling assert that fails when training and vLLM
share the same device
- vllm.patch (new)
- In async_scheduler, clamp num_output_placeholders with max(0, ...)
instead of assert; avoids EngineDeadError when KV cache is ~99%+ and
preemption drives the counter negative under colocate rollout load
- mindspeed.patch
- Add False defaults to getattr on fused RoPE / MLA-related flags
- Wrap features_manager validate hooks in try/except AttributeError for
TransformerConfig (no MindSpeed-specific fields)
- Guard MoE feature validators when args lack moe_fb_overlap /
moe_zero_memory fields
- megatron.patch
- Return arch version 10 on NPU in get_device_arch_version() so training
does not call torch.cuda on Ascend
* chore(npu): remove ascend-branch changes ahead of main Revert the working tree back to the ascend/npu merge-base (fa0b6e9), deleting all ascend-specific modifications, in preparation for migrating the npu branch implementation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(npu): migrate Ascend NPU support implementation Apply the full set of changes from the npu branch (origin/npu, 6 commits squashed) on top of the cleaned ascend base. Adds NPU support, A3 Dockerfile, setup guide, training scripts and unit tests. Squashed from npu branch: - add npu support. (vllm-project#179) - fix npu_4b. (vllm-project#194) - fix vllm_engine bug. - [NPU]add Ascend A3 Dockerfile (vllm-project#204) - [NPU] Add Ascend NPU setup guide and Qwen3-4B training script (vllm-project#212) - add qwen3-30b-npu script & change 4b script location (vllm-project#254) Co-Authored-By: flb <floatlibai@gmail.com> Co-Authored-By: Meihan-chen <zr010426ztt@outlook.com> Co-Authored-By: wangxiaoxin-sherie <wangxiaoxin7@huawei.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(npu): update Dockerfile.npu to use vllm-ascend verified commit Align docker/Dockerfile.npu with vllm-project#256: - clone vllm and vllm-ascend first, then check out vllm to the vllm-ascend verified commit and install it once (avoids the checkout-after-install mismatch flagged in the PR review) - install vllm-ascend after its verified vllm is in place - install the vllm-router dependency - use a glob for the torch_memory_saver wheel instead of a pinned filename - simplify the import smoke test (drop AutoBridge import) Co-Authored-By: yzeyu71 <532183776@qq.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> * chore(npu): remove docker/ changes from npu migration Delete the NPU docker assets introduced by the npu migration and the Dockerfile.npu update (docker/Dockerfile.npu and docker/npu_patch/*), restoring the docker/ directory to its pre-migration state. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: flb <floatlibai@gmail.com> Co-authored-by: wangxiaoxin-sherie <wangxiaoxin7@huawei.com> Co-authored-by: yzeyu71 <532183776@qq.com>
…llm-project#271) * fix(ascend): apply npu_patch from build context and fix pre-commit Apply Megatron/MindSpeed patches from /tmp/npu_patch (build context) instead of the git-cloned vime tree, so PR and local docker builds use the correct patch files. Also fix vllm_engine F821, restore mindspeed side-effect imports with noqa, run pre-commit formatters, and align npu_patch README with the ascend branch. Signed-off-by: kaiyuan <kyxiezju@163.com> * chore(ci): add Buildkite pipeline config to ascend branch The ascend branch was missing .buildkite/pipeline.yml, causing buildkite-agent pipeline upload to fail on PR builds. Copy the config from main so CPU CI gates can run. Signed-off-by: kaiyuan <kyxiezju@163.com> * test(ci): add CPU unit test stubs for optional vllm deps Sync tests/_unit_stubs.py from main (vllm-project#268) so Buildkite utils pytest can import ascend NPU modules without vllm/vllm_router installed. Also call install_vllm_cli_stubs in test_megatron_role_config for vime.utils.arguments imports. Signed-off-by: kaiyuan <kyxiezju@163.com> --------- Signed-off-by: kaiyuan <kyxiezju@163.com>
* fix qwen3-30b ep on npu Signed-off-by: flb <floatlibai@gmail.com> * refactor vLLMWorkerExtension to monkey patch vllm Signed-off-by: flb <floatlibai@gmail.com> * fix ci pre-commit Signed-off-by: flb <floatlibai@gmail.com> --------- Signed-off-by: flb <floatlibai@gmail.com>
…oject#281) Port the Dockerfile.npu improvements onto the ascend branch while keeping the ascend-specific build (colocate patches, build-context npu_patch, and the vime clone from the ascend branch): - drop the hardcoded VLLM_COMMIT arg and instead check out vllm to the vllm-ascend verified commit, then apply vllm.patch and install once - install the vllm-router dependency - simplify the import smoke test (drop AutoBridge import) Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> Co-authored-by: yzeyu71 <532183776@qq.com>
…ct#275) On NPU, raw mode QKV reshape causes high train_rollout_logprob_abs_diff (0.05-0.18 vs GPU ~0.025) and GRPO fails to converge. Default to bridge when is_npu() so users do not need to pass --megatron-to-hf-mode bridge explicitly. Signed-off-by: kaiyuan <kyxiezju@163.com>
* add gpu tau-bench example Signed-off-by: flb <floatlibai@gmail.com> * add npu tau-bench example script Signed-off-by: flb <floatlibai@gmail.com> * drop GPU tau-bench script from ascend branch Signed-off-by: flb <floatlibai@gmail.com> --------- Signed-off-by: flb <floatlibai@gmail.com>
…cript (vllm-project#287) * feat(npu): Enhance Dockerfile.npu with verified vllm commit and run script Signed-off-by: wuxiang <498160096@qq.com> * fix(docker): correct patch file name in Dockerfile.npu and README.md Signed-off-by: wuxiang <498160096@qq.com> * Migrate Qwen3-4B training script to use Ray Job Submission Signed-off-by: wuxiang <498160096@qq.com> --------- Signed-off-by: wuxiang <498160096@qq.com>
* feat: add npu pipeline Add a test CI pipeline based on NPU Signed-off-by: yangzeyu <532183776@qq.com> * feat: add npu pipeline Add a test CI pipeline based on NPU Signed-off-by: yangzeyu <532183776@qq.com> * feat/change-queue Signed-off-by: yangzeyu <532183776@qq.com> * feat/pipeline-add-npu Signed-off-by: yangzeyu <532183776@qq.com> * feat/resource-class-modify Signed-off-by: yangzeyu <532183776@qq.com> * feat/add-k8s-configuration Signed-off-by: yangzeyu <532183776@qq.com> * fix/command-on-group Signed-off-by: yangzeyu <532183776@qq.com> * fix/images-not-valid Signed-off-by: yangzeyu <532183776@qq.com> * feat/add-image-build Signed-off-by: yangzeyu <532183776@qq.com> * fix/delete-duplicate-steps Signed-off-by: yangzeyu <532183776@qq.com> * fix/delete-duplicate-steps Signed-off-by: yangzeyu <532183776@qq.com> * fix/add-depends Signed-off-by: yangzeyu <532183776@qq.com> * fix/change-resource-class Signed-off-by: yangzeyu <532183776@qq.com> * fix add ray start Failed to connect to the default Ray cluster address at 172.22.0.188:6379 Signed-off-by: yangzeyu <532183776@qq.com> * fix:resolve conflict Signed-off-by: yangzeyu <532183776@qq.com> --------- Signed-off-by: yangzeyu <532183776@qq.com> Signed-off-by: YZY <40078195+yzeyu71@users.noreply.github.com>
…vllm-project#285) * feat(npu): colocate IPC weight sync on Ascend Move packed-module remap into vime colocate worker extension and keep vllm-ascend patch limited to colocate init (free_memory check and profiling assert). NPU colocate uses is_checkpoint_format=False, skips layerwise_reload, and loads weights via direct param writes with QKV/gate_up remap. Signed-off-by: kaiyuan <kyxiezju@163.com> * feat(npu): colocate TMS env and CANN path for Ascend Add TMS torch mode, training region, colocate-only PYTORCH_NPU_ALLOC_CONF override, CANN PYTHONPATH for Ray workers, and safe NPU empty_cache. Complements IPC weight sync for bridge colocate on 8x910B. Signed-off-by: kaiyuan <kyxiezju@163.com> * fix(npu): align colocate with pr285 container validation Add npu_attention_patch before mindspeed import so Megatron train-side attention uses npu_fusion_attention on Ascend, matching pr266 colocate. Signed-off-by: kaiyuan <kyxiezju@163.com> * fix(npu): fallback IPC device id when NPU uuid is absent Ascend torch_npu device properties may lack uuid; use the same fallback on IPC send and receive paths. Signed-off-by: kaiyuan <kyxiezju@163.com> --------- Signed-off-by: kaiyuan <kyxiezju@163.com>
…m-project#305) Switch the base image to quay.io/ascend/vllm-ascend:v0.22.1rc1-a3, which already ships vllm and vllm-ascend (installed editable). Drop the steps that clone, checkout the verified commit, and reinstall vllm/vllm-ascend from source since the release image provides them. The NPU colocate patches (vllm.patch, vllm-ascend.patch) are still required, so apply them directly to the editable source trees the base image installs at /vllm-workspace/vllm and /vllm-workspace/vllm-ascend. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ct#323) * refactor 30b-npu script & remove gpu script Signed-off-by: flb <floatlibai@gmail.com> * remove 4b gpu scpript Signed-off-by: flb <floatlibai@gmail.com> * add mindspeed pg_colletion mismatch patch Signed-off-by: flb <floatlibai@gmail.com> * use sync entrypoint Signed-off-by: flb <floatlibai@gmail.com> --------- Signed-off-by: flb <floatlibai@gmail.com>
* [NPU] Add Search-R1 example Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Add Search-R1 example Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Add Search-R1 example Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Add Search-R1 example Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Add Search-R1 example Signed-off-by: gaofulin <gaofulin1@huawei.com> --------- Signed-off-by: gaofulin <gaofulin1@huawei.com> Co-authored-by: gaofulin <gaofulin1@huawei.com>
…de (vllm-project#318) Signed-off-by: wuxiang <498160096@qq.com>
* feat/ci-npu-patch 1. The NPU environment update and modification are extracted into the script for execution 2. CI adds judgment, and applies to the corresponding component when there is a patch update Signed-off-by: yangzeyu <532183776@qq.com> * feat:update image version Signed-off-by: yangzeyu <532183776@qq.com> * feat/NPU CI isolated from the vime CI Signed-off-by: yangzeyu <532183776@qq.com> Signed-off-by: yangzeyu <yangzeyu7@huawei.com> --------- Signed-off-by: yangzeyu <532183776@qq.com> Signed-off-by: yangzeyu <yangzeyu7@huawei.com>
* [NPU] Add Retool example Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Add Retool example Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Add Retool example Signed-off-by: gaofulin <gaofulin1@huawei.com> --------- Signed-off-by: gaofulin <gaofulin1@huawei.com>
…ject#314) * vlm Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> * vlm Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> * fix nomask Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> * fix vl thd Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> * fix weight sync Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> * align vllm-project#341 rollout Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> * fix Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> --------- Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Signed-off-by: yangzeyu <yangzeyu7@huawei.com> Co-authored-by: yangzeyu <yangzeyu7@huawei.com>
…ject#376) Signed-off-by: wuxiang <498160096@qq.com>
* Refactor NPU patch reconciliation Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> * Refine NPU patch reconciliation Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> * Refine NPU patch reconciliation review fixes Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> * vlm Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> * arg vime commit Signed-off-by: Meihan-chen <zr010426ztt@outlook.com> --------- Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
* Apply-MegatronAdaptor-NPU-migration-to-clean-branch --------- Signed-off-by: wangxiaoxin-sherie <wangxiaoxin7@huawei.com> Co-authored-by: wangxiaoxin-sherie <wangxiaoxin7@huawei.com>
Keep main/default CUDA paths at their original call sites while isolating Ascend-specific Ray, HCCL, IPC, Megatron, and vLLM behavior. Restore per-slot native IPC orchestration and add focused platform contracts. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
* [NPU] Add Qwen3.5-35B-A3B support Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Add missing newline at end of file Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Remove CI test config Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Add missing newline at end of file Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Update model filename Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Add download timeout Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Remove download timeout Signed-off-by: gaofulin <gaofulin1@huawei.com> * [NPU] Remove md files Signed-off-by: gaofulin <gaofulin1@huawei.com> --------- Signed-off-by: gaofulin <gaofulin1@huawei.com>
* fix ref-load torch_dist error Signed-off-by: flb_ <floatlibai@gmail.com> * change to megatron adaptor Signed-off-by: flb_ <floatlibai@gmail.com> * add testcases to verify args and convertion Signed-off-by: flb_ <floatlibai@gmail.com> * fix pre-commit Signed-off-by: flb_ <floatlibai@gmail.com> * combine to one testcase Signed-off-by: flb_ <floatlibai@gmail.com> * add ref-load test Signed-off-by: flb_ <floatlibai@gmail.com> * use exiting 30b case Signed-off-by: flb_ <floatlibai@gmail.com> * fix path Signed-off-by: flb_ <floatlibai@gmail.com> * remove torch_dist first Signed-off-by: flb_ <floatlibai@gmail.com> --------- Signed-off-by: flb_ <floatlibai@gmail.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Use native stateful HCCL/NPU IPC backends and rank-local payloads. Add native Qwen3-VL loading and fix IPC/optimizer memory lifecycles. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Register the NPU accelerator before MindSpeed bootstrap, preserve engine-group resource hooks, and handle empty optimizer/payload boundaries. Rebase vendor patches onto vLLM e6bfe03a and Ascend fd815467, isolate MRV2 KV allocations across sleep/wake, and disable NZ in the NPU E2E cases. Validation: Qwen3-4B, Qwen3-30B-A3B and Qwen3-VL-8B E2E passed on 2026-09-08. Existing runtime dependencies retained; NPU MTP, Omni and pull weights remain deferred. Original vendor IPC test fixture incompatibilities remain documented. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Record the resolved upstream/ascend merge from the existing index. Keep working-tree adaptation in a separate follow-up commit. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Record current MegatronAdaptor, patch-ordering, native VL and Qwen3.5 GDN adaptation with their tests. Keep torch_dist/ref-load opt-in and remove its temporary output-directory override as requested. Qwen3.5 serving/FLA environment isolation remains a follow-up working-tree change; this snapshot does not claim a passing Qwen3.5 E2E. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Preserve the current S7 Qwen3.5 serving diagnostics: isolate training FLA libraries before serving startup and select spawn for vLLM TP workers. CPU contracts pass (132 tests); the latest E2E passes startup and reaches rollout but is blocked by stale vLLM-Ascend conv1d binary schemas. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Revert f5b8491 (vllm-project#409) and the associated NPU adaptations in 7a27808 and 81ae79d. Defer Qwen3.5 validation to the next stage with a fresh, matched serving and training environment. Remove the NPU-only GDN interfaces, FLA bootstrap and serving isolation, forced spawn, dedicated scripts/tests/CI entry, and FLA build recipe. Restore the pre-409 Bridge pin and patch while preserving the common TransformerLayer implementation. Retain S7 vllm-project#385 training-stack migration, vllm-project#396 torch_dist/ref-load, Qwen3-VL fixes, and main Qwen3.5 model code. Serving patches and latest patches are unchanged. No installed environment is rolled back. Validation: 183 grouped CPU tests passed; Ruff and runner shell syntax passed; common-to-NPU Megatron and reverted Bridge patches pass clean-base apply checks. Existing basic E2E and torch_dist PASS logs are retained; no fresh post-revert E2E was run. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Default both prepare() and the script entry point to torch_dist conversion and reference checkpoint loading. Retain VIME_TEST_TORCH_DIST_REF_LOAD=0 for the native HF path, and cover the default and explicit overrides in CPU contracts. Validation: 9 targeted CPU tests passed; Ruff and diff whitespace checks passed. No E2E was run for this default switch. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
* feat: support GLM-4.7-Flash (MLA + MoE) on Ascend NPU
Add the GLM47MTPBridge megatron.bridge plugin for GLM-4.7-Flash
(Glm4MoeLiteForCausalLM): MLA (Multi-Head Latent Attention) + GLM-style
MoE with MTP support on Ascend 910B NPU. The bridge overrides
provider_bridge/mapping_registry with the MLA-aware versions (always
individual MLA projections + q_a_layernorm, never fused QKV) while
reusing GLM45Bridge's MTP plumbing and fused-expert handling.
A runtime monkey-patch (_patch_glm_mtp_graph_friendly, env-gated by
VIME_PATCH_GLM_MTP_GRAPH=1) rewrites the MTP drafter's bool-mask index
assignment as torch.where so it is NPU cudagraph-capturable
(aclnnNonzeroV2 otherwise fails under graph capture).
Register a new end-to-end GLM NPU smoke test
(tests/test_glm4.7_30B_A3B_npu.py) in the Buildkite NPU CI smk suite to
validate the MLA + MoE + MTP integration and prevent regressions. The
test loads HF weights via bridge mode, mirrors
scripts/run-glm4.7-30B-A3B-npu.sh (TP=4/EP=8, MTP speculative decoding
under cudagraph), and exercises the GLM MTP graph patch.
Signed-off-by: louyujing <louyujing@huawei.com>
* fix(npu): make GLM-4.7 bridge mapping_registry revision-safe
The CI image pins Megatron-Bridge at commit 7f0fb345, whose
GLM45Bridge._uses_fused_experts reaches self.hf_pretrained.state.source
via _hf_source_and_keys. On the peft adapter export path
(build_adapter_conversion_tasks -> mapping_registry) self.hf_pretrained
is a config-only object with no `state` attribute, so the inherited
helper raises AttributeError and update_weights crashes before any
weight is written back to vLLM.
The same path also reads self.hf_config (property), which the peft path
never populates, so MTP mappings were silently skipped ("No HF config
found").
Override _uses_fused_experts / _hf_expert_suffix with revision-safe
versions that read HF keys through getattr + try/except and fall back to
the documented GLM fused-expert default (True, no .weight suffix) when
no HF state is available. Add _glm_hf_config to locate the config across
both the _hf_config attribute (newer revisions) and the hf_config
property (7f0fb345), and use it for the MTP layer lookup.
Signed-off-by: louyujing <louyujing@huawei.com>
* fix(npu): default _uses_fused_experts to False for GLM-4.7-Flash
GLM-4.7-Flash HuggingFace checkpoints ship per-expert weights
(experts.<n>.gate_proj / up_proj / down_proj), not fused tensors
(gate_up_proj / down_proj). The config-only fallback in
_uses_fused_experts previously defaulted to True (inherited from the
GLM-4.5 assumption), which generated fused-expert mappings and raised
KeyError: 'model.layers.*.mlp.experts.gate_up_proj' during weight load.
Default to False on the config-only path so the per-expert mapping
branch is selected. When HF state keys are available, key-based
detection still overrides either default.
Verified end-to-end on CI-aligned image (Megatron-Bridge 7f0fb345):
update_weights + rollout + train step + second update_weights all
succeed (Job succeeded).
Signed-off-by: louyujing <louyujing@huawei.com>
* refactor(npu): move GLM MTP cudagraph fix from runtime patch to vllm.patch
Replace the runtime monkey-patch (_patch_glm_mtp_graph_friendly in
update_weight_from_tensor.py, env-gated by VIME_PATCH_GLM_MTP_GRAPH=1)
with a static patch in docker/npu_patch/vllm.patch that rewrites the
same line in vllm's Glm4MoeLiteMultiTokenPredictorLayer.forward:
`inputs_embeds[positions == 0] = 0` -> a cudagraph-friendly `torch.where`.
The patch is applied at image build time, so the runtime hook, its env
gate, and the VIME_PATCH_GLM_MTP_GRAPH export in the run script / e2e
test are no longer needed. Behaviour is unchanged: the MTP drafter's
bool-mask index assignment (aclnnNonzeroV2) is replaced by an
element-wise torch.where so it is NPU cudagraph-capturable.
Verified end-to-end on CI-aligned image (Megatron-Bridge 7f0fb345):
Job succeeded with the vllm patch applied and the runtime patch removed.
Signed-off-by: louyujing <louyujing@huawei.com>
* fix(test): drop --colocate from GLM-4.7 NPU e2e test
The colocate mode (train + rollout sharing the same NPUs) was inherited
from test_qwen3_30B_A3B_npu.py, but the GLM-4.7 run script
(run-glm4.7-30B-A3B-npu.sh) uses dedicated NPUs for train (8) and
rollout (8) on a 16-NPU node. Under colocate, CI observed MTP loss
3.12 (>1.0 threshold) and 0% speculative acceptance, indicating MTP
weight sync/alignment is broken in colocate mode. Dropping --colocate
matches the run script's dedicated-NPU layout and restores healthy MTP
loss (~0.66, well under the 1.0 CI gate).
Signed-off-by: louyujing <louyujing@huawei.com>
---------
Signed-off-by: louyujing <louyujing@huawei.com>
Co-authored-by: louyujing <louyujing@huawei.com>
Apply the pinned isort and Black formatting to the ten files reported by pre-commit-npu. Keep runtime logic, CI configuration, and vendor patches unchanged. Validation: all nine pre-commit hooks passed across all files; 118 targeted regression tests passed. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Merge ascend 083aea6 with main-friendly native loading and non-colocate training. Reuse the shared GLM model configuration, disable MTP until G2, and align NPU NZ and RoPE settings. Add GLM CI coverage and native MLA/MoE mapping contracts; keep obsolete Bridge files and deferred MTP vendor hunks out of the merge result. Validation: user-reported GLM G1 E2E passed; 291 CPU tests passed; Ruff, Black, isort, shell syntax and diff checks passed. Local TEST ONLY model/dataset overrides, companion tests and the existing Qwen3-30B checkpoint override remain uncommitted. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Add native Ascend worker draft target lifecycle and port the GLM MTP graph-friendly position mask. Preserve main's shared training and weight-transfer orchestration. Keep GLM G1 as default and expose independent MTP/eager test switches, with mapping and launch contracts. Validation: 319 CPU regression tests and all nine pre-commit hooks passed on the exact commit candidate. GLM G2 eager (20260910T122325Z) and graph (20260910T123259Z) passed with local 128-token responses, including two training steps and main/draft updates; graph capture completed. Production test retains 2048 tokens. Explicit only-MTP CI gating remains deferred; nonzero-GRPO G2 is not claimed validated. Exclude local model/dataset/checkpoint overrides, companion tests and the temporary response-length change. Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive support for Ascend NPU hardware across the vime codebase, including dedicated Buildkite CI pipelines, Docker configurations, and platform-specific abstractions for device discovery, Ray resource allocation, and weight synchronization. The review feedback identifies three critical issues: a NameError in vllm_engine.py due to an undefined args variable, a potential unhandled AttributeError in the memory cleanup logic of tool_sandbox.py, and a configuration compatibility issue in npu_attention_patch.py due to spelling variations of pre_tokens and next_tokens.
Comment on lines
+71
to
+75
| env = current_platform().vllm.subprocess_env( | ||
| env, | ||
| visible_devices=server_args_dict["_visible_devices"], | ||
| colocate=getattr(args, "colocate", False), | ||
| ) |
Contributor
There was a problem hiding this comment.
The variable args is not defined in the scope of the module-level function _build_subprocess_env, which will raise a NameError at runtime when attempting to access args.colocate. To fix this, retrieve the colocate flag from server_args_dict instead.
Suggested change
| env = current_platform().vllm.subprocess_env( | |
| env, | |
| visible_devices=server_args_dict["_visible_devices"], | |
| colocate=getattr(args, "colocate", False), | |
| ) | |
| env = current_platform().vllm.subprocess_env( | |
| env, | |
| visible_devices=server_args_dict["_visible_devices"], | |
| colocate=server_args_dict.get("colocate", False), | |
| ) |
Comment on lines
+65
to
+71
| try: | ||
| import ctypes | ||
|
|
||
| ctypes.CDLL(libc_path).malloc_trim(0) | ||
| break | ||
| except OSError: | ||
| continue |
Contributor
There was a problem hiding this comment.
If libc_path is loaded successfully but does not contain the malloc_trim attribute (such as libc.musl.so.1), accessing malloc_trim will raise an AttributeError. Since the except block only catches OSError, this will result in an unhandled exception and crash the process. Catching both OSError and AttributeError prevents this crash.
Suggested change
| try: | |
| import ctypes | |
| ctypes.CDLL(libc_path).malloc_trim(0) | |
| break | |
| except OSError: | |
| continue | |
| try: | |
| import ctypes | |
| ctypes.CDLL(libc_path).malloc_trim(0) | |
| break | |
| except (OSError, AttributeError): | |
| continue |
Comment on lines
+35
to
+36
| pre_tockens = getattr(self.config, "pre_tockens", 65536) | ||
| next_tockens = getattr(self.config, "next_tockens", 0) |
Contributor
There was a problem hiding this comment.
If the configuration defines pre_tokens or next_tokens (with the correct spelling), getattr(self.config, "pre_tockens") will fail to find them and fallback to the default values. Checking both spellings defensively ensures compatibility with standard configurations.
Suggested change
| pre_tockens = getattr(self.config, "pre_tockens", 65536) | |
| next_tockens = getattr(self.config, "next_tockens", 0) | |
| pre_tockens = getattr(self.config, "pre_tockens", getattr(self.config, "pre_tokens", 65536)) | |
| next_tockens = getattr(self.config, "next_tockens", getattr(self.config, "next_tokens", 0)) |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.