[Bug]: [NPU][ACL Graph] A3(910C) GLM-5.2-w8a8 fails to launch with enable_graph=true

[banxiaduhuo]

Your environment

Hardware: A3 with ARM
xLLM version: master

docker: quay.io/jd_xllm/xllm-ai:xllm-dev-a3-arm-cann9-20260605

startup parameters:
--max_memory_utilization=0.9
--max_tokens_per_batch=4096
--max_seqs_per_batch=16
--block_size=128
--enable_prefix_cache=true
--enable_chunked_prefill=true
--communication_backend="hccl"
--enable_schedule_overlap=true
--enable_graph=true
--ep_size=8
--dp_size=2
--nnodes=8

🐛 Describe the bug

ATB graph warmup crashes at capture_end with error code 107025 ("capture model contains a stream that was not joined to the original stream") on multi-die 910C. The same code, torch_npu, ATB, driver, image, and config run fine on single-die 910C.

The crash is on the first warmup bucket (bucket=1), ~1 second after warmup start. Forward completes successfully; the failure is at capture_end → aclmdlRICaptureEnd. The cleanup capture_end then fails with 107029 ("stream is not in capture status").

frame #12 of the crash stack lands in libhccl.so — the unjoined stream is created by HCCL during the MoE dispatch/combine collectives inside the captured forward, and is not joined back to the capture stream before capture_end.

Always happens on A3(910C).

Log from rank-0

I20260821 17:32:25.836668 84235 profile_manager.cpp:1283] Graph warmup started: bucket_count=1, configured_max_batch_size=2, allocatable_sequences=2, decode_seq_len=16
I20260821 17:32:25.937453 84730 acl_graph_executor_impl.cpp:384] [DIAG-107025] before capture: bucket=1 current_stream=0x564ec800 default_stream=0x564ec800 capture_stream=0x5645f200 graph_stream_=0x564ec800 current==default=1 current==capture=0
I20260821 17:32:25.937512 84730 acl_graph_executor_impl.cpp:408] [DIAG-107025] capture stream resolved: bucket=1 entered_switch_branch=1 graph_stream_=0x5645f200 capture_stream=0x5645f200 need_restore_stream=1
I20260821 17:32:25.937529 84730 acl_graph_executor_impl.cpp:422] [DIAG-107025] before capture_begin: bucket=1 current_stream=0x5645f200 graph_stream_=0x5645f200
I20260821 17:32:25.943053 84730 acl_graph_executor_impl.cpp:431] [DIAG-107025] after capture_begin (in capture): bucket=1 current_stream=0x5645f200 graph_stream_=0x5645f200
I20260821 17:32:26.286935 84730 acl_graph_executor_impl.cpp:442] [DIAG-107025] after forward (before capture_end): bucket=1 current_stream=0x5645f200 graph_stream_=0x5645f200 current==graph_stream=1
E20260821 17:32:26.321024 84730 acl_graph_executor_impl.cpp:464] ACL graph capture_end during cleanup failed: capture_end:torch_npu/csrc/core/npu/NPUGraph.cpp:247 NPU function error: c10_npu::acl::AclmdlRICaptureEnd(capture_stream_, &model_ri), error code is 107029
[ERROR] 2026-08-21-17:32:26 (PID:84235, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
EE9999: Inner Error!
EE9999[PID: 84235] 2026-08-21-17:32:26.320.711 (EE9999):  rtStreamEndCapture execution failed, reason=stream is not in capture status[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        TraceBack (most recent call last):
       end capture stream failed, runtime result = 107029[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]

Exception raised from capture_end at torch_npu/csrc/core/npu/NPUGraph.cpp:247 (most recent call first):
frame #0: c10::Error::Error(...) + 0xb0 (0xffffa942c700 in /usr/local/python3.11.15/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(...) + 0x68 (0xffffa93ca860 in /usr/local/python3.11.15/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10_npu::NPUGraph::capture_end() + 0xbec (0xffffb7e899ec in /usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x149f64c]
frame #4: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x14a3798]
frame #5: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x148d670]
frame #6: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13daaf4]
frame #7: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13dbff8]
frame #8: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13d9804]
frame #9: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13bd118]
frame #10: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0xe6dd58]
frame #11: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0xe6ea50]
frame #12: <unknown function> + 0x289da4 (0xffffb5e39da4 in /usr/local/Ascend/ascend-toolkit/latest/lib64/libhccl.so)
frame #13: <unknown function> + 0x7fbb4 (0xffffa65afbb4 in /lib64/libc.so.6)
frame #14: <unknown function> + 0xe7f4c (0xffffa6617f4c in /lib64/libc.so.6)
terminate called after throwing an instance of 'c10::Error'
  what():  capture_end:torch_npu/csrc/core/npu/NPUGraph.cpp:247 NPU function error: c10_npu::acl::AclmdlRICaptureEnd(capture_stream_, &model_ri), error code is 107025
[ERROR] 2026-08-21-17:32:26 (PID:84235, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
EE9999: Inner Error!
EE9999[PID: 84235] 2026-08-21-17:32:26.320.038 (EE9999):  rtStreamEndCapture execution failed, reason=capture model contains a stream that was not joined to the original stream[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        TraceBack (most recent call last):
       end capture stream failed, runtime result = 107025[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]

Exception raised from capture_end at torch_npu/csrc/core/npu/NPUGraph.cpp:247 (most recent call first):
frame #0: c10::Error::Error(...) + 0xb0 (0xffffa942c700 in /usr/local/python3.11.15/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(...) + 0x68 (0xffffa93ca860 in /usr/local/python3.11.15/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10_npu::NPUGraph::capture_end() + 0xbec (0xffffb7e899ec in /usr/local/python3.11.15/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x149edb8]
frame #4: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x14a3798]
frame #5: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x148d670]
frame #6: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13daaf4]
frame #7: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13dbff8]
frame #8: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13d9804]
frame #9: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0x13bd118]
frame #10: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0xe6dd58]
frame #11: /workspace/wangyuan/xllm_dev/xllm/build/lib.linux-aarch64-cpython-311/xllm/xllm() [0xe6ea50]
frame #12: <unknown function> + 0x289da4 (0xffffb5e39da4 in /usr/local/Ascend/ascend-toolkit/latest/lib64/libhccl.so)
frame #13: <unknown function> + 0x7fbb4 (0xffffa65afbb4 in /lib64/libc.so.6)
frame #14: <unknown function> + 0xe7f4c (0xffffa6617f4c in /lib64/libc.so.6)

[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!

910B (A2) with the identical config and software stack does NOT reproduce.
910C (A3) Always happens.

[XuZhang99]

add export HCCL_OP_EXPANSION_MODE="AIV" when launching xllm server.

[banxiaduhuo]

add export HCCL_OP_EXPANSION_MODE="AIV" when launching xllm server.

I tried HCCL_OP_EXPANSION_MODE=AIV — it did not help. It does not change how HCCL allocates/joins its internal communication streams during graph capture, which is where the crash occurs.

I also noticed vllm-ascend has reported a similar 107025 issue: vllm-project/vllm-ascend#13792