[NPU][A3] Garbled and repetitive MoE rollouts with inference EP after colocated IPC weight update

[Ulrica111]

Summary

We observe incorrect rollouts when running Qwen3-30B-A3B colocated RL training
on an Atlas A3 server with vLLM expert parallelism enabled.

After an actor-to-vLLM IPC weight update, some responses become garbled or
repetitive, often diverging from the first generated token.

Our current A/B results are:

Inference EP	MoE communication	Result
Disabled	Default non-EP path	Correct
EP=8	Native selection / MC2	Garbled or repetitive
EP=8	Forced ALLGATHER	Correct

Environment

• Server: Huawei Atlas A3
• NPU: 8 visible Ascend910 chips, physical IDs 0-7
• Host capacity: 16 physical chips on 8 dual-chip boards
• HBM per chip: 65536 MiB
• Model: Qwen3-30B-A3B
• Deployment: colocated RL training
• Weight synchronization: native NPU IPC
• Rollout topology: one 8-NPU engine
• Inference EP world size: 8
• Training EP size: 8
• CANN/Firmware: 9.0.0 / 9.0.0.6.200
• npu-smi: 26.1.0.b070

Relevant configuration

python train.py \
  --train-backend megatron \
  --actor-num-nodes 1 \
  --actor-num-gpus-per-node 8 \
  --rollout-num-gpus 8 \
  --rollout-num-gpus-per-engine 8 \
  --colocate \
  \
  --hf-checkpoint /path/to/Qwen3-30B-A3B \
  --load /path/to/Qwen3-30B-A3B \
  --megatron-to-hf-mode bridge \
  --weight-update-megatron-to-hf-mode bridge \
  \
  --rollout-backend vllm \
  --vllm-weight-sync-mode native \
  --vllm-enable-expert-parallel \
  --vllm-enable-sleep-mode \
  --vllm-enforce-eager \
  --vllm-gpu-memory-utilization 0.45 \
  --vllm-max-model-len 20480 \
  \
  --rollout-batch-size 8 \
  --n-samples-per-prompt 8 \
  --global-batch-size 64 \
  \
  --tensor-model-parallel-size 4 \
  --expert-model-parallel-size 8 \
  --expert-tensor-parallel-size 1

Weight-transfer verification

We inserted probes at the following stages:

1. actor-side exported weights;
2. tensors before IPC transfer;
3. tensors reconstructed by the receiver;
4. tensors passed to the native weight loader;
5. final vLLM runtime parameters.

We compared parameter names, shapes, dtypes, tensor values/hashes and
non-finite values, including packed MoE w13 and w2 weights.

No mismatch was found. This suggests that actor export, IPC transfer and native
weight loading are working correctly, and that the problem occurs during the
runtime MoE forward path.

MoE communication selection

vLLM-Ascend automatically selects the MoE communication backend according to
the device generation, EP topology, expert sharding and token count.

The relevant logic selects MC2 for the affected A3 workload when the token
count is within the MC2 capacity:

def _select_a3_moe_comm_method(
    num_tokens,
    mc2_tokens_capacity,
    vllm_config,
):
    if get_ascend_config().enable_fused_mc2 == 1:
        # FUSED_MC2 selection
        ...

    if num_tokens <= mc2_tokens_capacity:
        return MoECommType.MC2

    return MoECommType.ALLTOALL

For diagnosis, we added an opt-in workaround:

def _patch_npu_safe_moe_comm():
    if os.getenv("VIME_FORCE_MOE_ALLGATHER", "0").lower() not in {
        "1", "true", "yes"
    }:
        return

    import vllm_ascend.ascend_forward_context as ctx

    original_select = ctx.select_moe_comm_method

    def select_allgather(num_tokens, vllm_config, is_draft_model=False):
        selected = original_select(
            num_tokens, vllm_config, is_draft_model
        )
        if selected is None:
            return None
        return ctx.MoECommType.ALLGATHER

    ctx.select_moe_comm_method = select_allgather

With the same model, IPC weights, prompts, sampling configuration and EP
topology, forcing MoECommType.ALLGATHER produces coherent rollouts.

Additional observation

In some earlier configurations, coherent rollouts required both forced
ALLGATHER and decomposition of the fused residual AddRMSNorm path.

In the latest official environment, forced ALLGATHER is sufficient for the
samples tested so far. The exact boundary involving communication backend,
dynamic batching and residual AddRMSNorm is still under investigation. We will
share a separate A/B report once it is confirmed.

Questions

1. Should the vLLM-Ascend communication-selection policy avoid MC2 for
   colocated MoE RL workloads after live IPC weight updates?
2. Should selection consider additional runtime state, such as colocated mode,
   sleep/wake or live weight reload?
3. Is forcing ALLGATHER the intended correctness workaround for this topology?
4. Or should the underlying MC2 implementation/lifecycle be fixed so that MC2
   remains correct after IPC weight updates?
5. Are MC2 routing metadata, communication buffers or ACL graphs expected to be
   rebuilt after finish_weight_update or wake-up?

We would appreciate guidance on whether this should be addressed in Vime's
communication-selection policy or fixed more fundamentally in vLLM-Ascend's
MC2 implementation.

[floatlibai]

I wonder if mc2 is the root cause of the issue. Does vLLM-Ascend have accuracy issues when running single inference with EP enabled?

[Ulrica111]

I wonder if mc2 is the root cause of the issue. Does vLLM-Ascend have accuracy issues when running single inference with EP enabled?

We are currently conducting both vLLM-Ascend unit tests and VIME comparison tests to investigate the communication path. We strongly suspect that the issue is related to high concurrency. Specifically, we've observed that varying the vllm-server-concurrency parameter leads to significantly different rollout behaviors. We're actively debugging to isolate the root cause and will update once we have more findings.

[Ulrica111]

Follow-up: Vime IPC vs. Direct vLLM-Ascend Serving

Setup

• Hardware: one 8-NPU Ascend 910B A3 node.
• Image: quay.io/ascend/vime:0.3.0-a3-vllm0.22.1rc1.
• Model: Qwen3-30B-A3B.
• Test prompt: Please introduce yourself.

Both experiments used the native vLLM-Ascend communication selector. No forced ALLGATHER workaround was enabled. The probe only records the selected method.

Vime used IPC weight transfer, --colocate, inference EP=8, --vllm-enable-expert-parallel, --rollout-num-gpus-per-engine 8, and --vllm-gpu-memory-utilization 0.7.

The direct vLLM-Ascend test started a server and sent HTTP requests without Vime, IPC weight transfer, the training loop, or RolloutManager.

For Vime, I varied:

--vllm-server-concurrency=1/8/16/32/64

For direct serving, the client simultaneously sent 1/8/16/32/64 requests. Both use continuous batching, so configured concurrency is not the exact request count of each forward pass.

Results

Vime IPC End-to-End

Concurrency	Observed scheduler request counts	Communication	Rollout result
1	1	MC2	64 normal
8	1 through 8	MC2	64 normal
16	1, 3 through 16	MC2	64 normal
32	1, 24 through 32	MC2 + ALLTOALL	31 abnormal; 33 normal
64	1, 2, 51 through 60, 62 through 64	MC2 + ALLTOALL	62 abnormal; 2 normal

Direct vLLM-Ascend Serving

Concurrency	Observed scheduler request counts	Communication	Rollout result
1	1	MC2	1 normal
8	3, 5, 7, 8	MC2	8 normal
16	8, 15, 16	MC2	16 normal
32	2, 30, 31, 32	MC2 + ALLTOALL	32 normal
64	56, 63, 64	MC2 + ALLTOALL	64 normal

Direct serving also used continuous batching. At concurrency 32 and 64, the native selector chose ALLTOALL and all outputs remained normal.

Vime ALLTOALL Events

The prefill/decode composition is dynamically determined by scheduler state, not directly configured by concurrency.

Concurrency	Actual batch	Tokens	Communication	Result
32	31 prefill + 1 decode	373	ALLTOALL	The 31 newly admitted prefills were abnormal; the decode request was normal.
64	62 prefill + 2 decode	746	ALLTOALL	The 62 newly admitted prefills were abnormal; both decode requests were normal.

The number of abnormal rollouts exactly matched the number of newly admitted prefills.

Interpretation

Native mixed prefill/decode + ALLTOALL worked correctly in direct vLLM-Ascend serving with this model and EP=8. Therefore, the evidence does not indicate that ALLTOALL is generally broken.

Scenario	con32 ALLTOALL batch	con64 ALLTOALL batch	Result
Vime IPC end-to-end	31 prefill + 1 decode	62 prefill + 2 decode	Newly admitted prefills abnormal
Direct serving	30 prefill + 2 decode	56 prefill + 8 decode	All outputs normal

Although forcing ALLGATHER avoided corrupted rollouts in our previously affected Vime configuration, it is a workaround rather than a root-cause fix. Native ALLTOALL works in the direct-serving control, so globally replacing it does not explain the Vime-specific interaction.

The remaining investigation targets the Vime IPC/RL rollout path, including scheduler behavior, model-runner metadata, token layout, slot mapping, and IPC lifecycle state.