[Issue] RDMA PushRequest protocol mismatch causes ibv_post_send failure and data_num != 3 assertion on non-GDR (Ascend NPU) environment

[Dong-Jiahuan]

Summary

While porting StepMesh to a non-CUDA, Ascend NPU (ARM64) environment, Push operations crash consistently.
Initially the run failed with:

ibv_post_send failed.

But after further instrumentation, the crash was traced to the following protocol-layer assertion:

rdma_transport.h: PS_CHECK_EQ(buffer_ctx->data_num, 3)

This confirms the failure is not caused by RDMA verbs, but by an incorrect PushRequest data layout on the sender side.

---

Environment

• Hardware: Ascend NPU (ARM64)

• Architecture: aarch64

• StepMesh version: master (2025-01)

• RDMA: roce

• No CUDA / no GDR support

---

How I Identified the Real Root Cause

The initial crash happened inside ibv_post_send(). To rule out Inline SEND issues (EINVAL is common if inline size > device limit), I modified RDMAWriteWithImm:

if (inline_write) {
    wr.send_flags |= IBV_SEND_INLINE;
}
// Force-disable inline (important for ruling out EINVAL)

wr.send_flags &= ~IBV_SEND_INLINE;
if (prev_wr == nullptr) {

PS_CHECK_EQ(ibv_post_send(qp, &wr, &bad_wr), 0);

} else {

prev_wr->next = ≀

PS_CHECK_EQ(ibv_post_send(qp, prev_wr, &bad_wr), 0);

}

After disabling inline writes, the failure moved deterministically to:

rdma_transport.h: PS_CHECK_EQ(buffer_ctx->data_num, 3)

This shows:

1. RDMA verbs and QP are functional.

2. Inline was not the real cause.

3. The true bug is that the PushRequest sender is constructing an incorrect number of data segments.

Thus the crash is purely a protocol mismatch, not a hardware issue.

---

Expected Behavior

Per StepMesh internal RDMA protocol:

PushRequest (worker → server)
data_num = 3
segment[0] = keys
segment[1] = vals
segment[2] = lens

RecvPushRequest enforces this strictly:

PS_CHECK_EQ(buffer_ctx->data_num, 3);

rdma_transport

---

Actual Behavior on NPU Port

Sender constructs 1 or 2 segments, not 3.
As a result:

1. msg_buf->data.size() is wrong

2. SendRendezvousBegin sends a wrong data_num to peer

3. Receiver fails the assertion (data_num != 3)

4. Before disabling inline, this propagated as ibv_post_send errors (invalid WR state)

---

Root Cause

StepMesh currently relies on implicit, scattered assumptions for how many data segments each message type should contain.

The real contract is:

Message Type | Expected data_num | Required Segments -- | -- | -- PushRequest | 3 | keys / vals / lens PullRequest | 2 | keys / empty vals PullResponse | 3 | keys / vals / lens PushResponse | 0 | none

For PushRequest (non-GDR), the NPU sender must construct exactly 3 segments.
Because this convention is not centralized or validated, it is easy for non-GPU backends to violate it and cause fatal RDMA failures.

Attachments

Relevant source locations:

rdma_transport.h – receives PushRequest and asserts data_num == 3

rdma_van.h – converts msg.data into MessageBuffer

rdma_utils.h – Rendezvous structures

van.cc – SendMsg() code path
Closing

This issue is not RDMA-hardware related.
After disabling inline writes, the crash clearly originates from:

PushRequest sender not constructing 3 data segments.

A centralized protocol definition or normalization step would fix the issue and make StepMesh portable beyond CUDA/GDR environments.

[niehao100]

Hi,
Did you test build with no STEPMESH_USE_GDR flag in cmakelist ?

The original bytePS support the NO GDR mode and we start with it. The GDR support only put data on gpu and meta data on cpu, few modification can fix it.

We don't has Arm + 910 testbed. We also discuss with Ascend teams, as its roce function is implemented on NPU card, and verbs/fast sync is not supported, we believe the current version of stepmesh can not work on it without refactor, and it seems their teams are working on it.

If you are testing on a superpod, mayby ub interface is better choice.

[Dong-Jiahuan]

Hi, Did you test build with no STEPMESH_USE_GDR flag in cmakelist ?

The original bytePS support the NO GDR mode and we start with it. The GDR support only put data on gpu and meta data on cpu, few modification can fix it.

We don't has Arm + 910 testbed. We also discuss with Ascend teams, as its roce function is implemented on NPU card, and verbs/fast sync is not supported, we believe the current version of stepmesh can not work on it without refactor, and it seems their teams are working on it.

If you are testing on a superpod, mayby ub interface is better choice.

Yes, I am indeed building without the STEPMESH_USE_GDR flag enabled, so the error occurs outside the STEPMESH_USE_GDR-related code sections. Furthermore, I have implemented using the Ascend ACL interface according to your documentation (StepMesh/docs/backend.md) and registered the backend. I have also modified other parts of the code to adapt to the ARM architecture.

I am not sure if the current error is caused by my code modifications or if it is a general issue that arises when building without the STEPMESH_USE_GDR flag. I plan to test this in a GPU environment.

[Dong-Jiahuan]

Hi,
Did you test build with no STEPMESH_USE_GDR flag in cmakelist ?

The original bytePS support the NO GDR mode and we start with it. The GDR support only put data on gpu and meta data on cpu, few modification can fix it.

We don't has Arm + 910 testbed. We also discuss with Ascend teams, as its roce function is implemented on NPU card, and verbs/fast sync is not supported, we believe the current version of stepmesh can not work on it without refactor, and it seems their teams are working on it.

If you are testing on a superpod, mayby ub interface is better choice.

Hi,

I have fixed the PushRequest protocol mismatch
(PS_CHECK_EQ(buffer_ctx->data_num, 3)), and the communication proceeds further.

However, during communication tests I still hit a timeout:

PS_LOG(FATAL) << "request timeout " << timeout_ms << "ms, handler ..."

This happens in both cases:

• RDMA enabled (export DMLC_ENABLE_RDMA=1)
• RDMA disabled (no DMLC_ENABLE_RDMA)

Based on Postoffice::InitEnvironment, when RDMA is disabled the system falls back to the ZMQ Van by default. I would like to confirm if this understanding is correct.

Given that Ascend NPU does not support verbs / fast sync, I am wondering:

1. Is the ZMQ Van expected to support full data-plane communication (Push/Pull request-response) in StepMesh?

2. Is the observed timeout expected when running StepMesh on Ascend, even with ZMQ, due to platform or backend limitations?

3. Does Ascend support UCX-based communication?

I want to confirm whether this timeout indicates a misconfiguration on my side, or a current limitation of StepMesh on Ascend platforms.

Thanks.

[niehao100]

Hi @Dong-Jiahuan,

Thanks for you test. We didn't test ZMQ VAN for stepmesh.

I think maybe run cpu only mode with RDMA nic, and then DtoH/HtoD is a better choice.

[Dong-Jiahuan]

Hi @Dong-Jiahuan,

Thanks for you test. We didn't test ZMQ VAN for stepmesh.

I think maybe run cpu only mode with RDMA nic, and then DtoH/HtoD is a better choice.

Thanks for the suggestion. That is also what I was planning to try.
The server is currently unstable, so I will try it again later.
Thanks for the advice~