issues of single node example

[menggerSherry]

I run run_single_gpu.sh example, but got this issues:

[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/postoffice.cc:75: Creating Van: ibverbs. group_size=1
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:153: bind to DMLC_NODE_HOST: 29.27.177.139
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:601: Bind to [role=scheduler, id=1, ip=29.27.177.139, port=8123, is_recovery=0, aux_id=-1, num_ports=1]
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:178: Connecting to Node 1, My_Node=1
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7162
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7163
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7164
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 1 OnConnect to 1 with Transport=RDMA QP_NUM 7164
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7165
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 1 OnConnect to 1 with Transport=RDMA QP_NUM 7165
[16:07:32] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/postoffice.cc:75: Creating Van: ibverbs. group_size=1
[16:07:32] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:562:  interface and ip from env: enp194s0f1 (29.27.177.139)
[16:07:32] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:153: bind to DMLC_NODE_HOST: 29.27.177.139
[16:07:32] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:601: Bind to [role=worker, ip=29.27.177.139, port=58679, is_recovery=0, aux_id=-1, num_ports=1]
[16:07:32] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:178: Connecting to Node 1, My_Node=32767
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/postoffice.cc:75: Creating Van: ibverbs. group_size=1
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:562:  interface and ip from env: enp194s0f1 (29.27.177.139)
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:153: bind to DMLC_NODE_HOST: 29.27.177.139
[[16:07:3216:07:32] ] workerserver  00  /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc::198601: : qp created: pd=Bind to 0x4001dc001030[role=server, ip=29.27.177.139, port=53921, is_recovery=0, aux_id=-1, num_ports=1] , cq=
0x4001dc001160[, qp=16:07:327166] 
server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:178: Connecting to Node 1, My_Node=32767
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7167
[16:07:32] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030[ , cq=16:07:320x4001dc001160] , qp=scheduler7168 
0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7169
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7170
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 1 OnConnect to 32767 with Transport=RDMA QP_NUM 7169
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7171
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 1 OnConnect to 32767 with Transport=RDMA QP_NUM 7171
[16:07:32] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 32767 OnConnected to 1
could not set CPU affinity: gpu 0-> cpu18
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=[0x40030000116016:07:32, qp=] 7172worker
 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 32767 OnConnected to 1
[could not set CPU affinity: gpu 0-> cpu17[
16:07:3216:07:32] ] schedulerworker  00  /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc::731701: : rdmardma  132767	received: 	sent: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=29.27.177.139, port=58679, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=worker, ip=29.27.177.139, port=58679, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0

[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:312: AddNode (1/2): [role=worker, ip=29.27.177.139, port=58679, is_recovery=0, aux_id=0, num_ports=1]
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 1 OnConnect to 32767 with Transport=RDMA QP_NUM 7172
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7173
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 1 OnConnect to 32767 with Transport=RDMA QP_NUM 7173
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 32767 OnConnected to 1
could not set CPU affinity: gpu 0-> cpu13
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 1 OnConnected to 32767
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 1 OnConnected to 32767
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 1 OnConnected to 32767
[16:07:32[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:] 1097server:  10 OnConnected to  32767/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h
:1097: 32767 OnConnected to 1
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:148: Initialized BackendMemoryAllocator for GPU 0 with pd 0x40034c001030
could not set CPU affinity: gpu 0-> cpu12
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc[:16:07:32701] : schedulerrdma  032767 	sent: /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=server, ip=29.27.177.139, port=53921, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0731
: rdma 1	received: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=server, ip=29.27.177.139, port=53921, is_recovery=0, aux_id=0, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:211: rank detected for node [role=server, ip=29.27.177.139, port=53921, is_recovery=0, aux_id=0, num_ports=1]
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:211: rank detected for node [role=worker, ip=29.27.177.139, port=58679, is_recovery=0, aux_id=0, num_ports=1]
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:262: assign id=8 to node [role=server, ip=29.27.177.139, port=53921, is_recovery=0, aux_id=0, num_ports=1]
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:178: Connecting to Node 8, My_Node=1
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7174
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7175
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7176
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 32767 OnConnect to 1 with Transport=RDMA QP_NUM 7176
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:148: Initialized BackendMemoryAllocator for GPU 0 with pd 0x40034c001030
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7177
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 32767 OnConnect to 1 with Transport=RDMA QP_NUM 7177
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:148: Initialized BackendMemoryAllocator for GPU 0 with pd 0x40034c001030
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 1 OnConnected to 8
[16:07:32] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 32767 OnConnected to 1
[16:07:32[16:07:32] server 0 ] /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.hscheduler: 10970:  32767/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h OnConnected to :11097
: 1 OnConnected to 8
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:262: assign id=9 to node [role=worker, ip=29.27.177.139, port=58679, is_recovery=0, aux_id=0, num_ports=1]
[16:07:32] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:178: Connecting to Node 9, My_Node=1
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7178
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7179
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030 , cq=0x4001dc001160, qp=7180
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 32767 OnConnect to 1 with Transport=RDMA QP_NUM 7180
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030 , cq=0x4001dc001160, qp=7181
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 32767 OnConnect to 1 with Transport=RDMA QP_NUM 7181
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 1 OnConnected to 9
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 32767 OnConnected to 1
[[16:07:3316:07:33] ] schedulerworker  00  /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h::10971097: : 132767 OnConnected to  OnConnected to 91

[[16:07:3316:07:33] ] schedulerworker  00  /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc::701731: : rdmardma  132767	sent: 	received: ? => 9. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=server, id=8, ip=29.27.177.139, port=53921, is_recovery=0, aux_id=0, num_ports=1] [role=worker, id=9, ip=29.27.177.139, port=58679, is_recovery=0, aux_id=0, num_ports=1] [role=scheduler, id=1, ip=29.27.177.139, port=8123, is_recovery=0, aux_id=-1, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 01 => 32767. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=server, id=8, ip=29.27.177.139, port=53921, is_recovery=0, aux_id=0, num_ports=1] [role=worker, id=9, ip=29.27.177.139, port=58679, is_recovery=0, aux_id=0, num_ports=1] [role=scheduler, id=1, ip=29.27.177.139, port=8123, is_recovery=0, aux_id=-1, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0

[16:07:33] worker [[016:07:3316:07:33 ] ] /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.hschedulerserver:  17800:   Connecting to Node /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h8::, My_Node=7014359: : 
rdmaSendRendezvousReply (GDR Server): meta_len= 19441, data_len=	sent: 0
? => 8. Meta: request=0, timestamp=1, control={ cmd=ADD_NODE, node={ [role=server, id=8, ip=29.27.177.139, port=53921, is_recovery=0, aux_id=0, num_ports=1] [role=worker, id=9, ip=29.27.177.139, port=58679, is_recovery=0, aux_id=0, num_ports=1] [role=scheduler, id=1, ip=29.27.177.139, port=8123, is_recovery=0, aux_id=-1, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc[:16:07:33286] : serverThe scheduler is connected to  10 workers and  1/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h servers:
490: GDR Server Reply: meta_addr=400354010000, meta_rkey=247dfb, data_addr=0, data_rkey=0
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:731: rdma 32767	received: 1 => 32767. Meta: request=0, timestamp=1, control={ cmd=ADD_NODE, node={ [role=server, id=8, ip=29.27.177.139, port=53921, is_recovery=0, aux_id=0, num_ports=1] [role=worker, id=9, ip=29.27.177.139, port=58679, is_recovery=0, aux_id=0, num_ports=1] [role=scheduler, id=1, ip=29.27.177.139, port=8123, is_recovery=0, aux_id=-1, num_ports=1] } }. NOT DATA MSG!, Slave QP Count: 0
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:178: Connecting to Node 8, My_Node=8
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030 , cq=0x4001dc001160, qp=7182
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:701: rdma 1	sent: [16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:731: rdma 1	received: ? => 1. Meta: request=1, timestamp=2, control={ cmd=INSTANCE_BARRIER, barrier_group=7 }. NOT DATA MSG!, Slave QP Count: 0
1 => 1. Meta: request=1, timestamp=2, control={ cmd=INSTANCE_BARRIER, barrier_group=7 }. NOT DATA MSG!, Slave QP Count: 0
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:385: Instance barrier count for 7 : 1
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7183
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030 , cq=0x4001dc001160, qp=7184
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7185
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7186
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 8 OnConnect to 9 with Transport=RDMA QP_NUM 7186
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:148: Initialized BackendMemoryAllocator for GPU 0 with pd 0x40034c001030
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7187
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 8 OnConnect to 9 with Transport=RDMA QP_NUM 7187
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:148: Initialized BackendMemoryAllocator for GPU 0 with pd 0x40034c001030
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 9 OnConnected to 8
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7188
[[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 9 OnConnected to 8
16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:178: Connecting to Node 9, My_Node=9
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030 , cq=0x4001dc001160, qp=7189
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 8 OnConnect to 8 with Transport=RDMA QP_NUM 7188
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:148: Initialized BackendMemoryAllocator for GPU 0 with pd 0x40034c001030
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7190
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 8 OnConnect to 8 with Transport=RDMA QP_NUM 7190
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:148: Initialized BackendMemoryAllocator for GPU 0 with pd 0x40034c001030
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 8 OnConnected to 9
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 8 OnConnected to 9
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:148: Initialized BackendMemoryAllocator for GPU 0 with pd 0x40034c001030
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:178: Connecting to Node 9, My_Node=8
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030 , cq=0x4001dc001160, qp=7191
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030 , cq=0x4001dc001160, qp=7192
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7193
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 9 OnConnect to 9 with Transport=RDMA QP_NUM 7192
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030 , cq=0x4001dc001160, qp=7194
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 9 OnConnect to 9 with Transport=RDMA QP_NUM 7194
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7195
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030 , cq=0x4001dc001160, qp=7196
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 9 OnConnect to 8 with Transport=RDMA QP_NUM 7196
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:178: Connecting to Node 1, My_Node=9
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 8 OnConnected to 9
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030 , cq=0x4001dc001160, qp=7197
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 9 OnConnect to 8 with Transport=RDMA QP_NUM 7197
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 9 OnConnected to 8
[[16:07:33] worker[ 16:07:330]  server/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h :01097 : /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h9: OnConnected to 10978: 
8 OnConnected to 9
16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:148: Initialized BackendMemoryAllocator for GPU 0 with pd 0x40034c001030
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:178: Connecting to Node 1, My_Node=8
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030 , cq=0x4001dc001160, qp=7198
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7199
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x4001dc001030 , cq=0x4001dc001160, qp=7200
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7201
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x40034c001030 , cq=0x40034c001160, qp=7202
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 1 OnConnect to 9 with Transport=RDMA QP_NUM 7201
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7203
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 1 OnConnect to 9 with Transport=RDMA QP_NUM 7203
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 9 OnConnected to 1
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7204
[[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 9 OnConnected to 1
16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:520: W[9] is connected to others
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 1 OnConnect to 8 with Transport=RDMA QP_NUM 7204
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:198: qp created: pd=0x400300001030 , cq=0x400300001160, qp=7205
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1014: 1 OnConnect to 8 with Transport=RDMA QP_NUM 7205
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 8 OnConnected to 1
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 1 OnConnected to 9
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 1 OnConnected to 9
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:1097: 1 OnConnected to 8
[16:07:33[16:07:33] scheduler 0]  server/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h :01097 : /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h1: OnConnected to 10978: 
8 OnConnected to 1
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:148: Initialized BackendMemoryAllocator for GPU 0 with pd 0x40034c001030
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:520: S[8] is connected to others
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:[70116:07:33: ] rdmascheduler  90	sent:  ? => 1. Meta: request=1, timestamp=1, control={ cmd=INSTANCE_BARRIER, barrier_group=7 }. NOT DATA MSG!, Slave QP Count: 0/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc
:731: rdma 1	received: 9 => 1. Meta: request=1, timestamp=1, control={ cmd=INSTANCE_BARRIER, barrier_group=7 }. NOT DATA MSG!, Slave QP Count: 0
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:385: Instance barrier count for 7 : 2
[16:07:33] server 0 [/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc16:07:33:] 701scheduler:  rdma0  8/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc	sent: :731? => 1. Meta: request=1, timestamp=1, control={ cmd=INSTANCE_BARRIER, barrier_group=7 }. NOT DATA MSG!, Slave QP Count: 0: 
rdma 1	received: 8 => 1. Meta: request=1, timestamp=1, control={ cmd=INSTANCE_BARRIER, barrier_group=7 }. NOT DATA MSG!, Slave QP Count: 0
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:385: Instance barrier count for 7 : 3
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:701: rdma 1[	sent: 16:07:33? => 9. Meta: request=0, timestamp=3, control={ cmd=INSTANCE_BARRIER, barrier_group=0 }. NOT DATA MSG!, Slave QP Count: 0] 
worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:[731[16:07:33: 16:07:33] rdma] scheduler server 9 0	received: 0 1 => 9. Meta: request=0, timestamp=3, control={ cmd=INSTANCE_BARRIER, barrier_group=0 }. NOT DATA MSG!, Slave QP Count: 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc
/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h::701435: : rdmaSendRendezvousReply (GDR Server): meta_len= 2881, data_len=	sent: 0? => 8. Meta: request=0, timestamp=4, control={ cmd=INSTANCE_BARRIER, barrier_group=0 }. NOT DATA MSG!, Slave QP Count: 0

[[16:07:3316:07:33] ] serverscheduler  00  /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc::490701: : GDR Server Reply: meta_addr=rdma400354030000 , meta_rkey=13c252a	sent: , data_addr=? => 1. Meta: request=0, timestamp=5, control={ cmd=INSTANCE_BARRIER, barrier_group=0 }. NOT DATA MSG!, Slave QP Count: 00
, data_rkey=[016:07:33
] scheduler[ 16:07:330]  server/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc :0731 : /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.ccrdma: 7311: 	received: rdma1 => 1. Meta: request=0, timestamp=5, control={ cmd=INSTANCE_BARRIER, barrier_group=0 }. NOT DATA MSG!, Slave QP Count: 0 
8	received: 1 => 8. Meta: request=0, timestamp=4, control={ cmd=INSTANCE_BARRIER, barrier_group=0 }. NOT DATA MSG!, Slave QP Count: 0
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:701: rdma 1	sent: [? => 1. Meta: request=1, timestamp=6, control={ cmd=INSTANCE_BARRIER, barrier_group=7 }. NOT DATA MSG!, Slave QP Count: 0
16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:731: rdma 1	received: 1 => 1. Meta: request=1, timestamp=6, control={ cmd=INSTANCE_BARRIER, barrier_group=7 }. NOT DATA MSG!, Slave QP Count: 0
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:385: Instance barrier count for 7 : 1
KVWorker instance_idx,0
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/include/ps/kv_app.h:116: Enable worker zero-copy pull
could not set CPU affinity: gpu 0-> cpu19[
16:07:33[] [16:07:33server16:07:33]  ] workerscheduler  000   /home/bingxing2/home/scx7753/mxy/StepMesh-main/include/ps/af_tensor_app.h/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:::447701731: : : AFTensorServer runs on gpu rdmardma0  
91	sent: 	received: ? => 1. Meta: request=1, timestamp=2, control={ cmd=INSTANCE_BARRIER, barrier_group=6 }. NOT DATA MSG!, Slave QP Count: 09 => 1. Meta: request=1, timestamp=2, control={ cmd=INSTANCE_BARRIER, barrier_group=6 }. NOT DATA MSG!, Slave QP Count: 0

[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:385: Instance barrier count for 6 : 1
could not set CPU affinity: gpu 0-> cpu11
[[16:07:33] [server16:07:33 ] 0scheduler  /home/bingxing2/home/scx7753/mxy/StepMesh-main/include/ps/af_tensor_app.h0: 604/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc: :Start ResponseWorker 7310: 
rdma16:07:33 ] 1server	received:  8 => 1. Meta: request=1, timestamp=2, control={ cmd=INSTANCE_BARRIER, barrier_group=6 }. NOT DATA MSG!, Slave QP Count: 00
 [/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc16:07:33:] 701scheduler:  rdma0  8/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc	sent: :? => 1. Meta: request=1, timestamp=2, control={ cmd=INSTANCE_BARRIER, barrier_group=6 }. NOT DATA MSG!, Slave QP Count: 0385
: Instance barrier count for 6 : 2
[16:07:33] scheduler 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:701: rdma[ 16:07:331] 	sent: worker? => 9. Meta: request=0, timestamp=7, control={ cmd=INSTANCE_BARRIER, barrier_group=0 }. NOT DATA MSG!, Slave QP Count: 0 
0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:731: rdma[[ 16:07:3316:07:339] ] 	received: schedulerserver1 => 9. Meta: request=0, timestamp=7, control={ cmd=INSTANCE_BARRIER, barrier_group=0 }. NOT DATA MSG!, Slave QP Count: 0  
00  /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h::701435: : rdmaSendRendezvousReply (GDR Server): meta_len= 2881, data_len=	sent: 0? => 8. Meta: request=0, timestamp=8, control={ cmd=INSTANCE_BARRIER, barrier_group=0 }. NOT DATA MSG!, Slave QP Count: 0

[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:490: GDR Server Reply: meta_addr=400354050000, meta_rkey=3c454b, data_addr=0, data_rkey=0
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:731: rdma 8	received: 1 => 8. Meta: request=0, timestamp=8, control={ cmd=INSTANCE_BARRIER, barrier_group=0 }. NOT DATA MSG!, Slave QP Count: 0
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/include/ps/af_tensor_app.h:249: ZPush_ addr: 0x4001db800000 val_len: 32768
[[16:07:3316:07:33] ] workerserver  00  /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h::701435: : rdmaSendRendezvousReply (GDR Server): meta_len= 2969, data_len=	sent: 32776? => 8. Meta: request=1, timestamp=0, app_id=0, customer_id=0, simple_app=0, push=1, sid=0, head=1, key=0, dtype={ UINT64 OTHER }, Slave QP Count: 0 Body: { CPU(0)->CPU(0) data_size=[8,32768,] }

[16:07:33] worker [016:07:33 ] /home/bingxing2/home/scx7753/mxy/StepMesh-main/include/ps/af_tensor_app.hserver: 2490:  ZPush_ addr: 0x/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h4001db808000: val_len: 45032768: 
Alloc new gpu buffer: key=0, size=32768
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:452: GPU buffer allocated: key=0, addr=70377401233648, size=32768
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc:701: rdma 9	sent: ? => 8. Meta: request=1, timestamp=1, app_id=0, customer_id=0, simple_app=0, push=1, sid=0, head=2, key=1, dtype={ UINT64 OTHER }, Slave QP Count: 0 Body: { CPU(0)->CPU(0) data_size=[8,32768,] }
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/include/ps/af_tensor_app.h:249: ZPush_ addr: 0x4001db810000 val_len: 32768
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:202: Allocated new GPU buffer for key=0 addr=0x40034bc08000 size=32768
[[16:07:3316:07:33] ] serverworker  00  /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc::490701: : GDR Server Reply: meta_addr=rdma400354070000 , meta_rkey=93c494d	sent: , data_addr=? => 8. Meta: request=1, timestamp=2, app_id=0, customer_id=0, simple_app=0, push=1, sid=0, head=2, key=2, dtype={ UINT64 OTHER }, Slave QP Count: 0 Body: { CPU(0)->CPU(0) data_size=[8,32768,] }40034bc08000
, data_rkey=3c5358
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:435: SendRendezvousReply (GDR Server): meta_len=296, data_len=32776
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h[:16:07:33450] : workerAlloc new gpu buffer: key= 10, size= 32768/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/van.cc
:[70116:07:33: ] rdmaserver  90	sent:  ? => 8. Meta: request=1, timestamp=3, app_id=0, customer_id=0, simple_app=0, push=0, sid=0, head=3, key=0, dtype={ UINT64 OTHER }, Slave QP Count: 0 Body: { CPU(0)->CPU(0) data_size=[8,32768,] }/home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h
:452: GPU buffer allocated: key=1, addr=70377401235776, size=32768
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:202: Allocated new GPU buffer for key=1 addr=0x40034bc10000 size=32768
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:490: GDR Server Reply: meta_addr=400354090000, meta_rkey=3c5a5e, data_addr=40034bc10000, data_rkey=3c5d62
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:435: SendRendezvousReply (GDR Server): meta_len=296, data_len=32776
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:450: Alloc new gpu buffer: key=2, size=32768
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:452: GPU buffer allocated: key=2, addr=70377401237008, size=32768
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./././rdma_utils.h:202: Allocated new GPU buffer for key=2 addr=0x40034bc18000 size=32768
[16:07:33] server 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/././rdma_transport.h:490: GDR Server Reply: meta_addr=4003540b0000, meta_rkey=3c5f64, data_addr=40034bc18000, data_rkey=3c656b
[16:07:33] worker 0 /home/bingxing2/home/scx7753/mxy/StepMesh-main/include/dmlc/logging.h:301: [16:07:33] /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:785: Check failed: wc[i].status == IBV_WC_SUCCESS Failed status 
local protection error 4 0 81 128 RECV postoffice ptr: 0xaaaecfb860f0

Stack trace returned 6 entries:
[bt] (0) /home/bingxing2/home/scx7753/mxy/StepMesh-main/fserver_lib.cpython-310-aarch64-linux-gnu.so(+0x63a00) [0x400193063a00]
[bt] (1) /home/bingxing2/home/scx7753/mxy/StepMesh-main/fserver_lib.cpython-310-aarch64-linux-gnu.so(+0x63d14) [0x400193063d14]
[bt] (2) /home/bingxing2/home/scx7753/mxy/StepMesh-main/fserver_lib.cpython-310-aarch64-linux-gnu.so(ps::RDMAVan::PollCQ()+0x2b0) [0x4001930c44a0]
[bt] (3) /home/bingxing2/apps/anaconda/2021.11/envs/py310torch251cu121/lib/libstdc++.so.6(+0xdd5ec) [0x4000fa91d5ec]
[bt] (4) /usr/lib64/libpthread.so.0(+0x87ac) [0x4000d81e87ac]
[bt] (5) /usr/lib64/libc.so.6(+0xd60fc) [0x4000d84260fc]


terminate called after throwing an instance of 'dmlc::Error'
  what():  [16:07:33] /home/bingxing2/home/scx7753/mxy/StepMesh-main/src/./rdma_van.h:785: Check failed: wc[i].status == IBV_WC_SUCCESS Failed status 
local protection error 4 0 81 128 RECV postoffice ptr: 0xaaaecfb860f0

Stack trace returned 6 entries:
[bt] (0) /home/bingxing2/home/scx7753/mxy/StepMesh-main/fserver_lib.cpython-310-aarch64-linux-gnu.so(+0x63a00) [0x400193063a00]
[bt] (1) /home/bingxing2/home/scx7753/mxy/StepMesh-main/fserver_lib.cpython-310-aarch64-linux-gnu.so(+0x63d14) [0x400193063d14]
[bt] (2) /home/bingxing2/home/scx7753/mxy/StepMesh-main/fserver_lib.cpython-310-aarch64-linux-gnu.so(ps::RDMAVan::PollCQ()+0x2b0) [0x4001930c44a0]
[bt] (3) /home/bingxing2/apps/anaconda/2021.11/envs/py310torch251cu121/lib/libstdc++.so.6(+0xdd5ec) [0x4000fa91d5ec]
[bt] (4) /usr/lib64/libpthread.so.0(+0x87ac) [0x4000d81e87ac]
[bt] (5) /usr/lib64/libc.so.6(+0xd60fc) [0x4000d84260fc]


/var/spool/slurmd/job1063841/slurm_script: line 57: 3072967 Aborted                 DMLC_ROLE=worker numactl -m 0 python3 $THIS_DIR/$BIN.py $@
+ cleanup
+ echo 'kill all testing process of ps lite for user scx7753'

my script is

function cleanup() {
    echo "kill all testing process of ps lite for user $USER" # 定义清理函数 强行杀掉所有包含 test_remote_moe， test_fserver名字的
    # pkill -9 -f test_bench
    pkill -9 -f test_remote_moe
    pkill -9 -f test_fserver
    sleep 1
}
trap cleanup EXIT #在脚本退出时执行cleanup
# cleanup



# common setup
export ROLE=joint
export RNIC=enp194s0f1
export CUDA_VISIBLE_DEVICES=0,1,2,3      
# common setup
export BIN=${BIN:-test_fserver}
# export DMLC_INTERFACE=${RNIC:-brainpf_bond0}
export SCHEDULER_IP=$(ip -o -4 addr | grep ${RNIC} | awk '{print $4}' | cut -d'/' -f1)
export DMLC_NUM_WORKER=1
export DMLC_NUM_SERVER=1
export DMLC_PS_ROOT_URI=$SCHEDULER_IP  # scheduler's RDMA interface IP 
export DMLC_PS_ROOT_PORT=8123     # scheduler's port (can random choose)
export DMLC_ENABLE_RDMA=ibverbs
export DMLC_INTERFACE=auto
# export STEPMESH_BIND_CPU_CORE=1

export DMLC_NODE_HOST=${SCHEDULER_IP}
# export DMLC_INTERFACE=auto
export STEPMESH_SPLIT_QP_LAG=0
export STEPMESH_BIND_CPU_CORE=1
export STEPMESH_GPU=0
export PS_VERBOSE=2

DMLC_ROLE=scheduler numactl -m 0 python3 $THIS_DIR/$BIN.py &
export STEPMESH_CPU_START_OFFSET=10
DMLC_ROLE=server numactl -m 0 python3 $THIS_DIR/$BIN.py $@ &
# DMLC_ROLE=worker python3 $THIS_DIR/$BIN.py $@ &
# export STEPMESH_DROP_RATE=1
export STEPMESH_CPU_START_OFFSET=15
DMLC_ROLE=worker numactl -m 0 python3 $THIS_DIR/$BIN.py $@

wait

My machine is a A100 platform with 4 GPUs per node. Here is the information of the NIC:

$ibdev2netdev
mlx5_0 port 1 ==> enp194s0f0 (Up)
mlx5_1 port 1 ==> enp194s0f1 (Up)
mlx5_2 port 1 ==> enp226s0f0 (Up)
mlx5_3 port 1 ==> enp226s0f1 (Up)

$ lsmod | grep nvidia_peermem
nvidia_peermem        262144  0
ib_core               589824  9 rdma_cm,ib_ipoib,nvidia_peermem,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
nvidia              57081856  109 nvidia_uvm,nvidia_peermem,nvidia_modeset

$ ip addr show enp194s0f1
4: enp194s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4500 qdisc mq state UP group default qlen 1000
    link/ether 08:c0:eb:89:8a:fb brd ff:ff:ff:ff:ff:ff
    inet 29.27.26.104/16 brd 29.27.255.255 scope global noprefixroute enp194s0f1
       valid_lft forever preferred_lft forever

And I use ib_write_bw to test RDMA. It passed the test.

[niehao100]

Hi @menggerSherry，which commit did you test? It seems your cpu arch is aarch64, which we didn't test.
The error seems caused by ibv_reg_mr of gdr address(for tensor in GPU). Please try to set the tensor in test_fserver.py to CPU tensor for verification. If it works well, the root cause should be GDR misconfiguration.

[menggerSherry]

Hi @niehao100, Thanks for your help. I m tring to build on ARM platforms. Flowing your advice, I edit the test_fserver.py:

import torch, os
import time
import fserver_lib as f
print("test")
is_worker = os.environ.get('DMLC_ROLE') == 'worker'
is_server = os.environ.get('DMLC_ROLE') == 'server'

f.init()
print("test init")
if is_worker:
    gpu = os.environ.get('STEPMESH_GPU')
    # print(gpu)
    push_tensors = [
        torch.rand([1, 8192], dtype=torch.float32, device=f'cpu'),
        torch.rand([1, 8192], dtype=torch.float32, device=f'cpu'),
        torch.rand([1, 8192], dtype=torch.float32, device=f'cpu'),
    ]
    print(push_tensors)
    pull_tensors = [
        torch.rand([1, 8192], dtype=torch.float32, device=f'cpu')
    ]
    handler = f.push_pull(
        push_tensors, 
        [i for i in range(len(push_tensors))], 
        pull_tensors, 
        [i for i in range(len(pull_tensors))]
    )
    f.wait(handler)
    assert torch.allclose(sum(push_tensors), pull_tensors[0])
    print("worker test done")

elif is_server:
    gpu = os.environ.get('STEPMESH_GPU')
    torch.set_default_device('cpu')
    res = []
    while len(res) == 0:
        time.sleep(1)
        res = f.get_batch()
    print('results:',res)
    for r in res:
        comm_id, batch, _ = r
        f.respond([sum(batch)], comm_id)

f.stop()

and run script

THIS_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
function cleanup() {
    echo "kill all testing process of ps lite for user $USER"
    # pkill -9 -f test_bench
    pkill -9 -f test_remote_moe
    pkill -9 -f test_fserver
    sleep 1
}
trap cleanup EXIT
cleanup

# common setup
export ROLE=joint
export RNIC=enp194s0f1
export CUDA_VISIBLE_DEVICES=0,1,2,3      
export BIN=${BIN:-test_fserver}
export DMLC_INTERFACE=enp194s0f1
export SCHEDULER_IP=$(ip -o -4 addr | grep ${DMLC_INTERFACE} | awk '{print $4}' | cut -d'/' -f1)
export DMLC_NUM_WORKER=1
export DMLC_NUM_SERVER=1
echo $SCHEDULER_IP
export DMLC_PS_ROOT_URI=$SCHEDULER_IP  # scheduler's RDMA interface IP 
export DMLC_PS_ROOT_PORT=8123     # scheduler's port (can random choose)
export DMLC_ENABLE_RDMA=ibverbs
export DMLC_INTERFACE=auto

export DMLC_NODE_HOST=${SCHEDULER_IP}
# export DMLC_INTERFACE=auto
export STEPMESH_SPLIT_QP_LAG=0
export STEPMESH_GPU=0
export PS_VERBOSE=2

DMLC_ROLE=scheduler python3 $THIS_DIR/$BIN.py &
DMLC_ROLE=server python3 $THIS_DIR/$BIN.py $@ &
DMLC_ROLE=worker python3 $THIS_DIR/$BIN.py $@

wait

But still facing errors

kill all testing process of ps lite for user scx7753
29.27.144.188
/home/bingxing2/home/scx7753/.conda/envs/torch2.7/lib/python3.10/site-packages/torch/cuda/__init__.py:62: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/home/bingxing2/home/scx7753/.conda/envs/torch2.7/lib/python3.10/site-packages/torch/cuda/__init__.py:62: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/home/bingxing2/home/scx7753/.conda/envs/torch2.7/lib/python3.10/site-packages/torch/cuda/__init__.py:62: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
test
test
test
[[[14:48:4614:48:4614:48:46] ] ] workerschedulerserver   /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/postoffice.cc/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/postoffice.cc/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/postoffice.cc:::757575: : : Creating Van: Creating Van: Creating Van: ibverbsibverbsibverbs. group_size=. group_size=. group_size=111


[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:153: bind to DMLC_NODE_HOST: 29.27.144.188
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:600: Bind to [role=scheduler, id=1, ip=29.27.144.188, port=8123, is_recovery=0, aux_id=-1, num_ports=1]
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:178: Connecting to Node 1, My_Node=1
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47598, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47599, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47600, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 1 OnConnect to 1 with Transport=RDMA QP_NUM 47600
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47601, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 1 OnConnect to 1 with Transport=RDMA QP_NUM 47601
[[14:48:46] 14:48:46server]  worker/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc :/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc556:: 556automatic detect interface and ip from gpu: : enp226s0f1:0automatic detect interface and ip from gpu:  (enp226s0f1:029.27.192.91 ()29.27.192.91
)
[[14:48:4614:48:46] ] serverworker  /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h::153153: : bind to DMLC_NODE_HOST: bind to DMLC_NODE_HOST: 29.27.192.9129.27.192.91

[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:600: Bind to [role=server, ip=29.27.192.91, port=41793, is_recovery=0, aux_id=-1, num_ports=1]
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:178: Connecting to Node 1, My_Node=32767
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:600: Bind to [role=worker, ip=29.27.192.91, port=41147, is_recovery=0, aux_id=-1, num_ports=1]
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:178: Connecting to Node 1, My_Node=32767
[[14:48:4614:48:46] ] serverworker  /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h::202202: : qp created: pd=qp created: pd=0x4001680010900x40014c001090 , cq= , cq=0x4001680011c00x40014c0011c0, qp=, qp=4536445365, maxInline=, maxInline=572572

[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:[202: qp created: pd=14:48:460x40014c001090]  , cq=server0x40014c0011c0 , qp=/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h45366:, maxInline=202572: 
qp created: pd=0x400168001090 , cq=0x4001680011c0, qp=45367, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47602, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 1 OnConnect to 32767 with Transport=RDMA QP_NUM 47602
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47603, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 1 OnConnect to 32767 with Transport=RDMA QP_NUM 47603
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 32767 OnConnected to 1
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47604, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 1 OnConnect to 32767 with Transport=RDMA QP_NUM 47604
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 32767 OnConnected to 1
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47605, maxInline=572
[[14:48:46] 14:48:46scheduler]  worker/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h :/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h1009:: 10921:  OnConnect to 3276732767 OnConnected to  with Transport=1RDMA
 QP_NUM 47605
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:312: AddNode (1/2): [role=worker, ip=29.27.192.91, port=41147, is_recovery=0, aux_id=0, num_ports=1]
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 1 OnConnected to 32767
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 1 OnConnected to 32767
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 1 OnConnected to 32767
[14:48:46] server [/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h14:48:46:] 1092scheduler:  32767/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h OnConnected to :11092
: 1 OnConnected to 32767
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h:149: Initialized BackendMemoryAllocator for GPU 0 with pd 0x400168001090
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:211: rank detected for node [role=worker, ip=29.27.192.91, port=41147, is_recovery=0, aux_id=0, num_ports=1]
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:211: rank detected for node [role=server, ip=29.27.192.91, port=41793, is_recovery=0, aux_id=0, num_ports=1]
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:262: assign id=9 to node [role=worker, ip=29.27.192.91, port=41147, is_recovery=0, aux_id=0, num_ports=1]
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:178: Connecting to Node 9, My_Node=1
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47608, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47606, maxInline=572
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x40014c001090 , cq=0x40014c0011c0, qp=45368, maxInline=572
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 32767 OnConnect to 1 with Transport=RDMA QP_NUM 45368
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: [qp created: pd=14:48:460x40014c001090]  , cq=scheduler0x40014c0011c0 , qp=/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h45369:, maxInline=1092572: 
1 OnConnected to [914:48:46
] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 32767 OnConnect to 1 with Transport=RDMA QP_NUM 45369
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 32767 OnConnected to 1
[14:48:46] [scheduler14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 32767 OnConnected to 1
 /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 1 OnConnected to 9
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:262: assign id=8 to node [role=server, ip=29.27.192.91, port=41793, is_recovery=0, aux_id=0, num_ports=1]
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:178: Connecting to Node 8, My_Node=1
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47607, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47609, maxInline=572
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x400168001090 , cq=0x4001680011c0, qp=45370, maxInline=572
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 32767 OnConnect to 1 with Transport=RDMA QP_NUM 45370
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h:149: Initialized BackendMemoryAllocator for GPU 0 with pd 0x400168001090
[14:48:46] server[ /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x400168001090 , cq=0x4001680011c0, qp=45371, maxInline=572
14:48:46] scheduler[ 14:48:46/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h] :server1092 : /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h1: OnConnected to 10098: 
32767 OnConnect to 1 with Transport=RDMA QP_NUM 45371
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h:149: Initialized BackendMemoryAllocator for GPU 0 with pd 0x400168001090
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 32767 OnConnected to 1
[14:48:46] [scheduler14:48:46 ] /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.hserver: 1092/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h: :10921:  OnConnected to 327678 OnConnected to 
1[
14:48:46[] scheduler[ 14:48:46/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc14:48:46] :] server286worker  : /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:The scheduler is connected to :4411: 178 workers and SendRendezvousReply (GDR Server): meta_len=: 11944Connecting to Node  servers, data_len=9
0, My_Node=
9
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:492: GDR Server Reply: meta_addr=70374851149824, meta_rkey=1745435, data_addr=0, data_rkey=0
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:178: Connecting to Node 9, My_Node=8
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x40014c001090 , cq=0x40014c0011c0, qp=45372, maxInline=572
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x40014c001090 , cq=0x40014c0011c0, qp=45373, maxInline=572
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x40014c001090 , cq=0x40014c0011c0, qp=45374, maxInline=572
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x400168001090 , cq=0x4001680011c0, qp=45375, maxInline=572
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 9 OnConnect to 9 with Transport=RDMA QP_NUM 45374
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x400168001090 , cq=0x4001680011c0, qp=45376, maxInline=572
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x40014c001090 , cq=0x40014c0011c0, qp=45379, maxInline=572
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 9 OnConnect to 9 with Transport=RDMA QP_NUM 45379
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x40014c001090 , cq=0x40014c0011c0, qp=45380, maxInline=572
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 9 OnConnect to 8 with Transport=RDMA QP_NUM 45380
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 8 OnConnected to 9
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x40014c001090 , cq=0x40014c0011c0, qp=45385, maxInline=572
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 9 OnConnect to 8 with Transport=RDMA QP_NUM 45385
[[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:[109214:48:4614:48:46] server ] /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.hworker:1092:  8/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h OnConnected to 9
:178: [Connecting to Node 14:48:468] , My_Node=server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h9:: 149: Initialized BackendMemoryAllocator for GPU 0 with pd 0x400168001090
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:178: Connecting to Node 89, My_Node=8 OnConnected to 
8

[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 9 OnConnected to 8
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x400168001090 , cq=0x4001680011c0, qp=45386, maxInline=572
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x40014c001090 , cq=0x40014c0011c0, qp=45387, maxInline=572[
14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x400168001090 , cq=0x4001680011c0, qp=45388, maxInline=572
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x400168001090 , cq=0x4001680011c0, qp=45389, maxInline=572
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x40014c001090 , cq=0x40014c0011c0, qp=45390, maxInline=572
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 8 OnConnect to 8 with Transport=RDMA QP_NUM 45389
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h:149: Initialized BackendMemoryAllocator for GPU 0 with pd 0x400168001090
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x400168001090 , cq=0x4001680011c0, qp=45391, maxInline=572
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 8 OnConnect to 8 with Transport=RDMA QP_NUM 45391
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h:149: Initialized BackendMemoryAllocator for GPU 0 with pd 0x400168001090
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x400168001090 , cq=0x4001680011c0, qp=45392, maxInline=572
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 8 OnConnect to 9 with Transport=RDMA QP_NUM 45392
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h:149: Initialized BackendMemoryAllocator for GPU 0 with pd 0x400168001090
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h[:20214:48:46: ] qp created: pd=worker0x400168001090  , cq=/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h0x4001680011c0:, qp=109245393: , maxInline=9572 OnConnected to 
8
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 8 OnConnect to 9 with Transport=RDMA QP_NUM 45393
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h:149: Initialized BackendMemoryAllocator for GPU 0 with pd 0x400168001090
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:[1092: 914:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:178: Connecting to Node 1, My_Node=9
 OnConnected to 8
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h[:1092: 14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h:149: Initialized BackendMemoryAllocator for GPU 0 with pd 0x4001680010908 OnConnected to 9

[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:178: Connecting to Node 1, My_Node=[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:8
1092: 8 OnConnected to 9
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x400168001090 , cq=0x4001680011c0, qp=45394, maxInline=572
[14:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x400168001090 , cq=0x4001680011c0, qp=45395, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47610, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 1 OnConnect to 8 with Transport=RDMA QP_NUM 47610
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x40014c001090 , cq=0x40014c0011c0, qp=45396, maxInline=572
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202[: 14:48:46qp created: pd=] 0x40014c001090scheduler[ , cq= 14:48:460x40014c0011c0/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h] , qp=:server45397202 , maxInline=: /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h572qp created: pd=:
0x4004cc0010901092 , cq=: 0x4004cc0011c08, qp= OnConnected to 476131, maxInline=
572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 1 OnConnect to 8 with Transport=RDMA QP_NUM 47613
[14:48:46[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:202: qp created: pd=0x4004cc001090 , cq=0x4004cc0011c0, qp=47615, maxInline=] 572server
 /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:[1092: 814:48:46] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h:149: Initialized BackendMemoryAllocator for GPU [014:48:46 with pd ] 0x400168001090scheduler
 [/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h14:48:46:] 1009server:  1/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc OnConnect to :9520 with Transport=: RDMAS[8] QP_NUM  is connected to others47615

 OnConnected to 1
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 1 OnConnected to 8
[14:48:46[] 14:48:46scheduler]  worker/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h :/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h202:: 1092qp created: pd=: 0x4004cc0010909 , cq= OnConnected to 0x4004cc0011c01, qp=
47621, maxInline=572
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1009: 1 OnConnect to 9 with Transport=RDMA QP_NUM 47621
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 1 OnConnected to 8
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: 1 OnConnected to 9
[14:48:46[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:1092: ] 1worker OnConnected to  9/home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h
:1092: 9 OnConnected to 1
[14:48:46] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:520: W[9] is connected to others
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:385: Instance barrier count for 7 : 1
[14:48:46] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:385: Instance barrier count for 7 : 2
[14:48:47] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:385: Instance barrier count for 7 : 3
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:441: SendRendezvousReply (GDR Server): meta_len=288, data_len=0
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:492: GDR Server Reply: meta_addr=70374851280896, meta_rkey=1923504, data_addr=0, data_rkey=0
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/include/ps/af_tensor_app.h:432: AFTensorServer runs on gpu 0
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/include/ps/af_tensor_app.h:589: Start ResponseWorker 0
[14:48:47] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:385: Instance barrier count for 6 : 1
[14:48:47] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/include/ps/kv_app.h:115: Enable worker zero-copy pull
[14:48:47] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:385: Instance barrier count for 6 : 2
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:441: SendRendezvousReply (GDR Server): meta_len=288, data_len=0
test init
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:492: GDR Server Reply: meta_addr=70374851411968, meta_rkey=1896082, data_addr=0, data_rkey=0
test init
[tensor([[0.2406, 0.5999, 0.2618,  ..., 0.1305, 0.7256, 0.8594]]), tensor([[0.7216, 0.9704, 0.8005,  ..., 0.8948, 0.4491, 0.4099]]), tensor([[0.7043, 0.4091, 0.6620,  ..., 0.1652, 0.2163, 0.8058]])]
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:441: SendRendezvousReply (GDR Server): meta_len=296, data_len=32776
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:456: Alloc new gpu buffer: key=0, size=32768
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h:203: Allocated new GPU buffer for key=0 addr=0x40018fc08000 size=32768
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:492: GDR Server Reply: meta_addr=70374851543040, meta_rkey=1828652, data_addr=70375450902528, data_rkey=1810472
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:441: SendRendezvousReply (GDR Server): meta_len=296, data_len=32776
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:456: Alloc new gpu buffer: key=1, size=32768
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h:203: Allocated new GPU buffer for key=1 addr=0x40018fc10000 size=32768
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:492: GDR Server Reply: meta_addr=70374851674112, meta_rkey=1801501, data_addr=70375450935296, data_rkey=1829421
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:441: SendRendezvousReply (GDR Server): meta_len=296, data_len=32776
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:456: Alloc new gpu buffer: key=2, size=32768
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./././rdma_utils.h:203: Allocated new GPU buffer for key=2 addr=0x40018fc18000 size=32768
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:492: GDR Server Reply: meta_addr=70374851805184, meta_rkey=1786659, data_addr=70375450968064, data_rkey=1822763
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:441: SendRendezvousReply (GDR Server): meta_len=296, data_len=32776
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:456: Alloc new gpu buffer: key=0, size=32768
[14:48:47] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/././rdma_transport.h:492: GDR Server Reply: meta_addr=70374851936256, meta_rkey=1807655, data_addr=70375450902528, data_rkey=1810472
test init
[14:48:47] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:385: Instance barrier count for 7 : 1
results: [(0, [tensor([[0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0'), tensor([[0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0'), tensor([[0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')], [0, 1, 2])]
[14:48:48] scheduler /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/van.cc:385: Instance barrier count for 7 : 2
[14:48:48] server /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/include/dmlc/logging.h:301: [14:48:48] /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:779: Failed status 
local protection error 4 187656443519744 81 128 RECV postoffice ptr: 0xaaad902e4660

Stack trace returned 6 entries:
[bt] (0) /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/fserver_lib.cpython-310-aarch64-linux-gnu.so(+0x65080) [0x40013fe35080]
[bt] (1) /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/fserver_lib.cpython-310-aarch64-linux-gnu.so(+0x65314) [0x40013fe35314]
[bt] (2) /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/fserver_lib.cpython-310-aarch64-linux-gnu.so(ps::RDMAVan::PollCQ()+0x2cc) [0x40013fe98adc]
[bt] (3) /home/bingxing2/home/scx7753/.conda/envs/torch2.7/bin/../lib/libstdc++.so.6(+0xda2e0) [0x4000625da2e0]
[bt] (4) /usr/lib64/libpthread.so.0(+0x87ac) [0x40003c7f87ac]
[bt] (5) /usr/lib64/libc.so.6(+0xd60fc) [0x40003ca360fc]


terminate called after throwing an instance of 'dmlc::Error'
  what():  [14:48:48] /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:779: Failed status 
local protection error 4 187656443519744 81 128 RECV postoffice ptr: 0xaaad902e4660

Stack trace returned 6 entries:
[bt] (0) /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/fserver_lib.cpython-310-aarch64-linux-gnu.so(+0x65080) [0x40013fe35080]
[bt] (1) /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/fserver_lib.cpython-310-aarch64-linux-gnu.so(+0x65314) [0x40013fe35314]
[bt] (2) /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/fserver_lib.cpython-310-aarch64-linux-gnu.so(ps::RDMAVan::PollCQ()+0x2cc) [0x40013fe98adc]
[bt] (3) /home/bingxing2/home/scx7753/.conda/envs/torch2.7/bin/../lib/libstdc++.so.6(+0xda2e0) [0x4000625da2e0]
[bt] (4) /usr/lib64/libpthread.so.0(+0x87ac) [0x40003c7f87ac]
[bt] (5) /usr/lib64/libc.so.6(+0xd60fc) [0x40003ca360fc]


[14:49:01] worker /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/include/dmlc/logging.h:301: [14:49:01] /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:939: Check failed: 0 OnEvent: unknown event 11 (RDMA_CM_EVENT_DEVICE_REMOVAL)

Stack trace returned 6 entries:
[bt] (0) /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/fserver_lib.cpython-310-aarch64-linux-gnu.so(+0x65080) [0x400124fc5080]
[bt] (1) /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/fserver_lib.cpython-310-aarch64-linux-gnu.so(+0x65314) [0x400124fc5314]
[bt] (2) /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/fserver_lib.cpython-310-aarch64-linux-gnu.so(ps::RDMAVan::PollEvents()+0x828) [0x4001250201a8]
[bt] (3) /home/bingxing2/home/scx7753/.conda/envs/torch2.7/bin/../lib/libstdc++.so.6(+0xda2e0) [0x40004776a2e0]
[bt] (4) /usr/lib64/libpthread.so.0(+0x87ac) [0x4000219887ac]
[bt] (5) /usr/lib64/libc.so.6(+0xd60fc) [0x400021bc60fc]


terminate called after throwing an instance of 'dmlc::Error'
  what():  [14:49:01] /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/src/./rdma_van.h:939: Check failed: 0 OnEvent: unknown event 11 (RDMA_CM_EVENT_DEVICE_REMOVAL)

Stack trace returned 6 entries:
[bt] (0) /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/fserver_lib.cpython-310-aarch64-linux-gnu.so(+0x65080) [0x400124fc5080]
[bt] (1) /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/fserver_lib.cpython-310-aarch64-linux-gnu.so(+0x65314) [0x400124fc5314]
[bt] (2) /home/bingxing2/home/scx7753/mxy/sglang/StepMesh/fserver_lib.cpython-310-aarch64-linux-gnu.so(ps::RDMAVan::PollEvents()+0x828) [0x4001250201a8]
[bt] (3) /home/bingxing2/home/scx7753/.conda/envs/torch2.7/bin/../lib/libstdc++.so.6(+0xda2e0) [0x40004776a2e0]
[bt] (4) /usr/lib64/libpthread.so.0(+0x87ac) [0x4000219887ac]
[bt] (5) /usr/lib64/libc.so.6(+0xd60fc) [0x400021bc60fc]


tests/fserver/run_single_gpu.sh: line 35: 321744 Aborted                 DMLC_ROLE=server python3 $THIS_DIR/$BIN.py $@
tests/fserver/run_single_gpu.sh: line 35: 321745 Aborted                 DMLC_ROLE=worker python3 $THIS_DIR/$BIN.py $@
kill all testing process of ps lite for user scx7753
tests/fserver/run_single_gpu.sh: line 2: 321743 Killed                  DMLC_ROLE=scheduler python3 $THIS_DIR/$BIN.py

It seems the GPU address ptr error. So what should I do next ?