[BUG] Crashed when scaling out to 4 workers 4 servers.

[Zhangmj0621]

We successfully tested the StepMesh configuration with M=2 and N=2 workers and servers, but encountered a crash when configuring with M=4 and N=4.
Below is the error log. What steps should I take to resolve this issue?

RuntimeError: [02:11:04] /infrawaves/StepMesh/src/./rdma_van.h:206: Check failed: (getaddrinfo(node.hostname.c_str(), std::to_string(node.port).c_str(), nullptr, &remote_addr)) == (0) 

[niehao100]

Hi @Zhangmj0621. We tested stepMesh up to M=8 and N=2 on our testbed, it should work.
It seems like the ROCE ip distribution problem of scheduler at connection initiation.
Please try these method:

1. Set set PS_VERBOSE=2 in script to get more detailed log;
2. Print the info of node.hostname.c_str(), std::to_string(node.port).c_str(), &remote_addr .
3. Check the output of scheduler, you can modify the run_multi_gpu.sh and run scheduler separately to monitor its output.

[Zhangmj0621]

Hi @Zhangmj0621. We tested stepMesh up to M=8 and N=2 on our testbed, it should work. It seems like the ROCE ip distribution problem of scheduler at connection initiation. Please try these method:

1. Set set PS_VERBOSE=2 in script to get more detailed log;
2. Print the info of node.hostname.c_str(), std::to_string(node.port).c_str(), &remote_addr .
3. Check the output of scheduler, you can modify the run_multi_gpu.sh and run scheduler separately to monitor its output.

Thanks for reply. What bothers me is that every four nodes work will, but when it scale to 8 nodes, it crash.

[niehao100]

Hi thanks for your test.
Does 6 worker 2 server work?

[Zhangmj0621]

Hi thanks for your test. Does 6 worker 2 server work?

It also crashes.

[niehao100]

OK, I think it's configuration problem, please attach the detailed log and I'll check it later.

Hi thanks for your test. Does 6 worker 2 server work?

It also crashes.

[Zhangmj0621]

OK, I think it's configuration problem, please attach the detailed log and I'll check it later.

Hi thanks for your test. Does 6 worker 2 server work?

It also crashes.

root@node148:/gpfs/Stepmesh# ROLE=server NUM_SERVER=4 NUM_WORKER=4 NODE_RANK=0 BIN=../benchmark/bmk_comm_latency_multiserver bash tests/fserver/run_multi_gpu.sh 
Run server and scheduler, scheduler ip [10.10.117.20](http://10.10.117.20/) 
[03:09:05] scheduler /infrawaves/StepMesh/src/[postoffice.cc:75](http://postoffice.cc:75/): Creating Van: ibverbs. group_size=8
[03:09:05] scheduler /infrawaves/StepMesh/src/./rdma_van.h:153: bind to DMLC_NODE_HOST: [10.10.117.20](http://10.10.117.20/)
[03:09:05] scheduler /infrawaves/StepMesh/src/[van.cc:600](http://van.cc:600/): Bind to [role=scheduler, id=1, ip=[10.10.117.20](http://10.10.117.20/), port=-1, is_recovery=0, aux_id=-1, num_ports=1]
[03:09:05] scheduler /gpfs/Stepmesh/include/dmlc/logging.h:301: [03:09:05] /infrawaves/StepMesh/src/[van.cc:601](http://van.cc:601/): Check failed: (my_node_.port) != (-1) bind failed

Stack trace returned 10 entries:
[bt] (0) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(+0x53e5f) [0x7f0e3ca50e5f]
[bt] (1) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(+0x54193) [0x7f0e3ca51193]
[bt] (2) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(ps::Van::Start(int, bool)+0xac1) [0x7f0e3ca87d21]
[bt] (3) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(ps::Postoffice::Start(int, ps::Node::Role, int, bool, char const*)+0x2fa) [0x7f0e3ca7e3aa]
[bt] (4) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(+0x4e726) [0x7f0e3ca4b726]
[bt] (5) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(+0x61144) [0x7f0e3ca5e144]
[bt] (6) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(+0x6790b) [0x7f0e3ca6490b]
[bt] (7) python3(+0x15cb2e) [0x55fba35f1b2e]
[bt] (8) python3(_PyObject_MakeTpCall+0x25b) [0x55fba35e82db]
[bt] (9) python3(_PyEval_EvalFrameDefault+0x6b17) [0x55fba35e0d27]


Traceback (most recent call last):
  File "/gpfs/Stepmesh/tests/fserver/../benchmark/bmk_comm_latency_multiserver.py", line 67, in <module>
    f.init()
RuntimeError: [03:09:05] /infrawaves/StepMesh/src/[van.cc:601](http://van.cc:601/): Check failed: (my_node_.port) != (-1) bind failed

Stack trace returned 10 entries:
[bt] (0) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(+0x53e5f) [0x7f0e3ca50e5f]
[bt] (1) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(+0x54193) [0x7f0e3ca51193]
[bt] (2) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(ps::Van::Start(int, bool)+0xac1) [0x7f0e3ca87d21]
[bt] (3) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(ps::Postoffice::Start(int, ps::Node::Role, int, bool, char const*)+0x2fa) [0x7f0e3ca7e3aa]
[bt] (4) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(+0x4e726) [0x7f0e3ca4b726]
[bt] (5) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(+0x61144) [0x7f0e3ca5e144]
[bt] (6) /gpfs/Stepmesh/[fserver_lib.cpython-310-x86_64-linux-gnu.so](http://fserver_lib.cpython-310-x86_64-linux-gnu.so/)(+0x6790b) [0x7f0e3ca6490b]
[bt] (7) python3(+0x15cb2e) [0x55fba35f1b2e]
[bt] (8) python3(_PyObject_MakeTpCall+0x25b) [0x55fba35e82db]
[bt] (9) python3(_PyEval_EvalFrameDefault+0x6b17) [0x55fba35e0d27]


BindToCpuCore: gpu 0 -> cpu 10
[03:09:07] server /infrawaves/StepMesh/src/[postoffice.cc:75](http://postoffice.cc:75/): Creating Van: ibverbs. group_size=8
[03:09:08] server /infrawaves/StepMesh/src/[van.cc:556](http://van.cc:556/): automatic detect interface and ip from gpu: ens108np0 ([10.10.117.20](http://10.10.117.20/))
[03:09:08] server /infrawaves/StepMesh/src/./rdma_van.h:153: bind to DMLC_NODE_HOST: [10.10.117.20](http://10.10.117.20/)
[03:09:08] server /infrawaves/StepMesh/src/[van.cc:600](http://van.cc:600/): Bind to [role=server, ip=[10.10.117.20](http://10.10.117.20/), port=39063, is_recovery=0, aux_id=-1, num_ports=1]
[03:09:08] server /infrawaves/StepMesh/src/./rdma_van.h:178: Connecting to Node 1, My_Node=32767
[03:09:08] server /infrawaves/StepMesh/src/././rdma_transport.h:190: qp created: pd=0x7f518c001030 , cq=0x7f518c001140, qp=23228
[03:09:08] server /infrawaves/StepMesh/src/././rdma_transport.h:190: qp created: pd=0x7f518c001030 , cq=0x7f518c001140, qp=23229
[03:09:08] server /infrawaves/StepMesh/src/./rdma_van.h:1089: 32767 OnConnected to 1
[03:09:08] server /infrawaves/StepMesh/src/./rdma_van.h:1089: 32767 OnConnected to 1
BindToCpuCore: gpu 0 -> cpu 11
BindToCpuCore: gpu 1 -> cpu 15
[03:09:08] server /infrawaves/StepMesh/src/[postoffice.cc:75](http://postoffice.cc:75/): Creating Van: ibverbs. group_size=8
[03:09:08] server /infrawaves/StepMesh/src/[van.cc:556](http://van.cc:556/): automatic detect interface and ip from gpu: ens109np0 ([10.10.118.20](http://10.10.118.20/))
[03:09:08] server /infrawaves/StepMesh/src/./rdma_van.h:153: bind to DMLC_NODE_HOST: [10.10.118.20](http://10.10.118.20/)
[03:09:08] server /infrawaves/StepMesh/src/[van.cc:600](http://van.cc:600/): Bind to [role=server, ip=[10.10.118.20](http://10.10.118.20/), port=34011, is_recovery=0, aux_id=-1, num_ports=1]
[03:09:08] server /infrawaves/StepMesh/src/./rdma_van.h:178: Connecting to Node 1, My_Node=32767
[03:09:08] server /infrawaves/StepMesh/src/././rdma_transport.h:190: qp created: pd=0x7f6410001030 , cq=0x7f6410001140, qp=15669
[03:09:08] server /infrawaves/StepMesh/src/././rdma_transport.h:190: qp created: pd=0x7f6410001030 , cq=0x7f6410001140, qp=15674
[03:09:08] server /infrawaves/StepMesh/src/./rdma_van.h:1089: 32767 OnConnected to 1
[03:09:08] server /infrawaves/StepMesh/src/./rdma_van.h:1089: 32767 OnConnected to 1
[03:09:08] server /infrawaves/StepMesh/src/./././rdma_utils.h:148: Initialized BackendMemoryAllocator for GPU 1 with pd 0x7f6410001030
[03:09:08] server /infrawaves/StepMesh/src/[van.cc:700](http://van.cc:700/): rdma 32767	sent: ? => 1. Meta: request=0, timestamp=0, control={ cmd=ADD_NODE, node={ [role=server, ip=[10.10.118.20](http://10.10.118.20/), port=34011, is_recovery=0, aux_id=1, num_ports=1] } }. NOT DATA MSG!Slave QP Count: 0
BindToCpuCore: gpu BindToCpuCore: gpu 1 -> cpu 171 -> cpu 
16
BindToCpuCore: gpu 3 -> cpu 39
[03:09:08] server /infrawaves/StepMesh/src/[postoffice.cc:75](http://postoffice.cc:75/): Creating Van: ibverbs. group_size=8
[03:09:08] server /infrawaves/StepMesh/src/[van.cc:556](http://van.cc:556/): automatic detect interface and ip from gpu: ens111np0 ([10.10.120.20](http://10.10.120.20/))
[03:09:08] server /infrawaves/StepMesh/src/./rdma_van.h:153: bind to DMLC_NODE_HOST:

[niehao100]

The first error is bind failed, is 10.10.117.20 a ip of one ROCE NIC?

Check failed: (my_node_.port) != (-1) bind failed

[Zhangmj0621]

The first error is bind failed, is 10.10.117.20 a ip of one ROCE NIC?

Check failed: (my_node_.port) != (-1) bind failed

Yes, we have manally set STEPMESH_SPLIT_QP_LAG=0

[niehao100]

Hi @Zhangmj0621, I test the scheduler with NUM_SERVER=4 and NUM_WORKER=4 and didn't reproduce the error.

The errors shown in log seems due to the failure of the following code.

rdma_van.h 162

      if (rdma_bind_addr(listener_,
                         reinterpret_cast<struct sockaddr *>(&addr)) == 0) {
        break;
      }

It seems the problem of mismatch of DMLC_INTERFACE and SCHEDULER_IP. If your server have multiple RNIC, you may change RNIC to another one.

export DMLC_INTERFACE=${RNIC:-brainpf_bond0}
export SCHEDULER_IP=$(ip -o -4 addr | grep ${DMLC_INTERFACE} | awk '{print $4}' | cut -d'/' -f1)

In addition, this example provides a separate way to verify the aboving RDMA_CM api, please verify with it. https://github.com/animeshtrivedi/rdma-example/blob/master/src/rdma_server.c