Training on non nvlink communication multi GPU will get stuck?

[lohoz]

[INFO|grpo_trainer.py:1207] 2025-06-08 13:56:47,073 >> 
  [eval] global_step = 0
    completion length = [4822, 784, 2450, 373, 890, 1205, 610, 1241, 1664, 4645, 6123, 1266]
    accuracy reward   = [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
[rank0]:[E608 14:09:10.210136079 ProcessGroupNCCL.cpp:1484] [PG ID 0 PG GUID 0(default_pg) Rank 0] ProcessGroupNCCL's watchdog got stuck for 480 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API (e.g., CudaEventDestroy) hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api (for example, CudaEventDestroy), or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. 
[rank1]:[E608 14:09:10.268696153 ProcessGroupNCCL.cpp:1484] [PG ID 0 PG GUID 0(default_pg) Rank 1] ProcessGroupNCCL's watchdog got stuck for 480 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API (e.g., CudaEventDestroy) hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api (for example, CudaEventDestroy), or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. 
[rank2]:[E608 14:09:10.333263278 ProcessGroupNCCL.cpp:1484] [PG ID 0 PG GUID 0(default_pg) Rank 2] ProcessGroupNCCL's watchdog got stuck for 480 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API (e.g., CudaEventDestroy) hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api (for example, CudaEventDestroy), or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang. 
[rank0]:[F608 14:17:10.210908118 ProcessGroupNCCL.cpp:1306] [PG ID 0 PG GUID 0(default_pg) Rank 0] [PG ID 0 PG GUID 0(default_pg) Rank 0] Terminating the process after attempting to dump debug info, due to ProcessGroupNCCL watchdog hang.
[rank1]:[F608 14:17:10.269056642 ProcessGroupNCCL.cpp:1306] [PG ID 0 PG GUID 0(default_pg) Rank 1] [PG ID 0 PG GUID 0(default_pg) Rank 1] Terminating the process after attempting to dump debug info, due to ProcessGroupNCCL watchdog hang.
[rank2]:[F608 14:17:10.333598461 ProcessGroupNCCL.cpp:1306] [PG ID 0 PG GUID 0(default_pg) Rank 2] [PG ID 0 PG GUID 0(default_pg) Rank 2] Terminating the process after attempting to dump debug info, due to ProcessGroupNCCL watchdog hang.

The GPU topology is as follows：

        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV4     PXB     PXB     SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU1    NV4      X      PXB     PXB     SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU2    PXB     PXB      X      NV4     SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU3    PXB     PXB     NV4      X      SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU4    SYS     SYS     SYS     SYS      X      NV4     PXB     PXB     32-63,96-127    1               N/A
GPU5    SYS     SYS     SYS     SYS     NV4      X      PXB     PXB     32-63,96-127    1               N/A
GPU6    SYS     SYS     SYS     SYS     PXB     PXB      X      NV4     32-63,96-127    1               N/A
GPU7    SYS     SYS     SYS     SYS     PXB     PXB     NV4      X      32-63,96-127    1               N/A

[yflyzhang]

Thank you very much for raising this issue.
There was a bug in the gather/all_gather operation in function '_log_completions'. I’ve made a quick fix, and after preliminary testing, DDP should work with multiple devices now. ZeRO-2/3 configurations have not been tested yet.

Please try the latest version of the code (mainly src/grpo_trainer.py), and feel free to report any further issues.

[lohoz]

Thank you very much for raising this issue. There was a bug in the gather/all_gather operation in function '_log_completions'. I’ve made a quick fix, and after preliminary testing, DDP should work with multiple devices now. ZeRO-2/3 configurations have not been tested yet.

Please try the latest version of the code (mainly src/grpo_trainer.py), and feel free to report any further issues.

Thank you for your reply. After the fix, it can now run.