[Question] question about the data in the table

[15050188022]

Question

Which GPU is the data in your table based on, and what is the inter-machine RDMA bandwidth?

[15050188022]

add:
in the case of 6 seconds in the table, how long does it take to complete the two steps: 'Unified training-inference weight convert' and 'Global weight metadata calculation and exchange'?
Does weight transfer require allocating memory on the inference engine, or is it directly transferred to the memory location corresponding to the weights of the inference engine?
thanks a lot

[chaokunyang]

GPU is H20. RDMA bandwidth is 400gb.

Global weight metadata calculation and exchange happen only at initialize time. Not happen at exchange time.

And we don't collect metric for convert. Most weight convert are cheap. Only If some weights has different precision or is uncontinuous, then It will take some time.

6 seconds is end to end time.

No memory allocation happen on inference engine, weights are directly sent to inference memory location

[15050188022]

Whether in NCCL mode or RDMA mode, is it necessary to collect weights in the training engine before sending them during each weight update? (Similar to the Verl framework, this may cause a large peak memory usage on the head node.)

[chaokunyang]

No collect. I designed awex as a Zero-Redundancy weights exchange. It will build a auto mapping between training shards and inference shards. No collect/gather. It's pure p2p transfer

[15050188022]

No collect. I designed awex as a Zero-Redundancy weights exchange. It will build a auto mapping between training shards and inference shards. No collect/gather. It's pure p2p transfer没有收藏。我将awex设计为零冗余权重交换。它将在训练分片和推理分片之间建立自动映射。没有收集/聚集。这是纯粹的p2p传输

It seems that there may be some memory allocation during the process of weight conversion (such as the rearrangement of some QKV weights in Megatron format conversion). Does this lead to an increase in peak GPU memory usage during your actual RL (Reinforcement Learning) training?

[chaokunyang]

No, the allocation is small, and most of time it doesn't exceed the activation memory iof training. If it does use more memory, you can offlaod optimizer parameters and reload it back after finish the transfer.

If use RDMA for transfer, the weights convert can be streaminglined and copy to pined cpu memory, so no extra gpu memory overhead