vccl doesn't get benefit in megatron training

[dmndxld]

Hello! Thanks for the nice work. I've tested nccl-tests with vccl, and got good results.

So I used vccl in my v0.13 megatron following the doc.

But I didnt see any improvements in my training. Could you tell me what I did wrong and how to fix it?

I use slurm to train my model.
16nodes with TP=4 and PP = 4.

[leoda1]

Hi, did you add the change ./VCCL/asset/Megatron-change.patch to your Megatron?

[dmndxld]

Hi, did you add the change ./VCCL/asset/Megatron-change.patch to your Megatron?

Thanks for the quick reply!

Yes, I patched to my megatron by git apply.

And I do belive it works.

[leoda1]

You can try using the Megatron environment variables --account-for-embedding-in-pipeline-split and --account-for-loss-in-pipeline-split to put these layers into the PP stage. Accordingly, the NLAYERS value in your configuration script (e.g., Megatron/config/xxx.sh) should be reduced by 2.
This's beneficial cause SM-Free send-recv primarily eliminates SM resource competition between PP communication and computation. And, it is recommended to use the PyTorch Profiler to examine the activity on both the PP communication stream and the default stream. If the default stream's kernels are not computationally intensive, the SM resources saved from the PP stage may not yield a noticeable performance gain.