Different results in DDP and ZeRO-1 when training 4B model

[AiRyunn]

Hi, thank you for your work on this project!

I was training the 4B model with the provided script with 8×H100 GPUs (1 for vLLM server, 7 for training), but I got significantly lower results with DDP than with Zero-1. Zero-1 matches reported results, but DDP is much worse.

Looking at the configs, the main differences seem to be max_grad_norm and learning rate & scheduler type.

What settings should I use for DDP training to match Zero-1 performance?

[yflyzhang]

Thank you for your feedback and for trying out the project on your 8×H100 GPU setup!

In my setup, DDP and Zero-1 produce similar results, with eval/HuggingFaceH4/MATH-500/rewards/accuracy_reward around 0.8. I'm not sure why the discrepancy occurs, but I feel it could be due to differences in GPU architecture, which might affect precision or related optimization behavior.

If you find high train/grad_norm values (e.g., frequently above 5.0) during the training, this may indicate instability. Try reducing max_grad_norm (e.g., 0.5) first to stabilize the training.

If simply reduce max_grad_norm is not enough, you may need to further adjust learning_rate and lr_scheduler_type.

[AiRyunn]

Thanks for the quick reply. I wasn't sure where the model sync happens in DDP, so I tried the following:

context = (
    functools.partial(self.accelerator.no_sync, model=model)
    if self.accelerator.distributed_type != DistributedType.DEEPSPEED
    and not do_sync_step
    else contextlib.nullcontext
)

Using the same config as Zero-1 and a higher learning rate, DDP now reaches similar accuracy to Zero-1. Could it be possible that the earlier performance drop was caused by gradients not being synced during training?