NVIDIA-NeMo Automodel Q A · Discussions · GitHub

Challenges in Enabling PyTorch Native Pipeline Parallelism for Hugging Face Transformer Models
Announcements nemo-automation-bot[bot] bot
DGX Spark + NeMo: Model Training Power in the Palm of Your Hand
Announcements bernardwin
Accelerating Large-Scale Mixture-of-Experts Training in PyTorch with NeMo Automodel
Announcements bernardwin
Implementing large scale MoE training with native Pytorch in Nemo Automodel
Announcements nemo-automation-bot[bot] bot

Q&A Discussions

Ask the community for help

You must be logged in to vote

Why is loss scaled by dp group size before calling backward ?

sappenn asked Feb 27, 2026 in Q&A · Answered

1