ORBIT-2: EDM weak-scaling sweep (HSDP) + validated 1-8 node results

[ashwinma]

Summary

Enables and documents a weak-scaling sweep of the Bayes-CAST EDM model for ORBIT-2 across 1/2/4/8 MI355X nodes, keeping each GPU compute-saturated at every node count.

• run_scaling_study.sh — now honors ORBIT2_BATCH_SIZE / ORBIT2_MAX_EPOCH / ORBIT2_MAX_BATCHES overrides (previously hardcoded), and sets HSDP parallelism per node count: fsdp=8 within a node (XGMI) + simple_ddp=N across nodes (IB/ANP), instead of the render default fsdp=N/simple_ddp=8 which would shard weights across the slow inter-node fabric. This keeps per-rank work fixed for true weak scaling. Adds the EDM invocation and passes ORBIT2_ROOT / ORBIT2_ERA5_SPATIAL_RES / ORBIT2_FUSED_ATTN through to sbatch.
• recipes/train/README.md — adds the validated MI355X EDM weak-scaling table (bf16, per-rank batch 1024, ERA5 1.0°). Clean scaling is demonstrated through 2 nodes (efficiency ~1.01, flat step time = well-overlapped RCCL/IB comm); the 4/8-node efficiencies (>1.0) are explicitly flagged as a data-cap measurement artifact (steps/epoch collapse, so throughput = global_batch/step_time inflates), which motivates staging more ERA5.
• .cursor/skills/ai4science-earth-science/SKILL.md — captures the weak-vs-strong + HSDP layout lesson and the ">1 efficiency" pitfall so future sessions check steady_batch_count before quoting multi-node throughput.

Notes

• No site-specific content committed: only $AI4S_SHARED_DIR/env vars and placeholder partition/account; absolute cluster paths, account, and partition names were scrubbed from the recipe. Run artifacts (scaling_study.*) stay on shared storage, not in the repo.
• Terminology: this is documented as weak scaling (fixed per-rank batch); the collate "efficiency" is t_1node/t_Nnode, a comm-overlap/RCCL-bandwidth metric.

Test plan

• 1/2/4/8-node EDM sweep completes (jobs COMPLETED, loss sanity pass on all four)
• Confirm 1->2 node step time stays flat (weak-scaling baseline)
• Follow-up (tabled): stage more ERA5 so 4/8-node runs yield enough steps/epoch for trustworthy throughput