Skip to content

[EP 3/9] tpu: split optimizer param groups by device mesh in both TPU trainers - #183

Merged
copybara-service[bot] merged 2 commits into
mainfrom
tbajpai/ep-03-split-optimizer-by-mesh
Nov 18, 2026
Merged

copybara-service[bot] merged 2 commits into
mainfrom
tbajpai/ep-03-split-optimizer-by-mesh

Conversation

@tanvibajpai

@tanvibajpai tanvibajpai commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Under expert parallelism the routed experts live on an (efsdp, ep) mesh and the dense parameters on (fsdp,), but
torch_tpu's fused and foreach AdamW need every param group to share one mesh. Today only TpuGraphTrainer splits the
groups, with a private helper that runs after the base trainer has built the LR schedulers (so it rebuilds them), and
the eager TPUTrainer doesn't split at all.

This PR adds a shared split_optimizers_by_mesh() and runs it from the model spec's post_optimizer_build_fn in both
trainers. The base trainer calls that hook right after building the optimizers and before the LR schedulers and the
checkpointer, so both see the split groups and nothing has to be rebuilt.

Part of the EP stack, which adds expert parallelism to the shared TPU MoE path.

Changes

  • torchtitan/experiments/tpu/utils.py:
    • split_optimizers_by_mesh(): replaces a group whose params span several meshes with one group per mesh, keeping
      the group's options and matching param_names (the optimizer state dict is FQN-keyed, so its keys don't change).
      Meshes compare by equality, not identity (FSDP2 with TP builds equal but separate mesh objects); non-DTensor params
      share one group; a second call changes nothing; every group is validated before any is changed; it refuses to
      split once LR schedulers are attached.
    • with_split_optimizers_by_mesh(): chains the split after a model's own post_optimizer_build_fn.
  • torchtitan/experiments/tpu/train.py, graph_trainer/tpu_graph_trainer.py: both trainers install the hook; the
    private helper, the late split and the LR-scheduler rebuild are removed.
  • torchtitan/experiments/tpu/tests/test_split_optimizers_by_mesh.py (new).

Testing

Rebased on main after #182, 2026-11-09 nightly image (torch 2.16.0.dev20261108, jax 0.12.0.1):

  • CPU: test_split_optimizers_by_mesh 10 OK (one and two meshes, equal meshes, plain tensors, idempotence, the
    param_names check, the LR-scheduler guard, optimizer state dicts saved and loaded across a split, and both trainers
    splitting inside the hook before their LR schedulers are built, the GraphTrainer case with the in-graph optimizer);
    test_tpu_graph_trainer_config 169 OK.
  • TPU v6e-8 (8 ranks, EP = 1, so the hook runs but doesn't split):
    test_train_distributed_qwen3_fsdp_moe_use_gmm_kernel and test_train_distributed_qwen3_5_moe_simplefsdp OK;
    test_qwen3_5_graph_trainer_matches_eager and test_graph_trainer_distributed_gemma4_moe_simple_fsdp OK.

Notes

  • Nothing changes at EP = 1: every group is already on one mesh.
  • Checkpoints: the state dict is keyed by parameter FQN, so saving and loading works the same with or without the split
    (covered by a test).
  • train_minimal.py isn't changed: nothing runs TrainerMinimal with EP > 1.
  • Based on [TPU][Qwen3.5] Expert parallelism (EP > 1) for Qwen3.5 MoE #6 (Jialei Chen).

Comment thread torchtitan/experiments/tpu/train.py Resolved
@tanvibajpai
tanvibajpai force-pushed the tbajpai/ep-02-gmm-ragged-rows branch 2 times, most recently from 0c01c9f to 7e353d3 Compare November 18, 2026 00:43
Base automatically changed from tbajpai/ep-02-gmm-ragged-rows to main November 18, 2026 03:28
Under expert parallelism the routed experts live on an (efsdp, ep) mesh and
the dense parameters on (fsdp,), but torch_tpu's fused and foreach AdamW
need every param group to share one mesh. Only TpuGraphTrainer split the
groups, with a private helper that ran after the base trainer had built the
LR schedulers, so it rebuilt them afterwards. The eager TPUTrainer didn't
split at all.

- utils.py: public split_optimizers_by_mesh(). It replaces a group whose
  params span several meshes with one group per mesh, keeping the group's
  options and the matching param_names, so the FQN-keyed optimizer state dict
  keeps its keys. Meshes compare by equality rather than identity (FSDP2 with
  TP builds equal but separate mesh objects), non-DTensor params share one
  group, and a second call changes nothing. It validates every group before
  changing any, and refuses to split once LR schedulers are attached.
  with_split_optimizers_by_mesh() chains it after a model's own
  post_optimizer_build_fn.
- train.py, graph_trainer/tpu_graph_trainer.py: both trainers install that
  hook. The base trainer runs post_optimizer_build_fn right after building
  the optimizers and before the LR schedulers and checkpointer, so the late
  split, its LR-scheduler rebuild and the private helper are gone.
- tests/test_split_optimizers_by_mesh.py (new): one and two meshes, equal
  meshes, plain tensors, idempotence, the param_names check, the LR-scheduler
  guard, optimizer state dicts saved and loaded across a split, and both
  trainers splitting inside the hook before their LR schedulers are built
  (the GraphTrainer case with the in-graph optimizer).

Nothing changes at EP=1: every group is already on one mesh. Based on #6
(Jialei Chen).

Tests (2026-11-09 nightly image), rebased on main after #182:
- CPU: test_split_optimizers_by_mesh 10 OK; test_tpu_graph_trainer_config
  169 OK.
- TPU v6e-8, 8 ranks at EP=1 (the hook runs, nothing splits):
  test_train_distributed qwen3_fsdp_moe_use_gmm_kernel and
  qwen3_5_moe_simplefsdp OK; test_tpu_graph_trainer_distributed
  test_qwen3_5_graph_trainer_matches_eager and gemma4_moe_simple_fsdp OK.
@tanvibajpai
tanvibajpai force-pushed the tbajpai/ep-03-split-optimizer-by-mesh branch from be2ed21 to ae9a477 Compare November 18, 2026 05:55
@tanvibajpai tanvibajpai added the pull ready Signals Copybara to import this PR into Google3 label Nov 18, 2026
@copybara-service
copybara-service Bot merged commit 540a730 into main Nov 18, 2026
20 of 21 checks passed
@copybara-service
copybara-service Bot deleted the tbajpai/ep-03-split-optimizer-by-mesh branch November 18, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pull ready Signals Copybara to import this PR into Google3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants