Repository navigation
[EP 3/9] tpu: split optimizer param groups by device mesh in both TPU trainers - #183
Merged
copybara-service[bot] merged 2 commits intoNov 18, 2026
Merged
Conversation
This was referenced Nov 11, 2026
Merged
tanvibajpai
force-pushed
the
tbajpai/ep-03-split-optimizer-by-mesh
branch
from
November 12, 2026 21:32
954e384 to
be2ed21
Compare
tanvibajpai
force-pushed
the
tbajpai/ep-02-gmm-ragged-rows
branch
from
November 12, 2026 21:32
46df192 to
cc65b62
Compare
jialei777
approved these changes
Nov 17, 2026
tanvibajpai
force-pushed
the
tbajpai/ep-02-gmm-ragged-rows
branch
2 times, most recently
from
November 18, 2026 00:43
0c01c9f to
7e353d3
Compare
Under expert parallelism the routed experts live on an (efsdp, ep) mesh and the dense parameters on (fsdp,), but torch_tpu's fused and foreach AdamW need every param group to share one mesh. Only TpuGraphTrainer split the groups, with a private helper that ran after the base trainer had built the LR schedulers, so it rebuilt them afterwards. The eager TPUTrainer didn't split at all. - utils.py: public split_optimizers_by_mesh(). It replaces a group whose params span several meshes with one group per mesh, keeping the group's options and the matching param_names, so the FQN-keyed optimizer state dict keeps its keys. Meshes compare by equality rather than identity (FSDP2 with TP builds equal but separate mesh objects), non-DTensor params share one group, and a second call changes nothing. It validates every group before changing any, and refuses to split once LR schedulers are attached. with_split_optimizers_by_mesh() chains it after a model's own post_optimizer_build_fn. - train.py, graph_trainer/tpu_graph_trainer.py: both trainers install that hook. The base trainer runs post_optimizer_build_fn right after building the optimizers and before the LR schedulers and checkpointer, so the late split, its LR-scheduler rebuild and the private helper are gone. - tests/test_split_optimizers_by_mesh.py (new): one and two meshes, equal meshes, plain tensors, idempotence, the param_names check, the LR-scheduler guard, optimizer state dicts saved and loaded across a split, and both trainers splitting inside the hook before their LR schedulers are built (the GraphTrainer case with the in-graph optimizer). Nothing changes at EP=1: every group is already on one mesh. Based on #6 (Jialei Chen). Tests (2026-11-09 nightly image), rebased on main after #182: - CPU: test_split_optimizers_by_mesh 10 OK; test_tpu_graph_trainer_config 169 OK. - TPU v6e-8, 8 ranks at EP=1 (the hook runs, nothing splits): test_train_distributed qwen3_fsdp_moe_use_gmm_kernel and qwen3_5_moe_simplefsdp OK; test_tpu_graph_trainer_distributed test_qwen3_5_graph_trainer_matches_eager and gemma4_moe_simple_fsdp OK.
tanvibajpai
force-pushed
the
tbajpai/ep-03-split-optimizer-by-mesh
branch
from
November 18, 2026 05:55
be2ed21 to
ae9a477
Compare
copybara-service
Bot
deleted the
tbajpai/ep-03-split-optimizer-by-mesh
branch
November 18, 2026 20:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Under expert parallelism the routed experts live on an
(efsdp, ep)mesh and the dense parameters on(fsdp,), buttorch_tpu's fused and foreach AdamW need every param group to share one mesh. Today only
TpuGraphTrainersplits thegroups, with a private helper that runs after the base trainer has built the LR schedulers (so it rebuilds them), and
the eager
TPUTrainerdoesn't split at all.This PR adds a shared
split_optimizers_by_mesh()and runs it from the model spec'spost_optimizer_build_fnin bothtrainers. The base trainer calls that hook right after building the optimizers and before the LR schedulers and the
checkpointer, so both see the split groups and nothing has to be rebuilt.
Part of the EP stack, which adds expert parallelism to the shared TPU MoE path.
Changes
torchtitan/experiments/tpu/utils.py:split_optimizers_by_mesh(): replaces a group whose params span several meshes with one group per mesh, keepingthe group's options and matching
param_names(the optimizer state dict is FQN-keyed, so its keys don't change).Meshes compare by equality, not identity (FSDP2 with TP builds equal but separate mesh objects); non-DTensor params
share one group; a second call changes nothing; every group is validated before any is changed; it refuses to
split once LR schedulers are attached.
with_split_optimizers_by_mesh(): chains the split after a model's ownpost_optimizer_build_fn.torchtitan/experiments/tpu/train.py,graph_trainer/tpu_graph_trainer.py: both trainers install the hook; theprivate helper, the late split and the LR-scheduler rebuild are removed.
torchtitan/experiments/tpu/tests/test_split_optimizers_by_mesh.py(new).Testing
Rebased on
mainafter #182, 2026-11-09 nightly image (torch 2.16.0.dev20261108, jax 0.12.0.1):test_split_optimizers_by_mesh10 OK (one and two meshes, equal meshes, plain tensors, idempotence, theparam_namescheck, the LR-scheduler guard, optimizer state dicts saved and loaded across a split, and both trainerssplitting inside the hook before their LR schedulers are built, the GraphTrainer case with the in-graph optimizer);
test_tpu_graph_trainer_config169 OK.test_train_distributed_qwen3_fsdp_moe_use_gmm_kernelandtest_train_distributed_qwen3_5_moe_simplefsdpOK;test_qwen3_5_graph_trainer_matches_eagerandtest_graph_trainer_distributed_gemma4_moe_simple_fsdpOK.Notes
(covered by a test).
train_minimal.pyisn't changed: nothing runsTrainerMinimalwith EP > 1.