Repository navigation
[EP 5/9] tpu moe: train Qwen3.5 MoE with expert parallelism on the shared ragged transport - #186
Open
tanvibajpai wants to merge 4 commits into
Open
tanvibajpai wants to merge 4 commits into
tanvibajpai wants to merge 4 commits into
Conversation
This was referenced Nov 11, 2026
TpuGroupedExperts padded the routed rows to a multiple of tile_m and added the pad rows to the last expert's count. At EP=1 that is harmless: the pad rows are zeros and are sliced off the output. Under dropless expert parallelism, though, the receive buffer has a static worst-case row count whose rows past sum(num_tokens_per_expert_E) hold stale data. Growing the last expert's group over the padding pulls those rows into that expert's forward and dWeight. The dWeight TGMM kernels also need the row count to be a multiple of their own M tile, which can be larger than tile_m: with tile_m=128 and a 256-row TGMM tile, 300 rows were padded to 384, which that tile doesn't divide. - components/grouped_experts.py: pad the rows (and row_scale_R) to lcm(tile_m, TGMM M tile, W2 TGMM M tile) and pass num_tokens_per_expert_E unchanged. Every kernel skips rows past its sum. - kernels/paired_segment_matmul.py: the paired W1/W3 forward zeroes each output tile on its first visit. Its masked stores never write rows past sum(group_sizes), which kept stale VMEM contents in the partly routed tile once the pad rows stopped belonging to the last group. The docstrings now say group_sizes may sum to less than the row count. EP=1 outputs and gradients are unchanged: the pad rows were zeros, and now they're skipped instead (checked bit-exactly below). Will Constable measured the paired-kernel zero-init as neutral at EP=1 in his review of #142 (15.461 -> 15.490 ms). Based on #6 and #142 (Jialei Chen). Tests: - kernels/tests/test_paired_segment_matmul.py: NaN-filled rows past the routed count (a partly routed tile and an unrouted tile) don't reach the outputs, dInput or dWeight, against the unpaired kernel with zero rows. - kernels/tests/test_expert_segment_matmul.py: the same for the unpaired kernel (empty groups, a TGMM tile larger than tile_m, transpose_rhs), bit exact against zero rows in the last group. - tests/test_grouped_experts_tpu.py (new): 300 unaligned rows, paired and unpaired, with and without row_scale_R, bit exact against zero rows in the last expert. - tests/unit_tests/test_tpu_grouped_experts.py: every kernel gets the counts unchanged and rows padded to every M tile; NaN tail rows don't reach the outputs or gradients (CPU, kernels mocked with torch._grouped_mm). Results on a TPU v6e-8 and CPU, 2026-11-09 nightly image: test_expert_segment_matmul 7 OK, test_paired_segment_matmul 9 OK, test_grouped_experts_tpu 7 OK, transformers_modeling_backend test_pallas_experts_tpu 7 OK; test_tpu_grouped_experts.py 26 passed.
Under expert parallelism the routed experts live on an (efsdp, ep) mesh and the dense parameters on (fsdp,), but torch_tpu's fused and foreach AdamW need every param group to share one mesh. Only TpuGraphTrainer split the groups, with a private helper that ran after the base trainer had built the LR schedulers, so it rebuilt them afterwards. The eager TPUTrainer didn't split at all. - utils.py: public split_optimizers_by_mesh(). It replaces a group whose params span several meshes with one group per mesh, keeping the group's options and the matching param_names, so the FQN-keyed optimizer state dict keeps its keys. Meshes compare by equality rather than identity (FSDP2 with TP builds equal but separate mesh objects), non-DTensor params share one group, and a second call changes nothing. It validates every group before changing any, and refuses to split once LR schedulers are attached. with_split_optimizers_by_mesh() chains it after a model's own post_optimizer_build_fn. - train.py, graph_trainer/tpu_graph_trainer.py: both trainers install that hook. The base trainer runs post_optimizer_build_fn right after building the optimizers and before the LR schedulers and checkpointer, so the late split, its LR-scheduler rebuild and the private helper are gone. - tests/test_split_optimizers_by_mesh.py (new): one and two meshes, equal meshes, plain tensors, idempotence, the param_names check, the LR-scheduler guard, optimizer state dicts saved and loaded across a split, and both trainers splitting inside the hook before their LR schedulers are built (the GraphTrainer case with the in-graph optimizer). Nothing changes at EP=1: every group is already on one mesh. Based on #6 (Jialei Chen). Tests (2026-11-09 nightly image): - CPU: test_split_optimizers_by_mesh 10 OK; test_tpu_graph_trainer_config 169 OK. - TPU v6e-8, 8 ranks at EP=1 (the hook runs, nothing splits): test_train_distributed qwen3_fsdp_moe_use_gmm_kernel and qwen3_5_moe_simplefsdp OK; test_tpu_graph_trainer_distributed test_qwen3_5_graph_trainer_matches_eager and gemma4_moe_simple_fsdp OK.
… parallelism
Under EP > 1 the shared TPU dispatcher falls back to upstream's
AllToAllTokenDispatcher, whose variable-size all-to-all copies the token
splits to the host before exchanging tokens. That can't be traced by
TpuGraphTrainer. This adds a traceable, dropless transport to the shared
dispatcher, so any model using TpuTokenDispatcherConverter and
TpuRoutedExpertsConverter can opt in.
- components/token_dispatcher.py:
- TpuAllToAllTokenDispatcher.Config.ep_transport: "all_to_all" (default,
upstream's path, unchanged) or "ragged_all_to_all".
- ragged_all_to_all: all-gather the per-expert token counts, derive the
exchange offsets on device, and move exactly the routed rows with
torch.tpu.ops.ragged_all_to_all into a static expert-major receive buffer
of ep_size * T * top_k rows. No host sync and no dropped tokens,
whatever the routing skew. The combine sends the rows back with the
symmetric exchange and then uses the same fixed-K reduction as EP=1.
- TpuRaggedDispatchMetadata holds the offsets. It doesn't subclass
LocalDispatchMetadata, so TpuRoutedExperts' EP=1-only guard rejects it.
- The exchange is an autograd.Function with precomputed backward offsets,
which keeps it traceable. On TPU it requires a named process group; off
TPU a slow eager reference built from all-gathers runs CPU tests, and
tracing off TPU raises rather than returning zeros.
- TpuRoutedExperts.parallelize registers a non-persistent one-element
buffer holding the local EP rank and binds the dispatcher to it, so every
rank traces the same program (a Python int would bake each rank's index
into its graph). Dispatch raises if no owner is bound.
- tests/test_ragged_ep_dispatcher.py (new): EP=2 and EP=4 on CPU (gloo) with
random, skewed and empty-expert routing match the EP=1 path on the whole
batch, outputs and gradients, with NaN in the receive-buffer rows past the
routed count; plus config validation, tracing, the missing-owner error and
when parallelize registers the rank buffer.
- tests/test_token_dispatcher.py: EP=1 keeps the TPU fixed-K reduction with
either transport; the ragged metadata isn't LocalDispatchMetadata.
Memory: the receive buffer is sized for the worst case, ep_size * T * top_k
rows per rank. Bounded capacity, Ring-of-Experts and chunked overlap are
follow-ups. No recipe enables the new transport yet; Qwen3.5 does in a later
change. Based on #6 (Jialei Chen).
Tests: CPU (2026-11-09 nightly image): test_ragged_ep_dispatcher.py and
test_token_dispatcher.py pass (counts in the PR description).
…ed transport Qwen3.5 MoE recipes can now run with --parallelism.expert_parallel_degree > 1 on TPU, in both the TPU trainer and TpuGraphTrainer, on the shared pieces from the earlier PRs in this stack: the ragged all-to-all transport, the GMM tail handling and the optimizer split by mesh. Based on #6 (Jialei Chen), but on the shared dispatcher instead of a Qwen3.5-specific one. - components/expert_parallel.py (new): - use_expert_parallel(model_config, ep_degree=...): a no-op at EP=1. Under EP > 1 it turns plain RoutedExperts configs into TpuRoutedExperts (whose parallelize binds the EP rank buffer the ragged transport needs), turns the all-to-all dispatchers into TpuAllToAllTokenDispatcher with ep_transport="ragged_all_to_all", and turns off the paired Pallas GMM, which TpuGroupedExperts allows only at EP=1. - apply_expert_parallel_module_overrides(model, parallel_dims): under EP > 1 on TPU, switches tile-aligned experts that would use torch._grouped_mm to unpaired Pallas GMMs, because its backward reads uninitialized memory on empty expert segments (torch_tpu#2668). A no-op at EP=1. - qwen3_5/model.py, qwen3_5/__init__.py: TpuQwen35Model. Its Config.update_from_config calls use_expert_parallel and its parallelize calls the module overrides; every TPU Qwen3.5 config now builds it. At EP=1 the config update is identical to upstream's (tested), and state-dict keys don't change (the rank buffer is non-persistent). - qwen3_5/infra/parallelize.py: apply_qwen3_5_simple_fsdp wraps each MoE's routed experts on the expert data-parallel mesh, then applies SimpleFSDP to the rest. Upstream's apply_simple_fsdp does that only for TransformerBlock, which Qwen3.5's block isn't. - graph_trainer/qwen3_5.py: TpuGraphTrainer accepts EP for Qwen3.5 with SimpleFSDP (TP, CP and PP are still rejected), calls model.parallelize under EP and uses apply_qwen3_5_simple_fsdp. - tpu_job_config.py: the compile_grad_norm error now says to pass --compile.no-compile_grad_norm with EP > 1 or PP > 1; the eager grad norm handles the dense and expert meshes. - tests: - qwen3_5/tests/test_ep_parity_distributed.py (new): 8 TPU chips at EP=1/2/4/8 against a single-device CPU reference (loss, gradients and parameters over 3 steps) with the real ragged_all_to_all. - tests/test_train_distributed.py: Qwen3.5 MoE at EP=2 with SimpleFSDP and with the Pallas MoE kernels (TPU only). - graph_trainer/tests/test_tpu_graph_trainer_distributed.py: TpuGraphTrainer at EP=2 matches EP=1 at step 1. - qwen3_5/tests/test_qwen3_5_tpu.py: at EP=1 the config update matches upstream's; at EP=2 it switches to TpuRoutedExperts, the ragged transport and unpaired GMMs; the graph trainer rejects EP without SimpleFSDP. Not covered: the paired Pallas GMM under EP (it stays off, as in #6), the compiled grad norm over two meshes, scan + EP, and EP recipes (later changes). Tests (2026-11-09 nightly image, v6e-8): CPU: test_qwen3_5_tpu 46 OK (4 skipped), test_token_dispatcher.py 17 passed, test_ragged_ep_dispatcher.py 11 passed, test_tpu_grouped_experts.py 26 passed, test_split_optimizers_by_mesh 10 OK, test_tpu_graph_trainer_config 169 OK. TPU: test_train_distributed Qwen3.5 MoE at EP=1 and EP=2, each with SimpleFSDP and with the Pallas MoE kernels, 4 OK; test_ep_parity_distributed 5 OK; TpuGraphTrainer EP=2 vs EP=1 OK (step-1 loss 12.912108 vs 12.911852); TpuGraphTrainer vs eager OK.
tanvibajpai
force-pushed
the
tbajpai/ep-05-qwen35-ep-training
branch
from
November 12, 2026 21:32
a008185 to
ddada10
Compare
tanvibajpai
force-pushed
the
tbajpai/ep-04-ragged-a2a-transport
branch
from
November 12, 2026 21:32
022632f to
cac54a4
Compare
This was referenced Nov 17, 2026
tanvibajpai
force-pushed
the
tbajpai/ep-04-ragged-a2a-transport
branch
from
November 18, 2026 06:28
cac54a4 to
8ef6546
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Qwen3.5 MoE can now train with
--parallelism.expert_parallel_degree > 1on TPU, in both the TPU trainer andTpuGraphTrainer, on the shared pieces from this stack: the ragged all-to-all transport (#184), the GMM tail handling (#182) and the optimizer split by mesh (#183). Based on #6 (Jialei Chen), but on the shared dispatcher instead of a Qwen3.5-specific one.Changes
components/expert_parallel.py(new):use_expert_parallel(model_config, ep_degree=...): a no-op at EP = 1. Under EP > 1 it turns plainRoutedExpertsconfigs intoTpuRoutedExperts(whoseparallelizebinds the EP rank buffer the ragged transport needs), turns the all-to-all dispatchers intoTpuAllToAllTokenDispatcherwithep_transport="ragged_all_to_all", and turns off the paired Pallas GMM, whichTpuGroupedExpertsallows only at EP = 1.apply_expert_parallel_module_overrides(model, parallel_dims): under EP > 1 on TPU, switches tile-aligned experts that would usetorch._grouped_mmto unpaired Pallas GMMs, because its backward reads uninitialized memory on empty expert segments (torch_tpu#2668). A no-op at EP = 1.qwen3_5/model.py,qwen3_5/__init__.py:TpuQwen35Model. ItsConfig.update_from_configcallsuse_expert_paralleland itsparallelizecalls the module overrides; every TPU Qwen3.5 config now builds it. At EP = 1 the config update is identical to upstream's (tested), and state-dict keys don't change (the rank buffer is non-persistent).qwen3_5/infra/parallelize.py:apply_qwen3_5_simple_fsdpwraps each MoE's routed experts on the expert data-parallel mesh, then applies SimpleFSDP to the rest. Upstream'sapply_simple_fsdpdoes that only forTransformerBlock, which Qwen3.5's block isn't.graph_trainer/qwen3_5.py:TpuGraphTraineraccepts EP for Qwen3.5 with SimpleFSDP (TP, CP and PP are still rejected), callsmodel.parallelizeunder EP and usesapply_qwen3_5_simple_fsdp.tpu_job_config.py: thecompile_grad_normerror now says to pass--compile.no-compile_grad_normwith EP > 1 or PP > 1; the eager grad norm handles the dense and expert meshes.qwen3_5/tests/test_ep_parity_distributed.py(new): 8 TPU chips at EP = 1/2/4/8 against a single-device CPU reference (loss, gradients and parameters over 3 steps), with the realragged_all_to_all.tests/test_train_distributed.py: Qwen3.5 MoE at EP = 2 with SimpleFSDP and with the Pallas MoE kernels (TPU only).graph_trainer/tests/test_tpu_graph_trainer_distributed.py:TpuGraphTrainerat EP = 2 matches EP = 1 at step 1.qwen3_5/tests/test_qwen3_5_tpu.py: at EP = 1 the config update matches upstream's; at EP = 2 it switches toTpuRoutedExperts, the ragged transport and unpaired GMMs; the graph trainer rejects EP without SimpleFSDP.Testing
On a v6e-8 with the 2026-11-09 nightly image (torch 2.16.0.dev20261108, jax 0.12.0.1):
test_qwen3_5_tputest_token_dispatcher.py,test_ragged_ep_dispatcher.py,tests/unit_tests/test_tpu_grouped_experts.pytest_split_optimizers_by_mesh,test_tpu_graph_trainer_configtest_train_distributed: Qwen3.5 MoE at EP = 1 and EP = 2, each with SimpleFSDP and with the Pallas MoE kernelstest_ep_parity_distributed: EP = 1/2/4/8 vs a single-device CPU referenceTpuGraphTrainerEP = 2 vs EP = 1TpuGraphTrainervs the eager trainer (EP = 1)The new tests are picked up by
.ci/discover_tpu_tests.pyautomatically; the parity test takes about 5 minutes on a v6e-8.After rebasing the stack onto
mainat9c0aa673(wherespmd_typesbecame the default backend), re-run on #186's branch: every test in the table passes again.Notes
compile_grad_normstill requires EP = 1; EP runs use the eager grad norm.autotransport selection, token chunking to overlap EP communication, scan + EP, and Qwen3.5 EP recipes for the 35B and 397B models.