Repository navigation
[EP 2/9] tpu moe: let the Pallas expert GMMs skip rows past the routed count - #182
Merged
Merged
Conversation
This was referenced Nov 11, 2026
Merged
tanvibajpai
force-pushed
the
tbajpai/ep-02-gmm-ragged-rows
branch
from
November 12, 2026 21:32
46df192 to
cc65b62
Compare
tanvibajpai
force-pushed
the
tbajpai/ep-01-row-scale-ep-guard
branch
from
November 12, 2026 21:32
d32fd8b to
0ff96f8
Compare
jialei777
approved these changes
Nov 17, 2026
tanvibajpai
force-pushed
the
tbajpai/ep-02-gmm-ragged-rows
branch
from
November 18, 2026 00:19
cc65b62 to
0c01c9f
Compare
TpuGroupedExperts padded the routed rows to a multiple of tile_m and added the pad rows to the last expert's count. At EP=1 that is harmless: the pad rows are zeros and are sliced off the output. Under dropless expert parallelism, though, the receive buffer has a static worst-case row count whose rows past sum(num_tokens_per_expert_E) hold stale data. Growing the last expert's group over the padding pulls those rows into that expert's forward and dWeight. The dWeight TGMM kernels also need the row count to be a multiple of their own M tile, which can be larger than tile_m: with tile_m=128 and a 256-row TGMM tile, 300 rows were padded to 384, which that tile doesn't divide. - components/grouped_experts.py: pad the rows (and row_scale_R) to lcm(tile_m, TGMM M tile, W2 TGMM M tile) and pass num_tokens_per_expert_E unchanged. Every kernel skips rows past its sum. - kernels/paired_segment_matmul.py: the paired W1/W3 forward zeroes each output tile on its first visit. Its masked stores never write rows past sum(group_sizes), which kept stale VMEM contents in the partly routed tile once the pad rows stopped belonging to the last group. The docstrings now say group_sizes may sum to less than the row count. EP=1 outputs and gradients are unchanged: the pad rows were zeros, and now they're skipped instead (checked bit-exactly below). Will Constable measured the paired-kernel zero-init as neutral at EP=1 in his review of #142 (15.461 -> 15.490 ms). Based on #6 and #142 (Jialei Chen). Tests: - kernels/tests/test_paired_segment_matmul.py: NaN-filled rows past the routed count (a partly routed tile and an unrouted tile) don't reach the outputs, dInput or dWeight, against the unpaired kernel with zero rows. - kernels/tests/test_expert_segment_matmul.py: the same for the unpaired kernel (empty groups, a TGMM tile larger than tile_m, transpose_rhs), bit exact against zero rows in the last group. - tests/test_grouped_experts_row_padding_tpu.py (new): 300 unaligned rows, paired and unpaired, with and without row_scale_R, bit exact against zero rows in the last expert. Not test_grouped_experts_tpu.py: the internal test build drops the _tpu suffix to name test targets, so that name would collide with test_grouped_experts.py. - tests/test_grouped_experts.py: every kernel gets the counts unchanged and rows padded to every M tile; NaN tail rows don't reach the outputs or gradients (CPU, kernels mocked with torch._grouped_mm). Results on a TPU v6e-8 and CPU, 2026-11-09 nightly image, rebased on main after #181 together with the next commit: test_expert_segment_matmul 7 OK, test_paired_segment_matmul 9 OK, test_grouped_experts_row_padding_tpu 7 OK, transformers_modeling_backend test_pallas_experts_tpu 7 OK; test_grouped_experts.py 29 passed (6 new cases here, 3 in the next commit).
TpuGroupedExperts pads the rows to the lcm of tile_m and the two dWeight M tiles. M tiles that divide each other, like the power-of-two tiles the recipes use, need only the largest tile. Others can need a large multiple of it, all zero rows: lcm(384, 512) = 1536. The experts now raise a ValueError when the lcm is more than twice the largest M tile instead of padding that far. The check runs on the configured tile sizes before any kernel call. Every recipe's tiles divide each other, so no recipe is affected. Jialei Chen raised this in review. Tests: tests/test_grouped_experts.py: tiles that divide each other pad to the largest (512), lcm(256, 384) = 768 is still allowed, and lcm(384, 512) = 1536 raises before any kernel call (CPU, kernels mocked with torch._grouped_mm). Results in the 2026-11-09 nightly image, rebased on main after #181: test_grouped_experts.py 29 passed (CPU); test_grouped_experts_row_padding_tpu 7 OK (TPU v6e-8).
tanvibajpai
force-pushed
the
tbajpai/ep-02-gmm-ragged-rows
branch
from
November 18, 2026 00:43
0c01c9f to
7e353d3
Compare
tanvibajpai
added a commit
that referenced
this pull request
Nov 18, 2026
Under expert parallelism the routed experts live on an (efsdp, ep) mesh and the dense parameters on (fsdp,), but torch_tpu's fused and foreach AdamW need every param group to share one mesh. Only TpuGraphTrainer split the groups, with a private helper that ran after the base trainer had built the LR schedulers, so it rebuilt them afterwards. The eager TPUTrainer didn't split at all. - utils.py: public split_optimizers_by_mesh(). It replaces a group whose params span several meshes with one group per mesh, keeping the group's options and the matching param_names, so the FQN-keyed optimizer state dict keeps its keys. Meshes compare by equality rather than identity (FSDP2 with TP builds equal but separate mesh objects), non-DTensor params share one group, and a second call changes nothing. It validates every group before changing any, and refuses to split once LR schedulers are attached. with_split_optimizers_by_mesh() chains it after a model's own post_optimizer_build_fn. - train.py, graph_trainer/tpu_graph_trainer.py: both trainers install that hook. The base trainer runs post_optimizer_build_fn right after building the optimizers and before the LR schedulers and checkpointer, so the late split, its LR-scheduler rebuild and the private helper are gone. - tests/test_split_optimizers_by_mesh.py (new): one and two meshes, equal meshes, plain tensors, idempotence, the param_names check, the LR-scheduler guard, optimizer state dicts saved and loaded across a split, and both trainers splitting inside the hook before their LR schedulers are built (the GraphTrainer case with the in-graph optimizer). Nothing changes at EP=1: every group is already on one mesh. Based on #6 (Jialei Chen). Tests (2026-11-09 nightly image), rebased on main after #182: - CPU: test_split_optimizers_by_mesh 10 OK; test_tpu_graph_trainer_config 169 OK. - TPU v6e-8, 8 ranks at EP=1 (the hook runs, nothing splits): test_train_distributed qwen3_fsdp_moe_use_gmm_kernel and qwen3_5_moe_simplefsdp OK; test_tpu_graph_trainer_distributed test_qwen3_5_graph_trainer_matches_eager and gemma4_moe_simple_fsdp OK.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TpuGroupedExpertspadded the routed rows to a multiple oftile_mand added the pad rows to the last expert's count.That's harmless at EP = 1, where the pad rows are zeros and get sliced off. Under dropless expert parallelism the
receive buffer has a static worst-case row count, and its rows past
sum(num_tokens_per_expert_E)hold stale data:growing the last expert's group over the padding pulls them into that expert's forward and dWeight. The dWeight TGMM
kernels also need the row count to be a multiple of their own M tile, which can be larger than
tile_m.This PR pads to every kernel's M tile and leaves the counts alone; the kernels already skip rows past the routed count,
except the paired W1/W3 forward, which now zeroes each output tile on its first visit. M tiles whose lcm is more than
twice the largest are refused instead of padded that far (from Jialei's review).
Part of the EP stack, which adds expert parallelism to the shared TPU MoE path.
Changes
torchtitan/experiments/tpu/components/grouped_experts.py: pad the rows (androw_scale_R) tolcm(tile_m, TGMM M tile, W2 TGMM M tile); passnum_tokens_per_expert_Eunchanged. Raise aValueErrorbefore anykernel call when that lcm is more than twice the largest M tile, e.g. lcm(384, 512) = 1536, instead of padding that
far with zero rows. Every recipe's tiles divide each other, so no recipe is affected.
torchtitan/experiments/tpu/kernels/paired_segment_matmul.py: zero each forward output tile on its first visit(its masked stores never write rows past
sum(group_sizes)); docstrings saygroup_sizesmay sum to less than therow count.
kernels/tests/test_paired_segment_matmul.py: NaN rows past the routed count (a partly routed tile and anunrouted tile) don't reach outputs, dInput or dWeight.
kernels/tests/test_expert_segment_matmul.py: the same property for the unpaired kernel (empty groups, a TGMM tilelarger than
tile_m,transpose_rhs), bit exact against zero rows in the last group.tests/test_grouped_experts_row_padding_tpu.py(new): 300 unaligned rows, paired and unpaired, with and withoutrow_scale_R, bit exact against the old "zero rows in the last expert" padding.tests/test_grouped_experts.py: kernels get the counts unchanged and rows padded to every M tile; NaN tail rowsreach neither outputs nor gradients; tiles that divide each other pad to the largest, lcm(256, 384) = 768 is
allowed, lcm(384, 512) = 1536 raises (CPU, kernels mocked with
torch._grouped_mm).Testing
Rebased on
mainafter #181, 2026-11-09 nightly image (torch 2.16.0.dev20261108, jax 0.12.0.1):test_expert_segment_matmul7 OK,test_paired_segment_matmul9 OK,test_grouped_experts_row_padding_tpu7 OK,transformers_modeling_backend.tests.test_pallas_experts_tpu7 OK.tests/test_grouped_experts.py29 passed (9 new).main's code before the rebase: the paired NaN-tail kernel test fails(stale rows in the partly routed tile); the four
TpuGroupedExpertscases with a 256-row TGMM tile error out (300rows padded to 384); the 6 new CPU cases fail (the last expert's count is changed). The 128-row-tile cases pass on
maintoo, as expected, since the old padding was already correct there.Notes
zero-init as neutral at EP = 1 in his review of [tpu][qwen3_5] Fast dropless EP: Ring-of-Experts, Pallas ragged gather/reduce, chunked pipelining #142 (15.461 → 15.490 ms).
test_grouped_experts_tpu.pybecause the internal test build drops the_tpusuffixto name test targets, so it would collide with
test_grouped_experts.py.paired_segment_matmul.py: tpu moe: pipeline the routed-token gathers against the expert matmuls #148 and Q235: untile the W2 dWeight K axis #160–tpu: fuse Qwen3 QK-norm and RoPE elementwise passes into Pallas kernels #166. Whichever lands second rebases.