Repository navigation
[EP 1/9] tpu moe: reject scale_before_down_proj under expert parallelism - #181
Merged
Merged
Conversation
This was referenced Nov 11, 2026
Merged
TpuRoutedExperts.scale_before_down_proj multiplies each routed row by its routing score before W2, then reduces the rows by token locally. That only works with EP=1 local dispatch: under EP > 1 the scores stay on the source rank while W2 runs on the expert's rank, and the combine needs the return all-to-all. The forward already refused non-local dispatch metadata, but it checked isinstance(metadata, LocalDispatchMetadata), and upstream's AllToAllDispatchMetadata subclasses LocalDispatchMetadata. So with --parallelism.expert_parallel_degree > 1 on a recipe that sets scale_before_down_proj (the Qwen3 FSDP-only 30B and 235B recipes), the check passed: the forward scaled and reduced the received rows with the source rank's scores and token indices and skipped the return all-to-all, which gives a shape error or wrong results. - components/token_dispatcher.py: TpuRoutedExperts.parallelize raises when an EP mesh exists and scale_before_down_proj is set, so the combination fails when the model is parallelized rather than in the first step. The forward check also rejects AllToAllDispatchMetadata explicitly. - tests/test_token_dispatcher.py: the parallelize guard with and without an EP mesh, and the forward guard with AllToAllDispatchMetadata. No recipe enables both today, so this only changes a CLI override that produced wrong results. Supporting row scaling under EP (sending the scores along with the tokens) is left for a later change. Tests: torchtitan/experiments/tpu/tests/test_token_dispatcher.py: 14 passed (CPU, in the 2026-11-09 nightly image).
tanvibajpai
force-pushed
the
tbajpai/ep-01-row-scale-ep-guard
branch
from
November 12, 2026 21:32
d32fd8b to
0ff96f8
Compare
jialei777
approved these changes
Nov 17, 2026
tanvibajpai
added a commit
that referenced
this pull request
Nov 18, 2026
TpuGroupedExperts padded the routed rows to a multiple of tile_m and added the pad rows to the last expert's count. At EP=1 that is harmless: the pad rows are zeros and are sliced off the output. Under dropless expert parallelism, though, the receive buffer has a static worst-case row count whose rows past sum(num_tokens_per_expert_E) hold stale data. Growing the last expert's group over the padding pulls those rows into that expert's forward and dWeight. The dWeight TGMM kernels also need the row count to be a multiple of their own M tile, which can be larger than tile_m: with tile_m=128 and a 256-row TGMM tile, 300 rows were padded to 384, which that tile doesn't divide. - components/grouped_experts.py: pad the rows (and row_scale_R) to lcm(tile_m, TGMM M tile, W2 TGMM M tile) and pass num_tokens_per_expert_E unchanged. Every kernel skips rows past its sum. - kernels/paired_segment_matmul.py: the paired W1/W3 forward zeroes each output tile on its first visit. Its masked stores never write rows past sum(group_sizes), which kept stale VMEM contents in the partly routed tile once the pad rows stopped belonging to the last group. The docstrings now say group_sizes may sum to less than the row count. EP=1 outputs and gradients are unchanged: the pad rows were zeros, and now they're skipped instead (checked bit-exactly below). Will Constable measured the paired-kernel zero-init as neutral at EP=1 in his review of #142 (15.461 -> 15.490 ms). Based on #6 and #142 (Jialei Chen). Tests: - kernels/tests/test_paired_segment_matmul.py: NaN-filled rows past the routed count (a partly routed tile and an unrouted tile) don't reach the outputs, dInput or dWeight, against the unpaired kernel with zero rows. - kernels/tests/test_expert_segment_matmul.py: the same for the unpaired kernel (empty groups, a TGMM tile larger than tile_m, transpose_rhs), bit exact against zero rows in the last group. - tests/test_grouped_experts_tpu.py (new): 300 unaligned rows, paired and unpaired, with and without row_scale_R, bit exact against zero rows in the last expert. - tests/test_grouped_experts.py: every kernel gets the counts unchanged and rows padded to every M tile; NaN tail rows don't reach the outputs or gradients (CPU, kernels mocked with torch._grouped_mm). Results on a TPU v6e-8 and CPU, 2026-11-09 nightly image, rebased on main after #181 together with the next commit: test_expert_segment_matmul 7 OK, test_paired_segment_matmul 9 OK, test_grouped_experts_tpu 7 OK, transformers_modeling_backend test_pallas_experts_tpu 7 OK; test_grouped_experts.py 29 passed (26 of them from this change).
tanvibajpai
added a commit
that referenced
this pull request
Nov 18, 2026
TpuGroupedExperts pads the rows to the lcm of tile_m and the two dWeight M tiles. M tiles that divide each other, like the power-of-two tiles the recipes use, need only the largest tile. Others can need a large multiple of it, all zero rows: lcm(384, 512) = 1536. The experts now raise a ValueError when the lcm is more than twice the largest M tile instead of padding that far. The check runs on the configured tile sizes before any kernel call. Every recipe's tiles divide each other, so no recipe is affected. Jialei Chen raised this in review. Tests: tests/test_grouped_experts.py: tiles that divide each other pad to the largest (512), lcm(256, 384) = 768 is still allowed, and lcm(384, 512) = 1536 raises before any kernel call (CPU, kernels mocked with torch._grouped_mm). Results in the 2026-11-09 nightly image, rebased on main after #181: test_grouped_experts.py 29 passed (CPU); test_grouped_experts_tpu 7 OK (TPU v6e-8).
tanvibajpai
added a commit
that referenced
this pull request
Nov 18, 2026
TpuGroupedExperts padded the routed rows to a multiple of tile_m and added the pad rows to the last expert's count. At EP=1 that is harmless: the pad rows are zeros and are sliced off the output. Under dropless expert parallelism, though, the receive buffer has a static worst-case row count whose rows past sum(num_tokens_per_expert_E) hold stale data. Growing the last expert's group over the padding pulls those rows into that expert's forward and dWeight. The dWeight TGMM kernels also need the row count to be a multiple of their own M tile, which can be larger than tile_m: with tile_m=128 and a 256-row TGMM tile, 300 rows were padded to 384, which that tile doesn't divide. - components/grouped_experts.py: pad the rows (and row_scale_R) to lcm(tile_m, TGMM M tile, W2 TGMM M tile) and pass num_tokens_per_expert_E unchanged. Every kernel skips rows past its sum. - kernels/paired_segment_matmul.py: the paired W1/W3 forward zeroes each output tile on its first visit. Its masked stores never write rows past sum(group_sizes), which kept stale VMEM contents in the partly routed tile once the pad rows stopped belonging to the last group. The docstrings now say group_sizes may sum to less than the row count. EP=1 outputs and gradients are unchanged: the pad rows were zeros, and now they're skipped instead (checked bit-exactly below). Will Constable measured the paired-kernel zero-init as neutral at EP=1 in his review of #142 (15.461 -> 15.490 ms). Based on #6 and #142 (Jialei Chen). Tests: - kernels/tests/test_paired_segment_matmul.py: NaN-filled rows past the routed count (a partly routed tile and an unrouted tile) don't reach the outputs, dInput or dWeight, against the unpaired kernel with zero rows. - kernels/tests/test_expert_segment_matmul.py: the same for the unpaired kernel (empty groups, a TGMM tile larger than tile_m, transpose_rhs), bit exact against zero rows in the last group. - tests/test_grouped_experts_row_padding_tpu.py (new): 300 unaligned rows, paired and unpaired, with and without row_scale_R, bit exact against zero rows in the last expert. Not test_grouped_experts_tpu.py: the internal test build drops the _tpu suffix to name test targets, so that name would collide with test_grouped_experts.py. - tests/test_grouped_experts.py: every kernel gets the counts unchanged and rows padded to every M tile; NaN tail rows don't reach the outputs or gradients (CPU, kernels mocked with torch._grouped_mm). Results on a TPU v6e-8 and CPU, 2026-11-09 nightly image, rebased on main after #181 together with the next commit: test_expert_segment_matmul 7 OK, test_paired_segment_matmul 9 OK, test_grouped_experts_row_padding_tpu 7 OK, transformers_modeling_backend test_pallas_experts_tpu 7 OK; test_grouped_experts.py 29 passed (6 new cases here, 3 in the next commit).
tanvibajpai
added a commit
that referenced
this pull request
Nov 18, 2026
TpuGroupedExperts pads the rows to the lcm of tile_m and the two dWeight M tiles. M tiles that divide each other, like the power-of-two tiles the recipes use, need only the largest tile. Others can need a large multiple of it, all zero rows: lcm(384, 512) = 1536. The experts now raise a ValueError when the lcm is more than twice the largest M tile instead of padding that far. The check runs on the configured tile sizes before any kernel call. Every recipe's tiles divide each other, so no recipe is affected. Jialei Chen raised this in review. Tests: tests/test_grouped_experts.py: tiles that divide each other pad to the largest (512), lcm(256, 384) = 768 is still allowed, and lcm(384, 512) = 1536 raises before any kernel call (CPU, kernels mocked with torch._grouped_mm). Results in the 2026-11-09 nightly image, rebased on main after #181: test_grouped_experts.py 29 passed (CPU); test_grouped_experts_row_padding_tpu 7 OK (TPU v6e-8).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TpuRoutedExperts.scale_before_down_projmultiplies each routed row by its routing score before W2 and then reducesthe rows by token locally. That is only correct for EP = 1 local dispatch: under EP > 1 the scores stay on the source
rank while W2 runs on the expert's rank, and the combine needs the return all-to-all.
The forward already tried to refuse non-local dispatch, but it checked
isinstance(metadata, LocalDispatchMetadata),and upstream's
AllToAllDispatchMetadatasubclassesLocalDispatchMetadata. With--parallelism.expert_parallel_degree > 1on a recipe that setsscale_before_down_proj(the Qwen3 FSDP-only 30Band 235B recipes), the check passed, and the forward scaled and reduced the received rows with the source rank's
scores and token indices, skipping the return all-to-all. That's a shape error or silently wrong results.
This is the first PR of the EP stack, which adds expert parallelism to the shared TPU MoE path.
Changes
torchtitan/experiments/tpu/components/token_dispatcher.py:TpuRoutedExperts.parallelizeraises when an EP mesh exists andscale_before_down_projis set, so thecombination fails when the model is parallelized, before any step.
AllToAllDispatchMetadataexplicitly.torchtitan/experiments/tpu/tests/test_token_dispatcher.py: tests for the parallelize guard (with and without anEP mesh) and for the forward guard with
AllToAllDispatchMetadata.Testing
CPU, inside the 2026-11-09 nightly image (torch 2.16.0.dev20261108, jax 0.12.0.1), at
77b9b293:python -m pytest torchtitan/experiments/tpu/tests/test_token_dispatcher.py: 14 passed.python -m torchtitan.experiments.tpu.graph_trainer.tests.test_tpu_graph_trainer_config: 169 tests OK (builds theQwen3 recipes, including the ones that set
scale_before_down_proj).After rebasing the stack onto
mainat9c0aa673(wherespmd_typesbecame the default backend), re-run on #186's branch:test_token_dispatcher.py17 passed (it now includes #184's cases) andtest_tpu_graph_trainer_config169 OK.Notes
one EP-safe score-placement option shared with the planned Ring-of-Experts transport.