Skip to content

[EP 1/9] tpu moe: reject scale_before_down_proj under expert parallelism - #181

Merged
copybara-service[bot] merged 1 commit into
mainfrom
tbajpai/ep-01-row-scale-ep-guard
Nov 17, 2026
Merged

copybara-service[bot] merged 1 commit into
mainfrom
tbajpai/ep-01-row-scale-ep-guard

Conversation

@tanvibajpai

@tanvibajpai tanvibajpai commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

TpuRoutedExperts.scale_before_down_proj multiplies each routed row by its routing score before W2 and then reduces
the rows by token locally. That is only correct for EP = 1 local dispatch: under EP > 1 the scores stay on the source
rank while W2 runs on the expert's rank, and the combine needs the return all-to-all.

The forward already tried to refuse non-local dispatch, but it checked isinstance(metadata, LocalDispatchMetadata),
and upstream's AllToAllDispatchMetadata subclasses LocalDispatchMetadata. With
--parallelism.expert_parallel_degree > 1 on a recipe that sets scale_before_down_proj (the Qwen3 FSDP-only 30B
and 235B recipes), the check passed, and the forward scaled and reduced the received rows with the source rank's
scores and token indices, skipping the return all-to-all. That's a shape error or silently wrong results.

This is the first PR of the EP stack, which adds expert parallelism to the shared TPU MoE path.

Changes

  • torchtitan/experiments/tpu/components/token_dispatcher.py:
    • TpuRoutedExperts.parallelize raises when an EP mesh exists and scale_before_down_proj is set, so the
      combination fails when the model is parallelized, before any step.
    • The forward check also rejects AllToAllDispatchMetadata explicitly.
  • torchtitan/experiments/tpu/tests/test_token_dispatcher.py: tests for the parallelize guard (with and without an
    EP mesh) and for the forward guard with AllToAllDispatchMetadata.

Testing

CPU, inside the 2026-11-09 nightly image (torch 2.16.0.dev20261108, jax 0.12.0.1), at 77b9b293:

  • python -m pytest torchtitan/experiments/tpu/tests/test_token_dispatcher.py: 14 passed.
  • python -m torchtitan.experiments.tpu.graph_trainer.tests.test_tpu_graph_trainer_config: 169 tests OK (builds the
    Qwen3 recipes, including the ones that set scale_before_down_proj).

After rebasing the stack onto main at 9c0aa673 (where spmd_types became the default backend), re-run on #186's branch: test_token_dispatcher.py 17 passed (it now includes #184's cases) and test_tpu_graph_trainer_config 169 OK.

Notes

  • No recipe enables both today, so this only changes a CLI override that produced wrong results.
  • Supporting row scaling under EP (sending the scores along with the tokens) is left for a later change; the plan is
    one EP-safe score-placement option shared with the planned Ring-of-Experts transport.

TpuRoutedExperts.scale_before_down_proj multiplies each routed row by its
routing score before W2, then reduces the rows by token locally. That only
works with EP=1 local dispatch: under EP > 1 the scores stay on the source
rank while W2 runs on the expert's rank, and the combine needs the return
all-to-all.

The forward already refused non-local dispatch metadata, but it checked
isinstance(metadata, LocalDispatchMetadata), and upstream's
AllToAllDispatchMetadata subclasses LocalDispatchMetadata. So with
--parallelism.expert_parallel_degree > 1 on a recipe that sets
scale_before_down_proj (the Qwen3 FSDP-only 30B and 235B recipes), the check
passed: the forward scaled and reduced the received rows with the source
rank's scores and token indices and skipped the return all-to-all, which
gives a shape error or wrong results.

- components/token_dispatcher.py: TpuRoutedExperts.parallelize raises when an
  EP mesh exists and scale_before_down_proj is set, so the combination fails
  when the model is parallelized rather than in the first step. The forward
  check also rejects AllToAllDispatchMetadata explicitly.
- tests/test_token_dispatcher.py: the parallelize guard with and without an
  EP mesh, and the forward guard with AllToAllDispatchMetadata.

No recipe enables both today, so this only changes a CLI override that
produced wrong results. Supporting row scaling under EP (sending the scores
along with the tokens) is left for a later change.

Tests: torchtitan/experiments/tpu/tests/test_token_dispatcher.py: 14 passed
(CPU, in the 2026-11-09 nightly image).
@tanvibajpai
tanvibajpai force-pushed the tbajpai/ep-01-row-scale-ep-guard branch from d32fd8b to 0ff96f8 Compare November 12, 2026 21:32
@tanvibajpai tanvibajpai added the pull ready Signals Copybara to import this PR into Google3 label Nov 16, 2026
@tanvibajpai tanvibajpai changed the title tpu moe: reject scale_before_down_proj under expert parallelism [EP 1/9] tpu moe: reject scale_before_down_proj under expert parallelism Nov 17, 2026
@copybara-service
copybara-service Bot merged commit 6136182 into main Nov 17, 2026
21 checks passed
@copybara-service
copybara-service Bot deleted the tbajpai/ep-01-row-scale-ep-guard branch November 17, 2026 21:10
tanvibajpai added a commit that referenced this pull request Nov 18, 2026
TpuGroupedExperts padded the routed rows to a multiple of tile_m and added
the pad rows to the last expert's count. At EP=1 that is harmless: the pad
rows are zeros and are sliced off the output. Under dropless expert
parallelism, though, the receive buffer has a static worst-case row count
whose rows past sum(num_tokens_per_expert_E) hold stale data. Growing the
last expert's group over the padding pulls those rows into that expert's
forward and dWeight. The dWeight TGMM kernels also need the row count to be
a multiple of their own M tile, which can be larger than tile_m: with
tile_m=128 and a 256-row TGMM tile, 300 rows were padded to 384, which that
tile doesn't divide.

- components/grouped_experts.py: pad the rows (and row_scale_R) to
  lcm(tile_m, TGMM M tile, W2 TGMM M tile) and pass num_tokens_per_expert_E
  unchanged. Every kernel skips rows past its sum.
- kernels/paired_segment_matmul.py: the paired W1/W3 forward zeroes each
  output tile on its first visit. Its masked stores never write rows past
  sum(group_sizes), which kept stale VMEM contents in the partly routed tile
  once the pad rows stopped belonging to the last group. The docstrings now
  say group_sizes may sum to less than the row count.

EP=1 outputs and gradients are unchanged: the pad rows were zeros, and now
they're skipped instead (checked bit-exactly below). Will Constable measured
the paired-kernel zero-init as neutral at EP=1 in his review of #142
(15.461 -> 15.490 ms). Based on #6 and #142 (Jialei Chen).

Tests:
- kernels/tests/test_paired_segment_matmul.py: NaN-filled rows past the
  routed count (a partly routed tile and an unrouted tile) don't reach the
  outputs, dInput or dWeight, against the unpaired kernel with zero rows.
- kernels/tests/test_expert_segment_matmul.py: the same for the unpaired
  kernel (empty groups, a TGMM tile larger than tile_m, transpose_rhs), bit
  exact against zero rows in the last group.
- tests/test_grouped_experts_tpu.py (new): 300 unaligned rows, paired and
  unpaired, with and without row_scale_R, bit exact against zero rows in the
  last expert.
- tests/test_grouped_experts.py: every kernel gets the counts
  unchanged and rows padded to every M tile; NaN tail rows don't reach the
  outputs or gradients (CPU, kernels mocked with torch._grouped_mm).
Results on a TPU v6e-8 and CPU, 2026-11-09 nightly image, rebased on main
after #181 together with the next commit: test_expert_segment_matmul 7 OK,
test_paired_segment_matmul 9 OK, test_grouped_experts_tpu 7 OK,
transformers_modeling_backend test_pallas_experts_tpu 7 OK;
test_grouped_experts.py 29 passed (26 of them from this change).
tanvibajpai added a commit that referenced this pull request Nov 18, 2026
TpuGroupedExperts pads the rows to the lcm of tile_m and the two dWeight M
tiles. M tiles that divide each other, like the power-of-two tiles the
recipes use, need only the largest tile. Others can need a large multiple
of it, all zero rows: lcm(384, 512) = 1536. The experts now raise a
ValueError when the lcm is more than twice the largest M tile instead of
padding that far. The check runs on the configured tile sizes before any
kernel call. Every recipe's tiles divide each other, so no recipe is
affected. Jialei Chen raised this in review.

Tests: tests/test_grouped_experts.py: tiles that divide
each other pad to the largest (512), lcm(256, 384) = 768 is still allowed,
and lcm(384, 512) = 1536 raises before any kernel call (CPU, kernels mocked
with torch._grouped_mm). Results in the 2026-11-09 nightly image, rebased on main after #181:
test_grouped_experts.py 29 passed (CPU); test_grouped_experts_tpu 7 OK
(TPU v6e-8).
tanvibajpai added a commit that referenced this pull request Nov 18, 2026
TpuGroupedExperts padded the routed rows to a multiple of tile_m and added
the pad rows to the last expert's count. At EP=1 that is harmless: the pad
rows are zeros and are sliced off the output. Under dropless expert
parallelism, though, the receive buffer has a static worst-case row count
whose rows past sum(num_tokens_per_expert_E) hold stale data. Growing the
last expert's group over the padding pulls those rows into that expert's
forward and dWeight. The dWeight TGMM kernels also need the row count to be
a multiple of their own M tile, which can be larger than tile_m: with
tile_m=128 and a 256-row TGMM tile, 300 rows were padded to 384, which that
tile doesn't divide.

- components/grouped_experts.py: pad the rows (and row_scale_R) to
  lcm(tile_m, TGMM M tile, W2 TGMM M tile) and pass num_tokens_per_expert_E
  unchanged. Every kernel skips rows past its sum.
- kernels/paired_segment_matmul.py: the paired W1/W3 forward zeroes each
  output tile on its first visit. Its masked stores never write rows past
  sum(group_sizes), which kept stale VMEM contents in the partly routed tile
  once the pad rows stopped belonging to the last group. The docstrings now
  say group_sizes may sum to less than the row count.

EP=1 outputs and gradients are unchanged: the pad rows were zeros, and now
they're skipped instead (checked bit-exactly below). Will Constable measured
the paired-kernel zero-init as neutral at EP=1 in his review of #142
(15.461 -> 15.490 ms). Based on #6 and #142 (Jialei Chen).

Tests:
- kernels/tests/test_paired_segment_matmul.py: NaN-filled rows past the
  routed count (a partly routed tile and an unrouted tile) don't reach the
  outputs, dInput or dWeight, against the unpaired kernel with zero rows.
- kernels/tests/test_expert_segment_matmul.py: the same for the unpaired
  kernel (empty groups, a TGMM tile larger than tile_m, transpose_rhs), bit
  exact against zero rows in the last group.
- tests/test_grouped_experts_row_padding_tpu.py (new): 300 unaligned
  rows, paired and unpaired, with and without row_scale_R, bit exact
  against zero rows in the last expert. Not test_grouped_experts_tpu.py:
  the internal test build drops the _tpu suffix to name test targets, so
  that name would collide with test_grouped_experts.py.
- tests/test_grouped_experts.py: every kernel gets the counts unchanged
  and rows padded to every M tile; NaN tail rows don't reach the outputs
  or gradients (CPU, kernels mocked with torch._grouped_mm).

Results on a TPU v6e-8 and CPU, 2026-11-09 nightly image, rebased on main
after #181 together with the next commit: test_expert_segment_matmul 7 OK,
test_paired_segment_matmul 9 OK, test_grouped_experts_row_padding_tpu 7 OK,
transformers_modeling_backend test_pallas_experts_tpu 7 OK;
test_grouped_experts.py 29 passed (6 new cases here, 3 in the next commit).
tanvibajpai added a commit that referenced this pull request Nov 18, 2026
TpuGroupedExperts pads the rows to the lcm of tile_m and the two dWeight M
tiles. M tiles that divide each other, like the power-of-two tiles the
recipes use, need only the largest tile. Others can need a large multiple
of it, all zero rows: lcm(384, 512) = 1536. The experts now raise a
ValueError when the lcm is more than twice the largest M tile instead of
padding that far. The check runs on the configured tile sizes before any
kernel call. Every recipe's tiles divide each other, so no recipe is
affected. Jialei Chen raised this in review.

Tests: tests/test_grouped_experts.py: tiles that divide each other pad to
the largest (512), lcm(256, 384) = 768 is still allowed, and
lcm(384, 512) = 1536 raises before any kernel call (CPU, kernels mocked
with torch._grouped_mm). Results in the 2026-11-09 nightly image, rebased
on main after #181: test_grouped_experts.py 29 passed (CPU);
test_grouped_experts_row_padding_tpu 7 OK (TPU v6e-8).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pull ready Signals Copybara to import this PR into Google3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants