Skip to content

[EP 2/9] tpu moe: let the Pallas expert GMMs skip rows past the routed count - #182

Merged
copybara-service[bot] merged 2 commits into
mainfrom
tbajpai/ep-02-gmm-ragged-rows
Nov 18, 2026
Merged

copybara-service[bot] merged 2 commits into
mainfrom
tbajpai/ep-02-gmm-ragged-rows

Conversation

@tanvibajpai

@tanvibajpai tanvibajpai commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

TpuGroupedExperts padded the routed rows to a multiple of tile_m and added the pad rows to the last expert's count.
That's harmless at EP = 1, where the pad rows are zeros and get sliced off. Under dropless expert parallelism the
receive buffer has a static worst-case row count, and its rows past sum(num_tokens_per_expert_E) hold stale data:
growing the last expert's group over the padding pulls them into that expert's forward and dWeight. The dWeight TGMM
kernels also need the row count to be a multiple of their own M tile, which can be larger than tile_m.

This PR pads to every kernel's M tile and leaves the counts alone; the kernels already skip rows past the routed count,
except the paired W1/W3 forward, which now zeroes each output tile on its first visit. M tiles whose lcm is more than
twice the largest are refused instead of padded that far (from Jialei's review).

Part of the EP stack, which adds expert parallelism to the shared TPU MoE path.

Changes

  • torchtitan/experiments/tpu/components/grouped_experts.py: pad the rows (and row_scale_R) to
    lcm(tile_m, TGMM M tile, W2 TGMM M tile); pass num_tokens_per_expert_E unchanged. Raise a ValueError before any
    kernel call when that lcm is more than twice the largest M tile, e.g. lcm(384, 512) = 1536, instead of padding that
    far with zero rows. Every recipe's tiles divide each other, so no recipe is affected.
  • torchtitan/experiments/tpu/kernels/paired_segment_matmul.py: zero each forward output tile on its first visit
    (its masked stores never write rows past sum(group_sizes)); docstrings say group_sizes may sum to less than the
    row count.
  • Tests:
    • kernels/tests/test_paired_segment_matmul.py: NaN rows past the routed count (a partly routed tile and an
      unrouted tile) don't reach outputs, dInput or dWeight.
    • kernels/tests/test_expert_segment_matmul.py: the same property for the unpaired kernel (empty groups, a TGMM tile
      larger than tile_m, transpose_rhs), bit exact against zero rows in the last group.
    • tests/test_grouped_experts_row_padding_tpu.py (new): 300 unaligned rows, paired and unpaired, with and without
      row_scale_R, bit exact against the old "zero rows in the last expert" padding.
    • tests/test_grouped_experts.py: kernels get the counts unchanged and rows padded to every M tile; NaN tail rows
      reach neither outputs nor gradients; tiles that divide each other pad to the largest, lcm(256, 384) = 768 is
      allowed, lcm(384, 512) = 1536 raises (CPU, kernels mocked with torch._grouped_mm).

Testing

Rebased on main after #181, 2026-11-09 nightly image (torch 2.16.0.dev20261108, jax 0.12.0.1):

  • TPU v6e-8: test_expert_segment_matmul 7 OK, test_paired_segment_matmul 9 OK,
    test_grouped_experts_row_padding_tpu 7 OK, transformers_modeling_backend.tests.test_pallas_experts_tpu 7 OK.
  • CPU: tests/test_grouped_experts.py 29 passed (9 new).
  • Negative control, the row-padding tests on main's code before the rebase: the paired NaN-tail kernel test fails
    (stale rows in the partly routed tile); the four TpuGroupedExperts cases with a 256-row TGMM tile error out (300
    rows padded to 384); the 6 new CPU cases fail (the last expert's count is changed). The 128-row-tile cases pass on
    main too, as expected, since the old padding was already correct there.

Notes

Base automatically changed from tbajpai/ep-01-row-scale-ep-guard to main November 17, 2026 21:10
@tanvibajpai
tanvibajpai force-pushed the tbajpai/ep-02-gmm-ragged-rows branch from cc65b62 to 0c01c9f Compare November 18, 2026 00:19
TpuGroupedExperts padded the routed rows to a multiple of tile_m and added
the pad rows to the last expert's count. At EP=1 that is harmless: the pad
rows are zeros and are sliced off the output. Under dropless expert
parallelism, though, the receive buffer has a static worst-case row count
whose rows past sum(num_tokens_per_expert_E) hold stale data. Growing the
last expert's group over the padding pulls those rows into that expert's
forward and dWeight. The dWeight TGMM kernels also need the row count to be
a multiple of their own M tile, which can be larger than tile_m: with
tile_m=128 and a 256-row TGMM tile, 300 rows were padded to 384, which that
tile doesn't divide.

- components/grouped_experts.py: pad the rows (and row_scale_R) to
  lcm(tile_m, TGMM M tile, W2 TGMM M tile) and pass num_tokens_per_expert_E
  unchanged. Every kernel skips rows past its sum.
- kernels/paired_segment_matmul.py: the paired W1/W3 forward zeroes each
  output tile on its first visit. Its masked stores never write rows past
  sum(group_sizes), which kept stale VMEM contents in the partly routed tile
  once the pad rows stopped belonging to the last group. The docstrings now
  say group_sizes may sum to less than the row count.

EP=1 outputs and gradients are unchanged: the pad rows were zeros, and now
they're skipped instead (checked bit-exactly below). Will Constable measured
the paired-kernel zero-init as neutral at EP=1 in his review of #142
(15.461 -> 15.490 ms). Based on #6 and #142 (Jialei Chen).

Tests:
- kernels/tests/test_paired_segment_matmul.py: NaN-filled rows past the
  routed count (a partly routed tile and an unrouted tile) don't reach the
  outputs, dInput or dWeight, against the unpaired kernel with zero rows.
- kernels/tests/test_expert_segment_matmul.py: the same for the unpaired
  kernel (empty groups, a TGMM tile larger than tile_m, transpose_rhs), bit
  exact against zero rows in the last group.
- tests/test_grouped_experts_row_padding_tpu.py (new): 300 unaligned
  rows, paired and unpaired, with and without row_scale_R, bit exact
  against zero rows in the last expert. Not test_grouped_experts_tpu.py:
  the internal test build drops the _tpu suffix to name test targets, so
  that name would collide with test_grouped_experts.py.
- tests/test_grouped_experts.py: every kernel gets the counts unchanged
  and rows padded to every M tile; NaN tail rows don't reach the outputs
  or gradients (CPU, kernels mocked with torch._grouped_mm).

Results on a TPU v6e-8 and CPU, 2026-11-09 nightly image, rebased on main
after #181 together with the next commit: test_expert_segment_matmul 7 OK,
test_paired_segment_matmul 9 OK, test_grouped_experts_row_padding_tpu 7 OK,
transformers_modeling_backend test_pallas_experts_tpu 7 OK;
test_grouped_experts.py 29 passed (6 new cases here, 3 in the next commit).
TpuGroupedExperts pads the rows to the lcm of tile_m and the two dWeight M
tiles. M tiles that divide each other, like the power-of-two tiles the
recipes use, need only the largest tile. Others can need a large multiple
of it, all zero rows: lcm(384, 512) = 1536. The experts now raise a
ValueError when the lcm is more than twice the largest M tile instead of
padding that far. The check runs on the configured tile sizes before any
kernel call. Every recipe's tiles divide each other, so no recipe is
affected. Jialei Chen raised this in review.

Tests: tests/test_grouped_experts.py: tiles that divide each other pad to
the largest (512), lcm(256, 384) = 768 is still allowed, and
lcm(384, 512) = 1536 raises before any kernel call (CPU, kernels mocked
with torch._grouped_mm). Results in the 2026-11-09 nightly image, rebased
on main after #181: test_grouped_experts.py 29 passed (CPU);
test_grouped_experts_row_padding_tpu 7 OK (TPU v6e-8).
@tanvibajpai
tanvibajpai force-pushed the tbajpai/ep-02-gmm-ragged-rows branch from 0c01c9f to 7e353d3 Compare November 18, 2026 00:43
@copybara-service
copybara-service Bot merged commit 55ea7e4 into main Nov 18, 2026
20 of 21 checks passed
@copybara-service
copybara-service Bot deleted the tbajpai/ep-02-gmm-ragged-rows branch November 18, 2026 03:28
tanvibajpai added a commit that referenced this pull request Nov 18, 2026
Under expert parallelism the routed experts live on an (efsdp, ep) mesh and
the dense parameters on (fsdp,), but torch_tpu's fused and foreach AdamW
need every param group to share one mesh. Only TpuGraphTrainer split the
groups, with a private helper that ran after the base trainer had built the
LR schedulers, so it rebuilt them afterwards. The eager TPUTrainer didn't
split at all.

- utils.py: public split_optimizers_by_mesh(). It replaces a group whose
  params span several meshes with one group per mesh, keeping the group's
  options and the matching param_names, so the FQN-keyed optimizer state dict
  keeps its keys. Meshes compare by equality rather than identity (FSDP2 with
  TP builds equal but separate mesh objects), non-DTensor params share one
  group, and a second call changes nothing. It validates every group before
  changing any, and refuses to split once LR schedulers are attached.
  with_split_optimizers_by_mesh() chains it after a model's own
  post_optimizer_build_fn.
- train.py, graph_trainer/tpu_graph_trainer.py: both trainers install that
  hook. The base trainer runs post_optimizer_build_fn right after building
  the optimizers and before the LR schedulers and checkpointer, so the late
  split, its LR-scheduler rebuild and the private helper are gone.
- tests/test_split_optimizers_by_mesh.py (new): one and two meshes, equal
  meshes, plain tensors, idempotence, the param_names check, the LR-scheduler
  guard, optimizer state dicts saved and loaded across a split, and both
  trainers splitting inside the hook before their LR schedulers are built
  (the GraphTrainer case with the in-graph optimizer).

Nothing changes at EP=1: every group is already on one mesh. Based on #6
(Jialei Chen).

Tests (2026-11-09 nightly image), rebased on main after #182:
- CPU: test_split_optimizers_by_mesh 10 OK; test_tpu_graph_trainer_config
  169 OK.
- TPU v6e-8, 8 ranks at EP=1 (the hook runs, nothing splits):
  test_train_distributed qwen3_fsdp_moe_use_gmm_kernel and
  qwen3_5_moe_simplefsdp OK; test_tpu_graph_trainer_distributed
  test_qwen3_5_graph_trainer_matches_eager and gemma4_moe_simple_fsdp OK.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pull ready Signals Copybara to import this PR into Google3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants