Repository navigation
[EP 7/9] tpu moe: Ring-of-Experts EP transport, auto selection, and EP row scaling - #202
Open
tanvibajpai wants to merge 2 commits into
Open
tanvibajpai wants to merge 2 commits into
tanvibajpai wants to merge 2 commits into
Conversation
Adds kernels/ragged_gather_tc.py: MaxText's TensorCore ragged gather and ragged gather-reduce kernels (from its MLPerf Training TPU7x MoE submissions), bridged to PyTorch with @pallas.jax_op, plus the custom ops that the Ring-of-Experts (RoE) EP transport calls. Nothing in torchtitan uses the module yet; the RoE transport, later in the EP stack, is the first caller. - packed_stable_sort: stable expert order from one single-key uint32 sort when the key and position bits fit in 32 bits; a two-operand stable sort otherwise. - ragged_gather_tc: out[i] = x[indices[start + i]] for i < count into a static [cap, ...] buffer. The Pallas grid is cdiv(count, block_size), so HBM traffic scales with the routed count rather than with cap. - ragged_gather_reduce_tc: sums each token's gathered rows in FP32, in a deterministic order, optionally weighted by FP32 routing scores, and never reads rows >= count. - _ragged_scale_and_dscore_tc: backward kernel that scales the routed gradient by the routing weights and computes the FP32 dScore in one pass, masking rows >= count. - tt_moe::roe_* custom ops with autograd behind the public entry points roe_compute_routing, roe_sort_gather, roe_unsort_combine, roe_unsort_reduce and roe_gather_row_scales. The hidden size must be a multiple of 128. The VMEM limit is picked per TPU generation: 64 MiB on TPU7x, 32 MiB on v6e, 16 MiB otherwise. LICENSE lists the file as derived from MaxText. The kernel file comes unchanged from #142 (Jialei Chen); the test file only has its imports updated for the TPU directory reorg (#159). Jack Oh's drafts #110 and #107 port the same MaxText kernels; this module is functionally a superset. Tests (kernels/tests/test_ragged_gather_tc.py, new): - CPU and TPU: TPU generation parsing; packed and wide-key sorts against numpy's stable argsort; layout and empty-metadata edge cases. - TPU only, against a pure-PyTorch FP32 reference: routing metadata, gather and weighted / unweighted gather-reduce with their backwards over >= 6 gather steps and >= 8 reduce blocks (4-buffer wrap) with a partial last block; NaN rows past the routed count never reach outputs or gradients; the fused scale + dScore backward on full and partial blocks; the row-scale gather and its collision-free backward; a make_fx fake-tensor trace. Results on a TPU v6e-8 and CPU, 2026-11-09 nightly image: test_ragged_gather_tc "Ran 6 tests ... OK" on TPU; on CPU 6 tests OK with 3 skipped (TPU only).
…ling
Adds the dropless, static-shape Ring-of-Experts (RoE) expert-parallel
transport and `ep_transport="auto"` to the shared TPU dispatcher, and extends
`TpuRoutedExperts.scale_before_down_proj` to work under EP > 1 across both TPU
transports instead of introducing a separate RoE-only flag.
- components/token_dispatcher.py:
- `TpuAllToAllTokenDispatcher.Config.ep_transport`: adds `"ring_of_experts"`
and `"auto"` (`"ring_of_experts"` when `ep_size <= top_k`, otherwise
`"ragged_all_to_all"`).
- `ring_of_experts`: all-gathers `(ids, x, scores)` over `ep_mesh` via
`torch.tpu.ops.all_gather_dim`, computes global expert routing and gathers
this rank's `count` local-expert rows on TensorCore (`roe_compute_routing`,
`roe_sort_gather` from #189), reduces per-token partial outputs in FP32
(`roe_unsort_combine` or unweighted `roe_unsort_reduce`), and sums partials
across `ep_mesh` with `torch.tpu.ops.reduce_scatter_dim`. Off TPU, runs a
pure-PyTorch functional-collective reference for CPU gloo tests; tracing or
missing process groups raise instead of silently repeating shards.
- `TpuRoEDispatchMetadata` and `TpuRaggedDispatchMetadata`: carry explicit
`valid_count`, `row_scale_R` and `row_scale_applied`, without subclassing
`LocalDispatchMetadata`.
- `TpuRoutedExperts`: `scale_before_down_proj` now works under EP > 1 with
`ring_of_experts`, `ragged_all_to_all` and `auto` (gathering per-row
routing weights via `roe_gather_row_scales` or `ragged_all_to_all` and
skipping the post-W2 multiply in combine), while still rejecting upstream's
`all_to_all` at `parallelize` and `forward`. `parallelize` binds
`_ep_local_rank` for all TPU EP transports and enables `roe_zero_pad_tail`
when `inner_experts` does not use Pallas GMMs.
- components/expert_parallel.py, tpu_job_config.py, qwen3_5/model.py:
- `use_expert_parallel` defaults to `ep_transport="auto"` (preserving any
non-default TPU transport already set on a dispatcher) and accepts an
optional `scale_before_down_proj` override (clearing
`fuse_route_score_backward` when enabled).
- `TPUConfig` exposes optional `ep_transport` and
`ep_scale_before_down_proj` CLI overrides forwarded by `TpuQwen35Model`.
- components/remat_regions.py, graph_trainer/tpu_memory_policy.py:
- Adds `moe.ep_dispatch` (`MOE_EP_DISPATCH`) for Ring-of-Experts activation
all-gather outputs; `moe.routing` saves only `ids_all_GK`.
- graph_trainer/collectives.py, graph_trainer/config.py,
graph_trainer/passes/expert_all_gathers_under_splash.py:
- `classify_fsdp_roles` excludes integer collectives so saved-placeholder
routing gathers are never classified as FSDP parameter gathers or gradient
reductions.
- `_is_gathered_expert_weight` excludes `.token_dispatcher.` and integer
gathers so `expert_all_gathers_under_splash` ignores EP token/routing
exchanges.
- `_validate_graph_split` rejects `backward_partition_mode="fsdp_pipeline"`
when `expert_parallel_degree > 1`, directing callers to `by_layer`.
Based on #142 (Jialei Chen) and #105 / #111 / #112 (Jack Oh), with routing
weight placement unified under `scale_before_down_proj` per #142 review (Will
Constable). Not covered here: token chunking and chunk barriers (next PR),
scan + EP, and compiled multi-mesh grad norm.
Tests (2026-11-09 nightly image `sha256:c0080328...`, v6e-8):
- CPU (`ep07-c2`, `ep07-c3`): `test_token_dispatcher.py` (22 passed, 3.75s),
`test_ragged_ep_dispatcher.py` (11 passed, 35.77s),
`test_roe_ep_dispatcher.py` (24 passed, 102.63s),
`test_qwen3_5_tpu` (47 tests, 5 skipped, 103.97s),
`test_graph_split` (13 tests, 1.51s),
`test_tpu_graph_trainer_config` (170 tests, 13.41s),
`test_tpu_memory_policy` (21 tests, 5.07s).
- TPU (`ep07-t2`, `ep07-t3`):
`test_ring_of_experts_tpu_fx_trace_and_regions` (1 test, 4.05s),
`test_ragged_gather_tc` (6 tests, 58.67s),
`test_train_distributed_qwen3_5_moe_ep2_simplefsdp` (1 test, 36.73s),
`test_train_distributed_qwen3_5_moe_ep2_tpu_moe_kernels` (1 test, 33.41s),
`test_qwen3_5_moe_graph_trainer_ep2_matches_ep1` (1 test, 118.17s),
`test_ep_parity_distributed` (9 tests, 834.35s).
tanvibajpai
added a commit
that referenced
this pull request
Nov 18, 2026
…er EP guards Addresses the review of the ragged all-to-all transport. - components/token_dispatcher.py: under EP > 1, tracing the dispatch now raises when the bound TpuRoutedExperts' `_ep_local_rank` buffer is missing, on another device, or a real tensor under a fake trace (so not a graph input). It used to fall back to a constant, which bakes this rank's index into the graph and gives each rank a different program. EP=1 and eager dispatch keep their fallbacks. A comment notes that the expert counts stay integer because the GraphTrainer passes below key on it. - graph_trainer/passes/expert_all_gathers_under_splash.py, graph_trainer/collectives.py, graph_trainer/config.py (moved here from #202): `_is_gathered_expert_weight` skips integer gathers and `.token_dispatcher.` FQNs; `classify_fsdp_roles` maps integer collectives to None; `_validate_graph_split` rejects backward_partition_mode='fsdp_pipeline' with expert_parallel_degree > 1 and points to by_layer. The count gather runs under the routed experts' FQN, so without these the Splash pass would take it for an expert-weight gather and fsdp_pipeline for an FSDP parameter gather. - tests: - tests/test_ragged_ep_dispatcher.py: tracing without a usable rank buffer raises (missing, other device, not a graph input); a trace with TpuGraphTrainer's tracer reads each rank's offsets from its own buffer; eager dispatch without the buffer uses the mesh rank. - graph_trainer/tests/test_graph_split.py: the count gather, with the routed experts' FQN, is neither an expert-weight gather nor an FSDP gather; plus the EP collectives test from #202. - graph_trainer/tests/test_tpu_graph_trainer_config.py: EP rejects fsdp_pipeline and allows by_layer (from #202). Tests: CPU (2026-11-09 nightly image): test_ragged_ep_dispatcher.py 16 passed, test_token_dispatcher.py 17 passed, test_graph_split 14 OK, test_tpu_graph_trainer_config 170 OK.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EP stack: #181 (
[EP 1/9]) → #182 ([EP 2/9]) → #183 ([EP 3/9]) → #184 ([EP 4/9]) → #186 ([EP 5/9]) → #189 ([EP 6/9]) → this PR ([EP 7/9]).Summary
Adds the dropless, static-shape Ring-of-Experts (
ring_of_experts) transport andep_transport="auto"to the shared TPU token dispatcher, using the TensorCore ragged gather / gather-reduce kernels from #189, and extendsTpuRoutedExperts.scale_before_down_projto work under EP > 1 across both TPU transports.When
ep_size <= top_k(for example Qwen3.5 at EP = 2..8 withtop_k = 8), Ring-of-Experts moves aboutep_size * Tun-expanded token rows per rank viaall_gather_dim+reduce_scatter_diminstead oftop_k * Texpanded rows throughragged_all_to_all, and its traffic does not grow under skewed routing. Abovetop_k,ep_transport="auto"switches toragged_all_to_all(#184).Changes
components/token_dispatcher.py:TpuAllToAllTokenDispatcher.Config.ep_transport: adds"ring_of_experts"and"auto"("ring_of_experts"whenep_size <= top_k, else"ragged_all_to_all").ring_of_experts: all-gathers(ids, x, scores)overep_meshviatorch.tpu.ops.all_gather_dim, computes global expert routing and gathers this rank'scountlocal-expert rows on TensorCore (roe_compute_routing,roe_sort_gather), reduces per-token partials in FP32 (roe_unsort_combineor unweightedroe_unsort_reduce), and sums partials acrossep_meshwithtorch.tpu.ops.reduce_scatter_dim. Off TPU, runs a pure-PyTorch functional-collective reference for CPU gloo tests; tracing or missing process groups raise instead of silently repeating shards.TpuRoEDispatchMetadataandTpuRaggedDispatchMetadata: carry explicitvalid_count,row_scale_Randrow_scale_applied, without subclassingLocalDispatchMetadata.TpuRoutedExperts.scale_before_down_proj: single routing-weight placement setting covering EP = 1 local dispatch and EP > 1 (ring_of_experts,ragged_all_to_all,auto), gathering per-row routing weights to the expert rank and using unweighted combine; upstream'sall_to_allremains rejected atparallelizeandforward.components/expert_parallel.py,tpu_job_config.py,qwen3_5/model.py:use_expert_paralleldefaults toep_transport="auto"(preserving any non-default TPU EP transport already set on a dispatcher) and accepts an optionalscale_before_down_projoverride.TPUConfigadds optionalep_transportandep_scale_before_down_projCLI overrides forwarded byTpuQwen35Model.components/remat_regions.py,graph_trainer/tpu_memory_policy.py:moe.ep_dispatch(MOE_EP_DISPATCH) for Ring-of-Experts activation all-gather outputs;moe.routingsaves onlyids_all_GK.graph_trainer/collectives.py,graph_trainer/config.py,graph_trainer/passes/expert_all_gathers_under_splash.py:classify_fsdp_rolesexcludes integer collectives so saved-placeholder routing gathers are never classified as FSDP parameter gathers or gradient reductions._is_gathered_expert_weightexcludes.token_dispatcher.and integer gathers soexpert_all_gathers_under_splashignores EP token/routing exchanges._validate_graph_splitrejectsbackward_partition_mode="fsdp_pipeline"withexpert_parallel_degree > 1, directing callers toby_layer.tests/test_roe_ep_dispatcher.py(new): multi-process CPU (gloo) parity at EP = 2 and 4 against EP = 1 across random, skewed and empty-expert routing with NaN tail rows;ep_transport="auto"resolution and parity above and attop_k;scale_before_down_proj=Trueparity (outputs, input/score grads and expert weight grads) acrossring_of_experts,ragged_all_to_allandauto; rank-owner, off-TPU tracing and tail-padding guards.qwen3_5/tests/test_ep_parity_distributed.py: real TPU collectives on 8 chips against a single-device CPU reference forauto(EP = 1/2/4/8), explicitring_of_experts(EP = 4/8), andscale_before_down_proj=True(EP = 2/4).qwen3_5/tests/test_qwen3_5_tpu.py,tests/test_token_dispatcher.py,graph_trainer/tests/test_graph_split.py,graph_trainer/tests/test_tpu_graph_trainer_config.py,graph_trainer/tests/test_tpu_graph_trainer_distributed.py.Testing
All run on
v6e-8insidetorchtitan-container@sha256:c0080328e7682795e21e98147f2db390efd52d7435a21c4db166da75512f1011:ep07-c2,ep07-c3):torchtitan/experiments/tpu/tests/test_token_dispatcher.py:22 passed in 3.75storchtitan/experiments/tpu/tests/test_ragged_ep_dispatcher.py:11 passed in 35.77storchtitan/experiments/tpu/tests/test_roe_ep_dispatcher.py:24 passed in 102.63storchtitan.experiments.tpu.qwen3_5.tests.test_qwen3_5_tpu:Ran 47 tests in 103.970s OK (skipped=5)torchtitan.experiments.tpu.graph_trainer.tests.test_graph_split:Ran 13 tests in 1.512s OKtorchtitan.experiments.tpu.graph_trainer.tests.test_tpu_graph_trainer_config:Ran 170 tests in 13.411s OKtorchtitan.experiments.tpu.graph_trainer.tests.test_tpu_memory_policy:Ran 21 tests in 5.069s OKep07-t2,ep07-t3):Qwen35Test.test_ring_of_experts_tpu_fx_trace_and_regions:Ran 1 test in 4.054s OKtorchtitan.experiments.tpu.kernels.tests.test_ragged_gather_tc:Ran 6 tests in 58.665s OKTrainDistributedTest.test_train_distributed_qwen3_5_moe_ep2_simplefsdp:Ran 1 test in 36.731s OKTrainDistributedTest.test_train_distributed_qwen3_5_moe_ep2_tpu_moe_kernels:Ran 1 test in 33.409s OKTpuGraphTrainerDistributedTest.test_qwen3_5_moe_graph_trainer_ep2_matches_ep1:Ran 1 test in 118.171s OKtorchtitan.experiments.tpu.qwen3_5.tests.test_ep_parity_distributed:Ran 9 tests in 834.350s OKNotes
TpuRoutedExperts.scale_before_down_projper Will Constable's review on [tpu][qwen3_5] Fast dropless EP: Ring-of-Experts, Pallas ragged gather/reduce, chunked pipelining #142.[cap, D]buffer size and overlap EP collectives with expert matmuls) follow in the next PR (EP-8).