Repository navigation
[EP 4/9] tpu moe: dropless static-shape ragged all-to-all transport for expert parallelism - #184
Merged
Conversation
This was referenced Nov 11, 2026
tanvibajpai
force-pushed
the
tbajpai/ep-04-ragged-a2a-transport
branch
from
November 12, 2026 21:32
022632f to
cac54a4
Compare
tanvibajpai
force-pushed
the
tbajpai/ep-03-split-optimizer-by-mesh
branch
from
November 12, 2026 21:32
954e384 to
be2ed21
Compare
jialei777
approved these changes
Nov 17, 2026
tanvibajpai
force-pushed
the
tbajpai/ep-03-split-optimizer-by-mesh
branch
from
November 18, 2026 05:55
be2ed21 to
ae9a477
Compare
… parallelism
Under EP > 1 the shared TPU dispatcher falls back to upstream's
AllToAllTokenDispatcher, whose variable-size all-to-all copies the token
splits to the host before exchanging tokens. That can't be traced by
TpuGraphTrainer. This adds a traceable, dropless transport to the shared
dispatcher, so any model using TpuTokenDispatcherConverter and
TpuRoutedExpertsConverter can opt in.
- components/token_dispatcher.py:
- TpuAllToAllTokenDispatcher.Config.ep_transport: "all_to_all" (default,
upstream's path, unchanged) or "ragged_all_to_all".
- ragged_all_to_all: all-gather the per-expert token counts, derive the
exchange offsets on device, and move exactly the routed rows with
torch.tpu.ops.ragged_all_to_all into a static expert-major receive buffer
of ep_size * T * top_k rows. No host sync and no dropped tokens,
whatever the routing skew. The combine sends the rows back with the
symmetric exchange and then uses the same fixed-K reduction as EP=1.
- TpuRaggedDispatchMetadata holds the offsets. It doesn't subclass
LocalDispatchMetadata, so TpuRoutedExperts' EP=1-only guard rejects it.
- The exchange is an autograd.Function with precomputed backward offsets,
which keeps it traceable. On TPU it requires a named process group; off
TPU a slow eager reference built from all-gathers runs CPU tests, and
tracing off TPU raises rather than returning zeros.
- TpuRoutedExperts.parallelize registers a non-persistent one-element
buffer holding the local EP rank and binds the dispatcher to it, so every
rank traces the same program (a Python int would bake each rank's index
into its graph). Dispatch raises if no owner is bound.
- tests/test_ragged_ep_dispatcher.py (new): EP=2 and EP=4 on CPU (gloo) with
random, skewed and empty-expert routing match the EP=1 path on the whole
batch, outputs and gradients, with NaN in the receive-buffer rows past the
routed count; plus config validation, tracing, the missing-owner error and
when parallelize registers the rank buffer.
- tests/test_token_dispatcher.py: EP=1 keeps the TPU fixed-K reduction with
either transport; the ragged metadata isn't LocalDispatchMetadata.
Memory: the receive buffer is sized for the worst case, ep_size * T * top_k
rows per rank. Bounded capacity, Ring-of-Experts and chunked overlap are
follow-ups. No recipe enables the new transport yet; Qwen3.5 does in a later
change. Based on #6 (Jialei Chen).
Tests: CPU (2026-11-09 nightly image): test_ragged_ep_dispatcher.py and
test_token_dispatcher.py pass (counts in the PR description).
tanvibajpai
force-pushed
the
tbajpai/ep-04-ragged-a2a-transport
branch
from
November 18, 2026 06:28
cac54a4 to
8ef6546
Compare
…er EP guards Addresses the review of the ragged all-to-all transport. - components/token_dispatcher.py: under EP > 1, tracing the dispatch now raises when the bound TpuRoutedExperts' `_ep_local_rank` buffer is missing, on another device, or a real tensor under a fake trace (so not a graph input). It used to fall back to a constant, which bakes this rank's index into the graph and gives each rank a different program. EP=1 and eager dispatch keep their fallbacks. A comment notes that the expert counts stay integer because the GraphTrainer passes below key on it. - graph_trainer/passes/expert_all_gathers_under_splash.py, graph_trainer/collectives.py, graph_trainer/config.py (moved here from #202): `_is_gathered_expert_weight` skips integer gathers and `.token_dispatcher.` FQNs; `classify_fsdp_roles` maps integer collectives to None; `_validate_graph_split` rejects backward_partition_mode='fsdp_pipeline' with expert_parallel_degree > 1 and points to by_layer. The count gather runs under the routed experts' FQN, so without these the Splash pass would take it for an expert-weight gather and fsdp_pipeline for an FSDP parameter gather. - tests: - tests/test_ragged_ep_dispatcher.py: tracing without a usable rank buffer raises (missing, other device, not a graph input); a trace with TpuGraphTrainer's tracer reads each rank's offsets from its own buffer; eager dispatch without the buffer uses the mesh rank. - graph_trainer/tests/test_graph_split.py: the count gather, with the routed experts' FQN, is neither an expert-weight gather nor an FSDP gather; plus the EP collectives test from #202. - graph_trainer/tests/test_tpu_graph_trainer_config.py: EP rejects fsdp_pipeline and allows by_layer (from #202). Tests: CPU (2026-11-09 nightly image): test_ragged_ep_dispatcher.py 16 passed, test_token_dispatcher.py 17 passed, test_graph_split 14 OK, test_tpu_graph_trainer_config 170 OK.
Base automatically changed from
tbajpai/ep-03-split-optimizer-by-mesh
to
main
November 18, 2026 20:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Under EP > 1 the shared TPU dispatcher falls back to upstream's
AllToAllTokenDispatcher, whose variable-sizeall-to-all copies the token splits to the host before exchanging tokens, so
TpuGraphTrainercan't trace it.This PR adds a traceable, dropless transport to the shared dispatcher behind an opt-in knob, so any
model that uses
TpuTokenDispatcherConverterandTpuRoutedExpertsConvertercan use it.Part of the EP stack, which adds expert parallelism to the shared TPU MoE path.
Changes
torchtitan/experiments/tpu/components/token_dispatcher.py:TpuAllToAllTokenDispatcher.Config.ep_transport:"all_to_all"(default; upstream's path, unchanged) or"ragged_all_to_all".ragged_all_to_all: all-gather the per-expert token counts, derive the exchange offsets on device, and move exactlythe routed rows with
torch.tpu.ops.ragged_all_to_allinto a static expert-major receive buffer ofep_size * T * top_krows. No host sync and no dropped tokens, whatever the routing skew. The combine sends therows back with the symmetric exchange, then applies the same fixed-K reduction as EP = 1.
TpuRaggedDispatchMetadataholds the offsets. It doesn't subclassLocalDispatchMetadata, so the EP = 1-onlyguard in
TpuRoutedExperts([EP 1/9] tpu moe: reject scale_before_down_proj under expert parallelism #181) rejects it.autograd.Functionwith precomputed backward offsets, which keeps it traceable. On TPU itrequires a named process group. Off TPU, a slow eager reference built from all-gathers runs the CPU tests, and
tracing off TPU raises instead of returning zeros.
TpuRoutedExperts.parallelizeregisters a non-persistent one-element buffer with the local EP rank and binds thedispatcher to it, so every rank traces the same program (a Python int would bake each rank's index into its
graph). Dispatch raises if no owner is bound, and tracing at EP > 1 raises if the buffer is missing, on another
device, or not a graph input, instead of falling back to a constant.
torchtitan/experiments/tpu/graph_trainer/(moved here from [EP 7/9] tpu moe: Ring-of-Experts EP transport, auto selection, and EP row scaling #202, so the transport is safe with the graph-splitand Splash recipes as soon as it lands; the count gather runs under the routed experts' FQN):
passes/expert_all_gathers_under_splash.py:_is_gathered_expert_weightskips integer gathers (and.token_dispatcher.FQNs), so the count gather isn't taken for an expert-weight gather.collectives.py:classify_fsdp_rolesmaps integer collectives toNone, never to an FSDP gather or reduction.config.py:compile.graph_split.backward_partition_mode='fsdp_pipeline'is rejected withexpert_parallel_degree > 1; useby_layer.torchtitan/experiments/tpu/tests/test_ragged_ep_dispatcher.py(new): EP = 2 and 4 on CPU (gloo) with random, skewedand empty-expert routing match the EP = 1 path on the whole batch (outputs and gradients), with NaN in the receive
buffer's rows past the routed count; plus config validation, tracing, the missing-owner error, when
parallelizeregisters the rank buffer, tracing without a usable rank buffer (raises), a
minimal_fx_tracertrace that readseach rank's offsets from its own buffer, and the eager fallback.
torchtitan/experiments/tpu/tests/test_token_dispatcher.py: EP = 1 keeps the TPU fixed-K reduction with eithertransport; the ragged metadata isn't
LocalDispatchMetadata.torchtitan/experiments/tpu/graph_trainer/tests/test_graph_split.py: the count gather, with its real FQN, isneither an expert-weight gather nor an FSDP gather; integer EP collectives are never FSDP gathers (from [EP 7/9] tpu moe: Ring-of-Experts EP transport, auto selection, and EP row scaling #202).
torchtitan/experiments/tpu/graph_trainer/tests/test_tpu_graph_trainer_config.py: EP rejectsfsdp_pipelineandallows
by_layer(from [EP 7/9] tpu moe: Ring-of-Experts EP transport, auto selection, and EP row scaling #202).Testing
In the 2026-11-09 nightly image (torch 2.16.0.dev20261108, jax 0.12.0.1):
test_ragged_ep_dispatcher.py16 passed;test_token_dispatcher.py17 passed;test_graph_split14 OK;test_tpu_graph_trainer_config170 OK.test_train_distributed_qwen3_fsdp_moe_use_gmm_kernelandtest_train_distributed_qwen3_5_moe_simplefsdpOK;test_qwen3_5_graph_trainer_matches_eagerOK.--compile.graph_split.backward_partition_mode=by_layer):test_qwen3_5_moe_graph_trainer_ep2_matches_ep1OK,so TpuGraphTrainer traces the ragged path with the rank buffer as a graph input, and
test_train_distributed_qwen3_5_moe_ep2_simplefsdpOK.test_split_optimizers_by_mesh10 OK, and after rebasing thestack onto
mainat9c0aa673the CPU and TPU tests passed again on [EP 5/9] tpu moe: train Qwen3.5 MoE with expert parallelism on the shared ragged transport #186's branch.for Qwen3.5.
Notes
ep_size * T * top_krows per rank (+2.9 / +7.1 GiB per chip atEP = 4 / 8 on a 4-layer Qwen3.5-35B slice with [EP 5/9] tpu moe: train Qwen3.5 MoE with expert parallelism on the shared ragged transport #186). Bounded capacity, Ring-of-Experts and chunked overlap
are follow-ups.
against EP = 1 on TPU.