Skip to content

[EP 4/9] tpu moe: dropless static-shape ragged all-to-all transport for expert parallelism - #184

Merged
copybara-service[bot] merged 3 commits into
mainfrom
tbajpai/ep-04-ragged-a2a-transport
Nov 18, 2026
Merged

copybara-service[bot] merged 3 commits into
mainfrom
tbajpai/ep-04-ragged-a2a-transport

Conversation

@tanvibajpai

@tanvibajpai tanvibajpai commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Under EP > 1 the shared TPU dispatcher falls back to upstream's AllToAllTokenDispatcher, whose variable-size
all-to-all copies the token splits to the host before exchanging tokens, so TpuGraphTrainer can't trace it.
This PR adds a traceable, dropless transport to the shared dispatcher behind an opt-in knob, so any
model that uses TpuTokenDispatcherConverter and TpuRoutedExpertsConverter can use it.

Part of the EP stack, which adds expert parallelism to the shared TPU MoE path.

Changes

  • torchtitan/experiments/tpu/components/token_dispatcher.py:
    • TpuAllToAllTokenDispatcher.Config.ep_transport: "all_to_all" (default; upstream's path, unchanged) or
      "ragged_all_to_all".
    • ragged_all_to_all: all-gather the per-expert token counts, derive the exchange offsets on device, and move exactly
      the routed rows with torch.tpu.ops.ragged_all_to_all into a static expert-major receive buffer of
      ep_size * T * top_k rows. No host sync and no dropped tokens, whatever the routing skew. The combine sends the
      rows back with the symmetric exchange, then applies the same fixed-K reduction as EP = 1.
    • TpuRaggedDispatchMetadata holds the offsets. It doesn't subclass LocalDispatchMetadata, so the EP = 1-only
      guard in TpuRoutedExperts ([EP 1/9] tpu moe: reject scale_before_down_proj under expert parallelism #181) rejects it.
    • The exchange is an autograd.Function with precomputed backward offsets, which keeps it traceable. On TPU it
      requires a named process group. Off TPU, a slow eager reference built from all-gathers runs the CPU tests, and
      tracing off TPU raises instead of returning zeros.
    • TpuRoutedExperts.parallelize registers a non-persistent one-element buffer with the local EP rank and binds the
      dispatcher to it, so every rank traces the same program (a Python int would bake each rank's index into its
      graph). Dispatch raises if no owner is bound, and tracing at EP > 1 raises if the buffer is missing, on another
      device, or not a graph input, instead of falling back to a constant.
  • torchtitan/experiments/tpu/graph_trainer/ (moved here from [EP 7/9] tpu moe: Ring-of-Experts EP transport, auto selection, and EP row scaling #202, so the transport is safe with the graph-split
    and Splash recipes as soon as it lands; the count gather runs under the routed experts' FQN):
    • passes/expert_all_gathers_under_splash.py: _is_gathered_expert_weight skips integer gathers (and
      .token_dispatcher. FQNs), so the count gather isn't taken for an expert-weight gather.
    • collectives.py: classify_fsdp_roles maps integer collectives to None, never to an FSDP gather or reduction.
    • config.py: compile.graph_split.backward_partition_mode='fsdp_pipeline' is rejected with
      expert_parallel_degree > 1; use by_layer.
  • torchtitan/experiments/tpu/tests/test_ragged_ep_dispatcher.py (new): EP = 2 and 4 on CPU (gloo) with random, skewed
    and empty-expert routing match the EP = 1 path on the whole batch (outputs and gradients), with NaN in the receive
    buffer's rows past the routed count; plus config validation, tracing, the missing-owner error, when parallelize
    registers the rank buffer, tracing without a usable rank buffer (raises), a minimal_fx_tracer trace that reads
    each rank's offsets from its own buffer, and the eager fallback.
  • torchtitan/experiments/tpu/tests/test_token_dispatcher.py: EP = 1 keeps the TPU fixed-K reduction with either
    transport; the ragged metadata isn't LocalDispatchMetadata.
  • torchtitan/experiments/tpu/graph_trainer/tests/test_graph_split.py: the count gather, with its real FQN, is
    neither an expert-weight gather nor an FSDP gather; integer EP collectives are never FSDP gathers (from [EP 7/9] tpu moe: Ring-of-Experts EP transport, auto selection, and EP row scaling #202).
  • torchtitan/experiments/tpu/graph_trainer/tests/test_tpu_graph_trainer_config.py: EP rejects fsdp_pipeline and
    allows by_layer (from [EP 7/9] tpu moe: Ring-of-Experts EP transport, auto selection, and EP row scaling #202).

Testing

In the 2026-11-09 nightly image (torch 2.16.0.dev20261108, jax 0.12.0.1):

  • After the review fixes, on this PR's branch:
    • CPU: test_ragged_ep_dispatcher.py 16 passed; test_token_dispatcher.py 17 passed; test_graph_split 14 OK;
      test_tpu_graph_trainer_config 170 OK.
    • TPU v6e-8 (EP = 1, checks nothing regressed): test_train_distributed_qwen3_fsdp_moe_use_gmm_kernel and
      test_train_distributed_qwen3_5_moe_simplefsdp OK; test_qwen3_5_graph_trainer_matches_eager OK.
    • With [EP 5/9] tpu moe: train Qwen3.5 MoE with expert parallelism on the shared ragged transport #186 on top (its EP = 2 GraphTrainer test switched to
      --compile.graph_split.backward_partition_mode=by_layer): test_qwen3_5_moe_graph_trainer_ep2_matches_ep1 OK,
      so TpuGraphTrainer traces the ragged path with the rank buffer as a graph input, and
      test_train_distributed_qwen3_5_moe_ep2_simplefsdp OK.
  • Earlier, with this PR and the three below it applied: test_split_optimizers_by_mesh 10 OK, and after rebasing the
    stack onto main at 9c0aa673 the CPU and TPU tests passed again on [EP 5/9] tpu moe: train Qwen3.5 MoE with expert parallelism on the shared ragged transport #186's branch.
  • The EP = 2/4/8 parity check against EP = 1 on the VM's 8 chips comes with the next PR, which turns the transport on
    for Qwen3.5.

Notes

… parallelism

Under EP > 1 the shared TPU dispatcher falls back to upstream's
AllToAllTokenDispatcher, whose variable-size all-to-all copies the token
splits to the host before exchanging tokens. That can't be traced by
TpuGraphTrainer. This adds a traceable, dropless transport to the shared
dispatcher, so any model using TpuTokenDispatcherConverter and
TpuRoutedExpertsConverter can opt in.

- components/token_dispatcher.py:
  - TpuAllToAllTokenDispatcher.Config.ep_transport: "all_to_all" (default,
    upstream's path, unchanged) or "ragged_all_to_all".
  - ragged_all_to_all: all-gather the per-expert token counts, derive the
    exchange offsets on device, and move exactly the routed rows with
    torch.tpu.ops.ragged_all_to_all into a static expert-major receive buffer
    of ep_size * T * top_k rows. No host sync and no dropped tokens,
    whatever the routing skew. The combine sends the rows back with the
    symmetric exchange and then uses the same fixed-K reduction as EP=1.
  - TpuRaggedDispatchMetadata holds the offsets. It doesn't subclass
    LocalDispatchMetadata, so TpuRoutedExperts' EP=1-only guard rejects it.
  - The exchange is an autograd.Function with precomputed backward offsets,
    which keeps it traceable. On TPU it requires a named process group; off
    TPU a slow eager reference built from all-gathers runs CPU tests, and
    tracing off TPU raises rather than returning zeros.
  - TpuRoutedExperts.parallelize registers a non-persistent one-element
    buffer holding the local EP rank and binds the dispatcher to it, so every
    rank traces the same program (a Python int would bake each rank's index
    into its graph). Dispatch raises if no owner is bound.
- tests/test_ragged_ep_dispatcher.py (new): EP=2 and EP=4 on CPU (gloo) with
  random, skewed and empty-expert routing match the EP=1 path on the whole
  batch, outputs and gradients, with NaN in the receive-buffer rows past the
  routed count; plus config validation, tracing, the missing-owner error and
  when parallelize registers the rank buffer.
- tests/test_token_dispatcher.py: EP=1 keeps the TPU fixed-K reduction with
  either transport; the ragged metadata isn't LocalDispatchMetadata.

Memory: the receive buffer is sized for the worst case, ep_size * T * top_k
rows per rank. Bounded capacity, Ring-of-Experts and chunked overlap are
follow-ups. No recipe enables the new transport yet; Qwen3.5 does in a later
change. Based on #6 (Jialei Chen).

Tests: CPU (2026-11-09 nightly image): test_ragged_ep_dispatcher.py and
test_token_dispatcher.py pass (counts in the PR description).
@tanvibajpai
tanvibajpai force-pushed the tbajpai/ep-04-ragged-a2a-transport branch from cac54a4 to 8ef6546 Compare November 18, 2026 06:28
…er EP guards

Addresses the review of the ragged all-to-all transport.

- components/token_dispatcher.py: under EP > 1, tracing the dispatch now
  raises when the bound TpuRoutedExperts' `_ep_local_rank` buffer is
  missing, on another device, or a real tensor under a fake trace (so not a
  graph input). It used to fall back to a constant, which bakes this rank's
  index into the graph and gives each rank a different program. EP=1 and
  eager dispatch keep their fallbacks. A comment notes that the expert
  counts stay integer because the GraphTrainer passes below key on it.
- graph_trainer/passes/expert_all_gathers_under_splash.py,
  graph_trainer/collectives.py, graph_trainer/config.py (moved here from
  #202): `_is_gathered_expert_weight` skips integer gathers and
  `.token_dispatcher.` FQNs; `classify_fsdp_roles` maps integer collectives
  to None; `_validate_graph_split` rejects
  backward_partition_mode='fsdp_pipeline' with expert_parallel_degree > 1
  and points to by_layer. The count gather runs under the routed experts'
  FQN, so without these the Splash pass would take it for an expert-weight
  gather and fsdp_pipeline for an FSDP parameter gather.
- tests:
  - tests/test_ragged_ep_dispatcher.py: tracing without a usable rank
    buffer raises (missing, other device, not a graph input); a trace with
    TpuGraphTrainer's tracer reads each rank's offsets from its own buffer;
    eager dispatch without the buffer uses the mesh rank.
  - graph_trainer/tests/test_graph_split.py: the count gather, with the
    routed experts' FQN, is neither an expert-weight gather nor an FSDP
    gather; plus the EP collectives test from #202.
  - graph_trainer/tests/test_tpu_graph_trainer_config.py: EP rejects
    fsdp_pipeline and allows by_layer (from #202).

Tests: CPU (2026-11-09 nightly image): test_ragged_ep_dispatcher.py
16 passed, test_token_dispatcher.py 17 passed, test_graph_split 14 OK,
test_tpu_graph_trainer_config 170 OK.
Base automatically changed from tbajpai/ep-03-split-optimizer-by-mesh to main November 18, 2026 20:38
@tanvibajpai tanvibajpai added the pull ready Signals Copybara to import this PR into Google3 label Nov 18, 2026
@copybara-service
copybara-service Bot merged commit 442c453 into main Nov 18, 2026
20 of 21 checks passed
@copybara-service
copybara-service Bot deleted the tbajpai/ep-04-ragged-a2a-transport branch November 18, 2026 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pull ready Signals Copybara to import this PR into Google3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants