Skip to content

[EP 7/9] tpu moe: Ring-of-Experts EP transport, auto selection, and EP row scaling - #202

Open
tanvibajpai wants to merge 2 commits into
tbajpai/ep-05-qwen35-ep-trainingfrom
tbajpai/ep-07-ring-of-experts
Open

tanvibajpai wants to merge 2 commits into
tbajpai/ep-05-qwen35-ep-trainingfrom
tbajpai/ep-07-ring-of-experts

Conversation

@tanvibajpai

Copy link
Copy Markdown
Contributor

EP stack: #181 ([EP 1/9]) → #182 ([EP 2/9]) → #183 ([EP 3/9]) → #184 ([EP 4/9]) → #186 ([EP 5/9]) → #189 ([EP 6/9]) → this PR ([EP 7/9]).

Review note: Because #189 ([EP 6/9]) targets main independently, commit 1 of 2 on this branch (1e84166e8) is #189 cherry-picked onto #186 (ddada10e8); commit 2 (76e02f890) is the [EP 7/9] diff itself. Once #189 merges, that first commit drops out on rebase.

Summary

Adds the dropless, static-shape Ring-of-Experts (ring_of_experts) transport and ep_transport="auto" to the shared TPU token dispatcher, using the TensorCore ragged gather / gather-reduce kernels from #189, and extends TpuRoutedExperts.scale_before_down_proj to work under EP > 1 across both TPU transports.

When ep_size <= top_k (for example Qwen3.5 at EP = 2..8 with top_k = 8), Ring-of-Experts moves about ep_size * T un-expanded token rows per rank via all_gather_dim + reduce_scatter_dim instead of top_k * T expanded rows through ragged_all_to_all, and its traffic does not grow under skewed routing. Above top_k, ep_transport="auto" switches to ragged_all_to_all (#184).

Changes

  • components/token_dispatcher.py:
    • TpuAllToAllTokenDispatcher.Config.ep_transport: adds "ring_of_experts" and "auto" ("ring_of_experts" when ep_size <= top_k, else "ragged_all_to_all").
    • ring_of_experts: all-gathers (ids, x, scores) over ep_mesh via torch.tpu.ops.all_gather_dim, computes global expert routing and gathers this rank's count local-expert rows on TensorCore (roe_compute_routing, roe_sort_gather), reduces per-token partials in FP32 (roe_unsort_combine or unweighted roe_unsort_reduce), and sums partials across ep_mesh with torch.tpu.ops.reduce_scatter_dim. Off TPU, runs a pure-PyTorch functional-collective reference for CPU gloo tests; tracing or missing process groups raise instead of silently repeating shards.
    • TpuRoEDispatchMetadata and TpuRaggedDispatchMetadata: carry explicit valid_count, row_scale_R and row_scale_applied, without subclassing LocalDispatchMetadata.
    • TpuRoutedExperts.scale_before_down_proj: single routing-weight placement setting covering EP = 1 local dispatch and EP > 1 (ring_of_experts, ragged_all_to_all, auto), gathering per-row routing weights to the expert rank and using unweighted combine; upstream's all_to_all remains rejected at parallelize and forward.
  • components/expert_parallel.py, tpu_job_config.py, qwen3_5/model.py:
    • use_expert_parallel defaults to ep_transport="auto" (preserving any non-default TPU EP transport already set on a dispatcher) and accepts an optional scale_before_down_proj override.
    • TPUConfig adds optional ep_transport and ep_scale_before_down_proj CLI overrides forwarded by TpuQwen35Model.
  • components/remat_regions.py, graph_trainer/tpu_memory_policy.py:
    • Adds moe.ep_dispatch (MOE_EP_DISPATCH) for Ring-of-Experts activation all-gather outputs; moe.routing saves only ids_all_GK.
  • graph_trainer/collectives.py, graph_trainer/config.py, graph_trainer/passes/expert_all_gathers_under_splash.py:
    • classify_fsdp_roles excludes integer collectives so saved-placeholder routing gathers are never classified as FSDP parameter gathers or gradient reductions.
    • _is_gathered_expert_weight excludes .token_dispatcher. and integer gathers so expert_all_gathers_under_splash ignores EP token/routing exchanges.
    • _validate_graph_split rejects backward_partition_mode="fsdp_pipeline" with expert_parallel_degree > 1, directing callers to by_layer.
  • Tests:
    • tests/test_roe_ep_dispatcher.py (new): multi-process CPU (gloo) parity at EP = 2 and 4 against EP = 1 across random, skewed and empty-expert routing with NaN tail rows; ep_transport="auto" resolution and parity above and at top_k; scale_before_down_proj=True parity (outputs, input/score grads and expert weight grads) across ring_of_experts, ragged_all_to_all and auto; rank-owner, off-TPU tracing and tail-padding guards.
    • qwen3_5/tests/test_ep_parity_distributed.py: real TPU collectives on 8 chips against a single-device CPU reference for auto (EP = 1/2/4/8), explicit ring_of_experts (EP = 4/8), and scale_before_down_proj=True (EP = 2/4).
    • qwen3_5/tests/test_qwen3_5_tpu.py, tests/test_token_dispatcher.py, graph_trainer/tests/test_graph_split.py, graph_trainer/tests/test_tpu_graph_trainer_config.py, graph_trainer/tests/test_tpu_graph_trainer_distributed.py.

Testing

All run on v6e-8 inside torchtitan-container@sha256:c0080328e7682795e21e98147f2db390efd52d7435a21c4db166da75512f1011:

  • CPU (ep07-c2, ep07-c3):
    • torchtitan/experiments/tpu/tests/test_token_dispatcher.py: 22 passed in 3.75s
    • torchtitan/experiments/tpu/tests/test_ragged_ep_dispatcher.py: 11 passed in 35.77s
    • torchtitan/experiments/tpu/tests/test_roe_ep_dispatcher.py: 24 passed in 102.63s
    • torchtitan.experiments.tpu.qwen3_5.tests.test_qwen3_5_tpu: Ran 47 tests in 103.970s OK (skipped=5)
    • torchtitan.experiments.tpu.graph_trainer.tests.test_graph_split: Ran 13 tests in 1.512s OK
    • torchtitan.experiments.tpu.graph_trainer.tests.test_tpu_graph_trainer_config: Ran 170 tests in 13.411s OK
    • torchtitan.experiments.tpu.graph_trainer.tests.test_tpu_memory_policy: Ran 21 tests in 5.069s OK
  • TPU (ep07-t2, ep07-t3):
    • Qwen35Test.test_ring_of_experts_tpu_fx_trace_and_regions: Ran 1 test in 4.054s OK
    • torchtitan.experiments.tpu.kernels.tests.test_ragged_gather_tc: Ran 6 tests in 58.665s OK
    • TrainDistributedTest.test_train_distributed_qwen3_5_moe_ep2_simplefsdp: Ran 1 test in 36.731s OK
    • TrainDistributedTest.test_train_distributed_qwen3_5_moe_ep2_tpu_moe_kernels: Ran 1 test in 33.409s OK
    • TpuGraphTrainerDistributedTest.test_qwen3_5_moe_graph_trainer_ep2_matches_ep1: Ran 1 test in 118.171s OK
    • torchtitan.experiments.tpu.qwen3_5.tests.test_ep_parity_distributed: Ran 9 tests in 834.350s OK

Notes

Adds kernels/ragged_gather_tc.py: MaxText's TensorCore ragged gather and
ragged gather-reduce kernels (from its MLPerf Training TPU7x MoE
submissions), bridged to PyTorch with @pallas.jax_op, plus the custom ops
that the Ring-of-Experts (RoE) EP transport calls. Nothing in torchtitan
uses the module yet; the RoE transport, later in the EP stack, is the first
caller.

- packed_stable_sort: stable expert order from one single-key uint32 sort
  when the key and position bits fit in 32 bits; a two-operand stable sort
  otherwise.
- ragged_gather_tc: out[i] = x[indices[start + i]] for i < count into a
  static [cap, ...] buffer. The Pallas grid is cdiv(count, block_size), so
  HBM traffic scales with the routed count rather than with cap.
- ragged_gather_reduce_tc: sums each token's gathered rows in FP32, in a
  deterministic order, optionally weighted by FP32 routing scores, and
  never reads rows >= count.
- _ragged_scale_and_dscore_tc: backward kernel that scales the routed
  gradient by the routing weights and computes the FP32 dScore in one pass,
  masking rows >= count.
- tt_moe::roe_* custom ops with autograd behind the public entry points
  roe_compute_routing, roe_sort_gather, roe_unsort_combine,
  roe_unsort_reduce and roe_gather_row_scales.

The hidden size must be a multiple of 128. The VMEM limit is picked per TPU
generation: 64 MiB on TPU7x, 32 MiB on v6e, 16 MiB otherwise.

LICENSE lists the file as derived from MaxText. The kernel file comes
unchanged from #142 (Jialei Chen); the test file only has its imports
updated for the TPU directory reorg (#159). Jack Oh's drafts #110 and #107
port the same MaxText kernels; this module is functionally a superset.

Tests (kernels/tests/test_ragged_gather_tc.py, new):
- CPU and TPU: TPU generation parsing; packed and wide-key sorts against
  numpy's stable argsort; layout and empty-metadata edge cases.
- TPU only, against a pure-PyTorch FP32 reference: routing metadata, gather
  and weighted / unweighted gather-reduce with their backwards over >= 6
  gather steps and >= 8 reduce blocks (4-buffer wrap) with a partial last
  block; NaN rows past the routed count never reach outputs or gradients;
  the fused scale + dScore backward on full and partial blocks; the
  row-scale gather and its collision-free backward; a make_fx fake-tensor
  trace.
Results on a TPU v6e-8 and CPU, 2026-11-09 nightly image:
test_ragged_gather_tc "Ran 6 tests ... OK" on TPU; on CPU 6 tests OK with
3 skipped (TPU only).
…ling

Adds the dropless, static-shape Ring-of-Experts (RoE) expert-parallel
transport and `ep_transport="auto"` to the shared TPU dispatcher, and extends
`TpuRoutedExperts.scale_before_down_proj` to work under EP > 1 across both TPU
transports instead of introducing a separate RoE-only flag.

- components/token_dispatcher.py:
  - `TpuAllToAllTokenDispatcher.Config.ep_transport`: adds `"ring_of_experts"`
    and `"auto"` (`"ring_of_experts"` when `ep_size <= top_k`, otherwise
    `"ragged_all_to_all"`).
  - `ring_of_experts`: all-gathers `(ids, x, scores)` over `ep_mesh` via
    `torch.tpu.ops.all_gather_dim`, computes global expert routing and gathers
    this rank's `count` local-expert rows on TensorCore (`roe_compute_routing`,
    `roe_sort_gather` from #189), reduces per-token partial outputs in FP32
    (`roe_unsort_combine` or unweighted `roe_unsort_reduce`), and sums partials
    across `ep_mesh` with `torch.tpu.ops.reduce_scatter_dim`. Off TPU, runs a
    pure-PyTorch functional-collective reference for CPU gloo tests; tracing or
    missing process groups raise instead of silently repeating shards.
  - `TpuRoEDispatchMetadata` and `TpuRaggedDispatchMetadata`: carry explicit
    `valid_count`, `row_scale_R` and `row_scale_applied`, without subclassing
    `LocalDispatchMetadata`.
  - `TpuRoutedExperts`: `scale_before_down_proj` now works under EP > 1 with
    `ring_of_experts`, `ragged_all_to_all` and `auto` (gathering per-row
    routing weights via `roe_gather_row_scales` or `ragged_all_to_all` and
    skipping the post-W2 multiply in combine), while still rejecting upstream's
    `all_to_all` at `parallelize` and `forward`. `parallelize` binds
    `_ep_local_rank` for all TPU EP transports and enables `roe_zero_pad_tail`
    when `inner_experts` does not use Pallas GMMs.
- components/expert_parallel.py, tpu_job_config.py, qwen3_5/model.py:
  - `use_expert_parallel` defaults to `ep_transport="auto"` (preserving any
    non-default TPU transport already set on a dispatcher) and accepts an
    optional `scale_before_down_proj` override (clearing
    `fuse_route_score_backward` when enabled).
  - `TPUConfig` exposes optional `ep_transport` and
    `ep_scale_before_down_proj` CLI overrides forwarded by `TpuQwen35Model`.
- components/remat_regions.py, graph_trainer/tpu_memory_policy.py:
  - Adds `moe.ep_dispatch` (`MOE_EP_DISPATCH`) for Ring-of-Experts activation
    all-gather outputs; `moe.routing` saves only `ids_all_GK`.
- graph_trainer/collectives.py, graph_trainer/config.py,
  graph_trainer/passes/expert_all_gathers_under_splash.py:
  - `classify_fsdp_roles` excludes integer collectives so saved-placeholder
    routing gathers are never classified as FSDP parameter gathers or gradient
    reductions.
  - `_is_gathered_expert_weight` excludes `.token_dispatcher.` and integer
    gathers so `expert_all_gathers_under_splash` ignores EP token/routing
    exchanges.
  - `_validate_graph_split` rejects `backward_partition_mode="fsdp_pipeline"`
    when `expert_parallel_degree > 1`, directing callers to `by_layer`.

Based on #142 (Jialei Chen) and #105 / #111 / #112 (Jack Oh), with routing
weight placement unified under `scale_before_down_proj` per #142 review (Will
Constable). Not covered here: token chunking and chunk barriers (next PR),
scan + EP, and compiled multi-mesh grad norm.

Tests (2026-11-09 nightly image `sha256:c0080328...`, v6e-8):
- CPU (`ep07-c2`, `ep07-c3`): `test_token_dispatcher.py` (22 passed, 3.75s),
  `test_ragged_ep_dispatcher.py` (11 passed, 35.77s),
  `test_roe_ep_dispatcher.py` (24 passed, 102.63s),
  `test_qwen3_5_tpu` (47 tests, 5 skipped, 103.97s),
  `test_graph_split` (13 tests, 1.51s),
  `test_tpu_graph_trainer_config` (170 tests, 13.41s),
  `test_tpu_memory_policy` (21 tests, 5.07s).
- TPU (`ep07-t2`, `ep07-t3`):
  `test_ring_of_experts_tpu_fx_trace_and_regions` (1 test, 4.05s),
  `test_ragged_gather_tc` (6 tests, 58.67s),
  `test_train_distributed_qwen3_5_moe_ep2_simplefsdp` (1 test, 36.73s),
  `test_train_distributed_qwen3_5_moe_ep2_tpu_moe_kernels` (1 test, 33.41s),
  `test_qwen3_5_moe_graph_trainer_ep2_matches_ep1` (1 test, 118.17s),
  `test_ep_parity_distributed` (9 tests, 834.35s).
tanvibajpai added a commit that referenced this pull request Nov 18, 2026
…er EP guards

Addresses the review of the ragged all-to-all transport.

- components/token_dispatcher.py: under EP > 1, tracing the dispatch now
  raises when the bound TpuRoutedExperts' `_ep_local_rank` buffer is
  missing, on another device, or a real tensor under a fake trace (so not a
  graph input). It used to fall back to a constant, which bakes this rank's
  index into the graph and gives each rank a different program. EP=1 and
  eager dispatch keep their fallbacks. A comment notes that the expert
  counts stay integer because the GraphTrainer passes below key on it.
- graph_trainer/passes/expert_all_gathers_under_splash.py,
  graph_trainer/collectives.py, graph_trainer/config.py (moved here from
  #202): `_is_gathered_expert_weight` skips integer gathers and
  `.token_dispatcher.` FQNs; `classify_fsdp_roles` maps integer collectives
  to None; `_validate_graph_split` rejects
  backward_partition_mode='fsdp_pipeline' with expert_parallel_degree > 1
  and points to by_layer. The count gather runs under the routed experts'
  FQN, so without these the Splash pass would take it for an expert-weight
  gather and fsdp_pipeline for an FSDP parameter gather.
- tests:
  - tests/test_ragged_ep_dispatcher.py: tracing without a usable rank
    buffer raises (missing, other device, not a graph input); a trace with
    TpuGraphTrainer's tracer reads each rank's offsets from its own buffer;
    eager dispatch without the buffer uses the mesh rank.
  - graph_trainer/tests/test_graph_split.py: the count gather, with the
    routed experts' FQN, is neither an expert-weight gather nor an FSDP
    gather; plus the EP collectives test from #202.
  - graph_trainer/tests/test_tpu_graph_trainer_config.py: EP rejects
    fsdp_pipeline and allows by_layer (from #202).

Tests: CPU (2026-11-09 nightly image): test_ragged_ep_dispatcher.py
16 passed, test_token_dispatcher.py 17 passed, test_graph_split 14 OK,
test_tpu_graph_trainer_config 170 OK.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant