Skip to content

[EP 5/9] tpu moe: train Qwen3.5 MoE with expert parallelism on the shared ragged transport - #186

Open
tanvibajpai wants to merge 4 commits into
mainfrom
tbajpai/ep-05-qwen35-ep-training
Open

tanvibajpai wants to merge 4 commits into
mainfrom
tbajpai/ep-05-qwen35-ep-training

Conversation

@tanvibajpai

@tanvibajpai tanvibajpai commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Qwen3.5 MoE can now train with --parallelism.expert_parallel_degree > 1 on TPU, in both the TPU trainer and TpuGraphTrainer, on the shared pieces from this stack: the ragged all-to-all transport (#184), the GMM tail handling (#182) and the optimizer split by mesh (#183). Based on #6 (Jialei Chen), but on the shared dispatcher instead of a Qwen3.5-specific one.

Changes

  • components/expert_parallel.py (new):
    • use_expert_parallel(model_config, ep_degree=...): a no-op at EP = 1. Under EP > 1 it turns plain RoutedExperts configs into TpuRoutedExperts (whose parallelize binds the EP rank buffer the ragged transport needs), turns the all-to-all dispatchers into TpuAllToAllTokenDispatcher with ep_transport="ragged_all_to_all", and turns off the paired Pallas GMM, which TpuGroupedExperts allows only at EP = 1.
    • apply_expert_parallel_module_overrides(model, parallel_dims): under EP > 1 on TPU, switches tile-aligned experts that would use torch._grouped_mm to unpaired Pallas GMMs, because its backward reads uninitialized memory on empty expert segments (torch_tpu#2668). A no-op at EP = 1.
  • qwen3_5/model.py, qwen3_5/__init__.py: TpuQwen35Model. Its Config.update_from_config calls use_expert_parallel and its parallelize calls the module overrides; every TPU Qwen3.5 config now builds it. At EP = 1 the config update is identical to upstream's (tested), and state-dict keys don't change (the rank buffer is non-persistent).
  • qwen3_5/infra/parallelize.py: apply_qwen3_5_simple_fsdp wraps each MoE's routed experts on the expert data-parallel mesh, then applies SimpleFSDP to the rest. Upstream's apply_simple_fsdp does that only for TransformerBlock, which Qwen3.5's block isn't.
  • graph_trainer/qwen3_5.py: TpuGraphTrainer accepts EP for Qwen3.5 with SimpleFSDP (TP, CP and PP are still rejected), calls model.parallelize under EP and uses apply_qwen3_5_simple_fsdp.
  • tpu_job_config.py: the compile_grad_norm error now says to pass --compile.no-compile_grad_norm with EP > 1 or PP > 1; the eager grad norm handles the dense and expert meshes.
  • Tests:
    • qwen3_5/tests/test_ep_parity_distributed.py (new): 8 TPU chips at EP = 1/2/4/8 against a single-device CPU reference (loss, gradients and parameters over 3 steps), with the real ragged_all_to_all.
    • tests/test_train_distributed.py: Qwen3.5 MoE at EP = 2 with SimpleFSDP and with the Pallas MoE kernels (TPU only).
    • graph_trainer/tests/test_tpu_graph_trainer_distributed.py: TpuGraphTrainer at EP = 2 matches EP = 1 at step 1.
    • qwen3_5/tests/test_qwen3_5_tpu.py: at EP = 1 the config update matches upstream's; at EP = 2 it switches to TpuRoutedExperts, the ragged transport and unpaired GMMs; the graph trainer rejects EP without SimpleFSDP.

Testing

On a v6e-8 with the 2026-11-09 nightly image (torch 2.16.0.dev20261108, jax 0.12.0.1):

Test Where Result
test_qwen3_5_tpu CPU 46 OK (4 skipped)
test_token_dispatcher.py, test_ragged_ep_dispatcher.py, tests/unit_tests/test_tpu_grouped_experts.py CPU 17, 11 and 26 passed
test_split_optimizers_by_mesh, test_tpu_graph_trainer_config CPU 10 and 169 OK
test_train_distributed: Qwen3.5 MoE at EP = 1 and EP = 2, each with SimpleFSDP and with the Pallas MoE kernels TPU 4 OK
test_ep_parity_distributed: EP = 1/2/4/8 vs a single-device CPU reference TPU 5 OK
TpuGraphTrainer EP = 2 vs EP = 1 TPU OK (step-1 loss 12.912108 vs 12.911852)
TpuGraphTrainer vs the eager trainer (EP = 1) TPU OK

The new tests are picked up by .ci/discover_tpu_tests.py automatically; the parity test takes about 5 minutes on a v6e-8.

After rebasing the stack onto main at 9c0aa673 (where spmd_types became the default backend), re-run on #186's branch: every test in the table passes again.

Notes

TpuGroupedExperts padded the routed rows to a multiple of tile_m and added
the pad rows to the last expert's count. At EP=1 that is harmless: the pad
rows are zeros and are sliced off the output. Under dropless expert
parallelism, though, the receive buffer has a static worst-case row count
whose rows past sum(num_tokens_per_expert_E) hold stale data. Growing the
last expert's group over the padding pulls those rows into that expert's
forward and dWeight. The dWeight TGMM kernels also need the row count to be
a multiple of their own M tile, which can be larger than tile_m: with
tile_m=128 and a 256-row TGMM tile, 300 rows were padded to 384, which that
tile doesn't divide.

- components/grouped_experts.py: pad the rows (and row_scale_R) to
  lcm(tile_m, TGMM M tile, W2 TGMM M tile) and pass num_tokens_per_expert_E
  unchanged. Every kernel skips rows past its sum.
- kernels/paired_segment_matmul.py: the paired W1/W3 forward zeroes each
  output tile on its first visit. Its masked stores never write rows past
  sum(group_sizes), which kept stale VMEM contents in the partly routed tile
  once the pad rows stopped belonging to the last group. The docstrings now
  say group_sizes may sum to less than the row count.

EP=1 outputs and gradients are unchanged: the pad rows were zeros, and now
they're skipped instead (checked bit-exactly below). Will Constable measured
the paired-kernel zero-init as neutral at EP=1 in his review of #142
(15.461 -> 15.490 ms). Based on #6 and #142 (Jialei Chen).

Tests:
- kernels/tests/test_paired_segment_matmul.py: NaN-filled rows past the
  routed count (a partly routed tile and an unrouted tile) don't reach the
  outputs, dInput or dWeight, against the unpaired kernel with zero rows.
- kernels/tests/test_expert_segment_matmul.py: the same for the unpaired
  kernel (empty groups, a TGMM tile larger than tile_m, transpose_rhs), bit
  exact against zero rows in the last group.
- tests/test_grouped_experts_tpu.py (new): 300 unaligned rows, paired and
  unpaired, with and without row_scale_R, bit exact against zero rows in the
  last expert.
- tests/unit_tests/test_tpu_grouped_experts.py: every kernel gets the counts
  unchanged and rows padded to every M tile; NaN tail rows don't reach the
  outputs or gradients (CPU, kernels mocked with torch._grouped_mm).
Results on a TPU v6e-8 and CPU, 2026-11-09 nightly image:
test_expert_segment_matmul 7 OK, test_paired_segment_matmul 9 OK,
test_grouped_experts_tpu 7 OK, transformers_modeling_backend
test_pallas_experts_tpu 7 OK; test_tpu_grouped_experts.py 26 passed.
Under expert parallelism the routed experts live on an (efsdp, ep) mesh and
the dense parameters on (fsdp,), but torch_tpu's fused and foreach AdamW
need every param group to share one mesh. Only TpuGraphTrainer split the
groups, with a private helper that ran after the base trainer had built the
LR schedulers, so it rebuilt them afterwards. The eager TPUTrainer didn't
split at all.

- utils.py: public split_optimizers_by_mesh(). It replaces a group whose
  params span several meshes with one group per mesh, keeping the group's
  options and the matching param_names, so the FQN-keyed optimizer state dict
  keeps its keys. Meshes compare by equality rather than identity (FSDP2 with
  TP builds equal but separate mesh objects), non-DTensor params share one
  group, and a second call changes nothing. It validates every group before
  changing any, and refuses to split once LR schedulers are attached.
  with_split_optimizers_by_mesh() chains it after a model's own
  post_optimizer_build_fn.
- train.py, graph_trainer/tpu_graph_trainer.py: both trainers install that
  hook. The base trainer runs post_optimizer_build_fn right after building
  the optimizers and before the LR schedulers and checkpointer, so the late
  split, its LR-scheduler rebuild and the private helper are gone.
- tests/test_split_optimizers_by_mesh.py (new): one and two meshes, equal
  meshes, plain tensors, idempotence, the param_names check, the LR-scheduler
  guard, optimizer state dicts saved and loaded across a split, and both
  trainers splitting inside the hook before their LR schedulers are built
  (the GraphTrainer case with the in-graph optimizer).

Nothing changes at EP=1: every group is already on one mesh. Based on #6
(Jialei Chen).

Tests (2026-11-09 nightly image):
- CPU: test_split_optimizers_by_mesh 10 OK; test_tpu_graph_trainer_config
  169 OK.
- TPU v6e-8, 8 ranks at EP=1 (the hook runs, nothing splits):
  test_train_distributed qwen3_fsdp_moe_use_gmm_kernel and
  qwen3_5_moe_simplefsdp OK; test_tpu_graph_trainer_distributed
  test_qwen3_5_graph_trainer_matches_eager and gemma4_moe_simple_fsdp OK.
… parallelism

Under EP > 1 the shared TPU dispatcher falls back to upstream's
AllToAllTokenDispatcher, whose variable-size all-to-all copies the token
splits to the host before exchanging tokens. That can't be traced by
TpuGraphTrainer. This adds a traceable, dropless transport to the shared
dispatcher, so any model using TpuTokenDispatcherConverter and
TpuRoutedExpertsConverter can opt in.

- components/token_dispatcher.py:
  - TpuAllToAllTokenDispatcher.Config.ep_transport: "all_to_all" (default,
    upstream's path, unchanged) or "ragged_all_to_all".
  - ragged_all_to_all: all-gather the per-expert token counts, derive the
    exchange offsets on device, and move exactly the routed rows with
    torch.tpu.ops.ragged_all_to_all into a static expert-major receive buffer
    of ep_size * T * top_k rows. No host sync and no dropped tokens,
    whatever the routing skew. The combine sends the rows back with the
    symmetric exchange and then uses the same fixed-K reduction as EP=1.
  - TpuRaggedDispatchMetadata holds the offsets. It doesn't subclass
    LocalDispatchMetadata, so TpuRoutedExperts' EP=1-only guard rejects it.
  - The exchange is an autograd.Function with precomputed backward offsets,
    which keeps it traceable. On TPU it requires a named process group; off
    TPU a slow eager reference built from all-gathers runs CPU tests, and
    tracing off TPU raises rather than returning zeros.
  - TpuRoutedExperts.parallelize registers a non-persistent one-element
    buffer holding the local EP rank and binds the dispatcher to it, so every
    rank traces the same program (a Python int would bake each rank's index
    into its graph). Dispatch raises if no owner is bound.
- tests/test_ragged_ep_dispatcher.py (new): EP=2 and EP=4 on CPU (gloo) with
  random, skewed and empty-expert routing match the EP=1 path on the whole
  batch, outputs and gradients, with NaN in the receive-buffer rows past the
  routed count; plus config validation, tracing, the missing-owner error and
  when parallelize registers the rank buffer.
- tests/test_token_dispatcher.py: EP=1 keeps the TPU fixed-K reduction with
  either transport; the ragged metadata isn't LocalDispatchMetadata.

Memory: the receive buffer is sized for the worst case, ep_size * T * top_k
rows per rank. Bounded capacity, Ring-of-Experts and chunked overlap are
follow-ups. No recipe enables the new transport yet; Qwen3.5 does in a later
change. Based on #6 (Jialei Chen).

Tests: CPU (2026-11-09 nightly image): test_ragged_ep_dispatcher.py and
test_token_dispatcher.py pass (counts in the PR description).
…ed transport

Qwen3.5 MoE recipes can now run with --parallelism.expert_parallel_degree > 1
on TPU, in both the TPU trainer and TpuGraphTrainer, on the shared pieces from
the earlier PRs in this stack: the ragged all-to-all transport, the GMM tail
handling and the optimizer split by mesh. Based on #6 (Jialei Chen), but on
the shared dispatcher instead of a Qwen3.5-specific one.

- components/expert_parallel.py (new):
  - use_expert_parallel(model_config, ep_degree=...): a no-op at EP=1. Under
    EP > 1 it turns plain RoutedExperts configs into TpuRoutedExperts (whose
    parallelize binds the EP rank buffer the ragged transport needs), turns
    the all-to-all dispatchers into TpuAllToAllTokenDispatcher with
    ep_transport="ragged_all_to_all", and turns off the paired Pallas GMM,
    which TpuGroupedExperts allows only at EP=1.
  - apply_expert_parallel_module_overrides(model, parallel_dims): under EP > 1
    on TPU, switches tile-aligned experts that would use torch._grouped_mm to
    unpaired Pallas GMMs, because its backward reads uninitialized memory on
    empty expert segments (torch_tpu#2668). A no-op at EP=1.
- qwen3_5/model.py, qwen3_5/__init__.py: TpuQwen35Model. Its
  Config.update_from_config calls use_expert_parallel and its parallelize
  calls the module overrides; every TPU Qwen3.5 config now builds it. At EP=1
  the config update is identical to upstream's (tested), and state-dict keys
  don't change (the rank buffer is non-persistent).
- qwen3_5/infra/parallelize.py: apply_qwen3_5_simple_fsdp wraps each MoE's
  routed experts on the expert data-parallel mesh, then applies SimpleFSDP to
  the rest. Upstream's apply_simple_fsdp does that only for TransformerBlock,
  which Qwen3.5's block isn't.
- graph_trainer/qwen3_5.py: TpuGraphTrainer accepts EP for Qwen3.5 with
  SimpleFSDP (TP, CP and PP are still rejected), calls model.parallelize under
  EP and uses apply_qwen3_5_simple_fsdp.
- tpu_job_config.py: the compile_grad_norm error now says to pass
  --compile.no-compile_grad_norm with EP > 1 or PP > 1; the eager grad norm
  handles the dense and expert meshes.
- tests:
  - qwen3_5/tests/test_ep_parity_distributed.py (new): 8 TPU chips at
    EP=1/2/4/8 against a single-device CPU reference (loss, gradients and
    parameters over 3 steps) with the real ragged_all_to_all.
  - tests/test_train_distributed.py: Qwen3.5 MoE at EP=2 with SimpleFSDP and
    with the Pallas MoE kernels (TPU only).
  - graph_trainer/tests/test_tpu_graph_trainer_distributed.py: TpuGraphTrainer
    at EP=2 matches EP=1 at step 1.
  - qwen3_5/tests/test_qwen3_5_tpu.py: at EP=1 the config update matches
    upstream's; at EP=2 it switches to TpuRoutedExperts, the ragged transport
    and unpaired GMMs; the graph trainer rejects EP without SimpleFSDP.

Not covered: the paired Pallas GMM under EP (it stays off, as in #6), the
compiled grad norm over two meshes, scan + EP, and EP recipes (later changes).

Tests (2026-11-09 nightly image, v6e-8): CPU: test_qwen3_5_tpu 46 OK (4
skipped), test_token_dispatcher.py 17 passed, test_ragged_ep_dispatcher.py 11
passed, test_tpu_grouped_experts.py 26 passed, test_split_optimizers_by_mesh
10 OK, test_tpu_graph_trainer_config 169 OK. TPU: test_train_distributed
Qwen3.5 MoE at EP=1 and EP=2, each with SimpleFSDP and with the Pallas MoE
kernels, 4 OK; test_ep_parity_distributed 5 OK; TpuGraphTrainer EP=2 vs
EP=1 OK (step-1 loss 12.912108 vs 12.911852); TpuGraphTrainer vs eager OK.
@tanvibajpai
tanvibajpai force-pushed the tbajpai/ep-05-qwen35-ep-training branch from a008185 to ddada10 Compare November 12, 2026 21:32
@tanvibajpai
tanvibajpai force-pushed the tbajpai/ep-04-ragged-a2a-transport branch from 022632f to cac54a4 Compare November 12, 2026 21:32
@tanvibajpai tanvibajpai changed the title tpu moe: train Qwen3.5 MoE with expert parallelism on the shared ragged transport [EP 5/9] tpu moe: train Qwen3.5 MoE with expert parallelism on the shared ragged transport Nov 17, 2026
@tanvibajpai
tanvibajpai force-pushed the tbajpai/ep-04-ragged-a2a-transport branch from cac54a4 to 8ef6546 Compare November 18, 2026 06:28
Base automatically changed from tbajpai/ep-04-ragged-a2a-transport to main November 18, 2026 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant