Skip to content

[EP 8/9] tpu moe: sequence token chunking, FP32 cross-chunk dW, and chunk barriers - #207

Open
tanvibajpai wants to merge 1 commit into
tbajpai/ep-07-ring-of-expertsfrom
tbajpai/ep-08-token-chunking
Open

tanvibajpai wants to merge 1 commit into
tbajpai/ep-07-ring-of-expertsfrom
tbajpai/ep-08-token-chunking

Conversation

@tanvibajpai

Copy link
Copy Markdown
Contributor

EP stack: #181 ([EP 1/9]) → #182 ([EP 2/9]) → #183 ([EP 3/9]) → #184 ([EP 4/9]) → #186 ([EP 5/9]) → #189 ([EP 6/9]) → #202 ([EP 7/9]) → this PR ([EP 8/9]).

Review note: This PR is stacked on #202 (base tbajpai/ep-07-ring-of-experts at 76e02f890), so its diff is one commit (5b69ba31a). #181 and #182 have merged into main; the rest of the stack gets rebased onto main as it lands.

Summary

Adds sequence token chunking (num_token_chunks) and FP32 cross-chunk weight-gradient accumulation to the shared TpuRoutedExperts module, plus the insert_ep_chunk_barriers TpuGraphTrainer FX pass, which bounds simultaneously live chunks to W = max_inflight_chunks (default 2) so one chunk's EP collectives can overlap another chunk's expert matmuls.

Splitting a layer's T tokens into C chunks divides each chunk's worst-case [EP * (T / C) * top_k, D] expert buffer by C. The FSDP expert weights are still unsharded once per layer (no extra FSDP traffic), and the per-chunk dWeights are summed in FP32 rather than BF16. num_token_chunks = 1 (the default) leaves every existing path unchanged; chunking applies only at EP > 1 with the TPU transports (ring_of_experts, ragged_all_to_all, auto).

Changes

  • components/grouped_experts.py:
    • Extracts _unshard_local_weights() and adds unshard_only and pre_unsharded_weights to TpuGroupedExperts.forward (and weights to _native_experts_forward), so a chunked layer unshards the FSDP expert weights once, through the module call, which keeps module_fqn metadata for SimpleFSDP bucketing.
  • components/token_dispatcher.py:
    • TpuRoutedExperts.Config: adds num_token_chunks: int = 1, max_inflight_chunks: int = 2, and accumulate_chunk_weight_grads_fp32: bool = True.
    • TpuRoutedExperts._forward_ep_chunked: unshards the expert weights once outside the chunk loop and widens them to FP32 (when accumulate_chunk_weight_grads_fp32), so each chunk's .bfloat16() cast widens its dW to FP32 before autograd sums across chunks; computes per-chunk counts for ragged_all_to_all; supports scale_before_down_proj on both transports; and tags each chunk's ops with ep_chunk_idx / ep_max_inflight_chunks. Token counts not divisible by num_token_chunks use the largest divisor <= num_token_chunks, with a one-time warning. FSDP2-wrapped experts raise NotImplementedError (FSDP2 reshards the weights as soon as the unshard call returns), so chunking needs SimpleFSDP.
  • graph_trainer/passes/insert_ep_chunk_barriers.py (new), graph_trainer/tpu_backend_pass.py:
    • Inserts _optimization_barrier between chunk k - W's exits (including completed FP32 cross-chunk dW aten.add accumulators) and chunk k's entries in forward and backward, and keeps bf16 -> fp32 weight-widening casts out of forward barriers so XLA folds bf16(fp32(w_bf16)) back to w_bf16 instead of materializing FP32 weights in forward.
  • components/expert_parallel.py, tpu_job_config.py, qwen3_5/model.py:
    • Adds default_num_token_chunks(ep_degree) (ep_degree clamped to 4..16) and optional num_token_chunks / max_inflight_chunks arguments to use_expert_parallel, exposed as TPUConfig.ep_token_chunks / ep_max_inflight_chunks and forwarded by TpuQwen35Model.
  • Tests:
    • tests/test_ep_token_chunking.py (new): multi-process CPU (gloo) parity of chunked EP against unchunked EP = 1 for ring_of_experts, ragged_all_to_all and auto (with and without scale_before_down_proj, skewed and empty-expert routing, an indivisible token count), one unshard per layer, FP32 vs BF16 cross-chunk dW sums in the traced joint graph, the FSDP2 guard, and the barrier pass (barrier counts with W = 2, FP32 accumulators only in backward barriers, unchanged outputs).
    • qwen3_5/tests/test_ep_parity_distributed.py: real TPU collectives on 8 chips against a single-device CPU reference for 4-chunk EP = 2 (Ring-of-Experts via auto) and 4-chunk EP = 4 (ragged all-to-all via auto) with scale_before_down_proj=True.
    • graph_trainer/tests/test_tpu_graph_trainer_distributed.py: TpuGraphTrainer EP = 2, unchunked and with 4 chunks, matches EP = 1 at step 1.
    • qwen3_5/tests/test_qwen3_5_tpu.py: the chunking CLI overrides reach TpuRoutedExperts.

Testing

All run on v6e-8 inside torchtitan-container@sha256:c0080328e7682795e21e98147f2db390efd52d7435a21c4db166da75512f1011:

  • CPU (ep08r-c1):
    • torchtitan/experiments/tpu/tests/test_ep_token_chunking.py: 13 passed, 1 warning in 53.68s
    • torchtitan/experiments/tpu/tests/test_roe_ep_dispatcher.py: 24 passed, 1 warning in 99.03s
    • torchtitan/experiments/tpu/tests/test_ragged_ep_dispatcher.py: 11 passed, 1 warning in 32.03s
    • torchtitan/experiments/tpu/tests/test_token_dispatcher.py: 22 passed, 1 warning in 4.11s
    • tests/unit_tests/test_tpu_grouped_experts.py: 26 passed, 1 warning in 11.24s
    • torchtitan.experiments.tpu.tests.test_split_optimizers_by_mesh: Ran 10 tests in 0.052s OK
    • torchtitan.experiments.tpu.graph_trainer.tests.test_graph_split: Ran 13 tests in 1.501s OK
    • torchtitan.experiments.tpu.graph_trainer.tests.test_tpu_graph_trainer_config: Ran 170 tests in 13.788s OK
    • torchtitan.experiments.tpu.graph_trainer.tests.test_tpu_memory_policy: Ran 21 tests in 4.847s OK
    • torchtitan.experiments.tpu.qwen3_5.tests.test_qwen3_5_tpu: Ran 47 tests in 111.628s OK (skipped=5)
  • TPU (ep08r-t1, ep08r-t2):
    • Qwen35Test.test_ring_of_experts_tpu_fx_trace_and_regions: Ran 1 test in 3.946s OK
    • TrainDistributedTest.test_train_distributed_qwen3_5_moe_ep2_simplefsdp: Ran 1 test in 36.666s OK
    • TrainDistributedTest.test_train_distributed_qwen3_5_moe_ep2_tpu_moe_kernels: Ran 1 test in 33.473s OK
    • TpuGraphTrainerDistributedTest.test_qwen3_5_moe_graph_trainer_ep2_matches_ep1: Ran 1 test in 184.510s OK, step-1 loss {'ep1': 12.91185188293457, 'ep2': 12.912108421325684, 'ep2_chunk4': 12.912108421325684}
    • torchtitan.experiments.tpu.qwen3_5.tests.test_ep_parity_distributed: 10 of 11 passed in one run (Ran 11 tests in 1209.458s). libtpu aborted the eleventh, test_qwen3_5_moe_ep_matches_single_device_ep4_chunk4_w2, with SLICE_FAILURE_CHIP_DRIVER_ERROR from the TPU driver; rerun alone: Ran 1 test in 78.417s OK.

Notes

…iers

Splits MoE EP layers into `num_token_chunks` sequence chunks in the shared
`TpuRoutedExperts` module, accumulates the per-chunk expert weight gradients
in FP32, and adds the `insert_ep_chunk_barriers` GraphTrainer FX pass, which
bounds simultaneously live chunks to `W` (default `W = 2`) so one chunk's EP
collectives can overlap another chunk's expert matmuls without keeping every
chunk's buffers live.

- components/grouped_experts.py:
  - Extracts `_unshard_local_weights()` and adds `unshard_only` and
    `pre_unsharded_weights` to `TpuGroupedExperts.forward` (and `weights` to
    `_native_experts_forward`), so chunked MoE layers unshard FSDP expert
    weights once per layer while keeping `module_fqn` metadata for SimpleFSDP
    bucketing.
- components/token_dispatcher.py:
  - `TpuRoutedExperts.Config`: adds `num_token_chunks` (default `1`,
    unchunked), `max_inflight_chunks` (`2`), and
    `accumulate_chunk_weight_grads_fp32` (`True`).
  - `TpuRoutedExperts._forward_ep_chunked` (EP > 1 with `ring_of_experts`,
    `ragged_all_to_all` or `auto`): unshards the expert weights once, widens
    them to FP32 when `accumulate_chunk_weight_grads_fp32` is set so autograd
    sums the per-chunk dWeights in FP32 instead of BF16, computes per-chunk
    expert counts for `ragged_all_to_all`, supports `scale_before_down_proj`,
    and tags each chunk's ops with `ep_chunk_idx` / `ep_max_inflight_chunks`.
    Token counts not divisible by `num_token_chunks` use the largest divisor
    `<= num_token_chunks`, with a one-time warning. FSDP2-wrapped experts are
    rejected, since FSDP2 reshards the weights as soon as the unshard call
    returns: chunking needs SimpleFSDP.
- graph_trainer/passes/insert_ep_chunk_barriers.py (new),
  graph_trainer/tpu_backend_pass.py:
  - Inserts `_optimization_barrier` between chunk `k - W`'s exits (including
    completed FP32 cross-chunk weight-gradient `aten.add` accumulators) and
    chunk `k`'s entries in forward and backward, and keeps `bf16 -> fp32`
    weight-widening casts out of forward barriers so XLA folds
    `bf16(fp32(w_bf16))` back to `w_bf16` instead of materializing FP32
    weights in forward.
- components/expert_parallel.py, tpu_job_config.py, qwen3_5/model.py:
  - Adds `default_num_token_chunks(ep_degree)` (`ep_degree` clamped to
    4..16) and optional `num_token_chunks` / `max_inflight_chunks` arguments
    to `use_expert_parallel`, exposed as `TPUConfig.ep_token_chunks` /
    `ep_max_inflight_chunks` and forwarded by `TpuQwen35Model`.
- tests:
  - tests/test_ep_token_chunking.py (new): multi-process CPU (gloo) parity of
    chunked EP against unchunked EP = 1 for `ring_of_experts`,
    `ragged_all_to_all` and `auto` (with and without `scale_before_down_proj`,
    skewed and empty-expert routing, an indivisible token count), one unshard
    per layer, FP32 vs BF16 cross-chunk dWeight sums in the traced joint
    graph, the FSDP2 guard, and the barrier pass (barrier counts, FP32
    accumulators in backward barriers only, unchanged outputs).
  - qwen3_5/tests/test_ep_parity_distributed.py: 8-chip TPU parity against a
    single-device CPU reference for 4-chunk EP = 2 (Ring-of-Experts) and
    4-chunk EP = 4 (ragged all-to-all) with `scale_before_down_proj`.
  - graph_trainer/tests/test_tpu_graph_trainer_distributed.py: TpuGraphTrainer
    EP = 2 with 4 chunks matches EP = 1 at step 1.
  - qwen3_5/tests/test_qwen3_5_tpu.py: the chunking CLI overrides reach
    `TpuRoutedExperts`.

Based on #142 (Jialei Chen) and #106 (Jack Oh), with FP32 cross-chunk dWeight
accumulation per #142 review (Will Constable). Not covered: EP recipes (next
PR), chunking with FSDP2, and chunking at EP = 1.

Tests (2026-11-09 nightly image `sha256:c0080328...`, v6e-8):
- CPU (`ep08r-c1`): `test_ep_token_chunking.py` (13 passed, 53.68s),
  `test_roe_ep_dispatcher.py` (24 passed, 99.03s),
  `test_ragged_ep_dispatcher.py` (11 passed, 32.03s),
  `test_token_dispatcher.py` (22 passed, 4.11s),
  `tests/unit_tests/test_tpu_grouped_experts.py` (26 passed, 11.24s),
  `test_split_optimizers_by_mesh` (10 tests, 0.052s),
  `test_graph_split` (13 tests, 1.501s),
  `test_tpu_graph_trainer_config` (170 tests, 13.788s),
  `test_tpu_memory_policy` (21 tests, 4.847s),
  `test_qwen3_5_tpu` (47 tests, 5 skipped, 111.628s).
- TPU (`ep08r-t1`, `ep08r-t2`):
  `test_ring_of_experts_tpu_fx_trace_and_regions` (1 test, 3.946s),
  `test_train_distributed_qwen3_5_moe_ep2_simplefsdp` (1 test, 36.666s),
  `test_train_distributed_qwen3_5_moe_ep2_tpu_moe_kernels` (1 test, 33.473s),
  `test_qwen3_5_moe_graph_trainer_ep2_matches_ep1` (1 test, 184.510s; step-1
  loss 12.911852 at EP = 1 and 12.912108 at EP = 2, unchunked and with 4
  chunks), `test_ep_parity_distributed` (10 of 11 OK in one 1209.458s run; libtpu
  aborted the 4-chunk EP = 4 case there with SLICE_FAILURE_CHIP_DRIVER_ERROR,
  and it passed alone in 78.417s).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant