Repository navigation
[EP 8/9] tpu moe: sequence token chunking, FP32 cross-chunk dW, and chunk barriers - #207
Open
tanvibajpai wants to merge 1 commit into
Open
tanvibajpai wants to merge 1 commit into
tanvibajpai wants to merge 1 commit into
Conversation
…iers
Splits MoE EP layers into `num_token_chunks` sequence chunks in the shared
`TpuRoutedExperts` module, accumulates the per-chunk expert weight gradients
in FP32, and adds the `insert_ep_chunk_barriers` GraphTrainer FX pass, which
bounds simultaneously live chunks to `W` (default `W = 2`) so one chunk's EP
collectives can overlap another chunk's expert matmuls without keeping every
chunk's buffers live.
- components/grouped_experts.py:
- Extracts `_unshard_local_weights()` and adds `unshard_only` and
`pre_unsharded_weights` to `TpuGroupedExperts.forward` (and `weights` to
`_native_experts_forward`), so chunked MoE layers unshard FSDP expert
weights once per layer while keeping `module_fqn` metadata for SimpleFSDP
bucketing.
- components/token_dispatcher.py:
- `TpuRoutedExperts.Config`: adds `num_token_chunks` (default `1`,
unchunked), `max_inflight_chunks` (`2`), and
`accumulate_chunk_weight_grads_fp32` (`True`).
- `TpuRoutedExperts._forward_ep_chunked` (EP > 1 with `ring_of_experts`,
`ragged_all_to_all` or `auto`): unshards the expert weights once, widens
them to FP32 when `accumulate_chunk_weight_grads_fp32` is set so autograd
sums the per-chunk dWeights in FP32 instead of BF16, computes per-chunk
expert counts for `ragged_all_to_all`, supports `scale_before_down_proj`,
and tags each chunk's ops with `ep_chunk_idx` / `ep_max_inflight_chunks`.
Token counts not divisible by `num_token_chunks` use the largest divisor
`<= num_token_chunks`, with a one-time warning. FSDP2-wrapped experts are
rejected, since FSDP2 reshards the weights as soon as the unshard call
returns: chunking needs SimpleFSDP.
- graph_trainer/passes/insert_ep_chunk_barriers.py (new),
graph_trainer/tpu_backend_pass.py:
- Inserts `_optimization_barrier` between chunk `k - W`'s exits (including
completed FP32 cross-chunk weight-gradient `aten.add` accumulators) and
chunk `k`'s entries in forward and backward, and keeps `bf16 -> fp32`
weight-widening casts out of forward barriers so XLA folds
`bf16(fp32(w_bf16))` back to `w_bf16` instead of materializing FP32
weights in forward.
- components/expert_parallel.py, tpu_job_config.py, qwen3_5/model.py:
- Adds `default_num_token_chunks(ep_degree)` (`ep_degree` clamped to
4..16) and optional `num_token_chunks` / `max_inflight_chunks` arguments
to `use_expert_parallel`, exposed as `TPUConfig.ep_token_chunks` /
`ep_max_inflight_chunks` and forwarded by `TpuQwen35Model`.
- tests:
- tests/test_ep_token_chunking.py (new): multi-process CPU (gloo) parity of
chunked EP against unchunked EP = 1 for `ring_of_experts`,
`ragged_all_to_all` and `auto` (with and without `scale_before_down_proj`,
skewed and empty-expert routing, an indivisible token count), one unshard
per layer, FP32 vs BF16 cross-chunk dWeight sums in the traced joint
graph, the FSDP2 guard, and the barrier pass (barrier counts, FP32
accumulators in backward barriers only, unchanged outputs).
- qwen3_5/tests/test_ep_parity_distributed.py: 8-chip TPU parity against a
single-device CPU reference for 4-chunk EP = 2 (Ring-of-Experts) and
4-chunk EP = 4 (ragged all-to-all) with `scale_before_down_proj`.
- graph_trainer/tests/test_tpu_graph_trainer_distributed.py: TpuGraphTrainer
EP = 2 with 4 chunks matches EP = 1 at step 1.
- qwen3_5/tests/test_qwen3_5_tpu.py: the chunking CLI overrides reach
`TpuRoutedExperts`.
Based on #142 (Jialei Chen) and #106 (Jack Oh), with FP32 cross-chunk dWeight
accumulation per #142 review (Will Constable). Not covered: EP recipes (next
PR), chunking with FSDP2, and chunking at EP = 1.
Tests (2026-11-09 nightly image `sha256:c0080328...`, v6e-8):
- CPU (`ep08r-c1`): `test_ep_token_chunking.py` (13 passed, 53.68s),
`test_roe_ep_dispatcher.py` (24 passed, 99.03s),
`test_ragged_ep_dispatcher.py` (11 passed, 32.03s),
`test_token_dispatcher.py` (22 passed, 4.11s),
`tests/unit_tests/test_tpu_grouped_experts.py` (26 passed, 11.24s),
`test_split_optimizers_by_mesh` (10 tests, 0.052s),
`test_graph_split` (13 tests, 1.501s),
`test_tpu_graph_trainer_config` (170 tests, 13.788s),
`test_tpu_memory_policy` (21 tests, 4.847s),
`test_qwen3_5_tpu` (47 tests, 5 skipped, 111.628s).
- TPU (`ep08r-t1`, `ep08r-t2`):
`test_ring_of_experts_tpu_fx_trace_and_regions` (1 test, 3.946s),
`test_train_distributed_qwen3_5_moe_ep2_simplefsdp` (1 test, 36.666s),
`test_train_distributed_qwen3_5_moe_ep2_tpu_moe_kernels` (1 test, 33.473s),
`test_qwen3_5_moe_graph_trainer_ep2_matches_ep1` (1 test, 184.510s; step-1
loss 12.911852 at EP = 1 and 12.912108 at EP = 2, unchunked and with 4
chunks), `test_ep_parity_distributed` (10 of 11 OK in one 1209.458s run; libtpu
aborted the 4-chunk EP = 4 case there with SLICE_FAILURE_CHIP_DRIVER_ERROR,
and it passed alone in 78.417s).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
EP stack: #181 (
[EP 1/9]) → #182 ([EP 2/9]) → #183 ([EP 3/9]) → #184 ([EP 4/9]) → #186 ([EP 5/9]) → #189 ([EP 6/9]) → #202 ([EP 7/9]) → this PR ([EP 8/9]).Summary
Adds sequence token chunking (
num_token_chunks) and FP32 cross-chunk weight-gradient accumulation to the sharedTpuRoutedExpertsmodule, plus theinsert_ep_chunk_barriersTpuGraphTrainerFX pass, which bounds simultaneously live chunks toW = max_inflight_chunks(default2) so one chunk's EP collectives can overlap another chunk's expert matmuls.Splitting a layer's
Ttokens intoCchunks divides each chunk's worst-case[EP * (T / C) * top_k, D]expert buffer byC. The FSDP expert weights are still unsharded once per layer (no extra FSDP traffic), and the per-chunkdWeights are summed in FP32 rather than BF16.num_token_chunks = 1(the default) leaves every existing path unchanged; chunking applies only at EP > 1 with the TPU transports (ring_of_experts,ragged_all_to_all,auto).Changes
components/grouped_experts.py:_unshard_local_weights()and addsunshard_onlyandpre_unsharded_weightstoTpuGroupedExperts.forward(andweightsto_native_experts_forward), so a chunked layer unshards the FSDP expert weights once, through the module call, which keepsmodule_fqnmetadata for SimpleFSDP bucketing.components/token_dispatcher.py:TpuRoutedExperts.Config: addsnum_token_chunks: int = 1,max_inflight_chunks: int = 2, andaccumulate_chunk_weight_grads_fp32: bool = True.TpuRoutedExperts._forward_ep_chunked: unshards the expert weights once outside the chunk loop and widens them to FP32 (whenaccumulate_chunk_weight_grads_fp32), so each chunk's.bfloat16()cast widens itsdWto FP32 before autograd sums across chunks; computes per-chunk counts forragged_all_to_all; supportsscale_before_down_projon both transports; and tags each chunk's ops withep_chunk_idx/ep_max_inflight_chunks. Token counts not divisible bynum_token_chunksuse the largest divisor<= num_token_chunks, with a one-time warning. FSDP2-wrapped experts raiseNotImplementedError(FSDP2 reshards the weights as soon as the unshard call returns), so chunking needs SimpleFSDP.graph_trainer/passes/insert_ep_chunk_barriers.py(new),graph_trainer/tpu_backend_pass.py:_optimization_barrierbetween chunkk - W's exits (including completed FP32 cross-chunkdWaten.addaccumulators) and chunkk's entries in forward and backward, and keepsbf16 -> fp32weight-widening casts out of forward barriers so XLA foldsbf16(fp32(w_bf16))back tow_bf16instead of materializing FP32 weights in forward.components/expert_parallel.py,tpu_job_config.py,qwen3_5/model.py:default_num_token_chunks(ep_degree)(ep_degreeclamped to 4..16) and optionalnum_token_chunks/max_inflight_chunksarguments touse_expert_parallel, exposed asTPUConfig.ep_token_chunks/ep_max_inflight_chunksand forwarded byTpuQwen35Model.tests/test_ep_token_chunking.py(new): multi-process CPU (gloo) parity of chunked EP against unchunked EP = 1 forring_of_experts,ragged_all_to_allandauto(with and withoutscale_before_down_proj, skewed and empty-expert routing, an indivisible token count), one unshard per layer, FP32 vs BF16 cross-chunkdWsums in the traced joint graph, the FSDP2 guard, and the barrier pass (barrier counts withW = 2, FP32 accumulators only in backward barriers, unchanged outputs).qwen3_5/tests/test_ep_parity_distributed.py: real TPU collectives on 8 chips against a single-device CPU reference for 4-chunk EP = 2 (Ring-of-Experts viaauto) and 4-chunk EP = 4 (ragged all-to-all viaauto) withscale_before_down_proj=True.graph_trainer/tests/test_tpu_graph_trainer_distributed.py:TpuGraphTrainerEP = 2, unchunked and with 4 chunks, matches EP = 1 at step 1.qwen3_5/tests/test_qwen3_5_tpu.py: the chunking CLI overrides reachTpuRoutedExperts.Testing
All run on
v6e-8insidetorchtitan-container@sha256:c0080328e7682795e21e98147f2db390efd52d7435a21c4db166da75512f1011:ep08r-c1):torchtitan/experiments/tpu/tests/test_ep_token_chunking.py:13 passed, 1 warning in 53.68storchtitan/experiments/tpu/tests/test_roe_ep_dispatcher.py:24 passed, 1 warning in 99.03storchtitan/experiments/tpu/tests/test_ragged_ep_dispatcher.py:11 passed, 1 warning in 32.03storchtitan/experiments/tpu/tests/test_token_dispatcher.py:22 passed, 1 warning in 4.11stests/unit_tests/test_tpu_grouped_experts.py:26 passed, 1 warning in 11.24storchtitan.experiments.tpu.tests.test_split_optimizers_by_mesh:Ran 10 tests in 0.052s OKtorchtitan.experiments.tpu.graph_trainer.tests.test_graph_split:Ran 13 tests in 1.501s OKtorchtitan.experiments.tpu.graph_trainer.tests.test_tpu_graph_trainer_config:Ran 170 tests in 13.788s OKtorchtitan.experiments.tpu.graph_trainer.tests.test_tpu_memory_policy:Ran 21 tests in 4.847s OKtorchtitan.experiments.tpu.qwen3_5.tests.test_qwen3_5_tpu:Ran 47 tests in 111.628s OK (skipped=5)ep08r-t1,ep08r-t2):Qwen35Test.test_ring_of_experts_tpu_fx_trace_and_regions:Ran 1 test in 3.946s OKTrainDistributedTest.test_train_distributed_qwen3_5_moe_ep2_simplefsdp:Ran 1 test in 36.666s OKTrainDistributedTest.test_train_distributed_qwen3_5_moe_ep2_tpu_moe_kernels:Ran 1 test in 33.473s OKTpuGraphTrainerDistributedTest.test_qwen3_5_moe_graph_trainer_ep2_matches_ep1:Ran 1 test in 184.510s OK, step-1 loss{'ep1': 12.91185188293457, 'ep2': 12.912108421325684, 'ep2_chunk4': 12.912108421325684}torchtitan.experiments.tpu.qwen3_5.tests.test_ep_parity_distributed: 10 of 11 passed in one run (Ran 11 tests in 1209.458s). libtpu aborted the eleventh,test_qwen3_5_moe_ep_matches_single_device_ep4_chunk4_w2, withSLICE_FAILURE_CHIP_DRIVER_ERRORfrom the TPU driver; rerun alone:Ran 1 test in 78.417s OK.Notes
dWeightaccumulation per Will Constable's review on [tpu][qwen3_5] Fast dropless EP: Ring-of-Experts, Pallas ragged gather/reduce, chunked pipelining #142.default_num_token_chunksis only a suggested default here: nothing turns chunking on automatically. The Qwen3.5 35B and 397B EP recipes that use it follow in the final PR ([EP 9/9]).