Repository navigation
[EP 6/9] tpu moe: TensorCore ragged gather / gather-reduce Pallas kernels for EP - #189
Open
tanvibajpai wants to merge 2 commits into
Open
tanvibajpai wants to merge 2 commits into
tanvibajpai wants to merge 2 commits into
Conversation
Adds kernels/ragged_gather_tc.py: MaxText's TensorCore ragged gather and ragged gather-reduce kernels (from its MLPerf Training TPU7x MoE submissions), bridged to PyTorch with @pallas.jax_op, plus the custom ops that the Ring-of-Experts (RoE) EP transport calls. Nothing in torchtitan uses the module yet; the RoE transport, later in the EP stack, is the first caller. - packed_stable_sort: stable expert order from one single-key uint32 sort when the key and position bits fit in 32 bits; a two-operand stable sort otherwise. - ragged_gather_tc: out[i] = x[indices[start + i]] for i < count into a static [cap, ...] buffer. The Pallas grid is cdiv(count, block_size), so HBM traffic scales with the routed count rather than with cap. - ragged_gather_reduce_tc: sums each token's gathered rows in FP32, in a deterministic order, optionally weighted by FP32 routing scores, and never reads rows >= count. - _ragged_scale_and_dscore_tc: backward kernel that scales the routed gradient by the routing weights and computes the FP32 dScore in one pass, masking rows >= count. - tt_moe::roe_* custom ops with autograd behind the public entry points roe_compute_routing, roe_sort_gather, roe_unsort_combine, roe_unsort_reduce and roe_gather_row_scales. The hidden size must be a multiple of 128. The VMEM limit is picked per TPU generation: 64 MiB on TPU7x, 32 MiB on v6e, 16 MiB otherwise. LICENSE lists the file as derived from MaxText. The kernel file comes unchanged from #142 (Jialei Chen); the test file only has its imports updated for the TPU directory reorg (#159). Jack Oh's drafts #110 and #107 port the same MaxText kernels; this module is functionally a superset. Tests (kernels/tests/test_ragged_gather_tc.py, new): - CPU and TPU: TPU generation parsing; packed and wide-key sorts against numpy's stable argsort; layout and empty-metadata edge cases. - TPU only, against a pure-PyTorch FP32 reference: routing metadata, gather and weighted / unweighted gather-reduce with their backwards over >= 6 gather steps and >= 8 reduce blocks (4-buffer wrap) with a partial last block; NaN rows past the routed count never reach outputs or gradients; the fused scale + dScore backward on full and partial blocks; the row-scale gather and its collision-free backward; a make_fx fake-tensor trace. Results on a TPU v6e-8 and CPU, 2026-11-09 nightly image: test_ragged_gather_tc "Ran 6 tests ... OK" on TPU; on CPU 6 tests OK with 3 skipped (TPU only).
jialei777
approved these changes
Nov 17, 2026
| return int(match.group(1)) if match else None | ||
|
|
||
|
|
||
| def _tpu_vmem_limit_bytes() -> int: |
Contributor
There was a problem hiding this comment.
should this and the above _tpu_generation be moved outside this file and to some general tpu utils file?
Contributor
Author
There was a problem hiding this comment.
good call. moved _tpu_generation and _tpu_vmem_limit_bytes into kernels/pallas_vmem.py as tpu_generation / vmem_limit_bytes (705c7ea), so other kernels can share them. there are older copies of _tpu_generation in causal_conv1d and tokamax_gated_delta; i'll consolidate those in a follow-up.
This was referenced Nov 17, 2026
…llas_vmem.py Other Pallas kernels can import them instead of keeping their own copies. ragged_gather_tc.py now calls pallas_vmem.vmem_limit_bytes(). The helpers' bodies are unchanged: 64 MiB on TPU7x, 32 MiB on v6e, 16 MiB otherwise. The device-name parsing test moves to kernels/tests/test_pallas_vmem.py, which also checks the limit for each generation. The older copies of the generation parser in causal_conv1d.py and qwen3_5/tokamax_gated_delta.py are left for a follow-up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
torchtitan/experiments/tpu/kernels/ragged_gather_tc.py: MaxText's TensorCore ragged gather and ragged gather-reduce kernels (from its MLPerf Training TPU7x MoE submissions), bridged to PyTorch with@pallas.jax_op, plus the custom ops that the RoE transport calls. Nothing in torchtitan uses the module yet; the RoE transport, later in the EP stack, is the first caller.Why these kernels: under EP each rank's expert input is a static worst-case
[cap, D]buffer, but only the firstcountrows hold routed tokens. The gather and gather-reduce kernels run a dynamic Pallas grid ofcdiv(count, block_size), so their HBM traffic follows the routed count rather thancap, and they never read the rows pastcount.Changes
kernels/ragged_gather_tc.py(new):packed_stable_sort: stable expert order from one single-keyuint32sort when the key and position bits fit in 32 bits; a two-operand stable sort otherwise.ragged_gather_tc:out[i] = x[indices[start + i]]fori < countinto a static[cap, ...]buffer, with double-buffered async DMAs.ragged_gather_reduce_tcand its metadata builder: sums each token's gathered rows in FP32, in a deterministic order, optionally weighted by FP32 routing scores, through a 4-buffer VMEM ring._ragged_scale_and_dscore_tc: backward kernel that scales the routed gradient by the routing weights and computes the FP32 dScore in one pass, masking rows>= count.tt_moe::roe_*custom ops with autograd, behindroe_compute_routing,roe_sort_gather,roe_unsort_combine,roe_unsort_reduceandroe_gather_row_scales.safe_gather_block_size/safe_reduce_block_sizeclamp block sizes to the per-generation VMEM limit frompallas_vmem.vmem_limit_bytes().kernels/pallas_vmem.py(new):tpu_generation()parses the TPU generation from the device name, andvmem_limit_bytes()returns the VMEM limit for it (64 MiB on TPU7x, 32 MiB on v6e, 16 MiB otherwise). They're in their own module so other kernels can share them.kernels/tests/test_ragged_gather_tc.py(new):make_fxfake-tensor trace.kernels/tests/test_pallas_vmem.py(new): TPU generation parsing from device names, and the VMEM limit for each generation.LICENSE: listsragged_gather_tc.pyas derived from MaxText.Testing
2026-11-09 nightly image (torch 2.16.0.dev20261108, jax 0.12.0.1):
test_ragged_gather_tc"Ran 5 tests in 54.017s OK".test_ragged_gather_tc"Ran 5 tests in 0.323s OK (skipped=3)" (the three kernel tests need a TPU);test_pallas_vmem"Ran 2 tests in 0.001s OK".Notes
pallas_vmem.py, and the test file only has its imports updated for the TPU directory reorg (Reorganize the TPU experiment root into topic subpackages. #159).roe_compute_routingclamps each rank's rows tocap; with a droplesscapthe clamp never triggers.