Skip to content

[EP 6/9] tpu moe: TensorCore ragged gather / gather-reduce Pallas kernels for EP - #189

Open
tanvibajpai wants to merge 2 commits into
mainfrom
tbajpai/ep-06-ragged-gather-kernels
Open

tanvibajpai wants to merge 2 commits into
mainfrom
tbajpai/ep-06-ragged-gather-kernels

Conversation

@tanvibajpai

@tanvibajpai tanvibajpai commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds torchtitan/experiments/tpu/kernels/ragged_gather_tc.py: MaxText's TensorCore ragged gather and ragged gather-reduce kernels (from its MLPerf Training TPU7x MoE submissions), bridged to PyTorch with @pallas.jax_op, plus the custom ops that the RoE transport calls. Nothing in torchtitan uses the module yet; the RoE transport, later in the EP stack, is the first caller.

Why these kernels: under EP each rank's expert input is a static worst-case [cap, D] buffer, but only the first count rows hold routed tokens. The gather and gather-reduce kernels run a dynamic Pallas grid of cdiv(count, block_size), so their HBM traffic follows the routed count rather than cap, and they never read the rows past count.

Changes

  • kernels/ragged_gather_tc.py (new):
    • packed_stable_sort: stable expert order from one single-key uint32 sort when the key and position bits fit in 32 bits; a two-operand stable sort otherwise.
    • ragged_gather_tc: out[i] = x[indices[start + i]] for i < count into a static [cap, ...] buffer, with double-buffered async DMAs.
    • ragged_gather_reduce_tc and its metadata builder: sums each token's gathered rows in FP32, in a deterministic order, optionally weighted by FP32 routing scores, through a 4-buffer VMEM ring.
    • _ragged_scale_and_dscore_tc: backward kernel that scales the routed gradient by the routing weights and computes the FP32 dScore in one pass, masking rows >= count.
    • tt_moe::roe_* custom ops with autograd, behind roe_compute_routing, roe_sort_gather, roe_unsort_combine, roe_unsort_reduce and roe_gather_row_scales. safe_gather_block_size / safe_reduce_block_size clamp block sizes to the per-generation VMEM limit from pallas_vmem.vmem_limit_bytes().
  • kernels/pallas_vmem.py (new): tpu_generation() parses the TPU generation from the device name, and vmem_limit_bytes() returns the VMEM limit for it (64 MiB on TPU7x, 32 MiB on v6e, 16 MiB otherwise). They're in their own module so other kernels can share them.
  • kernels/tests/test_ragged_gather_tc.py (new):
    • CPU and TPU: packed and wide-key sorts against numpy's stable argsort; layout and empty-metadata edge cases.
    • TPU only, against a pure-PyTorch FP32 reference: routing metadata; gather and weighted / unweighted gather-reduce with their backwards over ≥ 6 gather steps and ≥ 8 reduce blocks (4-buffer wrap) with a partial last block; NaN rows past the routed count never reach outputs or gradients; the fused scale + dScore backward on full and partial blocks; the row-scale gather and its collision-free backward; a make_fx fake-tensor trace.
  • kernels/tests/test_pallas_vmem.py (new): TPU generation parsing from device names, and the VMEM limit for each generation.
  • LICENSE: lists ragged_gather_tc.py as derived from MaxText.

Testing

2026-11-09 nightly image (torch 2.16.0.dev20261108, jax 0.12.0.1):

  • TPU v6e-8: test_ragged_gather_tc "Ran 5 tests in 54.017s OK".
  • CPU: test_ragged_gather_tc "Ran 5 tests in 0.323s OK (skipped=3)" (the three kernel tests need a TPU); test_pallas_vmem "Ran 2 tests in 0.001s OK".
  • TPU7x: this PR's CI.

Notes

Adds kernels/ragged_gather_tc.py: MaxText's TensorCore ragged gather and
ragged gather-reduce kernels (from its MLPerf Training TPU7x MoE
submissions), bridged to PyTorch with @pallas.jax_op, plus the custom ops
that the Ring-of-Experts (RoE) EP transport calls. Nothing in torchtitan
uses the module yet; the RoE transport, later in the EP stack, is the first
caller.

- packed_stable_sort: stable expert order from one single-key uint32 sort
  when the key and position bits fit in 32 bits; a two-operand stable sort
  otherwise.
- ragged_gather_tc: out[i] = x[indices[start + i]] for i < count into a
  static [cap, ...] buffer. The Pallas grid is cdiv(count, block_size), so
  HBM traffic scales with the routed count rather than with cap.
- ragged_gather_reduce_tc: sums each token's gathered rows in FP32, in a
  deterministic order, optionally weighted by FP32 routing scores, and
  never reads rows >= count.
- _ragged_scale_and_dscore_tc: backward kernel that scales the routed
  gradient by the routing weights and computes the FP32 dScore in one pass,
  masking rows >= count.
- tt_moe::roe_* custom ops with autograd behind the public entry points
  roe_compute_routing, roe_sort_gather, roe_unsort_combine,
  roe_unsort_reduce and roe_gather_row_scales.

The hidden size must be a multiple of 128. The VMEM limit is picked per TPU
generation: 64 MiB on TPU7x, 32 MiB on v6e, 16 MiB otherwise.

LICENSE lists the file as derived from MaxText. The kernel file comes
unchanged from #142 (Jialei Chen); the test file only has its imports
updated for the TPU directory reorg (#159). Jack Oh's drafts #110 and #107
port the same MaxText kernels; this module is functionally a superset.

Tests (kernels/tests/test_ragged_gather_tc.py, new):
- CPU and TPU: TPU generation parsing; packed and wide-key sorts against
  numpy's stable argsort; layout and empty-metadata edge cases.
- TPU only, against a pure-PyTorch FP32 reference: routing metadata, gather
  and weighted / unweighted gather-reduce with their backwards over >= 6
  gather steps and >= 8 reduce blocks (4-buffer wrap) with a partial last
  block; NaN rows past the routed count never reach outputs or gradients;
  the fused scale + dScore backward on full and partial blocks; the
  row-scale gather and its collision-free backward; a make_fx fake-tensor
  trace.
Results on a TPU v6e-8 and CPU, 2026-11-09 nightly image:
test_ragged_gather_tc "Ran 6 tests ... OK" on TPU; on CPU 6 tests OK with
3 skipped (TPU only).
return int(match.group(1)) if match else None


def _tpu_vmem_limit_bytes() -> int:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this and the above _tpu_generation be moved outside this file and to some general tpu utils file?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call. moved _tpu_generation and _tpu_vmem_limit_bytes into kernels/pallas_vmem.py as tpu_generation / vmem_limit_bytes (705c7ea), so other kernels can share them. there are older copies of _tpu_generation in causal_conv1d and tokamax_gated_delta; i'll consolidate those in a follow-up.

@tanvibajpai tanvibajpai changed the title tpu moe: TensorCore ragged gather / gather-reduce Pallas kernels for EP [EP 6/9] tpu moe: TensorCore ragged gather / gather-reduce Pallas kernels for EP Nov 17, 2026
…llas_vmem.py

Other Pallas kernels can import them instead of keeping their own copies.
ragged_gather_tc.py now calls pallas_vmem.vmem_limit_bytes(). The helpers'
bodies are unchanged: 64 MiB on TPU7x, 32 MiB on v6e, 16 MiB otherwise.

The device-name parsing test moves to kernels/tests/test_pallas_vmem.py,
which also checks the limit for each generation.

The older copies of the generation parser in causal_conv1d.py and
qwen3_5/tokamax_gated_delta.py are left for a follow-up.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants