Repository navigation
feat(cuda-oxide): shared-memory i32/addrspace(3) codegen and CTA-local TMA without SharedPtr - #1656
Merged
nihalpasham merged 3 commits intoOct 27, 2026
Conversation
…and grid_constant in blockscale_gemm_cute Signed-off-by: nihalpasham <nihalp@nvidia.com>
…ts/casts and narrow MLIR shared GEP indices to i32 Signed-off-by: nihalpasham <nihalp@nvidia.com>
- Have `cuda_device::tma::cp_async_bulk_tensor_{1..5}d_g2s_cta` take `dst: *mut u8` directly, matching `cuda_intrinsics` and `barrier: *mut Barrier`.
- Remove `SharedPtr<T>`, `SharedSlot<T>`, and their special-case lowering hooks from `cuda-device`, `cuda-intrinsics-gen`, and `mir-importer`, along with the synthetic `SharedPtr` examples.
- Route `ctm_rs::tma::cp_async_bulk_tensor_shared_cta_global` to `cp_async_bulk_tensor_2d_g2s_cta` (`.shared::cta`) and register `CpAsyncBulkTensorG2sCtaTile2dOp` in the SM120 MLIR export pack, eliminating the `.shared::cluster` syscall fallback and 8-byte stack frame on `sm_120a`.
Signed-off-by: nihalpasham <nihalp@nvidia.com>
4 tasks done
nihalpasham
added a commit
that referenced
this pull request
Oct 27, 2026
…sable, and normalize tcgen05 shared pointers (#1662) ## What This PR Does Following up on #1656 (`addrspace(3)` preservation across `.add()` and `.cast()`), this PR fixes a 16K deadlock and 4K/8K register regression in `gemm_sol_final` on Blackwell (`sm_100a`) by making two compiler improvements: 1. **Add `#[unroll(never)]` and propagate `!llvm.loop.unroll.disable` (`cuda-macros`, `mir-transforms`, `mir-lower`, `llvm-export`)** 2. **Normalize non-generic `$0` pointer operands in `inline_asm_convergent` (`mir-lower`)** --- ### 1. Why #1656 Triggered a 16K Deadlock in `gemm_sol_final` `gemm_sol_final` has two kernels selected by problem size: ```text Matrix Size Kernel Selected by `GemmSelector` Status After #1656 ─────────── ───────────────────────────────────────────────────────── ────────────────────────────── 4096³ (4K) gemm_sol_clc_multicast_4_stage_pipeline (M256xN256) Runs (64 -> 69 regs, perf dip) 8192³ (8K) gemm_sol_clc_multicast_4_stage_pipeline (M256xN256) Runs (64 -> 69 regs, perf dip) 16384³ (16K) gemm_sol_clc_multicast_4_stage_pipeline_large (M512xN256) DEADLOCKS (96 -> 99 regs) ``` In both kernels, the epilogue stores TMEM accumulators to `SMEM_OUT` inside a nested loop: ```rust while n_chunk < 2 { // outer loop (2 chunks of 128 cols) while tmem_row_block < 2 { // middle loop (2 blocks of 16 rows) while col_block < 8 { // inner loop (8 blocks of 16 cols) tcgen05_ld_16x256b_pure(...); tcgen05_load_wait(); stmatrix_m8n8_x2(warp_scratch_u8.add(...), ...); } } warp::sync_mask(u32::MAX); // ... global stores ... warp::sync_mask(u32::MAX); } ``` When #1656 kept `(&raw mut SMEM_OUT as *mut u8).add(...)` in `addrspace(3)`, it removed 2 `addrspacecast` instructions inside the inner `while col_block` loop. That dropped the inner loop body just below LLVM `opt -O2`'s `LoopUnrollPass` cost threshold: ```text BEFORE #1656: `col_block < 8` stays 1x (loop) + `tmem_row_block < 2` & `n_chunk < 2` unroll 2x ──► 221 virtual .b32 regs, 96 physical regs, `bar.warp.sync` outside loop (1992 TFLOPS) AFTER #1656 (without `#[unroll(never)]`): LLVM `opt -O2` unrolls `col_block < 8` 8x (16 `tcgen05.ld` + 16 `stmatrix` in 1 block!) and leaves `n_chunk < 2` un-unrolled ──► 279 virtual .b32 regs, 99 physical regs, `bar.warp.sync` trapped inside `n_chunk` loop ──► multi-wave barrier/warp-sync deadlock on 16K ``` Furthermore, even when a loop carried `#[unroll(N)]`, `mir-transforms` partially unrolled it in MIR and erased `mir.unroll_hint` without emitting `!llvm.loop.unroll.disable` into the `.ll` file — allowing `opt -O2` to come behind `mir-transforms` and fully unroll the loop anyway. #### Fix: `#[unroll(never)]` + `!llvm.loop.unroll.disable` Propagation ```text Rust source: #[unroll] ──► full unroll in `mir-transforms` (removes loop header) #[unroll(N)] ──► partial unroll by N in `mir-transforms` + stamps `loop_unroll_disable` #[unroll(never)] ──► keeps 1x body in `mir-transforms` + stamps `loop_unroll_disable` │ ▼ `mir-lower` copies onto `llvm.br` backedge │ ▼ `llvm-export` emits `!llvm.loop !id` with `!{!"llvm.loop.unroll.disable"}` ``` Annotating the inner `while col_block` epilogue loops in `gemm_sol_final` with `#[unroll(never)]` keeps `SMEM_OUT` in `addrspace(3)` while restoring the exact 1x inner loop + 2x outer unroll structure (`64` regs on `M256xN256`, `96` regs on `M512xN256`). --- ### 2. Normalize Non-Generic Pointer `$0` in `inline_asm_convergent` (`mir-lower`) Several inline-PTX intrinsics (`tcgen05_alloc`, `tcgen05_alloc_cg2`, `tcgen05_commit*`, and `stmatrix` under `LibNvvm`) pass a raw pointer (`*mut u32` / `*mut u64` / `*mut u8`) as `$0` into inline PTX that expects a generic pointer (`addrspace(0)`): ```text Rust kernel: tcgen05_alloc_cg2(&raw mut TMEM_ADDR as *mut u32, 512); Inline PTX template: cvta.to.shared.u64 %shared64, $0; // <-- expects $0 in addrspace(0) (generic)! tcgen05.alloc.cta_group::2.sync.aligned.shared::cta.b32 [%shared32], $1; ``` After #1656, `&raw mut TMEM_ADDR as *mut u32` and `&raw mut ACCUM_FULL0 as *mut u64` stay in `addrspace(3)`. `inline_asm_convergent` now inserts an `AddrSpaceCastOp` to `addrspace(0)` when `$0` is a non-generic pointer and the template expects a generic pointer (`cvta.to.shared.u64 ..., $0;` or `tcgen05.commit.*`). --- ## Verification * **`ptxas -arch=sm_100a -v` (`gemm_sol_final`)**: * `gemm_sol_clc_multicast_4_stage_pipeline` (`M256xN256`): **`69 -> 64` registers**, `0` spills * `gemm_sol_clc_multicast_4_stage_pipeline_large` (`M512xN256`): **`99 -> 96` registers**, `0` spills * **Blackwell B200 (`sm_100a`) `gemm_sol_final` Correctness & Benchmark (`GEMM_SOL_MODE=both`)**: | Problem Size | Kernel Variant | Before This PR | **After This PR** | vs. Live `cuBLASLt` | |:-------------|:---------------|:---------------|:------------------|:--------------------| | **4096³** | `M256xN256` (`64` regs) | `1470.9 TFLOPS` (`69` regs) | **`1470.3 TFLOPS` (`93.48 us`)** | `79.9%` (`1839.3 TFLOPS`) | | **8192³** | `M256xN256` (`64` regs) | `1763.5 TFLOPS` (`69` regs) | **`1764.9 TFLOPS` (`622.98 us`)** | `85.0%` (`2077.2 TFLOPS`) | | **16384³** | `M512xN256` (`96` regs) | **DEADLOCK** (`99` regs) | **`1992.0 TFLOPS` (`4415.74 us`)** | **`96.5%` (`2063.9 TFLOPS`)** | * **CI Gates**: * `just fmt-check`, `just clippy`, `just doc-check`, `just check-intrinsics main`: **all PASS** * `tma_copy` + `tma_copy/verify-code-shape.sh`: **PASS** #### Test plan - [x] `cargo test -p cuda-macros -p dialect-mir -p mir-importer -p mir-transforms -p mir-lower -p llvm-export -p cuda-oxide-mlir-export` - [x] `just fmt-check && just clippy && just doc-check && just check-intrinsics main` - [x] `cargo oxide run tma_copy && bash crates/rustc-codegen-cuda/examples/tma_copy/verify-code-shape.sh` - [x] `GEMM_SOL_MODE=both cargo oxide run gemm_sol_final` on B200 (`sm_100a`) Generated with [Devin](https://devin.ai) Signed-off-by: nihalpasham <nihalp@nvidia.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What This PR Does
This PR makes three changes to shared-memory addressing and CTA-local TMA loads (following up on #1336, #1341, #1238, #1337, and #1249):
.add()and.cast()on shared pointers inaddrspace(3)(mir-importer)i32shared-memory offsets incuda-oxide-mlir-export(+#[grid_constant]inblockscale_gemm_cute)SharedPtr<T>(-3,141lines) and takedst: *mut u8directly incp_async_bulk_tensor_{1..5}d_g2s_cta(+ routectm-rsto.shared::cta)1. Keep
.add()and.cast()on Shared Pointers inaddrspace(3)(mir-importer)Previously,
SlotAddrSpaceMaponly preservedaddrspace(3)across plain_y = _xcopies. Calling.add(offset)or.cast::<T>()on a shared pointer demoted it to a 64-bit generic pointer (addrspace(0)), forcing anaddrspace(3) -> addrspace(0) -> addrspace(3)round-trip:2. Use 32-bit
i32Shared-Memory Offsets incuda-oxide-mlir-exportShared memory (
!llvm.ptr<3>) uses 32-bit addresses, butMirPtrOffsetOpand SM120mma/epiloguelowering incuda-oxide-mlir-exportwere extending 32-bit lane/warp indices to 64-biti64before indexing!llvm.ptr<3>.3. Remove
SharedPtr<T>and Takedst: *mut u8in.shared::ctaTMA (#1336, #1341, #1238, #1337, #1249)Why #1238 Happened on RTX 5090 (
sm_120a)Consumer Blackwell (
sm_120a) has no hardware support for cross-block TMA loads. When a kernel emits a Cluster TMA load (.shared::cluster),ptxasgenerates anif / elsewith a software emulationCALLin theelsebranch, which allocates an 8-byte stack frame and causesptxasto ignoresetmaxnreg(warningC7506):Why We Don't Need
SharedPtr<T>(Revising #1336 / #1341)PR #1336 (and #1341) introduced
SharedPtr<T>and lowered it toptr addrspace(3)across function signatures, return values, structs, and arrays (+3,141lines of plumbing). That created a fundamental layout mismatch:rustcalways sizesSharedPtr<T>at 8 bytes (*mut T).p3:32:32:32(libNVVM/--nvptx-short-ptr), LLVM sizesptr addrspace(3)at 4 bytes, causing return-type mismatches inllcandmisaligned addresscrashes in arrays/structs.With Change 1 (
SlotAddrSpaceMap) +convert_g2s_cta(which already accepts bothaddrspace(3)andaddrspace(0)pointers, just likebarrier: *mut Barrier),cp_async_bulk_tensor_{1..5}d_g2s_ctaworks directly withdst: *mut u8:We also updated
ctm_rs::tma::cp_async_bulk_tensor_shared_cta_globalto callcp_async_bulk_tensor_2d_g2s_cta(.shared::cta) instead ofcp_async_bulk_tensor_2d_g2s(.shared::cluster), eliminating the 8-byte stack frame and runtime CTA-id check inblockscale_gemm_ctm.Verification
scripts/smoketest.shon RTX 5090 (sm_120a), includingtma_copy/verify-code-shape.sh(.shared::ctaemitted,0cvta.to.shared,0shared::cluster).just check(fmt-check,clippy,test,test-cuda,check-guards,check-intrinsics main,doc-check): all PASS.scripts/check-primitives-api-compile.py(api,fp16,blockscale): all PASS.setmaxnreg+.shared::ctaonsm_120a(ptxas -v+nvdisasm):168registers,0stack frame,USETMAXREG.DEALLOC.CTAPOOL 0x28,USETMAXREG.TRY_ALLOC.CTAPOOL UP0, 0xe8,UTMALDG.2D, noCALL.nsys profilekernel-active median of 31 launches):10.880 us)reference/benchmark.py)cutlass-mlir10.880 us10.880 usblockscale_gemm_cutecutlass-mlir11.360 us10.912 us-0.032 us(-0.3%)blockscale_gemm_ctmLLVM/PTX11.616 us10.976 us-0.096 us(-0.9%)