Skip to content

Implement generated mbarrier_arrive_no_complete intrinsic - #1101

Merged
nihalpasham merged 3 commits into
NVIDIA:mainfrom
uurl:fix/mbarrier-arrive-no-complete
Aug 30, 2026
Merged

nihalpasham merged 3 commits into
NVIDIA:mainfrom
uurl:fix/mbarrier-arrive-no-complete

Conversation

@uurl

@uurl uurl commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

Closes #774.

Summary

Implements mbarrier_arrive_no_complete(bar, count) -> u64 through the
generated intrinsic catalog/ABI path and removes the stale handwritten stub.

The intrinsic maps to:

llvm.nvvm.mbarrier.arrive.noComplete.shared

and lowers to:

mbarrier.arrive.noComplete.shared.b64

with a dynamic u32 arrival count and returned u64 barrier state.

Changes

  • adds ABI entry i1016

  • adds mbarrier_arrive_no_complete to the mbarrier_basic catalog family

  • exposes:

    pub unsafe fn mbarrier_arrive_no_complete(
        bar: *const Barrier,
        count: u32,
    ) -> u64
  • removes the stale handwritten no-count/no-return implementation

  • generates the public CUDA API and compatibility path

  • adds MbarrierArriveNoCompleteSharedOp

  • adds MIR importer and lowering support

  • adds LLVM NVPTX lowering through:

    llvm.nvvm.mbarrier.arrive.noComplete.shared
    
  • adds libNVVM lowering through inline PTX:

    mbarrier.arrive.noComplete.shared.b64
    
  • adds verifier coverage for:

    • two operands
    • dynamic i32 count
    • i64 result
  • adds the generated LLVM probe

  • adds LLVM and CUDA 13.3/libNVVM evidence

  • updates the barrier example with a regression that verifies the no-complete phase behavior using the returned barrier state

Validation

  • cargo test -p cuda-intrinsics-gen

    • 293 passed
  • cargo run -p cuda-intrinsics-gen -- check

    • passed
  • cargo test -p dialect-nvvm

    • 51 passed
  • cargo test -p mir-lower

    • 263 passed
  • MIR lowering tests

    • 117 passed
  • cargo fmt --all -- --check

    • passed
  • git diff --check

    • passed
  • compile-only smoketest coverage

    • full suite: 215 / 216 passed
    • the remaining cutile_inter_kernel failure was traced to missing CUDA toolkit host headers; after completing the isolated CUDA 13.3 toolkit, its targeted compile-only rerun passed 1 / 1
    • all 216 examples were therefore observed compiling successfully on the same code revision

LLVM NVPTX

The typed NVVM intrinsic was compiled for sm_80 and emitted:

mbarrier.arrive.noComplete.shared.b64

with the dynamic count operand and returned barrier state preserved.

CUDA 13.3 ptxas accepted the generated PTX for sm_80.

libNVVM

CUDA 13.3 libNVVM successfully compiled the production barrier no-complete regression to LTOIR.

nvJitLink subsequently produced an sm_80 cubin containing the
barrier_no_complete_test kernel.

Runtime validation

Physical runtime validation on sm_80 hardware remains unexecuted.

The evidence profiles record this explicitly as:

runtime_validation: unexecuted

Compile, PTX assembly, libNVVM, and device-link validation for sm_80 have all completed successfully.

@nihalpasham nihalpasham added on-hold Paused: valid but deferred until a prerequisite decision or dependency resolves Depends Blocked on another PR/issue or an upstream dependency post-migration-pr intrinsics Device intrinsics and libdevice math mappings device-apis User-facing device-side and kernel-authoring APIs (cuda-device, cuda-macros) IR-lowering Lowering between dialects (dialect-mir to LLVM dialect) cuda-feature A CUDA hardware/toolkit feature (TMA, tcgen05, WGMMA) labels Aug 24, 2026
@nihalpasham

nihalpasham commented Aug 24, 2026 •

Copy link
Copy Markdown
Collaborator

Putting this PR on hold until the pointer-contract refactor is stable and the branch can be reviewed on that architecture.

Review can resume after the refactor rebase, ABI regeneration, and runtime evidence required by the Discord acceptance criteria.

uurl and others added 2 commits August 30, 2026 14:50
…ated catalog (NVIDIA#774)

Port of PR NVIDIA#1101 onto current main. The old hand-written stub in
cuda-device had no count argument and no importer arm, so any kernel
calling it died late in device codegen:

  before: mbarrier_arrive_no_complete(bar)         -> "Symbol
          cuda_device__barrier__mbarrier_arrive_no_complete not found"
  after:  mbarrier_arrive_no_complete(bar, count)  -> u64 opaque state,
          mbarrier.arrive.noComplete.shared.b64 $0, [$1], $2;

What lands here (the hand-written half; regeneration follows separately):

- overlay entry in mbarrier_basic.toml + ABI ledger entry
  (renumbered i1016 -> i1017; main took i1016 for ex2_approx_f16)
- generator recipe MbarrierBasicOperation::ArriveNoComplete and the
  ptx_modifier -> ptx_modifiers generalization (noComplete needs four
  modifiers, the old field hard-coded [op, "shared", "b64"])
- render arms: ABI docs, compat shim, dialect op
  MbarrierArriveNoCompleteSharedOp (2 opds / 1 result, u32 count +
  u64 state verifier), importer dispatch, lowering dispatch, probe,
  reference
- hand-written lowering convert_arrive_no_complete in mir-lower:
  LlvmNvptx -> llvm.nvvm.mbarrier.arrive.noComplete.shared,
  LibNvvm   -> inline PTX "=l,l,r,~{memory}"
- stub deletion in cuda-device/src/barrier.rs
- barrier example regression kernel: init(N+1), noComplete arrive(N)
  leaves the phase open (test_wait false), one final arrive completes
  it (true); host checks [0, 1]
- negative dialect tests + wrong-adapter and renderable-shape
  generator tests

Signed-off-by: Raul Estrada <raulestradaa@gmail.com>
… (i1017) + LLVM-23 evidence

Mechanical follow-up to the previous commit; no hand-written logic.

- cargo run -p cuda-intrinsics-gen -- generate (1304 outputs: catalog.json,
  device/dialect/importer/lowering/collector code, probes, reference doc)
- ABI id is i1017 on current main (PR NVIDIA#1101 recorded i1016, which main
  spent on ex2_approx_f16)
- evidence, re-recorded on this machine instead of porting the PR's
  LLVM-22 records:
  - rust-llvm-23.1.0-16696adc: typed llvm.nvvm.mbarrier.arrive.noComplete.shared
    probe; llc emitted exact mbarrier.arrive.noComplete.shared.b64 and
    CUDA 13.3 ptxas (V13.3.33) accepted it for sm_80
  - cuda-13.3-libnvvm-13.3.33: barrier example built via libNVVM to
    7,056-byte LTOIR, nvJitLink linked a 12,704-byte sm_80 cubin;
    cuobjdump shows barrier_no_complete_test sections and the counted
    arrival lowered as ATOMS.ADD on the barrier state
- gates run locally: generate, check, check-abi-history --base-ref
  origin/main (1016 base entries preserved, 1017 total), probe --all
  --skip-terminal --per-target (2366 target configurations)

Signed-off-by: nihalpasham <nihalp@nvidia.com>
@nihalpasham
nihalpasham force-pushed the fix/mbarrier-arrive-no-complete branch from 2596dac to 2026dcb Compare August 30, 2026 10:10
@nihalpasham

Copy link
Copy Markdown
Collaborator

Took this off hold and finished it ourselves; every blocker on it was churn from our side (the #1190 generator split, the LLVM 23 bump, #1104 taking your ABI id), and the design was already agreed in the #774 thread. Your commit is the base of the new head with authorship and sign-off intact.

What the port changed (mechanics only, zero design changes)

  • ABI id i1016 -> i1017 (main spent i1016 on ex2_approx_f16); ledger check passes against main: "preserves 1016 base entries, 1017 total"
  • generator/render/importer/lowering hunks re-homed into the post-split module layout (resolve.rs, render.rs and friends no longer exist as files)
  • evidence re-recorded rather than ported: your rust-llvm-22.1.2 profile is obsolete, so the record was re-attested end to end under rust-llvm-23.1.0 with the local CUDA 13.3 ptxas, and the libNVVM record's LTOIR/cubin sizes+hashes are from local runs (nvJitLink 13.3, SASS shows the counted arrival)

Runtime evidence, the part the hold was really about

Before (old stub, on main):  kernel calling mbarrier_arrive_no_complete
  -> [rustc_codegen_cuda] Device codegen failed: ... Symbol
     cuda_device__barrier__mbarrier_arrive_no_complete not found  (exit 101)

After (this branch, RTX 5090):  cargo oxide run barrier
  init(N+1); state = arrive_no_complete(&BAR, N)
  test_wait(state) == false     // phase held open   -> out[0] = 0
  arrive(&BAR)                  // one normal arrival
  test_wait(state) == true      // phase completes   -> out[1] = 1
  all three barrier tests pass

Gates on the final head: generate/check round-trip clean, probe gate (2366 target configs across 1017 routes), full workspace tests, fmt/clippy, per-example gates, lockfile check.

One ordering note: #1139 also wants the ledger tail. This branch assumes it lands first; whichever goes second regenerates, which is one command on our side. Dropping the on-hold/Depends labels; CI approval + landing next.

@nihalpasham nihalpasham added codegen Device code-generation pipeline (Rust MIR to IR to PTX) and removed Depends Blocked on another PR/issue or an upstream dependency on-hold Paused: valid but deferred until a prerequisite decision or dependency resolves labels Aug 30, 2026
The mlir-dialects page pins the catalog SHA and hand-kept op counts;
status-guard machine-checks the stamp. mbarrier_arrive_no_complete
moves it:

  stamp    49bcd626 -> 02a6a6b3
  ops      575 -> 576 (generated 549 -> 550), entries 1016 -> 1017
  row      cp_async/mbarrier_*/sync  34 -> 35

Both tables re-summed against the greps the guard prescribes.

Signed-off-by: nihalpasham <nihalp@nvidia.com>
@nihalpasham
nihalpasham merged commit 1c0b81a into NVIDIA:main Aug 30, 2026
52 checks passed
nihalpasham added a commit to almightychang/cuda-oxide that referenced this pull request Aug 30, 2026
…il to i1018-i1025

NVIDIA#1101 landed mbarrier_arrive_no_complete as i1017, which this branch
had claimed for the eight ordered sparse float forms. Resolution:

  - abi-v1.toml: main's i1017 kept, our eight entries appended as
    i1018-i1025 (same relative order)
  - the two count asserts move to the merged totals (1025 entries,
    65 shards); the pinned-binding tests move to 1018..=1025
  - every generated file resolved by regeneration, not by hand
  - book dialect page re-stamped (20bc41c2, 1025 entries; op counts
    unchanged, the float forms share the sparse-MMA structural op)

Re-validated: generate/check round-trip, ledger history vs main
(preserves 1017 base entries, 1025 total), probe gate (2374 configs /
1025 routes), per-target terminal probes of all eight plus the
mbarrier form, 310 generator tests, fmt, all nine status-guard
scripts, lockfile check, and a fresh RTX 5090 run of
sparse_mma_ordered_float (all 8 variants match the host GEMM).

Signed-off-by: nihalpasham <nihalp@nvidia.com>
@uurl
uurl deleted the fix/mbarrier-arrive-no-complete branch August 31, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working codegen Device code-generation pipeline (Rust MIR to IR to PTX) cuda-feature A CUDA hardware/toolkit feature (TMA, tcgen05, WGMMA) device-apis User-facing device-side and kernel-authoring APIs (cuda-device, cuda-macros) intrinsics Device intrinsics and libdevice math mappings IR-lowering Lowering between dialects (dialect-mir to LLVM dialect)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

importer does not recognize mbarrier_arrive_no_complete; kernels fail verification with symbol-not-found

2 participants