Skip to content

fix(cuda-oxide-mlir-export): saturate float to int casts - #1655

Merged
nihalpasham merged 3 commits into
NVIDIA:mainfrom
Rachit2323:fix/mlir-export-saturating-float-to-int
Oct 29, 2026
Merged

nihalpasham merged 3 commits into
NVIDIA:mainfrom
Rachit2323:fix/mlir-export-saturating-float-to-int

Conversation

@Rachit2323

@Rachit2323 Rachit2323 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What This PR Does

Fixes #1642 in cuda-oxide-mlir-export (cutlass-mlir backend) so Rust float-to-integer as casts (MirCastKindAttr::FloatToInt) saturate on out-of-range inputs and map NaN to 0, matching Rust semantics and mir-lower:

  1. Lower MirCastKindAttr::FloatToInt to llvm.fptosi.sat / llvm.fptoui.sat via llvm.call_intrinsic and reject integer destinations wider than 64 bits (cuda-oxide-mlir-export)
  2. Add unit tests covering f16 (FP16Type and MirFP16Type), f32, f64, and i128 / u128 rejection (cuda-oxide-mlir-export)

1. Why arith.fptosi / arith.fptoui Miscompiled Out-of-Range and NaN Casts

Previously, CastTranslation::translate in cuda-oxide-mlir-export/src/mir/core.rs lowered MirCastKindAttr::FloatToInt to MLIR arith.fptosi / arith.fptoui, which have poison semantics on NaN or out-of-range values:

BEFORE (arith.fptosi / arith.fptoui under cutlass-mlir on sm_120a):
  300.0f32 as u8    ──► 44          (wrapped low byte instead of clamping to 255)
  128.0f32 as i8    ──► -128        (wrapped instead of clamping to 127)
  f32::NAN as i64   ──► i64::MIN    (returned -9223372036854775808 instead of 0)

AFTER (llvm.call_intrinsic "llvm.fpto{s|u}i.sat.i{W}.f{F}"):
  300.0f32 as u8    ──► 255         (clamped via FMNMX + F2I)
  128.0f32 as i8    ──► 127         (clamped to i8::MAX)
  f32::NAN as i64   ──► 0           (mapped to 0 via DSETP.NAN + SEL)
  • Rejects destination integer widths > 64 bits (i128 / u128) with a clear diagnostic before reaching libCutlassCompiler.so (which does not support 128-bit fptosi.sat / fptoui.sat in libNVVM), matching mir-lower (#878).

Verification

  • RTX 5090 (sm_120a) cutlass-mlir GPU Suite: 838 / 838 runtime edge-case checks (NaN, +inf, -inf, 2147483648.0f32 as i32, -0.0, out-of-range positive/negative) across (f16, f32, f64) -> (i8, u8, i16, u16, i32, u32, i64, u64, isize, usize) pass under both cutlass-full-cute-mlir22 and cutlass-primitives-mlir22 (98 / 838 failed on main).
  • Unit & Integration Tests:
    • cargo test --all-targets -p cuda-oxide-mlir-export (88 unit + 70 integration tests): PASS
    • cargo clippy -p cuda-oxide-mlir-export --all-targets -- -D warnings & cargo oxide fmt --check: PASS
    • All 18 scripts/check-*.sh repository guards: PASS

Closes #1642

Fixes NVIDIA#1642.

Signed-off-by: Rachit2323 <atreyrachit23@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nihalpasham nihalpasham added the cuda-oxide SIMT programming model: rustc backend, cargo oxide, cuda-device, cuda-host, book label Oct 29, 2026
Signed-off-by: nihalpasham <nihalp@nvidia.com>
…int tests

Signed-off-by: nihalpasham <nihalp@nvidia.com>
@nihalpasham nihalpasham added bug Something isn't working codegen Device code-generation pipeline (Rust MIR to IR to PTX) IR-lowering Lowering between dialects (dialect-mir to LLVM dialect) labels Oct 29, 2026
@nihalpasham

Copy link
Copy Markdown
Collaborator

Thanks, emitting llvm.fptosi.sat / llvm.fptoui.sat and rejecting > 64-bit integer destinations brings cuda-oxide-mlir-export in line with mir-lower. Layered one small test commit (831bd1f76):

  • Added dialect_mir::types::MirFP16Type (the f16 type produced by mir-importer) and unsigned u128 coverage to the unit tests in mir/core.rs.

At 831bd1f76, all cuda-oxide-mlir-export tests, all 18 guard checks, and a 26-kernel cutlass-mlir GPU test suite on RTX 5090 (sm_120a, 838/838 runtime edge cases across f16/f32/f64 -> i8..u64) pass. Merging once CI is green.

@nihalpasham

Copy link
Copy Markdown
Collaborator

/ok to test 831bd1f

@nihalpasham
nihalpasham merged commit a259743 into NVIDIA:main Oct 29, 2026
69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working codegen Device code-generation pipeline (Rust MIR to IR to PTX) cuda-oxide SIMT programming model: rustc backend, cargo oxide, cuda-device, cuda-host, book IR-lowering Lowering between dialects (dialect-mir to LLVM dialect)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: cutlass-mlir backend doesn't clamp float -> int as casts

2 participants