Repository navigation
feat(compiler): keep stored shapes and lower integer conv and matmul - #1337
Draft
amd-mounikk wants to merge 15 commits into
Draft
amd-mounikk wants to merge 15 commits into
amd-mounikk wants to merge 15 commits into
Conversation
Decompose DynamicQuantizeLinear, ConvInteger, and MatMulInteger into existing HIP ops so a quantized graph can bufferize on both the HIP and rocMLIR paths. Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com> Made-with: Cursor
DynamicQuantizeLinear used an out-of-range axis on a vector, ReduceMin and ReduceMax rejected f32, and MatMulInteger checked batch broadcast without expanding it. An f32 dot product also drops bits once the reduction passes 258, so longer ConvInteger and MatMulInteger reductions now accumulate in int32. Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com> Made-with: Cursor
An i32 matmul passes B's batch stride so a shared matrix is not read past its end, and an integer convolution is not rewritten into hip.gemm. Wide i32 anchors stay on the HIP kernels. The f32 reduction clears the HIP last-error state before launch. Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com> Made-with: Cursor
Copying a rank-2 B into every batch allocates a separate matrix for a weight hip.matmul already reuses. Wide batched i32 matmul and an all-zero quantize now have exact numeric tests. Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com> Made-with: Cursor
The wide batched i32 matmul and the all-zero quantize now assert the execution provider output. A 259-channel i32 convolution stays hip.conv through the fusion pass. Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com> Made-with: Cursor
Co-authored-by: nvunnam57128 <261134987+nvunnam57128@users.noreply.github.com>
The i32 conv kernel was only checked on a 1x1 window. A reduction of 261 with stride, padding, two groups, and several output positions now checks the execution provider result. Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com> Made-with: Cursor
keepdims=0 drops the reduced axis, so trailing shape alignment treats a repeated extent as the inner suffix and reduces the wrong dimension. Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com> Made-with: Cursor
Widen 1-D NCL ConvInteger to the shared NC1L hip.conv path and collapse the result so int8 encoder convolutions bufferize. Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com> Made-with: Cursor
A data-dependent leading batch left ConvInteger and MatMulInteger unconverted. The runtime already reads that extent from the memref descriptor, so those ops can bufferize. Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com> Made-with: Cursor
A static rank-2 weight can be shared when the batch or M axis is dynamic. An uninferred K on A still accumulates in i32, and a per-row zero point still needs a static M. Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com> Made-with: Cursor
…t-integer Keep the int32 matmul declaration alongside the channels-last causal convolution and GatedDeltaNet declarations. Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com> Made-with: Cursor
…t-integer Keep the int32 matmul declaration alongside the DynamicSparseAttention and SparseAttentionIndexer declarations. Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com> Made-with: Cursor
A rule that only knows the rank was replacing a stored dimension with ?. Merge each rule with the stored type, read constant bound inputs, and rank AveragePool and attribute-form Upsample. Co-authored-by: Cursor <cursoragent@cursor.com>
amd-mounikk
requested review from
CaymanYang,
amd-bqiao,
amd-mingw,
asparkhi,
edelaye,
fhanuman,
qianglin-amd,
sambanerxilinxamd and
wcy123
as code owners
October 26, 2026 14:42
|
Thanks for opening a PR! This project follows LLVM's incremental-development and AI-tool-use Before requesting review, please check that:
Reviewers are assigned through |
github-actions
Bot
added
the
large-pr
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new shape-inference paths can produce incorrect extents and perform an undefined integer conversion for common Slice bounds.
3 open issues
What changed in this PR
Combines the stored-shape preservation fix from #1336 with the integer convolution and matmul support from #1248, extending hip-ep’s quantized graph compilation.
Changes:
- Preserve known result dimensions, infer additional ONNX shapes, and refine constant Split sizes.
- Lower DynamicQuantizeLinear, ConvInteger, and MatMulInteger with supporting runtime kernels and compiler routing.
- Add conversion, pipeline, and numeric tests, and document supported operations.
| File | Description |
|---|---|
| test/numeric/tests/test_reduce.py | Adds f32 ReduceMax coverage. |
| test/numeric/tests/test_reduce_min.py | Adds f32 ReduceMin coverage. |
| test/numeric/tests/test_quantized_conv.py | Tests dynamic quantization and integer convolution. |
| test/numeric/tests/test_matmul_integer.py | Tests integer matmul and batch handling. |
| test/lit/Transforms/HipFusionTransform/patch_embed_conv.mlir | Checks integer convolution avoids GEMM fusion. |
| test/lit/e2e/test_quantized_conv_model.mlir | Checks quantized convolution pipeline lowering. |
| test/lit/e2e/test_matmul_integer_wide.mlir | Checks wide reductions use integer matmul. |
| test/lit/e2e/test_matmul_integer_model.mlir | Checks quantization-to-matmul pipeline lowering. |
| test/lit/e2e/test_matmul_integer_dynamic.mlir | Covers dynamic matmul batches. |
| test/lit/e2e/test_matmul_integer_dynamic_sequence.mlir | Covers dynamic matmul sequence lengths. |
| test/lit/e2e/test_matmul_integer_dynamic_k.mlir | Covers uninferred contraction dimensions. |
| test/lit/e2e/test_integer_stays_on_hip.mlir | Checks integer operations remain on HIP. |
| test/lit/e2e/test_conv_integer_wide.mlir | Checks wide integer convolution routing. |
| test/lit/e2e/test_conv_integer_dynamic.mlir | Covers dynamic convolution batches. |
| test/lit/e2e/test_conv_integer_1d.mlir | Covers rank-3 convolution lowering. |
| test/lit/Dialect/hip-infer-loop-body-shapes.mlir | Tests shape preservation and added inference rules. |
| test/lit/Conversion/onnx-to-hip/test_split.mlir | Checks constant Split result extents. |
| test/lit/Conversion/onnx-to-hip/test_matmul_integer.mlir | Tests integer matmul conversion variants. |
| test/lit/Conversion/onnx-to-hip/test_dynamic_quantize_linear.mlir | Tests dynamic quantization decomposition. |
| test/lit/Conversion/onnx-to-hip/test_conv_integer.mlir | Tests integer convolution conversion variants. |
| test/lit/Conversion/hip-to-tosa/test_matmul.mlir | Checks integer matmul zero-point operands. |
| test/lit/Conversion/hip-to-tosa/test_conv.mlir | Checks integer convolution TOSA operands. |
| test/lit/Conversion/hip-to-llvm/test_reduce_min.mlir | Tests reduction strides with repeated dimensions. |
| test/lit/Conversion/hip-to-llvm/test_matmul.mlir | Checks shared-weight integer matmul stride. |
| lib/Runtime/real/reduce_min.cpp | Enables f32 ReduceMin dispatch. |
| lib/Runtime/real/reduce_max.cpp | Enables f32 ReduceMax dispatch. |
| lib/Runtime/real/matmul.cpp | Adds the integer matmul runtime wrapper. |
| lib/Runtime/real/conv.cpp | Enables i32 convolution dispatch. |
| lib/Runtime/mock/mock_gpu.cpp | Adds the mock integer matmul wrapper. |
| lib/Runtime/Kernels/include/hip_custom_kernels.h | Declares integer matmul and documents dtype support. |
| lib/Runtime/Kernels/hip/reduce_sum_kernel.hip | Adds f32 generic reduction dispatch. |
| lib/Runtime/Kernels/hip/matmul_i32_kernel.hip | Implements integer matmul accumulation. |
| lib/Runtime/Kernels/hip/conv_kernel.hip | Implements integer convolution accumulation. |
| lib/Runtime/Kernels/hip/cast_kernel.hip | Adds signed int8 casts and clears launch errors. |
| lib/Runtime/Kernels/CMakeLists.txt | Builds the integer matmul kernel. |
| lib/Runtime/hipdnn_ep_runtime.h | Declares the integer matmul runtime ABI. |
| lib/Dialect/Transforms/InferLoopBodyShapesPass.cpp | Preserves stored sizes and extends shape inference. |
| lib/Dialect/Transforms/fusion_pattern/patch_embed_conv_to_gemm.cpp | Excludes integer convolution from GEMM fusion. |
| lib/Dialect/Transforms/FuseROCMlir.cpp | Keeps i32 anchors on HIP kernels. |
| lib/Conversion/OnnxToHip/ReshapeConversion.cpp | Refines constant Split slice extents. |
| lib/Conversion/OnnxToHip/OnnxToHipUtils.h | Declares the new conversion registrations. |
| lib/Conversion/OnnxToHip/OnnxToHip.cpp | Registers the new ONNX conversions. |
| lib/Conversion/OnnxToHip/MatMulIntegerConversion.cpp | Decomposes integer matmul with accumulator selection. |
| lib/Conversion/OnnxToHip/DynamicQuantizeLinearConversion.cpp | Decomposes dynamic quantization into HIP operations. |
| lib/Conversion/OnnxToHip/ConvIntegerConversion.cpp | Decomposes integer convolution with accumulator selection. |
| lib/Conversion/OnnxToHip/CMakeLists.txt | Builds the new conversion sources. |
| lib/Conversion/HipToTosa/HipToTosa.cpp | Adds explicit integer zero-point operands. |
| lib/Conversion/HipToLLVM/ReduceLowering.cpp | Derives reduction strides from constant axes. |
| lib/Conversion/HipToLLVM/MatmulLowering.cpp | Routes i32 matmul to its runtime wrapper. |
| lib/Conversion/HipToLLVM/HipToLLVMUtils.h | Defines the integer matmul wrapper symbol. |
| docs/supported-operations.md | Documents quantized integer operation support. |
Review effort: Balanced
Comment on lines
+456
to
+462
| for (int64_t axis : *axisValues) { | ||
| if (axis < 0) | ||
| axis += rank; | ||
| if (axis < 0 || axis >= rank) | ||
| return {}; | ||
| axes.push_back(axis); | ||
| } |
Comment on lines
+763
to
+764
| int64_t out = quot + 1; | ||
| shape[2 + s] = out > 0 ? out : 0; |
Comment on lines
+1287
to
+1290
| for (double element : tensor->data) { | ||
| if (!std::isfinite(element) || element != std::trunc(element)) | ||
| return std::nullopt; | ||
| values.push_back(static_cast<int64_t>(element)); |
Kernel Unit Test Results
Leaves: 16 (0 failed) | Cases: 289 (0 failed) Mode: lookup | Coverage: 1 | Run: 7972 - Commit: |
L2 Accuracy Results (EP vs CPU)
Threshold: 0.01 | Run: 7972 - Commit: |
Binary sizesBaseline: run 40457802257 -
MorphiZen EP Performance Results
EPContext Export Performance
EPContext Import Performance
OGA Benchmark Results
OGA Wheel Smoke (Python benchmark_e2e.py)
VLM Benchmark Results
TTFT / TPS vs mainBaseline: run 40457802257 -
Run: 7972 - Commit: |
CI LLVM prefixes include AMDGPU and use a new cache key so rocMLIR can link. The mock job keeps the flag off because it has no ROCm SDK. Co-authored-by: Cursor <cursoragent@cursor.com>
pragyam32
force-pushed
the
gpuep/shape-merge-plus-quant-integer
branch
from
October 27, 2026 09:50
6e190c7 to
e7ec78e
Compare
amd-mounikk
marked this pull request as draft
October 27, 2026 12:41
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
main(7579ebf7).Related issue or design
Combines #1336 and #1248. #1336 follows up the Gemma-4 shape regression reported on #1306.
Why
The two changes do not touch the same lines. This branch is the single base that has the stored-shape compiler fix and the integer conv/matmul lowering together.
What
main: ConvInteger, MatMulInteger, DynamicQuantizeLinear, and the matching kernels and tests.43e43f96: merge each shape rule with the stored type, read constant bound inputs, rank AveragePool and attribute-form Upsample, and write a constant Split chunk size ontoextract_slice.Test plan
pre-commit run --fileson all 51 files changed versusmain(passed: whitespace, end-of-file, large files, BOM, line endings, merge conflict, lintrunner, licenseheaders)hip-mlir-opt --hip-infer-loop-body-shapes test/lit/Dialect/hip-infer-loop-body-shapes.mlir | FileCheck(passed)hip-mlir-opt test/lit/Conversion/onnx-to-hip/test_split.mlir --hip-add-context-arg --convert-onnx-to-hip | FileCheck(passed)Notes for reviewers
The #1248 commits are unchanged. The shape commit is the #1336 patch cherry-picked onto that tip.
Drafted with Cursor assistance. I reviewed the stack, the pre-commit result, and the two FileCheck runs.
Checklist