You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(onnx-to-hip): lower quantized integer conv and matmul - #1248
Lower ONNX DynamicQuantizeLinear, ConvInteger, and MatMulInteger through existing HIP ops so a quantized graph bufferizes and can run on both the HIP and rocMLIR paths.
DynamicQuantizeLinear becomes single-axis reductions, scale and zero-point math, and hip.quantize_linear with axis 0. ConvInteger and MatMulInteger accumulate in f32 when 65025 * K fits in the 24-bit mantissa, and in i32 otherwise.
A size-1 batch on A is expanded before hip.matmul. A rank-2 B, and a B whose batch dimensions are all 1, stays one shared matrix. A dynamic leading batch is preserved for ConvInteger. For MatMulInteger, a static rank-2 B may also be shared across a dynamic M and any other leading dimension. An uninferred K on A accumulates in i32. Add int8-to-f32 and int8-to-i32 casts so signed inputs of those integer ops have a kernel.
Related issue or design
None.
Why
A quantized graph failed one-shot bufferization on unconverted onnx.ConvInteger, onnx.DynamicQuantizeLinear, and onnx.MatMulInteger. With CPU fallback disabled, the execution provider skipped the graph. These ops decompose into HIP ops that already lower through hipDNN/hipBLASLt and through TOSA on the rocMLIR path.
hip.qlinear_conv and hip.qlinear_matmul are a different contract: they requantize. Graphs that use ConvInteger and MatMulInteger already cast, scale, and add bias afterward, so those fused ops would apply the scales twice.
A data-dependent leading batch also left both integer ops unconverted. The convolution and matmul runtimes already read that extent from the memref descriptor. MatMulInteger was still rejected when M, or any later leading dimension, was dynamic. The matmul runtime already reads M and the batch product from A's descriptor. B's K and N stay static, so the result shape and a per-column zero point remain compile-time values. When A's K is dynamic, the reduction length has no compile-time bound and the accumulator stays i32.
What
DynamicQuantizeLinear (uint8 only): chain of single-axis ReduceMin / ReduceMax to a scalar, scale (max - min) / 255 with a zero range selecting scale 1, ties-to-even zero point, then hip.quantize_linear at axis 0. Axis 0 is in range for a vector. ReduceMin and ReduceMax accept f32, which this decomposition reduces. The f32 reduction clears the HIP last-error state before launch.
A constant single reduce axis supplies inner_size. With keepdims=0, matching trailing extents treats a repeated dimension as the inner suffix and reduces the wrong axis. [1,6,5,5] reduced on the last axis now passes inner_size=1, not 5.
ConvInteger: rank-3 NCL and rank-4 NCHW. Rank-3 widens to the shared rank-4 NC1L hip.conv path and collapses the result. The batch may be dynamic when the input and result agree and the channels, spatial dimensions, and weights are static. The activation zero point is a scalar. The weight zero point is a scalar or one value per output channel, reshaped to [M, 1, 1, 1] so the 4D broadcast does not place it on W. An integer convolution is not rewritten to hip.gemm.
MatMulInteger: rank 2-4. A size-1 batch on A is expanded to the result batch, because hip.matmul counts batches from A. A rank-2 B, and a higher-rank B whose batch dimensions are all 1, stays one shared [K, N] matrix. Copying it once per batch would allocate a separate weight for every batch, and both the i32 kernel (batch stride 0) and tosa.matmul (A's batch folded into M) already reuse it. B is expanded only when a batch dimension is larger than 1. A static rank-2 B may be shared across dynamic batch and sequence dimensions, including a dynamic M. B's K and N stay static. An uninferred K on A still lowers and accumulates in i32; that runtime length must match B's K. A per-row zero point requires a static M, because its length is M. A per-column zero point is reshaped to [..., 1, N].
An 8-bit value minus an 8-bit zero point is at most 255 in magnitude, so each product is at most 65025. The f32 accumulator is used only when the reduction length is at most 258. A longer reduction casts to i32 and accumulates in int64 before storing a wrapping int32. A uint8 dot product of length 259 filled with 255 is 16,841,475.
Wide i32 matmul and convolution stay in main_graph under the rocMLIR outline pass, so they lower through wrap_matmul_i32 and wrap_conv. HIP-to-TOSA still builds those ops with explicit zero points and an i32 accumulator when a kernel asks for them.
hip_cast clears the HIP last-error state before launch and returns hipGetLastError() afterward. The int32 matmul and conv launches do the same.
Test plan
LIT conversion: test_dynamic_quantize_linear.mlir, test_conv_integer.mlir, test_matmul_integer.mlir, including rank-3 NCL ConvInteger, a dynamic batch on rank-3 and rank-4 ConvInteger, a dynamic leading batch on MatMulInteger, a dynamic M (1x?xK), a dynamic prefix (?x?xK), an uninferred K (?x?x?) that stays on the i32 accumulator, a rejected per-row zero point when M is dynamic, axis 0, a size-1 batch expanded on A, a rank-2 B left unexpanded, a size-1 batch on B collapsed to a shared matrix, and the i32 wide-K forms
LIT pipeline (--hipdnn-pipeline): test_conv_integer_1d.mlir, test_conv_integer_dynamic.mlir, test_quantized_conv_model.mlir, test_matmul_integer_model.mlir, test_matmul_integer_dynamic.mlir, test_matmul_integer_dynamic_sequence.mlir, test_matmul_integer_dynamic_k.mlir, test_conv_integer_wide.mlir (wrap_conv, no wrap_gemm), and test_matmul_integer_wide.mlir (wrap_matmul_i32, no hipBLASLt)
HIP-to-LLVM: an i32 matmul with A [2, 1, 2] and rank-2 B passes batch stride 0
HIP-to-LLVM: a constant last axis on [1,6,5,5] -> [1,6,5] passes inner_size=1, and axis 1 of [4,4,4] -> [4,4] passes inner_size=4 (test_reduce_min.mlir). test_reduce_sum.mlir and test_reduce_mean.mlir still pass
HIP-to-TOSA with verification: i32 conv and batched i32 matmul check explicit zero-point operands and an i32 accumulator
--rocmlir-pipeline on a wide i32 matmul and convolution leaves both ops in main_graph (test_integer_stays_on_hip.mlir)
--hip-fusion-transform: a 259-channel 1x1 i32 convolution stays hip.conv (patch_embed_conv.mlir)
GPU numeric, CPU fallback disabled: length-259 matmul with several rows, columns, and distinct per-batch weights; the same shape reusing one rank-2 weight; a 259-channel 1x1 convolution; a grouped, padded convolution with reduction 261, stride 2, and several output positions and channels; an all-zero quantize whose scale is 1 and zero point is 0; and f32 ReduceMin/ReduceMax. All passed.
Notes for reviewers
Rank-1 MatMulInteger (numpy's vector promotion) and rank above 4 stay unconverted. hip.matmul requires rank >= 2, and hip.sub rejects rank > 4. ConvInteger supports rank-3 or rank-4; rank-3 uses the shared rank-4 kernel path. The int32 convolution kernel is spatial rank 2. A dynamic channel, spatial, or non-leading batch dimension stays unconverted for ConvInteger. A dynamic K or N on B, and a per-row zero point with a dynamic M, stay unconverted for MatMulInteger.
Cursor Grok 4.7 drafted the conversions, the Copilot fixes, the tests, and this description.
Checklist
The change is focused, or links a design/series explaining its scope.
Relevant tests were added or updated and the results are documented.
User-facing or design documentation was updated when needed.
Substantial AI assistance is disclosed, and I reviewed and understand the result.
This project follows LLVM's incremental-development and AI-tool-use
guidance. See CONTRIBUTING.md
for the project workflow.
Before requesting review, please check that:
The change is focused. Substantial work links the relevant issue
or design discussion.
The PR documents relevant test results and updates affected
documentation.
If AI tools provided substantial assistance, the description
explains what was assisted and how it was validated, and commit
trailers identify the tool. The contributor has reviewed and
understands the result.
Reviewers are assigned through CODEOWNERS where ownership
is configured.
The only DynamicQuantizeLinear numeric input spans zero, so it never exercises the scale=1 branch. Add an all-zero input case asserting scale=1, zero_point=0, and an all-zero quantized output. The conversion-only FileCheck tests do not validate these numerical results.
Preserve rank-2 B instead of copying it into every batch. For A [32, 1, 4096] and B [4096, 4096], this expansion materializes a 2 GiB i32 weight buffer rather than sharing the 64 MiB matrix, adding substantial memory use and copy traffic. MatmulLowering.cpp already supports rank-2 B with batch stride 0, and HipToTosa.cpp supports it by flattening A's leading dimensions into M. Expand B only when it has batch dimensions.
B with only size-1 batch dimensions is also one shared matrix, but this branch copies it across every result batch. For A=[32,1,4096] and B=[1,4096,4096], the expanded i32 weight buffer is 2 GiB instead of 64 MiB, potentially causing an avoidable out-of-memory failure. Reshape B to rank 2 when all its batch dimensions are 1; LLVM lowering can then use batch stride 0, and TOSA can use its shared-matrix path. Keep expansion for other batched shapes.
Reviewed the four linked findings. Their fixes and regression coverage are already present in the branch: 7d5f4b9 adds f32 reduction dispatch and DynamicQuantizeLinear coverage; 86eb960 adds i32 HIP-to-TOSA and rocMLIR-path coverage; 5183a42 covers wide-K integer numerics and prevents i32 convolution fusion. pre-commit run --all-files passes. No additional diff was needed, so I did not create an empty commit; the latest branch commit is 0f0cae8.
Decompose DynamicQuantizeLinear, ConvInteger, and MatMulInteger into existing HIP ops so a quantized graph can bufferize on both the HIP and rocMLIR paths.
Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com>
Made-with: Cursor
DynamicQuantizeLinear used an out-of-range axis on a vector, ReduceMin and ReduceMax rejected f32, and MatMulInteger checked batch broadcast without expanding it. An f32 dot product also drops bits once the reduction passes 258, so longer ConvInteger and MatMulInteger reductions now accumulate in int32.
Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com>
Made-with: Cursor
An i32 matmul passes B's batch stride so a shared matrix is not read past its end, and an integer convolution is not rewritten into hip.gemm. Wide i32 anchors stay on the HIP kernels. The f32 reduction clears the HIP last-error state before launch.
Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com>
Made-with: Cursor
Copying a rank-2 B into every batch allocates a separate matrix for a weight hip.matmul already reuses. Wide batched i32 matmul and an all-zero quantize now have exact numeric tests.
Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com>
Made-with: Cursor
The wide batched i32 matmul and the all-zero quantize now assert the execution provider output. A 259-channel i32 convolution stays hip.conv through the fusion pass.
Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com>
Made-with: Cursor
The i32 conv kernel was only checked on a 1x1 window. A reduction of 261 with stride, padding, two groups, and several output positions now checks the execution provider result.
Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com>
Made-with: Cursor
keepdims=0 drops the reduced axis, so trailing shape alignment treats a repeated extent as the inner suffix and reduces the wrong dimension.
Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com>
Made-with: Cursor
Widen 1-D NCL ConvInteger to the shared NC1L hip.conv path and collapse the result so int8 encoder convolutions bufferize.
Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com>
Made-with: Cursor
A data-dependent leading batch left ConvInteger and MatMulInteger unconverted. The runtime already reads that extent from the memref descriptor, so those ops can bufferize.
Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com>
Made-with: Cursor
A static rank-2 weight can be shared when the batch or M axis is dynamic. An uninferred K on A still accumulates in i32, and a per-row zero point still needs a static M.
Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com>
Made-with: Cursor
…t-integer
Keep main's reduced_axes stride and the constant axes-memref fallback. Shape inference stays the last resort, so a repeated extent still selects the axis named by the constant memref.
Co-Authored-By: Cursor Grok 4.7 <cursoragent@cursor.com>
Made-with: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
large-prSoft size threshold (>500 LOC or >10 files). Reviewer signal, not a block.
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
Bot
added
the
large-pr
Summary
DynamicQuantizeLinear,ConvInteger, andMatMulIntegerthrough existing HIP ops so a quantized graph bufferizes and can run on both the HIP and rocMLIR paths.DynamicQuantizeLinearbecomes single-axis reductions, scale and zero-point math, andhip.quantize_linearwith axis 0.ConvIntegerandMatMulIntegeraccumulate in f32 when65025 * Kfits in the 24-bit mantissa, and in i32 otherwise.hip.matmul. A rank-2 B, and a B whose batch dimensions are all 1, stays one shared matrix. A dynamic leading batch is preserved forConvInteger. ForMatMulInteger, a static rank-2 B may also be shared across a dynamic M and any other leading dimension. An uninferred K on A accumulates in i32. Add int8-to-f32 and int8-to-i32 casts so signed inputs of those integer ops have a kernel.Related issue or design
None.
Why
A quantized graph failed one-shot bufferization on unconverted
onnx.ConvInteger,onnx.DynamicQuantizeLinear, andonnx.MatMulInteger. With CPU fallback disabled, the execution provider skipped the graph. These ops decompose into HIP ops that already lower through hipDNN/hipBLASLt and through TOSA on the rocMLIR path.hip.qlinear_convandhip.qlinear_matmulare a different contract: they requantize. Graphs that useConvIntegerandMatMulIntegeralready cast, scale, and add bias afterward, so those fused ops would apply the scales twice.A data-dependent leading batch also left both integer ops unconverted. The convolution and matmul runtimes already read that extent from the memref descriptor.
MatMulIntegerwas still rejected when M, or any later leading dimension, was dynamic. The matmul runtime already reads M and the batch product from A's descriptor. B's K and N stay static, so the result shape and a per-column zero point remain compile-time values. When A's K is dynamic, the reduction length has no compile-time bound and the accumulator stays i32.What
DynamicQuantizeLinear(uint8 only): chain of single-axisReduceMin/ReduceMaxto a scalar, scale(max - min) / 255with a zero range selecting scale 1, ties-to-even zero point, thenhip.quantize_linearat axis 0. Axis 0 is in range for a vector.ReduceMinandReduceMaxaccept f32, which this decomposition reduces. The f32 reduction clears the HIP last-error state before launch.inner_size. Withkeepdims=0, matching trailing extents treats a repeated dimension as the inner suffix and reduces the wrong axis.[1,6,5,5]reduced on the last axis now passesinner_size=1, not 5.ConvInteger: rank-3 NCL and rank-4 NCHW. Rank-3 widens to the shared rank-4 NC1Lhip.convpath and collapses the result. The batch may be dynamic when the input and result agree and the channels, spatial dimensions, and weights are static. The activation zero point is a scalar. The weight zero point is a scalar or one value per output channel, reshaped to[M, 1, 1, 1]so the 4D broadcast does not place it on W. An integer convolution is not rewritten tohip.gemm.MatMulInteger: rank 2-4. A size-1 batch on A is expanded to the result batch, becausehip.matmulcounts batches from A. A rank-2 B, and a higher-rank B whose batch dimensions are all 1, stays one shared[K, N]matrix. Copying it once per batch would allocate a separate weight for every batch, and both the i32 kernel (batch stride 0) andtosa.matmul(A's batch folded into M) already reuse it. B is expanded only when a batch dimension is larger than 1. A static rank-2 B may be shared across dynamic batch and sequence dimensions, including a dynamic M. B's K and N stay static. An uninferred K on A still lowers and accumulates in i32; that runtime length must match B's K. A per-row zero point requires a static M, because its length is M. A per-column zero point is reshaped to[..., 1, N].main_graphunder the rocMLIR outline pass, so they lower throughwrap_matmul_i32andwrap_conv. HIP-to-TOSA still builds those ops with explicit zero points and an i32 accumulator when a kernel asks for them.hip_castclears the HIP last-error state before launch and returnshipGetLastError()afterward. The int32 matmul and conv launches do the same.Test plan
test_dynamic_quantize_linear.mlir,test_conv_integer.mlir,test_matmul_integer.mlir, including rank-3 NCL ConvInteger, a dynamic batch on rank-3 and rank-4 ConvInteger, a dynamic leading batch on MatMulInteger, a dynamic M (1x?xK), a dynamic prefix (?x?xK), an uninferred K (?x?x?) that stays on the i32 accumulator, a rejected per-row zero point when M is dynamic, axis 0, a size-1 batch expanded on A, a rank-2 B left unexpanded, a size-1 batch on B collapsed to a shared matrix, and the i32 wide-K forms--hipdnn-pipeline):test_conv_integer_1d.mlir,test_conv_integer_dynamic.mlir,test_quantized_conv_model.mlir,test_matmul_integer_model.mlir,test_matmul_integer_dynamic.mlir,test_matmul_integer_dynamic_sequence.mlir,test_matmul_integer_dynamic_k.mlir,test_conv_integer_wide.mlir(wrap_conv, nowrap_gemm), andtest_matmul_integer_wide.mlir(wrap_matmul_i32, no hipBLASLt)[2, 1, 2]and rank-2 B passes batch stride 0[1,6,5,5] -> [1,6,5]passesinner_size=1, and axis 1 of[4,4,4] -> [4,4]passesinner_size=4(test_reduce_min.mlir).test_reduce_sum.mlirandtest_reduce_mean.mlirstill pass--rocmlir-pipelineon a wide i32 matmul and convolution leaves both ops inmain_graph(test_integer_stays_on_hip.mlir)--hip-fusion-transform: a 259-channel 1x1 i32 convolution stayship.conv(patch_embed_conv.mlir)Notes for reviewers
MatMulInteger(numpy's vector promotion) and rank above 4 stay unconverted.hip.matmulrequires rank >= 2, andhip.subrejects rank > 4.ConvIntegersupports rank-3 or rank-4; rank-3 uses the shared rank-4 kernel path. The int32 convolution kernel is spatial rank 2. A dynamic channel, spatial, or non-leading batch dimension stays unconverted forConvInteger. A dynamic K or N on B, and a per-row zero point with a dynamic M, stay unconverted forMatMulInteger.Checklist