Repository navigation
Conversation
…ectors A shape vector built from a dim and a small constant, e.g. the Gemma-4 KV-sharing layers' ConstantOfShape(Concat(Shape(q)[0:1], [0, 512])), is read back element by element to size the result. Upstream folds stop at the tensor.insert_slice and hip.constant has no folder, so every element reached runtime as a small device-to-host memref.copy plus a stream sync: 36 per decode step on Gemma-4 E4B. Phase 0 of --hip-infer-shapes now resolves a constant-index tensor.extract through the insert_slice chain to the scalar it reads (a tensor.from_elements element or an inline integer hip.constant element). Only that scalar is materialised; where the constant tensor is stored is still decided by hip-externalize-constants. Gemma-4 E4B AWQ uint4 decode on gfx1151 (default settings), over main: 51.63 -> 58.50 TPS (+13.3%, 388-token prompt), 50.08 -> 56.20 (+12.2%, 2k), 43.43 -> 47.67 (+9.8%, 16k); +9.4% to +11.6% with HIPDNN_EP_MATMUL_DP4A=0. Generated text unchanged. Co-Authored-By: Cursor Claude Opus 5.5 <cursoragent@cursor.com> Made-with: Cursor
donghanbj
requested review from
CaymanYang,
amd-bqiao,
amd-mingw,
asparkhi,
edelaye,
fhanuman,
qianglin-amd,
sambanerxilinxamd and
wcy123
as code owners
October 25, 2026 11:00
|
Thanks for opening a PR! This project follows LLVM's incremental-development and AI-tool-use Before requesting review, please check that:
Reviewers are assigned through |
L2 Accuracy Results (EP vs CPU)
Threshold: 0.01 | Run: 8249 - Commit: |
Binary sizesBaseline: run 41321195111 -
MorphiZen EP Performance Results
EPContext Export Performance
EPContext Import Performance
OGA Benchmark Results
OGA Wheel Smoke (Python benchmark_e2e.py)
VLM Benchmark Results
TTFT / TPS vs mainBaseline: run 41321195111 -
Run: 8249 - Commit: |
Contributor
Author
|
/run-test models=all node=xsjnucstrhalo09 |
Contributor
|
ROCm GenAI CI finished with result: UNSTABLE.
|
fhanuman
reviewed
Oct 28, 2026
| /// reads: an element of a tensor.from_elements, or of an inline integer | ||
| /// hip.constant. Only that scalar is materialised; where the constant tensor | ||
| /// itself is stored is still decided by hip-externalize-constants. | ||
| struct FoldConstantIndexShapeExtract |
Collaborator
There was a problem hiding this comment.
How is this related to shape inference? Can this be a separate pass?
| }; | ||
|
|
||
| Value src = op.getTensor(); | ||
| for (int depth = 0; depth < 16; ++depth) { |
Collaborator
There was a problem hiding this comment.
It's not clear why this threshold 16 was chosen?
… pass The fold does not depend on any hip-infer-shapes state, so it now runs as func.func(hip-fold-constant-extracts) immediately before hip-infer-shapes, which is back to its main version. - Walk the insert_slice chain without a depth cap; each step moves to a dominating operand, so the walk terminates. - Rewrite only the resolved extracts and erase only the producers they leave without uses, instead of a greedy rewrite of the whole module, so functions without a resolvable extract are unchanged. - Resolve any integer constant at the end of the chain, not only an inline hip.constant, and take the element attribute as is. - Cover multi-level walks, multi-dimensional indices, index elements and the unresolved slice forms in the lit test; document the pass. Co-Authored-By: Cursor Claude Opus 5.5 <cursoragent@cursor.com> Made-with: Cursor
donghanbj
changed the title
Contributor
Author
|
/run-test models=all node=strix-halo-genai |
Contributor
|
ROCm GenAI CI finished with result: UNSTABLE.
|
Contributor
Author
|
/run-test models=all node=xsjnucstrhalo09 |
Contributor
|
ROCm GenAI CI finished with result: SUCCESS.
|
Co-Authored-By: Cursor Claude Opus 5.5 <cursoragent@cursor.com> Made-with: Cursor
Contributor
Author
|
@fhanuman Thanks for the review. Both points are addressed in 6abcefe:
Could you take another look? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A new
func.func(hip-fold-constant-extracts)pass, run immediately before--hip-infer-shapes, folds constant-indextensor.extracts of vectors assembled withtensor.insert_slice, so those shape elements no longer reach runtime as small device-to-host copies. Gemma-4 E4B decode is 12-13% faster on gfx1151.Related issue or design
None.
Why
A shape vector built from a dim and a small constant, e.g. the Gemma-4 E4B KV-sharing layers'
ConstantOfShape(Concat(Shape(q)[0:1], [0, 512])), is read back element by element to size the result. Upstream folds stop at thetensor.insert_slice, andhip.constanthas no folder, so every element reached runtime as a small device-to-hostmemref.copy: 36hipMemcpyAsyncper decode step on Gemma-4 E4B (ktrace, 45 -> 9 per step with this PR), about 20 of which block the host for more than 100 us while the stream drains. With the copies gone, the host waits once at the existing end-of-step sync instead of stalling the GPU at every layer.What
hip-fold-constant-extracts(lib/Dialect/Transforms/FoldConstantExtracts.cpp): resolves a constant-indextensor.extractthrough static-offset, unit-strideinsert_slices to the scalar it reads: atensor.from_elementsoperand, an inline integerhip.constantelement, or another integer constant element. Dynamic indices, dynamic or strided or rank-reducing slices, and other producers keep the extract.hip-infer-shapes, so reification sees these scalars as constants, and beforehip-externalize-constants, which may replace the inlinehip.constantpayloads it reads. Where the constant tensor is stored is still decided byhip-externalize-constants.test/lit/Dialect/hip-fold-constant-extracts.mlir; pass documented indocs/pipeline_pass_menu.md,docs/design/hip-shape-inference.mdanddocs/design/constant-handling-design.md.Test plan
hip-mlir-opt/hip-compilerandhipgpu.dllon main 3c569ca:MorphizenMLIRLitTests664 passed (including the new test), 9 unsupported, 0 failed.For every compile of the models with no folds, the IR after the pass is identical to its input.
Performance
Decode TPS, Gemma-4 E4B AWQ uint4, gfx1151 (Windows), main 3c569ca, 128 generated tokens, mean of 2 rounds x 3 iterations:
ROCm GenAI CI, same model on node
xconucstrhalo17(gfx1151, Windows),LLM_ROCm_dynamic, 128 generated tokens, 3 repetitions after 1 warmup. The PR build is the current code (fc8e2ec; the later head only merges main again), and both packages are on maincf8ff09:TTFT is within 1% of main at every prompt length, and PPL (
PPL_VLM_seqallgpu) is identical: 246.481288 for both.The earlier
/run-test models=allresult above (Jenkins #2814) was measured on the first revision of this PR, before the fold moved into its own pass.Checklist
AI assistance (Cursor, Claude Opus 5.5) was used for profiling analysis, the implementation, the lit test and the benchmark runs. I reviewed the changes and validated them with the lit suite and the end-to-end runs above.