[cliu1003]

Summary

Improve correctness of dynamic-shape lowering, output allocation, device copies, and temporary-buffer handling.

The changes preserve runtime dimensions through Reshape and Slice lowering, keep eligible copies on device paths, allocate graph outputs with the correct external shape, and materialize host-written scalar buffers before generic allocation reuse.

Related issue or design

None.

Why

Several transformations relied on information that is insufficient for dynamic shapes:

• Equal dynamic tensor types do not prove that runtime dimensions are unchanged.
• Dynamic Slice bounds may be produced by operations that the local host-arithmetic analysis cannot interpret.
• A dynamic stride on a statically singleton dimension may still be derived from the physical inner layout.
• Rank-expanding output views may require multiple independent runtime dimensions that cannot be reconstructed from their product.
• Host-written scalar staging allocations must be separated from ordinary device allocations before lifetime-based reuse.

These cases could produce incorrect output extents, stale shape values, invalid host-side copies of device memory, or output descriptors with the wrong rank.

What

• Require proof that every runtime dimension is unchanged before removing a same-type dynamic Reshape.
• Preserve the existing host-arithmetic fast path for Slice bounds and use the shared synchronized scalar-readback path when the producer chain cannot be resolved.
• Derive effective singleton-dimension strides from the actual inner physical layout for device copies.
• Allocate rank-expanded graph outputs using their precise external dimensions while preserving ABI casts.
• Complete output view analysis before erasing the original allocation.
• Materialize recognized host-written scalar staging buffers before generic allocation reuse.
• Keep OptimizeMemRefs target-independent and run layout promotion after reuse.
• Add focused regression coverage for each behavior and for the production pass order.

Test plan

• Ran pre-commit on all modified files.
  • All hooks passed.
• Completed a full Release rebuild successfully.
• Ran targeted memory-copy, Reshape, Slice, output-allocation, and pipeline lit tests.
  • All targeted tests passed.
• Verified the public-helper output-allocation regression case.
  • The compiler completed without a crash.
• Ran text-and-image and text-and-audio end-to-end smoke tests.
  • Both paths completed successfully.
  • No shape-broadcast warnings or runtime errors were observed.

Performance

The pass reorder does not introduce additional kernels, copies, or synchronization.

The Slice fallback uses the shared scalar-readback infrastructure. In the validated graph, the required scalar was already host-visible through an earlier shape operation, so the readback count did not increase and no measurable performance regression was observed.

Ordinary device allocations continue to use OptimizeMemRefs; only recognized host-written scalar staging allocations are moved to host scratch before reuse.

Notes for reviewers

• Dynamic Reshape elimination remains conservative. Unknown or reordered runtime dimensions retain the Reshape.
• Slice no longer extends its local producer-chain dependencies. Unsupported producer chains use the shared scalar-readback path.
• MaterializeHostScalars handles target-specific host placement before the generic reuse pass.
• PromoteStridedHipOperands remains after OptimizeMemRefs so layouts introduced during reuse are normalized before pooling.
• The graph-output-count filtering experiment was removed because it was not required for the reproduced failure and incorrectly applied module metadata to all public functions.
• No user-facing documentation changes are required because this change only corrects compiler lowering behavior.

AI assistance disclosure

AI assistance was used to analyze compiler IR behavior, review the implementation, draft regression tests, investigate failures, and prepare the pull request description. All changes were reviewed, understood, built, and validated with pre-commit, targeted tests, and end-to-end smoke tests.

Checklist

• The change is focused, or links a design/series explaining its scope.
• Relevant tests were added or updated and the results are documented.
• User-facing or design documentation was updated when needed.
• Substantial AI assistance is disclosed, and I reviewed and understand the result.

[github-actions]

Thanks for opening a PR!

This project follows LLVM's incremental-development and AI-tool-use
guidance. See CONTRIBUTING.md
for the project workflow.

Before requesting review, please check that:

1. The change is focused. Substantial work links the relevant issue
   or design discussion.
2. The PR documents relevant test results and updates affected
   documentation.
3. If AI tools provided substantial assistance, the description
   explains what was assisted and how it was validated, and commit
   trailers identify the tool. The contributor has reviewed and
   understands the result.

Reviewers are assigned through
CODEOWNERS where ownership
is configured.

[github-actions]

L2 Accuracy Results (EP vs CPU)

Model	Combined L2	Total Elems	Skipped NaN/Inf
conv_test_hybrid	1.12256E-07	64	0
GroupQueryAttention_seq256	59.2561	2621440	0
MatMulNBits_o_seq128	259.905	368640	0
QMoE_seq128	24.2583	368640	0

Threshold: 0.01 | Run: 7880 - Commit: 459287c

[github-actions]

Binary sizes

Baseline: run 40209447587 - 702a948 (dll-sizes)

File	This (MB)	Main (MB)	Delta (MB)
hipgpu.dll	59.28	59.28	0.00
custom_kernels_gfx1170.dll	5.56	5.56	0.00
custom_kernels_gfx11-generic.dll	5.85	5.85	0.00

MorphiZen EP Performance Results

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	8.78	7.23	269	3	1158
GroupQueryAttention_seq128	11360.30	2.42436	1	5	230
matmul_down_seq128	508.48	3.13	124	3	276

EPContext Export Performance

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	8.82	48.90	245	3	15504

EPContext Import Performance

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	8.83	8.92	251	3	15639

OGA Benchmark Results

Model	Warmup	Reps	Prompt Len	Gen Tokens	TTFT (ms)	TPS	Peak Mem (GB)	GPU Mem (GB)
gpt-oss-20b-webgpu-int4-rtn-block-32	1	5	128	128	92.3	74.3	1.25	12.24
Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml	1	5	128	128	125.1	47.7	1.15	5.23

OGA Wheel Smoke (Python benchmark_e2e.py)

Model	TTFT (ms)	TPS
Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml	124	48.0

VLM Benchmark Results

Model	TTFT (ms)	TPS	Output
Qwen3.8-27B-fp16-ve-fp16-int4-k_quant-gs128-text-dml-onnx	4777.7	14.0	The user wants a detailed description of the image. Let me carefully observe all elements before writing. Subject: A white do

Run: 7880 - Commit: 459287c

[cliu1003]

Hi @fhanuman, could you please help to review again? thanks!

[fhanuman]

Could we also please make sure this change doesn’t cause any performance regression?

[cliu1003]

/run-test models=all

[amd-yanjunz]

ROCm GenAI CI finished with result: UNSTABLE.

• PR: fix(compiler): correct memory handling for dynamic shapes #1272
• Source comment: models=all
• TEST_MODE: performance,accuracy
• MODELZOO: LLM_ROCm_dynamic
• TEST_MODELS: (all)
• FORBID_MODELS: (none)
• NODE: (GenAI job default)
• CI_OPTIONS: (none)
• Jenkins build: http://xsjvitisaijenkins:8080/job/daily_hip_ep_genai/2584/
• Main baseline commit: 76c6664
• Main baseline: not usable, no baseline result for this commit yet
• Package build run: https://github.com/ROCm/hip-ep/actions/runs/39740180465
• NTS report: http://xsjnts.xilinx.com/ROCm/hipdnn-detail?database=development&collection=daily_hip_ep_genai&id=6ada110c848fa63c277aa7c3
• Compare with latest daily baseline (fallback): http://xsjnts.xilinx.com/ROCm/compare?base_url=http%3A%2F%2Fxsjnts.xilinx.com%2FROCm%2Fhipdnn-detail%3Fids%3D6ad9baedf27f47880d1b66a8%26collection%3DHIP-EP-Dynamic&base_job=&base=&curr_url=&curr_job=daily_hip_ep_genai&curr=2584

[cliu1003]

Hi @fhanuman All review comments have been addressed. CI showed some performance variation for a few models, but the baseline and current results were collected on different machines. I also ran a controlled same-machine A/B test with only the pass order changed. The TPS differences were 0.76% and 0.35%, which are within normal run-to-run variation, so no measurable regression was observed.

Could you please take another look? Thank you.

[fhanuman]

What is the expectation with this new change for the below IR? Won't this be a problem?

func.func @main_graph(
      %ctx: !hip.context, %a: index, %b: index)
      -> memref<4x4xf32> {
    // Assume %a = 4 and %b = 4 at runtime.
    %buffer = memref.alloc() : memref<16xf32>

    %dynamic = memref.cast %buffer
        : memref<16xf32> to memref<?xf32>

    %expanded = memref.expand_shape %dynamic [[0, 1]]
        output_shape [%a, %b]
        : memref<?xf32> into memref<?x?xf32>

    %result = memref.cast %expanded
        : memref<?x?xf32> to memref<4x4xf32>

    return %result : memref<4x4xf32>
  }

[fhanuman]

I see around a 20% drop in performance in some cases in the comparison report
(http://xsjnts.xilinx.com/ROCm/compare?base_url=http%3A%2F%2Fxsjnts.xilinx.com%2FROCm%2Fhipdnn-detail%3Fids%3D6ad9baedf27f47880d1b66a8%26collection%3DHIP-EP-Dynamic&base_job=&base=&curr_url=&curr_job=daily_hip_ep_genai&curr=2584),
so I am not sure we can attribute this to machine differences. Please confirm this once on the same machine, with and without all the changes in this PR for all the models that we run in the CI/CD.

For example, please see the PR below for how I have been qualifying the changes:

#724

We should make sure our compiler changes do not cause performance regressions. @amd-mingw, @qianglin-amd , could you please share your thoughts here?