[donghanbj]

Summary

Restores the VLM vision encoders that stopped compiling after #1306. On current main (7f2f3ce0), six of the seven VLMs we can run locally fail to compile; with this change all seven compile and run with no CPU fallback. Gemma-4 E4B decode at a 128-token prompt goes from 7.9 to 51.0 TPS.

Related issue or design

Regression from #1306 (d5979f76), reported on that PR. The NTS OGA dynamic dashboard has had no results for these VLMs since 10-24.

Why

#1306 runs the forward shape rules in hip-infer-loop-body-shapes on ranked results that still have a ?, and replaces the stored type with the rule's result. A rule can know less than the importer. For opset-10 Slice the bounds are operands, so the rule reports every extent dynamic. On Gemma-4 E4B the pass widened 736 result types and refined none. The vision RoPE Slice went from 1x?x12x32 to ?x?x?x?. Slice conversion then built tensor.extract_slice with static sizes and the widened result type, which fails verification:

error: expected type to be 'tensor<1x?x12x32xf16>' or a rank-reduced version. (size mismatch)
error: failed to legalize operation 'hip.rope' that was explicitly marked illegal
error: 'hip.loop' op result type #0 ('tensor<1x?x1152xf16>') must match v_init type #0 ('tensor<?x?x?xf16>')

With session.disable_cpu_ep_fallback=1, as in CI, the session cannot be created. Without it, eight Gemma-4 E4B subgraphs fall back to the CPU EP.

Separately, #1306 gives an equal Split of a static axis static chunk lengths. When the declared result type leaves that axis dynamic, the slice was still built with the declared type and failed verification.

What

• InferLoopBodyShapesPass: refineResultType merges the rule's type into the stored one per dimension. A static extent is kept, a dynamic one takes the rule's value, and a conflicting static extent leaves the result unchanged. An unranked result still takes the rule type. The forward-inference write-back uses it instead of setType(type).
• SplitToStdTensor: build tensor.extract_slice with the type inferred from its sizes, and tensor.cast to the declared type when the two differ.
• lit: refinement_keeps_known_extents in hip-infer-loop-body-shapes.mlir, and an equal split with a dynamic declared result in test_split.mlir.

Test plan

• lit on 7f2f3ce0 + this branch: 662 passed, 9 unsupported.
• pre-commit run --files on the four changed files: all hooks passed.
• VLMs on Strix Halo (gfx1151, Windows, OGA) with the CI AMDGPU genai_config (session.disable_cpu_ep_fallback=1), dog.jpg, 128-token prompt, 32 generated tokens, 1 iteration. Decode TPS:

Model	main 7f2f3ce	This PR
gemma3-4b-it-rtn-int4-128gs-fp16-onnx-gpu-pruned_lm_head	68.6	68.7
gemma-4-E4B-it-awq-uint4-onnx-split-fp16	compile error	47.5
gemma-4-12b-it-fp16-int4-bs32-kquant	compile error	22.2
google-gemma-4-26B-A4B-it-FP16W4-qmoe	compile error	44.9
Qwen3.5-9B-fp16-ve-fp16-int4-text-gs32-dml	compile error	36.4
Qwen3.6-27B-fp16-ve-fp16-int4-rtn-text-gs32-dml	compile error	12.9
Qwen3.8-27B-fp16-ve-fp16-int4-k_quant-gs128-text-dml-onnx	compile error	13.7

Qwen3.6-27B and Qwen3.8-27B ran at 13.3 and 14.2 TPS in the 10-23 NTS run, before #1306. Not covered locally: Nemotron-3-Nano-Omni (onnxruntime-genai rejects nemotron_omni) and Mistral-Small-4-119B (does not fit in memory on this machine).

Performance

Gemma-4 E4B (awq-uint4), prompt	main decode / TTFT	This PR decode / TTFT
128 (388 tokens incl. image)	7.9 TPS / 4261 ms	51.0 TPS / 337 ms
2k (2279 tokens)	not run	49.6 TPS / 1119 ms
16k (16375 tokens)	not run	42.7 TPS / 7986 ms

Hardware: Strix Halo (gfx1151), Windows, OGA vlm_benchmark, dog.jpg, 128 generated tokens, 1 warmup + 3 iterations, CPU fallback allowed. On main eight subgraphs fall back to the CPU EP. The 2k and 16k prompts were not run on main because of that fallback.

Notes for reviewers

Drafted with Cursor assistance. I reviewed the change, the lit and pre-commit results, and the measurements.

Checklist

• The change is focused, or links a design/series explaining its scope.
• Relevant tests were added or updated and the results are documented.
• User-facing or design documentation was updated when needed.
• Substantial AI assistance is disclosed, and I reviewed and understand the result.

[github-actions]

Thanks for opening a PR!

This project follows LLVM's incremental-development and AI-tool-use
guidance. See CONTRIBUTING.md
for the project workflow.

Before requesting review, please check that:

1. The change is focused. Substantial work links the relevant issue
   or design discussion.
2. The PR documents relevant test results and updates affected
   documentation.
3. If AI tools provided substantial assistance, the description
   explains what was assisted and how it was validated, and commit
   trailers identify the tool. The contributor has reviewed and
   understands the result.

Reviewers are assigned through
CODEOWNERS where ownership
is configured.

[github-actions]

L2 Accuracy Results (EP vs CPU)

Model	Combined L2	Total Elems	Skipped NaN/Inf
conv_test_hybrid	1.12256E-07	64	0
GroupQueryAttention_seq256	59.2561	2621440	0
MatMulNBits_o_seq128	259.905	368640	0
QMoE_seq128	24.2583	368640	0

Threshold: 0.01 | Run: 7958 - Commit: 0485a63

[github-actions]

Binary sizes

Baseline: run 40457802257 - ae1dc4f (dll-sizes)

File	This (MB)	Main (MB)	Delta (MB)
hipgpu.dll	59.32	59.31	+0.01
custom_kernels_gfx1170.dll	5.55	5.55	0.00
custom_kernels_gfx11-generic.dll	5.79	5.79	0.00

MorphiZen EP Performance Results

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	8.72	7.40	249	3	1158
GroupQueryAttention_seq128	11513.30	2.57128	1	5	229
matmul_down_seq128	512.25	3.32	125	3	273

EPContext Export Performance

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	8.82	48.16	249	3	15504

EPContext Import Performance

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	8.82	10.45	252	3	15638

OGA Benchmark Results

Model	Warmup	Reps	Prompt Len	Gen Tokens	TTFT (ms)	TPS	Peak Mem (GB)	GPU Mem (GB)
gpt-oss-20b-webgpu-int4-rtn-block-32	1	5	128	128	94.5	74.2	1.25	12.24
Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml	1	5	128	128	119.2	47.7	1.14	5.23

OGA Wheel Smoke (Python benchmark_e2e.py)

Model	TTFT (ms)	TPS
Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml	117	48.0

VLM Benchmark Results

Model	TTFT (ms)	TPS	Output
gemma-4-E4B-it-fp16-int4-kquant-prune-logits	3993.5	7.8	This is a high-quality, outdoor photograph featuring a beautiful, fluffy white dog, likely a West Highland White Terrier or a si
Qwen3.8-27B-fp16-ve-fp16-int4-k_quant-gs128-text-dml-onnx	4925.4	14.0	The user wants a detailed description of the image. Let me carefully observe all elements before writing. Subject: A white do

TTFT / TPS vs main

Baseline: run 40457802257 - ae1dc4f | Gate: TTFT +20% / TPS -20%

Suite	Model	TTFT (ms)	Main	Delta	TPS	Main	Delta
oga	gpt-oss-20b-webgpu-int4-rtn-block-32	94.5	91.5	+3.2%	74.2	74.2	0.0%
oga	Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml	119.2	116.9	+2.0%	47.7	47.6	+0.2%
vlm	gemma-4-E4B-it-fp16-int4-kquant-prune-logits	3993.5	4141.2	-3.6%	7.8	7.2	+7.9%
vlm	Qwen3.8-27B-fp16-ve-fp16-int4-k_quant-gs128-text-dml-onnx	4925.4	14900.2	-66.9%	14.0	14.0	+0.1%

Run: 7958 - Commit: 0485a63

[donghanbj]

Closing in favor of #1336, which now covers both fixes in this PR (per-dimension merge of inferred types, and the static equal-Split cast) and also fixes the hip.loop result/v_init mismatch in its second commit afa8642. On the same main (7f2f3ce), #1336 restores all 7 VLMs listed above, with the same decode TPS as this PR. #1346 (revert of #1306) also restores them.