[masteryi-0018]

Summary

The fused QMoE decode (hip_qmoe_decode_fused and its dp4a variant) now picks its fc1/fc2 GEMV tiling before the first call of a shape computes anything. It no longer samples candidates on real decode calls, so the same input gives the same output on every call. Before, the first ~100 decode calls of a process could differ in the low bits.

Related issue or design

• This is patch 0036 of the hip-ir-runtime downstream patch series on hip-ep, originally written by @xingy16-amd. It is upstreamed here unchanged, with an error-state fix, a numeric test and a kernel unit test added on top. It applies on main on its own.
• Conflicts with perf: speculative-decode verify on gfx1151 (QMoE bursts, LinearAttention burst, TopK) + TopK largest=0 fix #1264 (QMoE verify bursts). perf: speculative-decode verify on gfx1151 (QMoE bursts, LinearAttention burst, TopK) + TopK largest=0 fix #1264 extends the per-call sampling tuner that this PR replaces: it adds a burst_key per burst size to the same tuner keys and launch calls. Whichever lands second has to be rebased:
  • if this PR lands first, perf: speculative-decode verify on gfx1151 (QMoE bursts, LinearAttention burst, TopK) + TopK largest=0 fix #1264's bursts should be timed up front the same way;
  • if perf: speculative-decode verify on gfx1151 (QMoE bursts, LinearAttention burst, TopK) + TopK largest=0 fix #1264 lands first, this PR's timing has to cover T-token bursts.

Why

• The fused decode chose (BLOCK_SIZE, TILE_N) for fc1/fc2 by sampling every candidate on real decode calls, round-robin: 12 candidates × 9 rounds per shape.
• The block size sets the reduction order. So for the first ~100 calls of a process, the same inputs gave slightly different outputs depending on which candidate a call happened to sample.
• On Gemma 4 26B, decode logits were off by up to 0.07 from one sequence to the next, and by up to 4.4 on the dp4a path, where int8 activations magnify it.

What

• fix(qmoe): Choose the decode GEMV tiling before computing, not on real calls (original patch):
  • The first call of a shape times every candidate up front: one discarded round, then 8 timed rounds; the least total time wins.
  • Each timing launch runs on a different set of k experts, so the weights are read cold as in steady-state decode.
  • The timing writes only the call's scratch, which the real launches then overwrite. That call and every later one compute with the chosen tiling.
  • While the stream is being captured, nothing is timed: the default tiling is used and the choice is left for a later call.
  • hip_qmoe_decode_fused(_dp4a) take num_experts for the timing sets.
• fix(qmoe): clear the HIP error state before the fused decode launches:
  • Every launch in the two entry points returns hipGetLastError(), but neither entry point cleared it first.
  • An error left by an earlier, unrelated call was therefore reported as a QMoE decode failure. With the up-front timing, it would also abandon the timing and fall back to the default tiling.
  • The state is now cleared once on entry, as the other custom kernels do.
• test(qmoe): check the fused decode gives the same output on every call:
  • Runs the single-token fused decode 129 times in one session with the same input, and requires every output to be bit-identical to the first. 129 is more than the old tuner's 108 sampling calls.
  • It takes the dp4a decode, the one the EP runs with 32-aligned blocks and sizes. The fp decode is covered by the kernel unit test below.
  • The tiling choice is cached per shape for the process, so it uses a shape no other QMoE test uses.
  • It is a shape where the reduction order shows in the fp16 output. Several other shapes probed (e.g. 704/1408, 2048/4096, 2816/1408) never showed a difference.
• test(qmoe): add a kernel unit test for the fused decode:
  • New unit_test/qmoe_decode/Xfp16_Wu4_Yfp16 leaf calling hip_qmoe_decode_fused and hip_qmoe_decode_fused_dp4a directly, registered with the package build, the suite controller and the local Makefile.
  • fp and dp4a × block size 32 / 64 / 128 × zero points × bias, on shapes COVERAGE thins (up to GPT-OSS-20B and a 128-expert, k = 8 layer).
  • Each case checks the first call against a CPU reference that rounds and quantizes where the kernels do, then 128 more calls on the same inputs, each bit-identical to the first.
  • Every tier also runs the first call of an untimed shape over a pending HIP error, and the arguments the launchers must refuse.

Test plan

Linux, gfx1151 (Radeon 8060S), ./docker/run.sh build, this branch on main 84c0443. Numeric tests run with HIPDNN_EP_STRICT=1; the framework disables CPU fallback.

test_qmoe.py	main	this PR
New test_qmoe_decode_repeatable (dp4a)	fails: call 3 differs from call 0	passes
Rest of the file	20 pass, 2 xfail, 1 fail	identical

New kernel unit test qmoe_decode/Xfp16_Wu4_Yfp16, coverage 3 (80 cases), linked against three kernels:

	main's kernel	without the error-state fix	this PR
Accuracy against the reference	all pass	all pass	all pass (relL2 ≤ 7.6e-5)
128 repeat calls bit-identical	25 cases fail (both larger shapes, from call 2, 3 or 8)	all pass	all pass
First call over a pending HIP error	fp and dp4a fail (status 101)	dp4a fails (status 101)	both pass
Rejected arguments	all pass	all pass	all pass

• Main's kernel has no num_experts argument; it was built with main's header and linked by name, which the C calling convention allows (the extra trailing argument is ignored).
• The numeric failure is test_qmoe_gpt_oss_shape[1]: the fused decode exceeds atol=0.1, with cosine 0.99996. Its diff is identical on both branches. test_qmoe_gpt_oss_shape_mix_weights[1] is already xfailed for the same reason.
• With HIPDNN_EP_DEBUG=1, the numeric test logs one autotune line for fc1 and one for fc2, and all 129 calls take the dp4a decode.
• LIT (check-hip-mlir-lit): 663 passed, 9 unsupported. pre-commit on the changed files passes.

Performance

gfx1151 (Radeon 8060S), Linux.

Steady-state call time from the kernel unit test's shapes, after the 129 repeat calls (main has finished sampling by then). Each of 200 timed calls uses a different set of k experts, so the weights are read cold as in a real decode step; mean of the 200, block size 32 / 64 / 128:

Shape	Decode	main	this PR	this PR / main
GPT-OSS-20B (2880 / 2880, 32 experts, k 4)	dp4a	267.5 / 252.1 / 241.3 µs	260.5 / 246.1 / 239.4 µs	0.97 / 0.98 / 0.99
GPT-OSS-20B	fp	289.2 / 274.0 / 267.3 µs	254.2 / 240.7 / 231.8 µs	0.88 / 0.88 / 0.87
2048 / 768, 128 experts, k 8	dp4a	116.4 / 111.6 / 136.8 µs	114.8 / 102.8 / 100.5 µs	0.99 / 0.92 / 0.73
2048 / 768, 128 experts, k 8	fp	146.5 / 141.7 / 139.6 µs	111.5 / 100.0 / 96.7 µs	0.76 / 0.71 / 0.69

With the same experts on every call (hot weights, the regime main's tuner samples in), main's picks can look faster instead: on the 128-expert layer, dp4a at block size 32 took 42.1 µs on main against 53.8 µs here. A real decode step reads each layer's weights cold.

Through the EP, GPT-OSS-20B decode with dp4a (block size 32), 300 calls in one session; GPU times from rocprofv3 --kernel-trace, wall times host-side session.run:

	main	this PR
QMoE kernels per call, steady state (median of last 100)	248.5 µs	244.8 µs
session.run median, last 100 calls	0.341 ms	0.338 ms
First call	6.4 ms	34.6 ms
Calls 1-108 total	41.6 ms	36.8 ms

• First call: Timing is paid once per shape per process, ~30 ms. The MoE layers of one model normally share a shape. In exchange, calls 1-108 no longer synchronize on per-call timing events.

Notes for reviewers

• Pre-existing, not changed here: the fp decode needs a block size that is a multiple of 32. Both GEMVs read one group's scale and zero point per 32 values, but hip_qmoe_decode_fused accepts any even block size and wrap_qmoe routes single-token SwiGLU QMoE to it whenever dp4a does not apply. At block size 16 every output is wrong (relL2 0.2-1.1 against the reference, on main's kernel and on this branch's alike). With dp4a on (the default), 32-aligned blocks take dp4a, so the fp decode is reached by default exactly for those block sizes. The new kernel unit test therefore runs only 32, 64 and 128.
• Pending error on the fp decode: without the error-state fix, fp's first tiling launch consumes the stale error, so the call still succeeds but keeps the default tiling, which the test cannot see; dp4a reports the error as its own, which the test catches.
• Kernel unit test on Linux: unit_test/CMakeLists.txt skips the package build off Windows, and the leaf's Makefile drives hipcc.exe. On Linux the leaf was built with TheRock's hipcc using the same compile and link commands; the package registration is left to the Windows kernel-ut CI job.
• HIP graph capture: A call made while its stream is being captured uses the default tiling and leaves the choice for a later uncaptured call. A graph captured before the first uncaptured call therefore keeps the default tiling, while eager calls after it use the timed one. main does not capture streams today.
• Timing launches: They go to the call's stream and write only that call's scratch (act_out / slot_buf), which the real launches overwrite.
• Concurrency: The choice is made under the existing tuner mutex. Concurrent first calls of the same shape wait for one timing.

AI assistance

• The tiling change was written by @xingy16-amd with Claude Code.
• The error-state fix, the tests, the validation and profiling runs, and this description were prepared with Claude Code (Claude Opus 5.5).
• All commits carry Co-Authored-By trailers.

Checklist

• The change is focused, or links a design/series explaining its scope.
• Relevant tests were added or updated and the results are documented.
• User-facing or design documentation was updated when needed.
• Substantial AI assistance is disclosed, and I reviewed and understand the result.

🤖 Generated with Claude Code

[github-actions]

Thanks for opening a PR!

This project follows LLVM's incremental-development and AI-tool-use
guidance. See CONTRIBUTING.md
for the project workflow.

Before requesting review, please check that:

1. The change is focused. Substantial work links the relevant issue
   or design discussion.
2. The PR documents relevant test results and updates affected
   documentation.
3. If AI tools provided substantial assistance, the description
   explains what was assisted and how it was validated, and commit
   trailers identify the tool. The contributor has reviewed and
   understands the result.

Reviewers are assigned through
CODEOWNERS where ownership
is configured.

[github-actions]

L2 Accuracy Results (EP vs CPU)

Model	Combined L2	Total Elems	Skipped NaN/Inf
conv_test_hybrid	1.12256E-07	64	0
GroupQueryAttention_seq256	59.2561	2621440	0
MatMulNBits_o_seq128	259.905	368640	0
QMoE_seq128	24.2583	368640	0

Threshold: 0.01 | Run: 7968 - Commit: e48ff3c

[github-actions]

Binary sizes

Baseline: run 40457802257 - ae1dc4f (dll-sizes)

File	This (MB)	Main (MB)	Delta (MB)
hipgpu.dll	59.32	59.31	+0.01
custom_kernels_gfx1170.dll	5.56	5.55	+0.01
custom_kernels_gfx11-generic.dll	5.80	5.79	+0.01

MorphiZen EP Performance Results

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	8.77	7.49	247	3	1159
GroupQueryAttention_seq128	11288.10	2.56029	1	5	228
matmul_down_seq128	509.59	3.29	121	3	275

EPContext Export Performance

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	8.82	49.10	248	3	15505

EPContext Import Performance

Model	QPS	Session (s)	1st Infer (ms)	CPU%	Mem (MB)
full_model_seq128	8.82	10.06	252	3	15639

OGA Benchmark Results

Model	Warmup	Reps	Prompt Len	Gen Tokens	TTFT (ms)	TPS	Peak Mem (GB)	GPU Mem (GB)
gpt-oss-20b-webgpu-int4-rtn-block-32	1	5	128	128	92.4	74.2	1.25	12.24
Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml	1	5	128	128	117.9	47.8	1.15	5.23

OGA Wheel Smoke (Python benchmark_e2e.py)

Model	TTFT (ms)	TPS
Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml	117	47.9

VLM Benchmark Results

Model	TTFT (ms)	TPS	Output
gemma-4-E4B-it-fp16-int4-kquant-prune-logits	4021.3	7.9	This is a high-quality, outdoor photograph featuring a beautiful, fluffy white dog, likely a West Highland White Terrier or a si
Qwen3.8-27B-fp16-ve-fp16-int4-k_quant-gs128-text-dml-onnx	14719.8	14.0	The user wants a detailed description of the image. Let me carefully observe all elements before writing. Subject: A white do

TTFT / TPS vs main

Baseline: run 40457802257 - ae1dc4f | Gate: TTFT +20% / TPS -20%

Suite	Model	TTFT (ms)	Main	Delta	TPS	Main	Delta
oga	gpt-oss-20b-webgpu-int4-rtn-block-32	92.4	91.5	+0.9%	74.2	74.2	+0.1%
oga	Llama-3.1-8B-awq-g128-int4-asym-fp16-onnx-dml	117.9	116.9	+0.9%	47.8	47.6	+0.5%
vlm	gemma-4-E4B-it-fp16-int4-kquant-prune-logits	4021.3	4141.2	-2.9%	7.9	7.2	+9.3%
vlm	Qwen3.8-27B-fp16-ve-fp16-int4-k_quant-gs128-text-dml-onnx	14719.8	14900.2	-1.2%	14.0	14.0	+0.1%

Run: 7968 - Commit: e48ff3c

[github-actions]

Kernel Unit Test Results

Op	Leaf	Arch	Cases	Failed	Suite (ms)	Verdict
matmul_nbits	Xfp16_Wu2_Yfp16	gfx1151	12	0	6463	PASS
matmul_nbits	Xfp16_Wu3_Yfp16	gfx1151	12	0	6878.8	PASS
matmul_nbits	Xfp16_Wu4_Yfp16	gfx1151	12	0	4295.3	PASS
matmul_nbits	Xfp16_Wi8_Yfp16	gfx1151	12	0	5794	PASS
matmul_nbits	Xfp32_Wu2_Yfp32	gfx1151	12	0	6745.4	PASS
matmul_nbits	Xfp32_Wu3_Yfp32	gfx1151	12	0	6863.3	PASS
matmul_nbits	Xfp32_Wu4_Yfp32	gfx1151	12	0	4537.1	PASS
matmul_nbits	Xfp32_Wi8_Yfp32	gfx1151	12	0	5750	PASS
gqa/decode	Xfp16_Wfp16_Yfp16	gfx1151	17	0	3205.1	PASS
gqa/decode	Xfp16_Wi8_Yfp16	gfx1151	13	0	2502.2	PASS
gqa/prefill	Xfp16_Wfp16_Yfp16	gfx1151	26	0	5280.1	PASS
gqa/prefill	Xfp16_Wi8_Yfp16	gfx1151	9	0	3790.6	PASS
qmoe_ragged_mm	Xfp16_Wu4_Yfp16	gfx1151	40	0	1768.2	PASS
gather_block_quantized	Xi64_Wu4_Yfp16	gfx1151	32	0	283.7	PASS
gather_block_quantized	Xi64_Wu8_Yfp16	gfx1151	32	0	430.1	PASS
gather_block_quantized	Xi64_Wfp8_Yfp16	gfx1151	24	0	265.4	PASS
qmoe_decode	Xfp16_Wu4_Yfp16	gfx1151	32	0	722.8	PASS

Leaves: 17 (0 failed) | Cases: 321 (0 failed)

Mode: lookup | Coverage: 1 | Run: 7968 - Commit: e48ff3c