Skip to content

qwen3.8-mtp: vendor mmul proj-engine xclbins (decode 1.77 -> 2.84 tok/s) - #909

Merged
tawei-amd merged 2 commits into
mainfrom
feat/qwen38-proj-mmul-xclbins
Nov 1, 2026
Merged

tawei-amd merged 2 commits into
mainfrom
feat/qwen38-proj-mmul-xclbins

Conversation

@tawei-amd

Copy link
Copy Markdown
Collaborator

What

Replaces the two Qwen3.8-27B-MTP layer xclbins with builds of the q4_K projection engine on aie::mmul (internal IRON PR #135). Decode goes 1.77 -> 2.84 tok/s on a local NPU at unchanged MTP acceptance. The engine library is unchanged: no .so/.dll change, only the two kernels.

file sha256 IRON artifact
src/xclbins/Qwen3.8-27B-NPU2/deltanet_layer.xclbin 5881abeed388f52ecb7c4be5f1b71c6580876bb81d02dae5906f8a52fda8d162 xclbins-Qwen3_8_gate_delta_net_verify
src/xclbins/Qwen3.8-27B-NPU2/attention_layer.xclbin 1caee645f17fc942911e629ba75d10209fed2a479f86f62c2a102aacbce576d3 xclbins-Qwen3_8_gated_attention_verify (layer.xclbin, renamed to the name the engine registers)

IRON PR Check run: 41264160514 (the vendored files are exactly that run's artifacts; no local builds committed).

Verification

  • IRON CI green: xclbin builds, perf, GSM8K, DocVQA on XRT and HRX. CI decode for qwen3.8-mtp:27b: XRT +20..+80%, HRX +44..+115%; GSM8K 98.0 -> 98.0 (XRT), 96.0 -> 98.0 (HRX).
  • Local, these exact files: layer bench vs the stock kernel on identical input, delta-net 23.7 -> 12.3 ms (cos 0.99995), attention 20.6 -> 9.5 ms (cos 0.9998).
  • Local, production engine lib + these xclbins, six prompts each, temp 0, 160 tokens:
prompt set stock tok/s (MTP accept len) this PR tok/s (accept len)
English / code 1.77 (2.78) 2.84 (2.72)
multilingual / code / math 1.83 (2.65) 2.87 (2.56)

Output text diverges from stock after a few hundred characters (greedy decode with ~1-2% numeric noise in the layer), stays coherent.

Order

The IRON PR is still a draft. Per the usual order this should land after IRON merges; the kernel sources are not touched by the merge, but I will re-verify the merged run's artifacts against the sha256 above before this leaves draft.

🤖 Generated with Claude Code

deltanet_layer.xclbin and attention_layer.xclbin rebuilt with the q4_K block kernel on
aie::mmul (IRON PR AMD-ROCm-Internal/FastFlowLM_IRON#135, PR Check run 41264160514).
Files are the CI artifacts, not local builds:
  5881abeed388f52ecb7c4be5f1b71c6580876bb81d02dae5906f8a52fda8d162  deltanet_layer.xclbin
  1caee645f17fc942911e629ba75d10209fed2a479f86f62c2a102aacbce576d3  attention_layer.xclbin

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tawei-amd

Copy link
Copy Markdown
Collaborator Author

IRON#135 merged (5014861). Re-verified: the merge commit has no file differences from the PR head cf425f4 that run 41264160514 built, so these xclbins are the merged kernels. sha256 of the vendored files still match that run's artifacts (5881abee... deltanet, 1caee645... attention). Marking ready.

@tawei-amd
tawei-amd marked this pull request as ready for review November 1, 2026 05:25
…e xclbins

Decode tok/s at 1k/2k/4k/8k context: 1.06/1.86/1.50/1.32 -> 2.04/2.23/2.70/2.31 (IRON PR Check
run 41264160514, XRT perf-bench, the bench the page's numbers come from). Prefill and TTFT are
unchanged by this kernel and left as is.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tawei-amd

Copy link
Copy Markdown
Collaborator Author

Added the docs benchmark update: docs/docs/benchmarks/qwen3.8_results.md decode row 1.06/1.86/1.50/1.32 -> 2.04/2.23/2.70/2.31 tok/s (1k/2k/4k/8k) and a regenerated qwen38_decoding.png in the same style. Numbers are IRON PR Check run 41264160514 XRT perf-bench for this PR's kernels (the same bench the page's previous numbers match: its baseline gave 1.50/1.32 at 4k/8k and prefill within 0.2%). Prefill and TTFT are unchanged by this kernel and untouched.

@tawei-amd
tawei-amd merged commit 1c39813 into main Nov 1, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant