perf(qwen35): run the MTP head on the trunk flash policy with partials sized for the launch tile - #726
Open
alpineQ wants to merge 1 commit into
Open
perf(qwen35): run the MTP head on the trunk flash policy with partials sized for the launch tile#726alpineQ wants to merge 1 commit into
alpineQ wants to merge 1 commit into
Conversation
…s sized for the launch tile
Kaden-Schutt
added a commit
that referenced
this pull request
Sep 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The MTP head's attention ignored the trunk's flash policy and its flash-partials scratch was sized for a hardcoded 128-token tile; the head now follows
attention.flashlike the trunk (draft step at 33k: 10 ms → 0.3 ms) and sizes partials with the tile the launch actually picks, which removes a GPU page fault on gfx1100 atmax_seq ≤ 8192with MTP on.What was wrong
mtp_head_forward_block_only_with_pos_bufderives itsKvTierPlanfromkv.inner.tier_inputs(), which reportsflash_mode: 0. That pins the Q8 tier to the non-flashAttnQ8_0Kvkernel at every context, so each draft step of a K=3 chain at 33k context paid ~10 ms of attention against 0.3 ms on the trunk's flash tile (rocprofv2 kernel trace of a 33k-prompt MTP run).Qwen35MtpHeadScratch::newallocatedflash_partialsformax_seq / 128tiles.q8_flash_default_tile_sizereturns 32 on gfx1100 atmax_seq ≤ 8192(and 16 for the gfx12 small-dense shape), so once the head attends through the flash tile the kernel indexes 4× more tiles than were allocated. Reproduces on one card:Memory access fault by GPU node-1 … Page not presentinattention_flash_q8_0_tile, grid (24, 2), right after the head'skv_cache_write_q8_0Located with
AMD_SERIALIZE_KERNEL=3 AMD_LOG_LEVEL=3: the faulting address sits 3 960 832 bytes into the 1 585 152-byte partials buffer (= 24 heads × 64 tiles × 258 × 4), inside the 6 340 608 bytes the kernel needs at 256 tiles.What changes
hipfire_runtime::llama::attention_flash_mode(arch)/attention_flash_mode_for(mode, arch)are the (previously private) helpers that resolveattention.flash(never/auto/always,auto⇒ flash on gfx11/gfx12, tiered elsewhere). The MTP head uses it; the identical hand-writtenmatchinqwen35/forward.rsis replaced by the same call.Qwen35MtpHeadScratchsizesflash_partialswithrdna_compute::attention::q8_flash_tile_sizefor the head's own shape.Draft tokens only propose; the trunk verifies, so greedy output is unchanged. Measured on 2× and 1× gfx1100 with
qwen3.8-27b+.mtpsidecar at 33k: AR/MTP text byte-identical to the previous build, MTP decode 19.2 → 22.0 tok/s from the head attention alone.Which surface(s) does this touch?
llama.rsattention policy helper (spec path)hipfire-arch-qwen35(mtp_head.rs,qwen35/forward.rs)crates/hipfire-quantizeTest plan
./scripts/no-gpu-ci.sh: Rust check + no-GPU unit tests + env/docs drift check pass; the Python stage reports 838 passed / 6 failed intests/test_mq4c_repack.py, and those 6 fail identically on untouchedmasterin this environment (NixOS: no/bin/bash, and the test module resolves a differentmq4c_repackthan the script) — not touched by this changecargo build --releaseclean (--workspace --all-targets --locked)cargo test --lib --workspacepasses (incl. theattention_flash_mode_fortable test inllama.rs)serve_harness.pybattery + chain onqwen3.6:27b(registry fixture, MTP off): 5/5 turns each, no flagsqwen3.8-27b.mq4+.mtpsidecar,--mtp on: battery at--max-seq 8192and chain at defaultmax_seqcomplete with the head on the flash path (tau1.4–2.4 per turn). Two flags on the first two battery turns ([code]empty withfinish=None,[prose]runaway) reproduce identically with the head on master's non-flash path (A/B runq38-mtp-8192-battery-upstreampath(per-turn table attached), same binary minus this change), so they are the qwen3.8 template under--thinking off, not this changemax_seq 8192—Memory access fault by GPU node-1 … Page not presentinserve-fault.log, all five turns empty (q38-mtp-8192-flash-only.json); with the partials sized byq8_flash_tile_sizethe same battery runslocal serve_harness JSON
Hardware validation request (optional)
{ "routes": [ {"mode": "battery", "tag": "qwen3.6:27b"}, {"mode": "chain", "tag": "qwen3.6:27b"}, {"mode": "chain", "tag": "ornith-1.5:35b-a3b-mq4r"} ], "claim": "MTP head attends through the trunk flash policy with correctly sized partials; any MTP-enabled artifact at max_seq <= 8192 on gfx1100 no longer page-faults, and non-MTP serve is unchanged." }Architecture-trait change?
No.