perf(gfx11): optimize Qwen DFlash speculative verify

[HUSRCF]

Summary

Improve Qwen DFlash speculative verification on gfx11 by routing the B=16 full-attention workload through the batched flash/N64 path, and disable the gfx1100 MQ4 verify HipGraph by default where it does not provide a measurable benefit.

Changes

• Route speculative verification through batched flash attention on the eligible gfx11 path.
• Add the Qwen DFlash B=16, head_dim=128 N64 attention route.
• Keep HIPFIRE_VERIFY_GRAPH=1 as an explicit diagnostic opt-in, but default gfx1100 MQ4 verify graph capture off.

PM4 changes are intentionally excluded. The PM4 timeout quarantine fix is unrelated and should be reviewed separately.

Performance Evidence

Batched flash versus forced WMMA

On Precision (gfx1100), Qwen 27B, DFlash B=16, and real LongBench thinking inputs around 28K–30K tokens:

Verify attention path	End-to-end decode throughput
Batched flash	24.88–36.00 tok/s
Forced WMMA	12.10–16.64 tok/s

Acceptance and output lengths were materially the same. This is an end-to-end verifier comparison, not an isolated attention-kernel speedup claim.

HipGraph A/B

The verify graph path was confirmed to reach mode=replay in diagnostic runs. On the same gfx11 workload, replay windows were approximately 2–5% slower than direct execution.

Using the same three LongBench thinking prompts with --max-tokens 32768:

Configuration	Decode throughput
Graph off	29.5–30.0 tok/s
Graph on	29.5–29.9 tok/s

This is a neutral-to-slightly-negative result for graph capture at the current Qwen gfx1100 MQ4 shape. Graph remains available through HIPFIRE_VERIFY_GRAPH=1 for future diagnostics.

Correctness

• N64 parity coverage: 336 cases, 0 failures; maximum observed absolute difference was within the existing 1e-3 tolerance.
• Graph on/off runs completed with matching request completion behavior.
• No acceptance regression was observed in the graph A/B runs.
• PM4 remains disabled and is not part of this change.

Tests

cargo test -p hipfire-dispatch --lib
cargo test -p hipfire-runtime dflash --lib
cargo test -p hipfire-arch-qwen35 --lib dflash_verify_graph_env -- --nocapture

All listed tests pass on Precision. Existing compiler warnings are unrelated unused/dead-code warnings.

Scope and Follow-up

This PR does not change long-context thresholds or disable DFlash for long inputs. The optimization is in the verifier and attention routing paths. A future change can revisit graph capture after reducing graph-external synchronization and dynamic-shape overhead.

[HUSRCF]

Paired 10K-context verify HipGraph A/B

Follow-up correcting the earlier split campaign: this run used six controlled technical-document prompts at approximately 11.5K model-context tokens. Each GPU ran the same three prompts serially with Graph off and then Graph on.

Pinned configuration:

target: qwen3.6-27b.mq4
draft: qwen36-27b-dflash-mq4.hfq
HIPFIRE_DFLASH_CTX_CAP=16384
max_tokens=32768
sampling=greedy
seed=693
B=16

Graph execution was explicitly proven:

GPU0 Graph-on: warmup=1, capture=1, replay=154
GPU1 Graph-on: warmup=1, capture=1, replay=148

Case	Graph off	Graph on	Delta
1	59.7 tok/s	59.5 tok/s	-0.34%
2	59.1 tok/s	58.9 tok/s	-0.34%
3	11.7 tok/s	11.6 tok/s	-0.85%
4	57.5 tok/s	56.8 tok/s	-1.22%
5	61.5 tok/s	61.0 tok/s	-0.81%
6	51.4 tok/s	51.2 tok/s	-0.39%

All six pairs completed with identical generated-token counts, cycle counts, and tau between Graph off/on. Graph replay regressed every paired case by 0.34-1.22%, supporting the gfx1100 MQ4 default-off policy while retaining HIPFIRE_VERIFY_GRAPH=1 as a diagnostic override.

Case 3 is a reproducible workload-specific outlier in both arms (tau=5.17, 53 cycles, 326 generated tokens): Graph off/on are equally slow, and verifier windows remain approximately 72-75 ms. A standalone draft-subphase run also reproduces the low endpoint throughput, so this is not a HipGraph regression and is being investigated separately.

[HUSRCF]

Follow-up: 10K terminal-latency cliff isolated and fixed

During the paired long-context validation, synthetic prompt #3 reported only 11.9 tok/s, while adding one trailing ASCII period to the same prompt reported 63.0 tok/s. This was reproducible and was not a verifier-kernel, HipGraph, or context-length threshold regression.

Root cause

The original token trajectory encountered the terminal token in the middle of an 11-token speculative window (consumed=9). generate_spec correctly discarded the two unobserved tail tokens, but then reset target/drafter state and synchronously replayed the entire ~11.5K prompt plus generated prefix before returning an already-complete response. Outer-loop instrumentation measured 22.11 s in that terminal tail, versus ~0.1 ms when the terminal landed at the window boundary.

Commit 8e3e5210 changes terminal strict-prefix repair to reset the polluted target/drafter state, invalidate resident cache metadata, and let the next request use the normal cold-prefill path. Non-terminal strict-prefix paths retain the existing exact full-history realign behavior. Temporary profiling instrumentation is not included.

Correctness and performance evidence (gfx1100)

Arm	ctx	gen	cycles	tau	decode	request MD5
Before fix, original #3	11,555	326	54	5.06	11.9 tok/s	195fe17e1bd2e3bee2365241e39eb906
After fix, instrumented	11,555	326	54	5.06	62.6 tok/s	195fe17e1bd2e3bee2365241e39eb906
After fix, clean release build	11,555	326	54	5.06	62.8 tok/s	195fe17e1bd2e3bee2365241e39eb906

The clean run retained the same prompt MD5 (3c2f517bd549687460522c2c3d3161b3), output MD5, generated-token count, cycle count, tau, and stop result. No diagnostic timing code was present.

A same-daemon two-request test (#3 -> #1) confirmed the invalidated state is not reused: both requests took truthful cold-prefill (cached=0) and decoded at 62.1 / 60.1 tok/s, with their prior output MD5s unchanged.

Validation: cargo test -p hipfire-generate --test qwen_dflash_semantic_terminal_tests -> 65 passed, 0 failed. The added action test covers terminal strict-prefix reset, non-terminal realign, and full-window no-op.

[HUSRCF]

Exact pushed-source confirmation (8e3e5210): rebuilt hipfire-cli + hipfire-daemon on Precision and reran the unchanged original #3 fixture with no timing diagnostics. Result: 60.9 tok/s, ctx=11555, gen=326, cycles=54, tau=5.06, finish=stop; prompt MD5 3c2f517bd549687460522c2c3d3161b3 and request MD5 195fe17e1bd2e3bee2365241e39eb906 remain unchanged.

[alpineQ]

Two notes from running the same verifier shape on 2x gfx1100 (qwen3.8-27b, Q8 KV, 16k-50k ctx):

1. After this change the verify attend still re-reads the KV once per query row: attention_flash_q8_0_tile_batched grids [n_heads, tiles, ROW]. A multi-row variant (one block owns 4/8 rows, KV fetched once per group) measured 1.7x (4 rows) / 2.0x (8 rows) on the attend step at 33k and +12-25% end-to-end MTP decode at 16k-50k with tau unchanged. It only pays past ~8k ctx; below that the batched kernel wins. Happy to send it as a follow-up gated in dispatch_attend next to your batched route once this lands, if that shape is welcome.

2. ResetForTerminal clears conversation_tokens, both checkpoint sets and asst_turn_cache. The completed request returns fast, but the next turn of the same conversation cold-prefills the whole history (30k ctx ~= 60 s on gfx1100). An alternative that keeps the cache warm: roll the target and drafter back to the pre-window snapshot and replay only the consumed tail (O(window) instead of O(context)); I have that measured on the MTP path and can share the diff.

[HUSRCF]

@alpineQ Thanks, both observations are correct.

The multi-row verifier sounds appropriate as a separate follow-up. Gating the
4-row/8-row variant next to the current batched route at its measured crossover
(roughly >8K context) would keep the short-context path unchanged while giving
the new kernel independent correctness and performance coverage.

I addressed the terminal-cache issue in a14510a5 (current PR head
127a5624). The DFlash speculator now retains:

• the target recurrent state from immediately before the speculative window;
• a metadata watermark for the target-hidden cache.

When a terminal consumes only a strict prefix of that window, it restores the
snapshot and replays only:

window_seed + consumed[..last]

The final consumed token remains pending for the existing terminal flush.
Unsupported speculators retain the conservative reset fallback, and repair
failures still fail closed.

Exact-head gfx1100 validation used qwen3.6-27b, Q8 VMM, DFlash B=16,
max_tokens=32768, and the original long-context request:

• prompt MD5: 3c2f517bd549687460522c2c3d3161b3
• request MD5: 195fe17e1bd2e3bee2365241e39eb906
• ctx/gen/cycles/tau: 11555 / 326 / 54 / 5.06
• output MD5: 9a25382543d71869aa618e34143e8b67
• terminal repair: 9 tokens replayed

The immediately following turn in the same daemon reported:

• context: 11,911 tokens
• cached prefix: 11,881 tokens
• incremental suffix: 30 tokens
• incremental prefill: 202.5 ms
• trace: qwen-cache HIT dflash
• terminal repair: 13 tokens replayed

This preserves the fast completion behavior while removing the next-turn
full-history cold prefill.

Validation:

• target-hidden rollback unit test: 1/1 passed
• Qwen DFlash semantic terminal tests: 65/65 passed
• exact-head release build and two-turn gfx1100 E2E: passed
• daemon MD5: 9c4699cdbfb938ce4f7dca7aabebbfd9

[HUSRCF]

@alpineQ Your measured >8K multi-row verifier now looks directly relevant to the long-context report in #693. If the offer still stands, could you please share the follow-up or diff for the gfx1100 R4/R8 variant, including the dispatch_attend gate you used? We are reproducing the qwen3.8-27B ~20.7K-context case locally and will independently validate R4/R8 against the current batched route for correctness, attend-step time, and end-to-end decode.

[alpineQ]

Thanks — I split this out into one clean commit based directly on fdb750d6d3138269523ec1094fb189b465f1438c:

• commit: alpineQ/hipfire@c91f4b65
• branch: perf/gfx1100-multi-row-verifier
• raw patch

The kernel entry points are attention_flash_q8_0_rows{4,8}_d{4,8}. A block owns one (head, KV tile, row group) and reuses each Q8 K/V fetch across four or eight query rows. It writes the existing batched-partials layout, so the existing reducer is unchanged. Only the attention stage changes; projection GEMMs remain batched.

I also fixed the two merge caveats from the experimental version:

1. Admission is now fail-closed to the measured route: exact gfx1100, Q8 KV, head_dim in {128,256}, sequential/non-tree batches of 4..32 rows, logical context >4096, and graph capture off. The launcher independently rejects other architectures/shapes.
2. If the admission and launcher predicates ever drift, the call falls through to the established batched attention step. It no longer expands an unsupported case into one attention launch per row.

!capture_mode remains intentional: the multi-row tile grid is sized from the live logical context, so capturing it would retain the first cycle's grid. Graph/PM4 routes therefore keep the existing batched path.

I rebuilt and reran the clean commit on one RX 7900 XTX (gfx1100, wave32, HIP/ROCm 7.2.3), Q8 KV. General hardware kernel smoke passed 16/16. The dedicated oracle compared R4/R8 with attention_flash_q8_0_tile_batched; worst relative output error was 4.222e-7 (limit 1e-3). Final micro-harness MD5: d0f40e4a05cb2855cfdadbbe898a3e91.

context	R4 vs batched	R8 vs batched
2,048	0.87x	1.06x
4,096	1.11x	1.53x
20,676	1.90x	2.31x
32,768	1.90x	2.23x

The conservative >4096 default keeps the short-context route on the existing batched kernel; HIPFIRE_FA_PERTOKEN_MIN_CTX=0 disables the new route entirely.

Fresh-process end-to-end DFlash A/B used the same clean daemon binary (aa30f260e1fb208c7c663807092c863a), toggling only that route. The prompt had 21,550 actual tokens and generated 200 tokens:

route	samples	median
existing batched	27.1, 27.2, 26.9 tok/s	27.1 tok/s
R4/R8	35.7, 36.2, 36.3 tok/s	36.2 tok/s

That is +33.6% by median. All six runs had tau=1.52, 79 cycles, and identical output MD5 f958fb101df4a2f0506a2f1cc9fda671. Fixture MD5s: prompt b4d0b63cddcac872648ddf3cdd92cac2, target d1292b4d5bd6046693604201a6ca8074, DFlash draft 013395583cd04206c8aa68f4d061983d.

Static/unit validation also passed: hipfire-arch-qwen35 193/193, rdna-compute 241/241, crate maps, env-doc scan, changed-file rustfmt, fmt-bomb, and diff-check.

This improves the verifier cost but does not solve the separate low-tau case: this fixture remains at tau=1.52, so a prompt/draft with still worse agreement can still lose to AR. Also, the adjacent #693 observation still holds: speculation.dflash_adaptive_b is currently parsed into _adaptive_b by the daemon but is not wired into the generation loop, so adaptive_b=false is not yet a distinct daemon arm.

[Kaden-Schutt]

Completed exact inventory against beta af39905:2160d4c1 and8e3e5210 are direct ancestors; a14510a is patch-equivalent to landed5225cfdee (git cherry marks applied). The only unique remaining commit127a5624 is stale generated maps. Thus all substantive #695 code is already integrated; no additional beta cherry-pick needed. The multi-row verifier c91f4b6 discussed here is a separate alpineQ branch, not part of this PR; it needs its own current-beta review and canonical XT fixture evidence. Keeping this PR on #730 close-on-promotion list.

[alpineQ]

Done — I ported the R4/R8 follow-up onto current beta (b8092f7c7) and opened it separately as #741:

• PR: perf(gfx1100): share Q8 KV scans across verifier rows #741
• commit: de2448f07

The rerun now uses the pinned canonical XT target, not the earlier lookalike:

• target SHA-256 9f91556f7e0431a077d03756a7102d0154108757289e6e5fe9a2d204c0c9eeb7 (MD5 e45d15bfe0c9a87132697101d17cbed6)
• draft SHA-256 d0a74a232a0e2166d889f823e91e0fbf778d21dd9668d7de055cdecb065401bc (MD5 013395583cd04206c8aa68f4d061983d)
• committed 21,550-token prompt MD5 b4d0b63cddcac872648ddf3cdd92cac2

Fresh-process route A/B (off,on,on,off,off,on, same candidate daemon, route threshold the only arm delta):

route	samples	median
existing batched	33.4, 32.6, 33.4 tok/s	33.4
R4/R8	46.4, 42.9, 46.3 tok/s	46.3

That is +38.6% by median. All six samples retained tau=1.88, 69 cycles, 200 tokens, no errors, and identical output MD5 b501ab0e0102889bd63537f2006d4f61. A separately built and warmed clean-beta daemon gave 33.5 tok/s with the same output/tau/cycles, matching the disabled candidate arm.

The current-beta kernel oracle measured R4/R8 at 1.99x/2.40x for ctx 20,676 and 1.94x/2.28x for ctx 32,768; worst relative output error was 4.222e-7. R4/R8 hd256 compile at 106/186 VGPR, 41/58 SGPR, with no spills or private scratch. test_kernels passed 16/16; unit tests passed 193/193 and 242/242; the canonical-XT serve battery passed 5/5 with coherent decoded text.

One explicit caveat in #741: the canonical Redline PM4 kernel-bucket arm currently stops on gemv_mq4g256v2_residual private scratch on both clean beta and the candidate. Their capture identities match before the refusal, and the new route is excluded during capture, so this is a pre-existing gate blocker rather than a claimed PM4 pass.

[HUSRCF]

@alpineQ Got it, thanks a lot for your work! I'd be happy to help port it to gfx1201. Do you have any plans for that yet? If so, I'll leave that part to you and provide help elsewhere..

[HUSRCF]

Given that this PR was merged into beta, further work would be conducted in beta fork, so this would be turned off.

[alpineQ]

@HUSRCF I'd be happy to, but I don't have any gfx1201 hardware, so please feel free to take the port