perf(qwen36): Q8_0→Q4_K requant of full-attention q/o projections

[blinkeye-lcm]

Summary

One load-time weight requantization for the Qwen3.6 decode path, gated to the Qwen3.6 fingerprint and off by default on every other model.

Q8_0 → Q4_K full-attention q/o. Qwen3.6-35B-A3B UD ships the full-attention query and output projections (attn_q, attn_output on the 10 full-attn layers) as Q8_0 — 34 bytes/block on the per-token weight read. At short context, where weight bandwidth dominates decode, those matvecs are a measurable slice of the token. This requantizes them to Q4_K once at model load and decodes them through the existing int8 dp4a Q4_K MMVQ path — ~47% fewer bytes on those reads. No decode-kernel changes: the q/o projection sites already branch on the stored tensor type, so flipping Q8_0→Q4_K at load routes them through launch_mmvq_q4k automatically.

The fit is a new Lloyd-max coordinate-descent Q4_K quantizer (proj_q4k_lloyd.cu): each 32-value group alternates nearest-code assignment with a joint weighted least-squares solve for (scale, min) to convergence, then re-fits the 4-bit codes against the quantized super-block levels. It is applied only to the Q8_0 attention path; the Q6_K dense-FFN down requant keeps its existing affine fit.

On by default for the Qwen3.6 fingerprint (is_qwen35_or_qwen36_hybrid_moe); a strict no-op on the dense Qwythos path and any non-matching model.

A/B toggle: SPARKINFER_ATTN_REQUANT_Q4K=off restores native Q8_0 reads for the comparison below.

Proof of speedup

• Tested on RTX 5090 (sm_120)

Decode tok/s (end-to-end, qwen3_gguf_bench, Qwen3.6-35B-A3B-UD-Q4_K_M.gguf, 128 decode tokens), default vs SPARKINFER_ATTN_REQUANT_Q4K=off, same binary:

	decode tok/s (128)
before (=off, native Q8_0)	420.7
after (this PR, default)	434.2

# ctx=128:  off 420.73 -> default 434.18  (+3.20%)
# ctx=512:  off 416.37 -> default 428.89  (+3.01%)
# ctx=32k:  default 376.58  (win vs ~366 native; no regression at long context)

128 and 512 are the strongest contexts (short context is weight-bandwidth bound). 4k/16k/32k are neutral-to-positive — a byte-reducing requant cannot slow decode; the win just shrinks as KV work grows. Qwythos-9B is unchanged (no-op).

Accuracy. The requant is lossy but confined to the accuracy-tolerant attention projections (attn_q feeds q-norm + softmax; attn_output is an output proj). Teacher-forced top-1 agreement vs the native-Q8 baseline is 277/292 = 0.948 on a held-out text window; subject to the harness KL/top-1 gate (bench/scripts/accuracy.sh, top-1 ≥ 0.90, KL ≤ 0.20).

Relation to open PRs

The Q8_0→Q4_K requant of the full-attention q/o projections with the Lloyd-max fit is original — no open or merged PR does it. The two kernel files (kernels/csrc/cuda/quant/proj_q4k_lloyd.cu, proj_requant.h) are new paths touched by no other PR. The only shared file is runtime/src/models/qwen35.cpp (+16/−4). Added-line containment against every requant PR on that file is well under the guard thresholds: #350 = 15.4%, #323 = 33.0% (highest), #329/#308/#338 ≈ 0% (block ≥ 85%, warn ≥ 75%); per-function max is 77.6% on the format-dictated ggml Q4_K super-block packing, below the 92% single-function warn. The shared lines are generic CUDA launcher boilerplate. Distinct from #350 (Q8→Q4 on the GDN attn_qkv/attn_gate via make_qkx2): different layer class, different fitter, new file.

[skyrocket2026]

✅ sparkinfer auto-eval — d1192b3

metric	value
label	eval:XL
Qwen3.5 score	eval-qwen35:none (pass)
Qwen3.6 score	eval-qwen36:XL (pass)
Qwen3.5 vs same-box main	293.34 tok/s → +0.5% (+1.6)
Qwen3.5 scored decode (512 ctx · 512-context)	294.93 tok/s
Qwen3.5 correctness	top-1 95.3% · KL 0.0265
Qwen3.5 128-token no-regression gate	298.92 tok/s vs main 298.46 tok/s · pass
Qwen3.5 512-context no-regression gate	294.93 tok/s vs main 293.34 tok/s · pass
Qwen3.5 4k-context no-regression gate	283.89 tok/s vs main 283.3 tok/s · pass
Qwen3.6 vs same-box main	412.09 tok/s → +3.7% (+15.4)
Qwen3.6 scored decode (128 ctx · 128-context)	427.54 tok/s
Qwen3.6 correctness	top-1 94.7% · KL 0.0208
Qwen3.6 128-token no-regression gate	427.54 tok/s vs main 412.09 tok/s · pass
Qwen3.6 512-context no-regression gate	432.23 tok/s vs main 418.61 tok/s · pass
Qwen3.6 4k-context no-regression gate	414.9 tok/s vs main 402.5 tok/s · pass
Qwen3.6 16k-context no-regression gate	409.12 tok/s vs main 399.25 tok/s · pass
Qwen3.6 32k-context no-regression gate	391.48 tok/s vs main 382.52 tok/s · pass
Qwen3.5 optimize	eval:none · 294.93 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B guard accuracy	top-1 94.7% · KL 0.0208 · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 128	426.27 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 512	432.16 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 4k	415.0 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 16k	408.99 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 32k	391.23 tok/s · pass
Qwen3.6 optimize	eval:XL · 427.54 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) guard accuracy	top-1 95.3% · KL 0.0265 · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 128	299.02 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 512	294.37 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 4k	283.43 tok/s · pass

Verified speedup over same-box origin/main — 427.54 tok/s (main was 412.09 tok/s).

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

[skyrocket2026]

✅ Auto-merged as the round's merge-first winner — verified same-box speedup over main, all checks green. Thanks for the contribution!