perf(qwen35): Q4_K requant for Qwythos Q6 decode reads

[9876543210-tc-0123456789]

Summary

Requantize the remaining high-traffic Q6_K decode reads for Qwythos/Qwen3.5 dense-9B to Q4_K at load time, using the existing GGUF dequant/requant path and existing Q4_K MMVQ decode kernels. This covers dense FFN down weights, selected linear-attention QKV weights, and the LM head under the dense-9B shape guard.

The Qwythos dense-9B path is default-on with the upstream SPARKINFER_DOWN_REQUANT_Q4K semantics: unset or nonzero enables it, and SPARKINFER_DOWN_REQUANT_Q4K=0 keeps the native Q6_K reads.

Proof of speedup

⚠️ The on-device eval runs only when BOTH are true: (1) the box below is ticked, and
(2) the decode tok/s table shows a real end-to-end improvement (after > before,
filled from bench/scripts/bench.sh / end-to-end qwen3_gguf_bench, not an isolated-kernel microbenchmark).

Tick the box only if you actually ran it on an RTX 5090.

• Tested on RTX 5090 (sm_120)

Decode tok/s (end-to-end, from bench/scripts/bench.sh / qwen3_gguf_bench — required for evaluation):

	decode tok/s
before (main)	285.38
after (this PR)	304.59

# Qwythos-9B-Claude-Mythos-5-1M-Q4_K_M — RTX 5090, CUDA 13, sm_120
# Baseline: origin/main @ 95b93ce
# Candidate: PR #329 @ 47f2e8e
# Env: default requant env, SPARKINFER_BENCH_DEVICE_LOOP=1
# n=128, bs=1, ctx=0, median of 3 runs

=== before (main @ 95b93ce) ===
decode tg    : 284.89 tok/s  (ctx=0, n=128)
decode tg    : 285.38 tok/s  (ctx=0, n=128)
decode tg    : 285.56 tok/s  (ctx=0, n=128)
median       : 285.38 tok/s
VRAM used    : 8.0 GB

=== after (this PR @ 47f2e8e) ===
decode tg    : 304.49 tok/s  (ctx=0, n=128)
decode tg    : 304.61 tok/s  (ctx=0, n=128)
decode tg    : 304.59 tok/s  (ctx=0, n=128)
median       : 304.59 tok/s
VRAM used    : 7.6 GB

Delta        : +6.73%

Additional same-box medians on the rebased candidate:

context	before main tok/s	after PR tok/s	delta
512	282.92	301.90	+6.71%
4096	274.66	292.15	+6.37%

Accuracy

bench/scripts/accuracy.sh, same GGUF, seed q35-broad-requant-check:

positions             : 349
token-match (top-1)   : 317/349 = 0.908   (bar >= 0.90)
mean KL(llama||spark) : 0.0361 nats  (top-k approx)
PPL sparkinfer        : 7.947  (exact, full softmax)
PPL llama.cpp         : 10.740  (top-64+floor; inflated)
METRIC top1=0.908309 kl=0.036070 ppl_spark=7.9466 ppl_llama=10.7398

Implementation notes

• Generalizes the existing dense-down helper into dev_quant_requant_q4k(name, qtype, enabled).
• Keeps upstream dense-down behavior: SPARKINFER_DOWN_REQUANT_Q4K=0 disables, unset enables.
• Adds Qwythos dense-9B defaults for selected linear-attention QKV and LM-head requant.
• Adds explicit env overrides: SPARKINFER_ATTN_REQUANT_Q4K, SPARKINFER_ATTN_REQUANT_Q4K_QKV_LAYERS, SPARKINFER_ATTN_REQUANT_Q4K_QKV_LIMIT, and SPARKINFER_LMHEAD_REQUANT_Q4K.
• Adds no new kernels or headers; this reuses existing dequant/requant and Q4_K MMVQ building blocks.
• Broader qkv_all / all_attn sweeps were only about 0.3% faster than the safer default and include an accuracy-sensitive layer, so they are left as env-controlled options instead of defaults.

Test plan

• Rebases cleanly on origin/main@95b93ce
• git diff --check
• Built latest-main baseline and candidate in CUDA 13 container for sm_120
• Qwythos decode bench at ctx 0 / 512 / 4096 on RTX 5090
• Default-env ctx0 median: main 285.38 tok/s, PR 304.59 tok/s
• bench/scripts/accuracy.sh vs llama.cpp reference on held-out seed q35-broad-requant-check
• Checked current open PRs for overlap; this stays one-file scoped and reuses existing kernels without copying another PR

[skyrocket2026]

⏸ Merge conflict — rebase before eval

This branch conflicts with main, so the RTX 5090 eval is skipped until you rebase.

Please rebase onto main and push — the bot re-evaluates on the next poll once the PR is cleanly mergeable.

[skyrocket2026]

The round's merge-first PR was just merged. Please rebase this branch onto main — once you push the rebase the bot re-evaluates it against the new frontier (crediting your marginal gain on top of what merged).

[skyrocket2026]

The round's merge-first PR was just merged. Please rebase this branch onto main — once you push the rebase the bot re-evaluates it against the new frontier (crediting your marginal gain on top of what merged).

[skyrocket2026]

✅ sparkinfer auto-eval — 47f2e8e

metric	value
label	eval:M
Qwen3.5 score	eval-qwen35:M (pass)
Qwen3.6 score	eval-qwen36:none (pass)
Qwen3.5 vs same-box main	283.18 tok/s → +7.1% (+20.0)
Qwen3.5 scored decode (128 ctx · 128-context)	303.18 tok/s
Qwen3.5 correctness	top-1 95.2% · KL 0.0301
Qwen3.5 128-token no-regression gate	303.18 tok/s vs main 283.18 tok/s · pass
Qwen3.5 512-context no-regression gate	299.26 tok/s vs main 280.25 tok/s · pass
Qwen3.5 4k-context no-regression gate	289.19 tok/s vs main 272.14 tok/s · pass
Qwen3.6 vs same-box main	381.72 tok/s → +0.0% (+0.1)
Qwen3.6 scored decode (16384 ctx · 16k-context)	381.86 tok/s
Qwen3.6 correctness	top-1 97.3% · KL 0.0147
Qwen3.6 128-token no-regression gate	427.25 tok/s vs main 427.74 tok/s · pass
Qwen3.6 512-context no-regression gate	422.28 tok/s vs main 422.84 tok/s · pass
Qwen3.6 4k-context no-regression gate	405.57 tok/s vs main 405.82 tok/s · pass
Qwen3.6 16k-context no-regression gate	381.86 tok/s vs main 381.72 tok/s · pass
Qwen3.6 32k-context no-regression gate	354.47 tok/s vs main 354.56 tok/s · pass
Qwen3.5 optimize	eval:M · 303.18 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B guard accuracy	top-1 97.3% · KL 0.0147 · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 128	426.98 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 512	422.22 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 4k	405.41 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 16k	381.8 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 32k	354.45 tok/s · pass
Qwen3.6 optimize	eval:none · 381.86 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) guard accuracy	top-1 95.2% · KL 0.0301 · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 128	302.96 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 512	299.1 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 4k	289.24 tok/s · pass

Verified speedup over same-box origin/main — 303.18 tok/s (main was 283.18 tok/s).

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

[skyrocket2026]

✅ Auto-merged as the round's merge-first winner — verified same-box speedup over main, all checks green. Thanks for the contribution!