perf(qwen36): opt-in Q8→Q4 requant reward profile (~+6.3% @512)

[andriypolanski]

Summary

Load-time Q8_0 → Q4_K requant for Qwen3.6 UD GDN projections, gated behind a single SN74-facing knob:

SPARKINFER_Q80_REQUANT_PROFILE=reward

On Qwen3.6-35B-A3B-UD-Q4_K_M.gguf @ ctx=512 (128 generated tokens, RTX 5090), this trims ~47% of weight bytes on the hottest GDN qkv+gate reads on layers ≥15 while staying inside the accuracy gate (top-1 ≥ 0.90, KL ≤ 0.20).

Bottleneck: Qwen3.6 UD ships attn/GDN weights as Q8_0. At short context those MMVQ reads dominate decode (~36% GPU time @512). Requantizing once at load and decoding via the existing Q4_K int8 MMVQ path cuts bandwidth on the selected tensors without touching MoE experts, shared expert Q8 paths, or the LM head.

Scope: kernels/csrc/cuda/quant/ffn_down_requant.cu, kernels/include/sparkinfer/kernels/quant.h, runtime/src/models/qwen35.cpp only — per CONTRIBUTING.md miner paths.

Default: OFF. No env var → native Q8_0 weights (same path as main for Qwen3.6 UD).

Proof of speedup

• Tested on RTX 5090 (sm_120)

Decode tok/s (end-to-end, same binary, interleaved env on/off @ ctx=512, n=128, after merge with main):

	decode tok/s
before (no PROFILE, native Q8)	422.2
after (SPARKINFER_Q80_REQUANT_PROFILE=reward)	449.0

# Qwen3.6-35B-A3B-UD-Q4_K_M.gguf, ctx=512, 128 tokens, RTX 5090
# interleaved (before / after / before / after):
#   422.34  →  449.39  →  422.15  →  448.66 tok/s
# mean before ~422.2, mean after ~449.0  (+6.3% @512)

Correctness gate

Same binary, PROFILE=reward vs default OFF, held-out 100 positions (self_consistency.py):

config	top-1 vs before	KL	gate
before (no env)	—	—	—
SPARKINFER_Q80_REQUANT_PROFILE=reward	92/100	0.18	pass

Full-scope requant (all layers / all Q8 projections) fails KL (0.25–0.9+). The reward profile is the measured KL-safe slice:

knob	value
SPARKINFER_Q80_REQUANT_PROFILE	reward
effective scope	qkvz (GDN attn_qkv + attn_gate only)
effective layer filter	LAYER_OFF=15 (layers 0–14 stay native Q8_0)
format	Q4_K (default)

Equivalent explicit knobs:

SPARKINFER_Q80_REQUANT_Q4K=qkvz \
SPARKINFER_Q80_REQUANT_LAYER_OFF=15

Reproduce locally:

# build candidate (repo-root superbuild)
cmake -S . -B build-opt -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build-opt -j"$(nproc)" --target qwen3_gguf_bench qwen3_gguf_score

GGUF=/path/to/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
BIN=./build-opt/runtime/qwen3_gguf_bench

# before
unset SPARKINFER_Q80_REQUANT_PROFILE
$BIN "$GGUF" 128 512

# after
SPARKINFER_Q80_REQUANT_PROFILE=reward $BIN "$GGUF" 128 512

# accuracy (same IDS both runs, then self_consistency.py)
./build-opt/runtime/qwen3_gguf_score "$GGUF" 128 $IDS > /tmp/before.txt
SPARKINFER_Q80_REQUANT_PROFILE=reward \
  ./build-opt/runtime/qwen3_gguf_score "$GGUF" 128 $IDS > /tmp/after.txt
python3 bench/scripts/self_consistency.py /tmp/before.txt /tmp/after.txt

Implementation notes

• launch_q80_requant_q4k: dequant Q8_0 blocks to fp32 in registers, pack to ggml Q4_K super-blocks via llama.cpp-style make_qkx2 (same fitter upgraded for dense-FFN down requant).
• qwen35.cpp: at GGUF load, selected Q8_0 attn/GDN tensors are rewritten to Q4_K in VRAM; downstream decode uses existing launch_mmvq_q4k / GDN qkv+z fuse on requantized layers.
• Side effect on requantized layers: GDN qkv+gate become Q4_K, so SPARKINFER_GDN_QKVZ_FUSE can engage on layers ≥15 (further bandwidth win, no extra env).

Maintainer/debug scopes (SPARKINFER_Q80_REQUANT_Q4K, LAYER_MOD, LAYER_REM) remain for walking the KL gate; miners should use PROFILE=reward only.

Test plan

• Interleaved bench @ ctx=512: ~422 → ~449 tok/s (+6.3%)
• Accuracy with PROFILE=reward vs default OFF: top-1 92/100, KL 0.18 (gate pass)
• Confirm requant OFF by default (no env → ~422 tok/s, no weight rewrite)
• Guard contexts 128 / 4k / 16k for regressions (expected neutral or small win; no full-scope requant)
• Smoke Qwen3-30B (no unexpected Q80 requant on non-UD models)

Files changed

file	change
kernels/csrc/cuda/quant/ffn_down_requant.cu	llama-style Q4_K fitter + launch_q80_requant_q4k
kernels/include/sparkinfer/kernels/quant.h	declare Q80→Q4 load-time API
runtime/src/models/qwen35.cpp	PROFILE=reward, scope/layer filters, load-time hook