perf(qwen36): Q4_K shared-expert FFN kernels + Q8_0/Q5_K→Q4_K requant of the MoE FFN stage (+2.8–3.2%)

[carlh7777]

Summary

Two load-time weight requantizations targeting the MoE FFN stage of the Qwen3.6-35B-A3B decode path, gated to the Qwen3.6 fingerprint (is_qwen35_or_qwen36_hybrid_moe) and off by default on every other model. The FFN stage runs on every token across all 40 layers, so shrinking its per-token weight reads is a broad, context-flat decode win.

1. Shared-expert FFN Q8_0 → Q4_K (new kernels). Qwen3.6-35B-A3B UD ships the always-on shared expert (ffn_{gate,up,down}_shexp) as Q8_0 — 34 bytes/block on a read that fires every layer, every token. This requantizes it to Q4_K once at model load (~47% fewer weight bytes) and decodes it through two new Q4_K shared-expert MMVQ kernels (shared_gate_up_q4k_mmvq_kernel, shared_down_q4k_mmvq_kernel + launcher launch_shared_expert_q4k_mmvq). They mirror the existing Q8 shared-expert control flow (FNQ Q8_1 → gate/up·silu → quant_h Q8_1 → down), only the weight dot swaps to the int8 dp4a Q4_K path. Gated to the exact shared-expert shape the kernels specialize (<2048, 512>).
2. Routed-expert down Q5_K → Q4_K (no new kernel). The top-8 routed experts keep their down projection at Q5_K while gate/up are already Q4_K. That down read sits in the same decode-critical FFN stage, so requantizing ffn_down_exps Q5_K→Q4_K at load stacks a further byte reduction on the same stage and routes automatically through the pre-existing Q4_K routed-down MMVQ dispatch (down_type == 12).

Strict no-op on Qwythos-9B and any non-matching model.

A/B toggles (same binary): SPARKINFER_SHEXP_REQUANT_Q4K=0 restores native Q8_0 shared-expert reads; SPARKINFER_ROUTED_DOWN_REQUANT_Q4K=0 restores native Q5_K routed-down. Both set = the "before" column below.

Proof of speedup

• Tested on RTX 5090 (sm_120)

Decode tok/s (end-to-end, qwen3_gguf_bench, Qwen3.6-35B-A3B-UD-Q4_K_M.gguf, 128 decode tokens), default vs SPARKINFER_SHEXP_REQUANT_Q4K=off + SPARKINFER_ROUTED_DOWN_REQUANT_Q4K=off, same binary:

	decode tok/s (128)
before (=off, native)	437.1
after (this PR, default)	449.3

# ctx=128:   off 437.11 -> default 449.29  (+2.79%)
# ctx=512:   off 427.57 -> default 440.14  (+2.94%)
# ctx=4k:    off 408.02 -> default 419.86  (+2.90%)
# ctx=16k:   off 397.62 -> default 410.51  (+3.24%)
# ctx=32k:   off 375.55 -> default 387.46  (+3.17%)

All contexts clear the 2% gate. The FFN weight read fires every token regardless of KV depth, so the win is flat across context — a byte-reducing requant cannot slow decode. VRAM 23.7 → 22.0 GB. Qwythos-9B unchanged (fingerprint-gated no-op).

Cross-checked against a built-from-source origin/main @ a502a59 baseline (before = main, after = this branch, back-to-back same box): 438.5 → 448.7 @128 and 375.7 → 387.4 @32k — the off column above reproduces main within ~0.3% at every context.

Accuracy

The requant is lossy but confined to the accuracy-tolerant FFN projections. Teacher-forced top-1 agreement of the default (Q4_K) build vs the native (off) build on a held-out 299-token window:

positions        : 299
top-1 agreement  : 286/299 = 0.9565   (gate >= 0.90)
mean self-KL      : 0.0342 nats        (gate <= 0.20)
PPL  off 4.486 -> default 4.507

Subject to the harness KL/top-1 gate (bench/scripts/accuracy.sh, top-1 ≥ 0.90, KL ≤ 0.20).

Relation to open/merged PRs

Original — no open or merged PR requantizes the Qwen3.6 shared-expert FFN or adds Q4_K shared-expert kernels, and none requantizes the routed-expert down (Q5_K→Q4_K).

• The two new kernels + launcher live in kernels/csrc/cuda/moe/expert_ffn_q4k.cu / moe.h — the shared-expert Q4_K path is new code touched by no other PR.
• Distinct from perf(qwen36): Q8_0→Q4_K requant of full-attention q/o projections #353 (Q8→Q4 of full-attention q/o projections, Lloyd-max fit): different tensor class (MoE FFN vs attention), and this adds new kernels rather than reusing an existing MMVQ path.
• Distinct from perf(qwen35): requantize dense FFN down Q6_K→Q4_K at load (~5% Qwythos decode) #323 (dense FFN down, Qwythos), perf(qwen35): Q4_K requant for Qwythos Q6 decode reads #329 (Qwythos Q6 QKV/lm-head), perf(qwen36): opt-in Q8→Q4 requant reward profile (~+6.3% @512) #350 (GDN attn_qkv/attn_gate), perf(qwen3.6): Q4_K on-read LM head for the 35B-A3B hybrid-MoE (reuses #329) #308 (lm-head): none touch the shared-expert FFN or routed-expert down.
• Only shared file is runtime/src/models/qwen35.cpp (+43/−9); the reused lines are the generic dev_quant_requant_q4k helper (extended to accept a Q5_K source) and the load-loop call sites — standard requant boilerplate, not another PR's contribution.

Test plan

• Build qwen3_gguf_bench / qwen3_gguf_score on RTX 5090 (sm_120)
• Same-box A/B at 128/512/4k/16k/32k, default vs both-off
• Built-from-source origin/main baseline vs branch, back-to-back
• Teacher-forced top-1 / self-KL vs native (off) build
• Qwythos-9B guard: fingerprint-gated no-op

[skyrocket2026]

🚩 Flagged: eval-gaming

This PR involves an account blocked for gaming the SN74 emission mechanism (sybil / coordinated duplicate farming): carlh7777.

Per the project's no-gaming policy these accounts are blocked: the PR is not evaluated, scored, or merged. See .github/FLAGGED.md for the evidence and record.