perf(qwen36): Q4_K shared-expert FFN kernels + Q8_0/Q5_K→Q4_K requant of the MoE FFN stage (+2.8–3.2%)

[milosde111]

Summary

Two load-time weight requantizations targeting the MoE FFN stage of the Qwen3.6-35B-A3B
decode path, gated to the Qwen3.6 fingerprint (is_qwen35_or_qwen36_hybrid_moe) and off
by default on every other model. The FFN stage runs on every token across all 40 layers,
so shrinking its per-token weight reads is a broad, context-flat decode win.

• Shared-expert FFN Q8_0 → Q4_K (new kernels). Qwen3.6-35B-A3B UD ships the always-on
  shared expert (ffn_{gate,up,down}_shexp) as Q8_0 — 34 bytes/block on a read that fires
  every layer, every token. This requantizes it to Q4_K once at model load (~47% fewer weight
  bytes) and decodes it through two new Q4_K shared-expert MMVQ kernels
  (shared_gate_up_q4k_mmvq_kernel, shared_down_q4k_mmvq_kernel + launcher
  launch_shared_expert_q4k_mmvq). They mirror the existing Q8 shared-expert control flow
  (FNQ Q8_1 → gate/up·silu → quant_h Q8_1 → down); only the weight dot swaps to the int8
  dp4a Q4_K path. Gated to the exact shared-expert shape the kernels specialize (<2048, 512>).
• Routed-expert down Q5_K → Q4_K (no new kernel). The top-8 routed experts keep their down
  projection at Q5_K while gate/up are already Q4_K. Requantizing ffn_down_exps Q5_K→Q4_K at
  load stacks a further byte reduction on the same decode-critical stage and routes automatically
  through the pre-existing Q4_K routed-down MMVQ dispatch (down_type == 12).

Strict no-op on Qwythos-9B and any non-matching model.

A/B toggles (same binary): SPARKINFER_SHEXP_REQUANT_Q4K=0 restores native Q8_0 shared-expert
reads; SPARKINFER_ROUTED_DOWN_REQUANT_Q4K=0 restores native Q5_K routed-down. Both set = the
"before" column below.

Proof of speedup

• Tested on RTX 5090 (sm_120)

Decode tok/s (end-to-end, qwen3_gguf_bench, Qwen3.6-35B-A3B-UD-Q4_K_M.gguf, 128 decode tokens),
default vs SPARKINFER_SHEXP_REQUANT_Q4K=off + SPARKINFER_ROUTED_DOWN_REQUANT_Q4K=off, same binary:

	decode tok/s (128)
before (=off, native)	437.1
after (this PR, default)	449.3

# ctx=128:   off 437.11 -> default 449.29  (+2.79%)
# ctx=512:   off 427.57 -> default 440.14  (+2.94%)
# ctx=4k:    off 408.02 -> default 419.86  (+2.90%)
# ctx=16k:   off 397.62 -> default 410.51  (+3.24%)
# ctx=32k:   off 375.55 -> default 387.46  (+3.17%)

All contexts clear the 2% gate. The FFN weight read fires every token regardless of KV depth, so
the win is flat across context — a byte-reducing requant cannot slow decode. VRAM 23.7 → 22.0 GB.
Qwythos-9B unchanged (fingerprint-gated no-op).

Cross-checked against a built-from-source origin/main baseline (before = main, after = this branch,
back-to-back same box): 438.5 → 448.7 @128 and 375.7 → 387.4 @32k — the off column reproduces main
within ~0.3% at every context.

Accuracy. The requant is lossy but confined to the accuracy-tolerant FFN projections. Teacher-forced
top-1 agreement of the default (Q4_K) build vs the native (off) build on a held-out 299-token window is
286/299 = 0.9565 (gate ≥ 0.90), mean self-KL 0.0342 nats (gate ≤ 0.20), PPL off 4.486 → default 4.507;
subject to the harness KL/top-1 gate (bench/scripts/accuracy.sh, top-1 ≥ 0.90, KL ≤ 0.20).

Relation to open/merged PRs

Original — no open or merged PR requantizes the Qwen3.6 shared-expert FFN or adds Q4_K shared-expert
kernels, and none requantizes the routed-expert down (Q5_K→Q4_K). The two new kernels + launcher live in
kernels/csrc/cuda/moe/expert_ffn_q4k.cu / moe.h — the shared-expert Q4_K path is new code touched by
no other PR. Distinct from #353 (Q8→Q4 of full-attention q/o projections): different tensor class (MoE FFN
vs attention), and this adds new kernels rather than reusing an existing MMVQ path. Distinct from #323 /
#329 / #350 / #308: none touch the shared-expert FFN or routed-expert down. The only shared file is
runtime/src/models/qwen35.cpp, where the reused lines are the generic dev_quant_requant_q4k helper
(extended to accept a Q5_K source) and the load-loop call sites — standard requant boilerplate.

Files changed

• kernels/csrc/cuda/moe/expert_ffn_q4k.cu — new Q4_K shared-expert kernels + launcher.
• kernels/include/sparkinfer/kernels/moe.h — launcher declaration.
• runtime/src/models/qwen35.cpp — generic dev_quant_requant_q4k extended to accept a Q5_K source;
  shared-expert (Q8_0→Q4_K) and routed-down (Q5_K→Q4_K) load-time requant call sites; forward-time
  dispatch to the new Q4_K shared launcher when the shared weights are Q4_K.

Test plan

• Build qwen3_gguf_bench / qwen3_gguf_score on RTX 5090 (sm_120)
• Same-box A/B at 128/512/4k/16k/32k, default vs both-off
• Built-from-source origin/main baseline vs branch, back-to-back
• Teacher-forced top-1 / self-KL vs native (off) build
• Qwythos-9B guard: fingerprint-gated no-op

[skyrocket2026]

⚠️ sparkinfer auto-eval errored for 231a487 — re-run manually.

log tail

k-baseline P_LLAMA_32K_BASELINE] [--bidir]
                    [--p35-guard-128-baseline P35_GUARD_128_BASELINE]
                    [--p35-guard-512-baseline P35_GUARD_512_BASELINE]
                    [--p35-guard-4k-baseline P35_GUARD_4K_BASELINE]
                    [--g35-guard-128-baseline G35_GUARD_128_BASELINE]
                    [--g35-guard-512-baseline G35_GUARD_512_BASELINE]
                    [--g35-guard-4k-baseline G35_GUARD_4K_BASELINE] [--triple]
                    [--primary-quant {Q4_K_M,Q8_0,BF16}]
                    [--g36-guard-128-baseline G36_GUARD_128_BASELINE]
                    [--g36-guard-512-baseline G36_GUARD_512_BASELINE]
                    [--g36-guard-4k-baseline G36_GUARD_4K_BASELINE]
                    [--g36-guard-16k-baseline G36_GUARD_16K_BASELINE]
                    [--g36-guard-32k-baseline G36_GUARD_32K_BASELINE]
                    [--eval-mode {longctx,short}] [--reuse REUSE]
                    [--ssh HOST:PORT] [--keep] [--destroy] [--gpu GPU]
                    [--image IMAGE] [--reuse-timeout REUSE_TIMEOUT]
                    [--new-timeout NEW_TIMEOUT] [--no-recreate] [--pinned]
                    [--destroy-on-error] [--polaris] [--no-polaris]
                    [--baseline-only]
vast_eval.py: error: unrecognized arguments: --p35-guard-32k-baseline 241.17 --p35-guard-64k-baseline 204.26 --p35-guard-128k-baseline 159.13 --g35-guard-32k-baseline 241.17 --g35-guard-64k-baseline 204.26 --g35-guard-128k-baseline 159.13

[skyrocket2026]

❌ sparkinfer auto-eval — 231a487

metric	value
label	eval:REJECT
Qwen3.5 score	eval-qwen35:REJECT (fail)
Qwen3.6 score	eval-qwen36:REJECT (fail)
Qwen3.5 vs same-box main	240.98 tok/s → +0.4% (+0.9)
Qwen3.5 scored decode (32768 ctx · 32k-context)	241.88 tok/s
Qwen3.5 correctness	top-1 93.1% · KL 0.0359
Qwen3.5 128-token no-regression gate	298.65 tok/s vs main 298.7 tok/s · pass
Qwen3.5 4k-context no-regression gate	283.2 tok/s vs main 283.33 tok/s · pass
Qwen3.5 32k-context no-regression gate	241.88 tok/s vs main 240.98 tok/s · pass
Qwen3.6 vs same-box main	426.93 tok/s → -6.4% (-27.2)
Qwen3.6 scored decode (32768 ctx · 32k-context)	399.68 tok/s
Qwen3.6 correctness	top-1 93.6% · KL 0.0323
Qwen3.6 128-token no-regression gate	435.3 tok/s vs main 473.09 tok/s · fail
Qwen3.6 512-context no-regression gate	441.53 tok/s vs main 480.48 tok/s · fail
Qwen3.6 4k-context no-regression gate	424.54 tok/s vs main 458.25 tok/s · fail
Qwen3.6 16k-context no-regression gate	416.51 tok/s vs main 450.85 tok/s · fail
Qwen3.6 32k-context no-regression gate	399.68 tok/s vs main 426.93 tok/s · fail
Qwen3.5 optimize	eval:REJECT · 241.88 tok/s · fail
Qwen3.5 optimize — Qwen3.6-35B-A3B guard accuracy	top-1 93.6% · KL 0.0323 · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 128	435.61 tok/s · fail
Qwen3.5 optimize — Qwen3.6-35B-A3B 512	441.49 tok/s · fail
Qwen3.5 optimize — Qwen3.6-35B-A3B 4k	424.48 tok/s · fail
Qwen3.5 optimize — Qwen3.6-35B-A3B 16k	417.06 tok/s · fail
Qwen3.5 optimize — Qwen3.6-35B-A3B 32k	399.55 tok/s · fail
Qwen3.6 optimize	eval:REJECT · 399.68 tok/s · fail
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) guard accuracy	top-1 93.1% · KL 0.0359 · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 128	298.82 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 4k	282.69 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 32k	241.62 tok/s · pass
regressions	regression-qwen36-128, regression-qwen36-512, regression-qwen36-4k, regression-qwen36-16k, regression-qwen36-32k

No context cleared the 2% significance gate while at least one context regressed. Auto-closing this PR.

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.