perf(moe): Q5_K fp split-K Qwen specialization + gate_up F=512 dispatch (+4.3% Qwen3.6 decode) by inference2026 · Pull Request #267 · gittensor-ai-lab/sparkinfer

inference2026 · 2026-07-06T09:14:18Z

Summary

Speeds up Qwen3.6-35B-A3B decode +4.3% (254.5 → 265.3 tok/s at 128) via two compile‑time specializations for the Qwen3.6 shape (ffn=512, top_k=8).

Q5_K fp split‑K down (primary): when SPARKINFER_SPLITK=1, a new template kernel down_fp_splitk_qwen_kernel<S,NBLK,TOPK,DOWN_TYPE> makes NBLK=2, DBB=176, and TOTAL=16 compile‑time constants for the Q5_K (ggml type 13) expert‑down fp path on 37/40 layers. The compiler fully unrolls the work‑item loop and eliminates integer division/mod. When split‑K is off (default), falls through to the one‑warp‑per‑row kernel — same as main, zero regression.

Gate_up F=512 dispatch (secondary): adds {2048,512,8} instantiations for the gate/up mmvq2 templates.

Proof of speedup

Tested on RTX 5090 (sm_120)

Decode tok/s (end‑to‑end, from bench/scripts/bench.sh):

	decode tok/s
before (main)	254.5
after (this PR, splitk=1)	265.3

Per‑context:

context	main	this PR (splitk=1)	speedup
128	254.4	265.2	+4.3%
512	251.2	261.8	+4.2%
4k	255.2	266.4	+4.4%

With splitk=0 (eval default): matches main exactly — zero regression on all contexts.
Qwen3‑30B no‑regression guard: 497.3 tok/s — unchanged. ctest 7/7.

Qwen3.6-35B-A3B-UD-Q4_K_M · RTX 5090 (sm_120, CUDA 13) · qwen3_gguf_bench, n=128, bs=1
      ctx=128  main 254.4 → PR 265.2 (+4.3%)
      ctx=512  main 251.2 → PR 261.8 (+4.2%)
      ctx=4k   main 255.2 → PR 266.4 (+4.4%)
Qwen3-30B-A3B-Q4_K_M · 497.3 tok/s (unchanged)
ctest: 100% tests passed, 7/7

ai-hpc · 2026-07-06T10:37:09Z

⚪ sparkinfer auto-eval — `20df85f`

metric	value
label	`eval:none`
scored decode (512 ctx · 512-context · Qwen3.6)	252.23 tok/s
vs same-box main	251.87 tok/s → +0.1% (+0.4)
correctness (Qwen3.6 vs llama.cpp)	top-1 97.1% · KL 0.0161
Qwen3.6 128-token no-regression gate	255.02 tok/s vs main 254.72 tok/s · pass
Qwen3.6 512-context no-regression gate	252.23 tok/s vs main 251.87 tok/s · pass
Qwen3.6 4k-context no-regression gate	239.27 tok/s vs main 239.0 tok/s · pass
Qwen3-30B-A3B guard — accuracy	top-1 95.3% · KL 0.0196 · pass
Qwen3-30B-A3B guard — 128-token	498.01 tok/s · pass
Qwen3-30B-A3B guard — 512-context	473.04 tok/s · pass
Qwen3-30B-A3B guard — 4k-context	395.27 tok/s · pass
Qwen3-30B-A3B guard — 16k-context	331.54 tok/s · pass
Qwen3-30B-A3B guard — 32k-context	262.01 tok/s · pass

Within the significance gate — no verified speedup over same-box main.

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

ai-hpc · 2026-07-06T12:56:47Z

✅ sparkinfer auto-eval — `c42844c`

metric	value
label	`eval:M`
scored decode (128 ctx · 128-context · Qwen3.6)	278.76 tok/s
vs same-box main	254.72 tok/s → +9.4% (+24.0)
correctness (Qwen3.6 vs llama.cpp)	top-1 97.7% · KL 0.0165
Qwen3.6 128-token no-regression gate	278.76 tok/s vs main 254.72 tok/s · pass
Qwen3.6 512-context no-regression gate	275.13 tok/s vs main 251.87 tok/s · pass
Qwen3.6 4k-context no-regression gate	259.22 tok/s vs main 239.0 tok/s · pass
Qwen3-30B-A3B guard — accuracy	top-1 96.3% · KL 0.0337 · pass
Qwen3-30B-A3B guard — 128-token	481.04 tok/s · pass
Qwen3-30B-A3B guard — 512-context	458.7 tok/s · pass
Qwen3-30B-A3B guard — 4k-context	385.49 tok/s · pass
Qwen3-30B-A3B guard — 16k-context	326.91 tok/s · pass
Qwen3-30B-A3B guard — 32k-context	254.88 tok/s · pass

Verified speedup over same-box origin/main — 278.76 tok/s (main was 254.72 tok/s).

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

ai-hpc · 2026-07-06T13:06:21Z

The round's merge-first PR was just merged. Please rebase this branch onto main — the bot re-evaluates it on push (crediting your marginal gain on top of what merged).

ai-hpc · 2026-07-06T14:07:16Z