perf(qwen3.6): fused router GEMV+top-k + folded residual-norm (decode)

[fansilas]

Summary

Two decode critical-path latency cuts for Qwen3.6-35B-A3B (256-expert MoE), both bit-identical to main's output.

1. Fused router GEMV + top-k. The router today runs a split-K GEMV that writes 256 expert logits to global memory, then a separate single-block kernel reads them back and selects the top-8 with an O(num_experts^2) rank scan. On an nsys node trace that top-k kernel is ~6.6% of the 128-context token, sitting serial on the decode critical path (router -> gate/up). This PR fuses both into one kernel: the same split-K GEMV writes its logits, a grid-completion handshake (an atomicInc that wraps back to 0, so it is CUDA-graph-replay safe with no reset node) lets the last block run an in-kernel top-8. The select is a bitonic reduction tree that replaces the 8 serial warp-argmax passes with a 5-level tree of independent (ILP) shuffles, and the fusion removes the separate top-k launch plus its dependent-load gap. Selection is byte-identical to the existing rank-select (value descending, ties to the lower expert index; checked against a reference over 4000 tie-heavy cases and confirmed by a bit-exact decode score diff). Only the 256-expert path is fused; the Qwen3-30B guard (128 experts) keeps the existing rank-select unchanged.

2. Fold the routed+shared residual into the post-MoE norm. Each layer's residual_add(routed, shared) is folded into a 3-input add_rmsnorm3, deleting one graph node per layer on the latency-bound decode path. Bit-identical to main's bf16 rounding. Active only when n_shared > 0 (Qwen3.6); the guard (n_shared = 0) path is unchanged.

Distinct from open work: #237 tunes the router GEMV split factor and keeps the warp top-k (no fused kernel, no top-k rewrite); #188 / #191 remove a reload inside the 2-input add_rmsnorm2_q8 (a different function than the new 3-input fold).

Toggles for the A/B below (both default ON, 256-expert path): SPARKINFER_ROUTER_FUSED=0 restores the separate GEMV + top-k, SPARKINFER_ADDNORM3=0 restores the separate residual add.

Proof of speedup

• Tested on RTX 5090 (sm_120)

Decode tok/s (end-to-end, qwen3_gguf_bench, Qwen3.6-35B-A3B-UD-Q4_K_M.gguf, 128 decode tokens, 128-context):

	decode tok/s
before (main)	392.7
after (this PR)	409.3

Interleaved same-binary A/B (median of 3, env toggle flips both levers), all three scored contexts:

# ctx=128:  before 392.7 -> after 409.3  (+4.2%)
# ctx=512:  before 385.2 -> after 400.4  (+3.9%)
# ctx=4096: before 369.0 -> after 382.4  (+3.6%)

-- ctx=128 (before / after) --
392.93 409.42
392.77 409.25
392.46 409.18
-- ctx=512 --
385.50 400.47
385.28 400.29
384.83 400.32
-- ctx=4096 --
369.98 383.16
368.86 382.15
368.31 381.75

Accuracy: the fused router's selection and the folded norm are both byte-identical to main, so the decode score is unchanged (a top-5 score diff over 48 fuzzed prompts is empty). Qwen3-30B-A3B guard paths (128-expert router, n_shared=0 norm) are untouched.

Relation to open PRs

All new kernels here are original and implement optimizations no open PR does. moe_router_fused_kernel reuses the base-repo gemv_f32_sk<float,4> body verbatim (it is base code, unchanged by any PR) and adds the grid-completion + bitonic top-8; add_rmsnorm3 is a new 3-input sibling of the base add_rmsnorm2, not a copy of any PR.

The only file this shares with other open PRs is runtime/src/models/qwen35.cpp, whose forward_token many decode PRs edit in disjoint regions. The changed hunks here do not coincide with any of them:

• vs perf(qwen36): retune GEMV/router/MoE decode scheduling #237 (router.cu): perf(qwen36): retune GEMV/router/MoE decode scheduling #237 only edits launch_moe_router's dispatch; this PR does not touch that function and instead adds new symbols.
• vs perf(qwen3.6): fuse decode kernel launches — shared-expert + GDN conv/L2-norm #239 (shared-expert gate/up/swiglu fusion, ~line 556) and perf(moe): Q5_K fp split-K Qwen specialization + gate_up F=512 dispatch (+4.3% Qwen3.6 decode) #267 (shared-gate gemv_sigmoid, ~639, and GDN proj ~406): those edits sit between/outside this PR's router edit (~599) and residual-fold edit (~665-700); no shared changed line.
• vs perf(decode): occupancy-first adaptive n_splits — 64-split mid-context plateau (skip 128) #134 (adaptive n_splits ladder, ~207-225): ~380 lines away from any edit here.
• vs perf(decode): default gate/up to 1-row mmvq2_qwen + rmsnorm load elimination (sm_120) #188/perf(rmsnorm): drop redundant out_sum reload in fused add_rmsnorm2_q8 (bit-identical) #191 (add_rmsnorm2_q8 reload removal): this PR does not modify add_rmsnorm2_q8; its new add_rmsnorm3_q8 keeps the reload (the opposite change).

[ai-hpc]

📊 sparkinfer auto-eval — 7c889bb

metric	value
label	eval:BASELINE
scored decode (16384 ctx · 16k-context · Qwen3.6)	0.0 tok/s
correctness (Qwen3.6 vs llama.cpp)	top-1 97.0% · KL 0.0188
Qwen3.6 128-token no-regression gate	401.77 tok/s · pass
Qwen3.6 512-context no-regression gate	395.13 tok/s · pass
Qwen3.6 4k-context no-regression gate	380.86 tok/s · pass
Qwen3-30B-A3B guard — accuracy	top-1 96.5% · KL 0.0179 · pass
Qwen3-30B-A3B guard — 128-token	499.22 tok/s · pass
Qwen3-30B-A3B guard — 512-context	474.88 tok/s · pass
Qwen3-30B-A3B guard — 4k-context	397.24 tok/s · pass
Qwen3-30B-A3B guard — 16k-context	333.3 tok/s · pass
Qwen3-30B-A3B guard — 32k-context	262.33 tok/s · pass

No same-box main baseline was set; this run establishes one.

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

[ai-hpc]

📊 sparkinfer auto-eval — 7c889bb

metric	value
label	eval:BASELINE
scored decode (16384 ctx · 16k-context · Qwen3.6)	0.0 tok/s
correctness (Qwen3.6 vs llama.cpp)	top-1 98.2% · KL 0.0108
Qwen3.6 128-token no-regression gate	401.67 tok/s · pass
Qwen3.6 512-context no-regression gate	395.13 tok/s · pass
Qwen3.6 4k-context no-regression gate	380.84 tok/s · pass
Qwen3-30B-A3B guard — accuracy	top-1 95.8% · KL 0.0209 · pass
Qwen3-30B-A3B guard — 128-token	498.9 tok/s · pass
Qwen3-30B-A3B guard — 512-context	474.6 tok/s · pass
Qwen3-30B-A3B guard — 4k-context	397.09 tok/s · pass
Qwen3-30B-A3B guard — 16k-context	332.71 tok/s · pass
Qwen3-30B-A3B guard — 32k-context	262.26 tok/s · pass

No same-box main baseline was set; this run establishes one.

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

[ai-hpc]

✅ sparkinfer auto-eval — 7c889bb

metric	value
label	eval:BASELINE
commit	7c889bb
model	Qwen3.6-35B-A3B
guard model	Qwen3-30B-A3B
pass	true
scored decode	401.69 tok/s (128-context)
frontier	386.9 tok/s
delta	+14.79 tok/s (3.8% over frontier)
pct of llama	5.4%
difficulty mult	4.0×
effective pct	21.4%
top-1 match	96.8%
KL divergence	0.0140
guard verdict	speed: true, accuracy: true
guard speed	380.37 tok/s (128)

⚠️ This eval used Qwen3.6's llama.cpp ref (275.81) as difficulty anchor — inflated to XL. Re-running with the fix.

[ai-hpc]

⚠️ sparkinfer auto-eval errored for 7c889bb — re-run manually.

[ai-hpc]

⚪ sparkinfer auto-eval — 7c889bb

metric	value
label	eval:none
scored decode (128 ctx · 128-context · Qwen3.6)	381.39 tok/s
correctness (Qwen3.6 vs llama.cpp)	top-1 96.7% · KL 0.0184
vs same-box main	381.29 tok/s → +0.1% (+0.1)
Qwen3.6 128-token no-regression gate	381.39 tok/s vs main 381.29 tok/s · pass
Qwen3.6 512-context no-regression gate	375.90 tok/s vs main 374.93 tok/s · pass
Qwen3.6 4k-context no-regression gate	360.25 tok/s vs main 359.58 tok/s · pass
Qwen3-30B guard — accuracy	top-1 93.7% · KL 0.0395 · pass
Qwen3-30B guard — 128-token	459.60 tok/s · pass
Qwen3-30B guard — 512-context	437.28 tok/s · pass
Qwen3-30B guard — 4k-context	360.32 tok/s · pass
Qwen3-30B guard — 16k-context	306.56 tok/s · pass
Qwen3-30B guard — 32k-context	241.25 tok/s · pass

Within the significance gate — no verified speedup over same-box main.

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.