perf(attn): grid-target KV-split cap for int8-MMA long-context decode

[fansilas]

Summary

One decode scheduling fix for the int8 tensor-core flash-decode at long context, output-neutral to main (only the KV-work partition changes).

Grid-target n_splits cap. The depth-adaptive KV-split count jumps to 256 at >8k context — a value tuned for the older bf16 tile kernel. For the int8-MMA flash-decode (fa_split_gqa_mma_i8_kernel, 5 CTAs/SM), 256 over-splits: the num_kv_heads*256 grid spills a partial 2nd occupancy wave (tail), and the per-split chunk shrinks until the QK warps idle (gblk<8). On an nsys node trace the split kernel is ~27% of the 32k-context token and already runs at ~89% of memory bandwidth — so the win is the schedule, not the kernel.

This caps n_splits to a grid that fills ~3/4 of one occupancy wave: (num_sms*5*3/4)/num_kv_heads. The optimal grid is independent of seqlen — on the RTX 5090 (170 SMs) both 16k and 32k decode peak at grid ~640 (n_splits 159 at 4 KV heads). The cap only bites for higher KV-head counts: hd128 Qwen3-MoE (4 KV) -> 159; hd256 (2 KV -> cap 318, above the 256 tier) and short context (<8k, n_splits 32/128) are unchanged. The online-softmax combine is exact for any split count, so this is correctness-neutral.

A/B toggle: SPARKINFER_NSPLITS=256 restores the pre-cap fixed split count for the comparison below.

Proof of speedup

• Tested on RTX 5090 (sm_120)

Decode tok/s (end-to-end, bench/scripts/bench.sh, Qwen3-30B-A3B-Q4_K_M.gguf, 128 decode tokens):

	decode tok/s (32k)
before (main)	257.4
after (this PR)	287.1

Same-box A/B, both built on current main (v0.4.0), 2 reps each:

# ctx=16384: before 324.4 -> after 350.6  (+8.1%)
# ctx=32768: before 257.4 -> after 287.1  (+11.5%)

-- ctx=16384 (before / after) --
324.83 350.74
324.02 350.52
-- ctx=32768 --
257.47 287.65
257.34 286.64

128 / 512 / 4k are unchanged — the cap does not engage below 8k (int8 KV + the MMA path start at ctx>=8192). Correctness: the split count is a KV-work partition; the log-sum-exp combine is exact for any n_splits (main already ships 128 at 4k, 256 at 16k), so the attention output is unchanged up to float reduction order.

Relation to open PRs

The n_splits grid-target cap is original — no open PR does it. The only file touched is runtime/src/models/qwen35.cpp, shared with several open PRs (#329, #308, #294, #285, #283, #272, #267), but the change is a small self-contained addition to the depth-adaptive split block. Added-line containment against every one of those (and every merged/closed PR on this file) is <=14.3% — a single shared }. No kernel changes.

[skyrocket2026]

⚪ sparkinfer auto-eval — 0954339

metric	value
label	eval:none
Qwen3.5 score	eval-qwen35:none (pass)
Qwen3.6 score	eval-qwen36:none (pass)
Qwen3.5 vs same-box main	282.88 tok/s → +0.6% (+1.8)
Qwen3.5 scored decode (128 ctx · 128-context)	284.66 tok/s
Qwen3.5 correctness	top-1 96.8% · KL 0.0195
Qwen3.5 128-token no-regression gate	284.66 tok/s vs main 282.88 tok/s · pass
Qwen3.5 512-context no-regression gate	281.53 tok/s vs main 280.04 tok/s · pass
Qwen3.5 4k-context no-regression gate	272.64 tok/s vs main 271.88 tok/s · pass
Qwen3.6 vs same-box main	427.46 tok/s → -0.5% (-2.2)
Qwen3.6 scored decode (128 ctx · 128-context)	425.22 tok/s
Qwen3.6 correctness	top-1 96.1% · KL 0.0174
Qwen3.6 128-token no-regression gate	425.22 tok/s vs main 427.46 tok/s · pass
Qwen3.6 512-context no-regression gate	420.41 tok/s vs main 422.8 tok/s · pass
Qwen3.6 4k-context no-regression gate	403.36 tok/s vs main 405.75 tok/s · pass
Qwen3.6 16k-context no-regression gate	379.27 tok/s vs main 381.9 tok/s · pass
Qwen3.6 32k-context no-regression gate	351.92 tok/s vs main 354.31 tok/s · pass
Qwen3.5 optimize	eval:none · 284.66 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B guard accuracy	top-1 96.1% · KL 0.0174 · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 128	425.17 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 512	420.35 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 4k	403.29 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 16k	379.78 tok/s · pass
Qwen3.5 optimize — Qwen3.6-35B-A3B 32k	352.01 tok/s · pass
Qwen3.6 optimize	eval:none · 425.22 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) guard accuracy	top-1 96.8% · KL 0.0195 · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 128	284.36 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 512	281.0 tok/s · pass
Qwen3.6 optimize — Qwythos-9B (Q4_K_M) 4k	272.81 tok/s · pass

Within the significance gate — no verified speedup over same-box main.

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.