perf(decode): pack2 MMVQ for attention Q4_K Wq and O projections by claytonlin1110 · Pull Request #272 · gittensor-ai-lab/sparkinfer

claytonlin1110 · 2026-07-06T20:29:19Z

Summary

Halve CUDA-graph node count for the two largest attention Q4_K MMVQ projections (Wq @ 4096 rows, Wo @ 2048 rows) by packing two output rows per CTA with the same faithful 4-warp kfixed dot loop. Reduces bs=1 launch/graph-replay overhead on the decode hot path.

Closes #271, related #141.

Proof of speedup

Tested on RTX 5090 (sm_120)

Decode tok/s (end-to-end, from bench/scripts/bench.sh — required for evaluation):

	decode tok/s
before (main)	367.0
after (this PR)	367.27

# sparkinfer auto-eval — 59dd212 (RTX 5090, same-box main vs PR)
# scored context: 4k-context · Qwen3.6 · 128 generated tokens

Qwen3.6 4k-context no-regression gate | 367.27 tok/s vs main 367.0 tok/s → +0.1%
Qwen3.6 128-token no-regression gate  | 386.9 tok/s vs main 386.9 tok/s · pass
Qwen3.6 512-context no-regression gate| 380.84 tok/s vs main 380.92 tok/s · pass

Qwen3-30B-A3B guard — 128-token | 499.54 tok/s · pass
Qwen3-30B-A3B guard — 512-context | 475.11 tok/s · pass
Qwen3-30B-A3B guard — 4k-context | 396.83 tok/s · pass
Qwen3-30B-A3B guard — 16k-context | 332.78 tok/s · pass
Qwen3-30B-A3B guard — 32k-context | 262.6 tok/s · pass

correctness (Qwen3.6 vs llama.cpp) | top-1 95.9% · KL 0.0214
verdict: eval:none — within significance gate, no verified speedup over same-box main

ai-hpc · 2026-07-07T01:41:13Z

📊 sparkinfer auto-eval — `59dd212`

metric	value
label	`eval:BASELINE`
scored decode (16384 ctx · 16k-context · Qwen3.6)	0.0 tok/s
correctness (Qwen3.6 vs llama.cpp)	top-1 94.8% · KL 0.0138
Qwen3.6 128-token no-regression gate	387.15 tok/s · pass
Qwen3.6 512-context no-regression gate	380.96 tok/s · pass
Qwen3.6 4k-context no-regression gate	367.46 tok/s · pass
Qwen3-30B-A3B guard — accuracy	top-1 95.8% · KL 0.0175 · pass
Qwen3-30B-A3B guard — 128-token	499.59 tok/s · pass
Qwen3-30B-A3B guard — 512-context	474.87 tok/s · pass
Qwen3-30B-A3B guard — 4k-context	396.83 tok/s · pass
Qwen3-30B-A3B guard — 16k-context	332.66 tok/s · pass
Qwen3-30B-A3B guard — 32k-context	262.4 tok/s · pass

No same-box main baseline was set; this run establishes one.

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

ai-hpc · 2026-07-07T03:54:20Z

📊 sparkinfer auto-eval — `59dd212`

metric	value
label	`eval:BASELINE`
scored decode (16384 ctx · 16k-context · Qwen3.6)	0.0 tok/s
correctness (Qwen3.6 vs llama.cpp)	top-1 96.1% · KL 0.0231
Qwen3.6 128-token no-regression gate	387.07 tok/s · pass
Qwen3.6 512-context no-regression gate	381.19 tok/s · pass
Qwen3.6 4k-context no-regression gate	367.3 tok/s · pass
Qwen3-30B-A3B guard — accuracy	top-1 96.4% · KL 0.0192 · pass
Qwen3-30B-A3B guard — 128-token	498.46 tok/s · pass
Qwen3-30B-A3B guard — 512-context	473.25 tok/s · pass
Qwen3-30B-A3B guard — 4k-context	395.39 tok/s · pass
Qwen3-30B-A3B guard — 16k-context	331.93 tok/s · pass
Qwen3-30B-A3B guard — 32k-context	262.24 tok/s · pass

No same-box main baseline was set; this run establishes one.

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

ai-hpc · 2026-07-07T04:22:59Z

⚪ sparkinfer auto-eval — `59dd212`

metric	value
label	`eval:none`
scored decode (4096 ctx · 4k-context · Qwen3.6)	367.27 tok/s
vs same-box main	367.0 tok/s → +0.1% (+0.3)
correctness (Qwen3.6 vs llama.cpp)	top-1 95.9% · KL 0.0214
Qwen3.6 128-token no-regression gate	386.9 tok/s vs main 386.9 tok/s · pass
Qwen3.6 512-context no-regression gate	380.84 tok/s vs main 380.92 tok/s · pass
Qwen3.6 4k-context no-regression gate	367.27 tok/s vs main 367.0 tok/s · pass
Qwen3-30B-A3B guard — accuracy	top-1 94.9% · KL 0.0211 · pass
Qwen3-30B-A3B guard — 128-token	499.54 tok/s · pass
Qwen3-30B-A3B guard — 512-context	475.11 tok/s · pass
Qwen3-30B-A3B guard — 4k-context	396.83 tok/s · pass
Qwen3-30B-A3B guard — 16k-context	332.78 tok/s · pass
Qwen3-30B-A3B guard — 32k-context	262.6 tok/s · pass

Within the significance gate — no verified speedup over same-box main.

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

Co-authored-by: Cursor <cursoragent@cursor.com>

skyrocket2026 · 2026-07-09T00:24:40Z

⚪ sparkinfer auto-eval — `e39923b`

metric	value
label	`eval:none`
scored decode (128 ctx · 128-context · Qwen3.6)	413.26 tok/s
vs same-box main	412.63 tok/s → +0.2% (+0.6)
correctness (Qwen3.6 vs llama.cpp)	top-1 97.6% · KL 0.0194
Qwen3.6 128-token no-regression gate	413.26 tok/s vs main 412.63 tok/s · pass
Qwen3.6 512-context no-regression gate	405.69 tok/s vs main 405.33 tok/s · pass
Qwen3.6 4k-context no-regression gate	389.42 tok/s vs main 388.98 tok/s · pass
Qwen3.6 16k-context no-regression gate	355.96 tok/s vs main 412.63 tok/s · pass
Qwen3.6 32k-context no-regression gate	319.99 tok/s · pass
Qwen3-30B-A3B guard — accuracy	top-1 94.3% · KL 0.0141 · pass
Qwen3-30B-A3B guard — 128-token	485.01 tok/s · pass
Qwen3-30B-A3B guard — 512-context	462.9 tok/s · pass
Qwen3-30B-A3B guard — 4k-context	385.08 tok/s · pass
Qwen3-30B-A3B guard — 16k-context	320.55 tok/s · pass
Qwen3-30B-A3B guard — 32k-context	255.33 tok/s · pass

Within the significance gate — no verified speedup over same-box main.

RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.

* feat(polaris): verifiable eval receipts with Ed25519 signatures Adds cryptographic attestation to the eval pipeline so third parties can verify results without re-running GPU jobs. New files under eval/polaris/: - receipt.py: schema, canonicalization, Ed25519 sign/verify, AttestationBuilder, ReceiptValidator, compute_build_hash - judge.py: assembles unsigned attestation on eval box (no key needed) - verify.py: standalone CLI verifier — no GPU required - test_receipt.py: 29 unit tests (building, signing, verification, tamper detection, edge cases) - sparkinfer_eval.pub: Ed25519 public key trust anchor Modified: - pr_eval_bot.py: --polaris flag, key loading, attestation signing, receipt upload to sparkinfer-log, dashboard fields - vast_eval.py: --polaris flag, SSH command pins eval/polaris/ from origin/main, runs judge.py, passes POLARIS_ATTESTATION through Trust model: judge assembles on eval box (pinned to origin/main), bot signs on trusted host. Private key never touches the eval box. * feat(polaris): upgrade to Intel TDX hardware attestation via Polaris API Replace Ed25519 software signatures with Polaris TDX hardware attestation for scoring integrity. The scoring step (correctness gate, guard gate, label computation) now runs inside an Intel TDX enclave with DCAP-quoted receipts. New files: - eval/polaris/scoring.py: self-contained scoring script (stdlib only) that runs inside the TDX enclave, replicating evaluate_dual.sh lines 148-212 - eval/polaris/client.py: PolarisClient for the /v1/attest API Modified files: - eval/polaris/receipt.py: add build_polaris_receipt() and ReceiptValidator.verify_tdx() for TDX receipt assembly and verification. verify() auto-detects receipt type (tdx-quote vs Ed25519). - eval/polaris/verify.py: auto-detect TDX receipts, show Intel DCAP verification status, distinguish TDX vs Ed25519 in output. - eval/pr_eval_bot.py: when POLARIS_API_KEY is set, submit scoring to Polaris TDX enclave and build TDX receipts. Ed25519 signing is preserved as fallback when only SPARKINFER_POLARIS_PRIVATE_KEY is set. The Ed25519 path and all 29 existing tests continue to pass unchanged. * docs: add Polaris verifiable eval receipts documentation * dashboard: PR #272 -> eval:none (413.26 tok/s) * fix(eval): 16k/32k guard baseline fallback — bash expansion bug + display Two bugs caused Qwen3.6 16k/32k guard baselines to show wrong values: 1. evaluate_dual.sh: ${SPARKINFER_P_GUARD_16K_BASELINE:-338.55} treats the explicit "0" passed by the bot as a valid non-empty string, so the :- expansion never fires and the hardcoded default is never used. Fix: two-step fallback — try env var, then hardcoded default if still 0. 2. pr_eval_bot.py: the display fallback for ctx_16384_tps used frontier_tps (the 128-context baseline, ~412) instead of a proper 16k default (~339). ctx_32768_tps had no fallback at all. Fix: _GUARD_BASE_FALLBACK dict with per-context defaults matching evaluate_dual.sh. * dashboard: PR #282 -> eval:XL (427.28 tok/s) * fix(eval): trigger Qwen3.6 baseline sweep when any context is missing Previously the 5-context sweep only ran when 128 baseline was 0. Since the bot always passes a non-zero 128 baseline, the sweep never ran and 16k/32k baselines were never measured fresh — falling through to broken ${VAR:-default} expansions and eventually to wrong display values. Now the sweep triggers when ANY of 128/512/4k/16k/32k is 0, so passing only 128/512/4k (with 16k/32k as 0) still gets a fresh measurement on the eval box. * fix(eval): resolve vastai CLI path and update default instance to 44206573 - vast_eval.py: resolve vastai binary via shutil.which / ~/.local/bin/vastai so subprocess.run works from bot/cron contexts where PATH may be limited. - pr_eval_bot.py: update hardcoded VAST_DEFAULT_INSTANCE fallback to 44206573. * fix(eval): measure Qwen3.6 16k/32k baselines in bot pre-sweep The same-box Qwen3.6 baseline sweep now covers all 5 contexts (128, 512, 4k, 16k, 32k) instead of just 3. The measured values are passed as --p-guard-16k/32k-baseline to vast_eval.py, which forwards them to the eval box via SPARKINFER_P_GUARD_16K/32K_BASELINE env vars. This gives evaluate_dual.sh real measured baselines to use (instead of falling through to hardcoded defaults). * dashboard: PR #282 -> eval:XL (427.05 tok/s) * fix(eval): point eval log repo and URL to main sparkinfer repo Previously logs were pushed to gittensor-ai-lab/sparkinfer-log.git which skyrocket2026 lacks permission to push to. Move to the main sparkinfer repo and update the log page URL accordingly. * dashboard: PR #283 -> eval:none (415.04 tok/s) --------- Co-authored-by: Skyrocket <>

Co-authored-by: Cursor <cursoragent@cursor.com>

skyrocket2026 · 2026-07-09T16:54:46Z

⚠️ sparkinfer auto-eval errored for 938a7f5 — re-run manually.

log tail

_version":"","driver_version":"580.159.03"},"measurements":{"primary":{"model":"Qwen3.6-35B-A3B","model_sha256":"","ctx_128_tps":409.5,"ctx_512_tps":403.24,"ctx_4096_tps":387.47,"ctx_16384_tps":355.3,"ctx_32768_tps":318.88,"top1":0.9825,"kl":0.0116,"guard_128_baseline":408.95,"guard_128_ratio":1.0013,"guard_128_pass":true,"guard_512_baseline":402.91,"guard_512_ratio":1.0008,"guard_512_pass":true,"guard_4k_baseline":387.63,"guard_4k_ratio":0.9996,"guard_4k_pass":true,"guard_16k_baseline":372.39,"guard_16k_ratio":0.9541,"guard_16k_pass":false,"guard_32k_baseline":351.42,"guard_32k_ratio":0.9074,"guard_32k_pass":false},"guard":{"model":"Qwen3-30B-A3B","model_sha256":"","ctx_128_tps":489.29,"ctx_512_tps":468.84,"ctx_4096_tps":385.89,"ctx_16384_tps":325.04,"ctx_32768_tps":260.02,"top1":0.9579,"kl":0.0209,"speed_ok":true,"accuracy_ok":true}},"verdict":{"model":"Qwen3.6-35B-A3B","label":"REJECT","pass":false,"tps":409.5,"delta_tps":0.55,"pct_over_frontier":0.1,"score_context":128,"best_context_label":"128-context","context_gains_pct":{"128-context":0.13,"512-context":0.08,"4k-context":-0.04,"16k-context":-4.59,"32k-context":-9.26},"regression_labels":["regression-16k","regression-32k"],"guard_regression_labels":[],"reason":"16k-context decode no-regression gate: 355.30 tok/s < 98% of main 372.39 tok/s; 32k-context decode no-regression gate: 318.88 tok/s < 98% of main 351.42 tok/s"},"timestamp_utc":"2026-07-09T16:54:45Z"}
>> bare-metal box left running (ssh root@91.224.44.227:50200)

ai-hpc added area:kernels subsystem (emission weight 0.42) test-on-5090 Maintainer-approved to evaluate on RTX 5090 (greenlight) eval:BASELINE sparkinfer auto-eval verdict: BASELINE 16k-context UI-only: strongest measured context in sparkinfer eval labels Jul 7, 2026

ai-hpc added eval:BASELINE sparkinfer auto-eval verdict: BASELINE and removed eval:BASELINE sparkinfer auto-eval verdict: BASELINE labels Jul 7, 2026

ai-hpc added a commit that referenced this pull request Jul 7, 2026

dashboard: PR #272 -> eval:BASELINE (0.0 tok/s)

f9f5dd0

ai-hpc added eval:none sparkinfer auto-eval verdict: none 4k-context UI-only: strongest measured context in sparkinfer eval and removed eval:BASELINE sparkinfer auto-eval verdict: BASELINE 16k-context UI-only: strongest measured context in sparkinfer eval labels Jul 7, 2026

ai-hpc added a commit that referenced this pull request Jul 7, 2026

dashboard: PR #272 -> eval:none (367.27 tok/s)

81de1ab

claytonlin1110 and others added 2 commits July 8, 2026 10:10

perf(decode): pack2 MMVQ for attention Q4_K Wq and O projections

d64014d

Co-authored-by: Cursor <cursoragent@cursor.com>

perf(decode): fuse QKV pack2 mmvq to cut activation DRAM passes

e39923b

Co-authored-by: Cursor <cursoragent@cursor.com>

claytonlin1110 force-pushed the perf/attn-pack2-mmvq branch from 92bffb5 to e39923b Compare July 8, 2026 15:12

skyrocket2026 added a commit that referenced this pull request Jul 9, 2026

dashboard: PR #272 -> eval:none (413.26 tok/s)

6466408

perf(gdn): fuse wqkv+gate pack2 mmvq on Qwen3.6 linear layers

938a7f5

Co-authored-by: Cursor <cursoragent@cursor.com>

jimcody1995 mentioned this pull request Jul 9, 2026

perf(qwen36): decode fuses + FAGQA4 restore + n_splits=160 (+8.6% @32k) #294

Open

1 task

fansilas mentioned this pull request Jul 11, 2026

perf(attn): grid-target KV-split cap for int8-MMA long-context decode #345

Draft

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf(decode): pack2 MMVQ for attention Q4_K Wq and O projections#272

perf(decode): pack2 MMVQ for attention Q4_K Wq and O projections#272
claytonlin1110 wants to merge 3 commits into
gittensor-ai-lab:mainfrom
claytonlin1110:perf/attn-pack2-mmvq

claytonlin1110 commented Jul 6, 2026 •

edited

Loading

Uh oh!

ai-hpc commented Jul 7, 2026 •

edited

Loading

Uh oh!

ai-hpc commented Jul 7, 2026 •

edited

Loading

Uh oh!

ai-hpc commented Jul 7, 2026

Uh oh!

skyrocket2026 commented Jul 9, 2026

Uh oh!

skyrocket2026 commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

claytonlin1110 commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Proof of speedup

Uh oh!

ai-hpc commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 sparkinfer auto-eval — 59dd212

Uh oh!

ai-hpc commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 sparkinfer auto-eval — 59dd212

Uh oh!

ai-hpc commented Jul 7, 2026

⚪ sparkinfer auto-eval — 59dd212

Uh oh!

skyrocket2026 commented Jul 9, 2026

⚪ sparkinfer auto-eval — e39923b

Uh oh!

skyrocket2026 commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

claytonlin1110 commented Jul 6, 2026 •

edited

Loading

ai-hpc commented Jul 7, 2026 •

edited

Loading

📊 sparkinfer auto-eval — `59dd212`

ai-hpc commented Jul 7, 2026 •

edited

Loading

📊 sparkinfer auto-eval — `59dd212`

⚪ sparkinfer auto-eval — `59dd212`

⚪ sparkinfer auto-eval — `e39923b`