perf(qwen3.6): fuse Q-gate + O-proj Q8_1 quantize into one launch#283
Open
galuis116 wants to merge 1 commit into
Open
perf(qwen3.6): fuse Q-gate + O-proj Q8_1 quantize into one launch#283galuis116 wants to merge 1 commit into
galuis116 wants to merge 1 commit into
Conversation
On Qwen3.6 gated-Q attention layers the Q-gate (attn *= sigmoid(qgate), mul_sigmoid_kernel) writes bf16 attn, then the O-projection mmvq path immediately re-reads it to produce the Q8_1 activation (si_quantize_q8_1_blocks) — two back-to-back launches. mul_sigmoid_q8_kernel does both in one launch (one warp per 32-value Q8_1 block). Bit-identical to main: the fused kernel bf16-rounds x*sigmoid(gate) exactly as mul_sigmoid's bf16 store, then runs the amax/round/scale reduction copied verbatim from si_quantize_q8_1_blocks (mirrors the existing gated_norm_q8_warp fusion). SPARKINFER_GATE_Q8=0 reproduces main. No-op for Qwen3-30B (no q_has_gate) and non-mmvq wo paths. Decode tok/s (RTX 5090, Qwen3.6-35B-A3B-UD-Q4_K_M, n=256, ctx=0, bs=1), interleaved A/B on one binary (SPARKINFER_GATE_Q8 toggled), 3 runs each: before (GATE_Q8=0, == main): 405.79 after (default): 407.21 (+0.35%) ctest 7/7; compute-sanitizer memcheck 0 errors.
skyrocket2026
added
area:kernels
Member
⚪ sparkinfer auto-eval —
|
| metric | value |
|---|---|
| label | eval:none |
| scored decode (128 ctx · 128-context · Qwen3.6) | 415.04 tok/s |
| vs same-box main | 412.53 tok/s → +0.6% (+2.5) |
| correctness (Qwen3.6 vs llama.cpp) | top-1 99.0% · KL 0.0177 |
| Qwen3.6 128-token no-regression gate | 415.04 tok/s vs main 412.53 tok/s · pass |
| Qwen3.6 512-context no-regression gate | 407.28 tok/s vs main 405.25 tok/s · pass |
| Qwen3.6 4k-context no-regression gate | 390.79 tok/s vs main 388.62 tok/s · pass |
| Qwen3.6 16k-context no-regression gate | 357.36 tok/s vs main 355.78 tok/s · pass |
| Qwen3.6 32k-context no-regression gate | 320.69 tok/s vs main 319.97 tok/s · pass |
| Qwen3-30B-A3B guard — accuracy | top-1 95.4% · KL 0.0179 · pass |
| Qwen3-30B-A3B guard — 128-token | 485.1 tok/s · pass |
| Qwen3-30B-A3B guard — 512-context | 463.87 tok/s · pass |
| Qwen3-30B-A3B guard — 4k-context | 385.12 tok/s · pass |
| Qwen3-30B-A3B guard — 16k-context | 321.9 tok/s · pass |
| Qwen3-30B-A3B guard — 32k-context | 256.11 tok/s · pass |
Within the significance gate — no verified speedup over same-box main.
RTX 5090 (sm_120) · 128/512/4k/16k/32k guarded · scored vs same-box main · strongest context scores · built from source · correctness vs llama.cpp. Automated — not merged; merge manually after review.
skyrocket2026
added a commit
that referenced
this pull request
Jul 9, 2026
ai-hpc
pushed a commit
that referenced
this pull request
Jul 9, 2026
* feat(polaris): verifiable eval receipts with Ed25519 signatures Adds cryptographic attestation to the eval pipeline so third parties can verify results without re-running GPU jobs. New files under eval/polaris/: - receipt.py: schema, canonicalization, Ed25519 sign/verify, AttestationBuilder, ReceiptValidator, compute_build_hash - judge.py: assembles unsigned attestation on eval box (no key needed) - verify.py: standalone CLI verifier — no GPU required - test_receipt.py: 29 unit tests (building, signing, verification, tamper detection, edge cases) - sparkinfer_eval.pub: Ed25519 public key trust anchor Modified: - pr_eval_bot.py: --polaris flag, key loading, attestation signing, receipt upload to sparkinfer-log, dashboard fields - vast_eval.py: --polaris flag, SSH command pins eval/polaris/ from origin/main, runs judge.py, passes POLARIS_ATTESTATION through Trust model: judge assembles on eval box (pinned to origin/main), bot signs on trusted host. Private key never touches the eval box. * feat(polaris): upgrade to Intel TDX hardware attestation via Polaris API Replace Ed25519 software signatures with Polaris TDX hardware attestation for scoring integrity. The scoring step (correctness gate, guard gate, label computation) now runs inside an Intel TDX enclave with DCAP-quoted receipts. New files: - eval/polaris/scoring.py: self-contained scoring script (stdlib only) that runs inside the TDX enclave, replicating evaluate_dual.sh lines 148-212 - eval/polaris/client.py: PolarisClient for the /v1/attest API Modified files: - eval/polaris/receipt.py: add build_polaris_receipt() and ReceiptValidator.verify_tdx() for TDX receipt assembly and verification. verify() auto-detects receipt type (tdx-quote vs Ed25519). - eval/polaris/verify.py: auto-detect TDX receipts, show Intel DCAP verification status, distinguish TDX vs Ed25519 in output. - eval/pr_eval_bot.py: when POLARIS_API_KEY is set, submit scoring to Polaris TDX enclave and build TDX receipts. Ed25519 signing is preserved as fallback when only SPARKINFER_POLARIS_PRIVATE_KEY is set. The Ed25519 path and all 29 existing tests continue to pass unchanged. * docs: add Polaris verifiable eval receipts documentation * dashboard: PR #272 -> eval:none (413.26 tok/s) * fix(eval): 16k/32k guard baseline fallback — bash expansion bug + display Two bugs caused Qwen3.6 16k/32k guard baselines to show wrong values: 1. evaluate_dual.sh: ${SPARKINFER_P_GUARD_16K_BASELINE:-338.55} treats the explicit "0" passed by the bot as a valid non-empty string, so the :- expansion never fires and the hardcoded default is never used. Fix: two-step fallback — try env var, then hardcoded default if still 0. 2. pr_eval_bot.py: the display fallback for ctx_16384_tps used frontier_tps (the 128-context baseline, ~412) instead of a proper 16k default (~339). ctx_32768_tps had no fallback at all. Fix: _GUARD_BASE_FALLBACK dict with per-context defaults matching evaluate_dual.sh. * dashboard: PR #282 -> eval:XL (427.28 tok/s) * fix(eval): trigger Qwen3.6 baseline sweep when any context is missing Previously the 5-context sweep only ran when 128 baseline was 0. Since the bot always passes a non-zero 128 baseline, the sweep never ran and 16k/32k baselines were never measured fresh — falling through to broken ${VAR:-default} expansions and eventually to wrong display values. Now the sweep triggers when ANY of 128/512/4k/16k/32k is 0, so passing only 128/512/4k (with 16k/32k as 0) still gets a fresh measurement on the eval box. * fix(eval): resolve vastai CLI path and update default instance to 44206573 - vast_eval.py: resolve vastai binary via shutil.which / ~/.local/bin/vastai so subprocess.run works from bot/cron contexts where PATH may be limited. - pr_eval_bot.py: update hardcoded VAST_DEFAULT_INSTANCE fallback to 44206573. * fix(eval): measure Qwen3.6 16k/32k baselines in bot pre-sweep The same-box Qwen3.6 baseline sweep now covers all 5 contexts (128, 512, 4k, 16k, 32k) instead of just 3. The measured values are passed as --p-guard-16k/32k-baseline to vast_eval.py, which forwards them to the eval box via SPARKINFER_P_GUARD_16K/32K_BASELINE env vars. This gives evaluate_dual.sh real measured baselines to use (instead of falling through to hardcoded defaults). * dashboard: PR #282 -> eval:XL (427.05 tok/s) * fix(eval): point eval log repo and URL to main sparkinfer repo Previously logs were pushed to gittensor-ai-lab/sparkinfer-log.git which skyrocket2026 lacks permission to push to. Move to the main sparkinfer repo and update the log page URL accordingly. * dashboard: PR #283 -> eval:none (415.04 tok/s) --------- Co-authored-by: Skyrocket <>
3 tasks
Member
|
log tail |
3 tasks
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
One bit-identical launch fusion on the Qwen3.6 gated-Q attention decode path. On the full-attention (gated-Q) layers, the Q-gate
attn *= sigmoid(qgate)(mul_sigmoid_kernel, bf16 store) is immediately followed by the O-projection mmvq path re-readingattnto produce its Q8_1 activation (si_quantize_q8_1_blocks) — two back-to-back launches.mul_sigmoid_q8_kerneldoes both in one launch (one warp per 32-value Q8_1 block), deleting a per-layer graph node. Decode is CUDA-graph / launch-bound, so per-layer launches cost real time.Bit-identical to
main: the fused kernel bf16-roundsx*sigmoid(gate)(__bfloat162float(__float2bfloat16(...))) exactly asmul_sigmoid's bf16 store, then runs the amax/round/scale reduction copied verbatim fromsi_quantize_q8_1_blocks. It mirrors the existinggated_norm_q8_warp_kernel(same gated-op + quant fusion, already validated bit-identical).SPARKINFER_GATE_Q8=0reproducesmainexactly. No-op for Qwen3-30B (noq_has_gate) and for non-mmvqwopaths — the guard cannot regress them. Touches only the Q-gate→quant pair; does not modify any mmvq/GDN/MoE/router/shared-expert kernel.Proof of speedup
sm_120)Decode tok/s (end-to-end,
qwen3_gguf_bench Qwen3.6-35B-A3B-UD-Q4_K_M 256, ctx=0, bs=1). Interleaved A/B on one binary (SPARKINFER_GATE_Q8toggled), 3 runs each:SPARKINFER_GATE_Q8=0, == main)+1.42 tok/s (+0.35%). Small and honestly reported — the fusion applies only to the 10 full-attention layers (the 30 GDN-linear layers have no uncontested fusable pair).
SPARKINFER_GATE_Q8=0reproducesmainexactly.Correctness
mainby construction (bf16-rounded product → identical Q8_1 blocks; reduction order copied verbatim from the standalone quantizer).ctest7/7 passed (incl. both GPU tests).compute-sanitizer --tool memcheck0 errors.