[job] Qwen3.8-Flash-Next: long-context cold prefill to SGLang parity (stage 1 = #68)

@AzeezIsh

Status (updated 2026-09-28)

State: stage 1 (#68) merged on 2026-09-26: 31.5k cold TTFT went from 92.2 s to 24.3 s. The remaining levers are being priced.

Latest (reiner job 625, shared-expert A/B): moving the shared expert to BF16 MMA trades ppl against TTFT (+7.7% to +12.4% at 31.5k). No arm wins outright. spark-model: bf16-activation transposed MoE prefill kernels, opt-in (#74) #97's bf16a clones are closed per the neutral rule.
Next, in order of price:
1. MoE FP4 MMA over the resident decode layout (dequant is 1.96 s at 29.7k);
2. fuse hc_post into hc_pre_stage (~0.4 s, bit-identical);
3. a selection-stable QSA tensor-core formulation (−4.4 s, but loses 1/12 needles today).
Done bar: ≤ 16 s at 31.5k with ppl, needle and kl_drift gates green.

Kind

Feature for a model we already support

Model

nvidia/Qwen3.8-Flash-Next-NVFP4

What's the job?

Close the long-context prefill gap to SGLang (~2,250 tok/s cold at 27K ≈ 12 s at 27k). Stage 1 is draft PR #68: 31.5k cold TTFT 92.2 s → 24.3 s on main66 vs the campaign (GB10, job 239), greedy-identical. What is left, each priced:

MoE FP4 MMA over the resident [N,K/2] decode layout (no dequant, no transpose) — dequant alone is 1.96 s at 29.7k (probe 41d5deba).
fuse hc_post(L) into hc_pre_stage(L+1) — ~0.4 s, bit-identical by construction (needs a bitwise test).
QSA tensor-core kernels (ATLAS_QSA_ATTN_TC2/SCORE_TC, −4.4 s): perplexity neutral but lose 1/12 needles and top-1 91.7 % (job 243) — needs a selection-stable formulation before default-on.
Done = 31.5k cold TTFT ≤ 16 s on GB10 with ppl/needles/kl_drift gates green.

Impact radius

Shared code, other models could change

What else could it touch?

moe_w4a16_grouped_gemm.cu is a symlink shared by 7 targets; mHC kernels are Flash-Next-only; QSA kernels Flash-Next-only.

Who's waiting or willing to test?

@AzeezIsh (GB10)

Status (2026-09-25): stage 1 = #68 measured 92.2 → 24.3 s at 31.5k (3.8x) on GB10. Open questions being run on reiner: job 258 (QSA_ATTN_TC2 alone vs QSA_SCORE_TC alone — TC2 alone could take most of the −4.4 s), job 259 (build bisect of the +0.036 % dense-ppl drift / long-generation divergence to the mHC series), BFCL on #68 after job 254's memory answer. Details and the correction on long generations: #68 comments; living status: #76.

Attribution of the transposed-MoE +0.74 % dense perplexity (job 269): the _t prefill kernels are W8A8 FP8, not W4A16.

This comes from reading the source on main c2808e270, kernels/gb10/qwen3.8-flash-next/.../moe_w4a16_grouped_gemm.cu:

	transposed (`moe_w4a16_fused_gate_up_t_k64`, `…ptrtable_t_k64`; the default since #68)	untransposed (`ATLAS_MOE_TRANSPOSE=0`, `…grouped_gemm_ptrtable`)
activations into the MMA	bf16 → e4m3 with no scale: `cvt.rn.satfinite.e4m3x2` (`moe_bf16x4_to_e4m3x4`, ~:204-219, used at ~:656-660). That gives ~6 % relative rounding, subnormals below 2⁻⁶, and saturation at ±448	bf16
weights	e2m1 × e4m3 block × scale2, dequantized in f32 and then re-quantized to e4m3 (~:605-632)	dequantized to bf16
MMA	`mma.m16n8k32.e4m3.e4m3.f32` (W8A8)	`mma.m16n8k16.bf16.bf16.f32` (W4A16)
accumulation, SiLU intermediate, unpermute/combine	f32 / bf16 / fp32 weighted sum, shared by both paths	same

Ranked causes:

Unscaled e4m3 activations, which likely dominate.
The second rounding when weights are re-quantized to e4m3.

Decode isn't affected. Flash-Next NVFP4 on GB10 selects the UNIFIED tier, so decode also runs _t kernels, but those are bf16-activation warp GEMVs with f32 dequant (moe_shared_expert_fused_t.cu).

This is the same class of issue as #95 (27B dense-FFN W4A4 prefill → #96). For Flash-Next, reverting to the untransposed kernels would give up most of the #68 TTFT win, so the fix has to be in the kernel. Options, in order of expected value:

(a) Per-token-row e4m3 activation scale: compute the row absmax in the gather, scale into the e4m3 range, and apply the scale in the f32 epilogue. This removes underflow and clipping and keeps the FP8 MMA speed.
(b) Per-16 block-scaled FP8 or MXFP8 activations, if (a) isn't enough.
(c) A bf16-A MMA on the _t layout as a precision reference arm, which also bisects activations vs weights.

Each option is measured with the job-269 ppl split (first 2051 tokens as the control), 31.5k cold TTFT, and the ST-995 leg.

Correction to the attribution above (job 406). A bf16-activation clone of the transposed MoE prefill kernels (PR #97) is byte-identical to the untransposed kernels in the GPU test. Yet it moves dense perplexity only −0.17 % vs the e4m3 default (7.7112 → 7.6981), while ATLAS_MOE_TRANSPOSE=0 moves it −0.73 % (→ 7.6548, reproducing job 269). So the unscaled-e4m3 MoE arithmetic accounts for about 0.2 % of the transposed-path gap, and roughly 0.5 % comes from something else that ATLAS_MOE_TRANSPOSE=0 changes: the prefill kernel set outside the grouped GEMMs, the shared expert, or the UNIFIED-tier layout. After the QSA bound the sign flips: untransposed is +0.18 % worse than default, and bf16a −0.21 % better. Costs at 31.5k cold TTFT: e4m3 20.86 s, bf16a 21.77 s, untransposed 38.81 s. Decision: keep e4m3 as the default. The next step for the remaining ~0.5 % is to diff the prefill kernel dispatch between ATLAS_MOE_TRANSPOSE=0 and the default beyond the grouped GEMMs.

docs(status): #86/#96/#99 merged; recipes PR; open findings (#102, #74,

Triage (2026-09-26). Stage 1 (#68) landed: 31.5k cold TTFT went 92.2 → 24.3 s. The done bar is ≤ 16 s; SGLang does ~12 s at 27k.

Must know before quoting more perplexity numbers: #91 found that Atlas has been applying an extra final RMSNorm on qwen4_exp, scaling logits by 2/rms(x). Perplexity was 4.28 in Atlas against 2.29 in vLLM on the same NVFP4 weights.

Every ppl.py / kl_drift figure in this thread (job 243, 269's +0.74 %, 406) was measured on those logits.
Both arms shared the bug, so relative verdicts should hold. Absolute values and magnitudes need a re-baseline once the fix lands.

Scale back:

Close spark-model: bf16-activation transposed MoE prefill kernels, opt-in (#74) #97. bf16a recovers only ~0.2 % perplexity for +4.4 % TTFT, and the author recommends against merging. The remaining ~0.5 % of the transposed-path gap is an investigation, not another opt-in arm.

What's left, priced:

MoE FP4 over the resident layout (dequant alone is 1.96 s at 29.7k). This is really [job] GB10: raise the Atlas tensor-core GEMM ceiling (29-39 vs cuBLAS 75-206 TFLOP/s) #71's GEMM-template work; suggest tracking the kernel there and keeping this issue for the TTFT number.
hc_post(L) → hc_pre_stage(L+1) fusion: ~0.4 s, bit-identical by construction.
The TC2 / SCORE_TC default decision: job 387's paired McNemar vs 280. Please post it.

A lead on the remaining ~0.5 %. The transpose pass also builds transposed tables for the Flash-Next shared expert (shared_*_t). Once they exist, run_shared_expert_prefill takes w4a16_gemm_t, which quantizes activations to e4m3 and runs FP8 MMA, instead of the bf16-MMA w4a16_gemm. The NVFP4 qwen4_exp loader never installs the bf16 or fp8 shared-expert arms that take precedence. The bf16a routed clones (#97) don't touch this path, which fits them closing only 0.17 % of the gap. A smaller contributor with the same mechanism is the short-prefill tail through forward_batched (n ≤ 64), which uses the unified-T per-token kernels. The weight bytes themselves are identical: the transpose is a pure byte shuffle, with no requantization.

A/B queued as reiner job 474, on test/moe-tshared-bf16a = main + an ATLAS_MOE_T_SHARED=0 diagnostic lever (diag/moe-t-shared-lever @ 38d37add1) + #97. Four arms on one binary: default, shared untransposed, shared untransposed + bf16a routed, and ATLAS_MOE_TRANSPOSE=0. Each gets perplexity plus 31.5k TTFT. If the third arm lands on the untransposed perplexity, e4m3 in routed plus shared explains the whole gap, and a bf16-activation shared expert, which is small, becomes the cheap fix.

Localising the remaining ~0.56 % of the transposed-MoE perplexity gap (Qwen3.8-Max consult plus what's already in-tree).

Only prefill ops that change logits can move teacher-forced ppl:

the router/gating GEMM and top-k;
the shared expert;
the routed experts (already ruled out down to 0.17 % by spark-model: bf16-activation transposed MoE prefill kernels, opt-in (#74) #97's byte-identical bf16a clone);
the combine/unpermute reduction;
chunk boundaries, only if the flag changes chunking.

Decode-only tier differences can't move it.

One-run localisation instead of N A/B jobs:

ATLAS_OP_DUMP (spark-model: dump q/k post-norm/rope ops on chunked prefill too (ATLAS_OP_DUMP) #99, now covers chunked prefill) on the transposed arm, per MoE layer:
- router logits, top-k ids/weights;
- the token permutation;
- shared-expert in/out;
- routed outputs before combine;
- the combine result and the post-block hidden state.
Replay the untransposed path offline on the dumped inputs. The first tensor that diverges names the cause:
- top-k differs → router;
- expert outputs match but the MoE output differs → combine order/precision;
- only boundary states differ → chunking.

Two notes:

The queued moe-tshared-ab job tests the shared expert alone; keep it as the first bisection step.
All perplexity numbers in this thread predate qwen4_exp: no final RMSNorm after the hyper-connection mixer (split from #91) #111 (the qwen4_exp final-norm fix: ppl 7.711 → 4.416 dense on the same windows, job 527). Relative verdicts hold, but re-express the 0.73 % / 0.17 % / 0.56 % split on the fixed baseline before sizing any kernel work on it.

Shared-expert A/B for the remaining transposed-MoE ppl gap (reiner job 625).

Setup: Flash-Next NVFP4, util 0.88, one binary (test/moe-tshared-bf16a @ a4d34527a = main cace4bb + ATLAS_MOE_T_SHARED=0 lever + #97 bf16a clones). ppl is 8 paired 12k windows, split at the QSA bound. TTFT is the median of 3 cold 31.5k prefills.

arm	dense ppl (below bound)	QSA ppl (above bound)	31.5k TTFT
E, default (e4m3 routed + shared)	7.71124	4.38413	20.75 s
S, `ATLAS_MOE_T_SHARED=0` (shared on bf16 MMA)	7.70172 (−0.12 %)	4.40692 (+0.52 %)	22.35 s (+7.7 %)
SB, S + `ATLAS_MOE_T_BF16A=1`	7.67161 (−0.51 %)	4.37368 (−0.24 %)	23.33 s (+12.4 %)
U, `ATLAS_MOE_TRANSPOSE=0`	7.65477 (−0.73 %)	4.39191 (+0.18 %)	38.71 s (+87 %)

Greedy probe hashes are identical across all four arms.
SB closes 0.51 of the 0.73 % dense gap to U, about 70 %. The shared expert alone (S) accounts for 0.12 %. SB vs U: 0.22 % remains on dense, while SB is 0.42 % better on the QSA-selected part.
The QSA-part deltas mix signs (4 worse / 4 better windows for SB and U), so I read only the dense column as signal.
Cost: SB adds 12.4 % to 31.5k TTFT. U adds 87 %, which is why the transposed layout is the default.

My recommendation: keep the default. SB is a documented opt-in for users who want the extra ~0.5 % dense ppl at +12 % prefill. Deciding whether that trade is worth a default change is a maintainer call. It would also need an ST-995 run first.

GB10 Flash-Next prefill attribution with nsys (main). Reiner job 694, 2026-10-04.

Build/recipe: main 4405bce35 (wt-main440), Flash-Next NVFP4, job 659's recipe (util 0.88, seq 34000, prefill 16384, bf16 KV, no prefix cache, thinking off, no MTP).
Profile: nsys --trace=cuda, two cold requests with max_tokens 1.
TTFT: 5.18 s at 8,329 tokens (1,608 tok/s) and 15.61 s at 24,082 tokens (1,543 tok/s). SGLang reference: ~2,250 tok/s cold at 27k, so the gap at ~24k is ~1.46×.
GPU time: 21.15 s of kernels over both requests; 2.81 s of memory ops, of which HtoD is 1.79 s.

share	GPU ms	launches	kernel
13.8%	2,922	180	`qsa_prefill_attn_tc2`
12.4%	2,628	144	`moe_w4a16_fused_gate_up_t_k64`
7.5%	1,595	144	`moe_w4a16_grouped_gemm_ptrtable_t_k64`
6.5%	1,367	1,575	`dense_gemm_bf16_pipelined`
6.4%	1,351	288	`hc_post`
4.2%	895	108	`gated_delta_rule_chunk_fwd_o`
4.2%	894	1,746	`hc_pre_stage_bf16`
3.7%	782	288	`moe_transpose_u8_batched`
3.6%	753	72	`nvjet_sm121_tst_mma_128x208x64…` (cuBLASLt)
3.2%	682	180	`qsa_score_rows_exact`
3.2%	671	72	`fp8_fp8_gemm_t_m128`
3.1%	655	1,746	`hc_pre_mix`
3.0%	630	108	`gated_delta_rule_chunk_delta_h_pipe`
2.7%	563	144	`w4a16_gemm` (router)

What stands out:

moe_transpose_u8_batched runs on every prefill chunk: 288 launches = 3 chunks × 48 layers × gate_up/down, 782 ms (3.7%). Transposing MoE weights on every chunk looks like work that belongs at load time. Worth checking whether the transposed copy is dropped to save memory and rebuilt each chunk.
The routed MoE k64 pair is 19.9% (hottest-expert grid). On gfx1151 the F16-operand twins of these kernels just ran 4.3–4.7× faster per launch than the k64 BF16 ones (strix-hip: F16-operand prefill GEMM twins (exact NVFP4/FP8 dequant) for dense m128, routed MoE, w8a16 #155 + the engage commit). NVIDIA has its own lever list (MoE FP4 MMA).
QSA tc2 is still the single largest kernel at 13.8%.
The mHC kernels together are ~15%: hc_post + hc_pre_stage_bf16 + hc_pre_mix.
For comparison, gfx1151 after the HIP prefill train (winbox P4, 9,302 tokens, SYNC trace): 14.6 s of GPU. Its order is QSA 20%, MoE 18.5%, GDN 18%, BF16 GEMMs 16%.

Artifacts: reiner:~/jobqueue/done/694-gb10-fnext-nsys/ (.nsys-rep + .sqlite, per-request split).

@devin

Prefix-cache reuse at concurrency is the biggest agentic TTFT lever we've measured on GB10. From the MLPerf C=36 point (reiner job 688, Qwen3.6-35B-A3B NVFP4, --enable-prefix-caching true --ssm-cache-slots 128 --ssm-checkpoint-interval 128, 2,792 requests, mean prompt 35.5k tokens):

outcome	requests
KV prefix hit but no SSM snapshot → full re-prefill	1,315 (47%)
Marconi intermediate hit (still recomputes ~8.5k tokens on average)	1,419
exact/leaf hit	50

So ~21k prompt tokens recomputed per turn, when each agentic turn only appends a few hundred to ~2k. The snapshot pool (136 slots, auto-sized from 262144/2048) is shared by 36 conversations and evicts the anchor each one needs next turn. Median TTFT 24 s, TPOT p50 1.03 s — prefill-bound.

Plan (Claude): after #163 settles (it changes the checkpoint grid), measure hit rate vs snapshot pool size / checkpoint interval / eviction policy (keep each live session's latest anchor) on a fixed agentic replay. @devin flagging so it doesn't collide with the #74 nsys work (reiner 694).

ATLAS_QSA_ATTN_L8 gate result on strix-halo (Windows HIP, Flash-Next): hard crash, not a numeric divergence.

Ran the perplexity gate the env comment asks for (new scripts/ppl.py — echo+logprobs loglikelihood over the legacy /v1/completions call, stdlib-only; on dev/qsa-ppl-gate, verified working on strix-27B: ppl 5.82 over a docs window).

Same build both arms (atlas-p5 @ 35b4db9bc, winbox, seq 32768, 4 × ~18k-token windows of repo docs, temp 0):

arm	result
default QSA (REF)	all 4 windows scored, ppl = 4.5917 over 73,591 tokens, 0 faults, `lt-live=1`
`ATLAS_QSA_ATTN_L8=1`	serve died mid first window — client timeout, then 503s; teardown swept 229,520 allocations and `cuMemFree_v2` hit `status 719` (context destroyed)

Dispatch wiring confirmed: ATLAS_QSA_ATTN_L8 selects qsa_prefill_attn_l8 when qsa_prefill_attn_l8_ok(nq,nkv,hd) (crates/spark-model/src/layers/qsa_select_stages.rs:312-341), and kernels/strix-hip/qwen3.8-flash-next/nvfp4/qsa_indexer.cu is a symlink to the GB10 kernel — so the GB10-shaped _l8 launches on gfx1151 and faults there. Whether it also crashes on GB10 is untested here.

Verdict for this platform: do not enable _l8 on strix-halo — it's a hard fault at long context, not the ~0.9×-timing/3e-5-delta question the microbench framed. The gate does exist now for whichever host tries it next.

Out dir / evidence: winbox atlas-p5\out\serve-fnext-fnext-20261006-100144.log (L8ON arm) vs -095133.log (REF), plan win_plan_1006p9.ps1 / STATUS_plan1006p9.txt.

Cold prefill on FP4-Local, two GB10s at TP=EP=2: numbers and an nsys breakdown.

Setup: SparkQwen FP4-Local release (Enntity/atlas 9f64bf6b), nvidia/Qwen3.8-Flash-Next-FP4-Local, TP=EP=2 over our GB10 pair, prefill chunk 16,384. Method as in job 694: one request at a time, max_tokens 1, thinking off, no tools. A random nonce opens each prompt so the prefix cache cannot hit; the log confirms no hit.

TTFT, two passes each (repeatable within 0.3%):

prompt tokens	TTFT	tok/s
28,397	7.83 s	3,630
34,528	9.07 s	3,806
78,612	21.4 s	3,670
125,911	37.9 s	3,320

This is not like-for-like with job 694 (newer engine, two boxes). Per box it works out to about 1,800 tok/s.

Per 16K chunk at 126K: about 4.0 s, plus about 0.21 s more for each chunk of context already processed.

nsys on rank 0 (--trace=cuda), capturing a cold 34.5K and a cold 78.6K request. The GPU is busy 90% and 97% of the request time, so this is kernel-bound.

GPU time, rank 0	34.5K	78.6K
MoE (`moe_q38wn_gate_up_silu`, `moe_q38wn_down`, unpermute)	2.28 s (26%)	4.96 s (24%)
QSA (`qsa_prefill_attn_tc2r`, `qsa_score_rows_exact_v4`, top-k)	1.69 s (19%)	5.16 s (25%)
mHC (`hc_post_stage_bf16`, `hc_up_mix_*`)	1.48 s (17%)	3.22 s (16%)
GDN (`gated_delta_rule_*`)	1.23 s (14%)	2.79 s (14%)
dense GEMM (CUTLASS FP8 block-scaled + BF16)	1.21 s (14%)	2.68 s (13%)
norm / act / RDMA one-shot / misc	~0.8 s	~1.9 s

What changes on FP4-Local versus the status list above:

The MoE dequant lever is already gone. The only dequant kernel left is moe_q38_router_dequant (9 ms). The remaining MoE time is the grouped GEMM itself, which is where your FP4-local components come in.
hc_post_stage_bf16 alone is 10-11% of GPU time (854 and 1,807 launches). The "fuse hc_post into hc_pre_stage" item looks larger here than the ~0.4 s estimate.
QSA prefill attention is about a fifth to a quarter, growing slowly with context.

Unless you'd rather we didn't, we'll take the hc_post fusion next, since it doesn't overlap the MoE work. We'll hold it to a bitwise test against the current output before proposing anything.

Navigation Menu

Repository navigation

[job] Qwen3.8-Flash-Next: long-context cold prefill to SGLang parity (stage 1 = #68) #74

Status (updated 2026-09-28)

Kind

Model

What's the job?

Impact radius

What else could it touch?

Who's waiting or willing to test?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[job] Qwen3.8-Flash-Next: long-context cold prefill to SGLang parity (stage 1 = #68) #74

Description

Status (updated 2026-09-28)

Kind

Model

What's the job?

Impact radius

What else could it touch?

Who's waiting or willing to test?

Activity

AzeezIsh commented on Sep 25, 2026

AzeezIsh commented on Sep 26, 2026

AzeezIsh commented on Sep 26, 2026

AzeezIsh commented on Sep 26, 2026

AzeezIsh commented on Sep 26, 2026

AzeezIsh commented on Sep 27, 2026

AzeezIsh commented on Sep 28, 2026

AzeezIsh commented on Oct 4, 2026

AzeezIsh commented on Oct 5, 2026

AzeezIsh commented on Oct 6, 2026

maplepaladin73 commented on Oct 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions