Repository navigation
[job] Qwen3.8-Flash-Next: prompt-lookup + MTP hybrid drafting for agentic decode #130
Description
Status (updated 2026-09-29 11:40Z)
State: step 1 ran, but it couldn't measure K > 2. --num-drafts > 1 is ignored on Flash-Next: the scheduler logs num_drafts=8, yet every arm drafts one token per step.
-
Measured (reiner
643, mainb03dc7363, 8,076-token prompt above the QSA bound,ATLAS_QSA_VERIFY_ACTIVE=1, 384 output tokens, 3 requests per arm):arm step time tokens/step tok/s serial 57–60 ms 1.00 16.7–17.5 MTP K=2 78–82 ms 1.95–1.98 23.9–25.4 MTP "K=3..9" same as K=2 same same So a K=2 verify step costs ~1.38× a serial step. Every arm had 0 QSA declines and 0 errors.
-
Next: a small code change, a verify-cost microbenchmark: one forced K-token verify step on Flash-Next at batch 1 above the QSA bound, for K = 2..9. No existing flag can do this.
--num-draftsis ignored on Flash-Next, and the n-gram lane picks K from match length (graphed only for K = 2/3/4). The decision rule is unchanged: if K=9 costs more than 1.6× K=2, close this job.
Why: decode is 73% of the wall time on the Flash-Next agentic leg: #111's 1007-turn run, reiner job 596z, TTFT 1125 s, decode 3091 s, median TPOT 48 ms. MTP at --num-drafts 1 averages only 1.75 tokens/step (p1 = 0.753 over 1008 turns). Agentic output keeps re-emitting text the context already holds: paths, grep patterns, file lines, tool-call JSON.
Offline replay on the same traces (CPU only; ~/jobqueue/staged/campaign/ngram_replay.py on reiner). It uses Atlas's own lookup_match algorithm (2-gram index, 64 newest candidates, K=8) over prompt + output-so-far. At each output position it counts the accepted draft length, then simulates "prompt-lookup step where it matches, MTP otherwise":
| threshold (accepted ≥ T) | tokens covered by lookup | tokens/step | decode steps |
|---|---|---|---|
| T ≥ 1 | 40.8 % | 2.26 | −22.7 % |
| T ≥ 2 | 31.7 % | 2.23 | −21.6 % |
| T ≥ 4 | 22.7 % | 2.11 | −17.3 % |
The median turn has lookup-draftable spans at 16% of positions (p90: 38%).
The same replay on the 27B DFlash2 traces (146 turns, DFlash2 at 5.5 tokens/step) gives −8% at best, and likely about 0 in practice: the spans lookup covers are the ones DFlash2 already drafts well. So this job is scoped to MTP models only.
Why it's a job and not a flag flip: --ngram-speculative already exists, but it can't fire on this workload:
- The lane is confined to
seq_len + 4 <= verify_context_limit(), which is 2051 on Flash-Next (spec_step/ngram.rs). Agentic turns are 5–17k tokens. - Its verify is CUDA-graphed only for K = 2/3/4, while the replay's wins come from 5–9-token spans.
- It replaces MTP rather than falling back to it.
Plan, each step gated by the one before:
- Measure first (about 1 hour of GPU, no code): verify step time on Flash-Next at batch 1 for K = 2..9 above the QSA bound. At batch 1, extra tokens pull in extra MoE experts, and that cost converts the replay's step count into wall time. If a K=9 verify costs more than about 1.6× a K=2 step, the gain is gone: close this job and record the finding here.
- Hybrid proposer: prompt lookup when accepted-length confidence ≥ T, otherwise MTP. Verify through the path MTP already uses past the QSA bound.
- The gate is the ST-995 + agentic perf leg, against
596z(0.5053). Merge only if the agentic score improves by ≥ 8% with output unchanged (greedy hashes identical, which holds because verify is exact). Under the neutral-opt-in rule, a smaller gain doesn't merge as a flag.
Activity
maplepaladin73 commentedon Sep 29, 2026
Atlas-Inf measurement, 2026-09-29. Tree: Atlas-Inf/atlas @ b03dc73, rebuilt on spark(1x GB10) with ATLAS_TARGET_HW=gb10, ATLAS_TARGET_MODEL=qwen3.8-flash-next, ATLAS_TARGET_QUANT=nvfp4. Weights: nvidia/Qwen3.8-Flash-Next-NVFP4 @ fc694b54. Serve: batch 1, max-seq-len 8192, MTP forced.
K=2 (--num-drafts 1): request completed. 120 tokens, 16.7 tok/s, TTFT 3674 ms, tok/step 1.25, MTP share 0.03 (serial 0.97). The MTP verify-timing summary line never printed, so there is no clean verify-step ms. The prompt also chunked (first chunk 4097 of 5622), so this is not the requested single-step measurement just above the 2051 QSA bound.
K=9 (--num-drafts 8): the server accepts K=9 at boot (
num_drafts: 8 (K=9)), then the first verify returns HTTP 500:decode_verify_graphed_k4: qwen3_ssm mHC batched decode: no batched MoE arm for K=4. K=2/3 use forward_k2/k3 and K=4..8 the batched GEMV; this model's 512-expert MoE has no per-row staging under the highway yet.
It dies at K=4, before any K=9 verify step exists. The close rule (K=9 verify more than about 1.6x a K=2 step) cannot be evaluated. Do not close #130. The missing piece is a K=4..9 MoE verify arm on Atlas-Inf/atlas, not another timing run.
AzeezIsh commentedon Oct 2, 2026
Why
--num-drafts> 1 is ignored on Flash-Next:crates/spark-server/src/scheduler/mtp_step.rs:29-46clampsnum_draftstomodel.verify_max_drafts(). The mHC highway's batched MoE verify only has K=2/3 arms; asking past them used to kill requests after one token, so the clamp is total and silent. The serve log prints the clamp at boot (serve_load.rs:880).AzeezIsh commentedon Oct 2, 2026
Correction to my comment above. The one-draft clamp is not only "missing MoE arms". It is a measured optimum, documented at
qwen3_ssm/trait_layer.rs:189(GB10, 256-token completions):So lifting the clamp (my earlier ordering: recursive drafting → K = 4–8 arms) is not the lever. The lever is making the extra verify row cheaper. The +34 ms for one more row is far above what one more row of expert weights costs in bandwidth (~10 experts × 48 layers ≈ 1.3 GB at NVFP4, ~5–7 ms).
For context: Gufo's MTP on mixed text is ~1.25× its own serial (32.2 vs 25.9); Atlas GB10 K=2 is ~1.22×. MTP efficiency is at parity. The Flash-Next decode gap is the base serial step (Gufo 25.9 vs Atlas 17.7 GB10 / 9.5 Windows). That is #143's launch/kernel-cost question, which the Windows decode trace measures.
AzeezIsh commentedon Oct 2, 2026
A concrete source for the known MTP output drift on GB10 (from today's #153 work):
dense_gemv_bf16_batch2(qwen3_ssm/trait_decode_batched.rs, BF16 GDN is the qwen4_exp default). Plain decode usesdense_gemv_bf16.kernels/gb10/common/), the batch2 kernel uses 64 lanes per row plus a shared-memory reduction, and the single-row kernel uses 32 lanes. They sum in different orders, so the in-projection rounds differently on every verify step, on 36 layers. The batch2 header's "bit-identical" claim is false there.kernels/strix-hip/common/), both kernels use the same 32-lane walk and are bit-identical.gated_delta_rule_wy2) is a different algorithm from the decode one.None of this changes the rule that MTP is a non-exact accelerator. It does matter for any "MTP is output-preserving" claim and for the BFCL reading.
AzeezIsh commentedon Oct 2, 2026
Retraction of my earlier comment here. I claimed the GB10
dense_gemv_bf16_batch2sums in a different order fromdense_gemv_bf16(64 vs 32 lanes per row) and named it as a concrete MTP-drift source. That's wrong. The GB10dense_gemv_bf16also runs 64 lanes per row (BLOCK_SIZE 256 / N_PER_BLOCK 4; the// 32comment on that line is stale, and I trusted the comment over the arithmetic). A structural diff of the two kernels shows the same walk, products, shuffle tree and smem pair-sum; batch2 only adds a second accumulator. So the pair is bit-identical on GB10 as well as HIP, and it isn't a drift source. The GB10 drift must come from elsewhere; the verify GDN recurrence (gated_delta_rule_wy2vs the decode kernel) remains the main suspect. Sorry for the noise.AzeezIsh commentedon Oct 2, 2026
Re-baseline of Flash-Next MTP output drift on GB10, main
4405bce35(reiner job 665; util 0.90 / seq 8192, thinking off, temp 0, 8 prompts × 200 tokens, fresh server per arm):dense_gemv_bf16_batch2(bit-identical to the single-row kernel on GB10; my earlier claim is retracted above) and the BF16 LM head (lm_head: one-pass BF16 batch2 LM head on every target (stacked on #153; GB10 Flash-Next MTP verify) #159: byte-identical).gated_delta_rule_wy2vs decode kernel, conv1d update, MoE batch2 vs single, attention-per-token, hc collapse T=2 vs T=1), force the verify step through the single-token kernel and re-run this 8-prompt battery. Wherever drift drops to 0/8 is the source. The harness isreiner:~/jobqueue/done/665-gb10-mtp-drift/.This matters for any claim that MTP is lossless, and it shares a root theme with #158 (the pass layout changes numerics).
AzeezIsh commentedon Oct 2, 2026
The drift split (reiner job 666, same binary / settings / prompts as 665):
--exact-verify(#435, sequential-exact GDN verify chain) is already in-tree, so I measured it rather than bisecting by hand.--exact-verifygdn_flags.rscomment describes (#459: verify takes batch GEMVs, decode single-row GEMVs, ±1 ULP on ~5e-5 of lanes).dense_gemv_bf16_batch2does on both platforms; lm_head: one-pass BF16 batch2 LM head for K=2 verify on HIP (Flash-Next, −5.9 ms/verify step) #153/lm_head: one-pass BF16 batch2 LM head on every target (stacked on #153; GB10 Flash-Next MTP verify) #159), that's free. Thew4a16_gemv_batch*family is the remaining work. An audit of which batch kernels are bit-identical to their single-row siblings would scope it.AzeezIsh commentedon Oct 2, 2026
Scoping the remaining 4/8 (static audit of main
4405bce35):w4a16_gemv_batch2/3route through the bit-exactw4a16_gemv_batchm_impltemplate.w4a16_gemv_qg_batch2/3anddual_batch2/3are documented as "fifth family, now fixed" againstw4a16_gemv_qg, withw4a16_qg_batch_bitparity_microtest.rsas the gate. MoE*_shared_batch2_tuses the same per-output loop as*_shared_t.dense_gemv_bf16_batch2is identical (above). So thegdn_flags.rscomment's #459 attribution is probably stale.dense_gemm_ba_gates_prefillandgated_rms_norm_prefill_sigmoid(12 calls each, on the GDN layers), plusrms_norm_stridedand the hc collapse at T=2. Whether--exact-verifycovers the gate/norm arms or only the recurrence + conv needs checking inqwen3_ssm/trait_decode_batched*.--exact-verify. The arm that takes 4/8 → 8/8 is the source. Not starting it now; it's a dedicated bisect.