Repository navigation
qwen4_exp TP=2 profile: routed-MoE prefill runs NVFP4 experts at E4M3 (canonical NLL 1.0843 vs 1.0511 with exact weights) #214
Description
What
On the Qwen3.8-Flash-Next TP=2 profile (rc16, from #174), the routed-MoE prefill (ATLAS_QWEN4EXP_PREFILL_MOE=1, the q38 chain in moe_prefill_q38.cu / forward_prefill_q38.rs) runs the dequantized NVFP4 expert weights rounded to E4M3, and the activations cast to E4M3 (moe_q38_a_to_e4m3). Decode runs the experts at W4A16 (exact weights), so prefill and decode use different numerics for the same experts. #198's dtype table (doc §2) covered only decode and called the routed experts "native". It missed this.
Enntity measured the weight rounding at 3.7–4.7% error against the checkpoint (Enntity/atlas 63221c58e). They replaced the chain with W4A4 at the checkpoint's own input scales, which is NVIDIA's reference prefill for this card (b12x A4).
Measured (two GB10s, TP=EP=2, #198 head 0164cce9c, same binary, only the switch differs)
Arms:
- E: the shipped profile (rc16-resident).
- T: E plus
ATLAS_QWEN4EXP_PREFILL_MOE_BF16=1, which already exists: the same checkpoint planes, exact NVFP4 weights × BF16 activations on the TC decode numerics, no extra memory.
| E (shipped) | T (exact weights) | |
|---|---|---|
| canonical NLL, SparkQwen fidelity review's 27,034-ID request, positions 1..27,033 | 1.0843027707 | 1.0510869643 |
| ... tokens 1–4K | 1.7931 / 0.5585 | 1.7916 / 0.5569 |
| ... tokens 4K–8K / 8K–16K / 16K–27K | 0.6648 / 0.9668 / 1.4196 | 0.5904 / 0.9356 / 1.3885 |
nll_probe (32 × ~1.5K-token chunks) |
0.46113 | 0.46074 (lower in 13/32) |
| cold TTFT 2K / 8K / 16K / 28K (n=3, same session) | 0.93 / 2.68 / 5.76 / 9.58 s | 0.94 / 2.83 / 5.95 / ~10.0 s |
- Each arm's canonical NLL was run twice: all 27,033 positions were bit-identical.
- E's value equals the one @maplepaladin73 measured for qwen4_exp: serve FP4-Local at TP=EP=2 with the n-gram table resident, native MXFP8 sharding, BF16 routing #198 on their own pair (1.0843028).
- The gain is long-context only: short chunks are unchanged.
- T's greedy text differs from E on 8/8 prompts, so the switch engaged.
- Receipts and fingerprints (binary sha, profile hash, both ranks' env from
/proc) are on our pair under~/code/night1027/out/j01/.
SparkQwen 0.3.1 reports 0.995 on the same request on its own pair, with W4A4 prefill among other numerics differences. So this accounts for about 0.033 of the 0.089 gap, and the rest is somewhere else.
In progress
- A one-lever screen of the other prefill levers (
PREFILL_HC,_GDN,_QSA_SCORE,_SP,_ROWINV,_BF16_PROJ), measuring canonical NLL to find the rest of the gap. - Enntity's W4A4 routed prefill (
63221c58e+baa5d56a5, AGPL, credited), ported as an opt-in (ATLAS_QWEN4EXP_PREFILL_MOE_W4A4=1) on its own branch. - Once both are measured: pick the TP=2 default (T or W4A4) and gate it (exactness, cached == uncached, tools, BFCL ST-995).
#198 itself doesn't touch these files. It inherits this from the profile.
Activity
AzeezIsh commentedon Oct 27, 2026
One-lever screen of the other prefill switches (same binary
0164cce9c, canonical NLL, each override confirmed in both ranks'/proc/<pid>/environ; profile switches overridden through a per-arm copy of the profile, because the profile re-exports its own values):PREFILL_HC=0/PREFILL_GDN=0/PREFILL_QSA_SCORE=0/PREFILL_SP=0PREFILL_ROWINV=0PREFILL_BF16_PROJ=1(decode-precision prefill; impliesMOE_BF16)MOE_BF16alone)PREFILL_MOE_BF16=1(earlier)So the mHC, GDN, QSA-scorer and sequence-parallel prefill kernels are exact. Row-invariant prefill (on in the profile) helps. Among the existing switches,
PREFILL_MOE_BF16alone is the best.The profile matches SparkQwen 0.3.x's 8x32k profile on every prefill switch (they differ only in drafter / scheduling switches). So the rest of the gap to their 0.995 is code we don't have. Next hypothesis: the checkpoint is calibrated for activation quantization (W4A4 experts at its input scales, MXFP8 W8A8 projections). That would explain why their prefill, which quantizes activations more, scores lower NLL. Testing it next with the parked W8A8 projection branch and the W4A4 port.
AzeezIsh commentedon Oct 27, 2026
Correction: the single-document canonical NLL can't carry the conclusion I drew above. More arms (same pair; each value bit-reproducible):
ATLAS_QWEN4EXP_PREFILL_W8A8=1PREFILL_MOE_BF16=1PREFILL_MOE_BF16=1alone (earlier)If these were independent, systematic effects, the combined arm would land near 1.059. It lands at 1.105, worse than either change alone. Together with
ROWINV=0(+0.020) andBF16_PROJ=1(+0.018), it looks like one 27K-token document moves by ±0.02–0.05 under any change to prefill numerics. A plausible mechanism is QSA's discrete block selection flipping on small perturbations: per-position mean |Δ| is 0.13–0.19 nats, and the max is about 11.So "
MOE_BF16accounts for ~0.033 of the gap" was an overclaim. The −0.033 may be inside that spread. What still stands: the E4M3 routed prefill is a real departure from the checkpoint's weights, and the exact-arm values above are what they are. Next: a long-context NLL over many documents, with per-document paired comparisons, so a systematic effect can be told from noise. I'll post it here.AzeezIsh commentedon Oct 27, 2026
W4A4 routed prefill (Enntity
63221c58e+baa5d56a5, ported as opt-inATLAS_QWEN4EXP_PREFILL_MOE_W4A4=1), first gates on the pair. The branch is local for now (perf/fp4l-w4a4-prefill).nll_probe0.46525 vs 0.46113 (W4A4 lower on only 9/32 chunks). Together with W8A8 (also worse), that rules out my "checkpoint is calibrated for activation quantization" idea.ssm-state-poisoning-gategoes from 12/12 byte-identical (shipped) to 0/12 (12 jittered). Every replay round differs from the cold baseline round in the same way, so some restored/replayed prefill path doesn't produce the cold path's bytes under W4A4. Localizing it next.Status: W4A4 is a TTFT opt-in at most, not a default, and only after that replay defect is fixed. The long-context multi-document NLL (shipped vs
MOE_BF16vs W4A4 vs W8A8) is running now.AzeezIsh commentedon Oct 27, 2026
Long-context NLL over 16 documents (8 code, 8 prose, 18–30K tokens each, 362,388 scored tokens; teacher-forced via
/v1/completionsecho; same pair, one boot per arm; the shipped arm run twice was bit-identical, 16/16 ties). Per-document paired wins with a two-sided sign test, because the means turned out to be dominated by a few large shifts past 16K tokens:PREFILL_MOE_BF16=1(exact expert weights)PREFILL_ROWINV=0MOE_BF16Reading, worded to match the evidence:
nll_probeon short chunks).Decisions:
MOE_BF16costs +1–6% cold TTFT.MOE_BF16stays available as a fidelity opt-in.AzeezIsh commentedon Oct 27, 2026
The cache-replay defect is not W4A4-specific.
ssm-state-poisoning-gate(replays a 4-turn script 12×; demands byte-identical transcripts vs the cold round 0), same pair, one boot per arm:ATLAS_QWEN4EXP_PREFILL_MOE_BF16=1(an existing opt-in on main)PREFILL_MULTI=0PREFILL_SP=0Both non-E4M3 arms pass speculative == serial, cached == uncached behind a shared 4.4K-token prefix, and greedy C8 == C1. So the routed kernels themselves look row-invariant (Enntity's W4A4 check also shows 0 diffs across splits). My current hypothesis: some pass on the replay side (a Marconi checkpoint restore plus its "replaying N SSM tokens" pass, or the exact-hit re-run) reaches the routed experts through a path other than
try_q38_routed_prefill. That path matches the q38 E4M3 chain byte for byte, which is why the shipped profile is unaffected, but it doesn't match either alternative arm. Next: per-pass logging of which routed-prefill arm runs, then compare the cold round with a replay round. Until that's fixed,PREFILL_MOE_BF16=1shouldn't be recommended with prefix caching on.AzeezIsh commentedon Oct 27, 2026
Replay defect: the per-pass trace rules out my "bypass" hypothesis. I used a debug-only build that logs which routed arm every MoE call takes (layer 0) plus each prefill pass's start and row count, and ran
ssm-state-poisoning-gatewith the shipped arms (12/12) and with W4A4 (0/12):try_q38_routed_prefillwith the selected arm in both runs (q38-e4m3everywhere vsw4a4everywhere). Decode and verify take the sameforward/forward_rowspaths in both. Nothing bypasses the switch.MOE_BF16diverges first at turn 2, W4A4 already at turn 1.So with either non-E4M3 routed arm, some stage isn't invariant to how the positions are grouped into passes / restored from checkpoints, even though both arms pass the 4.4K-token prefix test and C8 == C1. Next, a cold-only test with no cache involved: the canonical request at
--max-prefill-tokens 8192vs 16384, compared position by position, per arm. If that splits the arms, it's chunk invariance; if not, it's the checkpoint capture/restore interacting with these arms.AzeezIsh commentedon Oct 27, 2026
Cold chunk invariance holds for every arm. I sent the same cold canonical request (27,034 tokens) at
--max-prefill-tokens 8192instead of 16384 (profile copy; confirmed in argv and in the server's "Prefill config" line), one boot per arm. All 27,033 scored logprobs are bit-identical to the 16K runs for the shipped arm,PREFILL_MOE_BF16, and W4A4.So the routed kernels, chunking, and state carried between chunks are all invariant. The replay defect is specific to restoring a prefix-cache checkpoint and computing the suffix from it. Next: the in-tree prefill determinism tracer (
ATLAS_GLM_DET_TRACE=2) at--max-prefill-tokens 1024, so the cold round's turn 1 runs as passes (0, 1024) + (1024, 30), exactly the shape of every replay's restored suffix (1024, 30). Comparing those passes' per-(layer, stage) hashes should name the first stage that differs after a restore.AzeezIsh commentedon Oct 27, 2026
Correction to the replay-defect attribution above: it is NOT caused by the routed-prefill arms. I traced with the in-tree prefill determinism tracer (
ATLAS_GLM_DET_TRACE=2,--max-prefill-tokens 1024) duringssm-state-poisoning-gate:So restored prefill == cold prefill, byte for byte, for every arm. The round-to-round divergence happens after prefill and depends on timing. W4A4 (faster prefill) and
MOE_BF16(slower prefill) only trigger it by shifting timing. The likely suspects are wall-clock-driven speculation controls (ATLAS_MTP_DYNAMIC_DEPTHchanges 3→4/5/6 within these requests, plus the MTP gate). If so, some depth or gate transition isn't output-preserving on this script, even though speculative == serial holds on the fixed 8-prompt sets. A run withMTP_DYNAMIC_DEPTH=0under the same tracing is next. I'm withdrawing "don't usePREFILL_MOE_BF16with prefix caching": the evidence doesn't single out that arm.AzeezIsh commentedon Oct 27, 2026
Dynamic depth ruled out. Shipped binary,
ATLAS_MTP_DYNAMIC_DEPTH=0(profile copy; 0 depth changes logged):DET_TRACE=2, 1K chunks):ssm-state-poisoning0/12 (all jittered);So any timing perturbation makes replays diverge from the cold round, even though every restored prefill pass is bit-identical (previous comment): the tracer here, and the faster W4A4 or slower
MOE_BF16prefill earlier. The divergence is in a post-prefill decision that reads wall-clock time. Next suspects: the MTP gate's timing windows (#200) andATLAS_MTP_SINGLE_DEPTH_ADAPT/ confidence stop. Next step: prefix caching on with speculation off, under the same tracing, then bisect those switches. The gate currently passes this as "jittered within bounds". Under exact verify, speculation choices shouldn't change tokens, so this is worth pinning down even though it isn't specific to any routed-prefill arm.data-angel commentedon Oct 27, 2026
A data point from Enntity's side, since much of the speculation code here came from our RC16 series.
We ran a port of
ssm-state-poisoning-gateagainst SparkQwen 0.3.3 on one GB10 pair. The port uses the same script, request body, canonical form and verdict rules ascrates/atlas-plugin/src/benchmarks/ssm_poison/on current main, in stdlib Python, with 12 replays. Setup: TP=EP=2, engine Enntity/atlasd0cead14, imageghcr.io/enntity/atlas-sparkqwen:e11549bf2ad1, FP4-Localb9ce18a6, spark mode, our 8x262k profile at util 0.80 plus--disable-thinking, prefix caching on, one boot per arm, no other traffic on the pair.ATLAS_GLM_DET_TRACE=2,--max-prefill-tokens=1024(your perturbation)The tracer engaged in the second arm (26,719
DETlines on rank 0, none in the first). Replays restored 1,024 / 1,072 / 1,152 / 1,280 cached tokens on turns 1-4, against 0 / 1,024 / 1,072 / 1,152 in the cold reference round, so every replay ran restore shapes the reference didn't. The two arms' reference rounds were also byte-identical to each other. One run per arm.Two things that may narrow your suspects:
ATLAS_MTP_GATE_FORCE=1, so the MTP throughput gate is disarmed (--mtp-gate force: MTP throughput gate DISARMEDin the log). If your rc16-resident profile leaves the gate armed, our result fits your next suspect, the gate's timing windows (scheduler: re-measure a full window after a regime change before the MTP gate probes (concurrency-sweep bimodal on main) #200). We didn't run an arm with the gate armed, so this is consistent with that, not a test of it.MTP dynamic depth: 3 -> 5 drafts, and that request finished atdepth=k6 depth_switches=1. All 12 replays of turn 4 finished atdepth=k4 depth_switches=0, and the 77-token output was byte-identical across all 13. So on this script one mid-request depth transition preserved output, which agrees with yourMTP_DYNAMIC_DEPTH=0result.ATLAS_MTP_SINGLE_DEPTH_ADAPT=1was also set; we can't tell from these logs whether it ever changed the draft count, so this run doesn't clear it.Raw transcripts, both ranks' logs and the driver are available if you want them.
Separately, on #201: our engine hit the same
n>1vision bug and fixed it independently in Enntity/atlas9fabbf1c("server: preserve shared image pixels for every blocking choice"), with a regression test incrates/spark-server/src/api/chat_blocking_vision_tests.rs.AzeezIsh commentedon Oct 27, 2026
Correction 2: it isn't timing, it's chunk geometry. My last two comments blamed timing (tracer stream syncs). That was confounded: every shipped-binary reproduction changed both the tracer and
--max-prefill-tokens(16384 → 1024). Separated, same binary and profile otherwise:ssm-state-poisoningMTP_GATE_FORCE=0/MTP_SINGLE_DEPTH_ADAPT=0/MTP_DYNAMIC_DEPTH=0What this shows:
MOE_BF16changed the turn-1 output length, which moved the positions, so they hit it at 16K.Next: establish which side matches an uncached reference (cold round vs replays) and which checkpoint kind (chunk-end vs mid-chunk tail at 1088) is inexact. I'm withdrawing the "any timing perturbation" wording from my previous comment.
AzeezIsh commentedon Oct 27, 2026
Resolved the attribution: on the shipped config (16K chunks) the cached replays of this conversation equal an uncached server text-for-text in every round (uncached reference via a no-cache, no-speculation boot; harness replays the gate's exact script and request body). At
--max-prefill-tokens 1024both the cold cached round and the replays differ from uncached, with or without speculation. That's a prefix-cache exactness bug that depends on chunk geometry. Filed separately as #218 with the repro. It isn't caused by the routed-prefill arms, which probably just move positions onto the same bug. This issue keeps the routed-prefill numerics results.AzeezIsh commentedon Oct 27, 2026
Closing the loop on
PREFILL_MOE_BF16's replay result: it was #218. On the #218 fix binary, withATLAS_QWEN4EXP_PREFILL_MOE_BF16=1and one boot per arm:ssm-state-poisoning-gateis 12/12 byte-identical (0/12 without the fix).MOE_BF16just moved positions onto the short-pass K/V mismatch fixed in #218. The same check for the W4A4 opt-in (W4A4 + the fix) is queued.AzeezIsh commentedon Oct 28, 2026
W4A4 too: its replay result was #218. With W4A4 + the #218 fix (
integ/w4a4-rowinvfix@b85e1b828), cached replays equal the uncached server in all 3 rounds, andssm-state-poisoning-gateis 12/12 byte-identical. Neither routed-prefill arm has a cache defect of its own. W4A4's remaining trade-off is the one above: TTFT −7 to −8% for a small NLL cost.AzeezIsh commentedon Oct 28, 2026
@data-angel thanks, that run helped narrow it. Your 12/12 at
--max-prefill-tokens 1024with the tracer on was the right control: it showed the trigger wasn't timing.mxfp8_proj, the prefill entry for FP4-Local's native MXFP8 projections, used GEMV tiers for passes of ≤ 32 rows and expand + GEMM above that. K/V or SSM rows written by a short pass (a short last chunk, or a restore suffix) then differed from what a longer uncached pass computes. SparkQwen keeps an exact BF16 copy for prefill (dense_mx.rs), which is why your engine is 12/12 under the same perturbation.ATLAS_MTP_GATE_FORCE=1. With speculation off entirely, the 1K-chunk divergence was identical, which rules outMTP_DYNAMIC_DEPTH,MTP_GATE_FORCEandMTP_SINGLE_DEPTH_ADAPT. That matches your observation that a mid-request depth switch preserved output.1aa1fa598(now in qwen4_exp: serve FP4-Local at TP=EP=2 with the n-gram table resident, native MXFP8 sharding, BF16 routing #198). The ssm gate is 12/12 at 1K and 16K chunks, and both routed-prefill opt-ins (MOE_BF16, W4A4) are clean on top of it.Thanks also for the pointer to
9fabbf1cfor the #201-class vision bug.