[AzeezIsh]

Summary

The concurrency-sweep gate (nvidia/Qwen3.8-27B-NVFP4) is bimodal on main. C=8 reads either ~54 or ~47 aggregate tok/s against a 50.5 floor, on main and on every branch measured:

	C=8 aggregate tok/s, 5 runs each (2026-10-20/21, sparky)
main e29809db9	54.4, 47.1, 54.3, 54.4, 47.1
#195 head 5ddfabb6e	46.9, 47.0, 53.5, 47.2, 55.7 (C=16 failed instead: 59.6)

Cause: the MTP throughput gate, not the model or the kernels.

• In the six runs whose server logs I read, every slow cell has MTP gate: switching Mtp -> Serial at its start, and no fast cell has one.
• After a width-regime change, note_width() pulls the probe forward so far that the first in-regime step closes a one-step window. That single step replaces the current mode's EWMA.
• When all prefills land before the first batched step, the batch jumps from width 1 straight into bucket N. The first step is then the bootstrap: one token per sequence, plus lazy CUDA-graph captures for the new shapes.
• From main, run-1792544951131605812: switching Mtp -> Serial (current 33.5 tok/s vs other 56.4 tok/s, margin 8.2, depth=734), while verify was delivering ~70 tok/s (8 / 0.113 s TPOT).
• Whether the jump is 1→N or 2/4→N is a prefill/decode timing race, hence the two modes.

Fix (crates/spark-server/src/scheduler/mtp_gate.rs):

• After a width or depth regime change, the current mode re-measures over one full WINDOW_STEPS window before the probe opens. WINDOW_STEPS' own doc already says one window "amortizes bootstrap/propose transients"; the one-step close violated that.
• arbitrate now refuses while either baseline is stale. Previously a probe that straddled a regime change could count a losing window: a fresh other-mode number against a stale current one.

Tests (mtp_gate/tests.rs):

• New one_bootstrap_step_cannot_set_a_new_regime_baseline replays the production cell's walls and token counts. On the old gate it switches to Serial; with the fix it stays Mtp.
• New stale_current_baseline_cannot_count_a_losing_window.
• Four regime-change tests updated to the full-window policy. All other gate tests are unchanged.

Evidence

All on sparky (GB10), 2026-10-21, nvidia/Qwen3.8-27B-NVFP4 through its own gate recipe (--pull-request-gate), interleaved, 180 s cooldown before every run. Fix = e29809db9 + this commit (binary sha256 cd3630581c3ec0a9); main = e29809db9 (98d17fab2a3ae5cc). Logs: ~/code/perf/nq-gatefix/.

concurrency-sweep, gate auto (default): aggregate tok/s; floors 16.2 / 33.5 / 50.5 / 72.0

run	record	C=1	C=4	C=8	C=16	gate switches	verdict
fix #1	run-1792567787441788471	24.3	42.5	54.3	77.4	0	PASS
main #1	run-1792568223349315164	24.3	38.8	55.2	59.2	1	FAIL
fix #2	run-1792568634145330563	24.3	43.4	54.6	77.7	0	PASS
main #2	run-1792569078596206207	24.3	40.1	46.9	59.3	3	FAIL
fix #3	run-1792569487975168857	24.3	43.8	56.3	77.2	0	PASS
main #3	run-1792569926943462361	24.3	37.4	53.7	59.3	3	FAIL
fix #4	run-1792570337375061294	24.3	39.7	58.7	78.4	0	PASS
fix #5	run-1792570743391640982	24.3	43.8	57.4	78.2	0	PASS

• Fix: 5/5 PASS, zero switches, including runs whose batch jumped 1→4 and 2→16. Before the fix, those jumps are exactly the ones that switched.
• Main, interleaved: 0/3 PASS, every failure with a Mtp→Serial switch at the start of the slow cell.
• Upper bound: with the gate bypassed entirely (ATLAS_MTP_GATE_FORCE=1), the same box earlier tonight gave C=8 56.8–58.5 and C=16 79.4–79.9 on both main and qwen4_exp: serve NVIDIA's Qwen3.8-Flash-Next FP4-Local on its native formats (one GB10) #195, 6/6 PASS. The fix sits just under that because the gate still pays its periodic serial refresh.

decode-floor (C=1): fix 25.2 tok/s median over 3 pinned runs (accept_len_mean 2.76), main 25.2 tok/s (2.76). Identical, as expected: C=1 never changes width regime.

All numbers are observed under this fingerprint, 5 and 3 repeats. The mechanism is shown by the switch lines, not inferred from the throughput.

Test plan

• cargo test -p spark-server --bins mtp_gate: red on the unfixed gate, green with the fix
• cargo fmt --all -- --check, cargo clippy -p spark-server --tests
• concurrency-sweep ×5 on the fix and ×3 on main, interleaved; decode-floor on both (sparky)
• Required-gate campaign at the final tip (this touches the scheduler, so every record is invalidated)

Generated with Devin