Repository navigation
[job] Strix Halo: multi-session serving (parallel or sequential) and long context, Flash-Next and the 27B #145
Description
Status (updated 2026-10-28 05:30Z)
State: in progress. Flash-Next serves on Linux (framework, 96 GB carve-out) from main since #197 merged (c3774309e, 2026-10-27). Post-merge build and serve were verified on both boxes at 8863dc5e5.
Done / measured
- Warm-turn slowdown root-caused.
ATLAS_HIP_MOE_GROUPED_MIN_TOKENSdefaulted to 1024, so every short prefill chunk ran MoE experts per token.- Setting it to 64 makes warm agent turns ~4× faster and cached-prefix prefill ~9× faster (f46, f49).
- ST-995 holds at 84.52/81.97 (f48).
- Agentic, same build: 1024 scores 50/50, Σwall 15,441 s (f51). 64 scores 45/50 (f47) and 47/50 (f53), Σwall 10,689 s.
- The 64 failures are all "no Cargo.toml was written", so 64 is not proposed.
- 256 is under test.
- Batched decode doesn't scale: steady C=8 is ~15 tok/s against Gufo's 125 (AR).
- Cause: routed and shared MoE run per row in the mHC batched decode (f50).
- The GB10 1–4-row kernel ported to HIP is bit-identical but no faster (f52), so it is parked.
ATLAS_QSA_VERIFY_ACTIVE=1brings back MTP past the QSA bound on gfx1151: 8k/32k decode 12.1 → 18.9/18.0 tok/s (f55).
Queued (framework):
- f56: agentic at 1024, second sample.
- f57: agentic at 256.
- f58: build the verify-active default branch and prove the flip.
- f59: ST-995 at 256.
Local branches (push disabled in the operator clone; the owner pushes):
perf/hip-qsa-verify-active-default88cffac78: ready.fix/hip-moe-grouped-min-645f57b08cc: on hold; the value is likely to become 256.perf/hip-qwen4exp-moe-decode750a18bfb: parked.
Next: a gfx1151-shaped batched MoE decode kernel (expert dedup across rows plus int8 dot). It is the C>1 limiter.
Why: Flash-Next on Strix Halo is served one request at a time today.
- The Windows recipe pins
--max-batch-size 1and defaults to an 8k context. 32k is verified by the 10/10 agentic-webserver leg on Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80. - Windows' WDDM commit wall caps resident memory at ~84.5 GiB (VGM 32 GB).
- Flash-Next hasn't been served on Linux yet. [job] Qwen3.8-Flash-Next: land the Strix Halo Windows port #57's Linux line: the build is verified; serving needs the 128 GB box booted into Linux.
- A second concurrent MTP request needs a batched MoE verify arm for K ≥ 3 rows, which doesn't exist. Qwen3.8-Flash-Next EXL3: MTP + native int8 decode for GDN/attention/lm_head (M6, follow-up to #91) #100 reports
no batched MoE arm for K=4at n=2. On GB10, reiner job 643's K sweep for [job] Qwen3.8-Flash-Next: prompt-lookup + MTP hybrid drafting for agentic decode #130 was clamped to K=2 for the same reason.
Multi-session (tester ask, 2026-09-30): a 128 GB tester wants 2–3 agent sessions on one box, for Flash-Next or the 27B. Memory decides which shape is possible:
- Parallel: batch 2–3. The 27B has room for it. Flash-Next on Windows sits close to the commit wall at batch 1.
- Sequential: one session runs at a time, but every session's prefix KV and SSM snapshots stay resident, so returning to a session doesn't re-prefill its history. Today the Windows Flash-Next recipe runs
--ssm-cache-slots 0with no prefix caching, so every turn re-prefills. In the running Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80 perf leg that means 60+ s TTFT at 20k+ context. - Stretch: spill idle sessions' KV and SSM snapshots to NVMe and restore them on their next turn.
Done bar (the 128 GB box under Linux; Windows limits documented beside it):
- C=8 greedy serving at ≥ 150 tok/s aggregate on short prompts, 0 faults, and no request refused for KV;
- 128k context at C=1, with the memory plan written into the recipe;
- C=2 on the agentic replay at 0 dropped turns, plus ST-995 unchanged (AGENTS.md platform rule);
- 3 interleaved agent sessions at batch 1 (the sequential shape) on Flash-Next and on the 27B. A revisit turn must restore its session's cache rather than re-prefill: the restore shows in the serve log, and revisit TTFT is reported against the cold TTFT. The
ssm-state-poisoning-gatemust pass under the recipe that does it.
Plan, each step gated by the one before:
- Serve Flash-Next under Linux on the 128 GB box, using the same launcher modes as Windows. Record the memory map: weights, KV, SSM pools, snapshots, and how much headroom is left.
- Batched MoE decode and verify arms for 2–8 sequences × K rows, which also unblocks MTP at C ≥ 2.
- A KV/SSM pool plan for C=8 at 32k and for C=1 at 128k. Then a C = 1/2/4/8 sweep on greedy short prompts and the agentic replay at C=2.
- Sequential multi-session: size the snapshot pool and prefix cache for 3 resident sessions inside each platform's memory wall, then measure revisit TTFT. The NVMe spill of idle sessions comes after, if residency doesn't fit.
Related: #57 (the port), #58 (DFlash2 batched concurrency, GB10), #100, #130, and #146 (thinking switch, flag cleanup, memory auto-sizing).
Activity
[-][job] Qwen3.8-Flash-Next on Strix Halo: concurrency and long context[/-][+][job] Strix Halo: multi-session serving (parallel or sequential) and long context, Flash-Next and the 27B[/+]AzeezIsh commentedon Oct 9, 2026
Unblocks multi-session and long context on winbox: the iGPU allocation went from 32 GB (~77 GB visible, ~63 GB usable) to 96 GB dedicated (107.9 GB HIP-visible) on 2026-10-09. Old KV and slot budgets measured on winbox are stale. Re-measure the commit ceiling before sizing. Details: #144 (comment)
AzeezIsh commentedon Oct 18, 2026
Status update (2026-10-18, framework 128 GB / aizeez): Flash-Next now serves on Linux.
Chain that works on
integ/amd-train-suballoc-leak(train +fix/hip-suballoc-145@ 969d28a, jobs f19/f20, bin4ac55d80be57):--gpu-memory-utilization 0.95+ATLAS_HIP_SUBALLOC=1(default on Linux) +--dangerously-allow-unresolved-kernel-lookups(hc_pre_down_mtisn't eager-registered on the strix-hip target — needs a real registration or the flag in the recipe).serve-amd.sh-adjacent fingerprint, max-seq 16384, MTP--num-drafts 1, thinking off).restored=2(the "restore, don't re-prefill" evidence this issue's done bar wants, at small scale).Also: the f17 "slabs leaked" FAIL was a test-bound bug, not a shim bug — the bound ignored the ~0.15 GB ROCm runtime pool (measured with
SUBALLOC=0). Shim retention was already ≤1 slab/class on this path; now provably bounded regardless of free order (969d28ab2, branchfix/hip-suballoc-145pushed). Strix s08: 27B outputs IDENTICAL with suballoc on vs off; nemotron golden 3/3 on framework.Still open here: full C=8/128k sweeps, batched MoE verify arms, and the kernel-audit registration for
hc_pre_down_mt.AzeezIsh commentedon Oct 19, 2026
FP4 dtype survey on the AMD track (2026-10-18) — decided: stay on NVFP4.
Looked at every FP4-family endpoint for these models on HF plus Atlas's loader support:
nvidia/Qwen3.8-Flash-Next-FP4-Local(~102 GB): routed experts are still NVFP4 (CSF scales); the delta is MXFP8 projections + F32 norms + 28.8 GB disk PLE. GB10 pre-KV ~93 GB. On framework (96 GB visible) that's ~0–4 GB KV at util 0.98 — fails the multi-session KV floor the NVFP4 pack meets (5.3 GB @0.95). Strix (60 GB) can't host it at all. Boot-boundary probe queued on framework as job f27 for the record.FreedomAISVR/Qwen3.6-35B-A3B-MXFP4-MOE-GGUF— GGUF, and Atlas's GGUF path covers classic quants only (Q4_0…Q8_K; no MXFP4 block, no qwen3moe arch). Not servable.Also: framework f22 landed the first FNext-on-Linux ST-995 — 84.12/82.33 (thinking default) + f24 TS2 84.32/82.97 — on
integ/amd-train-suballoc-v2; f23 first Linux perf sweep ~22 tok/s MTP decode. 35B-A3B serve queued (f25): it refused on strix's 60 GB but fits framework's 96 GB — result overnight.AzeezIsh commentedon Oct 20, 2026
Overnight consolidated status — the docked wins (2026-10-19/20, both gfx1151 boxes).
Now green on
integ/strix-hip-train-v1(= PR #197, the collapsed stack):7e430d0a18dc/ framework tv1) and on a fresh DevLab gfx1151 runner — where main itself fails (atlas_pdl.cuh+__ldcgon every kernel). So this stack is the difference between a buildable and un-buildable AMD target.restored=2, C-sweep 22.0→35.9 agg tok/s at C=8, 0 faults.FP4-Local-on-AMD — honest boundary + the viable path:
nvidia/Qwen3.8-Flash-Next-FP4-Local(~102 GB repo, ~93 GB pre-KV on GB10): cannot be staged on framework (74 GB free disk) or strix (58 GB total RAM < ~73 GB weights). Not a fit question — a capacity wall.integ/fp4local-hip-train, merge clean) — the live question is whethermxfp8_proj.cusurvives the HIP mirror; that's building on framework now (f32). If it compiles, the port is viable; the checkpoint still isn't.Flags for review:
hc_pre_down_mtstill rides--dangerously-allow-unresolved-kernel-lookups(needs eager registration); the s11 baseline serve died mid-bench once (retry clean — logged as flake).AzeezIsh commentedon Oct 23, 2026
Overnight verification ledger — every claim below is in
~/ovn-out/*/RESULTS.txt(strix + framework, job IDs noted).Build/portability
integ/strix-hip-train-v1(= PR strix-hip train: the validated gfx1151 stack (build fix + suballoc + carveout + hipBLASLt) #197) builds on both gfx1151 boxes and the DevLab pool — main does not (atlas_pdl.cuh/__ldcgper-kernel; DevLab run 39105011577 vs branch 39105009616 PASS).integ/fp4local-hip-train(fp4-local + train) builds on gfx1151 after one intrinsic shim (__float2bfloat16_rn) and serves the standard NVFP4 checkpoint clean — the port is viable; the checkpoint is not (102 GB repo, ~93 GB pre-KV; disk + KV-floor walls documented above).Output identity — suballoc on/off, all bit-identical (mean|dlp| 0.000): 27B (s13), 35B-A3B (f29), Flash-Next (f28).
Accuracy (ST-995, thinking default unless noted)
New capability gates
restored=2, C-sweep 22→35.9 agg tok/s (f26), 0 faults.Shortcomings measured (not yet fixed)
Infra bugs found & worked around tonight: f24 watchdog-subshell held
queue.lockforever (leaked pgid); two zombie claims from dispatcher-death-mid-claim (f25, s11); f30's nemotron-scoped build clobbered tv1's full-model binary; nemotron script'sVRAM < 90 GiBpreflight is wrong for the APU (4 GiB dedicated + GTT).AzeezIsh commentedon Oct 23, 2026
Prefill shortcoming localized (f37/f38/f39, framework + strix). The ~9.4 s TTFT at ~1k ISL on Flash-Next is not a fixed warmup — short reqs run 0.26-0.31 s while every ~1k req costs ~9.4 s, and the request-window log is wall-to-wall
MTP drafter context: prefill=ON carry=ONrepeating every ~200-800 ms (TTFT=9427.9 ms for an 812-tok prompt, mean_na=0). The drafter-context init appears to re-execute per chunk/step and dominate small-mid ISL prefill. A/B queued: f40 = same requests withATLAS_NO_MTP_DRAFTER_CONTEXT=0/1at ISL 900 + 8192 — if the 9.4 s collapses, that's the kernel-dispatch defect. Dense-27B strix: ~216-284 tok/s vs Gufo ~1,628 — the bigger ~7.5x gap.AzeezIsh commentedon Oct 27, 2026
2026-10-27 update: the ~1k-token TTFT scare is withdrawn. The cache-resume path has a real slowdown.
1. Correction to my 10-23 comment: the drafter context is not a prefill cost.
ATLAS_NO_MTP_DRAFTER_CONTEXT=0and=1, on thetv1binary. The results are identical either way:=0=1RUST_LOG=debugrun. Its "drafter context re-runs per chunk" lines are a log flood, not repeated work. The attention-projection prefill GEMM callsModelLevers::from_env()on every launch (qwen3_attention/prefill_weights.rs:44), andfrom_env()logs the drafter-context and GDN-deferred-commit lines each time. One info-level bench produced 1,218 of them.fix/levers-per-launch-log(d9a09f44f, off main). It adds a one-field accessor and makes no behaviour change. It is not pushed yet.2. New finding: resuming from a Marconi snapshot is ~13× slower than cold prefill (framework f41, Flash-Next NVFP4, b9ab4ed54 = main + #197 + #204).
Marconi SSM cache hit: 7344 tokens skipped … replaying 1867 SSM tokens to reach 9211): TTFT 34.8 s, about 54 tok/s on the 1,867 new tokens. A second repetition read 35.0 s.3. Gufo-matched baselines are running (greedy, thinking off, pp2048/tg128, cached depths, C = 1/2/4/8; both Gufo's sum-of-rates and the wall aggregate): f41 (Flash-Next: AR / MTP1 / MTP2) and s21 (27B: AR / MTP / DFlash2). First rows:
The full tables will go here when the jobs finish. The local ledger is at
~/code/notes/amd-campaign-2026-10-27/LEDGER.mdon the operator Mac.AzeezIsh commentedon Oct 27, 2026
Root cause of the Flash-Next warm-turn slowdown: a stale HIP MoE threshold sends any chunk under 1,024 rows through the per-token expert path. Framework f42, SYNC launch trace, Flash-Next NVFP4, b9ab4ed54.
What the trace shows (6,575-token prefill, GPU 14.7 s):
moe_expert_gate_up_shared_t+moe_expert_silu_down_shared_t, the decode GEMVs, take 7.1 s.Cause (
layers/moe/forward_prefill.rs:108-134):ATLAS_HIP_MOE_GROUPED_MIN_TOKENS(default 1024) goes toforward_batched, which dispatches experts per token.> 64.Being measured now (framework, the threshold is an env var, so no rebuild):
Side results:
quantize_bf16_to_fp8_blockscaledhas no HIP build. NVFP4 GDN doesn't fit (94.2 GB committed before KV). Details in [job] Qwen3.8-Flash-Next on Strix Halo: decode speed (serial and MTP) #143 to follow.AzeezIsh commentedon Oct 27, 2026
Fix measured: the HIP MoE grouped-GEMM threshold 1024 → 64 makes warm agent turns 4× faster. Framework f46, Flash-Next NVFP4, b9ab4ed54, MTP1, prefix caching on. The threshold is an environment variable, so this used the same binary at every value.
Cold TTFT (best of 2) at each
ATLAS_HIP_MOE_GROUPED_MIN_TOKENSvalue:Marconi resume on a ~6.1k prefix:
Greedy output: every value below 1024 gives the same 96-token answer, and it differs from the 1024 answer by one word ("linking" → "combining"). Short prefills now use the same grouped arithmetic that every prompt of 1,024+ tokens already used, so this is a numerics change, not a new code path.
Gates: framework f47 (agentic-webserver, 50 iterations) and f48 (ST-995, same profile as f05's 84.12/81.47) both run at 64. f47 was already running at 64; f48 reads the value from a file, now set to 64 to match. 32 wins only at 59 tokens and is left as a follow-up.
Code: local branch
fix/hip-moe-grouped-min-64(10a902576), based oninteg/strix-hip-train-v1, so it can ride #197 or follow it. It changes one default plus the comment. It is HIP-only behindcfg!(atlas_hip), so NVIDIA is byte-unchanged. It also covers the other NVFP4 MoE models on HIP (35B-A3B, Nemotron Super). Not pushed: push is disabled in this clone, so the owner pushes it.AzeezIsh commentedon Oct 27, 2026
First Flash-Next agentic soak on Linux, run at the new MoE threshold (64). framework f47, b9ab4ed54,
ATLAS_HIP_MOE_GROUPED_MIN_TOKENS=64, MTP1, prefix caching on, thinking at the checkpoint default. Recordrun-1793094528039258410.Pass rate:
Timing:
Caveats:
AzeezIsh commentedon Oct 27, 2026
The MoE threshold fix (1024 → 64) holds accuracy and makes cached-prefix prefill ~9× faster. Framework, Flash-Next NVFP4, b9ab4ed54.
ST-995, thinking at the checkpoint default, f05 profile (f48,
run-1793110320484201600):ATLAS_HIP_MOE_GROUPED_MIN_TOKENSThe +0.4 / +0.5 difference is within run-to-run noise for this leg. Read it as "no regression", not as a gain. 0 faults.
Gufo-matched prefill of ~1,850 new tokens on a cached prefix, AR (f49 vs f41, same build and serve flags):
Decode is unchanged (12.1–12.4 tok/s AR), as expected; this fix only touches prefill.
Gates at 64:
fix/hip-moe-grouped-min-64(10a902576, based oninteg/strix-hip-train-v1) is ready to ride strix-hip train: the validated gfx1151 stack (build fix + suballoc + carveout + hipBLASLt) #197 once f51 confirms. Push is disabled in this clone, so the owner pushes it.For context against Gufo: its Flash-Next pp at 8k depth is 1,721 tok/s; ours is now 499. Cold pp at depth 0 is 726 against 1,801. The remaining prefill gap is kernel throughput, tracked in #144.
AzeezIsh commentedon Oct 27, 2026
Same-build agentic A/B for the HIP MoE threshold: 64 cuts Σwall 31% and median TTFT 4.2×. The pass rate needs a replicate before the fix merges. Framework, Flash-Next NVFP4, b9ab4ed54, MTP1, prefix caching on, thinking at the checkpoint default, 50 iterations each.
ATLAS_HIP_MOE_GROUPED_MIN_TOKENSrun-1793128371095495787run-1793094528039258410Both arms ran with 0 faults.
Speed: the win is unambiguous. It matches the per-request sweep in f46.
Pass rate: 45 vs 50, from one run per arm.
Replicate: f53, same build, threshold 64 plus
ATLAS_QWEN4EXP_MOE_FAST=1.AzeezIsh commentedon Oct 28, 2026
MoE threshold 64: the replicate confirms the pass-rate gap, so the fix is not merge-ready as-is. Framework, Flash-Next NVFP4, agentic-webserver, 50 iterations.
run-1793128371095495787run-1793094528039258410MOE_FAST=1, bit-identical per f52)run-1793144150427619787Reading:
wrote_project. The single 1024 run has none.Queued on framework, ~8 h:
Decision: if f57 holds 50/50, the proposed default becomes 256, not 64. Branch
fix/hip-moe-grouped-min-64stays unproposed until then.AzeezIsh commentedon Oct 28, 2026
HIP MoE grouped-GEMM threshold: 256 is the value to ship. It keeps most of the warm-turn win with no agentic regression. Framework, Flash-Next NVFP4, agentic-webserver, 50 iterations, MTP1, prefix caching on, thinking at the checkpoint default.
ATLAS_HIP_MOE_GROUPED_MIN_TOKENSrun-1793128371095495787run-1793094528039258410run-1793144150427619787run-1793178905812470578Reading:
wrote_project. Neither the 1024 run nor the 256 run fails it.Code: local branch
fix/hip-moe-grouped-min-256(909dc8b20, rebased onto main8863dc5e5). It is a one-line HIP-only default change with the evidence in the comment, and it keeps #188'scfg(test)default of 1. ST-995 at 256 is queued (f59). If that holds, the branch is ready for the owner to push.AzeezIsh commentedon Oct 28, 2026
ST-995 holds at
ATLAS_HIP_MOE_GROUPED_MIN_TOKENS=256on Flash-Next (framework f59). Thinking default, prefix cache per recipe, recordrun-1793198886445328519.json.256 is within the 64↔1024 spread on both metrics. It was chosen because 64 caused the agentic "no Cargo.toml" failures and 1024 sent warm turns through the per-token MoE path. Faults: 0 errors, 0 panics, 0 illegal, 0 watchdog.
Branch
fix/hip-moe-grouped-min-256909dc8b20is ready for the owner to push (push is disabled in this clone). It changes one line of default incrates/spark-model/src/layers/moe/forward_prefill.rs,.unwrap_or(if cfg!(test) { 1 } else { 256 }), with the evidence in the comment.AzeezIsh commentedon Oct 29, 2026
Flash-Next
--max-prefill-tokens 8192: agentic pass rate holds, but agentic TTFT gets slightly worse, so the default stays at 2048 (framework f74 / f75). agentic-webserver, 50 iterations,perf/hip-qsa-attn-tc262e4d801d(tc2 on, MIN=256),--max-seq-len 40960, prefix cache ON, thinking default. Matched pair, differing only in chunk size:--max-prefill-tokensrun-1793246850093767652run-1793264442604964401Cold long prefills are 9–12% faster at 8192 (f69), and accuracy is unaffected (f70 ppl + needles, now this agentic pair). But warm multi-turn TTFT is slightly worse (+8% median, +12% p90), probably because prefix-cached suffixes don't benefit from bigger chunks while a bigger chunk holds the GPU longer. Neutral-or-worse doesn't change a default: 2048 stays the default; 8192 is a documented option for long single-shot documents.
(The first control, f72 at
--max-seq-len 32768, aborted at run 23/50 when an agent reached a 24,930-token prompt +max_tokens 8192. That's why the pair was rerun at 40960.)spark-runtime: don't report host MemAvailable as device free on a car…
hip shim: sub-allocate the request sizes hipMalloc rounds up wasteful…
hip shim: bound retained empty slabs to one per size class (Atlas-Inf…