[job] Qwen3.8-Flash-Next: land the Strix Halo Windows port #57

Activity

1. AzeezIsh

   added

   jobA piece of work anyone can pick up. Shows on atlasinference.dev/#jobsA piece of work anyone can pick up. Shows on atlasinference.dev/#jobs

   on Sep 23, 2026
2. AzeezIsh

   added a commit that references this issue on Sep 23, 2026

   site: jobs render plain text; snapshot carries the first three jobs

   ...

   da84a7a
3. AzeezIsh commented on Sep 24, 2026

   

   AzeezIsh

   on Sep 24, 2026

   ContributorAuthor

   More actions

   Status note: the merge vehicle remains #54 (stacked on #35). The Windows pipeline itself is healthy — winbox built and served the DFlash2 stack E2E today on port/dflash2-strix-windows-r2 (γ=4, 28.5 tok/s server-attested, 0 unresolved kernels), so the gfx1151 Windows toolchain is proven for the sibling model. Once #35 + #54 land and win_serve_flashnext_nvfp4.ps1 boots from a clean checkout, this can close.
4. AzeezIsh commented on Sep 24, 2026

   

   AzeezIsh

   on Sep 24, 2026

   ContributorAuthor

   More actions

   Status update for whoever picks up the Strix Windows leg: the DFlash2 Windows serve script is on main now (PR #60, merged 83d1bf52): scripts/strix-windows/win_serve_dflash2.ps1 — ASCII-clean for PowerShell 5.1, verified live on winbox at γ=4 (28.5 tok/s server-attested, 0 unresolved kernels). Also on port/dflash2-strix-windows-r2 (a4295fda6) with the nvfp4 variant.

   Config that OOM'd in the community test (64K ctx, SSM slots 0) is not the recipe profile — the shipped script pins seq 8192 / util 0.80 / ssm-cache-slots 20 / batch 1 / --no-fast-load. If the Flash-Next Windows port wants a sibling win_serve_flashnext_nvfp4.ps1, the DFlash2 script is the template.
5. AzeezIsh commented on Sep 25, 2026

   

   AzeezIsh

   on Sep 25, 2026

   ContributorAuthor

   More actions

   In flight (winbox, 2026-09-25): Flash-Next on Windows gfx1151, fnext-win-r2 build (atlas-fnext-win, binary 09-22 13:47), recipe win_serve_flashnext_nvfp4.ps1 (util 0.90/0.86, seq 8192, prefill 2048): serial vs MTP K=2 (+ ATLAS_QSA_VERIFY_ACTIVE=1), streaming decode tok/s on code/JSON/prose/~5k code, TTFT at ~2k/~6.5k. Reference: Gufo Flash-Next on Strix Halo Linux 1,628 tok/s pp / 59.4 tok/s tg; this branch's own last record ~8 tok/s serial / 16.7 MTP. Results land here.
6. AzeezIsh commented on Sep 25, 2026

   

   AzeezIsh

   on Sep 25, 2026

   ContributorAuthor

   More actions

   Result: Flash-Next on Windows gfx1151 (winbox, 2026-09-25).

   • Build: fnext-win-r2 @ 3d0684aa (PR Strix Halo Windows: Flash-Next port + newest DFlash fix stack (validation running) #54), nvidia NVFP4, boot-7g recipe.
   • Arms: serial at util 0.86 vs MTP K=2 at 0.90; seq 8192, prefill chunk 2048; streaming, 2 reps each.

   workload	serial tok/s	MTP tok/s	speedup
   code (36 tok prompt)	8.96 / 9.20	17.54 / 18.29	1.96x
   JSON	9.15 / 9.05	18.87 / 19.38	2.1x
   prose	8.93 / 8.86	15.18 / 16.30	1.75x
   code at 4.7k context	10.10 / 10.17	9.93 / 9.97	none

   prefill	serial TTFT	MTP TTFT	rate
   2.05k tok	27.3 / 28.1 s	24.4 / 26.0 s	~75 tok/s
   7.05k tok	101.5 / 101.6 s	93.9 / 93.5 s	~70-75 tok/s

   Reading:

   1. MTP roughly doubles short-context decode. Serial is byte-deterministic across reps; MTP prose is not (sha differs rep0 vs rep1), the same behavior as on GB10 (reiner job 257 is gating it).
   2. MTP turns off past the QSA bound here, so 4.7k shows no gain. The serve log reads serial=1.00 on long prompts. ATLAS_QSA_VERIFY_ACTIVE (qwen4_exp: MTP past the QSA inert bound (opt-in per-row verify) + three speculative-path safety fixes #51) is not on this branch, so the flag was a no-op. On GB10 it gives +56 % at 5.1k.
   3. Prefill is the problem: ~70-75 tok/s against a Gufo reference of 1,628 pp on the same silicon (Linux), a ~22x gap. Decode is 9 vs 59.4 tok/s, 6.5x.

   Next for winbox:

   • (a) Rebase fnext-win-r2 onto main. It is 35 ahead / 44 behind (merge base 1004b568). The rebase brings qwen4_exp: MTP past the QSA inert bound (opt-in per-row verify) + three speculative-path safety fixes #51 verify-active, the spark-server: drop hoisted SSM pool terms from inference_reserve #66 memory fix, and the rest of main.
   • (b) Rebuild and re-run this bench.
   • (c) Prefill kernel attribution on Windows, to see whether the MoE path is the 22x.
7. AzeezIsh commented on Sep 26, 2026

   

   AzeezIsh

   on Sep 26, 2026

   ContributorAuthor

   More actions

   Triage (2026-09-26).

   • The merge vehicle is Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80. Strix Halo Windows: Flash-Next port + newest DFlash fix stack (validation running) #54 and Strix Halo Windows (gfx1151, native HIP): Qwen3.8-Flash-Next NVFP4 port + DFlash2 recipe — ST-995 + perf legs #41 should close as superseded; both fail checks Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80 passes.
   • Before Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80 merges: an NVIDIA support-matrix boot, because it installs a cuBLASLt fallback at init for every model.

   Scope: measured Windows Flash-Next is ~70–75 tok/s prefill and 9 serial / 18 MTP decode, against Gufo's 1,628 / 59.4 on Strix Linux. The OS and engine are confounded in that comparison.

   • Suggestion: close this job when Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80 merges.
   • Don't open Windows-specific Flash-Next perf work until the prefill attribution is done. The 128 GB box is dual-boot, so the matched Linux comparison can be run there ([job] Qwen3.8-Flash-Next: run on Strix Halo Linux #56).
8. AzeezIsh commented on Sep 27, 2026

   

   AzeezIsh

   on Sep 27, 2026

   ContributorAuthor

   More actions

   Where the ~22x Windows prefill gap most likely lives (Qwen3.8-Max consult plus the facts in this thread).

   • Measured: Flash-Next prefill on winbox is about 70–75 tok/s at 2k and 7k. The dense 27B on the same silicon under Linux does 233 tok/s at 2k ([job] Qwen3.8-27B on Strix Halo: prefill 2.4x gap + DP4A verify tiers for K=5..9 #72). A sparse MoE whose active parameters are far fewer than 27B running 3x slower than the dense model points at a broken MoE compute path. WDDM and the OS alone don't explain it, because a commit wall shows up as stalls or faults (the 719 context kills), not as a steady, low rate.
   • Most likely: routed experts dispatching as many small per-expert kernels or a non-WMMA fallback, instead of one grouped GEMM. Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80's own notes fit this: pinning the MoE grouped/batched crossover and the transpose gave 2k TTFT about 2x.
   • One measurement splits it, and Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80 already carries the tool: ATLAS_TRACE_LAUNCH_SYNC per-kernel GPU time (rocprof isn't available on Windows). On one 2k prefill, record kernel count, GPU-busy vs wall, and the top kernels by time:
     • thousands of tiny kernels and low busy → launch-bound MoE fallback;
     • a few long kernels at a low FLOP rate → a scalar/BF16 fallback GEMM;
     • long idle gaps → WDDM commit/launch.
   • The matched comparison is the same box booted into Linux ([job] Qwen3.8-Flash-Next: run on Strix Halo Linux #56). Gufo's 1,628 is a Linux number, so until then OS and engine are confounded.

   For scope: land #80 through the #112 train, and run this attribution before opening any Windows-specific Flash-Next perf work.
9. AzeezIsh

   added a commit that references this issue on Sep 27, 2026

   strix-windows: derive the Flash-Next launcher's repo from its own loc…

   ...

   6627322
10. AzeezIsh commented on Sep 28, 2026

    

    AzeezIsh

    on Sep 28, 2026

    ContributorAuthor

    More actions

    Status 2026-09-28. #80 is the whole port and now ships as its own per-deployment PR; the mixed trains #112/#129 are closed.

    • e37ab5a0e on Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80 declares wy9..16 expected_absent on the strix-hip Flash-Next target. Without it, the boot gate refused the serve with 8 unresolved lookups.
    • The Windows Flash-Next agentic test on the Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80 build passed: 10/10 runs, 10/10 followed directions, 0 faults, at 32K with long-context speculation on (details on Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80).
    • Remaining gates: the Windows serial and MTP legs, after the 27B leg now building on winbox; a GB10 boot check (reiner 625zzz2); a Strix boot check (055d).
    • This job closes when Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80 merges.
11. AzeezIsh commented on Oct 1, 2026

    

    AzeezIsh

    on Oct 1, 2026

    ContributorAuthor

    More actions

    Update 2026-10-01 ~07:40Z.

    • Perf leg on the landing tree: ~440/1007 turns, 0 faults, host free memory steady at 9.4 GB, MTP on every request. ETA ~21:00Z.
    • Where its time goes: last 400 turns, TTFT median 64.5 s / p90 112 s vs decode 3.9 s (14.3 tok/s). That's 92% re-prefill, because the recipe has prefix caching off.
    • Launcher commits are pushed as a draft stacked on Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80: strix-windows: Flash-Next launchers + prefix caching on by default (stacked on #80) #149, which adds prefix caching on by default (4 Marconi slots, fits by shrinking KV from ~97k to ~77k tokens) and a fingerprint fix. Gate it after the current leg.
    • Accuracy bar: analysis on Strix Halo Windows: Flash-Next port on main + MoE prefill transpose (2k TTFT ~2x) (supersedes #54) #80. The normalized −0.63 is 2 irrelevance items. Recommend taking 84.22 / 83.61 as the record.
    • Teardown double frees ([bug] Flash-Next NVFP4: shutdown exits before teardown finishes, and teardown hits ~147k invalid frees #122): fixed in draft teardown: free SSM pool bases, not layer views; stop double-freeing store tensors after the MoE transpose (#122) #148. Flash-Next's ~147k were the MoE transpose plus WeightStore.
12. AzeezIsh commented on Oct 2, 2026

    

    AzeezIsh

    on Oct 2, 2026

    ContributorAuthor

    More actions

    Flash-Next decode attribution is on #143. One MTP verify cycle is 111 ms of GPU time and GPU-bound. Two bit-identical fixes are up as drafts: #153 (LM head read once for both verify tokens, −5.9 ms) and #154 (token-fused mHC collapse, up to −21 ms; bench in strix job 144). Both still need the AMD gates before merge.
13. AzeezIsh commented on Oct 2, 2026

    

    AzeezIsh

    on Oct 2, 2026

    ContributorAuthor

    More actions

    Done: the Strix Halo Windows Flash-Next port is on main (#80, merged as 693c9eee8).

    What's in it:

    • serve-amd.ps1 -Model qwen3.8-flash-next serves Flash-Next with MTP by default. It verifies past the QSA bound, so MTP stays on for long prompts.
    • -Spec dflash2 covers the 27B with DFlash2. Flash-Next + DFlash2 is refused at launch.
    • -Thinking default|off|on works on all modes.
    • build-amd.ps1 builds every strix-hip target in one binary.

    Evidence: listed on #80.

    • Landing-tree ST-995 84.22 / 83.61, the Windows Flash-Next record.
    • 1007-turn replay 1007/1007, 0 failed, 0 missing.
    • -Thinking smoke on all modes, and the DFlash2 poisoning gate.
    • A post-merge byte-identity check against main 428c0f957.

    Follow-ups, each tracked in its own PR or issue:

    • strix-windows: Flash-Next launchers + prefix caching on by default (stacked on #80) #149: prefix caching on by default, for the 92%-of-wall TTFT. Its gates are running on the merged tree.
    • qsa: O(complete) radix-select top-k on the device by default (Flash-Next decode past the 2051-token bound) #152 / lm_head: one-pass BF16 batch2 LM head for K=2 verify on HIP (Flash-Next, −5.9 ms/verify step) #153 / qwen4_exp mHC: token-fused decode collapse, bit-identical (Flash-Next verify, up to −20 ms/step target) #154: decode speed (QSA device top-k, one-pass LM head, token-fused mHC). These are gated as one "decode train" on winbox.
    • [job] Qwen3.8-Flash-Next on Strix Halo: decode speed (serial and MTP) #143 / [job] Qwen3.8-Flash-Next on Strix Halo: prefill and TTFT #144 / [job] Strix Halo: multi-session serving (parallel or sequential) and long context, Flash-Next and the 27B #145: Flash-Next decode, prefill and concurrency.

    Closing this job.
14. AzeezIsh

    closed this as completedon Oct 2, 2026