[job] Qwen3.8-27B on Strix Halo: prefill 2.4x gap + DP4A verify tiers for K=5..9

@AzeezIsh

Status (updated 2026-10-28 05:30Z)

State: in progress. strix (gfx1151, 58 GB), 27B NVFP4, on main 8863dc5e5. Measured this week; the tables are in the comments from 2026-10-27/28.

Accuracy (ST-995):

DFlash2 γ8: 84.12/74.65.
DFlash2 γ4: 83.52/73.31, reproduced exactly on main. The drop is systematic.
MTP K=4: 83.92/74.75.

Agentic (50/50 on all three):

DFlash2 γ4: Σwall 8,835 s.
DFlash2 γ8: 9,379 s.
MTP K=4: 9,926 s.

Speed:

Prose tg: γ4 12.4–13.5, MTP 13.2–13.4, AR 9.9–10.5, γ8 7.6–8.0 (γ8 over-speculates on prose).
pp is ~240 tok/s at depth 0, against Gufo's ~635.
Prefill is GEMM-bound at ~14 TF/s effective (s22).

Default: stays DFlash2 γ8. γ4 is a speed option at −1.3 normalized ST-995; that's the owner's call.

Concurrency: AR scales to C=4 at 25.8 tok/s steady, then drops at C=8 on this memory-tight box. MTP stops scaling after C=2.

Queued: merged-main gates for γ8 (s35 ST-995, s36 agentic) and MTP (s37, s38).

Next lever: prefill GEMM rate (w4a16 t_m128_f16a / w8a16 at ~14 TF/s, against ~30 TF/s possible with WMMA).

Kind

Feature for a model we already support

Model

nvidia/Qwen3.8-27B-NVFP4 (Strix Halo gfx1151)

What's the job?

Strix prefill is ~2.4x behind pwilkin's llama.cpp stack (104 vs 256.8 tok/s at 31.5k) and γ=8 DFlash2 verify runs 255 ms at K=9 because rows 5..9 land on the float w4a16_gemv_batch8/16 tiers, not DP4A. Steps: stage-6 prefill matrix (ATLAS_BF16_TC_PREFILL / FP8_M64 / NO_FFN_NVFP4_MMQ); MMQ for GDN/attn prefill; fused gate+up dual-accumulator GEMM; DP4A verify tiers for K=5..9 (check VGPRs first). Done = prefill ≥ 180 tok/s at 31.5k and γ=8 verify ≤ 150 ms, ST-995 within noise.

Impact radius

Shared code, other models could change

What else could it touch?

strix-hip kernels only; the W4A8 DP4A path changes numerics (5.58 % mean rel. error on the decode GEMV) so KL + ST-995 must re-gate.

Who's waiting or willing to test?

@AzeezIsh (GB10)

Stage-6 prefill matrix (strix, 2026-09-25) — no kernel-tier flag beats the default

main66 (5b07b953) on gfx1151, nvidia/Qwen3.8-27B-NVFP4, serve-amd.sh defaults (prefill chunk 2048), max-seq 34000. Prefill tok/s = prompt tokens / TTFT, median of 3, greedy hash identical in every cell (565339bc).

arm	~1.9k	~7.2k	~29k
default	233	202	139.8
`ATLAS_BF16_TC_PREFILL=1`	210	187	140.7
`ATLAS_FP8_M64_PREFILL=1`	176	159	123
`ATLAS_NO_FFN_NVFP4_MMQ=1`	205	192	140.6

Main has moved a lot since 09-13 (131.7 tok/s at ~2k then, 233 now). The gap to pwilkin (256.8 at 31.5k) is 1.8x, down from 2.4x.
The GEMM tier is not where the long-context gap is: every arm converges at ~140 at 29k, and throughput falls with length (233 → 140), which points at attention / GDN cost growing with context.
Next (running): rocprofv3 kernel ranking of one ~7k prefill, sliced to the request window, plus the prefill-chunk knob (2048 vs 8192) at ~29k.

Kernel ranking: Strix long prefill is GEMM-efficiency-bound (rocprofv3, 2026-09-25)

main66 on gfx1151, nvidia 27B, one 7,235-token prefill (chunk 2048), trace sliced to the request window: 33.0 s busy of a 33.17 s span (99.5 %).

kernel	share	calls	what	derived rate
`w4a16_gemm_t_m128`	47.2 % (15.59 s)	768	NVFP4 FFN gate/up/down, 64 layers × 3 × 4 chunks	2·3·5120·17408·7235·64 ≈ 248 TFLOP → ~16 TFLOP/s
`w8a16_gemm_n_m128`	29.5 % (9.74 s)	640	the pack's FP8 GDN / attention projections	~143 TFLOP (non-FFN params) → ~15 TFLOP/s
`inferspark_prefill_paged_64`	6.8 %	48	full attention
GDN (`chunk_delta_h_vfused` + `recompute_wu` + `chunk_fwd_o`)	8.2 %	576
everything else	< 8 %		norms, rope, conv, copies

The long-prefill gap is GEMM efficiency, not attention/GDN (at 7k; attention's share will grow toward 29k). Both GEMMs run at ~15-16 TFLOP/s, ~27 % of a ~59 TFLOP/s gfx1151 WMMA peak (estimate: 40 CU × 512 FLOP/clk × 2.9 GHz).
The stage-6 flags only move the FFN tier; nothing touched the FP8 projection GEMM (29.5 %).
Levers, in order: (1) the FP8 w8a16_gemm_n_m128 — the nvidia pack's GDN/attn projections are FP8; an in-SMEM FP8→BF16 dequant + WMMA with deeper pipelining, or requant them to NVFP4 (the fp8to4 checkpoint already exists and passed ST-995); (2) fused gate+up with dual accumulators (pwilkin mmb.cu) so the FFN's activation tile is read once and gate/up are never written to global; (3) then re-profile at 29k where attention grows.
Chunk-size knob (2048 vs 8192 at ~29k) is running now.

Strix prefill, part 2 (main66, gfx1151, 2026-09-25):

The chunk-size knob is not the long-context lever. At 29k: --max-prefill-tokens 2048 gives 144.6 / 138.5 tok/s; 8192 gives 139.6 / 137.2. That is the same within noise.

fp8to4 checkpoint profile (the nvidia-derived conversion; the FP8 weights become W4A16), 7.2k prompt, measured under rocprofv3: 212.4 tok/s. The nvidia pack's unprofiled default is 202 tok/s.

share	time	calls	kernel
69.6 %	23.58 s	1216	`w4a16_gemm_t_m128`
7.2 %	2.45 s	48	`inferspark_prefill_paged_64`
5.4 %	1.83 s	192	`w4a16_gemm_t`
4.1 %	1.38 s	192	`gated_delta_rule_chunk_delta_h_vfused`
3.2 %	1.10 s	45	`dense_gemm_bf16`
2.7 %	0.90 s	192	`gated_delta_rule_recompute_wu`
2.4 %	0.80 s	192	`gated_delta_rule_chunk_fwd_o`

(Profiler window: busy 33.88 s of 34.06 s.)

What this means. On both packs, prefill is one GEMM family. On the nvidia pack it is the W4A16 + W8A16 pair (47 % + 30 %); on fp8to4 it is a single kernel, w4a16_gemm_t_m128, at 70 %. That kernel runs at ~27 % of the estimated WMMA peak.

The only big Strix prefill lever is that GEMM's tile efficiency. Neither attention nor GDN comes to more than 7 %.

Next: occupancy and LDS analysis of w4a16_gemm_t_m128: tile shape, dequant placement, and the WMMA issue rate.

w4a16_gemm_t_m128: LDS bank conflicts removed, and throughput-neutral (draft #79).

Fix: padding smem_B_bf16 per 16 rows.
Counters: conflict cycles fell 88 % (36.4 % → 6.7 % of LDS-active cycles), LDS-active cycles −32 %.
Output: greedy byte-identical across 4 interleaved arms.
Prefill: unchanged at 1.9k / 7.2k / 29k (within ±2 % run-to-run drift).

So the kernel is not LDS-bound. The conflicts were real, but they were hidden behind something else.

Now running: a speed-of-light ablation on strix (timing only; the output is deliberately wrong).

X1 drops the per-element FP8 scale bit-decode.
X2 also drops the E2M1 LUT reads.
All loads are kept, so memory traffic is identical.
If either moves prefill tok/s, the in-register NVFP4 dequant is the limiter, and the fix is to hoist or share the scale decode (it runs ~8x redundantly across threads) or to pre-dequant.
If neither moves it, the limiter is WMMA issue or latency, and the lever is occupancy: ~41 KB LDS per CTA with 4 waves gives about 1 CTA per CU.

First Strix Halo 27B prefill win landed as a PR: #79 (ready for review).

Change: an exact, cheaper NVFP4 dequant in w4a16_gemm_t_m128.
Kernel: −17 % GPU time.
Prefill: +5.7 % at 1.9k, +4.7 % at 7.2k, medians of interleaved runs.
Output: greedy byte-identical in every arm.
Memory: no extra.

What is left in this kernel. The timing-only ceiling with a trivial scale is −26 % kernel time. The remaining gap is the exact E4M3 scale decode, which runs ~8x redundantly across the threads that share a scale group.

A cross-lane share (v_permlane16/x16) would cut that redundancy with no memory cost.
A precomputed fp16 scale (every E4M3 value is exact in fp16) would cut it at +0.125 B/weight.

Measurement note for anyone benchmarking on this box. Back-to-back 7k prefills throttle about 10 % (e.g. 203.7 → 189.8 → 183.8 tok/s within one server), so single sequential A/B runs are ±5 %. Interleave arms and compare medians by request position.

Tried and parked: sharing the exact E4M3 scale decode across lanes in w4a16_gemm_t_m128 (on top of #79 @ fd2fbeb; ISA only, not run).

Why it looked worth doing: in M128_STORE_TILE, lanes j, j+8, j+16, j+24 of each wave hold the same 16 scale bytes. Each of them decodes all 16 with e4m3_scale_half, so the decode is 4x redundant inside a wave and 8x across the CTA. The X3 ablation (trivial scale) puts the exact decode at about 7-9 % of kernel time.

The design:

Each lane decodes one column quarter.
It fetches the other three through row_xmask:8 DPP and two v_permlanex16.
Values are moved bit-for-bit, so the output is bit-identical by construction.

ISA, TheRock hipcc 10.0.0, -O3 -ffp-contract=off --offload-arch=gfx1151:

`w4a16_gemm_t_m128`	#79	scale-share
VALU (`v_*`)	2819	2510 (-11 %)
`v_permlanex16` / DPP	0 / 0	16 / 8
next_free_vgpr	256	256
spills / reloads	8 / 1	39 / 30
private segment	36 B	652 B
LDS	41216	41216

The kernel already sits at the 256-VGPR cap. Keeping the four shared scales and the per-lane column permutation live pushes it into real scratch traffic in the tile loop. So this lever needs VGPR headroom first, for example a smaller A/B register-prefetch footprint, before it can pay off. Not pursued further for now. Next on this kernel family: the w8a16_gemm_n_m128 speed-of-light ablation (30 % of the 7.2k prefill).

Triage (2026-09-26). Strix 27B prefill is trending the right way:

~2k: 131.7 → 233 tok/s since 09-13;
29k: ~140 on main66, against pwilkin's 256.8 (1.8x behind, down from 2.4x).

Ready to merge, all byte-identical and measured:

kernels(strix-hip): exact, cheaper NVFP4 dequant in w4a16_gemm_t_m128 — kernel −17 %, 27B prefill +5 %, bit-identical #79: NVFP4 dequant, kernel −17 %, prefill +5 %;
kernels(strix-hip): w8a16_gemm_n_m128 GROUP_M=8 CTA order + exact E4M3 decode — FP8 prefill GEMM −30 %, 27B prefill +6-13 %, bit-identical #94: FP8 GEMM swizzle, −30 % kernel, +6–13 % prefill;
qwen3_ssm + qwen3_attention (gfx1151): skip the transposed FP8 copies w8a16_gemm_n_m128 never reads (−7.2 GB on the 27B) #93: −5.5 GB;
strix (gfx1151): DP4A batch8 GEMVs for DFlash2 γ8 verify rows 5..8 (+8.5 % decode, output identical) + FP8 batch8 tier #101: DP4A batch8 verify, +8.5 % decode.

After those land, re-measure both done-bar numbers on one tree: prefill at ~29–31.5k (bar ≥ 180 tok/s) and γ8 verify ms per step (bar ≤ 150 ms, from 255).

Scale back:

The scale-share lever is correctly parked (VGPR cap).
strix (gfx1151): vl2 verify GEMVs — two logical lanes per thread for the γ8 batch8 tiers #105 (vl2) fails a dispatch test and has no measured delta yet. If its A/B is neutral, close it here.

Progress on the γ8 verify side (Strix Linux, gfx1151, 2026-09-26/27)

Where the time goes. One DFlash2 γ8 step on nvidia/Qwen3.8-27B-NVFP4 takes about 280 ms of GPU time. That's the rocprofv3 OFF arm of strix job 038d, MinHeap code prompt, 512 tokens, mean_na 5.66.

kernel	share of step
`w8a16_gemv_batch8` (FP8 GDN qkvz/out_proj + attn QKV)	28 %
`w4a16_gemv_dp4a_dual_batch8` (FFN gate+up)	28 %
drafter `dense_gemv_bf16_batchm` (BF16)	16 %
`w4a16_gemv_dp4a_batch8` (FFN down)	7 %
lm_head `w4a16_gemv_batch8`	7 % (9.3 ms/call, ~67 GB/s)

On this prompt, DFlash γ8 decodes about as fast as MTP K=4, ~24 tok/s each, so the verify GEMVs are the whole game.

PRs opened from this:

dflash_head: NVFP4 batch8 tier for the DFlash2 drafter's projection GEMMs at γ 5..8 #108, drafter NVFP4 batch8 tier. The 16 % drafter BF16 block moves to NVFP4. Measured −9 % GPU time and +10 % decode tok/s. The DFlash output sha changes, because speculation isn't output-neutral: the MTP reference stays fixed, but acceptance moves. An ST-995 γ8 leg on this build is queued before it can leave draft.
strix (gfx1151): vl2 verify GEMVs — two logical lanes per thread for the γ8 batch8 tiers #105 (stacked on strix (gfx1151): DP4A batch8 GEMVs for DFlash2 γ8 verify rows 5..8 (+8.5 % decode, output identical) + FP8 batch8 tier #101), vl2 runtime-M twins for the dual DP4A (−30 % per call) and w8a16 batch8 (−11 %). Output is bit-identical. The down projection stays vl1, since it's already at ~163 GB/s. The final A/B is queued.
qwen3_ssm + qwen3_attention (gfx1151): skip the transposed FP8 copies w8a16_gemm_n_m128 never reads (−7.2 GB on the 27B) #93, skip the dead transposed FP8 copies. Frees 7.2 GB, and prompts ≤128 tokens aren't slower (PR comment). This also turns out to be what makes γ8 fit on the 64 GB Linux box at 32k context.

Where that leaves the step:

GPU time per step is about 262 ms with vl2-dyn alone. With dflash_head: NVFP4 batch8 tier for the DFlash2 drafter's projection GEMMs at γ 5..8 #108, the estimate is about 236 ms.
The 150 ms target needs more. The next candidates:
- the w8a16 batch8 GEMV, still at ~75 GB/s effective;
- the lm_head batch8 GEMV (kernels/gb10/common/w4a16_gemv.cu, shared with gb10, so it needs a strix shadow);
- a vl2-style variant of the float w4a16_gemv_batch8, which the drafter now uses too.

Also this session:

The combined Strix build (carry + qwen3_ssm + qwen3_attention (gfx1151): skip the transposed FP8 copies w8a16_gemm_n_m128 never reads (−7.2 GB on the 27B) #93 + kernels(strix-hip): w8a16_gemm_n_m128 GROUP_M=8 CTA order + exact E4M3 decode — FP8 prefill GEMM −30 %, 27B prefill +6-13 %, bit-identical #94 + strix (gfx1151): DP4A batch8 GEMVs for DFlash2 γ8 verify rows 5..8 (+8.5 % decode, output identical) + FP8 batch8 tier #101) completed a 1007-turn agentic leg: 1007/1007, 0 failed, score 0.6197, 11,422 s, 0 KV / HIP / panic lines.
kernels(strix-hip): declare the K=9..16 WY verify lookups expected-absent on qwen3.8-27b #107 fixes current main's Windows kernel-gate failure: wy9..wy16 weren't declared absent on strix-hip.

Next levers for w4a16_gemm_t_m128 (Qwen3.8-Max consult, ranked, checked against the numbers in this thread).

Where it stands: about 16 TFLOP/s, roughly 27 % of an estimated gfx1151 WMMA peak, at the 256-VGPR cap with 8 spills.

kernels(strix-hip): exact, cheaper NVFP4 dequant in w4a16_gemm_t_m128 — kernel −17 %, 27B prefill +5 %, bit-identical #79's exact decode took −17 %.
The timing-only X3 ablation (trivial scale) bounds what's left in the scale decode at about −26 % kernel time.
The cross-lane scale share was parked at 39 spills.

#	lever	expected	cheap verification
1	GROUP_M CTA swizzle, same as #94 did for `w8a16_gemm_n_m128` (−30 % there, B-load bound)	0–15 %, and the cheapest thing to try	rasterisation change only; kernel time + `TCC_HIT`/miss; keep if > 3–5 %
2	Precomputed fp16 scales (every E4M3 value is exact in fp16), turning the decode into a load	10–15 %, capped by X3's −26 %	repack one layer and diff `SQ_INSTS_VALU`
3	Shrink the register prefetch to zero spills (fewer in-flight B fragments / shorter prefetch distance)	5–15 %	require 0 scratch loads/stores in the ISA; restore a stage if latency-bound
4	2 CTAs/CU with a smaller tile	0–10 %, conditional	only if LDS ≤ 32 KB and VGPRs fit
5	split-phase dequant into LDS	< 5 %	skip unless 3 fails
—	hipBLASLt on a bf16 copy	rejected	doesn't fit the 58 GB box

Cost check on #2: +0.125 B/weight on the dense FFN (5120 × 17408 × 3 × 64 ≈ 17.1 G weights) is about +2.1 GB. That only fits because #93 freed 5.5 GB on the same box, so budget for it explicitly against the DFlash2 γ8 recipe's headroom.

Suggested order: 1 → 3 → 2. Swizzle and spill removal cost no memory, and a fresh rocprof VGPR/spill readout comes before wiring anything, as with the reuse2 lesson.

Status 2026-09-28. The Strix 27B PRs ride the per-deployment HIP train #131: #79 (NVFP4 dequant, prefill +5%), #93 (−7.2 GB), #94 (FP8 prefill GEMM, prefill +6–13%), #101 (DP4A batch8 verify, decode +8.5%), #107 (wy9..16 manifest), #124 (C≥2 crash) and #123 (shutdown abort).

The Strix gate is 055c-hip-train-gate, running: agentic 1007 turns, 2 and 4 concurrent, ST-995, and clean shutdown.
strix (gfx1151): vl2 verify GEMVs — two logical lanes per thread for the γ8 batch8 tiers #105 (vl2 GEMVs) is stacked on HIP train (Strix Linux + Windows): #79 #93 #94 #101 #104 #107 #123 #124 #131. Its OFF/ON A/B is Strix 059/060, and it closes if the result is neutral.
spark-model: carry the DFlash2 drafter ctx across turns (Strix-validated ctx-carry stack onto main) #104 matters here too. Without it, main can't size memory for the 27B DFlash2 agentic recipe at 32K on Strix (054zzz1 died building the model, and the agentic serve got 0.8 GB of KV). Its GB10 crash fix (1dd752f54) is being verified in 625zzz4. Once it passes, it follows HIP train (Strix Linux + Windows): #79 #93 #94 #101 #104 #107 #123 #124 #131.

Kernel-level root cause of the gfx1151 NVFP4 prefill gap. Static ISA analysis is done; a GPU microbench is queued.

Method. Upstream clang 23 targeting gfx1151, with Atlas's HIP flags and ROCm 7.13 headers. The production sources compile unmodified (plus a one-line bit-cast shim for upstream clang's bf16 WMMA builtin signature). I counted the hot-loop instruction mix from the loop blocks.

Hardware facts (compiled probes):

f32→bf16 has no instruction on gfx1151: software RNE is 5 VALU.
f32→f16 is 1 VALU.
v_pk_mul_f16 exists; there is no packed bf16 math.
WMMA runs on the SIMD VALUs (~32 cycles per 16×16×16), so every dequant instruction displaces matrix issue.

Per-32-K-step hot loop:

kernel	instr	WMMA	instr/WMMA	sw bf16 rounds
`w4a16_gemm_t_m128` (dense prefill; 46% of 27B prefill per job 077)	829	32	26	32
`w8a16_gemm_n_m128` (FP8 layers; 26% per job 077)	772	16	48	16
`moe_w4a16_fused_gate_up_t` / `…ptrtable_t` (Flash-Next routed experts)	~687	16	43	32
F16 twin of m128 (new)	335	32	10.5	0
F16 twins of the MoE kernels (new)	267	16	16.7	0

Fix: F16 operands with an exact dequant.

f16 of every E2M1 value has a zero low byte, so one v_perm per four codes converts them.
The E4M3 scale goes to f16 via bits<<7 × 256, then one v_pk_mul_f16 multiplies two weights at a time.
Exact: proven exhaustively, 0 mismatches over all 254 non-NaN scale bytes × 16 codes, subnormals included. Today's kernels round every scaled weight to bf16.
The fp32 tensor/expert scale moves to the epilogue. Activations are converted bf16→f16 once per GEMM input.
The MoE twins also get the m128 register-prefetch / double-buffer pipeline. The MoE kernels were an older port: LDS lookup table, per-element operand building, two barriers per step.

Prediction: ~1.3–1.5× on the dense GEMM (≈25–33 TF/s from 19–24); more on the MoE kernels.

Qwen's adversarial review (overflow, denormals, possibly f16-rounded WMMA products) is covered by edge cases in the bench.
Activation overflow above 65504 would need a per-row power-of-two pre-scale. Real activation amax gets measured before wiring.

Validation: strix job 144 and a winbox slot after #80's ST-995.

Production kernels compiled verbatim beside the twins.
An FP64 host reference with exact weights.
27B and Flash-Next dense shapes at M=2048, plus Flash-Next routed MoE (512 experts, top-10).
Engine wiring and the AGENTS.md gates follow only if the bench shows the speedup and the accuracy.

Full write-up: atlas-audit 14_LOG_2026-10-02.md §12.

Prefill attention on gfx1151: quantified lead (static ISA, inferspark_prefill_paged via prefill_paged_compute.cuh; attention is 8.5% of 27B prefill at 8k and 33% at 29k).

Hot loop: 2,730 instructions per KV tile for 48 WMMA (57/WMMA). The top ops are 322 s_waitcnt, 288 v_cvt_f16_f32, 262 v_lshlrev, and 256 ds_load_u16.
Cause: P is already F16 (atlas-core: treat W4A16_NVFP4 as NVFP4 in quant detection (#88) #90), but the P·V V operand is built one element at a time: bb[k] = (__fp16)(float)smem_K[k_off + k][d_col]. That's a column-strided 2-byte LDS read plus a bf16→f32→f16 conversion per element, repeated for 2 K16 steps × 8 N-tiles in every warp. That accounts for exactly the 256 ds_load_u16 per lane, and roughly 700 instructions per tile (~25%).
Fix: stage V once per tile as F16, transposed (k-contiguous, 80-byte row stride so fragments are two ds_load_b128). That's ~64 LDS writes per thread vs 256 reads + 288 cvt.
Constraint: a transposed F16 V tile is ~20 KB of LDS. The kernel was tuned under 32 KB for 2 workgroups/CU (the occ2 K/V buffer-share), so this needs an aliasing design (e.g. with smem_S / Q after QK) or an occupancy A/B.
Not started; queued behind the F16 GEMM twins (job 144). Details in atlas-audit log §12.7.

The F16-operand GEMM twins discussed above are now up as draft #155: dense m128, routed MoE and w8a16 FP8, plus engine wiring. 914/914 tests pass. It isn't merge-ready until the GPU bench (strix job 144, also built on winbox) shows the speedup and accuracy, and then the AMD gates.

@14k

[amd-campaign · 2026-10-29 17:35Z · merged #259 → main c998a49] 27B prefill attention (regkv + V f16 + all-thread softmax), bit-identical: DevLab desktop +1.6% @7.4k / +3.0% @12.5k (4/4 rounds), strix +2..3% @14k (3/3), kernel −21% @2k / −23% @20k. Also merged #258 (DevLab ab_prefill default GPU_UTIL 0.88). Post-merge: s123 (strix), f95 (framework, after its agentic queue), DevLab bench 40922052191 (new main vs e9491be). s124 profiles 27B @~18k on new main to rank the next lever.

Thanks for letting me know, praise be 🙏

@18k

[amd-campaign s125/s126 · 2026-10-29 18:40Z · strix · GDN serial spine (shared by 27B + Flash-Next)] gated_delta_rule_chunk_delta_h_vfused (27B 5.7% @18k, Flash-Next 7.1% @6.6k), kernel harness at 27B geometry (48 v-heads, one 2048-token chunk):

Splitting the v columns over 2×/4× CTAs (bit-identical): 6.3 → 14.3 / 25.8 ms, slower. Inference: the per-chunk W/U/K LDS staging, one bf16 per thread per iteration, is the critical path, so fewer threads per CTA made it worse.
16-byte staging + decay expf across threads (bit-identical, hash equal): 6.0–6.7 → 3.8–3.9 ms (−40%), 3/3 rounds. Branch perf/hip-gdn-vec-stage; e2e A/B s127 (strix 27B) + DevLab bench 40931998079 running.
Also: post-merge DevLab bench of main c998a49 vs e9491be (run 40926928199): probes IDENTICAL, 12.5k +1.3–2%. Flash-Next agentic base f92: 48/50.

[amd-campaign · 2026-10-29 19:35Z · merged #260 → main a12bffa] GDN chunk_delta_h vector LDS staging, bit-identical: 27B prefill +2.2/+2.0/+1.6% @1.9k/7.4k/12.5k (DevLab desktop 4/4 rounds), kernel −40%. Post-merge: s128 (strix 27B); f96 (framework) checks Flash-Next on the shared kernel, e9491be vs a12bffa, probes must be IDENTICAL.

[amd-campaign · 2026-10-29 21:35Z · merged #261 → main ec054ef] GDN train 2 (recompute_wu VGPR row + 16-byte staging in recompute_wu/chunk_fwd_o), bit-identical: 27B prefill +1.4% @7.4k, +1.2% @12.5k (DevLab 4/4); kernels wu −38%, fwd_o −6%. Merged on main tonight, all bit-identical: #259 attention, #260 + #261 GDN. Post-merge: s131 (strix); f97 (framework) checks Flash-Next identity on the shared kernels.

[amd-campaign s138/s140 · 2026-10-30 14:00Z · strix (Z13 GZ302EA, Ryzen AI Max+ 395, 58 GB) · first same-box head-to-head vs Gufo, 27B]
Same client (gufo_bench.py, Gufo's pp2048/tg128 greedy thinking-off protocol, mixed prompts), same box, one engine at a time. Atlas main 1585c38 NVFP4 (serve-amd recipe + prefix caching) vs Gufo 72dc1a0 (its nix/cachix gfx1151 build in a container) Q4_K_XL (+ DFlash2 Q4_K_M draft), --sessions 8 --context 12288.

tok/s	Atlas MTP	Gufo DFlash2	Atlas AR	Gufo AR
C1 pp, depth 0	361	333	286	251
C1 tg, depth 0	14.8	9.3	10.0	7.8
C1 pp, 8k cached + 2k new	212	78	200	98
C1 tg, depth 8k	11.7	7.4	8.3	9.0
C4 steady aggregate	20.1	29.5	26.1	28.3
C8 steady aggregate	17.4	42.5	20.0	40.8

Reading (inferences labelled):

On the same box, Atlas leads single-user prefill, decode and cached-depth prefill. Gufo leads 2–2.4× at C8: concurrency scaling is the real gap. Atlas C8 < C4 in both modes.
Gufo's published 649 pp / 71.6 tg (27B) are from a 128 GB desktop; on this power-limited 58 GB tablet Gufo itself measures ~half of that, so published-vs-published comparisons overstate the gap.
Quants differ (Q4_K_XL vs NVFP4); an accuracy leg is still to do. Gufo ran with --context 12288 to fit 58 GB.

[amd-campaign f118 · 2026-10-30 18:40Z · framework (desktop Strix Halo) · main cd9f71a · 27B NVFP4, Gufo protocol] Desktop counterpart of the strix same-box h2h (s138/s140), serve-amd recipe + prefix caching, MAX_BATCH 8:

tok/s	Atlas MTP	Atlas AR
C1 pp depth 0 / 8k cached	488 / 366	501 / 369
C1 tg depth 0 / 8k	17.8 / 16.2	11.8 / 10.9
C4 steady aggregate	33.2	39.5
C8 steady aggregate	25.0	34.4

Gufo's published desktop 27B numbers (649 pp, up to 71.6 tg DFlash2, 128 C8) are not same-box; Gufo can't run on this box yet (~10 GB disk free). The shape matches strix: aggregate falls from C4 to C8 (MTP 33.2 → 25.0). #277 (merged → main 48f73b2): serve-amd keeps Flash-Next's GDN BF16 above MAX_BATCH 1, restoring C>1 KV room after #275.

[amd-campaign f119 · 2026-10-30 20:05Z · framework · main cd9f71a · 27B C8 AR decode, rocprofv3] 8 concurrent 256-token decodes: 2,048 tokens in 86.6 s, GPU 97% busy. Unlike Flash-Next, the 27B batches, but at M=8 the MLP of the 48 GDN layers runs on the prefill tile GEMMs, grouped by launch grid:

kernel @ decode	N	launches	per step	time
`w4a16_gemm_t`	17408 (gate/up)	24,480	96	17.6 s (20.9%)
`w4a16_gemm_t_k64`	5120 (down, K=17408)	12,240	48	12.4 s (14.7%)
`w4a16_gemv_dp4a_dual_batch8_d4_dyn_vl2`	—	4,080	16	2.8 s
So the 16 attention-layer MLPs take the batched-GEMV decode path while the 48 GDN-layer MLPs fall through to the tile GEMM at M=8. That is the "M>4 projection cliff" the GDN QKVZ arm already avoids.
Estimate (inference): ~117 ms/step on those tile GEMMs vs ~32 ms to stream the ~6.4 GB of MLP weights at ~200 GB/s, so ~25% on C8 decode if the GDN-layer FFN at M 5..8 takes the same batched GEMV path. That would also explain C8 < C4 (at M≤4 it likely already does). Numerics change (different kernel), so it needs ST-995. Implementing next.

[amd-campaign f120 · 2026-10-30 20:47Z · framework · main cd9f71a · 27B AR C>1, ATLAS_SSM_FFN_PREFILL_MIN_N] The M=8 tile-GEMM fallthrough from f119 is deliberate: ssm_ffn_prefill_min_n() defaults to 5 because the tile GEMM beat the batch-8 GEMV from n=5 on GB10. On gfx1151 it is the other way round (Gufo protocol, 2 interleaved rounds):

aggregate tok/s	C1	C4	C8
MIN_N=5 (default)	11.9 / 11.9	39.3 / 39.3	34.4 / 34.4
MIN_N=9	11.9 / 11.9	39.4 / 39.3	49.2 / 49.3
C8 +43%, and C8 now exceeds C4. Branch `perf/hip-ssm-ffn-minn9` makes 9 the `atlas_hip` default (5 elsewhere; env still overrides). f121 verifies the default takes and checks concurrency correctness (8 prompts solo vs concurrent, both settings) before the PR.

[amd-campaign s143 + f121 · 2026-10-30 21:15Z]

27B h2h accuracy leg (strix s143): Gufo 72dc1a0 Q4_K_XL AR on the ST-995 leg (bfcl-subset n=995, seed 42, temp 0, Atlas's runner as client): overall 84.22 / normalized 81.12 (hallucination 81.44, live 81.18, non_live 80.75). Atlas 27B NVFP4 on the same leg: 84.52 / 75.36 (s83). Overall is level; Gufo is +5.8 normalized (category-balanced). Record: run-1793394666667340468.json on strix.
perf/hip-ssm-ffn-minn9 (f121, framework): default takes (C8 49.3 vs 34.4 tok/s). Concurrency identity, 8 prompts greedy 64 tok solo vs all-at-once: 6/8 rows identical with MIN_N 9 vs 7/8 with MIN_N 5; every divergence starts after ~50 chars, and one row diverges under both (pre-existing batched-decode numerics). Because ST-995 runs sequentially and never exercises this path, f122 runs 8 ST-995 clients concurrently against one server as the gate.

[amd-campaign s144 vs s143 · 2026-10-31 03:00Z · strix · ST-995 per category, Atlas 27B NVFP4 vs Gufo Q4_K_XL] Paired sequential baseline on the current build (perf/hip-ssm-ffn-minn9 e321494, C1, serve-amd recipe): overall 84.12 / normalized 74.95 (run-1793414691707022897). Against Gufo (s143, 84.22 / 81.12):

category / subset	Atlas	Gufo
live	87.06	81.18
non_live	87.80	80.75
hallucination	50.00	81.44
↳ live_irrelevance	37.50	92.05
↳ irrelevance	62.50	70.83
Atlas is better at making the right call and much worse at declining to call a tool. The whole normalized gap is the irrelevance subsets.
Hypothesis (unverified): the serving recipe keeps XGrammar tool-call constrained decoding on (`qwen3_coder` parser, "constrained decoding ENABLED for tool"), which may push the model into a call when it should abstain. s145 reruns the identical leg with `--disable-tool-grammar`.

[amd-campaign s146 · 2026-10-31 07:40Z · strix · ST-995 hallucination gap: tool grammar ruled out] Same build, same recipe as s144, plus --disable-tool-grammar true (XGrammar off on tool_choice=auto): identical scores, overall 84.12 / normalized 74.95, live_irrelevance 37.50, irrelevance 62.50, every subset the same. So constrained decoding is not why Atlas calls a tool where Gufo (92.05 / 70.83) declines. Next candidates: the chat template / tool-prompt rendering for qwen3.8-27b (templates are picked by model_type), or the model's own output on these prompts. Check: diff Atlas's rendered prompt for a live_irrelevance item against the checkpoint's own template.

[amd-campaign s156 · 2026-10-31 11:18Z · strix · 27B same-box h2h re-run after #278 (merged) + #281 ladder (env)] Gufo protocol, prefix caching:

tok/s	Atlas MTP	Gufo DFlash2 (s140)	Atlas AR	Gufo AR
C1 pp d0 / 8k cached	380 / 213	333 / 78	278 / 198	251 / 98
C1 tg	15.2	9.3	10.0	7.8
C4 aggregate	29.5	29.5	25.9	28.3
C8 aggregate	28.3	42.5	30.2	40.8
C8 went from 0.41× to 0.67–0.74× of Gufo (Atlas MTP 17.4 → 28.3, AR 20.0 → 30.2), and C4 is at parity. Atlas still leads every C1 row.

[amd-campaign · 2026-10-31 17:40Z · strix s166 · main b73b73e] 27B same-box h2h, Atlas side (Gufo protocol, prefix cache on):

MTP: C1 pp 381 / tg 15.2, 8k-cached pp 212; steady C1/C4/C8 15.3 / 29.8 / 30.4
AR: C1/C4/C8 9.9 / 26.1 / 32.5
Gufo, same box (s140): DFlash2 C1 9.3, C4 29.5, C8 42.5; AR C8 40.8
Atlas leads C1, matches C4; C8 is 0.80x (AR) / 0.72x (MTP). It was 0.41x on 10-29.

Inference: at C8, MTP is now slower than AR (30.4 vs 32.5; s163 shows the same). s168 tests ATLAS_MTP_MAX_SEQS=4 (AR above 4 active) as a gfx1151 default.

[amd-campaign · 2026-10-31 19:05Z · merged #297 → main 4fd2736 · strix s169] gfx1151 now turns MTP off above 4 active sequences. Exact train, 27B NVFP4, MTP 3 drafts, Gufo protocol ×2: C8 29.2 → 34.0 (+16%), C1 14.3 → 14.45, C4 28.35 → 28.15. Flash-Next shows the same crossover (framework f142: MTP vs AR C6 28.4/30.0, C8 31.3/34.1; MTP wins C1 and C4). Open, measured, not yet merged: an A-row cache-set aliasing fix in the batched decode GEMVs. BF16 batchm is 1.45–3.4× on the kernel, byte-identical (framework f148); e2e A/B f149 and the FP8 n2 port f150 are running.

Navigation Menu

Repository navigation

[job] Qwen3.8-27B on Strix Halo: prefill 2.4x gap + DP4A verify tiers for K=5..9 #72

Status (updated 2026-10-28 05:30Z)

Kind

Model

What's the job?

Impact radius

What else could it touch?

Who's waiting or willing to test?

Stage-6 prefill matrix (strix, 2026-09-25) — no kernel-tier flag beats the default

Kernel ranking: Strix long prefill is GEMM-efficiency-bound (rocprofv3, 2026-09-25)

30 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[job] Qwen3.8-27B on Strix Halo: prefill 2.4x gap + DP4A verify tiers for K=5..9 #72

Description

Status (updated 2026-10-28 05:30Z)

Kind

Model

What's the job?

Impact radius

What else could it touch?

Who's waiting or willing to test?

Activity

AzeezIsh commented on Sep 25, 2026

Stage-6 prefill matrix (strix, 2026-09-25) — no kernel-tier flag beats the default

AzeezIsh commented on Sep 25, 2026

Kernel ranking: Strix long prefill is GEMM-efficiency-bound (rocprofv3, 2026-09-25)

AzeezIsh commented on Sep 25, 2026

AzeezIsh commented on Sep 25, 2026

AzeezIsh commented on Sep 25, 2026

AzeezIsh commented on Sep 26, 2026

AzeezIsh commented on Sep 26, 2026

AzeezIsh commented on Sep 27, 2026

AzeezIsh commented on Sep 27, 2026

AzeezIsh commented on Sep 28, 2026

AzeezIsh commented on Oct 2, 2026

AzeezIsh commented on Oct 2, 2026

AzeezIsh commented on Oct 2, 2026

30 remaining items

AzeezIsh commented on Oct 29, 2026

20k commented on Oct 29, 2026

AzeezIsh commented on Oct 29, 2026

AzeezIsh commented on Oct 29, 2026

AzeezIsh commented on Oct 29, 2026

AzeezIsh commented on Oct 30, 2026

AzeezIsh commented on Oct 30, 2026

AzeezIsh commented on Oct 30, 2026

AzeezIsh commented on Oct 30, 2026

AzeezIsh commented on Oct 30, 2026

AzeezIsh commented on Oct 31, 2026

AzeezIsh commented on Oct 31, 2026

AzeezIsh commented on Oct 31, 2026

AzeezIsh commented on Oct 31, 2026

AzeezIsh commented on Oct 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions