Strix Halo Vulkan stack: FA KV-quant, mmid, dense GEMM, delta-net prefill, DSv4 sparse attention - #17
Merged
Merged
Conversation
github-actions
Bot
added
documentation
This was referenced Sep 2, 2026
|
Ran this branch (cherry-picked onto the 68-commit halo-box catch-up, with #18 also stacked) against a tree sitting on current ggml-org master ( Setup: Strix Halo gfx1151, Vulkan/RADV, Fedora 44 mesa.
Decode flat to two decimals, exactly as you say. On prefill the two are within ±4% at every depth. The 2.65× at 32k for this model comes out at 1.04× here. The reason looks like the collapse itself: you measure halo-box master falling off a cliff at depth (Coder-30B 1558 → 96 t/s). On current upstream this model goes 416 → 269, a 35% decline rather than a collapse — so there is much less left to recover. None of that makes the work wrong; it makes the baseline old. Which is an argument for landing the fork's upstream sync sooner, and it lines up with your note that the Qwen3.8/DSv4 runtime fixes are waiting on exactly that. One incidental finding while testing: this tree cannot load unsloth's official shared MTP heads ( |
Author
|
Thanks for testing. Can you ty with f16 KV, or another kv quant type? Q8_0 is merged upstream but none of the others at present |
|
Follow-up on the KV-quant question, with measurements. Independent box: Ryzen AI 1. First, the premise: f16 and q4_0 KV + FA do run on the referencesYou said q8_0 is the only KV type merged upstream at present. On our box, both 2. The answer: the +/-4% parity does not survive a KV-type changeSame protocol as our q8_0 round: 5 rounds, arm order rotated, n=5 per cell.
Decode is flat (within 0.3%) at every depth. So at q8_0 KV the references are
Decode parity everywhere. This is consistent with the known RADV f16-KV 3. Correctness findings
4. Summary viewAt q8_0 KV this stack is roughly at parity on dense prefill and clearly ahead Assisted-by: Codex (measurements run and checked locally on real hardware by the poster). -Astra |
|
Hi @Nathanw1014 , |
…/q5_0/q5_1/iq4_nl) Evidence branch only - NOT for upstream. Extends the q8_0 dequant-once FA path to every KV-eligible quant type via per-type fused dequant+transpose shaders, plus a prefill-only fa_kv_ok gate for iq4_nl (no native coopmat1 path) and a GGML_VK_FA_DEQUANT env toggle. Correctness: dequant-once == native FA bit-exact for q4_0/q4_1/q5_0/q5_1; iq4_nl matches CPU. Finding: prefill is quant-type independent (all dequant to identical f16 scratch); iq4_nl is a poor KV type (ppl ~2x q4_0 at equal bits). Retained as gating evidence. Assisted-by: Claude Opus 4.8
The dequant-once path materializes a per-layer f16 K/V scratch (~2 KiB/token). On discrete devices that resident footprint can push the working set past free VRAM, at which point the driver silently pages device-local memory: measured ~15x prefill regression on an 8 GB card (RTX 3070, driver 591.86) at long context, with no error reported. Integrated/UMA devices have no separate device pool to overflow and are unaffected. Gate the path on this process's device-local usage against the physical heap size, keeping a conservative reserve for memory not observable in-process. heapBudget is deliberately not used as the signal: ggml_backend_vk_get_device_memory computes heapBudget - heapUsage in unsigned arithmetic, which wraps to a huge value exactly when the device is oversubscribed. The allocation cannot gate itself either - on WDDM vkAllocateMemory only fails at roughly physical heap size, which is above the free-VRAM level where paging begins, so a successful allocation is not evidence of a resident fit. Also fix the scratch size check: K and V are bound as one storage buffer, so their sum must fit maxStorageBufferRange, not each half independently. GGML_VK_FA_DEQUANT=0 forces the path off and =1 skips the capacity check; GGML_VK_FA_DEQUANT_RESERVE_MB overrides the reserve. Assisted-by: Claude (Opus 5)
…G, env-gated) The KV-cache view reaching FA has head-interleaved rows ([HS, NH, KV] physically), and the cm1 shader's direct-from-global coopMatLoads run ~2x slower on that stride than on per-head-contiguous K/V: a 16x16 tile touches 16 distant cache lines instead of 4. Measured on Strix Halo (RADV gfx1151), hd128/GQA8/kv10240/nb2048 f16: 29.8ms contiguous vs 63.1ms dense-permuted (the model layout; matches the in-model 59.9ms from the perf logger, where FLASH_ATTN_EXT was 72.6% of the graph at pp2048@d8192). GGML_VK_FA_KV_CONTIG=1 extends the dequant-once FA scratch to f16 K/V: dequant_f16_transpose.comp is a pure strided copy ([HS,NH,KV] -> [HS,KV,NH], same push-constant ABI and dispatch as the quant transpose shaders), engaged only when the rows are actually strided, prefill only (neq1 >= 64). FA op 63.1 -> 30.5ms (2.07x) incl. copy cost. Model-level (Qwen3-Coder-30B Q6_K_XL, ub2048, f16 KV, r=3, vs ROCm 571d0d5 nowmma): pp8192 877 -> 1199 t/s (ROCm 1216, was -28% now parity); pp2048@d4096/8192/16384: 776/513/300 -> 1120/850/580. Shallow prefill unchanged-to-better (pp2048 1542 -> 1633). Not the fix: shmem staging on AMD (loses on occupancy, 29.8 -> 54.4ms contiguous), bigger-tile/GQA-packed streaming (1-KV-head L2-resident probe runs identical -> kernel is issue-bound, not bandwidth-bound). test-backend-ops -o FLASH_ATTN_EXT green with the flag off and on (pre-existing iq4_nl+sinks failures unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
test_flash_attn_ext always built K/V as sparse views (physical dim1 doubled, half viewed), which can never satisfy the contiguize path's ggml_is_contiguously_allocated gate - so no permuted correctness case exercised it. Add a kv_view parameter (default true = unchanged) and dense-permuted eval cases matching the real KV-cache layout, including ALiBi and logit-softcap variants; all pass vs CPU with GGML_VK_FA_KV_CONTIG=1. Perf additions: Qwen3-Coder-30B prefill-at-depth shapes (hd128, 4 KV heads, GQA 8, kv up to 10240, nb 512/2048), the dense-permuted variant (model layout), and the probe set used to establish that the contiguous cm1 kernel is issue-bound: 32-distinct-KV-head MALL-spill (flat), 1-KV-head L2-resident (flat), mask=0 (-5.5%), f16 acc (-3%). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…ONTIG=0 opts out) Flip 442d7df from opt-in to default-on, matching the quant dequant-once path (GGML_VK_FA_DEQUANT) convention. The pass still self-gates: f16 K/V only, prefill only (neq1 >= 64), only when rows are actually strided, dense allocation, and the shared scratch-capacity check. Validated on Strix Halo (RADV gfx1151): FLASH_ATTN_EXT suite green with default env (dense-permuted cases exercise the pass) and with the opt-out; model-level pp2048@d8192 with no FA env matches the explicit GGML_VK_FA_KV_CONTIG=1 validation run (846.6 vs 847.9 t/s). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…ard-gate Rebased adaptation of the iq4_nl routing fix: upstream 8161641 made iq4_nl a native FA type, so the original motivation (iq4_nl had no native shader and silently read garbage outside the dequant-once path) no longer applies to any currently-admitted type. Keep the machinery as hardening: ggml_vk_fa_kv_native() is the one list, supports_op mirrors every hard condition of the dispatch-time dequant gate for any future non-native type, and dispatch asserts the invariant instead of falling back to a garbage-reading shader. Native list synced with upstream (iq4_nl in, q1_0 out to match current admission). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
iq4_nl has no native Vulkan FA shader; the dequant-once scratch path is its only route (1abdd92). The sweep's iq4_nl cases are all sparse-view, which that path correctly rejects, so iq4_nl had zero passing FA coverage. Add model-layout (dense [0,2,1,3]-permuted) cases at prefill batch size: iq4_nl/iq4_nl with sinks off and on, hd72+GQA with sinks, and mixed K=iq4_nl/V=f16. Validated on RADV gfx1151 at 146fb73: FLASH_ATTN_EXT 4765/4765 with default env and with GGML_VK_FA_KV_CONTIG=0/1; 4761/4761 with GGML_VK_FA_DEQUANT=0 (new cases correctly report unsupported); full test-backend-ops suite 15538/15538. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
The contiguize/dequant pass is dispatched inside the FLASH_ATTN_EXT node handler, and the perf logger writes one timestamp per graph node, so its cost was charged to FLASH_ATTN_EXT with no way to separate the two. That made the copy invisible and the kernel look correspondingly slower. Adds a sub-node timestamp: a handler can close an interval mid-node and have it logged under its own name. Measured on Coder-30B UD-Q6_K_XL, pp2048/ub2048 at d32768, f16 KV: graph total 4612.3 ms FLASH_ATTN_EXT 3490.6 ms 75.68% of graph FA_KV_CONTIGUIZE 33.1 ms 0.72% of graph (0.95% of FA) so the copy is under 1% of the graph and FA is 75.7% of it at that depth. Bench throughput is unchanged with the instrumentation compiled in (441.09 vs 442.14 t/s), since the marks are only emitted when the logger is on in per-op mode. Two latent bugs in the query-pool handling fall out of this and are fixed here: - The pool is created with n_nodes+100 slots but only the first n_nodes+1 were reset each graph, so anything using the headroom would read stale results. - The results buffer was sized n_nodes+1 while getQueryPoolResults was asked for query_idx entries. Equal today, but it is an overflow waiting for the first caller that writes an extra timestamp. Sub-op intervals log no flops, since they move bytes rather than doing math; attributing the node's flop count to them would corrupt the GFLOPS column for both halves. Assisted-by: Claude (Opus 5)
acc is an int32 sum of dotPacked4x8EXT results. With q8_0 both operands are full int8, so its bound is d_per_step*4*127*127, which overflows f16 when ACC_TYPE is f16 (GGML_PREC_DEFAULT): the score goes +inf, softmax is destroyed and the output comes back -FLT_MAX. Nibble types bound at ~30480 and stay in range, which is why only q8_0 tripped it. Apply the scales in fp32, then narrow. Identical arithmetic when ACC_TYPE is float, so the f32acc path is untouched. This is an upstream bug, not a fork regression, and it was fixed here once before - 61e77f4 carried it and the rebase onto b10133 dropped it. Only the scalar shader has MMQ, so the failing shape is hsk=128 + q8_0 K + prec=def + nb=1 + nr23=[1,1]; GQA>1 and nb>1 route to coopmat1 and escape. test-backend-ops -o FLASH_ATTN_EXT on gfx1151, quiet box: 13295/13295 twice, up from 13257/13295. All 38 failures were type_K=q8_0 prec=def. Both ggml_flash_attn_ext call sites in the tree force GGML_PREC_F32, so no model path reaches this - it is a landmine for the next person touching precision, not a live bug. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Nathanw1014
force-pushed
the
strix/vulkan-stack-only
branch
from
September 8, 2026 00:28
5450978 to
792acdf
Compare
Implements GGML_OP_LIGHTNING_INDEXER on Vulkan (scalar subgroup shader for small batches, coopmat 16x16 tiles for prefill, dedicated decode variant) and an indexed sparse flash-attention path that consumes the indexer's top-k selection directly via a new ggml_flash_attn_ext_add_top_k() API (FA src[5] + op_param[4] = n_kv_raw dense prefix), instead of attending densely over the full compressed KV. The sparse path engages only for V4's CSA shape (hd 512, 64 heads, MQA, f16 K==V latent) when dense_kv >= 3x active_kv; everything else falls through to the dense path, which stays correct because the kq_mask still carries the top-k selection. Dropped from the original: the mul_mat_id tokens-per-expert pipeline selection, which duplicates GGML_VK_MMID_SMALLN already on this branch. Originally by Gaetan Puleo (llama-cpp-nathan-toolbox-deepseek-v4-poc, branch deepseek-v4-flash-strix-halo); cherry-picked with the mmid hunk dropped.
- flash_attn_top_k.comp: remove the dead bounds check that would skip barriers for part of the workgroup if it ever fired (barrier divergence is UB; the dispatch gate sizes the grid exactly), pin per-lane sizing to LANES=64 instead of the SUBGROUP_SIZE spec constant (array bounds fold from the spec default at compile time), name the f16 mask threshold constant - ggml.h: document ggml_flash_attn_ext_add_top_k semantics (index base, dense prefix, backends-may-ignore contract) - tests: cover the scalar indexer variant (batch 4 and boundary 15), previously only the cm and decode-cm variants had eval parity cases Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Adds test_flash_attn_ext_top_k: builds the DeepSeek V4 CSA attention shape (hd 512, 64-head MQA, V as a view of K) with a consistent per-token top-k/mask pair, one deliberately invalid index, and cases on both sides of the Vulkan engagement gates. nb >= 64 cases are the first numerical parity coverage the sparse prefill shader has had; nb < 64 and sub-3x-kv cases pin the dense-fallback contract. Perf cases sweep kv 8k/32k/64k at nb 1/8/64/512 with a fixed active set. Measured on gfx1151: the sparse shader is flat vs kv at prefill (~2.2 TFLOPS active-only) while nb < 64 falls back to dense and scales with kv (1326 us at 64k, nb=1) - the gap a sparse decode path needs to close. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…ction The sparse prefill shader gates on q->ne[1] >= 64, so single-token decode attends densely over the whole compressed KV and its cost grows with context. This adds a gather pass (flash_attn_gather.comp): copy the active rows (dense prefix + top-k selection; MQA, so all 64 query heads share one set) plus their mask values into a compact contiguous scratch in prealloc_y, then run the ordinary dense FA over the compacted K/V/mask. V is the K latent, so one gather serves both. Invalid indices and padding get zeroed K and -inf mask. The FA function itself only has its inputs swapped: KV, mask geometry, strides and the K/V/mask bindings are overridden up front and every downstream decision (pipeline choice, split-k, workgroup sizing, use_mask_opt) sizes itself to the compact KV unchanged. Engages for the V4 CSA decode shape when kv >= 2x the padded active set; GGML_VK_FA_TOPK_GATHER=0 disables. Measured (gfx1151, test-backend-ops perf, active set 1536): kv=8192 nb=1: 255.6 us -> 55.8 us (4.6x) kv=32768 nb=1: 986.4 us -> 56.1 us (17.6x) kv=65536 nb=1: 1333.8 us -> 58.7 us (22.7x) Time is flat vs context. nb>1 still falls back to dense pending a union gather. FLASH_ATTN_EXT eval suite green incl. the top-k parity cases (the kv=4096 nb=1 case exercises this path end-to-end vs CPU). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Fold-in note for the toolbox branch: use_dequant_kv is additionally gated on !fa_compact.active (the compact scratch is already contiguous f16, and the two scratch layers must not stack), and the compact stride overrides chain through the toolbox's nb*_eff values so the contiguize/dequant path keeps its strides when the gather is inactive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Ports ggml-cuda/dsv4-hc.cu to Vulkan: hc_pre (mix the HC input streams down to one embedding), hc_comb (per-token 4x4 stream-mixing matrix - per-source softmax then eps-stabilized alternating column/row sinkhorn normalization, whole matrix in registers, one thread per token), and hc_post (redistribute the layer output back into the streams with the mixed residual). Same launch geometry as the CUDA kernels; plain f32 compute, no subgroup or coopmat requirements, so the pipelines are created unconditionally. The value at decode is dispatch-count collapse: the unfused fallback runs the decomposed graph (measured on gfx1151 config-a partial offload: SUM_ROWS alone 79 dispatches x 39.7us = 3.1ms per graph, plus DIV/MUL/ ADD shares at 4x4 shapes) where the fused form is 3 dispatches per layer. resolve_fused_ops now keeps all three fusions enabled on Vulkan instead of printing 'not supported, set to disabled'. Parity: upstream test-backend-ops DSV4_HC_PRE/COMB/POST cases green vs CPU on first build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…-ctk The fused indexer kernels read f16 keys only, so quantizing the small (128-dim) indexer cache silently disables them and falls back to the decomposed full-KV indexer path, whose contiguize cost grows superlinearly with depth (measured 0.8ms -> 97ms per dispatch by 12k context on Vulkan). Pin the indexer key cache to f16 in both DSA cache variants; the memory cost vs q8_0 is ~120 bytes per token per layer. Assisted-by: Claude Fable 5
Psh held P as [kv][query]. The GEMM2 UseA load therefore had to request ColumnMajor, and RADV only attaches an alignment hint to the internal column-major case, which for UseA means the RowMajor request. The load was emitted as 16 separate 16-bit shared reads per fragment. Store P as [query][kv] instead and request RowMajor. The producer now writes four scalar components rather than one vec4; those go to disjoint bytes, so there is no read-modify-write and no partially written vec4. Also updates the host shared-memory estimator, which mirrored the old stride and would otherwise disagree with the shader. Perf-neutral on gfx1151 (within 1% at hd128 and hd256), but it shrinks Psh: LDS 16384 -> 15360 B and code size 13896 -> 13596 at hd128, VGPRs unchanged at 96 with 0 spilled. Kept because it is free and removes a scalar shared-memory access pattern. FLASH_ATTN_EXT suite 5105/5105. Assisted-by: Claude Fable 5
get_fa_tuning_params_coopmat1 took device->subgroup_size unconditionally, so on a 64-wide device the coopmat1 FA path always ran wave64. The sibling scalar path already does AMD-specific wave selection; the coopmat1 path never got the equivalent. Narrowing is free exactly when it does not add an iteration to the O-accumulation loop, which runs ceil((HSV/4) / threads_per_rowgroup) per row. On a 64-wide device the test reduces to hsv <= 128. Above it the narrow subgroup issues 1.5x to 1.8x the instructions for the same SIMD passes and hd256 measures 6 to 18 percent slower, so the rule declines. Gated behind GGML_VK_FA_WAVE32 (=1 rule, =2 forces the pin regardless of head size, diagnostic only). Off by default. Measured on gfx1151 (RADV), model-level pp2048, Qwen3-Coder-30B-A3B: d0 +2.5%, d8192 +8.4%, d16384 +10.1%, d32768 +11.3%. Op-level across head sizes, largest where wave64 wastes the most lanes: hsv=64 +12.3%, hsv=96 +7.4%, hsv=128 +6.5%, hsv=256 correctly declined. Full test-backend-ops suite 15884/15884 at =0, =1 and =2; the pin was confirmed to engage in-band via pipeline VGPR statistics rather than inferred from the env var being set. Assisted-by: Claude Fable 5
Split from the fork's flag-flip commit (f7d804ee7): GGML_VK_FA_WAVE32 now defaults to 1 (apply the HSV-based narrowing rule); =0 opts out and =2 keeps its diagnostic force meaning. The subgroup_size_control device guard is unchanged, so devices without a 32-wide subgroup mode are unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Select the matmul_id tile by expected per-expert token count (nei1*nei0/n_expert) instead of aggregate nei1, which always picked the widest tile and left most N-lanes empty at MoE prefill (~16 rows/expert vs BN=64). Gated on GGML_VK_MMID_SMALLN=1, default off pending cross-model validation. Was null (+0.7%) standalone pre-row-lists: smaller tiles meant more workgroups each re-paying the per-WG id scan. With the Stage 1 row-list prepass the scan is gone and the occupancy win materializes. Strix Halo (RADV gfx1151), Qwen3.6-35B-A3B UD-Q5_K_XL, clean window, pp512 4-way (rowlists x smalln): 914.7 / 937.9 / 1005.3 / 1063.6 t/s (combined +16.3% vs baseline). MUL_MAT_ID q5_K 2.33->4.43 TFLOPS (1.91x), q6_K 1.99->3.57 (1.79x) cumulative. Hot pipelines verified via probe: matmul_id_*_f32 (y_f32 path, quantize_y does not engage for these quants); smalln flips tile _m -> _s. 790/790 MUL_MAT_ID both rowlists configs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Two env-gated overrides of the mmid small-tile config (KHR coopmat branch only, scoped to the mul_mat_id quant pipelines; dense untouched): - GGML_VK_MMID_TILE16: BN/WN 32->16. NEGATIVE (-3.8% e2e): experts with n_e>16 split into two column tiles and re-stream their full weight matrix; A-traffic scales with sum(ceil(n_e/BN)), so BN must not drop below the mean per-expert n. Kept as a documented dead end. - GGML_VK_MMID_BM64: BM 32->64, BLOCK_SIZE 64->128 (two warps). +1.3% e2e on top of rowlists+smalln: same A-traffic, half the ir-tiles so half the per-expert B re-reads, larger WGs hide latency. Strix Halo, Qwen3.6-35B-A3B UD-Q5_K_XL pp512, clean window, all on top of RL+SN control 1065.4 +/- 3.3: BM64 1078.9 +/- 3.5, TILE16 1024.6, BM64+TILE16 1007.5. Cumulative vs pre-Stage-1 baseline: 914.7 -> 1078.9 (+17.9%). 790/790 MUL_MAT_ID for all configs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
…ated) Extends the BM64 idea to the medium tile: GGML_VK_MMID_M128=1 raises the mmid m-tile to BM=128 / BLOCK_SIZE=4*subgroup (four warps), same A-traffic, half the ir-tiles and per-expert B re-reads. The medium tile is what the per-expert-n heuristic selects at n~64 (e.g. ub2048 on 256-expert/top-8 models, or ub1024 at 128 experts). Strix Halo, Qwen3.6-35B-A3B UD-Q5_K_XL pp2048, drain-verified window: ub2048 stack 1039.9 +/- 1.1 -> +M128 1091.6 +/- 2.6 (+5.0%); inert in the s-tile regime (ub1024 1137.2 vs 1147.9 ref). ub1024 remains the throughput sweet spot for this model. 790/790 MUL_MAT_ID. Also measured this session, NOT kept: caching counts+row lists across the three expert matmuls of a layer (they share one ids tensor) was correctness-clean but perf-null (1066.3 vs 1070.9 baseline) - the prepass dispatches and barriers are already free on this queue; reverted rather than carry the invalidation surface. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Force required subgroup size 32 on the KHR-coopmat mmid quant pipelines (pipeline_dequant_mul_mat_mat_id[*] only; dense untouched). Hypothesis: RDNA3.5 WMMA is wave32-native, so RADV may lower KHR_coopmat better at wave32 than at the reported default of 64. Mechanics: the cm1 path in mul_mm.comp derives its warp grid from the real subgroup (warp_i = gl_SubgroupID, tiw = gl_SubgroupInvocationID) while NUM_WARPS = BLOCK_SIZE/WARP (spec constants) sizes coopmat_stage[] and ballots_sh[] and warp_r/warp_c assume NUM_WARPS == (BM/WM)*(BN/WN). Forcing sg32 with WARP=64 would over-run those shared arrays and leave warp_c outside the tile, so the gate (a) sets WARP=32 in shadowed copies of the s/m/l mmid warptiles (composes after the BM64/M128 shadows) and (b) halves WM (or WN, keeping WM>=TM, WN>=TN) until the doubled subgroup count exactly tiles BM x BN again, asserting both invariants. BLOCK_SIZE is kept, so workgroup shape, load loops and shmem match the wave64 stack; per-lane accumulator footprint is also unchanged (half the lanes per subgroup, half the (WM/TM)*(WN/TN) fragments). The required size is passed via a scoped CREATE_MM redefinition adding a trailing required_subgroup_size arg (fp16-branch pattern), gated on subgroup_size_control covering 32 since ggml_vk_create_pipeline_func silently drops the required size otherwise. Correctness (test-backend-ops -o MUL_MAT_ID -b Vulkan0): 790/790 plain, 790/790 WAVE32=1, 790/790 WAVE32+SMALLN+BM64. Perf, Qwen3.6-35B-A3B-UD-Q5_K_XL, fa=1 b/ub=512 ctk/ctv=q8_0 pp512 r=3, atomic window, canary clean: stack (SMALLN+BM64), wave64 default: 1076.61 +/- 3.71 t/s stack + WAVE32: 1106.73 +/- 2.52 t/s (+2.8%) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Convert contiguous f32 activations (B) to f16 for quantized MUL_MAT_ID on KHR_coopmat devices and run the matmul_id_subgroup_<quant>_f16 kernels instead of the f32-B ones. Halves B bytes and buf_b shared memory. The _f16 SPIR-V already exists for every quant type; this adds a parallel env-gated pipeline array (pipeline_dequant_mul_mat_mat_id_f16b), extends the getter to return it for src1=F16 on non-coopmat2 devices, relaxes the src1-type assert, and forces the existing y_non_contig convert-to- prealloc_y plumbing (same as coopmat2). Default OFF, zero behavior change when unset. Measured on Radeon 8060S (RADV gfx1151), Qwen3.6-35B-A3B-UD-Q5_K_XL, -fa 1 -b 512 -ub 512 -ctk q8_0 -ctv q8_0 -p 512 -r 3, stacked on GGML_VK_MMID_SMALLN=1 GGML_VK_MMID_BM64=1: stack (f32 B, canary): pp512 1075.04 +/- 8.66 t/s stack + F16B: pp512 1100.66 +/- 2.37 t/s (+2.4%) test-backend-ops test -o MUL_MAT_ID -b Vulkan0: 790/790 with and without GGML_VK_MMID_F16B=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Assisted-by: Claude (Opus 5)
The MMQ J config is chosen from n, so a wave-partitioning error in one J tile only shows up when n sits on that tile's boundary. The neighbouring n picks a different J and passes, which hides it. The existing quantized cases stop at n=129 and the general MUL_MAT set jumps 64 -> 4096, so nothing lands on 256 or 512 and the whole class went untested. Sweep n over 255/256/257/511/512/513 for q8_0, q4_0, q4_K, q5_K and q6_K, in both MUL_MAT and MUL_MAT_ID. On an RDNA3.5 build that runs the J128 kernel with 16 wave32 waves over a 128-row tile these fail for every n that selects J128 and pass at n=513, which selects J112. A build predating that config passes the whole sweep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit b54cd8a) Assisted-by: Claude (Opus 5)
30d8bb0 raised mul_mat_id_param_count to 6 for the fused MUL epilogue and gave every mul_mat_id shader a binding 5 (FusedScale), including the coopmat2 one, which binds it purely for descriptor-layout parity. The coopmat2 pipeline creation block was left passing a literal 5, so two things go wrong there: - the pipelines are created with 5 descriptors while the shader declares 6 - PARAMCOUNT == mul_mat_id_param_count doubles as the "this is mul_mat_id" argument to ggml_vk_mul_mm_cm2_spec, so it went false and every coopmat2 mul_mat_id pipeline was specialized as a plain matmul, dropping the trailing spec constant Only reachable where device->coopmat2 is true. gfx1151 does not take that path and the v0.5 release predates the constant bump, so neither is affected. Not validated on hardware - no coopmat2 device here. The block does compile: built with the pinned shaderc (GL_NV_cooperative_matrix2 supported). The two OCP FP4 sites stay behind GL_EXT_float_e2m1, which that glslc does not support. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
Split from the fork's flag-flip commit (f7d804ee7): GGML_VK_MMID_F16B, BM64, M128, WAVE32 and SMALLN now default on with =0 opt-out. The wave32 gate keeps its subgroup_size_control device guard, so devices without a 32-wide subgroup mode are unaffected. GGML_VK_MMID_TILE16 stays opt-in (documented negative on gfx1151). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
RDNA3.x WMMA is wave32-native, so a wave64 subgroup issues each coopmat op as two halves. GGML_VK_MMID_WAVE32 already exploits this for mul_mat_id and leaves the dense pipelines at the driver default; this gives dense the same treatment, gated on measurement rather than a flag. BLOCK_SIZE is kept, so the subgroup count doubles and WM (or WN) halves until the warp grid tiles BM x BN again. Only the quantised tiles are retiled: on gfx1151 a standalone MUL_MAT microbench at both dense FFN shapes reads q6_K +5.2..+10.8%, q8_0 +5.4..+8.4%, q4_K +0.7..+9.1%, q4_0 -1.5..+1.8%, while f16 reads -6.7..+6.4% and bf16 ~0 - the float paths are bandwidth-bound on the weight stream, not issue-bound. The win tracks inline dequant instruction count (q6_K 3907 -> 3433 instructions, identical 192 VGPRs and 8 subgroups/SIMD). The required subgroup size is now the tile's own WARP for every dense coopmat pipeline. The cm1 shaders derive their warp grid from gl_SubgroupID and size shared arrays as NUM_WARPS = BLOCK_SIZE / WARP, so WARP and the real subgroup must agree; leaving that to the driver made the agreement incidental. Scoped to AMD coopmat1 on a wave64 default; other vendors keep the driver default. GGML_VK_DENSE_WAVE32=0 disables, =2 also retiles the float tiles. Qwen3-32B Q6_K_XL pp2048 +7.2% at ub256 / +3.9% at ub2048, Qwen3.8-27B +5.3% / +4.8%. PPL unchanged: 6.9496 +/- 0.24246 in both arms, all 20 per-chunk values identical, since the retile changes which warp owns an output sub-tile and not the K-reduction order within an element. Assisted-by: Claude Opus 5 (cherry picked from commit 448994e9637405610ced3e0ead02c7b6fa688314)
All default OFF, so the same binary A/Bs each change. GGML_VK_CONCAT_TRANSPOSE: delta-net does ggml_transpose() into a dim-0 ggml_concat(), which the generic concat reads fully de-coalesced. Route that shape through a 32x32 shared-memory tile transpose. CONCAT 11877 -> 957 us/op. GGML_VK_MMID_SCALE_EPILOGUE: apply the following MUL's per-(expert,token) broadcast scale as mul_mat_id writes out, removing a 134 MB write plus read back. Prefill only; the existing fusion is gated to mat-vec. Not implemented in the coopmat2 shader, so it is refused there. GGML_VK_FUSE_UNARY_MUL: silu(x)*y is two nodes in the delta-net path; run it as the existing swiglu split. 750 -> 443 us/op. GGML_VK_MMID_WG256: the RADV tuning gives the dense large tile 256 threads on a 128x128 tile but left the mul_mat_id variants at 128. Qwen3.6-35B-A3B UD-Q4_K_XL, gfx1151, pp2048 at ub2048: 1223.75 -> 1733.64 t/s. test-backend-ops CONCAT/MUL_MAT_ID/UNARY/MUL pass; generated text is unchanged with each flag on, and the disabled paths are bit-identical to before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 6f7a49e) Assisted-by: Claude (Opus 5)
The fused epilogue derives the scale index from row_ids as [token * nei0 + expert_slot], which carries no 4th dimension, but the gate admitted any ne[3] as long as the scale and the matmul agreed. A tensor with ne[3] > 1 would read the wrong scale for every batch past the first and return quietly wrong results. test-backend-ops never generates such a case, so the suite passed throughout; found by reading the gate against the shader. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit 016e906) Assisted-by: Claude (Opus 5)
The delta-net conv-state path transposes straight into a dim-0 concat, so the generic concat kernel walks src1 with a conv_channels * 4 byte stride. On qwen35 that is 40960 B, which is 160 * 256 B with 160 % 16 == 0, so every read lands on the same one of the 16 memory channels: 13.7 GB/s against 138.9 GB/s for the tiled path. The tiled-transpose route has been behind GGML_VK_CONCAT_TRANSPOSE=1 since it landed. Turn it on by default and keep GGML_VK_CONCAT_TRANSPOSE=0 as the opt-out. Qwen3.8-27B pp2048: +0.4% at ub 256, +4.7% at ub 1024, +7.2% at ub 2048. Assisted-by: Claude Opus 5
The scheduler's async input copies between graph splits land in the compute ctx on devices without a separate transfer queue, so the perf logger's fresh-ctx assert fired under partial offload (--n-cpu-moe). Assisted-by: Claude Fable 5
Nodes with no flops estimate (large copies, set_rows, mask fills) can pack a command buffer whose execution time grows with context length until it exceeds the amdgpu ring timeout (10s on the compute ring), causing the ring resets and DeviceLost reported at long context. Add a bytes-per-submit cap (default 8 GiB, GGML_VK_MAX_MB_PER_SUBMIT to override, 0 disables) alongside the existing flops and node-count gates. Assisted-by: Claude Fable 5
ggml-org#22789 replaced the fixed 30-entry split input array with a growable one and, in the same edit, changed the split-cutting heuristic from the constant to split->inputs_capacity: - if (split->n_inputs == GGML_SCHED_MAX_SPLIT_INPUTS) { + if (split->n_inputs >= split->inputs_capacity) { inputs_capacity starts at GGML_SCHED_MAX_SPLIT_INPUTS but doubles on demand and is never reset for the life of the sched, so once a split slot grows, the scheduler stops cutting there and the cut point ratchets up for every later graph build. Longer splits mean every cross-backend input copy is materialised at the split's start and stays live to its last use inside it, which raises the peak the compute-buffer allocator has to cover - n_copies times over under pipeline parallelism. Only multi-backend configurations can reach this. Keep the growable array, which is what fixes the original >30-input assert, and cut on the constant again as before ggml-org#22789. >= rather than == so the check keeps firing for splits that did have to grow. DeepSeek-V4-Flash UD-IQ3_XXS, gfx1151, -c 400000 -ub 2048 -fa 1 --fit off: Vulkan0 compute buffer 4714.00 MiB and 9157 graph nodes, byte-identical to the unpatched tree, and neither run grows a split past 30 inputs. Expected - one Vulkan device plus the CPU backend cannot exercise the path on this box. The reported case is 3 devices with pipeline parallelism. test-backend-ops -o FLASH_ATTN_EXT, run alone on gfx1151: 13257/13295 on both this and the unpatched tree, with the same 38 failing cases (identical case list, all type_K=q8_0 prec=def kv_view=1). Pre-existing on the branch, not touched by this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assisted-by: Claude (Opus 5)
The _f16 SPIR-V and pipeline_dequant_mul_mat_mat_f16 already exist, but are only populated in the coopmat2 branch. GGML_VK_DENSE_F16B populates them for coopmat1 too and routes B through the existing convert-to-prealloc_y path. Off by default: it helps large dense models and costs a little elsewhere. gfx1151, pp2048: Qwen3.8-27B and Qwen3-32B (hidden 5120) +5 to +7% for both q6_K and q8_0 weights, Qwen2.5-7B -1.2%, Qwen3-Coder-30B MoE -0.5%. Decode is untouched, ne1==1 does not reach this path. Numerically identical: mul_mm stages B into shared FLOAT_TYPE either way, so the f32-B kernel already rounds B to f16. Wikitext PPL matches to 4 dp. Assisted-by: Claude Opus 5
=auto restricts the f16-B path to ne10 == 5120, the only reduction width measured to gain, so it cannot fire on the widths that lose. =1 keeps the old all-shapes behaviour as a manual override. gfx1151, Qwen3.8-27B UD-Q6_K_XL pp2048, auto vs off: +5.8 / +6.0 / +5.9 / +5.3% at ub 256/512/1024/2048, which is 97% of the all-shapes win at ub256 and 82-86% above it. Qwen3-Coder-30B MoE is untouched, the gate never fires. The width equality is a stopgap until a per-shape predicate is derived; it will silently do nothing for a dense model of another width. Assisted-by: Claude Opus 5
The dense wave32 change rewrites s/m/l_warptile_mmq in place (WARP=32, WM/WN halved). The mul_mat_id tile block copied those rewritten vectors, so with GGML_VK_MMID_WAVE32=0 the mmid pipelines carried a WARP=32 spec constant but no required subgroup size and ran at wave64 with the wrong warp grid: test-backend-ops MUL_MAT_ID 700/7432 for every quantised type at n >= 16 (garbage output, not an error). Default settings were not affected; the opt-out was. Snapshot the dense tiles before the wave32 rewrite and build the mmid tiles from the snapshot. When the mmid wave32 pin is not applied, assert the coverage invariant and require the tile's own WARP as the subgroup size where the driver honours it, so the two can never disagree. Assisted-by: Claude (Opus 5)
Decode dispatches carry N = gqa_ratio query rows and the narrow subgroup loses there: Qwen3-Coder-30B q8_0 KV tg64 at d8192/d32768 measured 3.5 to 4 percent slower with the pin than without, while the prefill gain it was introduced for (up to 10 percent at d8192) needs the multi-row shapes. Apply the rule only when n_rows >= 32. Assisted-by: Claude (Opus 5)
Nathanw1014
force-pushed
the
strix/vulkan-stack-only
branch
from
September 8, 2026 11:21
792acdf to
1debd52
Compare
Nathanw1014
marked this pull request as ready for review
September 8, 2026 11:26
Author
|
Rebased onto current master (7449a0f), head is now 1debd52, conflicts gone. Description rewritten with a fresh grid against this master; the headline multipliers from the old body are gone. @dzannotti you were right that most of the earlier multiplier was the baseline. Against current master with your q8_0 protocol we get 1.24x / 1.37x / 1.51x prefill on Coder-30B at d0 / 8k / 32k on system RADV and 1.06x / 1.19x / 1.30x on Mesa main, and 1.25x to 1.28x flat across depth on Qwen3.6. Decode is flat on Qwen3.6; Coder was 1 to 3% down at depth, traced to the coopmat1 FA wave32 pin landing on decode dispatches, and a follow-up commit keeps the pin to multi-row dispatches (Coder tg64 at 32k now 48.2 vs 47.0 master, prefill unchanged). The Qwen3.6 lines in the old body were also against a bad reference build, so treat those as withdrawn. On shared MTP heads: we have those loading on our side and will send it separately. @voidsurfer thank you for the round, that is the KV-type effect we hoped someone would reproduce. On your questions:
If it helps review, we can split this by surface (FA KV-quant, mmid + dense, delta-net, DSv4); only DSv4 depends on the FA set. |
gaetan-puleo
approved these changes
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebased onto current halo-box master (7449a0f). 68 commits (66 rebased + 2 follow-ups from the QA below), cherry-picked directly onto master, no upstream sync bundled. Three commits from the previous revision were dropped because master already carries them (RADV LDS pad tuning and its driver gate, UMA bulk-read copy path).
What is in it
Perf gates default ON with
=0opt-outs (GGML_VK_FA_WAVE32,GGML_VK_MMID_{F16B,BM64,M128,WAVE32,SMALLN},GGML_VK_CONCAT_TRANSPOSE); Device scope: the wave32 pins are limited to AMD RADV (subgroup-size control on a wave64 default). The mmid tile reshapes, the f16 KV contiguize pass and dequant-once for q4/q5 KV apply to any KHR_coopmat device except Intel Xe1 and coopmat2 devices; those are perf-only and env-gated, they do not change results.Measured against current master (Strix Halo, gfx1151)
This replaces the earlier table. The earlier numbers were taken against a master that collapsed at depth (as @dzannotti showed, that was mostly the baseline), and the Qwen3.6 reference arm turned out to be a build with a known bad output, so the 1.44x/1.50x lines were wrong. Numbers below are the rebased tree (before the two follow-up commits) vs master (7449a0f), same session, ABBA order, 2 launches per arm, q8_0 KV (the conservative case: q8_0 FA is what upstream already has; f16 and q4_0 KV widen the gap at depth, see @voidsurfer's table below).
llama-bench -fa 1 -ctk q8_0 -ctv q8_0 -p 512 -n 32 -d 0,8192,32768 -ub 2048 -r 2Absolute numbers, raw llama-bench output and the run scripts are available on request. The dense row uses f16 KV and ub 256 (the dense-model setting); the MoE rows use q8_0 KV and ub 2048. Decode is flat on Qwen3.6 and 1 to 3% down on Coder at depth; that is the coopmat1 FA wave32 pin on decode dispatches (N = gqa_ratio rows), and a follow-up commit restricts the pin to multi-row dispatches (32+ query rows). With it, Coder-30B tg64 at d32768 is 48.2 vs master 46.7 to 47.2, and prefill keeps 1.37x / 1.47x at 8k / 32k.
DSv4 numbers (indexer, sparse FA, gather-compact decode) are unchanged from the previous revision and will be re-measured against this master separately; that model is not part of the grid above.
Validation
GGML_VK_FA_WAVE32=0; with the pin on, still identical on Qwen3.6 and Qwen3.8, and KLD 0.003 on Coder-30B. Same with CPU-offloaded experts. Every documented opt-out and opt-in run through its op suite alone.GGML_VK_MMID_WAVE32=0on its own produced wrong MUL_MAT_ID output because the dense wave32 change rewrote shared tile tables in place. Defaults were never affected.test-backend-ops -b Vulkan0: FLASH_ATTN_EXT 5220/5220, MUL_MAT_ID 7432/7432, MUL_MAT 7384/7387; the 3 failures (bf16/q5_1/q8_0, m=128 k=5120, fused CONCAT) also fail on halo-box master, separate issue to followNotes
Assisted-by:trailer per the contributing policy.AI usage disclosure: YES. Claude assisted with the rebase, conflict resolution, benchmarking and this description. Nathan Wilson owns and maintains the changes.
🤖 Generated with Claude Code