Repository navigation
Conversation
…election, and 8x8x8 coopmat1 Add device detection and matmul/MoE tile tuning for Apple AGX GPUs (M4 Max T6041 under Borealis and Mesa Honeykrisp): - Detect Borealis on Apple M4 Max (0x10005 / 0x6041) with GGML_VK_BOREALIS_TUNING (off|tiles|roll|all, default all) and disable SPIR-V BK loop unrolling on coopmat1 pipelines unless GGML_VK_CM1_ROLL_BK=1. - Add 8x8x8 VK_KHR_cooperative_matrix (f32matmad8x8) s/m/l dense and MoE warptiles sized for the 192-half-reg AGX G17P register file (<=16 8x8 accumulators per subgroup, BN=16 small MoE tile), subgroup-scoped barriers around coopMatStore, and guarded shared-memory bounds in mul_mm.comp. - Default disable_narrow_mul_mat on Apple AGX (with GGML_VK_DISABLE_NARROW_MUL_MAT=0|1 override) so 8x8x8 coopmat1 retains its tuned s/m/l tiles and split_k=1. - Select MUL_MAT_ID tiles by average rows per expert (GGML_VK_MMID_SELECT=pairs, default on Apple AGX coopmat1) with a dedicated matmul_id_tile_selector, and prefer FP16 B activations on AGX coopmat1 when nei1 > 32 (GGML_VK_MMID_F16_B). - Add tile/dispatch tuning knobs GGML_VK_CM1_TILES, GGML_VK_CM1_ID_TILES, GGML_VK_MMQ_S/M/L, GGML_VK_MMQID_S, GGML_VK_MMVID_MAX, GGML_VK_MMVID_ONE, and GGML_VK_DISABLE_INTEGER_DOT_PRODUCT. Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…gnostics Add configurable fence waiting, command-buffer submission policies, and env-gated graph instrumentation for CPU/GPU synchronization profiling: - Add GGML_VK_FENCE_WAIT=spin|block|wfe|nap|hybrid (with GGML_VK_FENCE_MARGIN_US, default 300 us) to control final graph fence waiting, including AArch64 WFE low-power polling and EMA-based hybrid sleep-then-spin after the almost-ready fence. - Add GGML_VK_SUBMIT_MODE=default|one|geo (with GGML_VK_SUBMIT_FIRST and GGML_VK_SUBMIT_GROWTH) and GGML_VK_FLOPS_PER_SUBMIT to control command-buffer submission granularity. - Add GGML_VK_GRAPH_STATS=1|2 (per-graph dispatch/barrier/submit/record/wait counters and per-dispatch pipeline+workgroup trace) and GGML_VK_DUP_OPS for non-in-place kernel timing ablations. Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…taNet recurrent state Add lossless BF16 storage for BF16-exact F32 gate weights and in-place Gated DeltaNet (GDN) recurrent state updates with speculative rollback safety: - Under GGML_GEN_BF16_EXACT=1, inspect F32 router, shared-expert gate, and GDN alpha/beta weights (.ffn_gate_inp.weight, .ffn_gate_inp_shexp.weight, .ssm_alpha.weight, .ssm_beta.weight, .ssm_beta_alpha.weight, .ssm_ba.weight) at load time; when every element has zero low 16 mantissa bits, store the tensor as BF16 and cast back to F32 above GGML_GEN_BF16_CAST_ABOVE columns (default 8) during graph build. - Under GGML_GEN_GDN_INPLACE=1, view recurrent cache rows directly in build_rs when the sequence state mapping is the identity (skipping GET_ROWS gather) while guarding against speculative rollback snapshot consumption when n_rs_seq != 0. - Add STATE_OUT variants of gated_delta_net.comp so single-step GDN writes its updated recurrent state directly into the destination cache slice and elides the trailing CPY node. Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…logue fusions Add Numerically Exact (0-ULP) RMS_NORM reductions and Tier-1 operator fusions gated by GGML_GEN_FUSE (all or comma-separated tokens): - Replace the non-subgroup shared-memory tail in rms_norm.comp with an exact 0-ULP subgroupShuffleXor tree reduction when gl_SubgroupSize == 32 and skip the redundant initial workgroup barrier. - Add lazy _glu mat-vec/mat-vec-id twin pipelines (specialization constant 3 = 1) so base mat-vec shaders incur zero extra register pressure. - Support GGML_GEN_FUSE=glu (mat-vec / mat-vec-id + SwiGLU), l2 (deferred RMS_NORM + SCALE with precise IEEE rounding for GDN L2 norm), gate (RMS_NORM * weight * SiLU(z) in one dispatch), mrope (M-RoPE / IM-RoPE in RMS_NORM_MUL_ROPE), mmgate (mat-vec + sigmoid and mat-vec + bias + softplus + mul), and shexp (mat-vec + per-token row scale). Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…roup DMMV, and outer row reuse Add row-split multi-subgroup dequant-mat-vec kernels, 0-ULP quantized fast paths, and paired mat-vec / mat-vec-id twin pipelines: - Add row-split multi-subgroup DMMV (local_size_y specialization constant 4 via GGML_VK_DMMV_NSG / GGML_VK_DMMV_ID_NSG), workgroup row/size knobs (GGML_VK_RM_STDQ/_ID, GGML_VK_RM_KQ/_ID, GGML_VK_DMMV_LARGE/_ID_LARGE), and 0-ULP Q8_0, Q4_K, Q5_K, and Q6_K vectorization / outer row reuse (GGML_VK_FAST_KQ). - Split paired mat-vec pipelines into lazy _pairmm (specialization constant 3 = 2) and _pairglu (3 = 3) twins so standalone and single-epilogue shaders keep minimum register footprint. - Support GGML_GEN_FUSE=gateup (single-dispatch gate + up + 0-ULP SwiGLU for MUL_MAT and MUL_MAT_ID) and GGML_GEN_FUSE=pairmm (sibling MUL_MAT sharing src1, including trailing sigmoid on the second output), with graph_optimize pairing and buffer lifetime dependencies. Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…ubmit barrier elision Add 0-ULP batch-invariant speculative verification mode and Series 8 graph scheduling / intra-submit barrier elision: - Under GGML_VK_BATCH_INVARIANT=1, size Flash Attention KV splits (including GQA ratios up to k <= 8 query tokens) identically to n=1, extend ADD+RMS_NORM partials to multi-row verification batches with an n=1 per-row 1 KB budget, and enable MUL_MAT + ADD fusion on the multi-column mat-vec path; also add GGML_VK_DISABLE_ADD_RMS. - Under GGML_GEN_FUSE=s8 (or all), fuse CONT + [view] + UNARY + MUL into a single strided unary_mul dispatch, lift external views out of RMS_NORM_MUL_SILU subgraphs, and co-schedule independent dispatches in ggml_vk_graph_optimize (1-workgroup TOPK_MOE with shared-expert gate+up+glu, ffn_moe_down with ffn_shexp so trailing residual ADDs coalesce into MULTI_ADD, and mixed-quant GDN projections) with explicit add_alloc_dep buffer lifetime constraints to elide intra-submit pipeline barriers. Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…d RMSNorm fusion Series 9-11 optimizations for Apple AGX (Honeykrisp / M4 Max): - Add per-graph Vulkan command-buffer replay cache for decode and speculative verification graphs with stable tensor buffer bindings. - Add 8-slot prealloc_y and prealloc_split_k ring buffers to eliminate intra-graph pipeline drain syncs across quantize_y and split-k dispatches. - Add multi_add_rms_mul.comp fusing residual ADD chains with RMS_NORM + MUL, fused ADD+SCALE, fused MoE top-k + scatter/gather epilogues, and fast FP16 dequantization/accumulation paths. Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
astaff
force-pushed
the
agx/m4-max-series8
branch
from
November 17, 2026 05:38
5d80a43 to
1a6738a
Compare
…WG=256 FlashAttention - Add mul_mv_mma.comp (8x8x8 simdgroup_matrix mat-vec/multi-col and grouped expert mul_mv_mma_id with fused paired-SwiGLU and expert-scale epilogues) and fused cm1 matmul/matmul_id epilogue variants (mul_mm_load_a_body.glsl). - Specialize single-wave subgroup shuffle gated_delta_net.comp (S_V <= 128) and tiled shared-memory ssm_conv.comp (NC == 4) with fused SiLU output. - Reduce 8x8x8 coopmat1 FlashAttention shared memory to <= 16 KB for dual workgroup occupancy per core (WG=256, 8 subgroups), add LPT GQA head/tile swizzle (Flags bit 256), and compact mask tiles for partial K blocks. Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…hread argmax, and greedy speculative fastpath - Add wide-tile RT=4/8 mul_mv_mma / mul_mv_mma_id variants with FP16 shared-memory activation staging (SHB=6, NT=1 for RT>=4 to eliminate AGX register spills on long-context prefill) and paired expert count_experts. - Add vec4 RMSNorm fast paths (32-lane 0-barrier d=128 QK-norm, dual F32+F16 outputs in rms_norm.comp and multi_add_rms_mul.comp to elide standalone cpy_f32_f16 passes before coopmat1 matmul) and fused 2-row Q+K RMSNorm. - Widen argmax.comp to 512 threads for large vocabularies (ne00 >= 65536) and optimize single-wave GDN/SSM kernels for multi-token prefill. - Add greedy common_sampler_sample_greedy() fastpath and skip redundant logit copies during greedy speculative decoding (MTP / draft verification). Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
astaff
force-pushed
the
agx/m4-max-series8
branch
from
November 17, 2026 06:02
1a6738a to
0588996
Compare
…used LoRA twins - Add 4-block (128-element) unrolled iter4_aligned_q8_0 / iter2_aligned_q8_0 mat-vec paths paired with 128-thread (4-subgroup) DMMV_WG_SIZE_LARGE routing for wide-K Q8_0 projections (reaching 90.04% of the 529.5 GB/s M4 Max DRAM ceiling on Qwen3.8-27B-Q8_0). - Replace unpack8 int-to-float Q8_0 coopmat conversions in mul_mv_mma.comp with 2-cycle IEEE-754 FP16 magic-bias extraction ((| 0x64006400u) - 1152.0h), hoist sh_b=12 on paired SwiGLU, lower GGML_VK_MMA_Q8_MIN_N to 8 (M <= 65536), and raise GGML_VK_MMA_MAX_N to 4096. - Hoist [lora_a] projections in ggml_vk_graph_optimize and fuse [mm, lora_b, scale, add] into lazy on-demand 11-binding gen_lora_twin pipelines with zero binding or register overhead on non-LoRA pipelines. - Refine common_sampler_init pure_argmax_fast detection across default sampler chains when penalty/DRY/XTC parameters are inactive. Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull requests for this build
/dev/dma_heap/systemPRIME backing and scales the UMA heap to 82% (52.3 GiB on 64 GB).26B123/26B131) firmware. Aura passes the 3-segment GPU firmware and display handoff data to Linux, and Linux enables the27.2AOP,ANE,GPU, andDCPfirmware interfaces.Test system and results
This pull request changes how the llama.cpp Vulkan backend runs inference on the Apple M4 Max GPU. It uses the Mesa Honeykrisp/Borealis Vulkan driver and the Asahi DRM kernel driver.
J616c, chipT604126B131,mBoot-20457.41.3f828324018296f80cb95dfd9bc8d13dfea772495, upstream based3c3e242cQwen3.6-35B-A3B-UD-Q4_K_XL(MoE with Gated DeltaNet) andQwen3.8-27B-Q8_0(dense hybrid with dynamic F32 LoRA)The tables give measured throughput in tokens per second. Prompt processing measures input tokens. Generation measures output tokens. MTP means multi-token prediction.
Qwen3.6-35B-A3B-UD-Q4_K_XL(MoE +Q8_0MTP2)tg128)pp4)pp16)pp32)pp256)pp512)pp1024)pp2048)pp4096)pp8192)Linux throughput was higher in all 13 tests in both configurations (all rows verified on
f82832401+ combined Mesa#354+#413+#440, against macOS 27.2.1 Metal in High Power Modepmset powermode = 2withTCMz ≤ 52 °CSMC cooldowns). The peak configuration used-b 2048 -ub 2048. The matched configuration used-ub 512and-bswhere applicable. With matched-bssettings, the 96-token MTP gain was 4.1% (134.2vs128.9 tok/s).Qwen3.8-27B-Q8_0(Dense 64-layer hybrid +Q8_0MTP2 + dynamic F32 LoRA)tg128, base)pp512, base)pp1024, base)pp2048, base)pp4096, base)pp8192, base)r=16F32 LoRA, prefillr=16F32 LoRA, decodegen_lora_twinat ~0.6–2.1% LoRA cost)Install the test software
1. Select an installation
If Linux already runs on the J616c, continue with step 2.
Aurora lists the J616c installer as experimental. The documented J616c boot used separate builds of Aura and Linux. A full test of this installation procedure on J616c is still necessary. J616c installer test record
2. Identify the board and firmware
uname -ato record the kernel version.tr '\0' '\n' </proc/device-tree/compatible. Look forapple,j616candapple,t6041.FW_BUILDto the build number from that record.26B131(27.2.1)26B123(27.2)3. Get the source code
Use an empty build directory. Run steps 3 and 4 in the same Bash session. The commands combine all three Mesa changes and keep the regression tests. For
26B131, they also combine both Linux changes.Record the output of
git rev-parse HEAD 'HEAD^{tree}'in each source directory. If the firmware differs from26B131, record whether the installed kernel includes Linux ggml-org#1951.4. Build and install the platform packages
For Aurora ALARM or Omarchy Aurora, use an ARM64 build machine with Python 3.11 or later. Use a regular account with access to the distribution package repositories. The package definitions supply the kernel configuration, Rust tools, J616c device tree and Mesa tests. Aurora package definitions
mesa-auroraandvulkan-aurorafromalarm-aurora/out/packages/.26B131, also select the locallinux-auroraandaurapackages.pacman -U.t6041-j616c.dtb.uname -a.For Aurora NixOS, use the same source revisions in the kernel, Aura and Mesa package definitions. Follow the NixOS Stage 2 procedure. Aurora NixOS · Stage 2 updates and recovery
5. Build llama.cpp and check the GPU
After the restart, open Bash in the build directory from step 3. Set
ROOT=$PWD.sudo pacman -S --needed base-devel git cmake ninja vulkan-headers vulkan-icd-loader vulkan-tools shaderc spirv-headers openssl.vulkaninfo --summary. Make sure the output lists the Apple GPU. The nameslavapipeandllvmpipeidentify software drivers.vulkaninfoforVK_KHR_cooperative_matrix.GL_KHR_cooperative_matrixsupport.llama-bench --list-deviceslists the devices, setDEVto the Apple GPU identifier.6. Measure prompt processing and token generation
Use the same GGUF model file for each test. The commands check the file against its SHA-256 checksum. The original test record identifies the model by name. The original checksum and the full server command are unknown. Model file for this procedure
Check the benchmark log for the Apple GPU, the number of layers on the GPU, and the GPU buffers. Record input and output tokens per second separately.
7. Measure speculative generation
Start the server in the same Bash session. This command permits up to two draft tokens.
In a second terminal, run both tests with the same prompt.
timings.predicted_per_secondand the output token count for each test.--spec-type none. Omit--spec-draft-n-max 2.For each comparison, record the firmware, kernel, device tree, Mesa version, llama.cpp commit, model checksum and power mode. Record all non-default
GGML_*andHK_*settings. Use a separate run for diagnostics such asGGML_VK_GRAPH_STATS=1.Changes to llama.cpp
GGML_VK_MMID_SELECT=pairs. Matrix multiplication and FlashAttention support native 8×8×8 cooperative matrices with FP32 accumulation.GGML_VK_FENCE_WAIT=blockselects a blocking wait.GGML_VK_GRAPH_STATS=1enables graph statistics.SET_ROWS. It can also combine a matrix-vector product withADD,MULorSIGMOID.GGML_VK_BATCH_INVARIANT=1selects batch-independent verification. The reported tests covered batch sizes from 1 to 8. The backend can omit redundant barriers within a submission.GGML_VK_SINGLE_FRONT=1enables command buffer reuse. The backend reuses temporary buffers. It can combine several additions with RMS normalization and multiplication.Q8_0roofline and dynamic LoRAKQ8_0matrix-vector products use 4-block (128-element) unrollediter4_aligned_q8_0paths with 128-thread workgroups.Q8_08×8×8 cooperative-matrix kernels use 2-cycle IEEE-754 FP16 magic-bias dequantization and hoistedsh_b=12. Dynamic LoRA adapters fuse[mm, lora_b, scale, add]into lazy on-demand 11-bindinggen_lora_twinpipelines with zero overhead on non-LoRA pipelines.Correctness test results
test-backend-opson the test system.GGML_VK_BATCH_INVARIANT=1, the maximum absolute difference between single-token logits and verification logits was zero fork=3,4.GGML_VK_EXACT_S8=1, modes0,1and4produced identical logits in the reported test.