Skip to content

Improve Vulkan inference on the Apple M4 Max GPU - #1

Draft
astaff wants to merge 10 commits into
masterfrom
agx/m4-max-series8
Draft

astaff wants to merge 10 commits into
masterfrom
agx/m4-max-series8

Conversation

@astaff

@astaff astaff commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Pull requests for this build

Pull requests Function
Linux #1951 GPU activity requests memory controller state DCS 5 and fabric state 3. The performance tests used this change.
Mesa #354, Mesa #413 and Mesa #440 The performance tests used all three changes. ggml-org#354 limits matrix register use and controls which GPU commands can overlap. ggml-org#413 improves shader code and command submission. ggml-org#440 adds /dev/dma_heap/system PRIME backing and scales the UMA heap to 82% (52.3 GiB on 64 GB).
Aura #373, Aura #265 and Linux #1954 These changes work together on macOS 27.2 (26B123 / 26B131) firmware. Aura passes the 3-segment GPU firmware and display handoff data to Linux, and Linux enables the 27.2 AOP, ANE, GPU, and DCP firmware interfaces.
Aura #68, U-Boot #42, Linux #525 and Mesa #85 These merged changes add boot support for the J616c and GPU support for all four shader clusters.
Mesa #275, #343, #350 and #355 The Mesa base includes these changes. They support 8×8 cooperative matrices, overlapping dispatches, BF16, and matrix products with mixed FP16 and FP32 inputs.
Linux #994 and Aurora Installer #97 ggml-org#994 supplies the J616c device tree for the installer. ggml-org#97 gives the installer test status for this board.

Test system and results

This pull request changes how the llama.cpp Vulkan backend runs inference on the Apple M4 Max GPU. It uses the Mesa Honeykrisp/Borealis Vulkan driver and the Asahi DRM kernel driver.

Item Test configuration
Computer 16-inch MacBook Pro, board J616c, chip T6041
GPU and memory 40 GPU cores, 64 GB of unified memory
Boot firmware macOS 27.2.1, build 26B131, mBoot-20457.41.3
llama.cpp Commit f828324018296f80cb95dfd9bc8d13dfea772495, upstream base d3c3e242c
Models Qwen3.6-35B-A3B-UD-Q4_K_XL (MoE with Gated DeltaNet) and Qwen3.8-27B-Q8_0 (dense hybrid with dynamic F32 LoRA)

The tables give measured throughput in tokens per second. Prompt processing measures input tokens. Generation measures output tokens. MTP means multi-token prediction.

Qwen3.6-35B-A3B-UD-Q4_K_XL (MoE + Q8_0 MTP2)

Test Linux Vulkan macOS 27.2.1 Metal Linux gain
Generation, 128 tokens (tg128) 96.04 90.44 +6.2%
Chat generation, mean 93.00 87.90 +5.8%
MTP, 2 draft tokens, 96 output tokens 134.2 133.0 +0.9%
MTP, 2 draft tokens, 512 output tokens 116.9 113.6 +2.9%
Prompt, 4 tokens (pp4) 224.0 213.6 +4.9%
Prompt, 16 tokens (pp16) 453.0 386.9 +17.1%
Prompt, 32 tokens (pp32) 653.0 510.0 +28.0%
Prompt, 256 tokens (pp256) 1593.1 1401.6 +13.7%
Prompt, 512 tokens (pp512) 1837.0 1649.9 +11.3%
Prompt, 1024 tokens (pp1024) 1984.4 1788.6 +10.9%
Prompt, 2048 tokens (pp2048) 2062.2 1828.6 +12.8%
Prompt, 4096 tokens (pp4096) 1997.3 1765.2 +13.1%
Prompt, 8192 tokens (pp8192) 1867.1 1663.4 +12.2%

Linux throughput was higher in all 13 tests in both configurations (all rows verified on f82832401 + combined Mesa #354 + #413 + #440, against macOS 27.2.1 Metal in High Power Mode pmset powermode = 2 with TCMz ≤ 52 °C SMC cooldowns). The peak configuration used -b 2048 -ub 2048. The matched configuration used -ub 512 and -bs where applicable. With matched -bs settings, the 96-token MTP gain was 4.1% (134.2 vs 128.9 tok/s).

Qwen3.8-27B-Q8_0 (Dense 64-layer hybrid + Q8_0 MTP2 + dynamic F32 LoRA)

Test Linux Vulkan macOS 27.2.1 Metal Linux gain / note
Generation, 128 tokens (tg128, base) 17.53 16.72 +4.8% (90.04% of 529.5 GB/s GPU DRAM ceiling)
MTP, 2 draft tokens, 96 output tokens 33.55 32.36 +3.7% (1.93× plain, 0-ULP bit-exact)
MTP, 2 draft tokens, 512 output tokens 33.98 32.81 +3.6% (1.95× plain, 0-ULP bit-exact)
Prompt, 512 tokens (pp512, base) 273.9 263.2 +4.1%
Prompt, 1024 tokens (pp1024, base) 273.4 263.4 +3.8%
Prompt, 2048 tokens (pp2048, base) 273.5 261.5 +4.6%
Prompt, 4096 tokens (pp4096, base) 269.5 253.2 +6.4%
Prompt, 8192 tokens (pp8192, base) 263.8 240.0 +9.9%
4096-token prompt + r=16 F32 LoRA, prefill 265.3 245.3 +8.1% (65/65 GPU layers, 131K ctx, 0 swap)
4096-token prompt + r=16 F32 LoRA, decode 16.83 16.44 +2.4% (fused gen_lora_twin at ~0.6–2.1% LoRA cost)

Install the test software

1. Select an installation

If Linux already runs on the J616c, continue with step 2.

Installation Instructions
Aurora Linux Check the J616c entry in the supported models list. Open Aurora Installer on macOS. Follow its Recovery and first boot instructions. Supported models · Installer releases
Omarchy desktop If Aurora Installer offers a signed Omarchy Aurora image for J616c, use that image. It uses the Aurora Arch Linux ARM (ALARM) packages. The standalone Omarchy Mac installer excludes M4 models. Omarchy Aurora · Omarchy Mac supported models

Aurora lists the J616c installer as experimental. The documented J616c boot used separate builds of Aura and Linux. A full test of this installation procedure on J616c is still necessary. J616c installer test record

2. Identify the board and firmware

  1. On Linux, run uname -a to record the kernel version.
  2. Run tr '\0' '\n' </proc/device-tree/compatible. Look for apple,j616c and apple,t6041.
  3. Read the boot firmware build from the installation record or the boot log. The macOS host version alone does not identify the Linux boot firmware.
  4. In Bash, set FW_BUILD to the build number from that record.
Boot firmware Instructions
26B131 (27.2.1) Use the paired Aura and Linux changes on a J616c that already boots with this firmware. This is the firmware for the performance results.
26B123 (27.2) Keep the Aura and Linux versions for this board and firmware. Check that the GPU can run Vulkan. Aurora uses this build for its production boot stub. Maintainer review
Other build Keep the Aura and Linux versions for that firmware. Record the build number with the test results.

3. Get the source code

Use an empty build directory. Run steps 3 and 4 in the same Bash session. The commands combine all three Mesa changes and keep the regression tests. For 26B131, they also combine both Linux changes.

set -euo pipefail
: "${FW_BUILD:?set FW_BUILD to the verified boot firmware build}"
ROOT=$PWD
kernel=511576514e0f3ae314a0746eaebf4d7cb88e78ef
fwkernel=e538e576af342201907f90eac3096b338c1832c5
aura=a7fee76fd421c41ea526e70949764f3f2e11a3d2
dispatch=1142bcb91d0a05b63e2a6f78117a1faf5afd4e8f
shader=3c643c4862814400e3ffc5bfa07ff6fdd0afa1fb
dmaheap=644ce6408064e089a7918abac6bed6bcd10571cb
common=57df4db849aac8f3a2815bb0875ed65868b01246
llama=f828324018296f80cb95dfd9bc8d13dfea772495

checkout() {
    local dir=$1 repo=$2 first=$3
    shift 3
    git init -q "$dir"
    git -C "$dir" remote add origin "https://github.com/$repo.git"
    git -C "$dir" fetch --no-tags --depth=32 origin "$first" "$@"
    git -C "$dir" switch -c repro "$first"
    git -C "$dir" config user.name 'Local reproduction'
    git -C "$dir" config user.email 'build@localhost'
}

if [[ $FW_BUILD == 26B123 || $FW_BUILD == 26B131 ]]; then
    checkout linux astaff/linux "$kernel" "$fwkernel"
    test "$(git -C linux merge-base "$kernel" "$fwkernel")" = 67dd90873461454284b05aa3fd64c9820d2c538e
    git -C linux merge --no-ff --no-edit "$fwkernel"
    checkout aura astaff/aura-fork "$aura"
fi

checkout mesa astaff/mesa "$dispatch" "$shader" "$dmaheap"
(
    cd mesa
    test "$(git merge-base "$dispatch" "$shader")" = "$common"
    test "$(git merge-base "$dispatch" "$dmaheap")" = "$common"
    git merge --no-ff --no-edit "$shader"
    git merge --no-ff --no-edit "$dmaheap"
)
checkout llama.cpp astaff/llama.cpp "$llama"

Record the output of git rev-parse HEAD 'HEAD^{tree}' in each source directory. If the firmware differs from 26B131, record whether the installed kernel includes Linux ggml-org#1951.

4. Build and install the platform packages

For Aurora ALARM or Omarchy Aurora, use an ARM64 build machine with Python 3.11 or later. Use a regular account with access to the distribution package repositories. The package definitions supply the kernel configuration, Rust tools, J616c device tree and Mesa tests. Aurora package definitions

checkout alarm-aurora aurora-silicon/alarm-aurora b5bddfdcee1e54795a239796418c2ad90554e2fa
python3 - "$ROOT" "$FW_BUILD" <<'PY'
import hashlib, pathlib, re, subprocess, sys
root = pathlib.Path(sys.argv[1])
packages = root / "alarm-aurora/packages"

def pin(repo):
    return subprocess.check_output(["git", "-C", root / repo, "rev-parse", "HEAD"], text=True).strip()

def archive(repo, prefix, output):
    subprocess.run(["git", "-C", root / repo, "archive", "--format=tar.gz",
                    "--prefix=" + prefix + "/", "-o", output, "HEAD"], check=True)
    with output.open("rb") as f:
        return hashlib.file_digest(f, "sha256").hexdigest()

def update(pkg, rev, edits):
    path = packages / pkg / "PKGBUILD"
    data, count = re.subn(r"(?m)^_commit=[0-9a-f]{40}$", "_commit=" + rev, path.read_text())
    assert count == 1, path
    for old, new in edits:
        assert old in data, (path, old)
        data = data.replace(old, new)
    path.write_text(data)

m = pin("mesa")
mh = archive("mesa", "mesa", packages / "mesa-aurora/mesa-src.tar.gz")
update("mesa-aurora", m, [('mesa::git+$url.git#commit=$_commit', 'mesa-src.tar.gz'),
                           ("sha256sums=('SKIP')", "sha256sums=('" + mh + "')")])
if sys.argv[2] == "26B131":
    k = pin("linux")
    kh = archive("linux", "linux-" + k, packages / ("linux-aurora/linux-" + k + ".tar.gz"))
    old = re.search(r"(?m)^_sha256=([0-9a-f]{64})$", (packages / "linux-aurora/PKGBUILD").read_text())[1]
    update("linux-aurora", k, [('::https://codeload.github.com/aurora-silicon/linux/tar.gz/${_commit}', ''), (old, kh)])
    update("aura", pin("aura"), [('git+$url.git', 'git+https://github.com/astaff/aura-fork.git')])
PY
(
    cd alarm-aurora
    if [[ $FW_BUILD == 26B131 ]]; then
        bash scripts/build-package aura
        bash scripts/build-package linux-aurora
    fi
    bash scripts/build-package mesa-aurora
)
  1. Keep a working boot entry for recovery.
  2. Select mesa-aurora and vulkan-aurora from alarm-aurora/out/packages/.
  3. If the boot firmware is 26B131, also select the local linux-aurora and aura packages.
  4. Install the selected files together with pacman -U.
  5. Keep the distribution's U-Boot and the firmware and calibration data for this computer.
  6. Use the distribution's Stage 2 update procedure for the kernel and t6041-j616c.dtb.
  7. Restart the computer.
  8. Check the active kernel version with uname -a.

For Aurora NixOS, use the same source revisions in the kernel, Aura and Mesa package definitions. Follow the NixOS Stage 2 procedure. Aurora NixOS · Stage 2 updates and recovery

5. Build llama.cpp and check the GPU

After the restart, open Bash in the build directory from step 3. Set ROOT=$PWD.

  1. On ALARM, install the build tools with sudo pacman -S --needed base-devel git cmake ninja vulkan-headers vulkan-icd-loader vulkan-tools shaderc spirv-headers openssl.
  2. Run vulkaninfo --summary. Make sure the output lists the Apple GPU. The names lavapipe and llvmpipe identify software drivers.
  3. Check the Apple GPU entry in vulkaninfo for VK_KHR_cooperative_matrix.
  4. Build llama.cpp with the commands below. Check the CMake output for GL_KHR_cooperative_matrix support.
  5. When llama-bench --list-devices lists the devices, set DEV to the Apple GPU identifier.
cd "$ROOT/llama.cpp"
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
    -DGGML_VULKAN=ON -DLLAMA_BUILD_TESTS=ON
cmake --build build --target llama-bench llama-server test-backend-ops
build/bin/llama-bench --list-devices
DEV=Vulkan0  # set to the Apple GPU identifier printed above
build/bin/test-backend-ops test -b "$DEV" \
    -o RMS_NORM,RMS_NORM_MUL_ADD,ADD,MUL_MAT,MUL_MAT_ID,FLASH_ATTN_EXT,GATED_DELTA_NET

6. Measure prompt processing and token generation

Use the same GGUF model file for each test. The commands check the file against its SHA-256 checksum. The original test record identifies the model by name. The original checksum and the full server command are unknown. Model file for this procedure

MODEL="$ROOT/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf"
curl -fL -C - -o "$MODEL" \
    'https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/resolve/b8efbbece76dc9103a6a45b7ed95b26c3932fc5f/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf'
printf '%s  %s\n' 55983c5a75a1ab969824077b3bb3de4146e82a9234072b48ad4e8f92ad3fe9f1 "$MODEL" | sha256sum -c -
for ub in 2048 512; do
    build/bin/llama-bench -v -m "$MODEL" -dev "$DEV" -ngl 99 -fa on \
        -p 4,16,32,256,512,1024,2048,4096,8192 -n 128 -b 2048 -ub "$ub" -r 5 \
        -o json > "$ROOT/bench-ub$ub.json" 2> "$ROOT/bench-ub$ub.log"
done

Check the benchmark log for the Apple GPU, the number of layers on the GPU, and the GPU buffers. Record input and output tokens per second separately.

7. Measure speculative generation

Start the server in the same Bash session. This command permits up to two draft tokens.

build/bin/llama-server -m "$MODEL" -dev "$DEV" -ngl 99 -fa on \
    -c 8192 -b 2048 -ub 2048 --spec-type draft-mtp --spec-draft-n-max 2 \
    --host 127.0.0.1 --port 8080

In a second terminal, run both tests with the same prompt.

for n in 96 512; do
    curl -fsS http://127.0.0.1:8080/completion -H 'Content-Type: application/json' \
        -d "{\"prompt\":\"Explain how unified memory affects inference.\",\"n_predict\":$n,\"temperature\":0,\"seed\":1,\"ignore_eos\":true,\"cache_prompt\":false}" \
        > "mtp2-$n.json"
done
  1. Save the server log with the two JSON files.
  2. Record timings.predicted_per_second and the output token count for each test.
  3. If the log gives the number of accepted draft tokens, record that number.
  4. For the control test, restart the server with --spec-type none. Omit --spec-draft-n-max 2.
  5. Change the output file names for the control test.
  6. Repeat both requests with the same prompt, sampling settings and warm-up procedure.

For each comparison, record the firmware, kernel, device tree, Mesa version, llama.cpp commit, model checksum and power mode. Record all non-default GGML_* and HK_* settings. Use a separate run for diagnostics such as GGML_VK_GRAPH_STATS=1.

Changes to llama.cpp

Component Change
GPU and memory The backend identifies the Apple GPU. It can use host memory without a copy. It selects tiles for expert operations with GGML_VK_MMID_SELECT=pairs. Matrix multiplication and FlashAttention support native 8×8×8 cooperative matrices with FP32 accumulation.
Command submission The backend can submit more graph nodes together. GGML_VK_FENCE_WAIT=block selects a blocking wait. GGML_VK_GRAPH_STATS=1 enables graph statistics.
Gated DeltaNet The backend uses exact FP32 values for the decay and beta gates. It writes the recurrent state in place.
Combined operations The backend combines operations for SwiGLU, Gated DeltaNet normalization and output gates, and M-RoPE with SET_ROWS. It can also combine a matrix-vector product with ADD, MUL or SIGMOID.
Matrix-vector products The kernels compute paired gate and up projections. They divide rows between subgroups and reuse rows.
Speculative verification GGML_VK_BATCH_INVARIANT=1 selects batch-independent verification. The reported tests covered batch sizes from 1 to 8. The backend can omit redundant barriers within a submission.
Command buffers GGML_VK_SINGLE_FRONT=1 enables command buffer reuse. The backend reuses temporary buffers. It can combine several additions with RMS normalization and multiplication.
GPU kernels The backend adds 8×8×8 matrix-vector kernels, single-wave Gated DeltaNet and tiled state space model (SSM) convolution. FlashAttention uses 256 threads and up to 16 KB of shared memory per workgroup.
Expert operations and sampling Wide tiles use FP16 shared memory. Normalization can produce FP32 and FP16 output together. The backend adds a 512-thread argmax and a fast path for greedy speculative sampling.
Q8_0 roofline and dynamic LoRA Wide-K Q8_0 matrix-vector products use 4-block (128-element) unrolled iter4_aligned_q8_0 paths with 128-thread workgroups. Q8_0 8×8×8 cooperative-matrix kernels use 2-cycle IEEE-754 FP16 magic-bias dequantization and hoisted sh_b=12. Dynamic LoRA adapters fuse [mm, lora_b, scale, add] into lazy on-demand 11-binding gen_lora_twin pipelines with zero overhead on non-LoRA pipelines.

Correctness test results

  • All seven operations in step 5 passed test-backend-ops on the test system.
  • With GGML_VK_BATCH_INVARIANT=1, the maximum absolute difference between single-token logits and verification logits was zero for k=3,4.
  • With GGML_VK_EXACT_S8=1, modes 0, 1 and 4 produced identical logits in the reported test.

…election, and 8x8x8 coopmat1

Add device detection and matmul/MoE tile tuning for Apple AGX GPUs (M4 Max
T6041 under Borealis and Mesa Honeykrisp):

- Detect Borealis on Apple M4 Max (0x10005 / 0x6041) with GGML_VK_BOREALIS_TUNING
  (off|tiles|roll|all, default all) and disable SPIR-V BK loop unrolling on
  coopmat1 pipelines unless GGML_VK_CM1_ROLL_BK=1.
- Add 8x8x8 VK_KHR_cooperative_matrix (f32matmad8x8) s/m/l dense and MoE
  warptiles sized for the 192-half-reg AGX G17P register file (<=16 8x8
  accumulators per subgroup, BN=16 small MoE tile), subgroup-scoped barriers
  around coopMatStore, and guarded shared-memory bounds in mul_mm.comp.
- Default disable_narrow_mul_mat on Apple AGX (with GGML_VK_DISABLE_NARROW_MUL_MAT=0|1
  override) so 8x8x8 coopmat1 retains its tuned s/m/l tiles and split_k=1.
- Select MUL_MAT_ID tiles by average rows per expert (GGML_VK_MMID_SELECT=pairs,
  default on Apple AGX coopmat1) with a dedicated matmul_id_tile_selector, and
  prefer FP16 B activations on AGX coopmat1 when nei1 > 32 (GGML_VK_MMID_F16_B).
- Add tile/dispatch tuning knobs GGML_VK_CM1_TILES, GGML_VK_CM1_ID_TILES,
  GGML_VK_MMQ_S/M/L, GGML_VK_MMQID_S, GGML_VK_MMVID_MAX, GGML_VK_MMVID_ONE,
  and GGML_VK_DISABLE_INTEGER_DOT_PRODUCT.

Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…gnostics

Add configurable fence waiting, command-buffer submission policies, and
env-gated graph instrumentation for CPU/GPU synchronization profiling:

- Add GGML_VK_FENCE_WAIT=spin|block|wfe|nap|hybrid (with GGML_VK_FENCE_MARGIN_US,
  default 300 us) to control final graph fence waiting, including AArch64 WFE
  low-power polling and EMA-based hybrid sleep-then-spin after the almost-ready
  fence.
- Add GGML_VK_SUBMIT_MODE=default|one|geo (with GGML_VK_SUBMIT_FIRST and
  GGML_VK_SUBMIT_GROWTH) and GGML_VK_FLOPS_PER_SUBMIT to control command-buffer
  submission granularity.
- Add GGML_VK_GRAPH_STATS=1|2 (per-graph dispatch/barrier/submit/record/wait
  counters and per-dispatch pipeline+workgroup trace) and GGML_VK_DUP_OPS for
  non-in-place kernel timing ablations.

Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…taNet recurrent state

Add lossless BF16 storage for BF16-exact F32 gate weights and in-place Gated
DeltaNet (GDN) recurrent state updates with speculative rollback safety:

- Under GGML_GEN_BF16_EXACT=1, inspect F32 router, shared-expert gate, and GDN
  alpha/beta weights (.ffn_gate_inp.weight, .ffn_gate_inp_shexp.weight,
  .ssm_alpha.weight, .ssm_beta.weight, .ssm_beta_alpha.weight, .ssm_ba.weight)
  at load time; when every element has zero low 16 mantissa bits, store the
  tensor as BF16 and cast back to F32 above GGML_GEN_BF16_CAST_ABOVE columns
  (default 8) during graph build.
- Under GGML_GEN_GDN_INPLACE=1, view recurrent cache rows directly in build_rs
  when the sequence state mapping is the identity (skipping GET_ROWS gather)
  while guarding against speculative rollback snapshot consumption when
  n_rs_seq != 0.
- Add STATE_OUT variants of gated_delta_net.comp so single-step GDN writes its
  updated recurrent state directly into the destination cache slice and elides
  the trailing CPY node.

Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…logue fusions

Add Numerically Exact (0-ULP) RMS_NORM reductions and Tier-1 operator fusions
gated by GGML_GEN_FUSE (all or comma-separated tokens):

- Replace the non-subgroup shared-memory tail in rms_norm.comp with an exact
  0-ULP subgroupShuffleXor tree reduction when gl_SubgroupSize == 32 and skip
  the redundant initial workgroup barrier.
- Add lazy _glu mat-vec/mat-vec-id twin pipelines (specialization constant 3 = 1)
  so base mat-vec shaders incur zero extra register pressure.
- Support GGML_GEN_FUSE=glu (mat-vec / mat-vec-id + SwiGLU), l2 (deferred
  RMS_NORM + SCALE with precise IEEE rounding for GDN L2 norm), gate
  (RMS_NORM * weight * SiLU(z) in one dispatch), mrope (M-RoPE / IM-RoPE in
  RMS_NORM_MUL_ROPE), mmgate (mat-vec + sigmoid and mat-vec + bias + softplus +
  mul), and shexp (mat-vec + per-token row scale).

Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…roup DMMV, and outer row reuse

Add row-split multi-subgroup dequant-mat-vec kernels, 0-ULP quantized fast
paths, and paired mat-vec / mat-vec-id twin pipelines:

- Add row-split multi-subgroup DMMV (local_size_y specialization constant 4 via
  GGML_VK_DMMV_NSG / GGML_VK_DMMV_ID_NSG), workgroup row/size knobs
  (GGML_VK_RM_STDQ/_ID, GGML_VK_RM_KQ/_ID, GGML_VK_DMMV_LARGE/_ID_LARGE), and
  0-ULP Q8_0, Q4_K, Q5_K, and Q6_K vectorization / outer row reuse
  (GGML_VK_FAST_KQ).
- Split paired mat-vec pipelines into lazy _pairmm (specialization constant
  3 = 2) and _pairglu (3 = 3) twins so standalone and single-epilogue shaders
  keep minimum register footprint.
- Support GGML_GEN_FUSE=gateup (single-dispatch gate + up + 0-ULP SwiGLU for
  MUL_MAT and MUL_MAT_ID) and GGML_GEN_FUSE=pairmm (sibling MUL_MAT sharing
  src1, including trailing sigmoid on the second output), with graph_optimize
  pairing and buffer lifetime dependencies.

Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…ubmit barrier elision

Add 0-ULP batch-invariant speculative verification mode and Series 8 graph
scheduling / intra-submit barrier elision:

- Under GGML_VK_BATCH_INVARIANT=1, size Flash Attention KV splits (including
  GQA ratios up to k <= 8 query tokens) identically to n=1, extend ADD+RMS_NORM
  partials to multi-row verification batches with an n=1 per-row 1 KB budget,
  and enable MUL_MAT + ADD fusion on the multi-column mat-vec path; also add
  GGML_VK_DISABLE_ADD_RMS.
- Under GGML_GEN_FUSE=s8 (or all), fuse CONT + [view] + UNARY + MUL into a
  single strided unary_mul dispatch, lift external views out of
  RMS_NORM_MUL_SILU subgraphs, and co-schedule independent dispatches in
  ggml_vk_graph_optimize (1-workgroup TOPK_MOE with shared-expert gate+up+glu,
  ffn_moe_down with ffn_shexp so trailing residual ADDs coalesce into
  MULTI_ADD, and mixed-quant GDN projections) with explicit add_alloc_dep
  buffer lifetime constraints to elide intra-submit pipeline barriers.

Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…d RMSNorm fusion

Series 9-11 optimizations for Apple AGX (Honeykrisp / M4 Max):
- Add per-graph Vulkan command-buffer replay cache for decode and speculative
  verification graphs with stable tensor buffer bindings.
- Add 8-slot prealloc_y and prealloc_split_k ring buffers to eliminate intra-graph
  pipeline drain syncs across quantize_y and split-k dispatches.
- Add multi_add_rms_mul.comp fusing residual ADD chains with RMS_NORM + MUL,
  fused ADD+SCALE, fused MoE top-k + scatter/gather epilogues, and fast FP16
  dequantization/accumulation paths.

Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
@astaff
astaff force-pushed the agx/m4-max-series8 branch from 5d80a43 to 1a6738a Compare November 17, 2026 05:38
@astaff astaff changed the title vulkan, llama: Apple AGX (M4 Max / Borealis & Honeykrisp) Series 8 inference stack vulkan, llama, common: Apple AGX (M4 Max / Honeykrisp) Series 14 inference stack Nov 17, 2026
…WG=256 FlashAttention

- Add mul_mv_mma.comp (8x8x8 simdgroup_matrix mat-vec/multi-col and grouped
  expert mul_mv_mma_id with fused paired-SwiGLU and expert-scale epilogues)
  and fused cm1 matmul/matmul_id epilogue variants (mul_mm_load_a_body.glsl).
- Specialize single-wave subgroup shuffle gated_delta_net.comp (S_V <= 128)
  and tiled shared-memory ssm_conv.comp (NC == 4) with fused SiLU output.
- Reduce 8x8x8 coopmat1 FlashAttention shared memory to <= 16 KB for dual
  workgroup occupancy per core (WG=256, 8 subgroups), add LPT GQA head/tile
  swizzle (Flags bit 256), and compact mask tiles for partial K blocks.

Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
…hread argmax, and greedy speculative fastpath

- Add wide-tile RT=4/8 mul_mv_mma / mul_mv_mma_id variants with FP16
  shared-memory activation staging (SHB=6, NT=1 for RT>=4 to eliminate AGX
  register spills on long-context prefill) and paired expert count_experts.
- Add vec4 RMSNorm fast paths (32-lane 0-barrier d=128 QK-norm, dual F32+F16
  outputs in rms_norm.comp and multi_add_rms_mul.comp to elide standalone
  cpy_f32_f16 passes before coopmat1 matmul) and fused 2-row Q+K RMSNorm.
- Widen argmax.comp to 512 threads for large vocabularies (ne00 >= 65536) and
  optimize single-wave GDN/SSM kernels for multi-token prefill.
- Add greedy common_sampler_sample_greedy() fastpath and skip redundant logit
  copies during greedy speculative decoding (MTP / draft verification).

Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
@astaff
astaff force-pushed the agx/m4-max-series8 branch from 1a6738a to 0588996 Compare November 17, 2026 06:02
@astaff astaff changed the title vulkan, llama, common: Apple AGX (M4 Max / Honeykrisp) Series 14 inference stack Improve Vulkan inference on the Apple M4 Max GPU Nov 17, 2026
…used LoRA twins

- Add 4-block (128-element) unrolled iter4_aligned_q8_0 / iter2_aligned_q8_0
  mat-vec paths paired with 128-thread (4-subgroup) DMMV_WG_SIZE_LARGE routing
  for wide-K Q8_0 projections (reaching 90.04% of the 529.5 GB/s M4 Max DRAM
  ceiling on Qwen3.8-27B-Q8_0).
- Replace unpack8 int-to-float Q8_0 coopmat conversions in mul_mv_mma.comp with
  2-cycle IEEE-754 FP16 magic-bias extraction ((| 0x64006400u) - 1152.0h),
  hoist sh_b=12 on paired SwiGLU, lower GGML_VK_MMA_Q8_MIN_N to 8 (M <= 65536),
  and raise GGML_VK_MMA_MAX_N to 4096.
- Hoist [lora_a] projections in ggml_vk_graph_optimize and fuse [mm, lora_b,
  scale, add] into lazy on-demand 11-binding gen_lora_twin pipelines with zero
  binding or register overhead on non-LoRA pipelines.
- Refine common_sampler_init pure_argmax_fast detection across default sampler
  chains when penalty/DRY/XTC parameters are inactive.

Signed-off-by: Artyom Astafurov <artyom.astafurov@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant