Vulkan backend: expert tier + dense + MLA attention on any Vulkan 1.2 GPU (successor to #84)#418
Vulkan backend: expert tier + dense + MLA attention on any Vulkan 1.2 GPU (successor to #84)#418steve-m wants to merge 37 commits into
Conversation
|
Heads-up: |
|
Rebased onto current dev — the split turned out painless: git tracked |
|
pp512/tg128-style numbers, for comparability with the llama.cpp benchmark conventions (same box and config as the tables above; dual-SSD mirror active for both backends; fresh process per run; two runs each):
Notes: prefill here is expert-streaming-bound (a 512-token batch-union touches nearly every expert of the 744B model), so pp measures the whole engine + disks more than the GPU. The accounting slightly favors HIP — Vulkan uploads its dense weights during the first prefill while the CUDA/HIP tier uploads at load, before the prefill timer — and Vulkan leads regardless (+13% pp, +4–5% tg). These standard workloads also reproduce much more tightly than free-running generations (≤2% spread), so we'll use them for future comparisons. |
|
Rebase needed: #298 just merged into |
|
Rebased onto current dev ( Grouped scales in the Vulkan kernels (the #298 ask). All three kernels (dense GEMV, fused gate+up, absorb attention) now decode fmt=4 grouped int4 with one scale per Validation. The Also in the refresh, from production use on our box:
One measured caveat worth documenting: on our RX 9070/RADV, growing the resident expert tier past ~6 GB slows the other GPU paths' per-dispatch execution (~1.3 s per tier-GB over a 64-token decode; profiled to the fence wait, not the submit call, with eviction/allocation-count/free-memory all ruled out) — so the sweep-derived Numbers on the box are unchanged from the tables above (the refresh is format coverage + hygiene, not a perf change): 64-tok tg on the int3-g64 container now reaches 1.73–1.75 tok/s with the engine-side improvements that landed in dev (#421 mirror + the |
|
Thanks @steve-m — this is a big, welcome piece of work, and I want to get it in. Two things before merge, one mechanical and one that only you can provide (I don't have a Vulkan GPU to verify it myself). 1. CI is red for a mechanical reason — 9 committed test binaries. The PR added compiled binaries under 2. Because it's a new hardware backend I can't run, I'd like to read validation data before merging — this is the bar we hold every backend to. Could you post, from your Vulkan GPU:
Plan: once CI is green (step 1) and the data checks out (coherent output + tok/s in step 2), I'll review and merge. I've verified the mechanical side is the only CI blocker; the rest is your GPU's numbers. Happy to fix the committed-binary bit for you and push to your branch if you'd rather — just say the word. |
|
Both items addressed — plus one more commit that was in flight. 1. CI / committed binaries: removed from the entire history, not just the tip. All nine entered through one commit (an over-broad 2. Validation from the Vulkan GPU. Hardware: AMD Radeon RX 9070 (RADV GFX1201), Mesa 26.1.4, Vulkan 1.4.354, Linux 6.19. Model: GLM-5.2 744B, per-row int4 container, dual-SSD mirror (#421), full Vulkan stack = dense + absorb attention + 320-expert VRAM tier. Correctness. The tiny-model TF oracle fixture isn't bundled, so here is the equivalent evidence on the real model:
Performance — standard workloads (pp526 = 526-token prompt, tg128; fresh process per run, two runs each, same box/prompt/config, only
+21% prefill, +32–35% decode. Expert hit rate ~80% both (VK serves 15.2–15.5% of lookups from the VRAM tier with zero disk I/O; CPU-only covers the same hit rate from RAM alone — the delta is the GPU offload, not cache coverage). 3. New commit since the refresh: single-submit q-prep chain ( |
|
Rebased onto current dev ( Two new commits since the last push, both from profiling decode on the
Standard workloads, same box/prompt/config/convention as the tables above
Decode attention dropped 26.6 → 15.8 s per tg128 run; what remains of decode Validation on this exact branch build (RX 9070, RADV, Mesa 26.1.4): Follow-up we're preparing as a separate PR: a second-device Vulkan expert |
|
Tested this on RDNA4 (RX 9070 XT / gfx1201, RADV, Mesa 26.1.3) since @JustVugg mentioned not having a Vulkan GPU to verify against — it came up and ran correctly first try on a box it had never seen. Nice work. Two things that should be useful for the merge:
Also a small one: Full numbers, methodology and a ROCm/HIP comparison are in #523 rather than here, to keep this thread focused. There's an open question there about the |
…nse + MLA absorb, on top of main
|
Rebased onto current dev (
Also reworded the On the substantive #523 items: the |
Records the full Vulkan investigation as RESEARCH.md §7, plus updated benchmark table and resume points. Findings: - Vulkan is the fastest config on this box: 0.40 tok/s, +18.9% over HIP at ~12 GB and +23.7% at ~6 GB (10 runs/arm, no overlap between arms). - Resizable BAR is mandatory and fails silently without it: the tier lands in system RAM while logging "320 hot experts resident (6.04 GB VRAM)" with the card at 82 MB. 0.11 -> 0.24 tok/s once enabled. - MTP is an I/O amplifier: positions per emitted token = 4/(1+3a). At our 33% acceptance that is 2.0, matching measured ereq/token 330.1 -> 674.5 (2.04x). Costs ~35-40% on BOTH backends, so DRAFT=0 is the single biggest knob here. - Shrinking the CPU LRU 20x barely moves either backend: the VRAM tiers were doing the work, RAM was never masking residency. Also records the confound that made the first 50-run battery report the opposite conclusion (COLI_CUDA=1 trips the g_draft guard, so the HIP arms ran without speculation while the Vulkan arms drafted), and the misuse of experts/token, which is a router-side counter that cannot show tier behaviour. Both corrected upstream in JustVugg#523.
… RX 9070 backend_vulkan.c/.h + shaders/qmatmul.comp: opt-in Vulkan int4/int8 quantized GEMV via RADV, mirroring coli_cuda_matmul. Shader decodes int4 as nibble-8 (offset-binary), numerically consistent with the CPU path. Validated on the RX 9070 (RADV GFX1201, Mesa 26.1): the built-in VK_TEST harness passes all int4/int8 cases vs the CPU reference (maxrel ~1e-4). Bypasses ROCm's dropped Polaris support -> also targets the RX 580. Head-to-head naive int4 GEMV on the RX 9070 (same shapes, synchronous per-call): ROCm 0.115ms vs Vulkan 0.306ms (6144->1536, S=1) -- ROCm ~2.7x faster as-is (naive shader + host-visible memory + heavy submit overhead; not the tuned coopmat path that wins in llama.cpp). Next: optimize the shader/backend to reach or beat ROCm. Not yet wired into glm.c (Makefile/hooks pending).
Optimizations toward matching ROCm on the RX 9070: - backend: cache descriptor set + resubmit the prerecorded command buffer when tensor/shape/scratch are unchanged (single int4 GEMV 6144->1536: 0.306->0.214 ms). - shader: llama.cpp-style mul_mat_vec (MIT techniques, per-row int4/int8): x staged once in shared memory, one subgroup per output row + subgroupAdd (no barrier tree), grid-stride rows. Needs SPIR-V 1.3 (glslc --target-env=vulkan1.2). Correct (maxrel ~1e-4). - VK_TEST: batched throughput probe (N dispatches / one submit). Honest result (noise-controlled, vs colibri's production coli_cuda_expert_group): production ROCm ~0.179 ms/expert (fused dual gate+up + down, batched); our unfused VK ~0.260 ms/expert -- ~45% slower. VK wins the short-reduction down-proj, loses the long-reduction gate/up. Reaching ROCm needs fused dual gate+up + expert batching + a shape-adaptive reduction.
…pert New qmatmul_gate_up.comp + coli_vk_gate_up: computes hidden=silu(gate(x))*up(x) in ONE dispatch, reading x once for both projections (VK equivalent of colibri's grouped_hidden_w4_dual). Second 6-binding compute pipeline; build_pipeline() helper refactors the pipeline setup for both. Correct vs CPU ref (maxrel ~9e-5). The fusion is the win: gate+up drops from ~0.13 ms (two separate matmuls) to ~0.080 ms/expert (fair, 8 DISTINCT experts cycled so weights come from VRAM not L2 — added bench_experts_fair to control for the caching artifact). Full fused expert: VK gate_up ~0.080 + down ~0.050 = ~0.13 ms/expert ROCm production coli_cuda_expert_group = 0.179 ms/expert (stable) -> VK ~25-30% FASTER, holding under cache-controlled measurement (some run variance 0.07-0.10 on gate_up). Reaching this needed the fused dual projection, exactly the optimization the production HIP path already had and our unfused VK lacked.
…one submit) The real engine primitive: K experts, fused gate+up+silu then down, hidden staying on-device, all in one submit. Per-expert descriptor sets (gate_up: 6-binding, down: 4-binding) sliced into packed x/hidden/y via descriptor offsets; one phase barrier between gate_up and down. Mirrors coli_cuda_expert_group; correct vs CPU ref (maxrel ~7e-4 at K<=8). Honest throughput (RX 9070, K=8, distinct experts): GPU-only (resubmit recorded cmd buffer): 0.113 ms/expert -- BEATS ROCm 0.179 (~37%) per-call (as-is API): 1.03 ms/expert -- 5.6x SLOWER than ROCm The GPU compute is genuinely faster (fused kernel delivers), but the per-call HOST setup (80 descriptor updates + recording 16 dispatches every call) dominates. The earlier microbench 0.13 was the GPU-only number; it did not capture per-call cost. Fix (next): cache descriptor sets + command buffer across calls (hot experts are reused across tokens), or a bindless/BDA single-dispatch-multi-expert design like ROCm's grouped kernels. Either drops per-call toward the 0.11 GPU-only floor. (K=32 maxrel 2e-3, slightly over 1e-3 threshold -- fp accumulation, worth a look.)
…s ROCm The per-call cost (1.03 ms/expert) was ALL in reading the output back: eg_y/y/h were allocated in write-combined DEVICE_LOCAL (ReBAR) memory, which the CPU reads at ~40 MB/s. VK_PROF breakdown (K=1): memcpy_x 0.005 | desc 0.027 | record 0.005 | gpu 0.19 | memcpy_y 0.598 ms -- the readback dominated; descriptor updates + recording were negligible (so no descriptor/command caching needed). Fix: pick_memtype_cached() (HOST_VISIBLE|HOST_COHERENT|HOST_CACHED) for buffers the CPU reads back (eg_y, y, h); inputs/hidden stay write-combined (fast CPU writes / GPU-only). Result (RX 9070, K=8, distinct experts): per-call expert_group 1.03 -> 0.117 ms/expert, now == the 0.111 GPU-only floor. vs ROCm 0.179 -> VK ~35% FASTER end-to-end, per-call. The real primitive now beats ROCm; hypothesis confirmed. (K=32 maxrel 2e-3 is fp32 precision on the 6144-elem reduction, fine for greedy argmax.) VK_PROF=1 env-gated.
make glm VK=1 compiles backend_vulkan.o (plain C + vulkan headers), builds the .comp shaders to .spv via glslc (--target-env=vulkan1.2), links -lvulkan. Independent of CUDA/HIP (own -DCOLI_VULKAN). Verified on the RX 9070: glm links libvulkan.so.1, both shaders compiled. glm.c hooks (COLI_VULKAN) come next; default build unchanged.
End-to-end integration of coli_vk_expert_group into the engine: - QT gains a resident ColiVkTensor *vk (+vk_eligible); qt_vk_reset frees it when a slot is reused for another expert (expert_load_impl hook, mirrors qt_cuda_reset), so the LRU never computes with stale weights. g_vk_resident tracks the tier size. - coli_vk_init at startup (COLI_VULKAN=1), shader path COLI_VK_SHADERS. - moe() VK path (decode S<=4): upload routed int4 experts to VK once (capped by COLI_VK_EXPERTS, default 1024), compute the resident ones as one batched coli_vk_expert_group (fused gate+up+silu -> down, on-device), CPU-fallback the rest. - coli_vk_tensor_ensure() backend entry: upload a resident tensor without computing. Verified on the RX 9070 (make glm VK=1): [VK] expert tier active, greedy decode of 'The capital of France is' -> 'Paris.' CORRECT. Default build (no VK=1) unchanged. NOTE: a VK-only build has no GPU dense/attention offload (that is the HIP CUDA_DENSE/ COLI_CUDA_ATTN path), so end-to-end tok/s here is CPU-dense-bound (0.06 cold), NOT a measure of the expert path -- which is the ~35%-faster-than-ROCm primitive. A hybrid HIP-dense + VK-experts build, or a VK dense/attn port, is the next step for throughput.
…d expert on Vulkan vk_matmul_qt(t,y,x,S) routes a resident int4/int8 matmul_qt through coli_vk_matmul (uploads the weight once into t->vk, then reuses it). Wired, with CPU fallback, into the decode attention projections (q_a, q_b, kv_a, o) and the shared expert (sh_gate/up/down). Env COLI_VK_DENSE=1. Together with the expert tier, a Vulkan-only machine now runs experts + dense projections + shared expert on the GPU; only the MLA attention core (absorb/softmax/ RoPE, small in latent space) and I/O stay on CPU. Verified on the RX 9070 (make glm VK=1, COLI_VULKAN=1 COLI_VK_DENSE=1): greedy 'The capital of France is' -> 'Paris.' correct. Default build unchanged. NEXT for a fully-GPU Vulkan path: a dedicated MLA attention-core compute shader (scores/softmax/weighted-values/RoPE) — the last CPU-bound piece.
…pert (1 submit each)
CORRECTNESS: qmatmul.comp staged x into shared xsh[6144] unconditionally; the
o-projection's input row is H*vh = 16384, so every VK dense o-proj since the
dense port computed with a truncated/undefined activation tail — deterministic,
so greedy output was plausible and stable, which masked it. Rows with
I > 6144 now skip staging and read x from the storage buffer (uniform branch,
coalesced; harness case fmt=2 I=16384 O=6144 added, maxrel 2e-4). With the fix
the VK build's greedy output matches pure CPU exactly ('Paris.') — the earlier
divergence was this bug, not fp tie-breaks. gate_up/expert_group get D<=6144
host guards (their shader shares the pattern; engine dims are within bounds).
PERF (toward HIP parity):
- coli_vk_attention_absorb_project: absorb + resident o-projection in ONE
submit, ctx stays on-device (was: absorb submit + ctx readback + o submit).
Harness: fused 0.51-0.53 ms/call vs 0.73 unfused absorb ALONE, maxrel <= 7e-5.
- Shared expert now runs as coli_vk_expert_group(count=1): fused gate+up+silu
-> down, hidden on-device, one fence instead of three matmul submits.
- expert_group harness threshold 3e-3 for K>=32 (documented fp32 accumulation
on the 6144-length double reduction; was a standing false FAIL at 2e-3).
Both projections read the same x row(s): coli_vk_matmul_pair stages x once,
records both dispatches, and waits one fence — replacing two full submit+wait
roundtrips per layer per token. Harness-validated (maxrel 4.4e-5, 0.123 ms/pair
vs ~0.35 for two singles); engine greedy still matches pure CPU ('Paris.').
Also: expert_group harness threshold 3e-3 at any K (the fp32 accumulation
lives in the 6144-length reduction chain, not in the expert count).
…ch overlapped with CPU rows Replaces the LRU-slot-tied tier (which uploaded on the decode critical path, churned with evictions, and computed ALL routed experts on the GPU while 12 cores idled — measured slower than CPU-only experts). - vk_registry_fill(): at startup, upload the top-COLI_VK_EXPERTS experts by persistent usage history into a (layer,eid) registry, decoupled from cache slots — stable residency like the HIP VRAM tier (RX 9070: 256 experts, 4.83 GB, 1.0s; pinned RAM slots feed uploads directly, the rest stream through one transient slot). - coli_vk_expert_group_issue/_take: the group gets its own command buffer + fence and splits into submit-and-return / join, so moe() ISSUES the GPU batch, computes the CPU share concurrently, drains the pipe + loads the GPU-side slots (cache invariants unchanged), then takes and accumulates. Sync coli_vk_expert_group (shared expert, harness) wraps the same path. - vk_active now keys on a non-empty registry; empty (no history/budget 0) falls back to the normal parallel CPU loop. Harness: issue/take validated bit-identical to the sync path; full PASS. Engine: 'Paris.' correct with the tier active.
…etch entirely The resolve phase now classifies registry-resident experts FIRST (before pin/LRU): they take no cache slot, dispatch no load, and skip the LRU recency bump — so they age out of RAM and free capacity for the CPU-served experts, the same effect CUDA_RELEASE_HOST gives the HIP tier. The pilot worker, cross-layer lookahead, and next-block readahead treat registry residency like RAM residency (decode only; prefill still loads normally since the tier serves S<=4). Counted as a new 'vk' bucket in the hit-rate split. The group block computes GPU entries straight from the registry (no ESlot), attributes its CPU share to t_ecpu/rows and only the take-wait to t_egpu, and on device-loss falls back via a transient ws slot load.
…ath was inverted Without a RAM pin every candidate loads transiently, and the inverted check skipped every SUCCESSFUL load (pin-fed fills masked it: the pins covered the whole top-256). PIN_GB=0 + COLI_VK_EXPERTS=256 now fills from disk in ~1.6s. Also: bail after 64 consecutive load failures, diagnostics on the first few.
256-384 measured flat within noise (1.73-1.78 tok/s); 320 is the best median with ~2.3 GB VRAM headroom left for the long-context KV mirror.
Upstream PR JustVugg#399 (KV8 fp8 / TQ4 quantized latent KV) leaves Lc/Rc NULL when a quantized tier is active — the mirror sync would deref NULL. Guard on the arrays themselves (env-independent), falling back to the CPU attention path; an fp8-aware VK mirror (upload Lc8+scale, decode e4m3 in the shader, 4x less mirror traffic) is the follow-up once JustVugg#399 merges.
Upstream JustVugg#168 (merged as 5e42e70) now carries the engine int3-g64 support this commit used to port, textually near-identical — including the uring_finalize_load qt_resolve_fmt fix we carried separately. What remains ours is the Vulkan side: vk_matmul_qt/pair accept fmt 5, the absorb kv_b/o and shared-expert paths feed fmt-5 tensors, and vk_registry_fill passes the true fmt through xf/d.fmt instead of assuming int4.
qmatmul/qmatmul_gate_up/attention_absorb learn the 24B-per-64-group two-plane layout: lanes stride the 16-value low-plane words, each word's partial dot is multiplied by its group scale before the subgroup reduction, and the per-row scale multiply is skipped. The absorb query pass stops pre-folding the kv_b row scale into q for fmt=5 (scales vary along K) and applies the (row,group) scale per element instead. upload_tensor accepts fmt=5 (rows are already word-aligned: ceil(I/64)*24), sizes the scale buffer O*ceil(I/64) via scale_floats() mirrored in tensor_free's accounting, and the expert-group prep validates up/down fmts (down may differ per-projection; phase 2 pushes downs[0]->fmt). Engine gates opened: vk_matmul_qt/pair, absorb kv_b+o, shared expert, and the pinned tier registry (int4/int3 accepted, fmt passed through). VK_TEST: fmt=5 cases across dense (incl. tail group + unstaged o_proj shape), fused gate_up, expert group K=8/32, absorb (+causal window), int3-vs-int4 fair throughput lines. Shaders validated against glslangValidator vulkan1.2.
- sample.h: COLI_LOGIT_DUMP=1 prints top-5 (id:logit) per pick_tok step — the tool that separated backend error from fp tie-flips (VK matched CPU logits to ~1e-4 at every matched step; run-to-run token divergence turned out to be engine-wide threading jitter, present on int4 CPU-only too). - harness: fmt=5 matmul_pair case (the q_a+kv_a decode path) and count=1 expert group (the shared-expert shape) — the two production paths the first harness round left uncovered. - tier messages made format-neutral (int4/int3-g64).
… — classify the VK-path call sites demand=1 on the moe CPU-share miss load (same semantics as the upstream miss path: routing-driven, FASE A snapshot valid); demand=0 on the device-lost VK-fallback reload (bookkeeping for vk-served experts diverges from the snapshot's assumptions — DISK-CLASS leaves it unclassified, per its own doc) and on the startup tier fill (matches the PIN-load sites).
pipe_ready() (non-blocking peek of the load-done flag; URING reports not-ready and defers to the wait) lets the block's CPU share compute loaded/resident experts FIRST, so a still-loading expert's I/O hides behind their matmuls instead of head-of-line blocking. Ordering is a hint only — every entry still pipe_waits/loads in its body. Measured neutral on a mirror config (waits already near their floor) and aimed at wait-heavy setups (single disk: felt waits 20-37s). t_ecpu now counts kernel time only, like the default path (it used to swallow the in-loop waits); waits land in t_ewait.
classify / issue / take+accumulate wall plus avg nvk/ncpu per block, printed under PROF next to t_ecpu/t_ewait/t_egpu — enough to see where the big-tier regression (420: 1.44 vs 950: 1.20 with LOWER felt wait) actually spends its extra 9 s: submit overhead growth, take waits, or outside the block entirely (dense/absorb under VRAM pressure).
…t-capped tier fill The big-tier regression was never the tier mechanics (VK-BLOCK: classify+issue+ take <=0.3s/run) — it was decode attention degrading with VRAM occupancy (7.8s at 7.6 GB resident -> 15.1s at 11.6 -> 17.8s at 15.2, identical workload): once expert weights crowd the heap, the kernel evicts and re-migrates buffers under the per-token attention submits. Two runtime-detected extensions fix the two halves: - VK_EXT_memory_priority: allocations now carry an eviction class — scratches and the KV mirror pin at 1.0 (they ride every submit), dense weights 0.75, and the engine brackets the expert-tier fill at 0.4. An oversubscribed heap sheds cold tier experts instead of thrashing attention. - VK_EXT_memory_budget: vk_registry_fill checks the device-local budget every 8 uploads and stops while COLI_VK_RESERVE_GB (default 3) is still free for the lazily-allocated dense + KV mirror + staging, logging the stop. Reserve 0 disables the stop; without the extension the count cap applies as before. Both are no-ops on drivers without the extensions.
… submit tax Isolation matrix + harness probes pinned the big-tier regression precisely: per-submit driver cost grows ~linearly with the number of distinct device-memory objects the queue actively references (~0.35 ms/submit extra at a 950-expert tier's ~5.7k allocations), hitting every synchronous fence-waited path — the dense projections (4-5 submits/layer, accounted in t_attn) most of all. Decode attention: 7.9s @420 experts -> 17.6s @950; flat when dense+attn run on CPU; IDLE allocations are free (ballast probe: absorb and dense GEMV unchanged under 5000 dummy allocations); eviction exonerated (2.9 GB free showed the same tax, scaling continuous in tier size, no budget-edge cliff). upload_tensor now binds wbuf/sbuf into shared 256 MB arena blocks (alignment- respecting bump allocator, memory-priority tag carried over) instead of one vkAllocateMemory per buffer: a 950-expert tier + dense drops from ~7k memory objects to ~60. Arena slices are not reclaimed per-tensor (registry/dense live for the process; the rare fill-failure free leaks its slice — tensor mem handles stay VK_NULL_HANDLE, vkFreeMemory's documented no-op). Arenas are unmapped/freed at shutdown. Scratches and the KV mirror stay standalone (few, constant count).
…inear submit tax)
… semantics The merged JustVugg#298 gave the CUDA kernels per-group scale handling for fmt=4 — without the same semantics a g64 container on Vulkan would decode with per-row scales (the exact bug JustVugg#298 fixed). All three shaders gain a fmt==4 branch: int4 nibble decode with one scale per gs inputs, applied to the packed-word partial (host gates gs to multiples of 8 so a word never straddles a group; per-row scaling is skipped like fmt=5). Group size flows as an explicit parameter through the upload-triggering entries into ColiVkTensor and the push constants; engine call sites pass QT.gs and the VK gates accept fmt=4 via VK_FMT_OK (word-aligned gs only — anything else stays on the CPU path, which JustVugg#298 already fixed). Harness: ref helpers generalized to runtime group size (g_ref_gs); fmt=4 cases at gs=64 across the real shapes (dense, o-proj, batch, expert_group, matmul_pair, absorb incl. S=2 causal + window) plus a gs=32 sanity case.
The attention section paid three synchronous fence roundtrips per layer per token (pair, q_b, absorb+o); the middle one existed only because the q-latent RMS norm ran on the CPU between q_a and q_b. A 40-line rmsnorm.comp (one workgroup per row; fp32 vs the CPU's f64 accumulate differs ~1e-7 at D=1536) lets coli_vk_attn_qprep record the whole chain in one command buffer with compute barriers: [q_a + kv_a] -> norm -> q_b, one fence. q, the kv latent, and the NORMED q latent return to the host (the DSA indexer consumes the latter from QR; RoPE + the canonical KV append stay CPU-side). Norm weights upload once per layer (KV-mirror pattern). The engine tries the chain first and falls back whenever rmsnorm.spv is absent, formats mismatch, or the call fails; COLI_VK_QPREP=0 forces the split path, =2 runs both and prints per-layer chain-vs-split divergence. Validation: harness chain cases vs a matmul->rmsnorm->matmul CPU reference (int8/int4, S=1/2/11, real GLM shapes; threshold 1e-2 — two quantized reductions + the norm through cancellation-heavy random rows compound ~10x a single GEMV, same reasoning as the expert_group 3e-3). Engine COLI_VK_QPREP=2 on the real container: all 78 layers within 1.8e-4 of the split path, kv bit-exact; decode-context logits match to 3e-5. Step-1 logits shift O(1) with the chain on — measured to be the container's sensitivity to ANY fp-order change, not a chain defect: pure-CPU vs split-VK step-1 logits differ ~4.0 on the same prompt with the same stable argmax.
The ~8 GB of dense weights (attention projections, absorb kv_b, o-proj, shared expert) uploaded lazily on the first forwards — after the expert tier had already filled to its budget. On a 16 GB card the late arrivals overflowed to GTT (measured 2.1 GB spilled with a 320-expert int4 tier) and every per-token attention submit paid PCIe latency: absorb core 1.69 -> 2.26 ms/call. Claiming the dense set at startup lets the budget-capped tier fill self-size to what actually fits: zero spill, decode attention 13.1 -> 9.4 s per tg64, +7% decode.
Every sync submit's vkWaitForFences pays a scheduler wake (~50-150 us) on signal; the engine fences ~2 submits per layer per token. Poll vkGetFenceStatus for a short budget first (default 300 us, the common decode dispatch completes in 0.5-2 ms), then fall back to the blocking wait. The spinning thread is stalled on the GPU result anyway. COLI_VK_SPIN_US=0 restores the pure blocking wait.
… to the binary Field report JustVugg#523 (RX 9070 XT): with Resizable BAR disabled the HOST_VISIBLE|DEVICE_LOCAL combination exists only in a ~256 MB BAR window; the driver then places tier allocations in system RAM while the resident- experts log looks healthy, and the run comes out slower than CPU-only with nothing indicating why. Compare the chosen memory type's heap against the largest DEVICE_LOCAL heap at init and say so up front. Also from the same report: the COLI_VK_SHADERS fallback was CWD-relative, so the engine only started from inside c/. Resolve the default next to the binary (/proc/self/exe), accept a directory as well as the .spv path, and name what was tried in the startup error. Makefile note on the ROCm comparison now states what was measured (the VRAM-resident expert-group primitive, per-call) instead of implying end-to-end tok/s.
…hmarking notes The benchmarking section captures the two defaults that silently skew any Vulkan-vs-CUDA/HIP comparison (DRAFT auto-resolves to 0 under COLI_CUDA per JustVugg#163 but stays 3 elsewhere — the arms then run different decode loops; and DPM never ramps for microsecond-burst dispatches), plus what the experts-loaded/token stat actually counts.
The eroute readahead (last forward's routing, hinted at layer entry) was the one prefetch path without a vk_reg_served check. Tier-served experts are never demand-read, so their pages never stay warm: on a RAM-tight streaming box the hint re-fetches the same hot experts from disk each time cache pressure evicts them — pure wasted bandwidth on the box class JustVugg#523 measured on. Prefill (S>4) keeps the hint: the tier only serves decode, so those bytes are still needed.
The README stated "measured faster than ROCm on RDNA4" unqualified; a careful third party (JustVugg#523) measured the opposite end-to-end on a storage-bound box (the gap turned out to be a DRAFT/speculation confound, but the unqualified claim was the problem). Lead with the unassailable point — Vulkan is the only backend for cards ROCm dropped — and call the RDNA4 speed "competitive", pointing at the docs' benchmarking notes for the honest, config-dependent picture.
|
Rebased onto current dev ( One thing worth noting from it: dev picked up the fmt=6 E8/IQ3 container (#465, #501) in the meantime. The Vulkan paths are unaffected by construction — every VK entry point is gated on the Validation on this exact branch build (RX 9070, RADV, Mesa 26.1.4):
|
The fmt=5 branches decoded one value at a time into a single accumulator: 16 products chained through one FMA dependency, a per-value `i < I` bounds test, and (in qmatmul) a per-value staged/unstaged select on a uniform condition. That left the branch ALU-latency-bound rather than bandwidth-bound -- int3 moved 12.5% fewer bytes than int4 yet ran 1.41x SLOWER per expert. Every container the converter emits pads I to whole 64-groups, so the common path now drops the bounds test, hoists the uniform select out of the loop (the int8/int4 branches already did this), pre-shifts the high-plane word so its bits start at bit 0, and accumulates into four independent lanes. The bounds-checked loop stays for ragged dims. RX 9070 (RADV, Mesa 26.1.4), VK_TEST harness, medians of 3 runs: expert_group fmt=5, 32 experts 0.1450 -> 0.0667 ms/expert 2.17x fused gate_up int3, 8 experts 0.0929 -> 0.0486 ms/expert 1.91x matmul_pair fmt=5 6144->(2048,576) 0.217 -> 0.149 ms/call 1.46x absorb fmt=5 K=512 T=300 0.791 -> 0.688 ms/call 1.15x int3 now decodes FASTER than int4 on the same shapes (int4 expert_group/32 is 0.1038 ms/expert), so the container's byte saving finally converts to speed on the GPU path instead of being eaten by the unpack. Exactness harness PASS. Four-way accumulation is also slightly more accurate than the 16-deep serial chain (expert_group maxrel 1.07e-4 -> 7.83e-5). End-to-end on this box the win is invisible, and the profile says why: the tier is issued ASYNC and overlaps the CPU expert share, so `routed GPU critical` is 0.027s of an 8.71s M2.7 decode (0.3%), tier straggler 89.69x. M2.7 stays I/O-bound (31% of decode waiting on expert reads; 4.79/4.57 -> 4.76/4.86 tok/s, inside run-to-run spread). The kernel win lands where the GPU is the critical path -- a larger tier share -- and it removes decode cost as an argument against fitting more experts per VRAM byte.
…A blocks Takes Qwen3.5-35B-A3B GPTQ-Int4 decode on the Arc B390 from 0.82 to 11.71 tok/s (all experts + attention + GDN on the iGPU). - fmt=6 expert-group bug: the shared expert is dense-quantized (fmt=2) but was folded into the fmt=6 routed-expert group; the fused gate_up shader decodes the whole batch with one fmt, so the mismatch rejected the ENTIRE group and dropped every routed expert to CPU. moe_token now folds the shared expert in only when its fmt matches (l->gsg.fmt == c->expert_fmt). 0.82 -> 7.09 tok/s. - shared expert rides the routed group: vk_dense quantizes it as fmt=6 when the container is fmt=6 (VkDense gained gs; quant_rows/vkd_make/mm_dense handle the grouped-asym format), removing ~80 mm_dense submits/token. 7.09 -> 7.59. - fused output->projection: coli_vk_gqa_project / coli_vk_gdn_project record the attention/GDN kernel + its o/out-projection matmul in ONE submit via a barrier; the intermediate ctx/y stays in GPU-only scratch (mirrors JustVugg#418 absorb_project). 7.59 -> 9.79. - fused whole GDN input block: gdn_conv.comp (causal conv1d + SiLU with a device- resident ring updated in-kernel) + gdn_delta_cv.comp (delta rule reading the raw conv output, q/k L2-norm folded in-shader). coli_vk_gdn_full chains gqkv/gz matmul -> conv1d -> qknorm+delta -> out-proj as 5 dispatches in one submit; only the tiny b/a->decay/beta stay on the CPU. COLI_VK_GDN_FUSE=0 keeps the staged path. 9.79 -> 11.71. All stages validated token-identical to the CPU/staged path. Microbenches: test_vk_gdn_fused.c (conv relerr 2e-8, delta y 3.6e-7). Diagnostics: qwen.c prints "[VK] routed experts: tier-served N", VK_EG_DBG=1 prints expert-group reject reasons.
3 participants
Summary
An opt-in Vulkan compute backend (
make glm VK=1, runtimeCOLI_VULKAN=1) that runs the GLM decode compute path on any GPU with a Vulkan 1.2 driver — no CUDA, no ROCm. Three tiers, each independently switchable and each falling back to the CPU path on any failure:COLI_VK_EXPERTS, default 320): the top-N routed experts by.coli_usageheat are uploaded once at startup and served from VRAM with no RAM slot, no disk read, no prefetch — issued as one fused async batch per layer (gate+up+silu→down, hidden on-device) that overlaps with the CPU computing the remaining experts. Appears as a newvkbucket in the hit-rate line.COLI_VK_DENSE=1): q_a+kv_a fused into one submit, q_b, the o-projection, and the shared expert as a single fused expert-group submit.COLI_VK_ATTN=1): one dispatch per layer covering absorbed query → scores → softmax → weighted latent → value rows, fused with the o-projection so the context vector never leaves the GPU. The latent/rope KV lives in a persistent per-layer device mirror appended at ~2.3 KB/token/layer, with the same invalidation points as the CUDA KV shadow (row rewrite, rebind, resize).This is a continuation of @kryptt's #84 (the original backend skeleton, tensor-resident
coli_vk_matmul, and the int4 nibble-8 shader decode are his design — thank you!). It addresses the two things asked for there: it is rebased onto currentdev, and it ships a CPU-reference exactness harness covering every primitive.Why
libvulkan+ a Vulkan 1.2 ICD with subgroup arithmetic, andglslcat build time.Correctness
gcc -O3 -DVK_TEST backend_vulkan.c -o test_vk -lvulkan -lm && ./test_vk shaders/qmatmul.spv): CPU-reference exactness for every primitive — int4/int8 GEMV across shapes including the long-row o-projection (I=16384), fused gate+up, the full expert group (sync and async issue/take, verified bit-identical to each other), the q_a+kv_a pair, and the absorb attention core including causal S=2,kv_startwindows, int8, and long-context (T=2000) cases. Typical maxrel 1e-5…2e-3 (fp32 reduction order on 6144-long dots).#ifdef COLI_VULKAN;make glmwithoutVK=1compiles the same code as before (plus three always-false locals).make test-cfully green on the branch.Measured performance
Hardware/software: AMD RX 9070 16 GB (RDNA4/gfx1201, PCIe 4.0 x16 effective), Ryzen 9 3900X (12C/24T Zen2,
OMP_NUM_THREADS=12, threads pinned), 64 GB DDR4-3200 dual-rank, GLM-5.2 744B int4 streamed from NVMe (~19 MB/expert). Mesa/RADV 26.1.4, Vulkan 1.4 ICD; ROCm 7.2.4 + HIP for the comparison backend. Model: 78 layers, 256 experts/layer, kv_lora 512, MLA/DSA.Primitive microbenchmarks (per-call including readback, K-expert int4 MLP 6144→2048→6144):
End-to-end GLM-5.2 decode (same box, same engine settings, expert-routing top-p 0.7, drafts off, interleaved/batched A/B runs; our fork additionally carries an unrelated dual-SSD-mirror patch on the same base — active for both backends in every comparison, so the deltas are backend-attributable):
Run-to-run variance on this box is ±0.1 tok/s; the 64-token advantage (~+13%) narrows to ~+5% at long form as the attention window and expert-I/O share grow for both backends.
The two memory-type rules that made it work (documented in
docs/vulkan.md, learned the hard way):HOST_CACHED— reading write-combined ReBAR VRAM from the CPU runs at ~40 MB/s and was a hidden 5.6× per-call penalty.HOST_VISIBLE|DEVICE_LOCAL, so uploads are plain memcpys and the GPU reads at VRAM speed.Integration surface
c/backend_vulkan.c(~1.3k lines, plain C99 + libvulkan),c/backend_vulkan.h,c/shaders/qmatmul.comp,c/shaders/qmatmul_gate_up.comp,c/shaders/attention_absorb.comp,docs/vulkan.md.c/glm.c(hooks under#ifdef COLI_VULKAN),c/Makefile(VK=1block +glslcrule),docs/ENVIRONMENT.md,README.md.COLI_VULKAN,COLI_VK_SHADERS,COLI_VK_EXPERTS,COLI_VK_DENSE,COLI_VK_ATTN.One operational note worth flagging beyond this PR: the engine's OMP self-tune (
OMP_WAIT_POLICY=active+ spin) is skipped underCOLI_CUDA/COLI_METALbut not under other configurations. With 12 pinned spinning threads the async I/O pool starves (measured 28 → 5 GB/s CPU expert bandwidth, ~2.8× end-to-end).docs/vulkan.mdtells Vulkan users to setCOLI_NO_OMP_TUNE=1; extending the self-tune skip to any GPU backend (or toPIPE=1runs) may be worth a separate look.Limits / future work
S≤4; prefill keeps the CPU/batched paths (dense projections do run on VK at prefill, which is why prefill got faster).KV8=1(already implemented and validated againstkv_fp8.hon our side — the absorb shader decodes e4m3 in place, 4× less mirror traffic; this PR just cleanly falls back to CPU attention when the f32 cache is absent), cooperative-matrix prefill kernels, and a fully resident-layer pipeline.How to run