docs(gpu): add GPU acceleration guide (Metal/CUDA/Vulkan + Apple Silicon on Asahi) - #930
Open
megabyte0x wants to merge 1 commit into
Open
megabyte0x wants to merge 1 commit into
megabyte0x wants to merge 1 commit into
Conversation
Document how QMD selects a llama.cpp backend, how to build a GPU-enabled node-llama-cpp binary when the packaged prebuilt is CPU-only, and the Apple-GPU-on-Linux (Asahi/Honeykrisp) Vulkan path. qmd doctor already tells users to configure Metal/CUDA/Vulkan but there was no how-to. - Add docs/GPU.md (backend selection, source build, per-distro deps, the spirv-headers gotcha, verification, benchmarks, troubleshooting) - Link it from the README Environment Variables section - Add an Unreleased changelog entry
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a GPU acceleration guide (
docs/GPU.md) and links it from the README.qmd doctoralready nudges users to "install/configure Metal, CUDA, or Vulkanfor faster embeddings," but there was no documentation on how. This fills that
gap, with particular attention to the Apple GPU on Linux (Asahi) path, which
isn't documented anywhere upstream.
Contents
docs/GPU.mdcovers:QMD_LLAMA_GPU,QMD_FORCE_CPU,auto), and thekey subtlety that
autocan only pick a backend that is compiled into theloaded
node-llama-cppbinary — the packaged prebuilt is CPU-only on someplatforms (e.g. Linux arm64), so
autoresolves to CPU there until you build one.node-llama-cpp source download --gpu ....(Mesa Honeykrisp) — distro dependencies, the easy-to-miss
spirv-headersrequirement (configure fails with
Could not find a package configuration file provided by "SPIRV-Headers"without it), andqmd doctorverification.Also adds a short pointer from the README Environment Variables section and an
## [Unreleased]changelog entry.Reference setup / benchmarks
Verified end-to-end on a MacBook Pro 14" (M2 Pro) running Asahi Linux. After
building the Vulkan backend and setting
QMD_LLAMA_GPU=vulkan,qmd doctorreports
GPU vulkan; offloading enabled; Apple M2 Pro (G14S B1); VRAM 15.6 GB:qmd vsearch(embedding)qmd query(expand + embed + rerank)CPU time for the deep pipeline dropped from ~64 s (all cores saturated) to ~2.3 s.
Notes
you'd prefer the guide stay more general.