Skip to content

docs(gpu): add GPU acceleration guide (Metal/CUDA/Vulkan + Apple Silicon on Asahi) - #930

Open
megabyte0x wants to merge 1 commit into
tobi:mainfrom
megabyte0x:docs/gpu-acceleration
Open

megabyte0x wants to merge 1 commit into
tobi:mainfrom
megabyte0x:docs/gpu-acceleration

Conversation

@megabyte0x

Copy link
Copy Markdown

What

Adds a GPU acceleration guide (docs/GPU.md) and links it from the README.

qmd doctor already nudges users to "install/configure Metal, CUDA, or Vulkan
for faster embeddings," but there was no documentation on how. This fills that
gap, with particular attention to the Apple GPU on Linux (Asahi) path, which
isn't documented anywhere upstream.

Contents

docs/GPU.md covers:

  • How QMD selects a backend (QMD_LLAMA_GPU, QMD_FORCE_CPU, auto), and the
    key subtlety that auto can only pick a backend that is compiled into the
    loaded node-llama-cpp binary
    — the packaged prebuilt is CPU-only on some
    platforms (e.g. Linux arm64), so auto resolves to CPU there until you build one.
  • Building a GPU-enabled binary via node-llama-cpp source download --gpu ....
  • Apple Silicon on Linux (Asahi): enabling the Apple GPU via Vulkan
    (Mesa Honeykrisp) — distro dependencies, the easy-to-miss spirv-headers
    requirement (configure fails with Could not find a package configuration file provided by "SPIRV-Headers" without it), and qmd doctor verification.
  • Measured before/after numbers and a troubleshooting section.

Also adds a short pointer from the README Environment Variables section and an
## [Unreleased] changelog entry.

Reference setup / benchmarks

Verified end-to-end on a MacBook Pro 14" (M2 Pro) running Asahi Linux. After
building the Vulkan backend and setting QMD_LLAMA_GPU=vulkan, qmd doctor
reports GPU vulkan; offloading enabled; Apple M2 Pro (G14S B1); VRAM 15.6 GB:

Operation CPU Vulkan (Apple GPU) Speedup
qmd vsearch (embedding) ~5.2 s ~2.2 s ~2.4×
qmd query (expand + embed + rerank) ~14.1 s ~2.4 s ~5.9×

CPU time for the deep pipeline dropped from ~64 s (all cores saturated) to ~2.3 s.

Notes

  • Docs-only; no code changes.
  • Happy to trim, adjust voice, or move the Asahi specifics into a subsection if
    you'd prefer the guide stay more general.

Document how QMD selects a llama.cpp backend, how to build a GPU-enabled
node-llama-cpp binary when the packaged prebuilt is CPU-only, and the
Apple-GPU-on-Linux (Asahi/Honeykrisp) Vulkan path. qmd doctor already
tells users to configure Metal/CUDA/Vulkan but there was no how-to.

- Add docs/GPU.md (backend selection, source build, per-distro deps, the
  spirv-headers gotcha, verification, benchmarks, troubleshooting)
- Link it from the README Environment Variables section
- Add an Unreleased changelog entry
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant