Docker image lacks CUDA Toolkit (nvcc): llama.cpp always falls back to CPU on Windows/WSL2, even with GPU passthrough working (related to #831, #2780)

[26leonardo]

Prerequisites

• I searched open issues and discussions and did not find an existing report of this bug.
• This is not a security vulnerability. (Vulnerabilities go to GitHub Security Advisories — see SECURITY.md.)
• I am running the latest code from the dev branch (the default branch you get on clone, where fixes land first) and the bug still reproduces there. Please git pull the latest dev before filing.

Install Method

Docker (docker compose up)

Operating System

Other

Steps to Reproduce

OS/Env: Windows 10 + WSL2 (Ubuntu), Docker Desktop

1. Clone the repo and cd odysseus.
2. Copy .env.example to .env and enable the NVIDIA GPU overlay:
   COMPOSE_FILE=docker-compose.yml:docker/gpu.nvidia.yml
3. Run docker compose up -d --build.
4. Confirm passthrough works: docker compose exec odysseus nvidia-smi -L
   shows the GPU correctly, and scripts/check-docker-gpu.sh reports
   "3 passed, 0 failed".
5. Open Cookbook → Dependencies, install llama.cpp (installs without errors).
6. Download a model in Cookbook that fits entirely in 16GB VRAM
   (e.g. mradermacher/Mistral-Small-24B-Instruct-2501-GGUF).
7. Serve the model from Cookbook.
8. Observe GPU usage in nvidia-smi while the model is loaded/generating.
9. Inference only working with CPU, even if GPU is found by the sistem

Expected Behaviour

With a working NVIDIA GPU detected by Cookbook and GPU passthrough confirmed
(nvidia-smi works both on host and inside the container), the served model
should offload layers to the GPU and show VRAM usage in nvidia-smi during
inference.

Actual Behaviour

The model always runs on CPU only, regardless of GPU layer settings. Cookbook
correctly detects the GPU and sets CUDA_VISIBLE_DEVICES / -ngl, but the
llama-server / llama-cpp-python binary built inside the container is CPU-only.
Reinstalling the serve engine from Cookbook → Dependencies does not fix it,
it rebuilds the same CPU-only binary every time.

Logs show (or an equivalent message):
warning: no usable GPU found, --gpu-layers option will be ignored
warning: one possible reason is that llama.cpp was compiled without GPU support

nvidia-smi confirms the GPU and driver are working fine on host and in the
container the whole time.

Logs / Screenshots

[odysseus] HF token: NOT SET — gated/private models will be denied. Add one in O
dysseus Cookbook -> Settings -> HuggingFace Token.
Native llama-server not found — building from source (one-time, may take a few m
inutes)...
[odysseus] No matching prebuilt llama-server for this host (arch=x86_64) — will
build from source.
[odysseus] WARNING: no HIP/CUDA/Vulkan toolchain found — building llama-server f
or CPU only.
[odysseus]   GPU inference will not be available for this llama.cpp build.
[odysseus]   Install Vulkan (libvulkan-dev) / ROCm for AMD GPUs or CUDA tooling
for NVIDIA, then re-launch this serve task.
Key excerpt from the Cookbook llama.cpp build inside the container — note it
only ever builds the CPU backend (ggml-cpu), never ggml-cuda:

[ 7%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/hbm.cpp.o
...
[ 69%] Linking CXX shared library ../bin/libllama.so
[ 69%] Built target llama
...
[100%] Built target llama-server

docker compose exec odysseus nvidia-smi -L:
GPU 0: NVIDIA GeForce RTX 4060 Ti (UUID: GPU-b23a5b76-...)

scripts/check-docker-gpu.sh output:
[PASS] GPU passthrough is working — the NVIDIA compose overlay should work.
=== Results: 3 passed, 0 failed ===

Model / Backend (if relevant)

Cookbook-managed llama.cpp (llama-server), model: mradermacher/Mistral-Small-24B-Instruct-2501-GGUF

Are you willing to submit a fix?

Yes — I can open a PR

Additional Information

Root cause: the final Dockerfile stage is FROM python:3.14-slim, which is
Debian "trixie" and does not include the CUDA Toolkit — no nvcc, no cuBLAS —
so CMake's ggml-cuda backend has nothing to compile against, even though GPU
passthrough (driver) works fine. This matches the root cause described in
odysseus-dev/odysseus discussion #789 ("the issue currently lies in the base
image lacking cuda support") and issues #831 / #2780, which report the same
symptom on different GPUs (GTX 1070 Ti, RTX 3060 Ti).

Two non-obvious gotchas hit while fixing this on the current base image:

1. NVIDIA's cuda-keyring package fails GPG verification on Debian trixie
   (SHA-1 self-signature rejected by newer apt/sqv policy since 2026-02-01).
   Workaround: add the NVIDIA repo directly via sources.list.d with
   [trusted=yes], scoped only to that one repo.
2. The cuda-toolkit-12-4 meta-package is not installable on trixie — it
   pulls Nsight Systems/Compute, which depend on libtinfo5 (removed in
   trixie, only libtinfo6 exists). Fix: install only cuda-nvcc-12-4,
   cuda-cudart-dev-12-4, cuda-driver-dev-12-4, cuda-nvrtc-dev-12-4,
   libcublas-dev-12-4 — skipping the profiler suite entirely.

I can also add a short "GPU passthrough works but llama.cpp still uses CPU"
troubleshooting section to docs/setup.md, since this seems to be a very
common report (#831, #2780, discussions #789, #2928 all hit the same root
cause). Willing to open a docs PR for that too if useful.