The reference SmoothNAS plugin for llama.cpp. Runs llama-server inside a SmoothNAS-managed LXC system container with GPU passthrough, tier-bound model storage, and bearer-injected auth from the SmoothNAS UI. Published images build upstream llama.cpp from the release baked into the SmoothNAS runner image.
This is the first reference plugin built against the SmoothNAS plugin system. It exists as a worked example for plugin authors and as a real workload SmoothNAS operators can install.
Three published variants, one per accelerator:
| Manifest | Image tag | Profile | Use when |
|---|---|---|---|
smoothnas-plugin.yaml |
ghcr.io/rakuensoftware/smoothnas-plugin-llama-cpp:VER-cuda |
gpu-nvidia |
Host has an NVIDIA GPU |
smoothnas-plugin-vulkan.yaml |
ghcr.io/rakuensoftware/smoothnas-plugin-llama-cpp:VER-vulkan |
gpu-amd |
Host has an AMD GPU. Uses Vulkan via /dev/dri — no ROCm runtime needed. |
smoothnas-plugin-cpu.yaml |
ghcr.io/rakuensoftware/smoothnas-plugin-llama-cpp:VER-cpu |
none | No GPU, or quick experiments before paying the GPU pull cost |
Pick the matching manifest at install time in the SmoothNAS UI's plugin install wizard, or via tierd-cli plugin install --tier <pool> <manifest.yaml>.
Upstream llama-server has no built-in HTTP auth. SmoothNAS phase-7 injects a bearer token at the nginx layer, but the plugin still has to check it — otherwise an operator who exposes the SmoothNAS UI publicly would also be exposing an unauthenticated inference server.
This repo's wrapper/ is a tiny Go binary (~150 LoC) that:
- Starts upstream
llama-server(/app/llama-serverin current upstream images) as a child on127.0.0.1:8081 - Listens on
:8080(the port the SmoothNAS nginx route forwards to) - Validates
Authorization: Bearer $SMOOTHNAS_BEARER_EXPECTEDon every request — constant-time compare, rejects with 401 on missing or wrong token - Forwards valid requests to the child via
httputil.NewSingleHostReverseProxywith streaming-friendly defaults (so SSE / token-by-token completions work)
SMOOTHNAS_BEARER_EXPECTED is auto-populated by tierd at install time and rotated via POST /api/plugins/llama-cpp/rotate-token. The operator never sees or touches the token directly — it's owned by SmoothNAS.
In the SmoothNAS UI:
- Install → paste this manifest into the wizard, pick a tier with NVME slot capacity, set
MODEL_URLto the GGUF download URL, install - Start → click Start; tierd pulls the wrapper image, creates the container, captures the bridge IP, writes the nginx route with the bearer token, and the wrapper downloads the configured model URL into
/models/model.gguf. IfSPEC_DRAFT_MODEL_URLis set, it also downloads that GGUF before starting llama.cpp. - Open → click Open on the card; the llama.cpp UI renders inside the SmoothNAS chrome at
https://<smoothnas>/plugins/llama-cpp/
CUDA and Vulkan manifests expose a GPU install-time field. Leave it
automatic to use the default SmoothNAS GPU profile, or select a host GPU to
narrow passthrough to that device.
The default GPU runtime profile is Qwen3.6 35B-A3B Q5 with conservative first-run settings. Increase context, parallel slots, or the memory limit only after confirming the selected model fits the host and GPU:
MODEL_URL=<https://.../*.gguf>CTX_SIZE=32768PARALLEL_SLOTS=1N_GPU_LAYERS=999on CUDA/Vulkan manifestsLLAMA_ARG_TEMP=0.8MEMORY_LIMIT=32GiBLLAMA_ARG_CACHE_TYPE_K=q8_0LLAMA_ARG_CACHE_TYPE_V=q8_0LLAMA_ARG_FLASH_ATTN=onLLAMA_ARG_N_CPU_MOE=10on CUDA/Vulkan manifestsLLAMA_ARG_FIT=onon CUDA/Vulkan manifestsLLAMA_ARG_REASONING_FORMAT=autoSPECULATIVE_MODE=none
Quantization is selected by the GGUF file itself. To use a different
quantization, update MODEL_URL and restart the plugin; the wrapper
keeps a single active model at /models/model.gguf and replaces it
when the URL changes.
The manifests expose llama.cpp's speculative decoding controls in the
plugin config UI. Leave SPECULATIVE_MODE=none for normal startup. When
SPEC_DRAFT_MODEL_URL is set and SPEC_DRAFT_MODEL_PATH is empty, the
wrapper caches the assistant GGUF at /models/mtp-head.gguf. CUDA and
Vulkan variants additionally expose draft GPU layers and draft CPU MoE
placement for upstream llama.cpp modes that support those flags.
For normal coding/completions against Qwen thinking models, set
LLAMA_ARG_REASONING_FORMAT=none when clients cannot handle
message.reasoning_content. This keeps llama.cpp from splitting the
model's thinking stream into a separate response field; clients may still
need to strip <think>...</think> blocks from message.content.
Uninstall via the UI's Danger Zone removes the container, the cached image, the nginx route, and the model files (per the parent doc's all-or-none policy). Back up /mnt/<tier>/.plugins/llama-cpp/models/ first if you care.
# wrapper smoke build + tests
cd wrapper && go build ./... && go test ./...
# image build (any variant)
CGO_ENABLED=0 go -C wrapper build -ldflags="-s -w" -o ../smoothnas-wrapper .
docker build \
--build-arg LLAMA_BACKEND=cuda \
--build-arg LLAMA_BUILD_BASE=nvidia/cuda:12.8.1-devel-ubuntu24.04 \
--build-arg LLAMA_RUNTIME_BASE=nvidia/cuda:12.8.1-runtime-ubuntu24.04 \
-t smoothnas-plugin-llama-cpp:dev-cuda .To sideload a dev image into a SmoothNAS dev VM, edit the matching manifest's artifact.image to your local tag and use tierd-cli plugin install.
.github/workflows/release.yml runs on tag push (v*):
- Builds three image variants (CUDA / Vulkan / CPU) via Docker + GHCR push
- Resolves the pushed digests and rewrites
artifact.digestin each manifest - Creates a GitHub release attaching the three rewritten manifests
- Pushes a release-channel
index.jsonso the SmoothNAS install wizard can offer the variants alphabetically
The smoke test that actually installs the plugin against a SmoothNAS dev VM lives in the SmoothNAS repo's CI, not here. Triggered nightly so a SmoothNAS release that broke the plugin contract is caught quickly.
Add a LICENSE file at publish time. The wrapper code in this repo is original; downstream images carry their own license terms (upstream llama.cpp is MIT).
This plugin is licensed under the AGPL-3.0.