llama-webui

Remote llama.cpp workbench for GGUF models with direct runtime control, persistent chats, RAG knowledge base, RPC split serving, and hardware-aware tuning from RK3588 boards to Windows CUDA systems.

What This Is

llama-webui is a standalone local web interface for llama.cpp with no Ollama dependency. Ollama was removed after CVE-2026-7482 (CVSS 9.1) exposed a critical RCE vulnerability. The serving stack is compiled llama.cpp, so KV cache quantization, context length, batch sizing, CPU affinity, GPU layer offload, and model-specific flags remain fully controllable.

This repository started as a board-local control surface for testing REAP-pruned Mixture-of-Experts models on a Radxa ROCK 5B+ with Rockchip RK3588. It now serves three purposes:

A practical remote web UI for loading and serving local GGUF models with llama.cpp
A documented benchmark and tuning harness for constrained ARM boards and more capable desktop machines
A serious-workload control plane for Windows CUDA hosts, including Windows-to-Windows llama.cpp RPC split serving across a LAN

The main point is simple: keep the convenience of a web UI without giving up the flags that decide whether local inference is actually usable.

What Makes It Different

Many local-AI UIs optimize for convenience first and runtime visibility second. llama-webui is deliberately built the other way around.

It talks to compiled llama.cpp directly — no Ollama dependency (removed after CVE-2026-7482).
It preserves the flags that actually decide whether a host feels usable:
- KV cache type
- context window
- batch and ubatch sizing
- CPU affinity
- GPU offload depth
- reasoning / no-thinking mode
It treats benchmark-backed hardware tuning as a feature, not background trivia.
It is intentionally credible on small ARM systems, especially the Radxa ROCK 5B+ with RK3588, not only on desktop GPUs.
It supports llama.cpp RPC split serving, so one coordinator can combine local and remote compute instead of being limited to a single box.

If you just want a generic chat shell, other tools already cover that. This repo is for people who want a usable web UI without giving up the runtime knobs that explain real behavior.

Who It Is For

engineers benchmarking GGUF models on llama.cpp
people serving models on ARM SBCs, headless Linux hosts, or mixed desktop hardware
developers who want a remote control plane for model switching and runtime tuning
anyone who needs to compare presets, flags, and model behavior with less guesswork

Compared To Other Options

Versus Ollama-based UIs

Ollama removed after CVE-2026-7482 (CVSS 9.1 critical RCE)
more direct control over llama.cpp startup flags
easier to reason about model-specific tuning and failure modes
better fit for benchmarking, profiling, and constrained-hardware tuning
no hidden abstraction layer between you and the runtime knobs

Versus desktop-first local AI tools

less GPU-desktop assumption baked into the defaults
friendlier to SBC and remote-host workflows
clearer path to RK3588 and CPU-first tuning

Versus ad hoc shell scripts

persistent chats
browser control plane
model inventory and presets
easier inspection of runtime state and configuration

Why RK3588 Is A First-Class Target

Most local-AI tools talk about ARM boards as a novelty. This repo does not.

It was shaped around real tuning work on a Radxa ROCK 5B+.
It exposes the settings that matter on RK3588: context, KV cache quantization, threads, affinity, and no-thinking mode.
It carries benchmark-backed presets instead of pretending the same defaults work equally well on laptops, workstations, and SBCs.
It is useful when the goal is stable, private local inference on constrained hardware, not just desktop demos.

Key Features

llama.cpp only — Ollama removed after CVE-2026-7482 (CVSS 9.1)
Browser-driven model loading and runtime config editing
GPU auto-detection for CUDA, ROCm, Metal, and CPU-first profiles
Streaming responses via Server-Sent Events
Persistent chat history in SQLite
Model discovery from configurable GGUF directories
Cross-platform support for ARM SBCs, Linux desktops, NVIDIA GPUs, and Windows CUDA hosts
llama.cpp RPC split mode for LAN-connected worker machines
Knowledge Base / RAG: index and search AI conversation history (Pi, Claude Code, Codex, Factory, Qwen Code, OpenCode) with hybrid BM25 + vector retrieval
Knowledge-augmented chat: inject retrieved context into model prompts with one toggle
RK3588-tested presets for fast daytime use and stronger overnight use
explicit no-thinking defaults for interactive serving, plus visible-response sanitation when models leak empty think wrappers anyway

Repository Layout

llama-webui/
├── README.md
├── AGENTS.md
├── CONTRIBUTING.md
├── REAP_RK3588_NOTES.md
├── WINDOWS_SETUP_GUIDE.md
├── SETUP_MISSING_COMPONENTS.md
├── .env.example
├── pyproject.toml
├── docs/
│   ├── architecture.md
│   ├── hardware-portability.md
│   ├── openapi.json
│   ├── reap-roadmap.md
│   └── rk3588-benchmarks.md
├── models/
│   └── .gitkeep
├── data/
│   └── .gitkeep
├── scripts/
│   ├── build_llama_cuda.sh
│   ├── build_llama_cuda.ps1
│   ├── configure_env.ps1
│   ├── reap_bakeoff.py
│   ├── reap_benchmark.py
│   ├── reap_status.sh
│   ├── run_reap_pipeline.sh
│   └── setup_windows.ps1
├── src/llama_webui/
│   ├── knowledge/        # RAG pipeline (DB, chunker, embedder, retriever, ingest)
│   └── ...
└── static/

Runtime artifacts under data/, downloaded models under models/, and local virtual environments are intentionally ignored by git.

Quick Start

Linux/macOS

Clone the repo and enter it.
Build llama.cpp.
Download at least one GGUF into ./models, ~/models, or a path listed in LLAMA_WEBUI_MODEL_DIRS.
Install and run:

git clone https://github.com/SergiioB/llamacpp-workbench.git
cd llamacpp-workbench

./scripts/build_llama_cuda.sh

uv venv
source .venv/bin/activate
uv pip install -e .
llama-webui

Windows

Run the setup script:

# Build from source
.\scripts\setup_windows.ps1

# Or use prebuilt binaries
.\scripts\setup_windows.ps1 -UsePrebuilt

Configure environment variables if needed:

.\scripts\configure_env.ps1 -CreateEnvFile

Start the app:

.venv\Scripts\Activate.ps1
llama-webui

See WINDOWS_SETUP_GUIDE.md and SETUP_MISSING_COMPONENTS.md for the Windows flow and dependency troubleshooting.

Open http://<host>:8095.

llama.cpp RPC Mode

For the exact Windows-to-Windows launch checklist and PowerShell helpers, use docs/rpc-remote-host-runbook.md.

The WebUI can start llama-server either on one machine or with llama.cpp RPC offload to another reachable device. In this mode there are two roles:

Coordinator host: runs llama-webui and starts llama-server.
RPC worker host: runs rpc-server and exposes one accelerator or CPU backend over TCP.

The worker must already be running before the coordinator starts or loads a model. The coordinator passes --rpc <host>:<port>, --split-mode layer, and optionally --tensor-split to llama-server.

This mode has been validated for real Windows-to-Windows local workloads: a Windows desktop coordinator can drive a Windows laptop RPC worker on the same LAN, combining CUDA devices while keeping the WebUI as the single control surface.

1. Prepare the RPC worker

Install or copy a compatible llama.cpp build onto the worker. Use the same llama.cpp build as the coordinator when possible; protocol mismatches can fail during model load.

On Windows, open PowerShell on the worker:

cd C:\path\to\llama.cpp\bin
.\rpc-server.exe -H 0.0.0.0 -p 50052 -c

On Linux, start the equivalent binary from the build directory:

cd /path/to/llama.cpp/build/bin
./rpc-server -H 0.0.0.0 -p 50052 -c

The worker log should print the endpoint, transport, and visible devices. For a CUDA worker, also check:

nvidia-smi

If the worker has a firewall enabled, allow inbound TCP on the selected port (50052 in the examples). On Windows, the rule can be added from an elevated PowerShell:

New-NetFirewallRule -DisplayName "llama.cpp RPC 50052" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 50052

Only expose this port on a trusted LAN. llama.cpp RPC is an experimental internal transport, not an authenticated public API.

2. Configure the WebUI coordinator

Open Settings in the WebUI and set:

Runtime -> Mode: RPC Split
RPC Host: the worker IP or DNS name, for example 192.168.1.146
Port: the worker port, for example 50052
Tensor Split: a comma-separated model-weight ratio, for example 34,66

Then click Check in the RPC panel. This verifies TCP reachability before any model is loaded.

After the RPC check passes, select a scanned model or preset and press Start or Load Preset. During load, llama-server uploads tensors to the worker. That upload can take a while and the worker may refuse extra TCP probes while the active coordinator connection owns the RPC server. The WebUI treats a managed healthy llama-server as authoritative instead of re-probing the busy RPC port.

3. Choose a tensor split

Tensor Split is a comma-separated ratio that controls how much model weight is placed on each backend. It must match the coordinator device plus each RPC device in order.

Examples:

50,50: roughly even split between local GPU and one RPC worker.
34,66: keep less model weight on a 12 GB display GPU and more on the remote worker.
70,30: keep more weight local when the worker has less available VRAM.

Tune the split against actual load success, free memory, prompt throughput, and token throughput. If the worker OOMs or disconnects during set_tensor, reduce the worker share or close competing GPU processes on the worker.

RPC mode uses --split-mode layer. Layer offload moves activations across the network at layer boundaries and is the practical default for normal LAN links. The default KV cache policy remains --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 so long-context serving keeps VRAM use predictable.

Troubleshooting

If Check RPC fails before model load, verify:

the remote rpc-server process is running
the host/IP and port match the WebUI fields
the firewall allows inbound TCP on the RPC port
both devices are on the same network or otherwise routable
the remote accelerator is visible to its local runtime, for example through nvidia-smi, rocminfo, clinfo, or the vendor tool for that device

If the model starts loading but then the worker logs recv failed, restart the worker rpc-server, stop the local llama-server, and try a smaller or more conservative split. A half-loaded local llama-server can hold VRAM even after the RPC worker died; use the WebUI Stop button or terminate that process before relaunching.

If the model is already loaded and chat works, a direct TCP probe to the worker may fail because the active llama-server connection is using the RPC server. In that state, trust /api/server/status or the WebUI server indicator over a standalone port probe.

Recommended RK3588 Defaults

For a Radxa ROCK 5B+ or other RK3588 board, the practical interactive baseline is:

Qwen3.5-2B for the fast tier
Qwen3.5-4B when quality matters more than latency
explicit no-thinking flags for chat-style serving
rk-llama.cpp where available for the better CPU path

The tracked board notes live in REAP_RK3588_NOTES.md.

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      Browser (Client)                       │
│  ┌─────────────────────────────────────────────────────┐   │
│  │  llama-webui  │  Chat UI  │  Model Settings  │ Logs │   │
│  └───────────────┴───────────┴──────────────────┴──────┘   │
└────────────────────────────┬────────────────────────────────┘
                             │ HTTP / SSE
┌────────────────────────────▼────────────────────────────────┐
│                     FastAPI Server                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │ Chat Router  │  │Model Manager │  │Download Manager  │  │
│  │   (SSE)      │  │              │  │                  │  │
│  └──────────────┘  └──────────────┘  └──────────────────┘  │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │  App State   │  │   Settings   │  │ Model Inventory  │  │
│  │  (SQLite)    │  │   (.env)     │  │                  │  │
│  └──────────────┘  └──────────────┘  └──────────────────┘  │
└────────────────────────────┬────────────────────────────────┘
                             │ Subprocess / IPC
┌────────────────────────────▼────────────────────────────────┐
│                    llama.cpp (Binary)                       │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │llama-server  │  │  llama-cli   │  │ Custom Flags     │   │
│  │ (HTTP API)   │  │ (Interactive)│  │ gpu_layers, ctx, │   │
│  │              │  │              │  │ threads, etc.    │   │
│  └──────────────┘  └──────────────┘  └──────────────────┘   │
└─────────────────────────────────────────────────────────────┘

Configuration

The app is path-portable. Machine-specific paths are handled through environment-driven defaults.

Supported variables:

LLAMA_WEBUI_DATA_DIR
LLAMA_WEBUI_MODEL_DIRS
LLAMA_WEBUI_DEFAULT_DOWNLOAD_DIR
LLAMA_WEBUI_LLAMA_SERVER
LLAMA_WEBUI_LLAMA_CLI
LLAMA_WEBUI_KNOWLEDGE_SOURCES

See .env.example for concrete examples.

Default behavior without overrides:

Data directory: ./data
Model search roots: ./models and ~/models
Default download target: first existing model root, otherwise ./models
llama-server and llama-cli: resolved from env, then PATH, then common local build locations under third_party/llama.cpp
Knowledge source paths: auto-discovered from ~/.pi, ~/.claude, ~/.codex, ~/.factory, ~/.local/share/opencode, ~/.qwen/projects

Building llama.cpp

CUDA Build (NVIDIA GPU)

Linux/macOS

./scripts/build_llama_cuda.sh

# Manual fallback
git clone https://github.com/ggerganov/llama.cpp.git third_party/llama.cpp
cd third_party/llama.cpp
mkdir build-cuda && cd build-cuda
cmake .. -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_CLI=ON
cmake --build . --config Release -j$(nproc)

Windows

# Recommended
.\scripts\setup_windows.ps1

# Build-only path
.\scripts\build_llama_cuda.ps1

# Prebuilt binaries
.\scripts\setup_windows.ps1 -UsePrebuilt -CudaVersion 12.4

Prerequisites for Windows CUDA builds:

Visual Studio 2022 Build Tools with the C++ desktop workload
CUDA Toolkit 12.4 or later, with 12.8+ recommended for RTX 50-series
CMake 3.18+

AMD ROCm Build (RX 7800 XT / RDNA3)

Tested on Ubuntu 24.04 with ROCm 6.4.4 and an RX 7800 XT (gfx1101).

# Prerequisites: ROCm 6.4+ installed and HSA_OVERRIDE_GFX_VERSION set if needed
git clone https://github.com/ggerganov/llama.cpp.git third_party/llama.cpp
cd third_party/llama.cpp
mkdir build-hip && cd build-hip

cmake .. \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS=gfx1101 \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP_ROCWMMA_FATTN=ON \
  -DGGML_HIP_GRAPHS=ON \
  -DGGML_HIP_MMQ_MFMA=ON \
  -DGGML_HIP_NO_VMM=ON

cmake --build . --config Release -j$(nproc)

Key build flags explained:

GGML_HIP_ROCWMMA_FATTN=ON — FlashAttention via AMD matrix cores
GGML_HIP_GRAPHS=ON — HIP graph capture for reduced launch overhead
GGML_HIP_MMQ_MFMA=ON — RDNA3 MFMA matrix multiply kernels
GGML_HIP_NO_VMM=ON — Avoids virtual memory overhead on RDNA3

The binary resolves automatically from third_party/llama.cpp/build-hip/bin/ when ROCm is detected.

CPU-only / RK3588 Build

cd third_party/llama.cpp
mkdir build-rk-opt && cd build-rk-opt
cmake .. -DGGML_CPU_VULKAN=ON -DGGML_CPU_BLAS=ON -DCMAKE_BUILD_TYPE=Release -march=armv8.2a+dotprod
cmake --build . --config Release -j$(nproc)

TurboQuant (TQ3_1S) for 16GB GPUs

TurboQuant enables 3.5-bit quantization to fit larger models on 16GB GPUs with a better size-to-quality tradeoff than plain Q4 variants.

git clone https://github.com/turbo-tan/llama.cpp-tq3.git third_party/llama.cpp-tq3
cd third_party/llama.cpp-tq3
mkdir build && cd build
cmake .. -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build . --config Release -j$(nproc)

GPU Backend Auto-Detection

Backend	Detection	Default Settings
CUDA	`nvidia-smi` available	`gpu_layers=99`, `parallel=4`, `batch_size=512`
ROCm	AMD GPU vendor ID (`0x1002`) or `rocm-smi`	`gpu_layers=99`, `flash_attn=on`, `parallel=1`, `kv=q8_0/q8_0`
Metal	Apple Silicon	Managed by llama.cpp
CPU	Fallback	`gpu_layers=0`, `parallel=1`, `batch_size=128`

The detected backend affects default gpu_layers, parallel, batch_size, ubatch_size, model presets, and build-path priority for llama-server. ROCm detection checks /sys/class/drm/card*/device/vendor for the AMD vendor ID (0x1002), with a rocm-smi fallback.

Tested Hardware

ARM / Embedded

Board: Radxa ROCK 5B+
SoC: Rockchip RK3588
Architecture: aarch64
Total cores: 8
RAM class: about 24 GiB
Tested models: GLM-4.7-Flash-REAP-23B-A3B, Qwen3.5-2B, Qwen3.5-4B, Qwen3.5-9B

Windows Desktop / Laptop

OS: Windows 11
CPU: Intel Core Ultra 9 285H
GPU: NVIDIA GeForce RTX 5060 Laptop GPU plus Intel Arc 140T
RAM: 32 GB
CUDA: 12.8+ for Blackwell-class support

AMD Linux Desktop (AI Node)

OS: Ubuntu 24.04 LTS (headless)
CPU: AMD Ryzen 7 3800X
GPU: Sapphire RX 7800 XT 16GB (gfx1101, RDNA3)
RAM: 16 GB DDR4
ROCm: 6.4.4
Access: Tailscale, sleeps 01:00-07:30
Tested models: Qwen3.6-35B-A3B (IQ3_XXS), GLM-4.7-REAP-23B (IQ4_XS), Qwen3-27B (Q4_K_M)

Windows-to-Windows RPC

Coordinator: Windows desktop running llama-webui and llama-server
Worker: Windows laptop running rpc-server
Transport: trusted LAN TCP with --rpc, --split-mode layer, and tuned --tensor-split
Workload class: serious interactive local inference, not only toy demos

See docs/rk3588-benchmarks.md and docs/hardware-portability.md.

Hardware Scope

Although validation is strongest on the ROCK 5B+, the project is not ARM-only. The UI works anywhere llama.cpp works, including:

ARM SBCs running CPU-only inference
Desktop Linux CPU-only systems
NVIDIA CUDA hosts using GPU offload
AMD ROCm hosts
Apple Silicon systems
Windows CUDA laptops and desktops

What changes per machine is not the app architecture, but the runtime tuning: gpu_layers, ctx_size, threads, parallel, batch_size, ubatch_size, and custom llama.cpp flags.

Best Configurations We Found

Best interactive quality-per-size on RK3588

Model:

GLM-4.7-Flash-REAP-23B-A3B-Q3_K_M.gguf

Validated settings:

CPU affinity: 4-7
Threads: 4
Parallel: 1
Context: 202752
Batch size: 128
KV cache quantization: --cache-type-k q8_0 --cache-type-v q4_0
Thinking disabled: --reasoning off --reasoning-budget 0 --reasoning-format none

Fast daytime fallback

Qwen3.5-2B-Q4_K_M.gguf

Reason:

Best measured latency / throughput balance for interactive RK3588 CPU use
Strong enough for chat, light coding, and day-to-day local work

Slower but stronger Qwen quality tier

Qwen3.5-4B-Q4_K_M.gguf

Reason:

Slower than 2B on RK3588, but structurally better on harder prompts
Better treated as a quality tier than as the universal default

RX 7800 XT 16GB — MoE 35B at 128K context

Model:

Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf (13 GB)
MoE: 35B total / 3B active per token
IQ3_XXS — most aggressive quant that stays coherent; anything larger loses KV cache headroom for 128K

Validated settings:

GPU layers: 99 (full offload)
Flash attention: ON (mandatory for 128K)
Context: 131072 (128K)
Batch size: 512
Micro batch: 128
KV cache: --cache-type-k q8_0 --cache-type-v q8_0 (halves VRAM vs FP16)
Reasoning: --reasoning off (10x fewer output tokens, higher benchmark scores)
Throughput: ~60 tok/s

Why it works:

IQ quants crush K-quants on RDNA3: same model in Q3_K_L (14 GB) → 17 tok/s vs IQ4_XS (12.6 GB) → 60 tok/s
MoE is bandwidth-bound, not compute-bound — power cap from 190W to 80W shows near-zero throughput loss
GPU draws 56-81W during inference

RX 7800 XT 16GB — Dense 27B at 160K context

Model:

Qwen3-27B-Q4_K_M.gguf or similar dense 27B

Validated settings:

GPU layers: 99
Flash attention: ON
Context: 163840 (160K)
Batch size: 512
Micro batch: 128
KV cache: --cache-type-k q8_0 --cache-type-v q8_0

Context ceiling on 16GB:

Dense 27B: 160K max (192K OOMs by only 505MB)
MoE 35B/3B: comfortable at 128K
MoE 26B/A4B (e.g. Gemma-4-26B-A4B): 256K possible with IQ2_M + Q4_0 KV

What We Learned

On AMD RDNA3 / ROCm

IQ quants massively outperform K-quants on gfx1101 — the IQ decompression kernels are significantly better optimized for this architecture
MoE inference is bandwidth-bound, not compute-bound — cutting power cap by >50% barely changes throughput
Flash attention is mandatory for any context above ~32K on 16GB VRAM
--reasoning off on Qwen3.6 MoE gives 10x fewer output tokens for the same quality — worth it for always-on inference
KV cache q8_0/q8_0 is the sweet spot: halves VRAM vs FP16, better coherence than q4_0

On ARM / RK3588

REAP-pruned MoE models deliver strong quality-per-size on constrained hardware
Disabling reasoning scratchpad improved interactive latency materially on RK3588
KV cache quantization mattered enough to keep enabled by default
On this board, the best interactive profile came from better runtime tuning, not from enabling more model thinking
Even with reasoning disabled server-side, some models can still leak empty <think> wrappers, so user-visible response sanitation is still worth keeping

Current Status

Feature	Status
Web UI	✅ Production-ready
Persistent chats	✅ Production-ready
Model scanning/loading	✅ Production-ready
Streaming responses	✅ Production-ready
Benchmark helpers	✅ Production-ready
Windows-to-Windows RPC split serving	✅ Validated
RK3588-tuned presets	✅ Production-ready
ROCm/RX 7800 XT presets	✅ Production-ready
Windows setup flow	✅ Available
Knowledge Base / RAG	✅ Available
Knowledge-augmented chat	✅ Available
Corpus ingestion (multi-source)	✅ Available
Retrieval / memory pages	🔜 Planned
Personalized REAP build	🔜 Planned

Contributing

Contributions are welcome.

Fork the repo
Create a feature branch
Run ruff check . && mypy src/
Submit a PR

See CONTRIBUTING.md for contribution guidance.

License

MIT

Name		Name	Last commit message	Last commit date
Latest commit History 62 Commits
.github		.github
assets		assets
data		data
docs		docs
models		models
scripts		scripts
src/llama_webui		src/llama_webui
static		static
tests		tests
.env.example		.env.example
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
AGENTS.md		AGENTS.md
CONTRIBUTING.md		CONTRIBUTING.md
README.md		README.md
REAP_RK3588_NOTES.md		REAP_RK3588_NOTES.md
SETUP_MISSING_COMPONENTS.md		SETUP_MISSING_COMPONENTS.md
WINDOWS_SETUP_GUIDE.md		WINDOWS_SETUP_GUIDE.md
pyproject.toml		pyproject.toml
start_webui.bat		start_webui.bat
start_webui.ps1		start_webui.ps1
test_import.py		test_import.py

Folders and files

Latest commit

History

Repository files navigation

llama-webui

What This Is

What Makes It Different

Who It Is For

Compared To Other Options

Versus Ollama-based UIs

Versus desktop-first local AI tools

Versus ad hoc shell scripts

Why RK3588 Is A First-Class Target

Key Features

Repository Layout

Quick Start

Linux/macOS

Windows

llama.cpp RPC Mode

1. Prepare the RPC worker

2. Configure the WebUI coordinator

3. Choose a tensor split

Troubleshooting

Recommended RK3588 Defaults

Architecture

Configuration

Building llama.cpp

CUDA Build (NVIDIA GPU)

Linux/macOS

Windows

AMD ROCm Build (RX 7800 XT / RDNA3)

CPU-only / RK3588 Build

TurboQuant (TQ3_1S) for 16GB GPUs

GPU Backend Auto-Detection

Tested Hardware

ARM / Embedded

Windows Desktop / Laptop

AMD Linux Desktop (AI Node)

Windows-to-Windows RPC

Hardware Scope

Best Configurations We Found

Best interactive quality-per-size on RK3588

Fast daytime fallback

Slower but stronger Qwen quality tier

RX 7800 XT 16GB — MoE 35B at 128K context

RX 7800 XT 16GB — Dense 27B at 160K context

What We Learned

On AMD RDNA3 / ROCm

On ARM / RK3588

Current Status

Contributing

License

Related Links

About

Topics

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages