A local coding agent on an ordinary 8 GB graphics card, two to nine times faster than download-and-go. The card was never the limit: the RAM next to it decides how large a model it runs.
One command builds llama.cpp, fetches a model and sets up the
pi coding agent against it, tuned
to the last few hundred megabytes of the card. Then qwen-pi in your project: an agent that reads
your files, runs your tests and edits your code. No API key, no rate limit, nothing leaves the
machine.
Mixture-of-experts models only use a few billion parameters per token, so their experts can live in system RAM while the card holds the rest. More RAM, a bigger model:
| RAM next to the card | Model | Speed | Window | Good for |
|---|---|---|---|---|
| any | Ternary-Bonsai-2-27B, all on the card | 40 tok/s | 64k | small, well-scoped tasks |
| 32 GB | Qwen3.6-35B-A3B | 52-65 tok/s | 131k | most work: fast, and the default |
| 64 GB | Qwen3.8-Flash-Next, 125B | ~19 tok/s | 131k | the strongest, for when it may take longer |
./install.sh reads your RAM, shows which of the three run, and asks. How they did as agents on
the same task: the agent sessions.
You need an 8 GB NVIDIA or AMD card, Linux or WSL2, and the RAM from the table above.
git clone https://github.com/voxlo-dev/qwen-agent-8gb.git
cd qwen-agent-8gb
./install.sh # NVIDIA
BACKEND=vulkan ./install.sh # AMDIt checks your machine first and stops with a list of anything missing, before it downloads or compiles. Then a 10-30 minute build and the download; re-running is safe. On Windows, set up WSL2 first: Windows. Disk and the rest: Requirements.
qwen-pi # in your project directory; arguments go to piqwen-pi starts the server in the background, waits for the model and stops it after the last
session. To keep it running, start it yourself:
qwen-server # terminal 1, ready at "listening on http://127.0.0.1:8080"
qwen-pi # terminal 2The server is also a plain OpenAI-compatible endpoint at http://127.0.0.1:8080/v1 for any other
tool. qwen-studio opens the same model with the same settings in
Unsloth Studio's chat UI, at the same speed and VRAM:
details.
The same card and the same models, three levels of effort. Generation in tok/s, RTX 4060 Ti 8 GB, Ryzen 7 7800X3D, 64 GB RAM:
| Bonsai | Qwen3.6 | Qwen3.8-Flash | |
|---|---|---|---|
| Download and go: the GGUF with default settings | ~4 | ~25 | ~4 |
| This repo's configuration: the right llama.cpp tree, expert placement, KV cache, multi-token prediction; under WSL2 | 36 | 39-45 | 9-10 |
| Plus the system: native Linux, no display on the card | 39.6 | 52-65 | ~19 |
The biggest single step after the configuration: do not let this GPU drive your monitor. A
desktop takes 0.5-1.2 GB of VRAM and GPU time, both out of the model; plug the monitor into the
mainboard's iGPU. If you cannot, PROFILE=display makes room. Bonsai's WSL2 figure predates its
current llama.cpp pin, which is 8 % faster on native Linux. Where each number comes from:
docs/performance.md and the pages per model.
Speed falls as the context fills. Qwen3.8-Flash held 16.4 tok/s at 64k and 14.3 at 110k, up from
12.9 and 10.5, once the model's sparse attention stopped running over the whole cache at decode:
a patch in patches/unsloth/ (details).
Strata is an engine written for Qwen3.8-Flash alone. On a card with 12 GB or more, it is the faster choice: 53-94 tok/s on an RTX 5070 by its own numbers. On an 8 GB card with the same GGUF, this repo is the better fit:
| Same file (UD-IQ4_XS), same 8 GB card | this repo | Strata |
|---|---|---|
| Window | 131k | 4.6-8k, chosen by its setup on a card this size |
| Experts per token | all 10, exact | missed low-ranked experts skipped by default, about one quant step of accuracy by its own measurement |
| Decode | 19.0-19.6 tok/s short, 14.3 at 110k | 14.3 and 19.4 on two short requests at 8k |
| Coding agent | pi set up against it, the context budget measured | pi's requests are refused at that window until fit_max_tokens is set |
Strata is faster on 8 GB in one setup: its own 3-bit quant, measured less accurate than this repo's, ran an 8 GB laptop at ~29 tok/s with a 262k window. Nothing of Strata's transfers to llama.cpp by a flag, but reading it found the dense attention above. The full comparison: docs/qwen38-flash.md.
Each model has two profiles: dedicated (default) for a card that drives no display, display
for one that does. Settings live in config.env, each with a comment; an
environment variable always wins, and arguments go straight to llama-server:
PROFILE=display ./install.sh pi && PROFILE=display qwen-pi
MODEL=qwen38-flash qwen-pi
qwen-server --port 9000After changing the profile, the window or the port, run ./install.sh pi again so pi's copy
matches the server. The window and pi's budget values constrain each other:
Context budget.
Vision, off by default: the model reads images, e.g. the screenshots of its own browser tests,
through pi's read. The projector runs on the CPU, so it costs no VRAM, ~1 GB of RAM and ~10 s
per screenshot; it stays on once installed, VISION=0 turns it off:
details.
VISION=1 ./install.sh model piMeasured, each with a logged run:
| Card | System | Qwen3.6 | Qwen3.8-Flash | Bonsai |
|---|---|---|---|---|
| RTX 4060 Ti 8 GB | Linux Mint 22.3, native | 52-65 tok/s | ~19 tok/s | 39.6 tok/s |
| RTX 4060 Ti 8 GB | Windows 11 + WSL2, Ubuntu 26.04 | 39-45 tok/s | 9-10 tok/s | 36 tok/s |
| AMD RX 570 8 GB | Debian 13, Vulkan (Mesa 26.1) | ~25 tok/s | - | 7 tok/s; this repo's Vulkan patch reads prompts 35 % faster |
Expected to work, unmeasured: RTX 20xx to 40xx and RDNA2/3 cards with 8 GB or more; RTX 50xx needs CUDA >= 12.8. If you run one, a hardware report is the most useful thing you can send. Not supported: less than 8 GB of VRAM, ROCm, Metal, CPU-only.
Disk: 22 GB for Qwen3.6, 88 GB for Qwen3.8-Flash, 6 GB for Bonsai, plus ~10 GB for the build and
the CUDA toolkit. Drivers, toolkit versions, Node for pi and what deps installs per distro:
docs/setup.md.
./install.sh runs deps build model pi link in order; name steps to run only those.
| Step | Does |
|---|---|
deps |
the apt toolchain: CUDA or Vulkan, cmake, gcc (asks for sudo) |
build |
the model's llama.cpp tree at its pinned version (FORCE=1 rebuilds) |
model |
the GGUF, from the Hugging Face cache or downloaded, checksummed |
pi |
its own pinned pi with a config for the model and pi-bg-bash for background commands; your own pi and ~/.pi stay untouched |
link |
qwen-server, qwen-pi and qwen-studio into ~/.local/bin |
The model you pick is recorded in $QWEN_HOME/model.env and used by every qwen-* command. A
second one installs next to it with MODEL=bonsai ./install.sh and runs with MODEL=bonsai qwen-pi.
Everything lands in ~/.local/share/qwen-local (QWEN_HOME); an install from before the rename,
in ~/.local/share/bonsai-local, stays where it is and keeps running Bonsai.
Every non-default setting in this repo answers a failure seen on real hardware, and the
measurement behind it is in docs/. Speed, VRAM and windows are reproducible.
Agent quality is a handful of single sessions on one prompt, enough to put Qwen3.6 ahead of Bonsai
and not more; a reproducible test is next. The starting point was a study of eight local models
as coding agents on an 8 GB card: docs/model-comparison.md.
rm -rf ~/.local/share/qwen-local ~/.local/bin/qwen-server ~/.local/bin/qwen-pi ~/.local/bin/qwen-studio
rm -rf ~/.local/state/pi-bg # pi-bg-bash's task logsThat includes the models, pi and its sessions (bonsai-local for an install from before the
rename). The apt packages from deps stay.
Issues, pull requests and hardware reports welcome. CONTRIBUTING.md has the one rule: a non-default choice arrives with the measurement that justifies it. Working on the code: AGENTS.md.
This fills an 8 GB card to within a few hundred megabytes and keeps the GPU, and for the Qwen models tens of gigabytes of RAM, busy for as long as a session runs. Nothing here overclocks, raises a power limit or touches a fan curve. Your cooling, power supply and driver are yours. Provided as is, no warranty: see LICENSE.
- The Qwen team for Qwen3.6-35B-A3B and Qwen3.8-Flash-Next
- Unsloth for the GGUFs, the llama.cpp tree both Qwen models run on, and Unsloth Studio
- PrismML for Ternary Bonsai 2 27B and the llama.cpp fork that loads it
- ggml-org/llama.cpp, including whoever wrote the IQ-grid Vulkan shaders the patch follows, and the reporter of PrismML-Eng/llama.cpp#185, who made that work findable
- Mesa and RADV, and pi by earendil-works
The weights are their authors', under the licenses on their model cards, downloaded at install time and never redistributed here. The model comparison and the localagent workflow come from a project thesis at Technische Hochschule Mittelhessen by this repo's author.
License: MIT, including the PTQ1_0 Vulkan loader in patches/vulkan,
which is offered upstream under the same terms.