Local LLMs on an AMD RX 9070 XT + a two-tier LLM workflow

Running capable LLMs locally on an RDNA4 Radeon GPU, and pairing it with Claude for higher planning and complex reasoning.

Status: early. This is a working setup I use daily, but it's young. Numbers and claims here are what I'm seeing so far, not settled truth. I'm testing it over the coming weeks and will confirm things that hold (or don't).

Two things this repo is about

1. Getting LLMs running well on an AMD RX 9070 XT (RDNA4). Almost every guide points at ROCm. On RDNA4, that was a fight:

• Missing kernels — gfx1201 wasn't in vLLM's tuned paths. Slow fallback or nothing.
• Silent FP8 trap — ROCm dequantized FP8 weights to FP32 at load. Half speed, no warning.
• Vulkan won — Mesa RADV runs inference out of the box and beat ROCm in every test.

Engine	Tokens/sec
llama.cpp (Vulkan)	62
vLLM (ROCm)	48

A 35B model generates at roughly 60 tokens a second on 16GB VRAM. ROCm still earns its place for image gen and fine-tuning, but Vulkan won inference.

2. A two-tier workflow: local Qwen + Claude Opus. Local model for the bulk and the private work. Cloud model for the hard thinking. They split cleanly:

• Local layer — Qwen3.6 35B A3B, served by llama-server (Vulkan) and driven from the terminal with OpenCode and Aider. Free, fully private, on-GPU. This handles most agent work and anything I don't want leaving the machine.
• Cloud layer — Claude Opus via Claude Code, for planning, architecture, and the reasoning-heavy passes where a frontier model earns its cost.

The rough pattern: plan with Claude, execute with Qwen. Keep sensitive work on the local side. Reach for the cloud when the problem is more intricate.

Every backend speaks the OpenAI API, so the front ends (editor, scripts, tools) don't care which engine is underneath. The model is just a service at localhost:8080.

The hardware

• GPU: AMD RX 9070 XT, 16GB VRAM (RDNA4)
• CPU: Ryzen 7 9800X3D
• RAM: 64GB DDR5
• OS: Ubuntu 26.04

The 16GB of VRAM is the constraint everything bends around. A 35B fits because Qwen3.6 is a Mixture-of-Experts model: 35B total, only ~3B active per token, so the hot parts stay on the GPU and the rest spill to system RAM.

What I've built on it

Each folder is its own thing:

Project	Status	What it does
Continue.dev	✅ daily driver	Coding assistant — chat + edit
RAG	✅ working	Search over my docs — local Qdrant + reranker, refuses to guess
Tool calling	✅ working	Agent loop — model picks tools, I run them, it continues
Structured output	✅ working	Grammar-constrained JSON — parses every time
Flashcard gen	✅ working	Notes → Anki cards (built end-to-end by the local model)

Building the demos with the setup itself

I didn't write most of the showcase folders by hand. I had the local model build them from the terminal. Good test of whether any of this holds up for real work.

OpenCode (driving Qwen) won easily over Aider on the same jobs. It rewrites whole files, which suits a local model. Aider's default surgical-diff edits tripped Qwen up until I switched it to whole-file mode too. Lesson logged: for a local model, whole-file editing beats diff editing.

The output always needed a once-over. Small bugs, off-spec markup, code copy-pasted instead of reused. All caught in review. The local model does real work and saves real time. It doesn't run unsupervised.

Where it falls short ⚠️

• 16GB is still 16GB. MoE fits a 35B, but big dense models are out.
• Context capped at 32k. The model can do more, but every token of context is VRAM reserved up front.
• No editor autocomplete. Not enough spare VRAM to run a second model alongside the 35B.
• The local model needs supervision. Real work, but small mistakes on most jobs. Review everything.

What's next 🔜

• More detailed README — document the entire setup and thought process deeply
• Ollama — best ecosystem, but its RDNA4 Vulkan support was still flaky when I set this up
• ROCm, for what it's good at — image generation and fine-tuning lean on it. It comes back for those.
• Speculative decoding — pays off most on big dense models; marginal on a sparse MoE, deferred
• RAG tuning — bigger embeddings, hybrid search, smarter chunking. The pipeline works; these sharpen it

What's in this repo

• rag/ — retrieval over your own docs (local Qdrant + reranker + Qwen)
• tool-calling/ — a working function-calling agent loop
• structured-output/ — schema-constrained, guaranteed-valid output
• flashcard-gen/ — a notes-to-flashcards demo built end to end by the local model
• USAGE.md — day-to-day commands and when to use what