A hub for fast, verified Qwen serving recipes on consumer hardware. Every number here was measured on real machines with the exact commands published next to it. No screenshots without flags, no "trust me" throughput.
Community project, not affiliated with Alibaba or the Qwen team.
Current fleet: one box, "saturn": 6x RTX 5090 32 GB (sm120, PCIe, no NVLink), AMD 5955WX, 125 GB RAM. Recipes state how many of the six cards they use. More hardware will join as it arrives; recipes are tagged per configuration.
Newest first: the top row is always the most recently added or updated recipe.
| model | engine | hardware | context | decode | prefill @depth | KV pool | recipe |
|---|---|---|---|---|---|---|---|
| Qwen3.8-27B ternary, Bonsai 2 (PQ2_0 GGUF, 1.72 bpw) | llama.cpp, PrismML fork 5d80cff0 + 4 patches | 1x 5090 | 262,144 native | 139-141 tok/s on code, 540 on copy-heavy edits (n-gram drafts, thinking off) | 3,445 tok/s @30K, 2,303 @100K | 1 slot x 262,144 | recipe |
| Qwen3.8-Flash-Next (NVFP4) | vLLM nightly eed1f3d0 + overlay r10 | 4x 5090 | 393,216 (YaRN 1.5) | 319-331 tok/s on code, 106 engine steps/s | ~7,800 tok/s @36K-100K | 736,359 tok | recipe |
| Qwen3.8-Flash-Next (UD-Q4_K_XL GGUF) | llama.cpp PR #28243 + Unsloth MTP head | 4x 5090 | 131,072 | 130-156 tok/s on code (n-max 2-3), 218 on copy-heavy edits | 1,456-1,712 tok/s @4K | 1 slot x 131,072 | recipe |
| Qwen3.8-27B (NVFP4) | vLLM v0.28.0, no patches | 2x 5090 | 262,144 native | 157 tok/s (212-236 on code) | 4,600-5,050 tok/s | 833,295 tok | recipe |
Decode is single-stream on code-shaped prompts, greedy, warmup discarded.
Prefill is cold, measured at 99K-186K context. Full methodology in
tools/: every recipe is benchmarked with llm-bench, seeded and
cache-honest (fresh salt per run against a live server).
Mia'a AI Lab serves the same Qwen3.8-27B through
an in-house exllamav3 fork
(DFlash2 and MTP drafting, NVFP4/FP8 KV cache, GB10/aarch64 as well as x86
CUDA), with matching quants:
EXL3 3.5bpw target
and a
DFlash2 EXL3 5.0bpw draft,
plus a
deployment kit.
Mia'a AI Lab's numbers: 47.5 tok/s on a DGX Spark (4.43 tokens accepted
per step) and ~30-34.5 tok/s on 24 GB cards, hardware this repo does not
cover. That fork is also where our fp8 QSA-KV patch learned the sm12x rule
that fp8 operands need a bf16 upcast before tl.dot. Credit where it is
due.
Some recipes need patches that have not merged upstream yet. We do not ship prebuilt images: each recipe carries a small Dockerfile (pinned base digest + COPY of the patch files, all published in the repo) so you build in one command and can read every line of what you run. Each patch names its upstream thread and retires when the fix merges.
The findings behind these recipes are contributed back where they belong:
- 2026-09-19, in the recipe, not yet filed: llama.cpp CUDA gate/up/GLU fusion refused on 24 of 64 layers of a Qwen3.5-architecture model because the graph allocator places the GLU output on the shared activation's memory; on the quantized matvec path the kernel only reads a private q8_1 copy, so the overlap check can ignore that tensor
- 2026-09-19, in the recipe, not yet filed: dual-output matvec for llama.cpp's CUDA backend, running two batch-1 projections of the same input (
ssm_alpha+ssm_beta,attn_k+attn_v) as one launch - 2026-09-05, in the recipe, not yet filed: sm120 skinny GEMM plan table for the Qwen4Exp dense projections, opening upstream's CuTe DSL low-latency kernel (normally gated to sm90/sm103) to sm120
- 2026-09-05, in the recipe, not yet filed: local-argmax draft sampling for the Flash-Next MTP head, gathering (value, id) pairs from the local vocab shard instead of moving the full logits tensor per draft step
- 2026-09-05, in the recipe, not yet filed: custom IPC all-gather routing for the target's logits gather, reusing vLLM's existing custom-all-reduce buffers for a collective NCCL was handling before
- 2026-09-05, in the recipe, not yet filed: fused GDN qkvz+ba input projection, merging two per-layer GEMMs into one on all Gated DeltaNet layers
- 2026-09-05, in the recipe, not yet filed: FlashInfer all-reduce sm120 table and RDMA-flag fixes, adding a compute-capability-120 entry to vLLM's FlashInfer all-reduce size tables and gating FlashInfer's GPUDirect RDMA flag on the device actually supporting it
- 2026-09-05, in the recipe, not yet filed: custom all-reduce PCIe gate, allowing vLLM's one-shot IPC all-reduce kernel on more than two PCIe-only GPUs when NVML reports no NVLink on any pair
- 2026-09-05, in the recipe, not yet filed: three mixed-precision loader fixes for nvidia's Qwen3.8-Flash-Next-NVFP4 checkpoint (FP8 PLE table support in mixed ModelOpt checkpoints, the MTP layer-index remap for the quantized-layers map, and FP8_BLOCK_SCALES routed experts routed to vLLM's block-FP8 MoE method)
- 2026-08-28 vllm-project/recipes#870: Flash-Next verified at 200 tok/s decode on 4x RTX 5090, with the sm120 knobs explained
- 2026-08-28 vllm-project/vllm#54275: fix for the kv-cache-memory advisory that OOMs when other processes hold GPU memory
- 2026-08-28 fp8 KV for QSA, offered on vllm#53896: pool +78%, outputs md5-identical to bf16