Merlin is a vLLM fork tuned for NVIDIA Ampere and Ada GPUs. It runs seven open models at their full 262K context (Nemotron 3.5 Super VL at its 1M context) on four RTX 3090s, behind one OpenAI-compatible API with vision, reasoning and tool calling. One install, one command with pre-tuned profiles for your detected GPUs.
git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine && ./setup.shThe setup script checks your GPUs, suggests a model, shows each download and its size before it starts, then
launches the server on 127.0.0.1 (port 8000; 8100 for the 27B and ThinkingCap). Nothing is installed system-wide. Run it again to restart the server.
The How to serve guide covers the rest: Merlin's commands and flags, talking to the server, serving on your network, picking GPUs and switching models.
Manual install
git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine
curl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.1/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl
MERLIN_WHEEL="$PWD/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl" bash release/install-env.sh ./venv
venv/bin/hf download halt95/Qwen3.8-Flash-Next-W4A16-Merlin --local-dir ~/models/flash-next
(cd ~/models/flash-next && sha256sum -c "$OLDPWD/models/flash-next/checkpoint.sha256")
venv/bin/merlin serve ~/models/flash-nextThe wheel is checked against release/merlin-wheel.sha256. Without MERLIN_WHEEL the installer builds the engine
from source, which takes about 1.5 hours on 8 cores. The first start compiles kernels; the server is ready when it
prints Application startup complete.
| Model | Type | GPUs | KV pool, tokens | Decode at 262K | 8 agents, thinking on | 8 agents, thinking off | GSM8K-200, thinking on | GSM8K-200, thinking off |
|---|---|---|---|---|---|---|---|---|
| Flash-Next | 125B MoE | 4 | 924,993 | 197 tok/s | 677 tok/s | 664 tok/s | 197 | 199 |
| Swift1.5 | Flash-Next fine-tune | 4 | 924,993 | 186 tok/s | 673 tok/s | 650 tok/s | 199 | 197 |
| Swift1.5-Huihui | Swift1.5, abliterated | 4 | 924,993 | 196 tok/s | 703 tok/s | 681 tok/s | 198 | 198 |
| Qwen3.8-27B | 27B dense | 1 / 2 / 4 | 271,506 / 826,689 / 2,012,105 | 71 / 143 / 238 tok/s | – / 705 / 954 tok/s | – / 615 / 854 tok/s | 198 (4 GPUs) | 195 / 194 / 197 |
| ThinkingCap | 27B fine-tune | 2 / 4 | 826,689 / 2,012,105 | 160 / 238 tok/s | 697 / 952 tok/s | 633 / 832 tok/s | 198 (4 GPUs) | 193 / 195 |
| Nemotron 3.5 Super VL | 120B MoE, 1M context | 4 | 1,856,367 | 93 tok/s | 432 tok/s | 432 tok/s | 197 | 195 |
| Ornith 1.5 | 122B MoE | 4 | 894,618 | 105 tok/s | 528 tok/s | 486 tok/s | 196 | 190 |
Merlin 3.0.1. Measured on 4 x RTX 3090 at 220 W with P2P enabled (aikitoria's open-gpu-kernel-modules P2P driver) and Zagitta's gpu-unlock. Decode is one stream with thinking on (reasoning effort medium where the checkpoint has one; the Nemotron 3.5 Super VL and Ornith 1.5 templates have no effort levels), at the full 262,144-token context. 8 agents is total throughput for eight simultaneous follow-ups on 16K conversations; with thinking on (effort medium) a 512-token follow-up may end inside the reasoning, and reasoning tokens count. GSM8K-200 is correct answers out of 200: thinking on at effort medium on this release, thinking off from the Merlin 3.0.0 runs (the model weights are unchanged). All models use an FP8 KV cache and MTP speculative decoding. The bench cards have the full figures and method.
- More context in the same VRAM. Embedding tables live in host RAM, so the cards hold more KV cache. Flash-Next keeps three full 262K conversations in memory on four 3090s.
- FP8 KV cache on Ampere, with our own kernels for cards that lack native FP8.
- Faster prefill. Up to 33 % faster at 262K than Merlin 3.0.0, with the optional gpu-unlock host upgrade.
- Built for agents. Follow-up turns resume from cache, even when more conversations are open than the cache can hold.
- KV RAM cache (experimental).
--kv-ram-cache 24gkeeps conversation prefixes the GPU evicts in host RAM and restores them with identical output: on Flash-Next a 262K restore takes 1.2–1.4 s instead of about 45 s to recompute. - Swarm mode (experimental).
--swarmraises the session cap on four RTX 3090s, from 8 to 24 on the Flash-Next family and from 24 to 32 on the 27B family, with a 48 GB KV RAM cache. Tested on Flash-Next and the 27B with P2P. - DFlash2 for the Flash-Next family.
--spec dflash2on four RTX 3090s with P2P decodes 5–18 % faster than MTP, which stays the default. - Vision, reasoning and tool calling on every model, with images up to 8 MP (Nemotron: its native ~3.4 MP; the 27B on one card: 4 MP).
- Low idle power. Four idle 3090s draw 110 W.
- One command.
merlin serve <model>picks the best layout for the GPUs it finds.
What Merlin adds to vLLM
More context in the same VRAM
- Token embedding tables in host RAM. The token embedding tables of the main model and its MTP drafter are read from pinned host memory, on by default for the Flash-Next and Qwen3.8-27B families (about 1.2 GiB per card freed on 27B at TP=2), with no speed loss.
- N-gram embedding table in host RAM. Flash-Next's n-gram embedding table (about 47.7 GiB) lives in system memory, one copy shared by every GPU, so the VRAM goes to KV cache: a 924,993-token pool on four RTX 3090s, three full 262K sessions resident at once.
- FP8 KV cache on Ampere. An sm_86 FP8 QSA attention reader for Flash-Next, and an sm_86 FP8 small-query verify tile for 27B, so every checkpoint keeps an FP8 KV cache on cards with no native FP8.
Faster decode and prefill
- RecoverSSM on every checkpoint. RecoverSSM recovers the GDN and Mamba-2 state for MTP speculative decoding, on by default for Flash-Next, Swift1.5, Swift1.5-Huihui, Qwen3.8-27B, ThinkingCap, Nemotron 3.5 Super VL and Ornith 1.5. The Flash-Next family drafts over the full vocabulary with a BF16 draft head.
- Faster prefill in 3.0.1, with the optional gpu-unlock host upgrade. Cold prefill of a full 262K prompt on four RTX 3090s against Merlin 3.0.0: +17 % on Flash-Next, +32 % on Qwen3.8-27B and ThinkingCap, +33 % on Nemotron 3.5 Super VL and Ornith 1.5. The engine wheel is unchanged. A stock-versus-unlocked A/B ties the gain to the unlocked host on Flash-Next and the 27B; Super VL and Ornith 1.5 had no A/B.
- DFlash2 drafter for the Flash-Next family.
--spec dflash2runs incoai's block-diffusion drafter on four RTX 3090s with P2P: decode 5–18 % faster than MTP, and the server holds three 262K sessions instead of six 131K ones. MTP stays the default. - All-reduce tuned for PCIe P2P. Flash-Next on P2P hosts uses a PCIe IPC all-reduce for decode-sized messages and NCCL's Simple protocol for prefill.
- Marlin W8A16 for Flash-Next's INT8 GDN projections.
Built for agents
- Follow-ups resume from cache. Prefix-cache retention keeps each conversation's prefix on a 64-token grid instead of the 4,096-token block, so the next turn starts from cache instead of re-reading the whole conversation (Flash-Next v2.5.1: median follow-up time to first token 0.96 s, was 1.56 s).
- Cache-thrash protection, on by default. Cost-aware admission and session-coherent yields keep follow-ups cached when more sessions are open than the pool holds: under 8 × 131K and 12 × 80K oversubscription, follow-up turns reused 74.7 % and 82.1 % of their prompt from cache.
- KV RAM cache (experimental).
--kv-ram-cache SIZEkeeps prefixes the GPU evicts in host RAM and restores them instead of recomputing them, with exactly the same output as a GPU cache hit. Flash-Next family with MTP and Qwen3.8-27B; off by default. - Swarm mode (experimental).
--swarmraises the session cap on four RTX 3090s, with or without P2P: 8 to 24 on Flash-Next, Swift1.5 and Swift1.5-Huihui, 24 to 32 on Qwen3.8-27B and ThinkingCap, with a 48 GB KV RAM cache. Tested on Flash-Next and the 27B with P2P; Swift1.5, Swift1.5-Huihui and every model without P2P use the same cap untested (⚫, expected to work). - A chat template that keeps the cache. Every Qwen recipe serves a fixed Qwen3.8 template that accepts system messages after the first turn, so coding agents' per-turn notes no longer break the prefix cache.
- Vision, reasoning and tool calling on every checkpoint, with images up to 8 MP (Nemotron its native ~3.4 MP,
one-card 27B 4 MP) and a per-image pixel cap (
MAX_PIXELS).
Steady in long runs
- Repeatable greedy output and fault detection on any rank, with faster shutdown.
- Idle cards drop to P8 with the runtime bounds guards on: 110 W idle on four RTX 3090s, against 446 W when the guard reader kept them awake.
One install, one command
./setup.shchecks the host, recommends a checkpoint and layout for your GPUs, shows every download with its size, installs the prebuilt engine (about 1.5 minutes, no compiling, nothing system-wide) and starts the server.merlin serve <checkpoint>recognises the checkpoint, picks the largest tested layout for the GPUs it finds, and starts the server;merlin doctorchecks the host first. Every layout is a profile oftools/merlin/launch.py; every knob (PORT,HOST,KV_BYTES,MAX_PIXELS, ...) is an environment variable.- Profiles beyond the reference host: RTX 3090 without P2P, eight RTX 3090s, RTX 4090 (24 and 48 GB), RTX A6000 / A40, A100, A30 and CMP 170HX.
- One prebuilt wheel for sm_80, sm_86 and sm_89, installed from a hash-locked requirements file.
- New variants just work. A Flash-Next-family checkpoint that ships its own FP8 KV scales is served with them, with no repository update.
Prefill and time to first token at 262K
| Model | GPUs | Prefill | Time to first token |
|---|---|---|---|
| Flash-Next | 4 | 6,803 tok/s | 38 s |
| Swift1.5 | 4 | 6,783 tok/s | 39 s |
| Swift1.5-Huihui | 4 | 6,866 tok/s | 38 s |
| Qwen3.8-27B | 1 / 2 / 4 | 579 / 1,115 / 2,101 tok/s | 452 / 235 / 125 s |
| ThinkingCap | 2 / 4 | 1,114 / 2,099 tok/s | 235 / 125 s |
| Nemotron 3.5 Super VL | 4 | 3,916 tok/s | 67 s |
| Ornith 1.5 | 4 | 2,261 tok/s | 116 s |
One stream, cold cache, 261,888-token prompt.
Bench cards: Flash-Next · Swift1.5 · Swift1.5-Huihui · Qwen3.8-27B · ThinkingCap · Nemotron 3.5 Super VL · Ornith 1.5
| GPUs | Status |
|---|---|
| 2× or 4× RTX 3090 | 🟢 Tested and tuned |
| 4× RTX 3090 without P2P | 🟠 Partly tested (Ornith 1.5, Flash-Next PP4), tuning in progress |
| 4× RTX 4090 | 🟡 Testing on cloud-hosted cards, tuning in progress |
| 2× or 4× CMP 170HX 64 GB | 🟡 Testing on cloud-hosted cards, tuning in progress |
| 8× RTX 3090, RTX 4090 48 GB, A100, A30 | ⚫ Profile included, not yet tested by us |
Any NVIDIA sm_80, sm_86 or sm_89 card can run Merlin. The How to serve guide lists which models fit on which cards, and how many full-length conversations each setup holds.
On RTX 3090s, PCIe P2P and Zagitta's gpu-unlock are optional host upgrades: Merlin runs without them, and our 3.0.1 bench numbers are measured with both. In a stock-versus-unlocked A/B the unlock sped up prefill by about 16–29 %; decode and output are unchanged.
| OS | Linux x86-64, glibc 2.38 or newer |
| Driver | NVIDIA R580 or later (CUDA 13) |
| Python | 3.13 with uv |
| Compiler | GCC 13 or newer |
| Host RAM | 72 GiB for the Flash-Next family on four cards (tested minimum; more with --kv-ram-cache or --swarm) |
| Disk | About 115 GiB for Flash-Next, 66 GiB for Ornith 1.5 |
merlin doctor checks your system before anything starts.
Models: Qwen · UkisAI (Swift 1.5) · huihui-ai (abliteration) · BottleCap AI (ThinkingCap) · NVIDIA (Nemotron) · ornith-ai (Ornith 1.5, a fine-tune of Qwen's Qwen3.5-122B-A10B)
Quantisation: Intel (AutoRound) · RadixArk (FP8 PLE table) · DominikBucko (MTP packing) · philbert (AWQ)
Engine: vLLM, FlashInfer, compressed-tensors and the vLLM community's
PLE-offload work · Sumandora (abliteration method) ·
froggeric (chat template) · aikitoria
(P2P on consumer Ampere) · Zagitta (GPU unlock) · incoai (DFlash2 drafter) ·
club-3090 (sm_80 FP8 KV decoder port) · Sudzpel (GPU unlock proposal and measurements) ·
brandon-nuttall (YaRN past 262K report, SEQS/MAXLEN launcher fix, use_local_argmax_reduction, TP2×PP4 community rig) · R-omk (YaRN past 262K crash report and recipes) · CoolShades (idle-power report and first fix behind the v2.5.2 engine fix) ·
Neiko2002 · Yarikdevcom (independent work on the same hardware class)
Upstream fixes are credited in the engine repository.
Merlin is built and tested on one home machine. If it saves you time, you can support it on Ko-fi.
The engine is Apache-2.0, like vLLM; see LICENSE and NOTICE. Each model has its own licence on its Hugging Face card.
Coming from qwen38-flash-next-3090s? This repository replaces it. Run
git remote set-url origin https://github.com/halt95/merlin-engine.git, then reinstall with ./setup.sh.