From 213b6bdc57f1ae88c3895a099586f565bad233ce Mon Sep 17 00:00:00 2001 From: tilloh-bot Date: Sat, 5 Sep 2026 16:43:11 +0000 Subject: [PATCH] feat(llama.cpp): add experimental DFlash2 preset for Qwen3.8-27B 200ctx MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add [Qwen3.8-27B-UD-Q4_K_M-200ctx-q4_0-dflash2] as an experimental twin of the 200ctx MTP section: same target GGUF, 200k ctx, q4_0/q4_0 KV and thinking sampling, but speculation on the external DFlash2 block-diffusion drafter (incoai/Qwen3.8-27B-DFlash2-GGUF, Q4_K_M, 1.1 GB) instead of the embedded MTP head. spec-draft-n-max 7 per the drafter card; draft KV q4_0 follows the Muse-Glimmer dflash section. Drafter added to models.list. Experimental: 200k load with drafter and speed delta vs draft-mtp are not measured on hermine yet (no GPU in this environment). DFlash2 support is in stock llama.cpp (PR #27816, merged 2026-08-27; present in v0.4.0). Rationale and open questions: docs/models/Qwen3.8-27B.md. Co-Authored-By: Tim Köster <40566351+tilloh-dev@users.noreply.github.com> --- docs/models/Qwen3.8-27B.md | 9 +++++++ llama.cpp/models.list | 1 + llama.cpp/presets/models.example.ini | 39 ++++++++++++++++++++++++++++ 3 files changed, 49 insertions(+) diff --git a/docs/models/Qwen3.8-27B.md b/docs/models/Qwen3.8-27B.md index cbd79ce..ecac811 100644 --- a/docs/models/Qwen3.8-27B.md +++ b/docs/models/Qwen3.8-27B.md @@ -43,3 +43,12 @@ The section now has a deliberate twin for real-scenario A/B testing instead of a **q4_1 KV is disqualified on CUDA despite winning the KLD ladder.** It measured KLD 0.0074 (between q8_0's 0.0039 and q4_0's 0.0099) but generation drops to 30.5 t/s short (q4_0: 43.2) and prompt processing collapses to ~30 t/s vs q4_0's 1989 t/s (~65x) - a 103k prompt was 22 % done after 13 minutes and hit the client timeout. VRAM also lands oddly lower (20618 vs 21241 MiB), consistent with a different, non-fused kernel path. Sharpened rule: **matching K and V is necessary but not sufficient on CUDA - only pairs with a fused FA path are usable, and on this build/model that is q8_0/q8_0 and q4_0/q4_0 (throughput-verified); every other pair needs its own probe before use.** **Renamed 2026-08-15 (same day):** the twins are now `[Qwen3.8-27B-small]` (ex `[Qwen3.8-27B]`) and `[Qwen3.8-27B-large]` (ex `[Qwen3.8-27B-maxctx]`) in both preset files and in the hermine OpenCode provider (models map + whitelist, live and backup). lieselotte keeps the plain `Qwen3.8-27B` ID - its GGUF and preset were never verified on that box, and renaming the OpenCode entry without touching that machine's models.ini would have broken the ID linkage. This doc file keeps the base-model name. + +## DFlash2 speculation section (added 2026-09-05, unmeasured) + +`[Qwen3.8-27B-UD-Q4_K_M-200ctx-q4_0-dflash2]` is an **experimental** twin of the 200ctx MTP section: same target GGUF, ctx 200000, q4_0/q4_0 KV, thinking sampling, mmproj on CPU — but `spec-type = draft-dflash` with the external block-diffusion drafter `incoai/Qwen3.8-27B-DFlash2-GGUF` (Q4_K_M, 1.1 GB, added to `models.list`) instead of the embedded MTP head. `spec-draft-n-max = 7` and the Q4_K_M drafter follow the drafter's HF card (incoai/Qwen3.8-27B-DFlash2-GGUF); draft KV q4_0/q4_0 follows the only other dflash section in this file (Muse-Glimmer-30B). + +Upstream status: dflash support landed in stock llama.cpp via PR #27816 (merged 2026-08-27), so the prebuilt bootstrap build picks it up; the "build from PR #27342" instructions on the HF card are stale. + +**Nothing on this box is measured yet:** whether 200k + target + drafter + draft KV fits in 24 GB (the MTP section alone already needed ~2.6 GiB of embedded-head overhead at 229376; the external drafter adds its own 1.1 GB weights + KV), and whether dflash beats the 1.85× MTP number on this model. The drafter card reports GSM8K acceptance length ~5.1–5.4 (Q4_K_M: 5.39) on the *ggml-org* Q4_K_M target at their sampling settings — a different target quant, different engine, not comparable to the hermine MTP runs. First load should go through `llama-fit-params` / real `nvidia-smi`, and an A/B short-probe against the 200ctx MTP section before this section is trusted or wired into any OpenCode provider. + diff --git a/llama.cpp/models.list b/llama.cpp/models.list index 1470fa7..2749801 100644 --- a/llama.cpp/models.list +++ b/llama.cpp/models.list @@ -13,6 +13,7 @@ unsloth/Qwen3.8-27B-GGUF | Qwen3.8-27B-UD-Q4_K_M.gguf | Qwen3.8-27B unsloth/Qwen3.8-27B-GGUF | Qwen3.8-27B-UD-Q4_K_XL.gguf | Qwen3.8-27B unsloth/Qwen3.8-27B-GGUF | mmproj-F16.gguf | Qwen3.8-27B +incoai/Qwen3.8-27B-DFlash2-GGUF | Qwen3.8-27B-DFlash2-Q4_K_M.gguf | Qwen3.8-27B unsloth/Qwen3.6-27B-MTP-GGUF | Qwen3.6-27B-IQ4_XS.gguf | Qwen3.6-27B unsloth/Qwen3.6-35B-A3B-MTP-GGUF | Qwen3.6-35B-A3B-UD-IQ4_XS.gguf | Qwen3.6-35B-A3B unsloth/gemma-4-26B-A4B-it-qat-GGUF | gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf | gemma-4-26B-A4B diff --git a/llama.cpp/presets/models.example.ini b/llama.cpp/presets/models.example.ini index 2dd1a7e..3636afb 100644 --- a/llama.cpp/presets/models.example.ini +++ b/llama.cpp/presets/models.example.ini @@ -54,6 +54,45 @@ cache-ram = 16384 load-mode = dio kv-unified = true +# Experimental twin of the 200ctx section above: same target GGUF, same 200k +# / q4_0 KV / sampling, but speculation runs on the external DFlash2 +# block-diffusion drafter (incoai/Qwen3.8-27B-DFlash2-GGUF, Q4_K_M, ~1.1 GB) +# instead of the embedded MTP head. Upstream llama.cpp support (spec-type +# draft-dflash), n-max 7 per the drafter's HF card. NOT yet measured on +# hermine: 200k load with drafter + draft KV, and speed delta vs draft-mtp, +# are open. Draft cache q4_0/q4_0 follows the Muse-Glimmer dflash section. +# Drafter download: models.list (incoai/Qwen3.8-27B-DFlash2-GGUF). +[Qwen3.8-27B-UD-Q4_K_M-200ctx-q4_0-dflash2] +alias = Qwen3.8-27B-UD-Q4_K_M-DFlash2 +model = C:/Users/Anwender/AppData/Local/llama.cpp/models/Qwen3.8-27B/Qwen3.8-27B-UD-Q4_K_M.gguf +mmproj = C:/Users/Anwender/AppData/Local/llama.cpp/models/Qwen3.8-27B/mmproj-F16.gguf +mmproj-offload = false +model-draft = C:/Users/Anwender/AppData/Local/llama.cpp/models/Qwen3.8-27B/Qwen3.8-27B-DFlash2-Q4_K_M.gguf +device = CUDA0 +ctx-size = 200000 +n-gpu-layers = -1 +cache-type-k = q4_0 +cache-type-v = q4_0 +cache-type-k-draft = q4_0 +cache-type-v-draft = q4_0 +flash-attn = on +jinja = true +reasoning-format = auto +reasoning-preserve = true +parallel = 1 +cache-reuse = 256 +fit = off +spec-type = draft-dflash +spec-draft-n-max = 7 +temp = 1.0 +top-p = 0.95 +top-k = 20 +min-p = 0.0 +presence-penalty = 0.0 +cache-ram = 16384 +load-mode = dio +kv-unified = true + [Qwen3.8-27B-UD-Q4_K_M-150ctx-q8_0] alias = Qwen3.8-27B-small model = C:/Users/Anwender/AppData/Local/llama.cpp/models/Qwen3.8-27B/Qwen3.8-27B-UD-Q4_K_M.gguf