Feature Request: Support for Laguna S 2.1 DFlash

[sdroege]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

poolside's branch has support for this https://github.com/poolsideai/llama.cpp/tree/laguna . @joerowell worked on that.

I don't know the status of that but I assume it works. Unfortunately the branches diverged a bit but with a bit of effort I could get the relevant changes from there to apply on master. I don't understand enough about the code to submit a PR though, and there are other differences than just the DFlash support in the branch so that probably needs pulling things out into separate PRs.

Motivation

Laguna S 2.1 and DFlash in general are already supported and the DFlash can improve performance considerably.

Possible Implementation

No response

[joerowell]

Hey @sdroege! I plan to open a PR for this sometime this week, I just need to make sure it works properly on main.

[sdroege]

I don't know if it works correctly, but it works and I get something like 30-60% hit rate. It's unfortunately slower than not using DFlash on my setup but that's probably expected (limited by PCI / RAM bandwidth)

[sdroege]

This is the patch I'm currently running btw. Based on the poolside branch and merge conflicts fixed without much understanding. 0001-Full-laguna-support.patch

In case this helps, but feel free to ignore it :)

[wafir645fcqs98]

Thanks for sharing this. The patch is definitely helpful for understanding the current Laguna support work. I’ll take a look at it and the poolside changes.

…

On Fri, Aug 14, 2026 at 9:30 AM Sebastian Dröge ***@***.***> wrote: *sdroege* left a comment (ggml-org/llama.cpp#26669) <#26669 (comment)> This is the patch I'm currently running btw. Based on the poolside branch and merge conflicts fixed without much understanding. 0001-Full-laguna-support.patch <https://github.com/user-attachments/files/31058941/0001-Full-laguna-support.patch> In case this helps, but feel free to ignore it :) — Reply to this email directly, view it on GitHub <#26669?email_source=notifications&email_token=CKRFICDVT255DH4PHGSDUAD5J25YNA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRZGA3TENRXGY22M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-5290726765>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CKRFICHDFTVI4TM2REZMEND5J25YNAVCNFSNUABFKJSXA33TNF2G64TZHM3DCMRTGU2DOOBUHNEXG43VMU5TKMBXHEYTIMZTHA3KC5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/CKRFICBQK5ZEVTIW6QA4RWT5J25YNA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRZGA3TENRXGY22M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSVGM33PORSXEX3JN5ZQ> and Android <https://github.com/notifications/mobile/android/CKRFICGCUCOMOIZQJC3GOUL5J25YNA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRZGA3TENRXGY22M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSXGM33PORSXEX3BNZSHE33JMQ>. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[sjvrensburg]

Still reproduces on current master — ca3d5a3e1 (b10665), Vulkan/RADV, Strix Halo (gfx1151), target Laguna-S-2.1-UD-Q3_K_XL, drafter laguna-s-2.1-DFlash-BF16.gguf from poolside/Laguna-S-2.1-GGUF (unchanged upstream, lastModified 2026-08-06):

common_speculative_init_result: loading draft model '.../laguna-s-2.1-DFlash-BF16.gguf'
llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 76, got 69

One warning that seems worth adding now, because it is a very convincing false lead for anyone landing here this week: b10f9ca58 ("spec : add DFlash2 support (local convolution + candidate selector)", #27816, merged 2026-08-27) adds exactly seven new tensor types to the dflash arch, against a "expected 76, got 69" error. It is not related. The new tensors are created behind ml->get_tensor_meta("selector_hidden.weight") in src/models/dflash.cpp, and this drafter has no selector or conv tensors at all, so the branch never fires. Confirmed by dumping the file and by the run above.

The actual gap, in case it helps whoever splits the poolside branch into upstream PRs. The drafter holds 12 per-block tensors × 6 blocks + 4 globals = 76:

blk.{0..5}.{attn_norm, attn_q, attn_k, attn_v, attn_output,
            attn_q_norm, attn_k_norm, attn_gate,
            ffn_norm, ffn_gate, ffn_down, ffn_up}
fc.weight, enc.output_norm.weight, enc.aux_norm.weight, output_norm.weight

src/models/dflash.cpp creates all of these except two:

• blk.%d.attn_gate.weight — 6 tensors, [3072, 72] (per-head gate, head_count = 72). LLM_TENSOR_ATTN_GATE already exists at src/llama-arch.cpp:451 and src/models/laguna.cpp creates it for the target arch; the dflash builder just doesn't.
• enc.aux_norm.weight — 1 tensor, [3072, 6] (6 = block_count, and dflash.target_layers has 6 entries, so presumably one norm per tapped target layer). No such name exists anywhere in the tree.

6 + 1 = 7, which accounts for the whole 76/69 discrepancy.

Relevant KVs, all present and parsed fine:

general.architecture              = dflash
dflash.decoder_arch               = laguna
dflash.block_count                = 6
dflash.embedding_length           = 3072
dflash.attention.head_count       = 72
dflash.block_size                 = 16
dflash.target_layers              = [2, 11, 20, 30, ...]  (6 entries)
dflash.attention.sliding_window   = 512

So the load-side delta looks small and self-contained, whatever else the graph needs. @sdroege's patch presumably already covers this — this is only meant to name the minimal tensor delta explicitly, and to head off the DFlash2 red herring.

Happy to test patches on gfx1151/Vulkan.

[sdroege]

0001-Implement-Laguna-S-2.1-DFlash-support.patch here the minimal version of the changes required to make DFlash work without the other changes. Applies to latest master.

[sjvrensburg]

Tested — the patch works. Thanks for splitting the minimal version out.

Applies cleanly to master 557614e02 (b10713), builds clean on Vulkan. The expected 76, got 69 load
failure is gone: enc.aux_norm + the six blk.N.attn_gate.weight load, and the drafter initialises
with block_size=16, mask_token_id=12, n_extract=6, sample_from_anchor=true. Output is coherent, no
aborts, no NaNs — the sanitized … non-finite target feature values warning never fired on Vulkan
across ~4000 generated tokens, so the massive-activation f16 overflow you guarded for looks
Metal-specific.

Setup. Strix Halo (Ryzen AI Max+ 395, gfx1151, 128 GB), Vulkan/RADV, Mesa 26.1.8, kernel 7.1.10.
Target poolside/Laguna-S-2.1-GGUF UD-Q3_K_XL, drafter laguna-s-2.1-DFlash-BF16.gguf (the stock
upload, not reconverted). -fa on --ctx-size 8192 --parallel 1, temperature 0, cache_prompt: false.
Three prompts: 300-token code answer, 300-token prose answer, 40-token JSON answer. Control =
identical server, -md omitted; control decode is 33.7 / 32.9 / 33.8 t/s and is byte-identical
across three separate runs.

It is a win, but only below n_max = 8, and the docs example picks the wrong side of that.

--spec-draft-n-max	code	prose	struct	accept (code/prose/struct)
control	1.00x	1.00x	1.00x	—
2	1.20x	1.07x	1.31x	69.2 / 52.8 / 80.0 %
3	1.24x	1.00x	1.60x	58.1 / 38.5 / 84.8 %
4	1.17x	1.00x	1.58x	48.4 / 35.9 / 72.5 %
6	1.07x	0.81x	1.65x	36.2 / 23.3 / 64.6 %
8	0.38x	0.32x	0.59x	26.2 / 19.5 / 48.4 %
12	0.33x	0.27x	0.58x	17.8 / 12.9 / 38.1 %
15 (docs example)	0.31x	0.25x	0.54x	14.3 / 10.4 / 30.5 %

Two things there are worth separating.

The gentle part is ordinary speculative-decoding economics: acceptance decays with block length, so
past some point you pay for drafts you throw away. That alone would explain 6 → 8.

The part that does not look ordinary is the size of the step between 6 and 8: 36.0 → 12.7 t/s, a
2.8x drop, while acceptance only moves 36.2% → 26.2%. Between 8 and 15 the curve is flat again
(12.7 → 10.3) even though acceptance nearly halves. So the cliff is not the acceptance curve — it
reads like a batch-size threshold in the drafter or verify path rather than anything Laguna-specific.
I have not chased it further; flagging it in case it is a cheap win, and because anyone who copies
--spec-draft-n-max 15 out of the docs/speculative.md block this patch adds will land squarely in
it and conclude DFlash is a 3x regression. That is exactly what I concluded for the first hour. On
this box the peak is n_max = 3.

(Unrelated to this patch, but it cost me a measurement: the speculative.n_max request field is
silently ignored by llama-server — n_max 8, 4 and 2 sent per-request produced byte-identical
draft_n. It has to be the server flag.)

One caveat that is not a regression from this patch. With the drafter attached, greedy output is
not reproducible: at temperature 0 the two 300-token answers differ from the control, and differ
between two runs of the same drafted server, while the control is byte-identical across three runs.
The 40-token JSON answer matches everywhere. I see the same thing on this machine with --spec-type draft-mtp on two unrelated model/drafter pairs, so I read it as a property of the speculative path
on this build rather than anything you introduced here — but it does mean DFlash on Laguna is not
lossless in practice, and anyone using it where exact arguments matter (tool calls) should know that
before switching it on.

Happy to re-run any of this, or to test a fix for the n_max ≥ 8 cliff.