Feature Request: adaptive-kv-streaming

[munchmunch77]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Please implement adaptive-kv-streaming

Motivation

I can run higher quants for both model and KV and have a much higher context with minimal drop in performance.

Possible Implementation

Add a flag to enable or disable it.

[mgscreativa]

Really appreciated if merged! now I'm experimenting with kvarn5 and MTP on 200k context on my RTX3090 24GB and this looks promising.

[Midaychi]

What's the difference between this and just lowering your -c context?
I tried an adaption of this and the context beyond the 'arena' window seems to functionally not exist for the model.

[munchmunch77]

You can lower the context if you happy with a smaller context or running at lower KV quants

[Anbeeld]

I checked the initial repo and the article and I failed to find the most important thing there: KLD or some other measurements of how this affects output. If you are already a user of this fork, could you please benchmark that, or create an issue asking for that?

The thing is, if output quality does not match the full resident even roughly, then there's barely any point in the feature as a whole.

[Midaychi]

You can lower the context if you happy with a smaller context or running at lower KV quants

That's what I mean though. If you use this then sure the context number can be bigger but the model still can't actually read all the context at once it can only address it in arena-sized windows, and I DON'T mean dynamically as it generates.

Setting an 8 GB arena but setting max context vs constraining your context number till it only uses 8 GB is functionally identical except that the former's window slides through that big context (but the model can only ever see that window and that window doesn't move WHILE it's generating.).

Well that and it tends to show the model repeating gibberish at the cut off point s

[munchmunch77]

@Midaychi Took me a while to figure out what you meant and I think I understand now.

@Anbeeld
I did generate KLD and it was very similar, close to a rounding error comparing F16 cache to q4_0 on stock llama.cpp and then comparing q4_0 from the fork against F16 from stock.

q4_0 stock llama.cpp vs F16 stock llama.cpp
====== KL divergence statistics ======
Mean KLD: 0.003116 ± 0.000023
Maximum KLD: 0.488065
99.9% KLD: 0.089937
99.0% KLD: 0.024898
95.0% KLD: 0.010323
90.0% KLD: 0.006790
Median KLD: 0.001594
10.0% KLD: 0.000021
5.0% KLD: 0.000004
1.0% KLD: -0.000000
0.1% KLD: -0.000002
Minimum KLD: -0.000068

q4_0 phase arena fork vs F16 stock llama.cpp
====== KL divergence statistics ======
Mean KLD: 0.003028 ± 0.000055
Maximum KLD: 5.046766
99.9% KLD: 0.072822
99.0% KLD: 0.023757
95.0% KLD: 0.009928
90.0% KLD: 0.006562
Median KLD: 0.001540
10.0% KLD: 0.000020
5.0% KLD: 0.000003
1.0% KLD: -0.000000
0.1% KLD: -0.000003
Minimum KLD: -0.000039

The big outlier is the Maximum KLD which spiked to over 5.

My main issue though was my tests were extremely limited due to my limited hardware, and I was only able to test at around 100 000ctx due to system ram running out.

[Anbeeld]

@munchmunch77 can you please clarify what model and quant was used, what was the baseline (wikitext-2?), and have you actually offloaded q4_0 to RAM in the fork test, and how much of it?

[munchmunch77]

As explained, I am not well equipped to run these tests, and not even sure if I am doing it right. But the base model was Qwen3.8-27B-UD-Q3_K_XL at F16/F16 and q4_0 was compared to that on the same model for the fork and llama.cpp. I did set a value for kv-stream-stage-mib but it's unclear if it offloaded or not. Normally q4_0 does not fit but my 48GB RAM was maxed out during the tests, so I really don't have room to try anything else. You can disregard my tests for that reason, but I tried.

[Anbeeld]

The fork probably requires adding a custom argument that would force offload of KV cache into VRAM after a specific point, in order to be able to A/B test KLD at all. I'll try it when I have some free time.