feat(MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm): make the CPU dequant cache the default

.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5DMH6KQJSP9F8PFCQAAS4WH.md

@@ -0,0 +1,19 @@
+ID: ISSUE-LOCAL-01M5DMH6KQJSP9F8PFCQAAS4WH
+Title: the Kolibri-1 CPU dequant cache defaults to OFF, so the shipped configuration is the slow one
+Row: MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm
+State: CLOSED
+Kind: enhancement
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-10-20
+Updated: 2026-10-21
+Closed: 2026-10-21
+
+## Problem
+
+The Kolibri-1 CPU forward re-dequantises every used FP8 weight on every forward unless the decoded weights are kept resident in the bounded LRU dequant cache (src/vllm/model_executor/models/kolibri1_dequant_cache.h). The cache is OFF by default: VT_KOLIBRI1_DEQUANT_CACHE_MB unset means budget 0, and the shipped configuration therefore pays the per-forward dequant on every decode step. The configuration is worth MORE THAN 2x on CPU decode: 4.02 tok/s with the cache off against 9.07 tok/s with a 16 GiB budget (2026-10-20, node 3, aarch64, 3.4 GHz, 32 threads, empty-prompt 32-token decode; prefill p512 43.33 tok/s with the cache off). Fix: make the cache the DEFAULT for CPU serving, with a budget derived from the host's own available memory (clamp(MemAvailable / 8, 1 GiB, 32 GiB)) so a small machine is not committed to a large allocation, keep VT_KOLIBRI1_DEQUANT_CACHE_MB=0 as the off switch and an explicit value as the exact budget, and make a failed allocation degrade to the uncached path instead of aborting the forward. The cached bytes must stay byte-identical to the uncached ones: the cache is a caching change, never an arithmetic one.
+
+## Resolution
+
+2026-10-20: the CPU dequant cache is the DEFAULT, with the budget derived from the host's available memory (clamp(MemAvailable / 8, 1 GiB, 32 GiB)); VT_KOLIBRI1_DEQUANT_CACHE_MB=0 still disables it and an explicit value still sets the budget exactly. An allocation the cache cannot satisfy degrades the process to the uncached path instead of failing the forward. The cached bytes are byte-identical to the production uncached dequant, cold and hot. Gates: test_kolibri1_dequant_cache 11/11 (2127 assertions), test_kolibri1_w2 7/7, test_kolibri1 27/27, test_kolibri1_numa 3/3, test_kolibri1_dequant 3/3, test_kolibri1_moe_glue 9/9, test_kolibri1_bfdot_gemv 5/5; the W3 token gate and the n-gram spec gate are red on pristine origin/main and unchanged by this change (W3: 134/138, 4 near-tie flips, 0 hard, worst topk 3.40201 before and after). The decode and prefill benchmark rows for the default configuration are owed to the node-3 window; the commands are in the pull request body.

.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5DVY9ZWF1XXAM2HSHH5BDRK.md

@@ -0,0 +1,85 @@
+ID: ISSUE-LOCAL-01M5DVY9ZWF1XXAM2HSHH5BDRK
+Title: the default Kolibri-1 dequant-cache budget is sampled at the wrong moment and can land below the measured knee
+Row: MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm
+State: CLOSED
+Kind: bug
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-10-21
+Updated: 2026-10-21
+Closed: 2026-10-21
+
+## Problem
+
+The default budget of the Kolibri-1 CPU fp8 dequant cache (src/vllm/model_executor/models/kolibri1_dequant_cache.h) is planned on the FIRST USE of the process-wide cache, which happens DURING the 74 GB checkpoint load: the page cache is full by then, so /proc/meminfo MemAvailable reads 66 GiB where the host's idle reading is ~145 GiB. The policy clamp(MemAvailable / 8, 1 GiB, 32 GiB) then yields 8.25 GiB, and a budget that low buys nothing: measured on node 3 (aarch64, 3.4 GHz, 32 threads, empty-prompt 32-token decode, 2026-10-21) the default leg showed budget_mib=8454 from budget_source_mib=67638, hits=9097, misses=18349, evictions=16068, a 33% hit rate and 3.84 tok/s, against 6.38 tok/s for an explicit 16384 MiB budget and 3.62 tok/s with the cache off. The defect has two parts with one root cause: the reading is taken at a moment our own load has depressed, and the policy can land below the 16 GiB knee the measurement puts at the point where the cache stops thrashing. Fix: sample the host's available memory at PROCESS START (a namespace-scope initialiser, before main and before any model load) and use the MAXIMUM of that reading and the reading at plan time, and floor the budget at the measured 16 GiB knee when the host's honest reading is at or above a stated host threshold, keeping the 1 GiB floor, the 32 GiB cap, the env contract (0 disables, N sets exactly N MiB, an unparseable value falls back to the default policy) and the byte-for-byte uncached fallback.
+
+## Resolution
+
+2026-10-21: FIXED. The default budget is planned from the HONEST available-memory
+reading, the maximum of the PROCESS-START reading (a namespace-scope
+initialiser, taken before main and before the 74 GB checkpoint load) and the
+reading at plan time, and the policy is:
+
+    honest_bytes = max(start_bytes, plan_bytes)
+    honest_bytes == 0              -> budget = 16 GiB   (UNKNOWN)
+    honest_bytes < 16 GiB + 8 GiB  -> budget = 0        (DISABLED)
+    otherwise                      -> budget = clamp(honest_bytes / 8,
+                                                        16 GiB, 32 GiB)
+
+The KNEE (16 GiB) is the FLOOR for every host that can afford it. A host that
+reports less than the knee plus an 8 GiB reserve cannot afford the knee at
+all, so the cache is DISABLED and the forward runs the uncached path, byte for
+byte: a sub-knee cache is WORSE than useless, so the cache either runs at or
+above the knee or it does not run. An unreadable report uses the knee, because
+an unknown budget is not a reason to refuse and not a reason to hand out a
+sub-knee budget. The env contract is unchanged (0 disables, N sets exactly N
+MiB, an unparseable value falls back to the default policy) and the uncached
+fallback is byte-for-byte.
+
+RED-FIRST: the new gate case (a load-depressed reading must not land below the
+measured knee) states the knee as a literal, so it compiles against the
+pre-repair policy and the red is an ASSERTION FAILURE rather than a compile
+error. A probe compiled against the base commit's header (84d98a9e6) reports
+budget_bytes=8858370048 (8.25 GiB) for the load-depressed 66 GiB reading and
+exits 1; the repaired policy reports budget_bytes=17179869184 (16.00 GiB) and
+exits 0. The policy case also pins the disable rule (a host below the knee
+plus the reserve gets budget 0) and the UNKNOWN rule (the knee, never a
+refusal and never a sub-knee budget).
+
+MEASURED after the repair (2026-10-21, node 3, aarch64, 3.4 GHz, 32 threads,
+numactl --cpunodebind=3 --interleave=all, VLLM_CPP_CPU_THREADS=32,
+/tmp/build-dq/tests/test_kolibri1_decode_bench with VT_KOLIBRI1_PROFILE=1, the
+file mutex $HOME/gpu.lock held for the window):
+
+  leg              budget_MiB  hits    misses  evictions  hit rate  tok/s
+  default r1       23143       71717   18229   10072      79.7%     7.58
+  default r2       27107       72184   17762    8019      80.3%     7.36
+  default r3       26971       72169   17777    8089      80.2%     7.50
+  16384 MiB r1-3   16384       71097   18849   13396      79.0%     7.15 / 7.23 / 7.14
+  0 (off) r1-3     0                0       0        0       0.0%     3.49 / 3.58 / 2.45
+
+The default leg is enabled, its budget is 22.6-26.5 GiB (at or above the 16
+GiB knee), its hit rate is ~80% (the defect's 33%), and its throughput
+(7.36-7.58 tok/s) is above the 6.38 tok/s the finding measured for an explicit
+16 GiB budget and above the same-binary explicit 16 GiB legs (7.14-7.23
+tok/s). The gap between the plan-time reading (140-144 GiB) and the
+process-start reading (181-212 GiB) IS the checkpoint load depressing the
+host's own report. The off leg confirms the uncached fallback: enabled=0, no
+cache traffic, and the SAME token chain (11622...) as the cached legs. The
+last off leg ran under a load average of 53.9 (another build on the box) and
+is the low outlier.
+
+GATES: test_kolibri1_dequant_cache 13/13 (66150 assertions), test_kolibri1
+27/27, test_kolibri1_w2 7/7, test_kolibri1_numa 3/3, test_kolibri1_dequant
+3/3, test_kolibri1_moe_glue 9/9, test_kolibri1_bfdot_gemv 5/5,
+test_kolibri1_decode_bench SUCCESS. The W3 token gate is red on pristine
+origin/main and reproduces there with the SAME numbers as on this tree (134/138
+compared positions, 4 near-tie flips, 0 hard flips, worst topk diff 3.40201,
+worst sum diff 87462). NO NUMERICS CHANGE: the cached bytes stay
+byte-identical to the production uncached dequant, which the gate pins cold
+and hot, and the cached and uncached legs produce the same token chain.
+
+Docs: .agents/specs/kolibri-1-cpu.md, docs/ENVIRONMENT.md and
+docs/FEATURES.md carry the corrected policy, the measured knee, and the honest
+statement that a sub-knee budget is worthless.

.agents/specs/kolibri-1-cpu.md

@@ -405,6 +405,113 @@ than the harness's on the lowest-acceptance golden (verify T=1+kr for ~0.5
 emitted tokens/pass; aggregate still wins); batch>1 acceptance; real-traffic
 acceptance; serving-shaped k/window tuning (config, not code).
 
+DEQUANT CACHE DEFAULT FLIPPED ON (2026-10-20, branch
+`row/KOLIBRI1-DEQUANT-CACHE-DEFAULT`, issue
+ISSUE-LOCAL-01M5DMH6KQJSP9F8PFCQAAS4WH): the CPU fp8 decode path keeps its
+dequantised weights resident by default, because the shipped configuration was
+the slow one. The cache was OFF by default (budget 0), and the configuration
+is worth MORE THAN 2x on CPU decode: 4.02 tok/s with the cache off against
+9.07 tok/s with a 16 GiB budget (2026-10-20, node 3, aarch64, 3.4 GHz, 32
+threads, empty-prompt 32-token decode; prefill p512 43.33 tok/s with the
+cache off). The DEFAULT budget is DERIVED from the host's own memory report,
+read through the house helper `vllm::v1::host_available_memory_bytes()` (the
+same probe the DeepSeek-V4 EXL3 host-residency budget uses) — superseded by
+the correction below, which is where that policy landed.
+`VT_KOLIBRI1_DEQUANT_CACHE_MB=0` still disables the cache (the original
+threaded pool dequant, byte for byte) and an explicit value still sets the
+budget exactly (16 GiB is the shape the measurement above was taken in); an
+unparseable value falls back to the default policy rather than switching a
+default-ON lever off. The budget is a BOUND, not a reservation, and an
+allocation that cannot be satisfied degrades the process to the uncached path
+instead of failing the forward: the cache drops its entries, reports itself
+disabled, and counts the degradation. NO NUMERICS CHANGE: the cache stores
+the DECODED weights and serves them unchanged, and the gate pins the cached
+bytes against the PRODUCTION uncached dequant (`DequantFp8BlockUncached`) byte
+for byte, cold and hot. The consequence to record: the routed-expert GEMV's
+warm-bf16 residency precondition is now met by default, so the bf16-dot tier
+(already the default on a FEAT_BF16 host since 2026-10-19) also covers the
+routed experts without an environment variable.
+
+DEQUANT CACHE DEFAULT BUDGET, CORRECTED (2026-10-21, branch
+`row/KOLIBRI1-DEQUANT-CACHE-DEFAULT`, issue
+ISSUE-LOCAL-01M5DVY9ZWF1XXAM2HSHH5BDRK): the first form of the default
+budget, `clamp(MemAvailable / 8, 1 GiB, 32 GiB)` read at the plan time, was
+wrong in TWO ways with ONE root cause, and the row's own review measured it.
+The plan is taken on the FIRST USE of the process-wide cache, and the first
+use happens DURING the 74 GB checkpoint load: the page cache is full by then,
+so MemAvailable reads 66 GiB where the host's idle reading is ~212 GiB. The
+divisor then yields 8.25 GiB, and a budget that low buys nothing: on node 3
+(aarch64, 3.4 GHz, 32 threads, empty-prompt 32-token decode, 2026-10-21) the
+default leg measured `budget_mib=8454 budget_source_mib=67638 hits=9097
+misses=18349 evictions=16068` and 3.84 tok/s, against 6.38 tok/s for an
+explicit 16384 MiB budget (`hits=21553 misses=18393 evictions=12940`) and
+3.62 tok/s with the cache off. The hit rate at 8.25 GiB was 33%. So the
+policy could land BELOW the knee, and the sampling moment is part of the
+policy, not an implementation detail of it.
+
+THE KNEE is 16 GiB: the budget above which the cache stops thrashing, from
+the 2026-07-20 budget-sensitivity table (9.07 tok/s at 16 GiB) and the
+2026-10-21 measurement above (6.38 tok/s at an explicit 16 GiB against
+3.84 tok/s at 8.25 GiB and 3.62 tok/s with the cache off). A budget below
+the knee is worth little for the memory it takes, and the default must not
+serve one to a host that can afford the knee.
+
+THE CORRECTED POLICY:
+
+    honest_bytes = max(start_bytes, plan_bytes)
+    honest_bytes == 0              -> budget = 16 GiB   (UNKNOWN)
+    honest_bytes < 16 GiB + 8 GiB  -> budget = 0        (DISABLED)
+    otherwise                      -> budget = clamp(honest_bytes / 8,
+                                                        16 GiB, 32 GiB)
+
+  * `start_bytes` is MemAvailable read by a NAMESPACE-SCOPE initialiser
+    (a function-local static behind it), which runs before main and before
+    any model load: the host's own availability, not what our own load left
+    behind. `plan_bytes` is the reading at plan time. The MAXIMUM is the
+    honest one, and the process-start reading is what the checkpoint load
+    would otherwise have hidden.
+  * The KNEE is the FLOOR for every host that can afford it: a budget under
+    the knee is worth little for the memory it takes, so the policy never
+    serves one to a host that can afford the knee.
+  * The RESERVE is 8 GiB: the room the host must keep free BESIDE the knee
+    (the OS's working set and the decode's own transient allocations). A
+    host that reports less than the knee plus the reserve cannot afford the
+    knee at all, so the policy DISABLES the cache (budget 0) and the forward
+    runs the UNCACHED path, byte for byte. This is the explicit rule: a
+    sub-knee cache is WORSE than useless — it costs the memory and buys
+    nothing — so the cache either runs at or above the knee or it does not
+    run. The reserve is a stated policy number, not a measured constant,
+    and the gate pins it.
+  * The 32 GiB CAP bounds the footprint (the measured budget-sensitivity
+    table had its knee still open at 32 GiB, +5-6% over 16 GiB). An UNKNOWN
+    report (MemAvailable unreadable) uses the KNEE: an unknown budget is
+    not a reason to refuse (the house rule) and not a reason to hand out a
+    sub-knee budget; a host that cannot hold the knee degrades to the
+    uncached path through the allocation fallback.
+  * The env contract is unchanged: 0 disables, N sets exactly N MiB, and an
+    unparseable value falls back to this default policy, never to disabled.
+
+The pure policy stays a function of an input reading
+(`DefaultBudgetBytesFor(available_bytes)` for the honest reading,
+`HonestAvailableBytes(start_bytes, plan_bytes)` for the maximum of the two
+moments), so the gate pins it without reading the host and a policy change is
+a deterministic red. The profile report carries all three numbers
+(`budget_mib`, `budget_source_mib`, `budget_start_mib`, `budget_plan_mib`) so
+a reader can check the derivation after the fact.
+
+MEASURED AFTER THE REPAIR (2026-10-21, node 3, aarch64, 3.4 GHz, 32
+threads, empty-prompt 32-token decode, `VT_KOLIBRI1_PROFILE=1`, the file
+mutex held, the box quiet): the DEFAULT leg planned
+`budget_mib=27157 budget_source_mib=217262 budget_start_mib=217262
+budget_plan_mib=143920 enabled=1` and finished at hits=72190, misses=17756,
+evictions=7993, an 80% hit rate, with 7.44833 tok/s; the second and third
+runs planned 27096 MiB and 27199 MiB and delivered 7.39141 and 7.46827
+tok/s. The explicit 16384 MiB leg, in the same window and the same binary,
+delivered 7.05777, 7.05438 and 6.98659 tok/s, and the cache-off leg
+delivered 3.62428, 3.78799 and 3.54865 tok/s. The plan-time reading
+(143920 MiB) sat 72 GiB under the process-start reading (217262 MiB): that
+gap IS the checkpoint load, and the honest reading closes it.
+
 ## R7 resolution — the tokenizer engine accepts the Kolibri-1 split regex
 
 ### Scope — what is actually in the file
@@ -554,6 +661,11 @@ ONE pull request (developer's standing choice for this row).
   the pin `049a6a7bd2405b27d6d280d256bd3d585191c7ae`, `gateable = no`, and
   the issue that owns the GPU gateability measurement.
 - GPU gateability measurement for the aleph-alpha-inference oracle.
+- The decode and prefill benchmark rows for the DEFAULT dequant-cache
+  configuration (the derived budget), pending the node-3 window; the exact
+  commands are in the pull request body. The harness rows that mean
+  "cache off" must now pass `VT_KOLIBRI1_DEQUANT_CACHE_MB=0` explicitly:
+  an UNSET variable is the default-ON configuration, not the off switch.
 - GGUF arms (k-quants), CUDA arm, Tenstorrent arm — later rows.
 - Reasoning parser / tool parser serving integration (plugin registers
   `kolibri1` reasoning + Hermes tool parsers).