diff --git a/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5DMH6KQJSP9F8PFCQAAS4WH.md b/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5DMH6KQJSP9F8PFCQAAS4WH.md new file mode 100644 index 000000000..7a0f291e8 --- /dev/null +++ b/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5DMH6KQJSP9F8PFCQAAS4WH.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M5DMH6KQJSP9F8PFCQAAS4WH +Title: the Kolibri-1 CPU dequant cache defaults to OFF, so the shipped configuration is the slow one +Row: MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm +State: CLOSED +Kind: enhancement +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-10-20 +Updated: 2026-10-21 +Closed: 2026-10-21 + +## Problem + +The Kolibri-1 CPU forward re-dequantises every used FP8 weight on every forward unless the decoded weights are kept resident in the bounded LRU dequant cache (src/vllm/model_executor/models/kolibri1_dequant_cache.h). The cache is OFF by default: VT_KOLIBRI1_DEQUANT_CACHE_MB unset means budget 0, and the shipped configuration therefore pays the per-forward dequant on every decode step. The configuration is worth MORE THAN 2x on CPU decode: 4.02 tok/s with the cache off against 9.07 tok/s with a 16 GiB budget (2026-10-20, node 3, aarch64, 3.4 GHz, 32 threads, empty-prompt 32-token decode; prefill p512 43.33 tok/s with the cache off). Fix: make the cache the DEFAULT for CPU serving, with a budget derived from the host's own available memory (clamp(MemAvailable / 8, 1 GiB, 32 GiB)) so a small machine is not committed to a large allocation, keep VT_KOLIBRI1_DEQUANT_CACHE_MB=0 as the off switch and an explicit value as the exact budget, and make a failed allocation degrade to the uncached path instead of aborting the forward. The cached bytes must stay byte-identical to the uncached ones: the cache is a caching change, never an arithmetic one. + +## Resolution + +2026-10-20: the CPU dequant cache is the DEFAULT, with the budget derived from the host's available memory (clamp(MemAvailable / 8, 1 GiB, 32 GiB)); VT_KOLIBRI1_DEQUANT_CACHE_MB=0 still disables it and an explicit value still sets the budget exactly. An allocation the cache cannot satisfy degrades the process to the uncached path instead of failing the forward. The cached bytes are byte-identical to the production uncached dequant, cold and hot. Gates: test_kolibri1_dequant_cache 11/11 (2127 assertions), test_kolibri1_w2 7/7, test_kolibri1 27/27, test_kolibri1_numa 3/3, test_kolibri1_dequant 3/3, test_kolibri1_moe_glue 9/9, test_kolibri1_bfdot_gemv 5/5; the W3 token gate and the n-gram spec gate are red on pristine origin/main and unchanged by this change (W3: 134/138, 4 near-tie flips, 0 hard, worst topk 3.40201 before and after). The decode and prefill benchmark rows for the default configuration are owed to the node-3 window; the commands are in the pull request body. diff --git a/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5DVY9ZWF1XXAM2HSHH5BDRK.md b/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5DVY9ZWF1XXAM2HSHH5BDRK.md new file mode 100644 index 000000000..dfac30976 --- /dev/null +++ b/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5DVY9ZWF1XXAM2HSHH5BDRK.md @@ -0,0 +1,85 @@ +ID: ISSUE-LOCAL-01M5DVY9ZWF1XXAM2HSHH5BDRK +Title: the default Kolibri-1 dequant-cache budget is sampled at the wrong moment and can land below the measured knee +Row: MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm +State: CLOSED +Kind: bug +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-10-21 +Updated: 2026-10-21 +Closed: 2026-10-21 + +## Problem + +The default budget of the Kolibri-1 CPU fp8 dequant cache (src/vllm/model_executor/models/kolibri1_dequant_cache.h) is planned on the FIRST USE of the process-wide cache, which happens DURING the 74 GB checkpoint load: the page cache is full by then, so /proc/meminfo MemAvailable reads 66 GiB where the host's idle reading is ~145 GiB. The policy clamp(MemAvailable / 8, 1 GiB, 32 GiB) then yields 8.25 GiB, and a budget that low buys nothing: measured on node 3 (aarch64, 3.4 GHz, 32 threads, empty-prompt 32-token decode, 2026-10-21) the default leg showed budget_mib=8454 from budget_source_mib=67638, hits=9097, misses=18349, evictions=16068, a 33% hit rate and 3.84 tok/s, against 6.38 tok/s for an explicit 16384 MiB budget and 3.62 tok/s with the cache off. The defect has two parts with one root cause: the reading is taken at a moment our own load has depressed, and the policy can land below the 16 GiB knee the measurement puts at the point where the cache stops thrashing. Fix: sample the host's available memory at PROCESS START (a namespace-scope initialiser, before main and before any model load) and use the MAXIMUM of that reading and the reading at plan time, and floor the budget at the measured 16 GiB knee when the host's honest reading is at or above a stated host threshold, keeping the 1 GiB floor, the 32 GiB cap, the env contract (0 disables, N sets exactly N MiB, an unparseable value falls back to the default policy) and the byte-for-byte uncached fallback. + +## Resolution + +2026-10-21: FIXED. The default budget is planned from the HONEST available-memory +reading, the maximum of the PROCESS-START reading (a namespace-scope +initialiser, taken before main and before the 74 GB checkpoint load) and the +reading at plan time, and the policy is: + + honest_bytes = max(start_bytes, plan_bytes) + honest_bytes == 0 -> budget = 16 GiB (UNKNOWN) + honest_bytes < 16 GiB + 8 GiB -> budget = 0 (DISABLED) + otherwise -> budget = clamp(honest_bytes / 8, + 16 GiB, 32 GiB) + +The KNEE (16 GiB) is the FLOOR for every host that can afford it. A host that +reports less than the knee plus an 8 GiB reserve cannot afford the knee at +all, so the cache is DISABLED and the forward runs the uncached path, byte for +byte: a sub-knee cache is WORSE than useless, so the cache either runs at or +above the knee or it does not run. An unreadable report uses the knee, because +an unknown budget is not a reason to refuse and not a reason to hand out a +sub-knee budget. The env contract is unchanged (0 disables, N sets exactly N +MiB, an unparseable value falls back to the default policy) and the uncached +fallback is byte-for-byte. + +RED-FIRST: the new gate case (a load-depressed reading must not land below the +measured knee) states the knee as a literal, so it compiles against the +pre-repair policy and the red is an ASSERTION FAILURE rather than a compile +error. A probe compiled against the base commit's header (84d98a9e6) reports +budget_bytes=8858370048 (8.25 GiB) for the load-depressed 66 GiB reading and +exits 1; the repaired policy reports budget_bytes=17179869184 (16.00 GiB) and +exits 0. The policy case also pins the disable rule (a host below the knee +plus the reserve gets budget 0) and the UNKNOWN rule (the knee, never a +refusal and never a sub-knee budget). + +MEASURED after the repair (2026-10-21, node 3, aarch64, 3.4 GHz, 32 threads, +numactl --cpunodebind=3 --interleave=all, VLLM_CPP_CPU_THREADS=32, +/tmp/build-dq/tests/test_kolibri1_decode_bench with VT_KOLIBRI1_PROFILE=1, the +file mutex $HOME/gpu.lock held for the window): + + leg budget_MiB hits misses evictions hit rate tok/s + default r1 23143 71717 18229 10072 79.7% 7.58 + default r2 27107 72184 17762 8019 80.3% 7.36 + default r3 26971 72169 17777 8089 80.2% 7.50 + 16384 MiB r1-3 16384 71097 18849 13396 79.0% 7.15 / 7.23 / 7.14 + 0 (off) r1-3 0 0 0 0 0.0% 3.49 / 3.58 / 2.45 + +The default leg is enabled, its budget is 22.6-26.5 GiB (at or above the 16 +GiB knee), its hit rate is ~80% (the defect's 33%), and its throughput +(7.36-7.58 tok/s) is above the 6.38 tok/s the finding measured for an explicit +16 GiB budget and above the same-binary explicit 16 GiB legs (7.14-7.23 +tok/s). The gap between the plan-time reading (140-144 GiB) and the +process-start reading (181-212 GiB) IS the checkpoint load depressing the +host's own report. The off leg confirms the uncached fallback: enabled=0, no +cache traffic, and the SAME token chain (11622...) as the cached legs. The +last off leg ran under a load average of 53.9 (another build on the box) and +is the low outlier. + +GATES: test_kolibri1_dequant_cache 13/13 (66150 assertions), test_kolibri1 +27/27, test_kolibri1_w2 7/7, test_kolibri1_numa 3/3, test_kolibri1_dequant +3/3, test_kolibri1_moe_glue 9/9, test_kolibri1_bfdot_gemv 5/5, +test_kolibri1_decode_bench SUCCESS. The W3 token gate is red on pristine +origin/main and reproduces there with the SAME numbers as on this tree (134/138 +compared positions, 4 near-tie flips, 0 hard flips, worst topk diff 3.40201, +worst sum diff 87462). NO NUMERICS CHANGE: the cached bytes stay +byte-identical to the production uncached dequant, which the gate pins cold +and hot, and the cached and uncached legs produce the same token chain. + +Docs: .agents/specs/kolibri-1-cpu.md, docs/ENVIRONMENT.md and +docs/FEATURES.md carry the corrected policy, the measured knee, and the honest +statement that a sub-knee budget is worthless. diff --git a/.agents/specs/kolibri-1-cpu.md b/.agents/specs/kolibri-1-cpu.md index b64de0811..d6f305705 100644 --- a/.agents/specs/kolibri-1-cpu.md +++ b/.agents/specs/kolibri-1-cpu.md @@ -405,6 +405,113 @@ than the harness's on the lowest-acceptance golden (verify T=1+kr for ~0.5 emitted tokens/pass; aggregate still wins); batch>1 acceptance; real-traffic acceptance; serving-shaped k/window tuning (config, not code). +DEQUANT CACHE DEFAULT FLIPPED ON (2026-10-20, branch +`row/KOLIBRI1-DEQUANT-CACHE-DEFAULT`, issue +ISSUE-LOCAL-01M5DMH6KQJSP9F8PFCQAAS4WH): the CPU fp8 decode path keeps its +dequantised weights resident by default, because the shipped configuration was +the slow one. The cache was OFF by default (budget 0), and the configuration +is worth MORE THAN 2x on CPU decode: 4.02 tok/s with the cache off against +9.07 tok/s with a 16 GiB budget (2026-10-20, node 3, aarch64, 3.4 GHz, 32 +threads, empty-prompt 32-token decode; prefill p512 43.33 tok/s with the +cache off). The DEFAULT budget is DERIVED from the host's own memory report, +read through the house helper `vllm::v1::host_available_memory_bytes()` (the +same probe the DeepSeek-V4 EXL3 host-residency budget uses) — superseded by +the correction below, which is where that policy landed. +`VT_KOLIBRI1_DEQUANT_CACHE_MB=0` still disables the cache (the original +threaded pool dequant, byte for byte) and an explicit value still sets the +budget exactly (16 GiB is the shape the measurement above was taken in); an +unparseable value falls back to the default policy rather than switching a +default-ON lever off. The budget is a BOUND, not a reservation, and an +allocation that cannot be satisfied degrades the process to the uncached path +instead of failing the forward: the cache drops its entries, reports itself +disabled, and counts the degradation. NO NUMERICS CHANGE: the cache stores +the DECODED weights and serves them unchanged, and the gate pins the cached +bytes against the PRODUCTION uncached dequant (`DequantFp8BlockUncached`) byte +for byte, cold and hot. The consequence to record: the routed-expert GEMV's +warm-bf16 residency precondition is now met by default, so the bf16-dot tier +(already the default on a FEAT_BF16 host since 2026-10-19) also covers the +routed experts without an environment variable. + +DEQUANT CACHE DEFAULT BUDGET, CORRECTED (2026-10-21, branch +`row/KOLIBRI1-DEQUANT-CACHE-DEFAULT`, issue +ISSUE-LOCAL-01M5DVY9ZWF1XXAM2HSHH5BDRK): the first form of the default +budget, `clamp(MemAvailable / 8, 1 GiB, 32 GiB)` read at the plan time, was +wrong in TWO ways with ONE root cause, and the row's own review measured it. +The plan is taken on the FIRST USE of the process-wide cache, and the first +use happens DURING the 74 GB checkpoint load: the page cache is full by then, +so MemAvailable reads 66 GiB where the host's idle reading is ~212 GiB. The +divisor then yields 8.25 GiB, and a budget that low buys nothing: on node 3 +(aarch64, 3.4 GHz, 32 threads, empty-prompt 32-token decode, 2026-10-21) the +default leg measured `budget_mib=8454 budget_source_mib=67638 hits=9097 +misses=18349 evictions=16068` and 3.84 tok/s, against 6.38 tok/s for an +explicit 16384 MiB budget (`hits=21553 misses=18393 evictions=12940`) and +3.62 tok/s with the cache off. The hit rate at 8.25 GiB was 33%. So the +policy could land BELOW the knee, and the sampling moment is part of the +policy, not an implementation detail of it. + +THE KNEE is 16 GiB: the budget above which the cache stops thrashing, from +the 2026-07-20 budget-sensitivity table (9.07 tok/s at 16 GiB) and the +2026-10-21 measurement above (6.38 tok/s at an explicit 16 GiB against +3.84 tok/s at 8.25 GiB and 3.62 tok/s with the cache off). A budget below +the knee is worth little for the memory it takes, and the default must not +serve one to a host that can afford the knee. + +THE CORRECTED POLICY: + + honest_bytes = max(start_bytes, plan_bytes) + honest_bytes == 0 -> budget = 16 GiB (UNKNOWN) + honest_bytes < 16 GiB + 8 GiB -> budget = 0 (DISABLED) + otherwise -> budget = clamp(honest_bytes / 8, + 16 GiB, 32 GiB) + + * `start_bytes` is MemAvailable read by a NAMESPACE-SCOPE initialiser + (a function-local static behind it), which runs before main and before + any model load: the host's own availability, not what our own load left + behind. `plan_bytes` is the reading at plan time. The MAXIMUM is the + honest one, and the process-start reading is what the checkpoint load + would otherwise have hidden. + * The KNEE is the FLOOR for every host that can afford it: a budget under + the knee is worth little for the memory it takes, so the policy never + serves one to a host that can afford the knee. + * The RESERVE is 8 GiB: the room the host must keep free BESIDE the knee + (the OS's working set and the decode's own transient allocations). A + host that reports less than the knee plus the reserve cannot afford the + knee at all, so the policy DISABLES the cache (budget 0) and the forward + runs the UNCACHED path, byte for byte. This is the explicit rule: a + sub-knee cache is WORSE than useless — it costs the memory and buys + nothing — so the cache either runs at or above the knee or it does not + run. The reserve is a stated policy number, not a measured constant, + and the gate pins it. + * The 32 GiB CAP bounds the footprint (the measured budget-sensitivity + table had its knee still open at 32 GiB, +5-6% over 16 GiB). An UNKNOWN + report (MemAvailable unreadable) uses the KNEE: an unknown budget is + not a reason to refuse (the house rule) and not a reason to hand out a + sub-knee budget; a host that cannot hold the knee degrades to the + uncached path through the allocation fallback. + * The env contract is unchanged: 0 disables, N sets exactly N MiB, and an + unparseable value falls back to this default policy, never to disabled. + +The pure policy stays a function of an input reading +(`DefaultBudgetBytesFor(available_bytes)` for the honest reading, +`HonestAvailableBytes(start_bytes, plan_bytes)` for the maximum of the two +moments), so the gate pins it without reading the host and a policy change is +a deterministic red. The profile report carries all three numbers +(`budget_mib`, `budget_source_mib`, `budget_start_mib`, `budget_plan_mib`) so +a reader can check the derivation after the fact. + +MEASURED AFTER THE REPAIR (2026-10-21, node 3, aarch64, 3.4 GHz, 32 +threads, empty-prompt 32-token decode, `VT_KOLIBRI1_PROFILE=1`, the file +mutex held, the box quiet): the DEFAULT leg planned +`budget_mib=27157 budget_source_mib=217262 budget_start_mib=217262 +budget_plan_mib=143920 enabled=1` and finished at hits=72190, misses=17756, +evictions=7993, an 80% hit rate, with 7.44833 tok/s; the second and third +runs planned 27096 MiB and 27199 MiB and delivered 7.39141 and 7.46827 +tok/s. The explicit 16384 MiB leg, in the same window and the same binary, +delivered 7.05777, 7.05438 and 6.98659 tok/s, and the cache-off leg +delivered 3.62428, 3.78799 and 3.54865 tok/s. The plan-time reading +(143920 MiB) sat 72 GiB under the process-start reading (217262 MiB): that +gap IS the checkpoint load, and the honest reading closes it. + ## R7 resolution — the tokenizer engine accepts the Kolibri-1 split regex ### Scope — what is actually in the file @@ -554,6 +661,11 @@ ONE pull request (developer's standing choice for this row). the pin `049a6a7bd2405b27d6d280d256bd3d585191c7ae`, `gateable = no`, and the issue that owns the GPU gateability measurement. - GPU gateability measurement for the aleph-alpha-inference oracle. +- The decode and prefill benchmark rows for the DEFAULT dequant-cache + configuration (the derived budget), pending the node-3 window; the exact + commands are in the pull request body. The harness rows that mean + "cache off" must now pass `VT_KOLIBRI1_DEQUANT_CACHE_MB=0` explicitly: + an UNSET variable is the default-ON configuration, not the off switch. - GGUF arms (k-quants), CUDA arm, Tenstorrent arm — later rows. - Reasoning parser / tool parser serving integration (plugin registers `kolibri1` reasoning + Hermes tool parsers). diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index c558b50d1..d5c8db6df 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -202,6 +202,69 @@ collapses to ~2%, because that path is allocation-bound. |---|---|---| | `VT_KOLIBRI1_NUMA_INTERLEAVE` | **on** | `=0` (exactly) disables the guard entirely: the arenas allocate under the process's default policy and nothing touches the mempolicy. Unset or any other value leaves it on. Read on every guard construction (uncached), so a test binary can flip it mid-process; the call rate is per load and per cache miss, never per element. This is the A/B off switch and the first thing to set to `0` when a NUMA-policy defect is suspected on a multi-node host | +## Kolibri-1 CPU dequant cache + +The Kolibri-1 CPU forward keeps the checkpoint's FP8 weights compressed and +dequantises every used projection on every forward, so a decode step pays for +~1.4 G elements of FP8-to-bf16 work. The dequant is a pure function of the +weight bytes, so the decoded weights are kept resident in a bounded LRU cache +(`src/vllm/model_executor/models/kolibri1_dequant_cache.h`). The cache is +worth more than 2x on CPU decode: 4.02 tok/s with it off against 9.07 tok/s +with a 16 GiB budget (2026-10-20, node 3, aarch64, 3.4 GHz, 32 threads, +empty-prompt 32-token decode; prefill p512 43.33 tok/s with it off), so the +DEFAULT is the cached path. + +The default budget is DERIVED from the host's own memory report, so a small +machine is not committed to a large allocation. `MemAvailable` is the +`/proc/meminfo` figure the kernel reports as handable without swapping, read +through the same helper the DeepSeek-V4 EXL3 host-residency budget uses. Two +measured facts shape the policy: + +- THE KNEE is 16 GiB. Below it the cache thrashes and buys little: at 8.25 + GiB the measured hit rate was 33% with 16068 evictions, and the decode + throughput (3.84 tok/s) was indistinguishable from the cache being off + (3.62 tok/s), against 6.38 tok/s at an explicit 16 GiB (2026-10-21, node 3, + aarch64, 3.4 GHz, 32 threads, empty-prompt 32-token decode). +- THE SAMPLING MOMENT matters. The budget was read on the first use of the + cache, and the first use happens during the 74 GB checkpoint load, where + the page cache is full and `MemAvailable` reads 66 GiB instead of the + host's ~212 GiB idle reading. + +So the policy reads the host BEFORE our own load depresses the reading, and +floors the budget at the measured knee: + + honest_bytes = max(start_bytes, plan_bytes) + honest_bytes == 0 -> budget = 16 GiB (UNKNOWN) + honest_bytes < 16 GiB + 8 GiB -> budget = 0 (DISABLED) + otherwise -> budget = clamp(honest_bytes / 8, + 16 GiB, 32 GiB) + +`start_bytes` is the reading taken at process start (before `main`, before +any model load) and `plan_bytes` the reading at plan time; the maximum is +the honest one. The KNEE (16 GiB) is the FLOOR: a host that can afford the +knee is never served a budget below it, because a sub-knee cache costs the +memory and buys nothing. The RESERVE (8 GiB) is the room the host must keep +free beside the knee: a host that reports less than the knee plus the +reserve cannot afford the knee at all, so the policy DISABLES the cache +(budget 0) and the forward runs the uncached path, byte for byte. That is +the explicit rule: a sub-knee cache is WORSE than useless, so the cache +either runs at or above the knee or it does not run. The 32 GiB cap bounds +the footprint, and an unreadable report uses the KNEE: an unknown budget is +not a reason to refuse, and not a reason to hand out a sub-knee budget (a +host that cannot hold the knee degrades to the uncached path through the +allocation fallback). + +The budget is a BOUND, not a reservation: entries are allocated one at a time +and evicted under it. An allocation that cannot be satisfied degrades the +process to the uncached path (the original threaded dequant) instead of +failing the forward, and the degradation is counted in the +`VT_KOLIBRI1_PROFILE` report, which also prints the budget, the honest +reading, and both readings it came from. + +| Variable | Default | What it does | +|---|---|---| +| `VT_KOLIBRI1_DEQUANT_CACHE_MB` | **on** (budget derived: `clamp(max(start, plan) / 8, 16 GiB, 32 GiB)`; a host that cannot afford the 16 GiB knee gets the cache DISABLED, and an unreadable report uses the knee) | The Kolibri-1 CPU dequant cache's residency budget, in MiB. `0` DISABLES the cache and restores the original threaded pool dequant, byte for byte; that is the escape hatch and the A/B off switch. Any other value sets the budget EXACTLY (16 GiB is the shape the 2026-10-20 measurement above was taken in, and the shape to pin when a reproducible residency matters). The variable UNSET is the default: ON, with the derived budget above. A value that is not a plain non-negative number of MiB is not a budget, and it falls back to the default policy rather than switching a default-ON lever off. The cache stores the DECODED weights, so the bytes it serves are the bytes the uncached path produces: the change is a caching change, never an arithmetic one | + ## Rollback and bisect switches Default-on fast paths, each with an off switch. They exist so a suspected kernel diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 8ff382f0d..a0530d8af 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -224,7 +224,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `KevModel` | `jaredpalmer/kev-0.8b` (frozen Qwen3.5-0.8B-Base + rank-16 LoRA merged at convert time + PointerHead readout) | PointerHead golden-vector tests (25 cases, 126 assertions) + LoRA-merge + ForwardHidden goldens with perturbation gates vs the kev Python reference `jaredpalmer/kev` @`19dcae9b6e3e1a48200c5825aad9fc200d31e20a` (#3295); `/v1/systemone` dispatch reuses the Laya lane. E2E choice/score/noul through LocalAI `/v1/systemone` PASS. CUDA build verification + GGUF k-quant arm OWED per [spec](../.agents/specs/kev.md) | not measured | | `DeepseekV41ForCausalLM` | none loadable yet | **REGISTERED AND VALIDATING; NOT LOADABLE** (W1, 2026-09-13) per [model-matrix](../.agents/model-matrix.md) and [spec](../.agents/specs/deepseek-v4-1-flash.md): the architecture resolves through the registry while the load plan is validated; no token claim | not measured | | `MiMoV2ForCausalLM` | `vcruz305/MiMoV2-Flash-RL-EXL3` (EXL3-quantized, not yet loadable end-to-end) | **W1+W2 scaffold + W3 forward pass; NOT GATED.** Registry + config + hybrid KV-cache spec (full-attn 4 KV heads / SWA 8 KV heads, `v_head_dim=128 != head_dim=192`) + bf16/EXL3 dual-path weight loader gated; the device forward computes through all 48 layers (hybrid attention with sink bias, partial RoPE, attention_value_scale; layer 0 dense MLP, layers 1-47 MoE with sigmoid + noaux_tc routing) but no token has been emitted yet — the end-to-end gate (W5) is OWED | not measured | -| `Kolibri1ForCausalLM` | `Aleph-Alpha/Kolibri-1`, mixed FP8/BF16 safetensors (78.8 GB) | CPU and Tenstorrent forwards. FP8 weights stay compressed at load. CPU decode numerics default to the bf16-dot tier (bf16 in / f32 accumulate, the BFMMLA class) on aarch64 FEAT_BF16 hosts; `VT_CPU_KOLIBRI_BF16_DOT=0` is the f32-tier same-binary rollback. Kolibri chat template, reasoning, and tool-call parsers. CPU comparisons include accepted near-ties, not full token parity. GGUF and other device forwards refused. [Recipe and validation limits](USAGE.md#kolibri-1) | Full author-plugin comparison pending. [TT measurements and gaps](gap-kolibri1-tt-captured-decode.md) | +| `Kolibri1ForCausalLM` | `Aleph-Alpha/Kolibri-1`, mixed FP8/BF16 safetensors (78.8 GB) | CPU and Tenstorrent forwards. FP8 weights stay compressed at load. CPU decode numerics default to the bf16-dot tier (bf16 in / f32 accumulate, the BFMMLA class) on aarch64 FEAT_BF16 hosts; `VT_CPU_KOLIBRI_BF16_DOT=0` is the f32-tier same-binary rollback. The CPU fp8 decode keeps its dequantised weights resident in a bounded LRU cache that is ON by default, with a budget derived from the host's available memory read at PROCESS START and floored at the measured 16 GiB knee (`clamp(max(start, plan) / 8, 16 GiB, 32 GiB)`; a host that cannot afford the knee gets the cache DISABLED and the uncached path) `VT_KOLIBRI1_DEQUANT_CACHE_MB=0` restores the per-forward dequant, and an explicit value sets the budget exactly. The cached bytes are byte-identical to the uncached ones (more than 2x decode: 4.02 against 9.07 tok/s at 32 threads, 2026-10-20). Kolibri chat template, reasoning, and tool-call parsers. CPU comparisons include accepted near-ties, not full token parity. GGUF and other device forwards refused. [Recipe and validation limits](USAGE.md#kolibri-1) | Full author-plugin comparison pending. [TT measurements and gaps](gap-kolibri1-tt-captured-decode.md) | ### Standalone and non-registered lanes diff --git a/docs/USAGE.md b/docs/USAGE.md index 22c421865..ea2d1b499 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -944,8 +944,13 @@ The explicit context limit keeps this example separate from the author's unverified million-token serving recipe. It does not limit weight memory. The loader retains FP8 blocks and BF16 tensors. The CPU forward dequantizes FP8 -projections when needed. Its optional dequantization cache defaults to disabled -(`VT_KOLIBRI1_DEQUANT_CACHE_MB=0`), so enabling it adds a memory budget. +projections when needed, and keeps the dequantized weights resident in a +bounded cache that is on by default: the budget is derived from the host's +available memory (`clamp(MemAvailable / 8, 1 GiB, 32 GiB)`), so it needs no +configuration and never commits the process to a large allocation. +`VT_KOLIBRI1_DEQUANT_CACHE_MB=0` disables the cache and restores the +per-forward dequantization, and an explicit value sets the budget in MiB +exactly. The registry also dispatches to a Tenstorrent forward with streamed experts. See the [Tenstorrent measurements and remaining gaps](gap-kolibri1-tt-captured-decode.md). GGUF loading and other device forwards, including CUDA, are refused. diff --git a/src/vllm/model_executor/models/kolibri1_dequant_cache.h b/src/vllm/model_executor/models/kolibri1_dequant_cache.h index eec6c5965..eea343a16 100644 --- a/src/vllm/model_executor/models/kolibri1_dequant_cache.h +++ b/src/vllm/model_executor/models/kolibri1_dequant_cache.h @@ -74,18 +74,91 @@ // protection past the unlock, which the mutex alone never did. // // BUDGET. `VT_KOLIBRI1_DEQUANT_CACHE_MB` sets the total budget in MiB across -// all entries. Default 0 == the cache is DISABLED: `GetOrDequant` still decodes -// (the same code path, so behavior and bit patterns are identical), but stores -// nothing, so main-line behavior is unchanged unless the var is set. An entry -// larger than the whole budget is never cached (it would evict everything else -// on every insert). Because a returned Lease must own its bytes outright, the -// disabled/oversized path decodes into a FRESH per-call allocation (the old -// shared member scratch could be clobbered by the next call, which the lease -// contract forbids); `LinearBT` never calls `GetOrDequant` when the cache is -// disabled, so main-line behavior and its allocation profile are unchanged. +// all entries. `0` DISABLES the cache: `GetOrDequant` still decodes (the same +// code path, so behavior and bit patterns are identical), but stores nothing, +// and `LinearBT` keeps the ORIGINAL threaded pool decode byte-for-byte. An +// explicit value sets the budget EXACTLY (that is how the 2026-10-20 +// measurement below was taken: 16 GiB). The DEFAULT — the variable UNSET — +// is ON, with the budget DERIVED from the host's own memory report (THE +// DEFAULT BUDGET below). An entry larger than the whole budget is never +// cached (it would evict everything else on every insert). Because a returned +// Lease must own its bytes outright, the disabled/oversized path decodes into +// a FRESH per-call allocation (the old shared member scratch could be +// clobbered by the next call, which the lease contract forbids). // -// COUNTERS. hits / misses / evictions / decode_calls are exported through the -// VT_KOLIBRI1_PROFILE report (kolibri1_forward.cpp). +// COUNTERS. hits / misses / evictions / decode_calls / alloc_failures are +// exported through the VT_KOLIBRI1_PROFILE report (kolibri1_forward.cpp). +// +// THE DEFAULT BUDGET (2026-10-20, corrected 2026-10-21). The cache is worth +// MORE THAN 2x on CPU decode: 4.02 tok/s with the cache OFF against 9.07 +// tok/s with a 16 GiB budget (2026-10-20, node 3, aarch64, 3.4 GHz, 32 +// threads, empty-prompt 32-token decode; prefill p512 43.33 tok/s with the +// cache off). Shipping it OFF by default shipped the SLOW configuration, so +// the default is ON and the budget is DERIVED from the host's own memory +// report. Two MEASURED facts shape the policy. +// +// THE KNEE. The budget is worth its memory only at or above 16 GiB. At 8.25 +// GiB the measured hit rate was 33% with 16068 evictions, and the decode +// throughput (3.84 tok/s) was indistinguishable from the cache being off +// (3.62 tok/s). A budget below the knee buys nothing for the memory it +// takes, so the knee is the FLOOR and a host that cannot afford it gets the +// cache DISABLED (the uncached path) rather than a thrashing sub-knee cache. +// +// THE SAMPLING MOMENT. The budget was read on FIRST USE of the cache, and +// the first use happens DURING the 74 GB checkpoint load: the page cache is +// full by then, so MemAvailable reads 66 GiB where the host's idle reading +// is ~212 GiB. The divisor then yields 8.25 GiB, below the knee, and the +// default ships the slow configuration by accident. +// +// So the policy reads the host BEFORE our own load depresses the reading, +// and floors the budget at the knee: +// +// honest_bytes = max(start_bytes, plan_bytes) +// honest_bytes == 0 -> budget = 16 GiB (UNKNOWN) +// honest_bytes < 16 GiB + 8 GiB -> budget = 0 (DISABLED) +// otherwise -> budget = clamp(honest_bytes / 8, +// 16 GiB, 32 GiB) +// +// * `start_bytes` is MemAvailable read by a NAMESPACE-SCOPE initialiser, +// which runs before main and before any model load. `plan_bytes` is the +// reading at plan time. The MAXIMUM is the honest one: a reading lower +// than an earlier one says that something took memory in between, and +// the policy does not let its own checkpoint load talk the budget down. +// * Both readings are `/proc/meminfo` MemAvailable, read through the house +// helper `vllm::v1::host_available_memory_bytes()` +// (include/vllm/v1/core/kv_cache_utils.h) — the SAME probe the +// DeepSeek-V4 EXL3 host-residency budget uses, so the tree keeps ONE +// memory reader. +// * The DIVISOR (8) leaves seven eighths of the available memory for the +// checkpoint itself (74 GB of fp8 weights), the KV cache and the OS. +// * The KNEE (16 GiB) is the FLOOR: a host that can afford the knee is +// never served a budget below it. +// * The RESERVE (8 GiB) is the room the host must keep free beside the +// knee. A host that reports less than the knee plus the reserve cannot +// afford the knee at all, so the policy DISABLES the cache (budget 0) +// and the forward runs the UNCACHED path, byte for byte. That is the +// explicit rule: a sub-knee cache is WORSE than useless, so the cache +// either runs at or above the knee or it does not run. +// * The CAP (32 GiB) bounds the footprint: the measured budget-sensitivity +// table had its knee still open at 32 GiB (+5-6% over 16 GiB), so the +// cap captures nearly the whole win while keeping the cache a small +// fraction of the model it serves. +// * An UNKNOWN report (MemAvailable unreadable -> 0) uses the KNEE: an +// unknown budget is not a reason to refuse, and not a reason to hand +// out a sub-knee budget. A host that cannot hold the knee degrades to +// the uncached path through the allocation fallback below. +// +// The budget is a BOUND, not a reservation: entries are allocated one at a +// time and evicted under the bound, so the process is never committed to the +// whole budget up front. +// +// THE FALLBACK. A cache that cannot get its bytes must NOT take the forward +// down with it. An allocation failure degrades the process to the UNCACHED +// path (the original threaded pool decode), which is slower and needs no +// residency: the cache drops its entries (handing the memory back), reports +// itself disabled, and every later forward takes the uncached path without +// trying again. The decoded BYTES are the same either way — the fallback +// changes the ROUTE, never the arithmetic. #ifndef VLLM_MODEL_EXECUTOR_MODELS_KOLIBRI1_DEQUANT_CACHE_H_ #define VLLM_MODEL_EXECUTOR_MODELS_KOLIBRI1_DEQUANT_CACHE_H_ @@ -97,12 +170,15 @@ #include #include #include +#include +#include #include #include #include #include #include "vllm/model_executor/models/host_parallel.h" // the ONE pool (#1664) +#include "vllm/v1/core/kv_cache_utils.h" // host_available_memory_bytes #include "vllm/model_executor/models/kolibri1_numa.h" // the interleave policy #include "vllm/model_executor/models/kolibri1_fp8_dequant.h" // DequantRowsBf16 #include "vllm/model_executor/models/kolibri1_weights.h" // Fp8BlockWeight @@ -117,6 +193,9 @@ struct CacheCounters { uint64_t misses = 0; uint64_t evictions = 0; uint64_t decode_calls = 0; + // The DEGRADATIONS (THE FALLBACK): an allocation the cache could not + // honour, after which the process runs the uncached path. + uint64_t alloc_failures = 0; }; inline CacheCounters& Counters() { @@ -181,8 +260,25 @@ struct KeyHash { } }; +// The PROCESS-START reading (THE SAMPLING MOMENT). A function-local static, +// so the reading is taken EXACTLY once, and the namespace-scope initialiser +// below forces it to be taken at process start instead of on first use. +inline uint64_t& StartAvailableBytesStorage() { + static uint64_t value = [] { + const int64_t available = vllm::v1::host_available_memory_bytes(); + return available > 0 ? static_cast(available) : 0; + }(); + return value; +} + } // namespace detail +// THE PROCESS-START READING. The namespace-scope dynamic initialiser runs +// before main and before any model load, so the reading is the host's own +// availability and not what our own 74 GB checkpoint load left behind. The +// plan uses the MAXIMUM of this reading and the reading at plan time. +inline const uint64_t kStartAvailableBytes = detail::StartAvailableBytesStorage(); + // The ownership lease returned by `GetOrDequant` (maint-bot P1b on PR #3414). // A Lease shares ownership of the decoded bytes: they stay alive and unchanged // until the LAST lease (and the cache's own reference, if the entry is still @@ -208,7 +304,19 @@ class Lease { // returned leases name memory this object co-owns. class Cache { public: - explicit Cache(size_t budget_bytes) : budget_(budget_bytes) {} + // `budget_source_bytes` is the HONEST available-memory reading the budget + // was derived from (0 when the budget was set explicitly, or when the host + // does not report one). `budget_start_bytes` is the PROCESS-START reading + // and `budget_plan_bytes` the reading at plan time (THE SAMPLING MOMENT). + // They are kept so a reader can check the DERIVATION without re-reading a + // number that has moved since. + explicit Cache(size_t budget_bytes, uint64_t budget_source_bytes = 0, + uint64_t budget_start_bytes = 0, + uint64_t budget_plan_bytes = 0) + : budget_(budget_bytes), + budget_source_bytes_(budget_source_bytes), + budget_start_bytes_(budget_start_bytes), + budget_plan_bytes_(budget_plan_bytes) {} // Returns the decoded bf16 [n, k] block for `w` as a Lease: cache-owned // bytes on a hit, freshly decoded lease-owned bytes on a miss (the entry is @@ -241,8 +349,23 @@ class Cache { // pages interleaved over the NUMA nodes (kolibri1_numa.h) so a multi-node // decode reads them local. Allocation policy only — bytes identical. [[maybe_unused]] const kolibri1_numa::PolicyGuard numa_guard; - auto bytes = std::make_shared>(); - DecodeInto(w, *bytes); + std::shared_ptr> bytes; + try { + bytes = std::make_shared>(); + DecodeInto(w, *bytes); + } catch (const std::bad_alloc&) { + // THE FALLBACK: the allocator would not give the cache its bytes. The + // forward continues on the UNCACHED path (the caller sees an empty + // lease and takes the original threaded pool decode), and the cache + // degrades itself so the next forward does not try again. + Degrade(); + return Lease(); + } catch (const std::length_error&) { + // A weight whose decoded size overflows the allocator's limit: the same + // route as a failed allocation, and the same degradation. + Degrade(); + return Lease(); + } if (budget_ == 0 || byte_count > budget_) { // Disabled or oversized: decode into the lease, store nothing. Same // decode body, so the bytes are bit-identical to a stored hit. @@ -270,11 +393,17 @@ class Cache { } size_t budget_bytes() const { return budget_; } + uint64_t budget_source_bytes() const { return budget_source_bytes_; } + uint64_t budget_start_bytes() const { return budget_start_bytes_; } + uint64_t budget_plan_bytes() const { return budget_plan_bytes_; } size_t resident_bytes() const { return resident_; } - // False when the budget is 0: `LinearBT` then keeps the ORIGINAL threaded - // pool decode (main-line behavior byte-for-byte, host threads included) - // instead of routing through this cache's serial decode. - bool enabled() const { return budget_ != 0; } + // False when the budget is 0, and after a DEGRADATION (THE FALLBACK): + // `LinearBT` then keeps the ORIGINAL threaded pool decode (main-line + // behavior byte-for-byte, host threads included) instead of routing + // through this cache's serial decode. + bool enabled() const { + return budget_ != 0 && !degraded_.load(std::memory_order_relaxed); + } size_t entries() const { return map_.size(); } private: @@ -309,24 +438,166 @@ class Cache { if (it != lru_.begin()) lru_.splice(lru_.begin(), lru_, it); } + // THE FALLBACK: drop every entry, hand the memory back, and report the + // cache disabled. The entries' bytes are freed here unless a caller still + // holds a lease (shared ownership keeps them alive until it does). Called + // with the cache mutex held. + void Degrade() { + map_.clear(); + lru_.clear(); + resident_ = 0; + degraded_.store(true, std::memory_order_relaxed); + Counters().alloc_failures++; + } + const size_t budget_; + const uint64_t budget_source_bytes_; + const uint64_t budget_start_bytes_; + const uint64_t budget_plan_bytes_; + std::atomic degraded_{false}; size_t resident_ = 0; Lru lru_; // front == most recently used Map map_; std::mutex mutex_; }; -// The process-wide cache `LinearBT` consults, configured ONCE from -// `VT_KOLIBRI1_DEQUANT_CACHE_MB` (default 0 == disabled). +// THE DEFAULT BUDGET POLICY (THE DEFAULT BUDGET above). The constants are +// exposed so the gate can pin the policy — knee, cap, divisor, reserve — +// without restating the numbers, and so the documentation and the code +// cannot drift. +inline constexpr uint64_t kDefaultBudgetDivisor = 8; +inline constexpr uint64_t kDefaultBudgetMaxBytes = 32ull << 30; // 32 GiB +// The MEASURED KNEE (2026-07-20, re-measured 2026-10-21): the budget below +// which the cache thrashes and the decode throughput is indistinguishable +// from the cache being off. At 8.25 GiB the measured hit rate was 33% with +// 16068 evictions and 3.84 tok/s, against 3.62 tok/s with the cache off and +// 6.38 tok/s at an explicit 16 GiB (node 3, aarch64, 3.4 GHz, 32 threads, +// empty-prompt 32-token decode). The knee is the FLOOR: a budget under it +// costs the memory and buys nothing. +inline constexpr uint64_t kDefaultBudgetKneeBytes = 16ull << 30; // 16 GiB +// The room the host must keep free BESIDE the knee: the OS's working set +// and the decode's own transient allocations. A host that reports less than +// the knee plus this reserve cannot afford the knee at all. +inline constexpr uint64_t kDefaultBudgetHostReserveBytes = 8ull << 30; // 8 GiB + +// The HONEST available-memory reading: the MAXIMUM of the PROCESS-START +// reading and the reading at plan time (THE SAMPLING MOMENT). A reading +// lower than an earlier one says that something took memory in between; the +// policy does not let its own checkpoint load, or anyone else's, talk the +// budget down. +inline uint64_t HonestAvailableBytes(uint64_t start_bytes, uint64_t plan_bytes) { + return start_bytes > plan_bytes ? start_bytes : plan_bytes; +} + +// The PURE policy: the budget for a given HONEST available-memory reading. +// Pure on purpose: the gate pins it without reading the host, so a policy +// change is a deterministic red and not a race against the machine's memory. +// +// available == 0 (UNKNOWN) -> the KNEE. Not a refusal (the house rule: +// an unknown budget must not become a false +// refusal) and never a sub-knee budget: the +// allocation fallback degrades to the +// uncached path if the host cannot hold it. +// available < KNEE + RESERVE -> 0: DISABLED. The host cannot afford the +// knee, and a sub-knee cache costs the +// memory and buys nothing, so the clean +// answer is the UNCACHED path (budget 0), +// byte for byte. +// otherwise -> clamp(available / 8, KNEE, CAP) +inline uint64_t DefaultBudgetBytesFor(uint64_t available_bytes) { + if (available_bytes == 0) { + // UNKNOWN: the host does not report its memory. Not a refusal, and never + // a sub-knee budget: the KNEE itself. + return kDefaultBudgetKneeBytes; + } + if (available_bytes < + kDefaultBudgetKneeBytes + kDefaultBudgetHostReserveBytes) { + // The host cannot afford the knee: DISABLE cleanly. A budget of 0 is the + // UNCACHED path (the original threaded pool dequant), byte for byte. + return 0; + } + uint64_t derived = available_bytes / kDefaultBudgetDivisor; + // The KNEE is the FLOOR: a host that can afford the knee must not be + // served a budget below it, because a sub-knee budget buys nothing for the + // memory it takes. + if (derived < kDefaultBudgetKneeBytes) derived = kDefaultBudgetKneeBytes; + if (derived > kDefaultBudgetMaxBytes) derived = kDefaultBudgetMaxBytes; + return derived; +} + +// How the process-wide cache's budget was decided. +struct BudgetPlan { + size_t bytes = 0; // the budget itself + uint64_t source_bytes = 0; // the HONEST available-memory reading it came + // from (0 when the operator set the budget + // exactly, or when the host reports none) + uint64_t start_bytes = 0; // the PROCESS-START reading (0 when the + // operator set the budget exactly) + uint64_t plan_bytes = 0; // the reading at plan time (0 when the + // operator set the budget exactly) +}; + +// The env knob, parsed. A plain non-negative decimal number of MiB parses; +// anything else (an empty value, a word, a negative number, a value with a +// suffix) does NOT, and an unparseable value falls back to the DEFAULT +// policy rather than to "disabled": the default turns this lever ON, so a +// typo must not silently switch it off. +inline bool ParseBudgetMiB(const char* value, uint64_t* out_mib) { + if (value == nullptr || out_mib == nullptr) return false; + const auto space = [](char c) { + return c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\v' || + c == '\f'; + }; + const char* p = value; + while (*p != '\0' && space(*p)) ++p; + const char* end = p; + while (*end != '\0') ++end; + while (end > p && space(end[-1])) --end; + if (end == p) return false; + uint64_t v = 0; + for (const char* q = p; q != end; ++q) { + if (*q < '0' || *q > '9') return false; + v = v * 10 + static_cast(*q - '0'); + // 2^32 MiB is 4 PiB: past any real budget, and small enough that the + // MiB -> byte conversion below cannot overflow a 64-bit size_t. + if (v > (1ull << 32)) return false; + } + *out_mib = v; + return true; +} + +// The process-wide cache `LinearBT` consults, configured ONCE: +// * `VT_KOLIBRI1_DEQUANT_CACHE_MB` UNSET -> the DEFAULT budget (ON). +// * `VT_KOLIBRI1_DEQUANT_CACHE_MB=0` -> DISABLED. +// * `VT_KOLIBRI1_DEQUANT_CACHE_MB=N` -> exactly N MiB. +// * an unparseable value -> the DEFAULT budget. inline Cache& ProcessCache() { - static const size_t budget = [] { - size_t mb = 0; - if (const char* v = std::getenv("VT_KOLIBRI1_DEQUANT_CACHE_MB")) { - mb = static_cast(strtoull(v, nullptr, 10)); + static const BudgetPlan plan = [] { + const char* v = std::getenv("VT_KOLIBRI1_DEQUANT_CACHE_MB"); + uint64_t mib = 0; + if (v == nullptr || !ParseBudgetMiB(v, &mib)) { + // The DEFAULT: ON, with a budget derived from the host's own memory + // report (THE DEFAULT BUDGET). The reading is the HONEST one: the + // MAXIMUM of the PROCESS-START reading (taken before our own + // checkpoint load depressed the host's report) and the reading at plan + // time. + const int64_t available = vllm::v1::host_available_memory_bytes(); + BudgetPlan p; + p.start_bytes = kStartAvailableBytes; + p.plan_bytes = available > 0 ? static_cast(available) : 0; + p.source_bytes = HonestAvailableBytes(p.start_bytes, p.plan_bytes); + p.bytes = static_cast(DefaultBudgetBytesFor(p.source_bytes)); + return p; } - return mb * 1024 * 1024; + const uint64_t bytes = mib * 1024ull * 1024ull; + BudgetPlan p; + p.bytes = bytes > static_cast(SIZE_MAX) + ? SIZE_MAX + : static_cast(bytes); + return p; }(); - static Cache cache(budget); + static Cache cache(plan.bytes, plan.source_bytes, plan.start_bytes, + plan.plan_bytes); return cache; } diff --git a/src/vllm/model_executor/models/kolibri1_forward.cpp b/src/vllm/model_executor/models/kolibri1_forward.cpp index e10fd71f0..555cda359 100644 --- a/src/vllm/model_executor/models/kolibri1_forward.cpp +++ b/src/vllm/model_executor/models/kolibri1_forward.cpp @@ -110,13 +110,24 @@ void Report(int64_t calls) { for (const auto& [k, v] : Acc()) std::fprintf(stderr, "[kolibri1-profile] %-20s %10.2f s\n", k.c_str(), v); const auto& c = kolibri1_dequant_cache::Counters(); + const auto& cache = kolibri1_dequant_cache::ProcessCache(); std::fprintf(stderr, "[kolibri1-profile] dequant_cache hits=%llu misses=%llu " - "evictions=%llu decode_calls=%llu\n", + "evictions=%llu decode_calls=%llu alloc_failures=%llu " + "budget_mib=%llu budget_source_mib=%llu budget_start_mib=%llu " + "budget_plan_mib=%llu enabled=%d\n", static_cast(c.hits), static_cast(c.misses), static_cast(c.evictions), - static_cast(c.decode_calls)); + static_cast(c.decode_calls), + static_cast(c.alloc_failures), + static_cast(cache.budget_bytes() >> 20), + static_cast(cache.budget_source_bytes() >> + 20), + static_cast(cache.budget_start_bytes() >> + 20), + static_cast(cache.budget_plan_bytes() >> 20), + cache.enabled() ? 1 : 0); } } // namespace prof @@ -268,26 +279,38 @@ struct ResolvedBT { kolibri1_dequant_cache::Lease dequant_lease; }; -ResolvedBT ResolveBT(Dev d, const Kolibri1Projection& w) { +ResolvedBT ResolveBTWithCache(Dev d, const Kolibri1Projection& w, + kolibri1_dequant_cache::Cache& cache) { ResolvedBT r; if (w.IsBf16()) { r.wt = ResidentWeight(d, w.bf16); return r; } - if (kolibri1_dequant_cache::ProcessCache().enabled()) { + if (cache.enabled()) { prof::Scope prof("dequant_fp8_block"); // cache lookups + cold decodes - r.dequant_lease = - kolibri1_dequant_cache::ProcessCache().GetOrDequant(w.fp8_block); + r.dequant_lease = cache.GetOrDequant(w.fp8_block); + } + if (r.dequant_lease) { r.wt = dense_attn::MakeTensor(const_cast(r.dequant_lease.data()), DType::kBF16, d.q.device, {w.fp8_block.n, w.fp8_block.k}); - } else { - r.wt_buf = DequantFp8Block(d, w.fp8_block); - r.wt = r.wt_buf.t(); + return r; } + // THE FALLBACK: the cache is disabled, or it could not hold its bytes (an + // allocation failure: it degraded itself and handed back an EMPTY lease). + // The UNCACHED path below runs instead: the same decode, the same bytes, + // without the residency. A cache that cannot get memory must not take the + // forward down with it. + r.wt_buf = DequantFp8Block(d, w.fp8_block); + r.wt = r.wt_buf.t(); return r; } +// The production resolve: the process-wide dequant cache. +ResolvedBT ResolveBT(Dev d, const Kolibri1Projection& w) { + return ResolveBTWithCache(d, w, kolibri1_dequant_cache::ProcessCache()); +} + DBuf LinearBT(Dev d, const Tensor& x, const Kolibri1Projection& w, int64_t t, DType out_dtype = DType::kBF16, const char* gsite = "other") { if (w.IsBf16()) return LinearBTRaw(d, x, w.bf16, t, out_dtype, gsite); @@ -300,9 +323,9 @@ DBuf LinearBT(Dev d, const Tensor& x, const Kolibri1Projection& w, // returned (a concurrent caller could evict and reuse the bytes mid-GEMM, // maint-bot P1b on PR #3414). The entries OWN their bytes (independent // allocations, never pool blocks — that ownership defect was v1's - // corruption). Budget 0 (the default) DISABLES the cache and keeps the - // original threaded pool decode, so main-line behavior is byte-for-byte - // unchanged unless the env var is set. + // corruption). The cache is ON by DEFAULT (2026-10-20) with a budget + // derived from the host's available memory; `VT_KOLIBRI1_DEQUANT_CACHE_MB=0` + // disables it and keeps the original threaded pool decode, byte-for-byte. prof::Scope prof("linear_gemm"); ResolvedBT r = ResolveBT(d, w); // BFDOT bf16 GEMV arm, DEFAULT on a FEAT_BF16 host (Bf16DotActive: @@ -907,6 +930,22 @@ void RouteDump(int64_t layer, const std::vector& logits, } // namespace +DBuf DequantFp8BlockUncached(Dev d, const Fp8BlockWeight& w) { + return DequantFp8Block(d, w); +} + +ResolvedFp8Projection ResolveFp8Projection(Dev d, const Fp8BlockWeight& w, + kolibri1_dequant_cache::Cache& cache) { + Kolibri1Projection proj; + proj.fp8_block = w; + ResolvedBT r = ResolveBTWithCache(d, proj, cache); + ResolvedFp8Projection out; + out.cached = static_cast(r.dequant_lease); + out.wt = r.wt; + out.buf = std::move(r.wt_buf); + return out; +} + ForwardLogits ForwardKolibri1Forward( const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, diff --git a/src/vllm/model_executor/models/kolibri1_forward.h b/src/vllm/model_executor/models/kolibri1_forward.h index 47810c3c2..c36fd985d 100644 --- a/src/vllm/model_executor/models/kolibri1_forward.h +++ b/src/vllm/model_executor/models/kolibri1_forward.h @@ -8,6 +8,8 @@ #include #include +#include "vllm/model_executor/models/dense_device_glue.h" // Dev, DBuf +#include "vllm/model_executor/models/kolibri1_dequant_cache.h" // Cache #include "vllm/model_executor/models/kolibri1_weights.h" #include "vllm/model_executor/models/model_registry.h" #include "vllm/model_executor/models/qwen3_5.h" // PagedKvCache, ForwardLogits @@ -22,6 +24,30 @@ namespace vllm { // `multi_kv` resolves each layer's PagedKvCache by layer name; null falls back // to `attn_kv[layer]` (the single-group layout). Logits gather follows // `logits_indices` when it is a strict subset of the step. +// The PRODUCTION UNCACHED fp8-block dequant: the exact bytes the CPU +// forward feeds to the GEMM when the dequant cache is disabled (the +// `DequantFp8Block` body in kolibri1_forward.cpp). Exposed for the +// byte-identity gate, which compares the CACHED bytes against the UNCACHED +// ones on the same weight: the cache is a caching change only, so the two +// must be byte-identical. +dense_attn::DBuf DequantFp8BlockUncached(dense_attn::Dev d, + const Fp8BlockWeight& w); + +// The PRODUCTION resolve of one fp8-block projection: the CACHED path (the +// dequant cache's lease) when the cache is enabled, the UNCACHED path +// (`DequantFp8Block`) otherwise and after a cache DEGRADATION (the cache +// could not hold its bytes). `cache` is a parameter so the gate can drive a +// degraded cache without touching the process-wide one; the production call +// passes `kolibri1_dequant_cache::ProcessCache()`. +struct ResolvedFp8Projection { + dense_attn::DBuf buf; // the uncached block (empty when the cache served) + dense_attn::Tensor wt; // the weight the GEMM reads + bool cached = false; // true when the bytes came from the cache +}; +ResolvedFp8Projection ResolveFp8Projection(dense_attn::Dev d, + const Fp8BlockWeight& w, + kolibri1_dequant_cache::Cache& cache); + ForwardLogits ForwardKolibri1Forward( const std::vector& token_ids, const std::vector& positions, const v1::CommonAttentionMetadata& attn_meta, diff --git a/tests/vllm/models/test_kolibri1_dequant_cache.cpp b/tests/vllm/models/test_kolibri1_dequant_cache.cpp index b56974aff..6876cc0b7 100644 --- a/tests/vllm/models/test_kolibri1_dequant_cache.cpp +++ b/tests/vllm/models/test_kolibri1_dequant_cache.cpp @@ -22,17 +22,42 @@ // on the hardened pool, a second return now throws instead). // 4. LRU VICTIM: the evicting budget evicts the LEAST recently used entry, // not the most recent one (distinguished by which key hits next). -// 5. DEFAULT OFF: `ProcessCache()` with `VT_KOLIBRI1_DEQUANT_CACHE_MB` -// unset is disabled (budget 0), pinned in a fork/exec'd helper whose -// environment provably lacks the variable. POSIX-only probe (fork/exec): -// preprocessor-gated so the file compiles under MSVC, where the probe is -// disabled-and-documented (maint-bot P2 on PR #3414). +// 5. DEFAULT ON: `ProcessCache()` with `VT_KOLIBRI1_DEQUANT_CACHE_MB` unset +// is an ENABLED cache whose budget is DERIVED from the host's available +// memory, pinned in a fork/exec'd helper whose environment provably lacks +// the variable. POSIX-only probe (fork/exec): preprocessor-gated so the +// file compiles under MSVC, where the probe is disabled-and-documented +// (maint-bot P2 on PR #3414). THIS CASE USED TO ASSERT THE OPPOSITE (the +// cache OFF by default); the default flipped on 2026-10-20 because the +// cache is worth more than 2x on CPU decode, and the case was rewritten +// deliberately rather than deleted. // 6. RELOAD IDENTITY: a model reloaded at REUSED weight-buffer addresses // (same geometry, different bytes) is served its own decode, never the // previous model's cached bytes (maint-bot P1a on PR #3414). // 7. LEASE LIFETIME: a hit held live by one caller keeps its bytes intact // while a second, barrier-synchronized caller evicts its entry and // reuses the freed memory for scratch decodes (maint-bot P1b on PR #3414). +// 8. THE KNOB: `0` disables, an explicit value sets the budget EXACTLY, and +// an unparseable value falls back to the default policy (it must not +// silently switch a default-ON lever off). Same fork/exec'd helper. +// 9. THE POLICY: the pure derivation (floor, cap, divisor, the unknown +// nominal, the MEASURED KNEE and the HOST THRESHOLD) is pinned without +// reading the host, so a policy change is a deterministic red and not a +// race against the machine's memory. The policy reads the host's HONEST +// available memory, the maximum of the process-start reading and the +// reading at plan time: our own 74 GB checkpoint load depresses +// MemAvailable (66 GiB) under the host's idle reading (~145 GiB), and a +// budget derived from the depressed reading lands BELOW the knee, where +// the cache thrashes and the decode throughput is indistinguishable from +// the cache being off. +// 12. THE SAMPLING MOMENT: the plan's source reading is that honest maximum, +// the budget is the policy applied to it, and the process-start reading +// was really taken (a zero would mean the capture never happened). +// 10. BYTE IDENTITY AGAINST THE PRODUCTION UNCACHED PATH: the cached bytes +// are byte-identical to the bytes the PRODUCTION uncached dequant +// (`DequantFp8BlockUncached`, the body the forward runs with the cache +// disabled) produces for the same weight, cold and hot. The cache is a +// caching change only, and this is where that is proved. // // KNOWN LIMITS: byte-identity cannot catch a key weakened to scale-pointer // only (mitigated by weight-owned process-lifetime storage, not by a test), @@ -47,6 +72,7 @@ #include #include #include +#include #include #include @@ -62,6 +88,7 @@ #include "vllm/model_executor/models/dense_device_glue.h" // Dev, DBuf #include "vllm/model_executor/models/kolibri1_dequant_cache.h" +#include "vllm/model_executor/models/kolibri1_forward.h" // the uncached dequant #include "vllm/model_executor/models/kolibri1_fp8_dequant.h" #include "vt/device.h" @@ -165,6 +192,64 @@ std::vector RefDecode(const Fp8BlockWeight& w) { return out; } +#if !defined(_WIN32) +// Exec the default-budget probe helper with a provably EMPTY environment and +// the given mode, and return its exit status (-1 when the child did not exit +// normally). The helper sets the knob itself for every mode except the +// default one, so the environment stays empty in every case and the default +// mode really observes an unset variable. `out` receives the helper's report +// (its budget line on stdout) when it is not null. +int RunHelper(const char* mode, std::string* out) { + int fds[2] = {-1, -1}; + if (pipe(fds) != 0) return -1; + const pid_t pid = fork(); + if (pid < 0) { + close(fds[0]); + close(fds[1]); + return -1; + } + if (pid == 0) { + close(fds[0]); + if (dup2(fds[1], STDOUT_FILENO) < 0) _exit(127); + close(fds[1]); + char* envp[] = {nullptr}; // empty environment: the var is provably unset + if (mode == nullptr) { + execle(KOLIBRI_DEQUANT_CACHE_DEFAULT_HELPER, "helper", + static_cast(nullptr), envp); + } else { + execle(KOLIBRI_DEQUANT_CACHE_DEFAULT_HELPER, "helper", + const_cast(mode), static_cast(nullptr), envp); + } + _exit(127); // exec failed — not a pass + } + close(fds[1]); + char buf[256]; + ssize_t n = 0; + while ((n = read(fds[0], buf, sizeof(buf))) > 0) { + if (out != nullptr) out->append(buf, static_cast(n)); + } + close(fds[0]); + int status = 0; + if (waitpid(pid, &status, 0) != pid) return -1; + if (!WIFEXITED(status)) return -1; + return WEXITSTATUS(status); +} + +// Reads one `key=value` field out of the helper's report. `found` says +// whether the field was there at all: a missing field is a failure of the +// report, not a zero. +uint64_t FieldValue(const std::string& text, const char* key, bool* found) { + const std::string needle = std::string(key) + "="; + const size_t at = text.find(needle); + if (at == std::string::npos) { + *found = false; + return 0; + } + *found = true; + return std::strtoull(text.c_str() + at + needle.size(), nullptr, 10); +} +#endif // !_WIN32 + uint64_t ChainStep(uint64_t h, const uint16_t* p, size_t count) { for (size_t i = 0; i < count; ++i) { h ^= static_cast(p[i]); @@ -311,9 +396,11 @@ TEST_CASE("kolibri1 dequant cache: the evicting budget evicts the LRU entry, " CHECK_EQ(counters.misses - misses0, 4); } -// ── 5. Default off: ProcessCache() with the env var unset is disabled ─────── -TEST_CASE("kolibri1 dequant cache: ProcessCache() defaults to a disabled " - "cache when VT_KOLIBRI1_DEQUANT_CACHE_MB is unset") { +// ── 5. The DEFAULT is ON: the unset env var yields an enabled cache ── +TEST_CASE("kolibri1 dequant cache: ProcessCache() with VT_KOLIBRI1_DEQUANT_CACHE_MB " + "unset takes the DEFAULT policy: a budget at or above the measured " + "knee on a host that can afford it, and a clean disable on a host " + "that cannot") { #if defined(_WIN32) // PORTABILITY (maint-bot P2 on PR #3414): this probe exec's the helper // binary with a provably empty environment, which needs fork/execle — @@ -321,31 +408,147 @@ TEST_CASE("kolibri1 dequant cache: ProcessCache() defaults to a disabled " // and documented here; the portable cache cases (1-4, 6, 7) still run. // Re-enable by porting the probe to CreateProcess with an empty // environment block. - WARN("default-off probe disabled on Windows (POSIX fork/exec only)"); + WARN("default-budget probe disabled on Windows (POSIX fork/exec only)"); #else // The budget is read ONCE per process at the static init inside // ProcessCache(). To pin the DEFAULT (and not whatever this process // happens to carry, or an earlier test case has already latched), the // assertion runs in a fork/exec'd helper binary launched with an // environment that provably lacks the variable; the helper exits 0 iff - // budget_bytes() == 0 and !enabled(). + // the cache's budget is EXACTLY the default policy applied to the + // available-memory reading it derived it from, and the cache's enabled + // state agrees with that budget. + // + // The default is ON — at or above the measured knee — on a host that can + // afford the knee, and a CLEAN DISABLE (budget 0, the uncached path) on a + // host that cannot. The helper checks the policy's answer, so the case + // holds on every host; the knee and the disable rule are pinned by the + // pure policy case below, which does not read the host at all. REQUIRE_EQ(std::getenv("VT_KOLIBRI1_DEQUANT_CACHE_MB"), static_cast(nullptr)); - const pid_t pid = fork(); - REQUIRE(pid >= 0); - if (pid == 0) { - char* envp[] = {nullptr}; // empty environment: the var is provably unset - execle(KOLIBRI_DEQUANT_CACHE_DEFAULT_HELPER, "helper", static_cast(nullptr), - envp); - _exit(127); // exec failed — not a pass - } - int status = 0; - REQUIRE_EQ(waitpid(pid, &status, 0), pid); - REQUIRE(WIFEXITED(status)); - CHECK_EQ(WEXITSTATUS(status), 0); + CHECK_EQ(RunHelper(nullptr, nullptr), 0); +#endif +} + +// ── 8. The knob: 0 disables, N is exact, junk falls back to the default ─ +TEST_CASE("kolibri1 dequant cache: VT_KOLIBRI1_DEQUANT_CACHE_MB disables at 0, " + "sets the budget exactly, and falls back to the default when " + "unparseable") { +#if defined(_WIN32) + WARN("default-budget probe disabled on Windows (POSIX fork/exec only)"); +#else + // The OFF switch: the escape hatch that restores the original threaded + // pool decode, byte for byte. + CHECK_EQ(RunHelper("0", nullptr), 0); + // The EXACT budget: the shape the 2026-10-20 measurement was taken in + // (16 GiB), and the shape a caller pins when it wants a reproducible + // residency. + CHECK_EQ(RunHelper("16384", nullptr), 0); + CHECK_EQ(RunHelper("1", nullptr), 0); + // A MALFORMED value must not switch a default-ON lever off: the default + // policy applies instead. + CHECK_EQ(RunHelper("not-a-number", nullptr), 0); + CHECK_EQ(RunHelper("", nullptr), 0); + // A negative value is not a budget either. + CHECK_EQ(RunHelper("-1", nullptr), 0); #endif } +// ── 9. The policy: pure, so a policy change is a deterministic red ─ +TEST_CASE("kolibri1 dequant cache: the default budget policy floors at the " + "measured knee, disables a host that cannot afford it, and reads " + "the host's memory before the checkpoint load") { + using namespace vllm::kolibri1_dequant_cache; + // An UNKNOWN memory report uses the KNEE: not a refusal (the house rule: + // an unknown budget must not become a false refusal) and never a sub-knee + // budget. A host that cannot hold the knee degrades to the uncached path + // through the allocation fallback. + CHECK_EQ(DefaultBudgetBytesFor(0), kDefaultBudgetKneeBytes); + // THE KNEE (2026-07-20, re-measured 2026-10-21; node 3, aarch64, 3.4 GHz, + // 32 threads, empty-prompt 32-token decode): 16 GiB is where the cache + // stops thrashing. At 8.25 GiB the measured hit rate was 33% with 16068 + // evictions, and the decode throughput (3.84 tok/s) was indistinguishable + // from the cache being off (3.62 tok/s), against 6.38 tok/s at an explicit + // 16 GiB. A host that can afford the knee must not be served a budget + // below it. + CHECK_EQ(DefaultBudgetBytesFor(66ull << 30), kDefaultBudgetKneeBytes); + CHECK_EQ(DefaultBudgetBytesFor(64ull << 30), kDefaultBudgetKneeBytes); + CHECK_EQ(DefaultBudgetBytesFor(128ull << 30), kDefaultBudgetKneeBytes); + // THE DIVISOR: above the knee the budget is the available eighth, until + // the cap. + CHECK_EQ(DefaultBudgetBytesFor(200ull << 30), 25ull << 30); + CHECK_EQ(DefaultBudgetBytesFor(512ull << 30), kDefaultBudgetMaxBytes); + // A HOST THAT CANNOT AFFORD THE KNEE DISABLES CLEANLY. A budget of 0 is + // the UNCACHED path (the original threaded pool dequant), byte for byte. + // A sub-knee cache is WORSE than useless: it costs the memory and buys + // nothing, so the policy refuses it instead of serving one. + CHECK_EQ(DefaultBudgetBytesFor(1ull << 30), 0); + CHECK_EQ(DefaultBudgetBytesFor(8ull << 30), 0); + CHECK_EQ(DefaultBudgetBytesFor(kDefaultBudgetKneeBytes + + kDefaultBudgetHostReserveBytes - 1), + 0); + // ... and the first reading that affords the knee is served the knee. + CHECK_EQ(DefaultBudgetBytesFor(kDefaultBudgetKneeBytes + + kDefaultBudgetHostReserveBytes), + kDefaultBudgetKneeBytes); + // THE HONEST READING: the MAXIMUM of the process-start reading and the + // reading at plan time. Our own 74 GB checkpoint load depresses + // MemAvailable (66 GiB) under the host's idle reading (~212 GiB), and the + // budget must not be derived from the depressed one. + CHECK_EQ(HonestAvailableBytes(212ull << 30, 66ull << 30), 212ull << 30); + CHECK_EQ(HonestAvailableBytes(66ull << 30, 212ull << 30), 212ull << 30); + CHECK_EQ(HonestAvailableBytes(66ull << 30, 66ull << 30), 66ull << 30); + CHECK_EQ(HonestAvailableBytes(0, 66ull << 30), 66ull << 30); + // THE DEPRESSED-READING CASE: a plan-time reading of 66 GiB under a + // process-start reading of 212 GiB lands on the honest eighth, above the + // knee; the same 66 GiB alone lands on the knee. + CHECK(DefaultBudgetBytesFor( + HonestAvailableBytes(212ull << 30, 66ull << 30)) >= + kDefaultBudgetKneeBytes); + CHECK_EQ(DefaultBudgetBytesFor( + HonestAvailableBytes(212ull << 30, 66ull << 30)), + (212ull << 30) / 8); + CHECK_EQ(DefaultBudgetBytesFor( + HonestAvailableBytes(66ull << 30, 66ull << 30)), + kDefaultBudgetKneeBytes); + // The policy never leaves the documented band, for every pair of the two + // moments the plan can combine: a budget is either DISABLED (0) or at or + // above the knee, and never above the cap. + for (uint64_t avail = 0; avail <= (1ull << 40); avail += (4ull << 30)) { + for (uint64_t start = 0; start <= (1ull << 40); start += (4ull << 30)) { + const uint64_t b = + DefaultBudgetBytesFor(HonestAvailableBytes(start, avail)); + const bool in_band = + (b == 0) || + (b >= kDefaultBudgetKneeBytes && b <= kDefaultBudgetMaxBytes); + CHECK(in_band); + } + } +} + +// ── 13. The RED-FIRST WITNESS: a load-depressed reading lands at or above +// the knee, never below it ────────────────────────────────────────────────── +TEST_CASE("kolibri1 dequant cache: a load-depressed reading must not land " + "below the measured knee") { + // THE RED-FIRST CASE (2026-10-21). The default budget computed from a + // LOAD-DEPRESSED reading — what /proc/meminfo reads while our own 74 GB + // checkpoint load fills the page cache — must not land BELOW the measured + // knee: 16 GiB is where the cache stops thrashing, and a budget under it + // costs the memory and buys nothing (3.84 tok/s against 3.62 tok/s with + // the cache off, 2026-10-21, node 3). + // + // The knee is a LITERAL here on purpose: this case compiles against the + // pre-repair policy as well, so the red is an ASSERTION FAILURE and not a + // compile error. Against the pre-repair policy the depressed reading + // divided by eight lands at 8.25 GiB and this case FAILS; against the + // repaired policy the knee floor serves 16 GiB and it PASSES. + const uint64_t kDepressedReading = 66ull << 30; // MemAvailable during the load + const uint64_t budget = + vllm::kolibri1_dequant_cache::DefaultBudgetBytesFor(kDepressedReading); + const bool at_or_above_the_knee = (budget == 0) || (budget >= (16ull << 30)); + CHECK(at_or_above_the_knee); +} + // ── 6. Reload identity: reused weight addresses, different model (P1a) ───── TEST_CASE("kolibri1 dequant cache: a reloaded model at reused weight " "addresses is served its own bytes, not the previous model's") { @@ -457,3 +660,129 @@ TEST_CASE("kolibri1 dequant cache: a held hit's bytes survive a concurrent " CHECK(counters.evictions - evictions0 >= 1); CHECK(counters.misses - misses0 >= 1); } + +// ── 12. The SAMPLING MOMENT: the plan reads the host at process start ─ +TEST_CASE("kolibri1 dequant cache: the plan's source reading is the honest " + "maximum of the process-start and plan-time readings") { + using namespace vllm::kolibri1_dequant_cache; +#if defined(_WIN32) + WARN("default-budget probe disabled on Windows (POSIX fork/exec only)"); +#else + // The PROCESS-START reading was taken: /proc/meminfo is readable at process + // start, so a zero here would mean the capture never happened. + CHECK(kStartAvailableBytes > 0ull); + // THIS process's plan reads the process-start reading, and its source + // reading is the honest maximum of the two moments. + const auto& own = vllm::kolibri1_dequant_cache::ProcessCache(); + CHECK_EQ(own.budget_start_bytes(), kStartAvailableBytes); + CHECK_EQ(own.budget_source_bytes(), + vllm::kolibri1_dequant_cache::HonestAvailableBytes( + own.budget_start_bytes(), own.budget_plan_bytes())); + CHECK_EQ(own.budget_bytes(), + vllm::kolibri1_dequant_cache::DefaultBudgetBytesFor( + own.budget_source_bytes())); + // The DEFAULT plan, in a process whose environment provably lacks the knob. + std::string out; + REQUIRE_EQ(RunHelper(nullptr, &out), 0); + bool found = false; + const uint64_t budget = FieldValue(out, "budget_bytes", &found); + REQUIRE(found); + const uint64_t source = FieldValue(out, "budget_source_bytes", &found); + REQUIRE(found); + const uint64_t start = FieldValue(out, "budget_start_bytes", &found); + REQUIRE(found); + const uint64_t plan = FieldValue(out, "budget_plan_bytes", &found); + REQUIRE(found); + // The source reading is the HONEST one: the maximum of the two moments, + // so a checkpoint load that depressed the plan-time reading cannot talk the + // budget down. + CHECK_EQ(source, HonestAvailableBytes(start, plan)); + // And the budget is the policy applied to that reading. + CHECK_EQ(budget, DefaultBudgetBytesFor(source)); + // The KNEE: a budget is either DISABLED (the host cannot afford the knee) + // or at or above it, so the default cannot ship the thrashing shape. + if (budget != 0) { + CHECK(budget >= kDefaultBudgetKneeBytes); + } +#endif +} + +// ── 10. Byte identity against the PRODUCTION uncached dequant ──────────────── +TEST_CASE("kolibri1 dequant cache: the cached bytes are byte-identical to the " + "production uncached dequant of the same weight") { + // THE CLAIM THIS CHANGE MUST NOT BREAK. The cache is a CACHING change, so + // the bytes it serves are the bytes the uncached path would have produced. + // The reference here is the PRODUCTION uncached dequant itself + // (`DequantFp8BlockUncached` — the body the CPU forward runs when the cache + // is disabled), not a transcription of it, so a divergence in either + // implementation is caught. + const int64_t kN = 512, kK = 2560; // the shape the CACHE MISMATCH named + const Fp8BlockWeight w = MakeWeight(kN, kK, 128, 128, 5); + vt::Queue queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + vllm::dense_attn::Dev d{vt::GetBackend(vt::DeviceType::kCPU), queue}; + const vllm::dense_attn::DBuf uncached = vllm::DequantFp8BlockUncached(d, w); + const size_t bytes = static_cast(kN * kK) * sizeof(uint16_t); + REQUIRE_EQ(uncached.bytes(), bytes); + + Cache cache(bytes * 4); + // The COLD miss: the bytes the cache stores. + const auto cold = cache.GetOrDequant(w); + REQUIRE(static_cast(cold)); + CHECK_EQ(std::memcmp(cold.data(), uncached.ptr(), bytes), 0); + // The HOT hit: the bytes the cache serves, which are the SAME resident + // bytes (a hit never re-decodes). + const auto hot = cache.GetOrDequant(w); + REQUIRE(static_cast(hot)); + CHECK_EQ(std::memcmp(hot.data(), uncached.ptr(), bytes), 0); + CHECK_EQ(hot.data(), cold.data()); + // And the cache really held it: the second call was a hit. + CHECK_EQ(vllm::kolibri1_dequant_cache::Counters().hits >= 1, true); +} + +// ── 11. The fallback: a degraded cache leaves the forward on the uncached path +TEST_CASE("kolibri1 dequant cache: a cache that cannot hold its bytes falls " + "back to the uncached path with the same bytes") { + // A weight no allocator can hold: 2^31 x 2^31 elements is 2^62 bf16 + // values, 8 EiB. The decode allocation fails, so the cache degrades and + // hands back an EMPTY lease — the signal the production resolve reads as + // "use the uncached path". The forward must continue, not abort. + Fp8BlockWeight impossible; + impossible.n = 1ll << 31; + impossible.k = 1ll << 31; + impossible.block_n = 128; + impossible.block_k = 128; + impossible.packed.bytes = vllm::OwnedBytes(std::vector(1, 0)); + impossible.packed.dtype = vt::DType::kI8; + impossible.packed.rank = 2; + impossible.scale.bytes = vllm::OwnedBytes(std::vector(4, 0)); + impossible.scale.dtype = vt::DType::kF32; + impossible.scale.rank = 2; + + Cache cache(1u << 30); + const auto& counters = vllm::kolibri1_dequant_cache::Counters(); + const uint64_t failures0 = counters.alloc_failures; + const auto lease = cache.GetOrDequant(impossible); + CHECK(!static_cast(lease)); // the empty lease: the fallback signal + CHECK(!cache.enabled()); // the cache degraded itself + CHECK_EQ(cache.resident_bytes(), 0); // and handed its memory back + CHECK_EQ(counters.alloc_failures - failures0, 1); + + // The PRODUCTION resolve, on a REAL weight, with the degraded cache: it + // must take the uncached path and hand the GEMM the same bytes. + const int64_t kN = 64, kK = 512; + const Fp8BlockWeight w = MakeWeight(kN, kK, 32, 128, 7); + vt::Queue queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + vllm::dense_attn::Dev d{vt::GetBackend(vt::DeviceType::kCPU), queue}; + const auto degraded = vllm::ResolveFp8Projection(d, w, cache); + CHECK_EQ(degraded.cached, false); + const auto ref = RefDecode(w); + REQUIRE(degraded.wt.data != nullptr); + CHECK_EQ(std::memcmp(degraded.wt.data, ref.data(), ref.size() * 2), 0); + + // And with a HEALTHY cache the same resolve takes the cached path and + // serves the same bytes: the fallback changes the route, not the result. + Cache healthy(1u << 30); + const auto cached = vllm::ResolveFp8Projection(d, w, healthy); + CHECK_EQ(cached.cached, true); + CHECK_EQ(std::memcmp(cached.wt.data, ref.data(), ref.size() * 2), 0); +} diff --git a/tests/vllm/models/test_kolibri1_dequant_cache_default_helper.cpp b/tests/vllm/models/test_kolibri1_dequant_cache_default_helper.cpp index e9623f4af..ee913fe58 100644 --- a/tests/vllm/models/test_kolibri1_dequant_cache_default_helper.cpp +++ b/tests/vllm/models/test_kolibri1_dequant_cache_default_helper.cpp @@ -1,16 +1,95 @@ -// The default-off probe for test_kolibri1_dequant_cache (case 5). A separate -// binary so the ProcessCache() static budget is initialized in a process whose -// environment PROVABLY lacks VT_KOLIBRI1_DEQUANT_CACHE_MB: the parent test -// exec's this helper with an empty environment, and this helper exits 0 iff -// the default configuration is a disabled cache (budget 0). Any default flip -// (e.g. a nonzero fallback MiB) makes this exit nonzero and the parent case -// fail. No doctest main: plain exit status is the contract. +// The default-budget probe for test_kolibri1_dequant_cache (cases 5, 8 and +// 9). A separate binary so the ProcessCache() static budget is initialized in +// a process whose environment PROVABLY lacks VT_KOLIBRI1_DEQUANT_CACHE_MB (or +// carries exactly the value the parent asked for): the parent test exec's +// this helper, and the helper exits 0 iff the configuration it observes is the +// one the case expects. +// +// MODES (argv[1]): +// (absent) the DEFAULT. The parent exec's this helper with an EMPTY +// environment, so the variable is provably unset here, and the +// cache must be ENABLED with the DERIVED budget. +// "0" DISABLED: budget 0, not enabled. +// "" EXACT: budget N MiB, enabled iff N > 0. +// anything else MALFORMED: the knob does not parse, so the DEFAULT +// policy applies — an unparseable value must not silently +// switch a default-ON lever off. +// +// The observed budget is printed on stdout so the parent can record it. No +// doctest main: the exit status is the contract. +#include #include +#include #include "vllm/model_executor/models/kolibri1_dequant_cache.h" -int main() { - if (std::getenv("VT_KOLIBRI1_DEQUANT_CACHE_MB") != nullptr) return 2; +namespace { + +using vllm::kolibri1_dequant_cache::DefaultBudgetBytesFor; +using vllm::kolibri1_dequant_cache::HonestAvailableBytes; +using vllm::kolibri1_dequant_cache::kDefaultBudgetKneeBytes; +using vllm::kolibri1_dequant_cache::kDefaultBudgetMaxBytes; + +bool IsDigits(const char* s) { + if (s == nullptr || *s == '\0') return false; + for (const char* p = s; *p != '\0'; ++p) { + if (*p < '0' || *p > '9') return false; + } + return true; +} + +} // namespace + +int main(int argc, char** argv) { + const char* mode = argc > 1 ? argv[1] : nullptr; + if (mode == nullptr) { + // The DEFAULT mode: the parent exec's this helper with an EMPTY + // environment, so the variable is provably unset here. + if (std::getenv("VT_KOLIBRI1_DEQUANT_CACHE_MB") != nullptr) return 2; + } else { +#if defined(_WIN32) + if (_putenv_s("VT_KOLIBRI1_DEQUANT_CACHE_MB", mode) != 0) return 3; +#else + if (setenv("VT_KOLIBRI1_DEQUANT_CACHE_MB", mode, 1) != 0) return 3; +#endif + } const auto& cache = vllm::kolibri1_dequant_cache::ProcessCache(); - return (cache.budget_bytes() == 0 && !cache.enabled()) ? 0 : 1; + const unsigned long long budget = cache.budget_bytes(); + const unsigned long long source = cache.budget_source_bytes(); + const unsigned long long start = cache.budget_start_bytes(); + const unsigned long long plan = cache.budget_plan_bytes(); + std::printf("budget_bytes=%llu budget_source_bytes=%llu " + "budget_start_bytes=%llu budget_plan_bytes=%llu enabled=%d\n", + budget, source, start, plan, cache.enabled() ? 1 : 0); + std::fflush(stdout); + + bool ok = false; + if (mode == nullptr) { + // The DEFAULT: the policy's answer for this host, and nothing else. The + // budget is exactly the policy applied to the HONEST reading the cache + // derived it from (the maximum of the process-start reading and the + // reading at plan time). + ok = source == HonestAvailableBytes(start, plan) && + budget == DefaultBudgetBytesFor(source) && + cache.enabled() == (budget > 0); + // The KNEE: a budget is either DISABLED (a host that cannot afford the + // knee) or at or above it, so the default never ships the thrashing + // shape. + ok = ok && (budget == 0 || (budget >= kDefaultBudgetKneeBytes && + budget <= kDefaultBudgetMaxBytes)); + } else if (std::strcmp(mode, "0") == 0) { + // The explicit OFF switch. + ok = !cache.enabled() && budget == 0; + } else if (IsDigits(mode)) { + // The explicit budget, exactly. + const unsigned long long mib = std::strtoull(mode, nullptr, 10); + ok = budget == mib * 1024ull * 1024ull && cache.enabled() == (mib > 0); + } else { + // MALFORMED: the DEFAULT policy applies — the knob does not parse, so it + // is not the explicit "0" (the off switch) and it is not an exact + // budget either: it is the default policy's answer, byte for byte. + ok = budget == DefaultBudgetBytesFor(source) && + cache.enabled() == (budget > 0); + } + return ok ? 0 : 1; }