kolibri1 CPU: adopt the bf16-dot compute class as the aarch64 default

.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5AA1RRQYAT1JG3MDSKNEYAH.md

@@ -0,0 +1,19 @@
+ID: ISSUE-LOCAL-01M5AA1RRQYAT1JG3MDSKNEYAH
+Title: test_kolibri1_spec_probe anchor stale since the FMA-contract re-anchor
+Row: MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm
+State: CLOSED
+Kind: bug
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-10-19
+Updated: 2026-10-19
+Closed: 2026-10-19
+
+## Problem
+
+The spec probe's plain-chain anchor is 109726 (captured main 323f81ca6, pre-FMA-contract). The f32 production chain has been constant 33382 since the paged-attention FMA-contract re-anchor (decode bench re-anchor comment), so the probe's GATE (ii) CHECK(plain.chain.back()==kAnchorLastToken) has been red on main's f32 tier; evidenced 2026-10-19 by VT_CPU_KOLIBRI_BF16_DOT=0 (bit-identical to main production) printing CHECK( 33382 == 109726 ). Found during row MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm bf16-dot adoption; repaired in the same flow with per-tier anchors (f32 33382, bf16-dot default 11622).
+
+## Resolution
+
+2026-10-19: per-tier anchors landed with the bf16-dot adoption on row/kolibri-bf16-dot-cpu; the f32 leg (VT_CPU_KOLIBRI_BF16_DOT=0, bit-identical to main production) now passes GATE (ii) with anchor 33382 and 8/8 spec-vs-plain byte-identity; the bf16-dot default tier anchors 11622 (3/3 identical chains).

.agents/specs/kolibri1-bf16-dot-cpu.md

@@ -0,0 +1,97 @@
+# SPEC: kolibri1-bf16-dot-cpu — adopt the bf16-dot compute class as the default on aarch64
+
+Row: MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm
+Status: ACTIVE (spec committed before implementation; one PR carries spec + implementation — recorded developer preference)
+Decision authority: developer ruling 2026-10-19 — "we can support bf16 in hardware both on aarch64 and p150 so makes no sense not doing it" (resolves the direction of #3438).
+
+## Problem
+
+Kolibri-1's CPU production path dequantizes fp8 to f32 and multiplies in f32.
+The primary oracle (aleph-alpha-inference via vLLM 0.31.0+cpu) computes the
+same matmuls in the bf16-dot class (BFMMLA, 11,328 instances, #3438 evidence).
+The three-way analysis (#3469, landed a70efb8cb) shows production and the
+C draft are a TIE against the primary (~98% top-1 agreement either way) — so
+production f32-dequant is the outlier, not the reference. The bf16-dot arms
+exist behind default-off knobs (#3464) but nothing reaches them in production.
+
+## Scope
+
+Flip to default-on (hardware-gated, not env-gated) the kolibri1 arms that
+consume bf16 inputs directly on aarch64 when `DetectArmBf16()` proves
+FEAT_BF16 at runtime:
+
+1. Routed-expert GEMV — `kolibri1_bfdot_gemv.cpp` (was `VT_CPU_KOLIBRI_BFDOT_GEMV=1` opt-in).
+2. Paged-attention decode dot — `cpu_paged_attn_bfdot_arm.cpp` (was `VT_CPU_PAGED_ATTN_BFDOT=1` opt-in).
+3. The lm_head / router bf16 extensions from #3464 (same gating).
+
+Out of scope: x86 (no bf16-dot tier exists; the f32 path is unchanged and
+remains the only tier there), non-bf16 aarch64 (fallback tier unchanged),
+the P150 (separate surface — the band-gate re-score is a follow-up row),
+the silu LUT (already unconditional and bit-exact — untouched), W8A8
+(falsified, stays off).
+
+## Design
+
+- Dispatch: feature-gated at runtime (`DetectArmBf16()` + the existing
+  per-arm usability checks). No environment variable gates the DEFAULT path;
+  vLLM dispatches numerics kernels on CPU feature probes, not env knobs.
+  An escape hatch remains: `VT_CPU_KOLIBRI_BF16_DOT=0` forces the f32 tier
+  for the arms above (single knob, replaces the three per-arm opt-ins; the
+  old knobs keep working as explicit `=1` for one release, then go away).
+- Numerics contract: bf16 inputs, f32 accumulate (BFDOT semantics) — the
+  class the primary oracle uses. The f32 fallback tier's byte stream is
+  unchanged, so a non-bf16 host never moves.
+- Gate re-adjudication: the CPU goldens scored against
+  `kolibri1_goldens.json` (the torch transcription) move to the primary
+  artifacts (`kolibri1_goldens_primary.json` + the top-100 rows, landed via
+  #3463). Expected: first-token agreement with the primary RISES (the C
+  draft already matched it on the streams where production flipped). The
+  greedy chain gates get re-recorded under the new default, with the old
+  f32-tier streams preserved under the escape hatch as the A/B leg.
+
+## Upstream anchors
+
+- vLLM CPU bf16 GEMM dispatch: runtime feature probes, no env gates.
+- Primary oracle numerics: BFMMLA class, #3438 evidence thread.
+- `scripts/gen-kolibri1-primary-goldens.py` / `-topk-rows.py` — the
+  denominators for re-adjudication.
+
+## Tests
+
+- Existing arm tests (from #3464) flip from knob-gated-red to
+  default-path-green on aarch64 CI; add one test asserting the DEFAULT
+  dispatch selects the bf16-dot arm when `DetectArmBf16()` is true, and the
+  escape hatch returns the f32 stream.
+- Full kolibri1 CPU suite + the three-way capture test
+  (`test_kolibri1_threeway_capture.cpp`) green under the new default.
+- Non-aarch64 CI: zero behavioral change (the source-level guarantee the
+  #3464 arch-guard incident taught us to verify explicitly).
+
+## Gates
+
+- aarch64 gate (this host) and the x86 CI lanes green.
+- Token gate: greedy decode vs the primary oracle rows — per-position
+  agreement recorded BEFORE (f32 production) and AFTER (bf16-dot default);
+  the AFTER number must be >= the BEFORE number for the row to land.
+  This is the row's red line: adoption may not move AWAY from the primary.
+
+## Evidence
+
+- Before/after agreement numbers, the re-recorded goldens, and the bench
+  number for the flipped arms (the C draft measured +21% on the GEMV leg
+  historically; record the production-shape number, not the micro).
+
+## Risks / stop conditions
+
+- If the AFTER agreement drops below BEFORE, STOP: file the divergence,
+  revert to opt-in, and take it to the developer — the ruling sanctions the
+  class, not a regression.
+- If the escape hatch cannot reproduce the old f32 stream bit-for-bit
+  (dispatch accidentally altered the fallback), STOP: that is a dispatch
+  bug, not a numerics trade.
+
+## Owed
+
+- P150 band-gate re-score (separate row; needs the developer's explicit
+  band-gate confirmation).
+- W8A8 stays falsified/off — no arm.

docs/ENVIRONMENT.md

@@ -281,6 +281,7 @@ portable/reference path. In normal operation leave them unset.
 | `VT_CPU_REF` | off | Set on to force the portable reference path (dequantize-everything oracle), the standard "is this a kernel bug?" bisect switch |
 | `VT_CPU_MATMUL_TIER` | `auto` | Force the elementwise CPU GEMM tier in the same adaptive binary: `portable`/`neon` on arm64, or `portable`/`sse2`/`sse2+f16c`/`avx2`/`avx512` on x86_64. An unsupported or cross-architecture tier fails closed after the exact OS capability probe |
 | `VT_CPU_Q8_DOT` | `auto` | Select the Arm Q8_0 x Q8_0 inner dot: `portable`, compiler-generated `sdot`, or the Linux-only `a76-asm`. `auto` selects the A76 schedule on Cortex-A76, otherwise SDOT on any Linux/Darwin DotProd-capable host, otherwise portable. A forced unsupported tier fails closed |
+| `VT_CPU_KOLIBRI_BF16_DOT` | **on** (aarch64 with FEAT_BF16) | The kolibri1 bf16-dot CPU compute class is the DEFAULT wherever the HWCAP2 FEAT_BF16 probe (`DetectArmBf16`) proves the hardware: the routed-expert/attention decode GEMV over the warm-cache bf16 weights (`vbfdotq_f32`, bf16 in / f32 accumulate), the paged-attention decode K dot, and the lm_head / router-gate decode GEMV (`src/vllm/model_executor/models/kolibri1_bfdot_gemv.cpp`, `src/vt/cpu/cpu_paged_attn_bfdot_arm.cpp`). This mirrors vLLM, which dispatches CPU numerics kernels on feature probes, not env knobs. The f32 tier it replaces stays the only tier on x86 and on non-bf16 aarch64, so those hosts never move. `=0` is the single escape hatch: it forces the f32 tier for ALL of these arms in the same binary (the A/B leg, bit-identical to the pre-adoption production stream). The paged-attention engagement rides the NEON lane body (K dot on the BFDOT arm, V accumulation on the lane's widen+vfma path), so the lane's load-contention caveat (`VT_CPU_PAGED_ATTN_NEON` row) applies to the decode dot too. The routed-expert GEMV additionally needs the dequant cache (`VT_KOLIBRI1_DEQUANT_CACHE_MB`) for its warm-bf16 residency precondition. The legacy per-arm opt-ins `VT_CPU_KOLIBRI_BFDOT_GEMV=1` and `VT_CPU_PAGED_ATTN_BFDOT=1` keep working for one release and then go away (SPEC `.agents/specs/kolibri1-bf16-dot-cpu.md`; #3438) |
 | `VT_CPU_PAGED_ATTN_NEON` | off | Set `1` to run the aarch64 NEON lane for the CPU paged-attention K dot-product and V accumulation (`src/vt/cpu/cpu_paged_attn.cpp`). Not bit-identical to the scalar path (the K reduction runs four fused-multiply lanes): outputs agree to f32 noise plus one bf16 output ulp, and greedy chains can flip near-ties — the kolibri gates adjudicate them; the decode-bench anchor is byte-exact and has no near-tie instrument yet, which is why the default is off (docs/bench-evidence/kolibri1-neon-paged-attn-20261008.md). Measured: attn_core halves (4.9→2.3 s at t32 over the 64-step decode bench). Widths that are not multiples of 4 in head or v-head dim, and every non-aarch64 build, use the scalar path regardless |
 | `VT_CPU_QUANT_MMLA` | `auto` | Select the Arm i8mm quant-dot tier. `portable` (also `0`/`off`/`false`) disables it; `i8mm` (also `1`/`on`/`true`) forces it and fails closed unless Linux `HWCAP2_I8MM` or Darwin `hw.optional.arm.FEAT_I8MM` is present |
 | `VT_CPU_QUANT_REPACK` | `auto` | Select the Arm i8mm+DotProd Q8_0 repack tier. `portable` (also `0`/`off`/`false`) disables it; `i8mm` (also `1`/`on`/`true`) forces it and fails closed unless the exact Linux HWCAP/HWCAP2 or Darwin sysctl capabilities are present |

docs/FEATURES.md

@@ -224,7 +224,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks.
 | `KevModel` | `jaredpalmer/kev-0.8b` (frozen Qwen3.5-0.8B-Base + rank-16 LoRA merged at convert time + PointerHead readout) | PointerHead golden-vector tests (25 cases, 126 assertions) + LoRA-merge + ForwardHidden goldens with perturbation gates vs the kev Python reference `jaredpalmer/kev` @`19dcae9b6e3e1a48200c5825aad9fc200d31e20a` (#3295); `/v1/systemone` dispatch reuses the Laya lane. E2E choice/score/noul through LocalAI `/v1/systemone` PASS. CUDA build verification + GGUF k-quant arm OWED per [spec](../.agents/specs/kev.md) | not measured |
 | `DeepseekV41ForCausalLM` | none loadable yet | **REGISTERED AND VALIDATING; NOT LOADABLE** (W1, 2026-09-13) per [model-matrix](../.agents/model-matrix.md) and [spec](../.agents/specs/deepseek-v4-1-flash.md): the architecture resolves through the registry while the load plan is validated; no token claim | not measured |
 | `MiMoV2ForCausalLM` | `vcruz305/MiMoV2-Flash-RL-EXL3` (EXL3-quantized, not yet loadable end-to-end) | **W1+W2 scaffold + W3 forward pass; NOT GATED.** Registry + config + hybrid KV-cache spec (full-attn 4 KV heads / SWA 8 KV heads, `v_head_dim=128 != head_dim=192`) + bf16/EXL3 dual-path weight loader gated; the device forward computes through all 48 layers (hybrid attention with sink bias, partial RoPE, attention_value_scale; layer 0 dense MLP, layers 1-47 MoE with sigmoid + noaux_tc routing) but no token has been emitted yet — the end-to-end gate (W5) is OWED | not measured |
-| `Kolibri1ForCausalLM` | `Aleph-Alpha/Kolibri-1`, mixed FP8/BF16 safetensors (78.8 GB) | CPU and Tenstorrent forwards. FP8 weights stay compressed at load. Kolibri chat template, reasoning, and tool-call parsers. CPU comparisons include accepted near-ties, not full token parity. GGUF and other device forwards refused. [Recipe and validation limits](USAGE.md#kolibri-1) | Full author-plugin comparison pending. [TT measurements and gaps](gap-kolibri1-tt-captured-decode.md) |
+| `Kolibri1ForCausalLM` | `Aleph-Alpha/Kolibri-1`, mixed FP8/BF16 safetensors (78.8 GB) | CPU and Tenstorrent forwards. FP8 weights stay compressed at load. CPU decode numerics default to the bf16-dot tier (bf16 in / f32 accumulate, the BFMMLA class) on aarch64 FEAT_BF16 hosts; `VT_CPU_KOLIBRI_BF16_DOT=0` is the f32-tier same-binary rollback. Kolibri chat template, reasoning, and tool-call parsers. CPU comparisons include accepted near-ties, not full token parity. GGUF and other device forwards refused. [Recipe and validation limits](USAGE.md#kolibri-1) | Full author-plugin comparison pending. [TT measurements and gaps](gap-kolibri1-tt-captured-decode.md) |
 <!-- supported-arch-table:end -->
 
 ### Standalone and non-registered lanes

docs/bench-evidence/kolibri1-bf16dot-adoption-20261019/report.md

@@ -0,0 +1,75 @@
+# Kolibri-1 bf16-dot CPU adoption: default-flip gate evidence (2026-10-19)
+
+Row: MODEL-TEXT-kolibri-1 (CPU lane). Branch `row/kolibri-bf16-dot-cpu`
+(fresh from `origin/main` `4b3a0091c`; spec
+`.agents/specs/kolibri1-bf16-dot-cpu.md` committed before implementation).
+
+## Question
+
+Does flipping the kolibri1 bf16-dot arms (routed-expert GEMV, paged-attention
+decode K dot, lm_head / router gate) from opt-in knobs to the hardware-gated
+DEFAULT on aarch64 FEAT_BF16 hold the token gate (top-1 agreement with the
+primary oracle must not drop) and what does the decode GEMV leg cost or save
+in the production shape?
+
+## Method
+
+One binary, same-tree A/B; the f32 leg is the escape hatch
+`VT_CPU_KOLIBRI_BF16_DOT=0`, the bf16 leg is the unset default. Host:
+aarch64 Ampere-1a (128 threads, FEAT_BF16 via HWCAP2), contended shared box —
+the A/B is same-binary and interleaved-legs-in-one-session, but the absolute
+numbers are contended-host numbers; only the RATIO is evidence. Weights:
+`/mnt/models/Aleph-Alpha/Kolibri-1` fp8 checkpoint, unchanged from the
+previous reports. `VT_KOLIBRI1_DEQUANT_CACHE_MB=16384` on BOTH legs so the
+GEMV arm's warm-bf16 residency precondition holds on both sides and the only
+difference is the compute class.
+
+## Token gate (the red line)
+
+`tests/vllm/models/test_kolibri1_threeway_capture.cpp` dumps, and
+`scripts/kolibri1-threeway-analysis.py` scores, the same 202 aligned
+positions against the primary oracle's published top-100 rows
+(`kolibri1_primary_topk_rows.json`):
+
+| mode | top-1 | top-5 | top-100 | gap mean (nats) | cos mean | KL mean (nats) |
+|---|---|---|---|---|---|---|
+| BEFORE f32 (`=0`) | 97.0% | 100.0% | 100.0% | 5.842 | 0.9812 | 6.994 |
+| AFTER bf16-dot default | **98.0%** | 100.0% | 100.0% | 5.854 | 0.9804 | 7.018 |
+
+AFTER >= BEFORE: the gate holds and moves TOWARD the primary, as the spec
+predicted from the C-draft tie. Depth-growth: the gain concentrates in the
+6-15 bin (96.8% -> 98.4%) and the 16-31 bin (98.0% -> 99.0%).
+
+## Decode bench (production shape, 128-in / 63 greedy steps)
+
+`tests/test_kolibri1_decode_bench`, 3 runs per leg, median:
+
+| leg | decode s (median) | tok/s (median) | runs |
+|---|---|---|---|
+| f32 (`=0`) | 14.068 | 4.478 | 4.453 / 4.478 / 4.287 |
+| bf16-dot default | 11.948 | 5.273 | 5.201 / 5.284 / 5.273 |
+
+**+17.7% decode tok/s** on the flipped GEMV leg (1.177x), consistent with the
+C draft's historical +21% micro measurement. Chains: f32 constant 33382
+(3/3), bf16 constant 11622 (3/3) — both tier anchors re-recorded in the bench
+gate.
+
+## W3 torch-golden adjudication
+
+- f32 leg (`=0`): 141/145 chain matches, 4 near-tie flips, 0 hard;
+  fingerprint band green — bit-identical to main's production gate.
+- bf16-dot default: 134/138 chain matches, 4 near-tie flips, 0 hard; the
+  torch-f32 topk fingerprint band (2.5 nats) no longer binds (worst 3.40) —
+  the fingerprints are the torch f32 transcription, so the default tier is
+  adjudicated against the PRIMARY artifacts above, per the spec's Design
+  section. The W3 gate is now tier-scoped, not relaxed.
+
+## spec probe
+
+The probe's anchor was stale on main (expected 109726, the f32 tier's chain
+has been 33382 since the FMA-contract re-anchor — red before this change;
+hatch run 2026-10-19 shows 33382). Re-anchored per tier. spec-vs-plain
+byte-identity holds 8/8 on the f32 tier; the bf16-dot default measures 6/8
+(golden[3], golden[5] — the two knife-edge prompts, where the plain t==1
+BFDOT arms and the batched verify rows are different numerics), recorded as
+the tier floor.

src/vllm/model_executor/models/kolibri1_bfdot_gemv.cpp

@@ -25,6 +25,11 @@ bool EnvOne(const char* name) {
   return e != nullptr && e[0] == '1';
 }
 
+bool EnvZero(const char* name) {
+  const char* e = std::getenv(name);
+  return e != nullptr && e[0] == '0';
+}
+
 }  // namespace
 
 bool KnobRequested() { return EnvOne("VT_CPU_KOLIBRI_BFDOT_GEMV"); }
@@ -44,6 +49,18 @@ bool KnobActive() {
   return true;
 }
 
+bool Bf16DotActive() {
+  // Legacy per-arm opt-in keeps working as an explicit `=1` (deprecated,
+  // one-release bridge): it keeps the refuse-by-name contract above.
+  if (KnobRequested()) return KnobActive();
+  // The DEFAULT dispatch is feature-gated, not env-gated: vLLM dispatches
+  // CPU numerics kernels on feature probes (SPEC kolibri1-bf16-dot-cpu
+  // Design). The single escape hatch VT_CPU_KOLIBRI_BF16_DOT=0 forces the
+  // f32 tier for every kolibri1 bf16-dot arm.
+  if (EnvZero("VT_CPU_KOLIBRI_BF16_DOT")) return false;
+  return vt::cpu::DetectArmBf16();
+}
+
 template <bool kF32Out>
 inline void StoreOut(void* out_ptr, int64_t row, float acc) {
   if (kF32Out)

src/vllm/model_executor/models/kolibri1_bfdot_gemv.h

@@ -43,8 +43,18 @@ bool KnobRequested();
 
 // Requested AND FEAT_BF16-probed. `=1` without the hardware feature refuses
 // BY NAME (VT_CHECK), so the knob can never silently stay off.
+// DEPRECATED as a default-shaping knob: kept one release as an explicit `=1`
+// opt-in; the default dispatch is `Bf16DotActive` below.
 bool KnobActive();
 
+// The DEFAULT dispatch (SPEC kolibri1-bf16-dot-cpu): feature-gated at
+// runtime — true when `DetectArmBf16()` proves FEAT_BF16, independent of any
+// environment variable. Escape hatch: `VT_CPU_KOLIBRI_BF16_DOT=0` forces the
+// f32 tier (returns false) on every kolibri1 bf16-dot arm. The legacy
+// `VT_CPU_KOLIBRI_BFDOT_GEMV=1` opt-in still selects the arm explicitly and
+// keeps its refuse-by-name contract.
+bool Bf16DotActive();
+
 // out[1, n] (bf16) = x[1, k] (bf16) @ w[1, n*k] (bf16 [n, k] row-major)^T —
 // the t==1 decode GEMV. `w` is the warm-cache bf16 tensor reused DIRECTLY
 // (no conversion, no repack). Threaded over output rows through the ONE