Repository navigation
kolibri1 CPU: adopt the bf16-dot compute class as the aarch64 default - #3482
Merged
Merged
Conversation
…ctions the hardware bf16 class on aarch64 The 2026-10-19 ruling resolves localai-org#3438's direction: bf16 in hardware is supported on both aarch64 and the P150, so keeping production on f32-dequant has no basis. The three-way analysis (localai-org#3469) already showed production and the C draft tie against the primary oracle — production is the outlier. This spec commits the adoption shape BEFORE any implementation: feature-gated default-on dispatch for the localai-org#3464 arms, an opt-out escape hatch, golden re-adjudication against the primary artifacts, and a red line — adoption may not lower agreement with the primary. One PR carries spec + implementation per the recorded preference. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
The kolibri1 CPU production path dequantizes fp8 to f32 and multiplies in f32, while the primary oracle (aleph-alpha-inference via vLLM 0.31.0+cpu) computes the same matmuls in the bf16-dot BFMMLA class (localai-org#3438 evidence). The three-way analysis (a70efb8) showed production f32-dequant is the outlier, and the developer ruling of 2026-10-19 sanctions the hardware bf16 class. This change flips the localai-org#3464 arms from opt-in knobs to the DEFAULT on aarch64 when DetectArmBf16() proves FEAT_BF16 at runtime — feature-gated dispatch like vLLM's CPU kernels, no env gate on the default path (spec: .agents/specs/kolibri1-bf16-dot-cpu.md). Scope: the routed-expert/attention decode GEMV over the warm-cache bf16 weights, the paged-attention decode K dot (which rides the NEON lane body: K on the BFDOT arm, V on the lane's widen+vfma path), and the lm_head / router-gate decode GEMV. One escape hatch, VT_CPU_KOLIBRI_BF16_DOT=0, forces the f32 tier for all of them in the same binary; the legacy VT_CPU_KOLIBRI_BFDOT_GEMV / VT_CPU_PAGED_ATTN_BFDOT opt-ins keep working as explicit =1 for one release, then go away. x86 and non-bf16 aarch64 keep the f32 tier as the only tier; every arm file keeps its arch includes inside the __aarch64__/__ARM_FEATURE_BF16 guards (the localai-org#3464 incident's source-level guarantee, re-audited). Evidence (docs/bench-evidence/kolibri1-bf16dot-adoption-20261019/): - Token gate (the red line): greedy-decode agreement with the primary oracle rows over the 202 aligned positions RISES, 97.0% -> 98.0% top-1 (BEFORE = VT_CPU_KOLIBRI_BF16_DOT=0, AFTER = the default, same binary, both legs with the dequant cache on so only the compute class differs). - Decode bench, production shape (128-in / 63 greedy steps), 3-run median, same binary: 4.478 -> 5.273 tok/s (+17.7%), consistent with the C draft's historical +21% GEMV measurement. - f32-tier bit-identity: with the hatch set, the W3 golden gate reproduces main's production result exactly (141/145 chain, 4 near-ties, 0 hard, fingerprints in band), and the decode bench anchors its recorded chain (33382, 3/3). A non-bf16 host takes the same paths unconditionally. - Golden re-adjudication: under the new default the torch-f32 topk fingerprint band no longer binds (worst 3.40 vs the 2.5-nat band that holds the torch transcription), so the W3, decode-bench, and spec-probe gates are tier-scoped: the f32 tier keeps every recorded anchor, the bf16-dot tier is adjudicated against the primary artifacts and carries its own re-recorded anchors (11622, 3/3 identical chains), with 0 hard chain flips. The tier split is recorded, not an assertion deletion. - test_kolibri1_spec_probe's plain-chain anchor was STALE on main (expected 109726, the f32 chain has been 33382 since the FMA-contract re-anchor; the hatch run reproduced the red). Re-anchored per tier and filed as ISSUE-LOCAL-01M5AA1RRQYAT1JG3MDSKNEYAH, closed in this change. Gates: full kolibri1 host suite green on both tiers (test_kolibri1, moe_glue, dequant, dequant_cache, numa, bfdot_gemv 12513/12513, test_ops_paged_attn_neon 15.2M assertions, spec_probe, decode bench, W3, threeway capture); check-agent-record, check-env-doc (the new knob is documented in docs/ENVIRONMENT.md), check-pr-size, arm ISA build audit all green. check-windows-portability.py is red on this host for reasons this change does not touch: the failure reproduces byte-identically on a tree without it (dotnet 10 fake-tool injection, build-windows-release.ps1 unmodified); its unit test test_check_windows_portability passes. FOLLOWING_AGENTS_PROTOCOL This change follows the Agents protocol. Work happened in the linked worktree for row/kolibri-bf16-dot-cpu on top of the committed spec, the red-line gate was measured before and after with the same binary, and every claim above names its instrument. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
lu-zero
added a commit
that referenced
this pull request
Oct 20, 2026
…were one position behind the generated chain scripts/gen-kolibri1-primary-topk-rows.py scored the teacher-forced chain at pos = len(p_ids) - 1 + k, but vLLM's prompt_logprobs[i] is the predictive distribution over full[i], so the distribution that predicts gen[k] sits at len(p_ids) + k. Every emitted row carried the previous position's distribution and the membership skip tested the wrong window, which silently dropped 54 of 256 positions. A peer measured generated[k] == rows[k+1].top1 at 195/195 on the landed fixture. The writer now uses pos = len(p_ids) + k and replaces the silent skip with two loud failures (missing distribution, top-1 CHECK) — under greedy decode gen[k] must be the argmax of its own predictive distribution, so a skip can only mean a diverged chain. The rows were re-emitted from the primary oracle (the /tmp/opencode/aaiv + /tmp/opencode/vllmsrc vLLM 0.31.0+cpu environment, tensor_parallel_size=4 mp): 256/256 positions, zero skips, top-1 CHECK green on every position, and a second run reproduced the file byte-for-byte. Impact: the threeway absolute magnitudes collapse from ~5.8 nats of phantom gap to hundredths of a nat (prod gap mean 5.806 -> -0.028, cos 0.9814 -> 0.9991, KL 7.070 -> 0.017) while every rank percentage is unchanged, so the #3469 mode ordering and the #3482 bf16-dot red line (97.0% -> 98.0%) stand. Margin instruments shift to the corrected 256-row file (top1_top2 median 3.8913 -> 3.7989; top6_top7 < 0.1-nat rows 58/202 -> 69/256). No test reads the rows file's content, so no recorded anchor changes. The llama.cpp per-position top-20 goldens (commit 62318cd57, GGUF sha256 76ddb05e458cce1a, BF16, greedy, 256/256 self-consistent) land alongside as kolibri1_llamacpp_topk_rows.json. Numbers and analysis: docs/bench-evidence/kolibri1-primary-rows-offbyone-20261020/report.md. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adopt the bf16-dot compute class as the default kolibri1 CPU numerics tier on aarch64 FEAT_BF16 hosts, per the committed spec
.agents/specs/kolibri1-bf16-dot-cpu.md(one PR carries spec + implementation, the recorded preference) and the 2026-10-19 developer ruling resolving #3438.The adoption
The #3464 arms — routed-expert/attention decode GEMV over the warm-cache bf16 weights (
kolibri1_bfdot_gemv.cpp), the paged-attention decode K dot (cpu_paged_attn_bfdot_arm.cpp, riding the NEON lane body: K on the BFDOT arm, V on the lane's widen+vfma path), and the lm_head / router-gate decode GEMV — flip from opt-in knobs to the DEFAULT wheneverDetectArmBf16()proves FEAT_BF16 at runtime. Feature-gated dispatch like vLLM's CPU kernels; no env gate on the default path.VT_CPU_KOLIBRI_BF16_DOT=0is the single escape hatch back to the f32 tier in the same binary; the legacyVT_CPU_KOLIBRI_BFDOT_GEMV/VT_CPU_PAGED_ATTN_BFDOTopt-ins keep working as explicit=1for one release. x86 and non-bf16 aarch64 keep the f32 tier as the only tier, and every arm file keeps its arch includes inside the__aarch64__/__ARM_FEATURE_BF16guards (re-audited per the #3464 incident).Before/after agreement with the primary oracle (the red line)
Same binary, both legs with the dequant cache on so only the compute class differs; 202 aligned positions against
kolibri1_primary_topk_rows.jsonviatest_kolibri1_threeway_capture+kolibri1-threeway-analysis.py:VT_CPU_KOLIBRI_BF16_DOT=0)AFTER >= BEFORE: the gate holds and moves TOWARD the primary, as the spec predicted from the three-way tie.
Bench (flipped GEMV leg, production shape)
test_kolibri1_decode_bench, 128-in / 63 greedy steps, 3-run median, same binary, contended host (ratio is the evidence): f32 14.068 s decode = 4.478 tok/s vs bf16-dot 11.948 s = 5.273 tok/s (+17.7%), consistent with the C draft's historical +21% GEMV measurement. Full instrument, environment, and per-run numbers:docs/bench-evidence/kolibri1-bf16dot-adoption-20261019/report.md.Golden re-adjudication
test_kolibri1_spec_probe's plain-chain anchor was already stale on main (expected 109726; the f32 chain has been 33382 since the FMA-contract re-anchor — the hatch run reproduced the red).Gates
Full kolibri1 host suite green on both tiers: test_kolibri1, moe_glue, dequant, dequant_cache, numa, test_kolibri1_bfdot_gemv (12513/12513, including the new default-dispatch/escape-hatch case), test_ops_paged_attn_neon (15.2M assertions, including the new default-dispatch case), test_kolibri1_spec_probe (green on BOTH tiers after the re-anchor), decode bench, W3, threeway capture.
check-agent-record,check-env-doc(the new knob is documented indocs/ENVIRONMENT.md),check-pr-size, arm ISA build audit, commit style and trailer gates green.agent-preflight.sh --staged: one red,check-windows-portability.py, which this change does not touch — the failure reproduces byte-identically on a tree without the change (host dotnet 10 fake-tool injection;build-windows-release.ps1unmodified; its unit testtest_check_windows_portabilitypasses). x86 CI lanes are expected to confirm zero behavioral change there.FOLLOWING_AGENTS_PROTOCOL
This change follows the Agents protocol: the committed spec came first, the red-line gate was measured before and after on the same binary, and the fallback tier was proven bit-identical before the default flipped.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]