From 6fee3fcfc18a0057f5946d10a69024ac81c33bd2 Mon Sep 17 00:00:00 2001 From: Luca Barbato Date: Tue, 20 Oct 2026 15:46:26 +0200 Subject: [PATCH] kolibri1: cross-check the CPU gates against llama.cpp 62318cd57 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add upstream llama.cpp's freshly-merged Kolibri-1 support (#30673, merge 62318cd57, 2026-10-17) as a third independent reference for the row's correctness record, and record the three-way agreement measurement. No product code changes: this is the oracle pin, its gateability evidence, and one local issue. The pin advances from 11fe0215 to 62318cd57 because that revision is the first upstream object that loads the released Aleph-Alpha/Kolibri-1 checkpoint at all. The gateability run built CPU-only from a clean tree at the exact commit (GGML_NATIVE=ON, aarch64 FEAT_BF16, the same hardware class as our production bf16-dot default), converted the fp8 checkpoint through the upstream converter to a bf16 GGUF (903 tensors, 149069.2 MiB, sha256 76ddb05e458cce1a023d42cd4f13dc8e8152fbb6501c1f680918fc7d2a0fcefd), and generated coherently end-to-end. The measurement: over the primary oracle's 202 aligned teacher-forced positions, llama.cpp top-1 is 198/202 = 98.0% — identical to our committed production bf16-dot number. Chain decode vs the golden chains is 126/256 = 49.2% with 3 of 8 prompts token-exact, the near-tie re-rank shape our own chains show; the aligned numbers are the numerics statistic, the chain number is not a quality verdict. Nothing was tuned on either side. Closes .agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ ISSUE-LOCAL-01M5CRZ3J4CFX1KEJD6ZRE4A8R. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:zai/glm-5.3-flash [maki] --- .../ISSUE-LOCAL-01M5CRZ3J4CFX1KEJD6ZRE4A8R.md | 19 +++ .agents/oracles/llama-cpp.md | 22 +++- .../kolibri1-llamacpp-crosscheck-20261020.md | 114 ++++++++++++++++++ 3 files changed, 151 insertions(+), 4 deletions(-) create mode 100644 .agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5CRZ3J4CFX1KEJD6ZRE4A8R.md create mode 100644 docs/bench-evidence/kolibri1-llamacpp-crosscheck-20261020.md diff --git a/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5CRZ3J4CFX1KEJD6ZRE4A8R.md b/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5CRZ3J4CFX1KEJD6ZRE4A8R.md new file mode 100644 index 0000000000..a94bd51283 --- /dev/null +++ b/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M5CRZ3J4CFX1KEJD6ZRE4A8R.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M5CRZ3J4CFX1KEJD6ZRE4A8R +Title: Cross-check kolibri1 CPU gates against upstream llama.cpp 62318cd57 +Row: MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm +State: CLOSED +Kind: task +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-10-20 +Updated: 2026-10-20 +Closed: 2026-10-20 + +## Problem + +Our kolibri1 correctness record has two references (primary aleph-alpha-inference, torch transcription). Upstream llama.cpp merged Kolibri-1 support (62318cd57, #30673, 2026-10-17); the records need a third independent reference: pin it, gate it on the released checkpoint, and measure three-way agreement. No product code changes. + +## Resolution + +2026-10-20: llama.cpp 62318cd57 built CPU (clean tree), converted /mnt/models/Aleph-Alpha/Kolibri-1 fp8 to bf16 GGUF (903 tensors, 149069.2 MiB, sha256 76ddb05e458cce1a023d42cd4f13dc8e8152fbb6501c1f680918fc7d2a0fcefd), served and generated end-to-end (gateable). Three-way: llama.cpp top-1 vs primary 198/202 = 98.0% over the 202 aligned positions, identical to our committed production bf16-dot number; llama.cpp chain vs golden 126/256 = 49.2% (3/8 token-exact, near-tie re-ranks). Recorded in docs/bench-evidence/kolibri1-llamacpp-crosscheck-20261020.md; oracle pin advanced to 62318cd57 in .agents/oracles/llama-cpp.md. diff --git a/.agents/oracles/llama-cpp.md b/.agents/oracles/llama-cpp.md index 8d40a38e8d..66fabedf18 100644 --- a/.agents/oracles/llama-cpp.md +++ b/.agents/oracles/llama-cpp.md @@ -164,13 +164,27 @@ id = llama-cpp role = secondary upstream = https://github.com/ggml-org/llama.cpp scope = CPU and GGUF k-quant speed and memory floors, quant-matched against the same weights -pin = 11fe02151f79c41d0d4af7da708755d73b9c0da6 -pin_label = llama.cpp main, 2026-10-04 -pinned_on = 2026-10-03 +pin = 62318cd57dc63900ecb153272c0b10f78550855d +pin_label = llama.cpp main, 2026-10-17 (kolibri-1 merge 62318cd57) +pinned_on = 2026-10-20 gateable = yes -evidence = docs/bench-evidence/oracle-llamacpp-11fe0215-gateable-20261003.md +evidence = docs/bench-evidence/kolibri1-llamacpp-crosscheck-20261020.md ``` +**Prior pin: `11fe02151f79c41d0d4af7da708755d73b9c0da6` (llama.cpp main, +2026-10-04), pinned 2026-10-03, gateable evidence +[oracle-llamacpp-11fe0215-gateable-20261003.md](../../docs/bench-evidence/oracle-llamacpp-11fe0215-gateable-20261003.md).** +Every gate recorded against `11fe0215` stays attributed to it. The pin advanced +on 2026-10-20 because `11fe0215` predates the Kolibri-1 (Aleph-Alpha) support +merge `62318cd57` (#30673, 2026-10-17): the kolibri-1 three-way cross-check +(row `MODEL-TEXT-kolibri-1`) needs a third independent reference and that +revision is the first upstream object that loads the released checkpoint at +all. `62318cd57` was verified gateable on the Aleph-Alpha/Kolibri-1 fp8 +checkpoint converted to bf16 GGUF through the upstream converter; identity, +build, executed path and agreement numbers are in the evidence file above. +Non-kolibri denominators recorded against `11fe0215` keep that revision in +their own records; this advancement does not re-base them. + **Prior pin: `10bf611e533d81f739128304991c5e133c6aebd8` (`b10451`), pinned 2026-08-16, gateable evidence [oracle-llamacpp-b10451-gateable-20260822.md](../../docs/bench-evidence/oracle-llamacpp-b10451-gateable-20260822.md).** diff --git a/docs/bench-evidence/kolibri1-llamacpp-crosscheck-20261020.md b/docs/bench-evidence/kolibri1-llamacpp-crosscheck-20261020.md new file mode 100644 index 0000000000..89006511cb --- /dev/null +++ b/docs/bench-evidence/kolibri1-llamacpp-crosscheck-20261020.md @@ -0,0 +1,114 @@ +# Kolibri-1 llama.cpp cross-check: third independent reference vs the primary oracle (2026-10-20) + +Row: MODEL-TEXT-kolibri-1 (`kolibri1-for-causal-lm`). Branch +`row/KOLIBRI-LLAMACPP-CROSSCHECK` (fresh from `origin/main` `61d333e52`). +Developer-approved task: add upstream llama.cpp's freshly-merged Kolibri-1 +support as a third independent reference and cross-check our CPU gates +against it. Measurement only — no vllm.cpp product code changes. + +## What landed upstream + +llama.cpp merged Kolibri-1 (Aleph-Alpha) support in `62318cd57` +(#30673, 2026-10-17): arch `kolibri1`, converter `Kolibri1Model(AfmoeModel)` +(`conversion/kolibri1.py`), the released FP8 checkpoint goes through the +generic `weight_scale_inv` dequant path (`conversion/base.py`, `quant_method +== "fp8"`). Upstream follow-up #30865 (router precision overrides) is AFTER +the pin and is NOT in this measurement. + +## Pin and build identity + +- Oracle revision: `62318cd57dc63900ecb153272c0b10f78550855d` (llama.cpp main). +- Source tree: `/home/lu_zero/Sources/llama.cpp`, `git status --porcelain` + EMPTY at the pinned commit. Read-only against ggml-org; the PR head ref + (`pull/30673/head` = `566df8ce4`) was fetched only to locate the merge. +- Build: CPU-only, `cmake -DGGML_NATIVE=ON -DGGML_CUDA=OFF -DGGML_VULKAN=OFF`, + GNU 16.2.0, Linux aarch64 (Ampere-1a, FEAT_BF16/dotprod — the same hardware + class as our production bf16-dot default). `version: 0.7.0-dev (build 11709, + commit 62318cd57)`. + - `llama-cli` sha256 `05d0f29c20da702b482646d27b804b9cad5b9c315d311999dd5bc3f5a4a3295a` + - `llama-server` sha256 `fdff524b31d6e6b8d2462d5c78229ad4421e463ad95c9aedaa880216e1322afc` +- Executed path: `llama-server -t 16`, all defaults otherwise (repack ON, + flash-attention AUTO, no `-nr`/`-ub` overrides). Per the oracle record's + path-pinning rule this is the path that produced every number below. + +## The GGUF artifact + +Produced with the pin's own converter, FP8 through the native path: + +- Command: `convert_hf_to_gguf.py /mnt/models/Aleph-Alpha/Kolibri-1 + --outfile /tmp/kolibri1-llamacpp-auto.gguf` (default `--outtype auto`). +- Result: 903 tensors, 156.3 GiB logical, `general.architecture = kolibri1`, + `general.file_type = 32` (`MOSTLY_BF16`), `expert_count = 384`, + `expert_shared_count = 1`, sigmoid gating. +- File: `/tmp/kolibri1-llamacpp-auto.gguf`, 149,069.2 MiB on disk, + sha256 `76ddb05e458cce1a023d42cd4f13dc8e8152fbb6501c1f680918fc7d2a0fcefd`. + Not committed; re-derivable from the command above. + +The runtime LOADED the artifact and generated coherent text end-to-end on all +eight prompts: this oracle pin is gateable for kolibri-1. + +## Method + +The primary oracle's committed fixture +(`tests/vllm/models/kolibri1_primary_topk_rows.json`, 202 aligned positions +over the 8-prompt battery, per-prompt scored counts 32/30/24/20/30/20/14/32) +is the reference. Two measurements through llama.cpp: + +1. **Aligned-position top-1** (the same statistic as our committed 98.0% + number). PROVISIONAL LABEL: the primary top-100 rows on main carry a + confirmed off-by-one (the writer stored each row one position behind the + generated chain; repair in flight on row/PRIMARY-ROWS-OFFBYONE). Both + sides of this comparison used the same lagged alignment, so the equality + of the two percentages below is the meaningful statistic — two + independent implementations agreeing exactly — while the absolute 98.0% + label is provisional until the repaired rows land: for each aligned position k, the prompt ids plus the primary's + generated ids `0..k-1` were sent as a token-array completion with + `n_predict=1, temperature=0`, and the sampled id was compared with the + primary's emitted token. Both sides therefore score position k under the + IDENTICAL prefix — a numerics distance, not a chain artifact. +2. **Chain decode**: the prompt ids alone, `n_predict=32, temperature=0`, + compared position-wise with the golden chains + (`kolibri1_goldens.json` `generated_ids`, the torch-transcription / + primary chains). + +Raw results: `/tmp/kolibri-xcheck/llamacpp_results.json` (not committed; the +tables below are the record). + +## Three-way agreement + +| side vs PRIMARY emitted token | aligned positions | top-1 agreement | +|---|---|---| +| OURS — production bf16-dot default (committed, `docs/bench-evidence/kolibri1-bf16dot-adoption-20261019/report.md`) | 202 | **198/202 = 98.0%** | +| llama.cpp `62318cd57`, bf16 GGUF (this run) | 202 | **198/202 = 98.0%** | + +Both independent implementations miss the same four-or-fewer knife-edge +positions; ours and llama.cpp agree with each other at the reference at +least wherever both match it. + +| llama.cpp chain vs golden chain | match | first divergence per prompt (0-7) | +|---|---|---| +| 126/256 = 49.2% | 3 of 8 prompts token-exact through 32 steps | 0, 1, —, 0, —, 5, 7, — | + +The chain number is LOW while the aligned number is HIGH — the same shape our +own chain runs show: single-token near-ties at small logit margins re-rank, +and once a chain diverges the later positions compare different contexts and +are meaningless as a numerics statistic. Do not read the 49.2% as model +quality; read the 202-position teacher-forced numbers. + +llama.cpp's four aligned mismatches: (p0, k0, primary 116026 → 121598), +(p1, k1, 35423 → 46252), (p3, k3, 2042 → 1584), (p6, k16, 13111 → 47354). +No tuning was applied to either side; this is a measurement. + +## What this changes + +`.agents/oracles/llama-cpp.md` advances its pin from `11fe0215` to +`62318cd57` with this file as the gateability evidence for the kolibri-1 +artifact. `docs/USAGE.md` owes no line: the kolibri section's promises +(commands, workflow, gate provenance) did not change, and this run adds no +command, config key, or workflow. + +## Not measured here + +Speed and memory floors vs this oracle remain owed by the oracle record's +re-take debt; this run pins correctness only. Upstream #30865 (router +precision overrides removal) is newer than the pin and unaudited here.