[lu-zero]

The primary-oracle top-100 rows fixture stored each generated position one step behind its own predictive distribution, and every downstream absolute number built on it was a cross-position comparison.

The bug

scripts/gen-kolibri1-primary-topk-rows.py scored the teacher-forced chain at pos = len(p_ids) - 1 + k. vLLM's prompt_logprobs[i] is the predictive distribution OVER full[i], so the distribution that predicts gen[k] sits at len(p_ids) + k. Every emitted row (gen_index = k) therefore held the previous position's distribution, and the skip (if not any(tid == gid)) tested membership in that wrong window — silently dropping 54 of 256 positions. A peer measured generated[k] == rows[k+1].top1 at 195/195 positions on the landed fixture (#3463).

The fix

pos = len(p_ids) + k, the stale index comment corrected, and the silent skip replaced by two loud failures (missing distribution; a top-1 CHECK). Under greedy decode gen[k] must be the argmax of the distribution that predicted it, so the old skip path can now only mean a diverged chain, and the generator aborts instead of emitting a shifted row.

Re-emission

Reran the generator against the primary oracle in the same environment as the original run (/tmp/opencode/aaiv venv + /tmp/opencode/vllmsrc source build, vLLM 0.31.0+cpu, aleph_alpha_inference plugin, tensor_parallel_size=4 mp on this host, greedy, prompt_logprobs=100), overwriting tests/vllm/models/kolibri1_primary_topk_rows.json:

• 256/256 generated positions emitted, zero skips (old: 202/256, 54 dropped);
• top-1 CHECK green on every position: generated[k] == rows[k].top1 at 256/256 (old: 63/202);
• a second independent run reproduced the file byte-for-byte.

Corrected numbers (old lagged vs new)

kolibri1-threeway-analysis.py over the four captured dump dirs, same 202 dump positions:

mode	top-1	gap mean old	gap mean new	cos old	cos new	KL old	KL new
prod	98.0%	5.806	-0.028	0.9814	0.9991	7.070	0.017
draft	97.0%	5.836	0.002	0.9812	0.9989	6.971	0.020
before_f32	97.0%	5.842	0.008	0.9812	0.9989	6.994	0.021
after_bf16dot	98.0%	5.854	0.020	0.9804	0.9981	7.018	0.051

Margin instruments (all 8 prompts): top1_top2 median 3.8913 -> 3.7989 (p10 1.0849 -> 0.4812); top6_top7 median 0.1855 -> 0.1901, min 0.0028 -> 0.0018; rows with top6_top7 < 0.1 nats 58/202 -> 69/256. Every same-index old margin changed (0/202 identical) — no old margin label is trustworthy.

Impact on prior analyses

• #3469 (three-way numerics): the mode ORDERING stands — rank percentages are unchanged in every mode, and the corrected table keeps prod/after_bf16dot at the top with the same 98%/97% split. The absolute gap/cos/KL columns are superseded by the table above: they measured our position k against the primary's position k-1. Our production bf16-dot default's real logprob gap to the primary is -0.03 nats, cos 0.9991, KL 0.017.
• #3482 red line (bf16-dot adoption gate): STANDS. The gate was top-1 agreement computed on a consistent-but-lagged alignment; corrected alignment gives the same 97.0% -> 98.0% BEFORE -> AFTER move. The corrected absolute values still rank after_bf16dot as the loosest mode (gap +0.020 nats, KL 0.051), same conclusion.
• W3 near-tie/hard-flip classification: untouched — test_kolibri1_w3.cpp classifies flips from OUR logit gaps, not from this file. Instruments that quoted primary margins shift to the corrected column; the corrected file shows a slightly larger knife-edge tail (69 positions below 0.1 nats on top6_top7).
• No test reads the rows file's content (grep -rl primary_topk_rows tests/ finds only the path macro and manifest provenance), so no recorded anchor changes and nothing was loosened.

llama.cpp goldens

tests/vllm/models/kolibri1_llamacpp_topk_rows.json lands with this change: 8 prompts x 32 positions of per-position token-ids + top-20 logprobs from llama.cpp commit 62318cd57, GGUF sha256 76ddb05e458cce1a023d42cd4f13dc8e8152fbb6501c1f680918fc7d2a0fcefd, BF16, greedy; 256/256 rows with top-1 == emitted token.

Gates

check-agent-record, staged preflight (style suites, trailer suites, record gates) green. The single preflight failure, check-windows-portability.py, is environmental on this host (dotnet/pwsh invocation) and fails identically on pristine origin/main 61d333e52; this change touches no Windows surface. Full numbers and method: docs/bench-evidence/kolibri1-primary-rows-offbyone-20261020/report.md. Local issue ISSUE-LOCAL-01M5CQ4GMFF9ZAYYMFYDV6WZ3Q closed with the dated resolution.

The change was produced under the repository agent protocol; the human submitter reviewed and owns it.

NOTE ON GOLDENS-CHAIN OVERLAP: the re-emitted rows' generated_ids are byte-identical to the old file's, and they do NOT match the committed kolibri1_goldens.json chains on every prompt (3/8 exact, 126/256 positions). That is pre-existing and expected: the rows are emitted from the primary oracle serving Kolibri-1-bf16-clean through vLLM, while the goldens are the pure-torch reference over Kolibri-1-bf16 — different artifacts, same weights. The near-tie re-rank shape of the differences is documented in the three-way analysis; nothing in this PR changes it.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]