benchmark: public competitive evidence and qualification release gate

[luke-n-alpha]

Goal

Produce independently defensible, multilingual competitive evidence for naia-memory and keep publication fail-closed until the exact benchmark artifacts, authorization chain, deployment trust anchor, and signed qualification can be reproduced and verified.

Public-ready acceptance criteria

• Competitive claim is based on independently authored/native-reviewed held-out data, not generated diagnostics alone.
• Naia, Mem0, Graphiti, Hindsight, and any other declared comparator run under a frozen, engine-neutral contract with equivalent model/runtime disclosure.
• Korean and multilingual results disclose per-language relevance, contradiction/forbidden-result rates, latency, cost, and uncertainty.
• Dataset custody, preregistration, participant delivery, execution authorization, blind adjudication, and non-peeking constraints verify end to end.
• A release-owned public deployment policy provisions a static Ed25519 trust anchor without committing private keys.
• A signed competitive qualification binds all required artifacts and passes the canonical public gate from a clean checkout.
• Independent adversarial review examines benchmark overfitting, Naia-specific metric advantage, comparator parity, and claim scope.
• Final Korean/English public report separates demonstrated results, limitations, and unproven claims.

Existing evidence and history

Earlier benchmark/qualification progress was accidentally posted to #39, whose actual scope is conflict-aware structured facts. Those comments remain immutable historical records but are not treated as the canonical tracker:

• feat(memory): conflict-aware structured facts with multilingual-safe retrieval #39 comment 5388427700 — v5 manifest qualification sealing
• feat(memory): conflict-aware structured facts with multilingual-safe retrieval #39 comment 5388454998 — 37-artifact fail-closed integration
• feat(memory): conflict-aware structured facts with multilingual-safe retrieval #39 comment 5388505109 — pinned qualification success path
• feat(memory): conflict-aware structured facts with multilingual-safe retrieval #39 comment 5388578300 — secret-free trust-anchor provisioning

Relevant pushed commits include 094ba5d, 7392980, a9f00b9, and 6ce69b1.

Current status (2026-08-24)

The public gate and provisioning machinery are materially stronger, but the repository remains intentionally not public-qualified: the committed deployment anchor is null, no release-owned policy/key has been provisioned, and the final independent multilingual comparator campaign has not been sealed against that exact trust-store digest.

GPU1 is out of scope and must not be used.

[luke-n-alpha]

Qualification anchor provenance stage pushed: 0210a2f (feat(benchmark): verify qualification anchor provenance).

What changed:

• CLI now supports atomic --output provisioning and exact-byte --check drift verification against the deployment-owned public policy.
• Missing/unreadable/drifted modules fail closed.
• Output cannot target the policy itself or a symbolic-link alias.
• Regression coverage proves unrecognized private material is not rendered; the trust-store projects only allowlisted public fields.

Validation:

• Biome: pass
• focused qualification/public-gate suite: 5 files, 22 tests pass
• benchmark TypeScript check: pass
• git diff --check: pass
• adversarial review: initial REJECT (path alias/private leakage concerns), fixes applied, final OpenCode review ACCEPT. A second independent OpenCode run was inconclusive because it wandered into unrelated full-suite failures; it is not counted as ACCEPT.
• broad quality suite remains non-clean due pre-existing/concurrent MIRACL true-batch source-hash mismatch (2 files / 8 tests), outside this stage and not used to claim cleanliness.

Public release status remains NOT QUALIFIED: no release-owned production policy/key or final signed clean-checkout campaign exists yet. GPU1 was not used.

[luke-n-alpha]

Stage update — clean-checkout qualification release gate

Implemented and pushed c4e6431 (feat(benchmark): gate clean-checkout qualification release).

• Added pnpm benchmark:semantic-qualification-release.
• The command reads a bounded public deployment policy, derives its qualification trust anchor, and requires an exact deep match with the checkout-provisioned static anchor before delegating to the existing sealed-manifest public gate.
• Missing anchor, unreadable/invalid policy, policy drift, ambiguous invocation, or downstream gate failure remain fail-closed (promotable: false).
• Removed test dependency injection from the public CLI runner so callers cannot substitute a fake anchor or fake successful sealed-gate runner.
• Validation: Biome clean; 22 focused tests pass; benchmark TypeScript compile passes; git diff --check passes. GPU1 was not used.

Adversarial-review record:

• First OpenCode response was discarded because untracked new files were omitted from its diff view.
• Final staged diff was then fully read by OpenCode, but the provider hit a rate limit before verdict.
• A second OpenCode model produced no verdict before the 180s timeout.
• Therefore this stage does not claim formal review-pass CLEAN; reviewer status is inconclusive/NOT_RUN.
• Formal deterministic preflight also remains blocked by the pre-existing unrelated untracked .cache/tools/trec_eval-ba38899/ path; it was not modified.

Qualification status: NOT QUALIFIED. This stage supplies release-path integrity only. A release-owned production public policy/static anchor plus the final independently executed, signed and timestamped clean-checkout campaign are still required before any public competitive claim.

[luke-n-alpha]

Stage update — provisioned release wiring evidence

Pushed 73b575e (test(benchmark): verify provisioned release wiring).

This closes the test-only gap left by c4e6431 without adding any runtime dependency-injection or environment override:

• module-isolated integration test proves an exact static-anchor/policy match delegates exactly the five sealed-manifest arguments;
• drifted static anchor proves zero downstream calls and asserts the exact fail-closed reason;
• downstream sealed-gate rejection code is propagated rather than converted into success;
• default wrapper anchor is asserted to be the exact generated-module export;
• mocked anchor remains bound to SemanticQualificationTrustAnchor | null.

Validation: Biome clean; 2 focused files / 8 tests pass; benchmark TypeScript compile passes; diff checks pass. GPU1 was not used.

Adversarial review: Claude headless returned ACCEPT on the initial staged test and identified four non-blocking assurance gaps. All four were addressed. A required second post-fix review returned no verdict before 180s timeout, so this stage records one ACCEPT plus one inconclusive timeout and does not claim formal two-round review-pass CLEAN.

Qualification remains NOT QUALIFIED. The tests prove release wiring only; they do not substitute for a release-owned policy/key or independently executed signed multilingual comparator campaign.

[luke-n-alpha]

[naia-memory] benchmark evidence update

• Pushed 056ad7e (test(benchmark): enforce execution configuration parity) to feat/memory-bench-harness-and-fixes.
• Competitive-candidate v5 evidence now fails closed on repetition drift, unsafe endpoints, missing runtime disclosures, direct-comparator provider asymmetry, mutable server identities, and Graphiti current/historical runtime mismatch.
• Verification: 4 test files / 33 tests passed; tsc --noEmit and staged diff check passed.
• Adversarial review: one OpenCode/DeepSeek round returned CLEAN; additional OpenCode and Claude rounds independently reran type/full-path checks but timed out before a final verdict. Formal review-pass remains NOT_CLEAN because the repository contains a pre-existing untracked .cache/tools/trec_eval-ba38899/; it was not modified or removed.
• Release status remains NOT QUALIFIED: no final anchored, independently reproduced multilingual campaign exists yet.

Next: cryptographically bind configurationSha256 to the disclosed runtime/provider configuration, then rerun adversarial and deterministic gates.

[luke-n-alpha]

[naia-memory] configuration receipt binding complete

• Pushed 0cb9c9d (test(benchmark): bind receipts to disclosed configuration).
• v5 competitive execution receipts now require each signed configurationSha256 to equal the canonical hash of that engine's raw disclosed configuration; only per-run executionSeed is excluded.
• The binding chain is now: raw disclosure -> artifact SHA-256 -> campaign SHA-256 -> signed engine receipt. Naia selection policy must resolve to the same configuration hash.
• Added a valid-signature/mismatched-configuration negative path and an all-six-engine positive path.
• Verification: 4 files / 36 tests passed; both TypeScript project typechecks passed; staged diff check passed.
• Adversarial review: two consecutive OpenCode/DeepSeek reviews returned CLEAN on the same staged snapshot. Formal review-pass repository preflight remains NOT_CLEAN solely due to the pre-existing untracked .cache/tools/trec_eval-ba38899/, which was not touched.
• Public release status remains NOT QUALIFIED pending an anchored, independent, multilingual final campaign.

[luke-n-alpha]

[naia-memory] Hindsight runtime evidence hardening completed and pushed.\n\n- Commit: 10372da (fix(benchmark): bind Hindsight embedding runtime evidence)\n- Branch: feat/memory-bench-harness-and-fixes\n- Hindsight competitive receipts now fail closed unless engine image digest, LLM provider/model, and embedding provider/model/revision/dimensions are disclosed.\n- Shared Naia/Mem0 provider paths now require an explicit BENCHMARK_EMBEDDING_REVISION on custom OpenAI-compatible, gateway, and direct Gemini paths; model-name fallback was removed.\n- Archived disclosure validation rejects missing/blank embedding identity and non-positive dimensions.\n- Validation: Biome clean; 4 files / 39 tests passed; TypeScript typecheck passed.\n- Adversarial review: Claude headless found and drove three test/evidence hardening rounds; final diff then received two consecutive independent CLEAN verdicts.\n- Formal review-pass preflight remains externally NOT_CLEAN only because of the pre-existing untracked .cache/tools/trec_eval-ba38899/ path, which was preserved and not staged.\n\nPublic competitive status remains NOT QUALIFIED. This closes a reproducibility/evidence gap; it does not create a new performance claim. Next bottleneck: Letta embedding runtime identity/revision parity.

[luke-n-alpha]

Letta runtime-evidence hardening completed and pushed.

• Commit: 2334163 (fix(benchmark): bind Letta embedding runtime evidence)
• Competitive Letta receipts now fail closed unless embedding provider, model, operator-asserted revision, and positive dimensions are disclosed alongside engine version, immutable image digest, and LLM model.
• Archived evidence validation independently requires the same fields.
• Scope remains competitive-candidate only; diagnostic campaigns are unchanged.
• Validation: Biome clean; related tests 30/30 passed; benchmark + project TypeScript checks passed.
• Adversarial review: two independent Claude headless reviews, both VERDICT: CLEAN.
• Semantics: Letta’s official model API exposes provider/model/dimensions but no universal immutable embedding revision, so LETTA_EMBEDDING_REVISION is recorded as an explicit operator assertion, not claimed as cryptographically immutable. Image digest remains the immutable runtime anchor.

Public status remains NOT QUALIFIED. This closes an evidence-integrity gap; it does not create a competitive-performance claim.

[luke-n-alpha]

[naia memory] Graphiti comparator evidence hardening completed (scoped review; public gate remains NOT QUALIFIED).\n\n- Added live contacted-runtime binding for graphiti-core, Neo4j driver, google-genai adapter, exact Gemini client/embedder classes, models/dimensions, deployed sidecar hash, and server lock hash.\n- Identity is checked before and after execution.\n- Adversarial review found a real downstream gap: campaign validation trusted configurationAuthority without reconstructing nested contacted-runtime checks. Fixed; forged/missing/drifted nested identities now fail closed.\n- Corrected unchanged tree received two independent scoped CLEAN conclusions.\n- Verification: 62/62 selected tests; typecheck/Biome/py_compile/diff-check pass. Clean clone at Graphiti 993e081... applied both patches and uv lock/sync, observing graphiti-core 0.28.2, neo4j 5.28.1, google-genai 1.62.0.\n- Honest boundary: Git revision, image digest, and provider-side embedding revision remain operator-attested. This improves comparator evidence integrity, not benchmark performance.\n- Repository-wide formal review remains NOT_CLEAN because unrelated user-owned .cache/tools/trec_eval-ba38899/ fails global preflight; preserved untouched.\n\nResearch note: reports/quality/graphiti-contacted-runtime-evidence-review-2026-08-24.md

[luke-n-alpha]

[English primary gate adversarial review — 2026-08-24]

Status: NOT AUTHORIZED / NOT PUBLIC-CLAIM ELIGIBLE. opencode returned ACCEPT_WITH_CHANGES; Claude returned REJECT because execution provenance was not yet closed.

Implemented remediation (independent of the source-frozen Arabic CLI):

• canonical raw ranking + qrels digest is now recomputed and checked; fixed seeded 64-query diagnostic
• vector/passages artifact digest is recomputed and checked
• frozen execution-policy digest pinned: d2d6e0c505dbe11ff8e34999a7ddfff02f82ef977f739e7884153d63771e5856
• active Arabic Qdrant port excluded; English collection identity isolated by source+policy digest
• public/equivalence/noninferiority/throughput/multilingual-transfer/SOTA eligibility all explicitly false
• duplicate sample identities rejected

Verification: focused Vitest 10/10 PASS; TypeScript noEmit PASS; Biome PASS.

Remaining blockers before authorization: source-lock-derived corpus extraction receipt, direct topics/qrels byte verification, producer source-set closure, live Qdrant verification, checkpoint/attempt ledger, positive authorization tests, and mandatory evaluation CLI wiring after Arabic evidence closure.

Arabic exact run remains CPU-only and untouched; GPU1 is not used.

Report: reports/quality/miracl-en-primary-execution-preregistration-2026-08-24.md

[luke-n-alpha]

[2026-08-24 진행 업데이트] MIRACL-en primary 사전등록/증거 바인딩 구현

커밋: a390de7 (bench: preregister MIRACL English primary run)

완료:

• 영어 전체 코퍼스 primary 실행 정책과 1회 실행/전 결과 공개 경계를 사전등록
• 소스 잠금 영수증에서 직접 8,192 passage(8개 byte-length strata × 1,024)와 64 query를 결정론적으로 선택
• passage/qrels/query identity를 canonical SHA-256으로 preflight와 authorization에 결속
• 임의 해시 문자열 대신 실행 진입점+정적 import closure의 source manifest를 증거에 포함
• 승인 파일 검증 시 source manifest를 현재 repository bytes와 재계산하여 stale/forged producer 차단
• Qdrant port 6344, CPU-only, 고정 output/checkpoint, overwrite 금지 강제
• sampler bounded insertion/early reject로 full-corpus sampling 메모리 상한 유지

검증:

• focused 3 test files / 16 tests PASS
• tsc -p tsconfig.benchmark.json --noEmit PASS
• 위조 producer manifest, qrels digest drift, strata 위조, 비정규 evidence 거부 테스트 포함

적대리뷰:

• 앞선 Claude 리뷰의 유효 지적(층별 개수, canonical comparator, Qdrant positive pin, temp ownership, qrels binding, sampler complexity)을 반영
• 이번 Claude headless 재리뷰는 2분 이상 무응답이라 판정 근거에서 제외
• OpenCode 무료 모델 리뷰는 포트/조건문을 오독하여 신뢰 불가로 분류

현재 공개 판정: NOT QUALIFIED. 이 커밋은 결과가 아니라 영어 실험의 score-shopping/출처변조를 막는 실행 게이트다.

다음 차단 요소:

1. source-derived sample을 실제로 소비해 per-item/batch embedding 및 retrieval raw artifact를 생성하는 preflight CLI 완성
2. preflight PASS 후 고정된 MIRACL-en full-corpus primary 1회 실행
3. 진행 중 MIRACL-ar 전체 코퍼스 실행 완료 및 ko/en/ar 비교
4. Graphiti/Hindsight 등 경쟁군 독립 재현과 held-out/외부 benchmark까지 통과 후 공개 경쟁력 보고서 작성

Arabic 상태(기록 시점): Qdrant green, CPU-only, 1,891,840 / 2,061,414 points. GPU1 미사용.

[luke-n-alpha]

English primary preflight hardening update (2026-08-24)

• Commit: ffcd1a4 (bench: bind English retrieval corpus relevance)
• Closed a false-confidence path: a random 8,192-passage sample could contain no selected qrel documents, allowing both modes to agree at NDCG=0.
• Retrieval corpus is now the deterministic union of every selected-qrel document and the preregistered 8 x 1,024 length-stratified sample, deduplicated by docid and ordered by corpus ordinal.
• Receipt now binds the explicit sorted relevant docid set, its digest, the complete retrieval-corpus passage digest, and rejects missing/changed stratified passages.
• Execution authorization binds preflight retrieval evidence to that corpus digest and recomputes selected relevant-docid identity from locked qrels.
• Verification: 18 focused tests PASS; benchmark TypeScript noEmit PASS; Biome PASS; git diff check PASS.
• Adversarial review: Claude headless and OpenCode headless both stalled without findings and are NOT counted as review evidence. Independent fail-open inspection found and fixed the missing-negative validation gap. Formal cross-review remains pending tool availability.
• Arabic full-corpus CPU run remains healthy: 1,918,976 / 2,061,414 points (~93.09%), Qdrant green/optimizer ok. GPU1 is not used.

Public status remains NOT QUALIFIED. Next: implement the executable English preflight while avoiding oversized JSON vector artifacts, then run it only after artifact/provenance contracts and resource isolation are validated.

[luke-n-alpha]

English primary gate execution update (2026-08-24 KST)

• Implemented a CPU-only MIRACL-en preflight runner for the frozen per-item-v1 vs padded-array-batch-v1 contract.
• Evidence integrity: flat Float32 binary artifact, canonical passage digest binding, complete-write loop, immutable hard-link publication, actual on-disk SHA-256 verification, and recoverable vector-first/evidence-last publication.
• Retrieval integrity: exact cosine ranking extracted into a tested pure function with deterministic ordinal/docid tie-breaking; query/corpus dimension, finite-value, and nonzero-norm checks fail closed.
• Deterministic validation: 3 focused test files PASS, 20 tests total; TypeScript noEmit PASS; Biome PASS; git diff --check PASS.
• Adversarial review status: OpenCode and Claude headless attempts both returned no final verdict within 90s, therefore recorded as NOT_RUN (not claimed as CLEAN). Formal review-pass preflight remains NOT_CLEAN solely because of unrelated user-owned .cache/tools/trec_eval-ba38899/; no push/publication.
• MIRACL-ar full-corpus CPU run remains healthy: 1,942,016 / 2,061,414 points (94.21%), Qdrant green, evaluator PID 409189 CPU-bound. Our evaluator PIDs are absent from nvidia-smi; GPU1 is not used.
• Public competitive status remains NOT QUALIFIED. Next gate: finish Arabic run, preserve its receipts, complete a valid adversarial review of the English runner, then execute the preregistered English preflight without overlapping the heavy Arabic CPU run.

[luke-n-alpha]

Follow-up evidence-chain hardening (2026-08-24 KST)

Adversarial inspection found an authorization gap: the English execution authorizer previously trusted the vector digest stated inside preflight JSON without re-hashing the actual .vectors.f32 artifact. Fixed locally:

• authorization now streams the artifact in 1 MiB chunks and requires its SHA-256 to equal the digest sealed in canonical preflight evidence;
• authorization receipt includes the independently recomputed vector artifact digest;
• added regression coverage proving exact bytes pass and tampered bytes fail;
• focused gate is now 21/21 tests PASS, TypeScript noEmit PASS, Biome PASS.

OpenCode/Claude headless attempts still produced no final verdict within their 90s limits, so external adversarial review remains honestly NOT_RUN. Formal review-pass remains NOT_CLEAN due only to unrelated user-owned .cache/tools/trec_eval-ba38899/; changes remain unpushed. Public status remains NOT QUALIFIED.

[luke-n-alpha]

English primary preflight hardening update (2026-08-24 KST)

Implemented and independently revalidated in the intended MIRACL English preflight scope:

• fail-closed exact cosine now rejects non-finite/zero norms, non-finite dot products/scores, and avoids avoidable intermediate overflow by multiplying separately computed square roots;
• added regression coverage for finite inputs whose naïve norm-product overflows, plus zero/overflow rejection;
• vector artifact publication now completes partial writes, fsyncs the temporary file before immutable hard-link publication, and fsyncs the containing directory after publication;
• evidence JSON publication now has the same file+directory durability barrier;
• primary authorization streams and hashes the actual .vectors.f32 bytes and binds the independently recomputed digest into the authorization receipt; byte tampering fails closed.

Deterministic evidence:

• focused MIRACL English suite: 4 files / 22 tests PASS;
• tsc --noEmit -p tsconfig.benchmark.json: PASS;
• Biome on intended files: PASS;
• git diff --check on intended files: PASS.

Adversarial review status:

• OpenCode muse-spark read the exact five-file scope but did not return a verdict inside the 90-second bound: external pass is NOT_RUN, not CLEAN.
• Internal adversarial review found and fixed the norm-product overflow and crash-durability gaps above.
• Formal review-pass deterministic preflight remains NOT_CLEAN because the complete changed-set gate rejects user-owned untracked .cache/tools/trec_eval-ba38899/. It was not deleted or modified; no remote push is claimed.

Arabic full-corpus control remains CPU-only and healthy:

• Qdrant green, 1,966,592 / 2,061,414 points (~95.40%);
• evaluator PIDs remain absent from nvidia-smi; GPU1 is not used.

Public competitive status remains NOT QUALIFIED. Next gate: finish Arabic full corpus and seal receipts, then obtain a completed isolated adversarial verdict for the English preflight/authorization before authorizing the English primary run.