[chenncy]

Which issue or RFC does this PR close?

Refs #1977.

Depends on #2004 for the versioned Source evidence declarations and exact historical evidence-read API required by #1975.

Rationale for this change

Atomic Memory candidate pools can be dominated by repeated representations or shared lineage. Before proposing retrieval or lifecycle changes, we need reproducible measurements that distinguish duplication from legitimate facts sharing the same evidence.

This adds a bounded evaluation that measures candidate-pool concentration and final PreparedContext fact coverage without changing production behavior.

What changes are included in this PR?

• Add lineage-fixture and lineage-compare commands to the existing LongMemEval-V2 evaluation package.
• Group exact Scope-local Artifact and Source evidence conservatively, keeping missing evidence explicitly unknown.
• Add versioned fixtures covering identical entries, paraphrases, legitimate shared-source facts, revisions, historical evidence, recurring automation, unknown evidence, and CJK/English query controls.
• Compare the baseline with an in-memory exact-equality measurement projection and deterministic lexical-neighbor annotations.
• Report concentration, duplicate occupancy, evidence/fact coverage, recall, false suppression, task regressions, budget omissions, latency, and model cost separately.
• Generate pinned manifests, aggregate-only JSON/Markdown reports, a backend capability matrix, reproducible commands, artifact hashes, and a decision record.
• Add focused tests and English/Chinese documentation.

Are there any user-facing changes?

Two evaluation CLI commands are added.

Production ranking, Memory writes, lifecycle state, citations, public HTTP/MCP schemas, Runtime events, and PreparedContext delivery behavior are unchanged.

The current recommendation is no product change. Equality projection improved the concentration fixture, while lexical annotations produced 26 false-relation pairs across query samples. These bounded synthetic results do not justify production deduplication or reranking.

How was this change tested?

Validation used the exact evidence-capable Runtime snapshot from #2004:

25d9e7a7d3514a7110761c472f0efb947b2e5826

• Full LongMemEval-V2 evaluation test suite: 243 passed, 1 skipped.
• Focused compatibility checks against current main: 16 passed, 1 skipped.
• Real SQLite fixture execution: 9 isolated Scopes, 18 queries, 3 evaluation arms, and 2 held-out partitions.
• No model calls; model cost USD 0.
• Aggregate report privacy, source digests, manifest compatibility, and artifact hashes verified.
• Ruff lint and formatting checks passed.
• Changed evaluation source type checks and repository-wide Linux-target static type checks passed.
• Repository-wide pre-commit checks passed except for existing Windows type errors involving POSIX-only APIs in unchanged files.
• git diff --check passed.

Test commands:

• python -m pytest -c evaluation/pyproject.toml evaluation/memory/longmemeval_v2/tests -q
• ruff check evaluation/memory/longmemeval_v2/src evaluation/memory/longmemeval_v2/tests
• ruff format --check evaluation/memory/longmemeval_v2/src evaluation/memory/longmemeval_v2/tests

Limitations:

• The evidence-capable validation uses an unmerged dependency snapshot; validation should be rerun after feat(memory): persist versioned Source evidence and neutral Atomic Memory projection #2004 merges.
• OceanBase execution was not performed because no test connection was configured. It is explicitly reported as not_run.
• Native scoped search and the in-process Builder were exercised; HTTP/MCP and host delivery were not exercised.

AI usage statement

OpenAI Codex assisted with implementation, fixture design, regression tests, documentation, and validation.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[hidb4ai]

One evidence-accounting issue remains in the fixture evaluation.

[hidb4ai]

[P2] Include Artifact references in evidence coverage and loss

These sets only collect direct Source identities, ignoring entry.artifacts; measure_pool() has the same omission in unique_evidence_count. I reproduced this with a native SQLite replay on the pinned #2004 Runtime: two byte-identical entries have no direct Sources and cite different retained Artifact revisions. The baseline delivers both entries and the equality arm delivers one, but both report unique_evidence_count = 0 and the alternative reports lost_baseline_evidence = 0. This incorrectly reports no evidence loss when the delivered context loses an independent exact Artifact citation, undermining the coverage comparison required by #1977. Include exact scoped Artifact references, preserving family and revision, in coverage/loss accounting or report Source and Artifact metrics separately, and cover this Artifact-only case.

[hidb4ai]

Rechecked the current PR head, 25c8facd2f9525846abc581ad74ed9be87502c3f, which is unchanged from the reviewed commit. A fresh-database SQLite replay against the PR's pinned #2004 Runtime snapshot (25d9e7a7) still reproduces this: the baseline delivers 2 entries and the equality arm delivers 1, while both report unique_evidence_count = 0 and the alternative reports lost_baseline_evidence = 0, despite dropping one distinct exact Artifact citation. The P2 remains reproducible. Please account for exact scoped Artifact references in coverage and loss, or expose separate Source/Artifact metrics, with an Artifact-only regression case.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[hidb4ai]

LGTM