evaluation(recall): measure lineage concentration and near-duplicate occupancy #1977

Scope

Run a bounded, Scope-local evaluation of Atomic Memory candidate-pool lineage concentration and near-duplicate occupancy before changing retrieval order, admission, retention, or write behavior.

This complements #1942. #1942 evaluates whether Jev can classify a proposed candidate against existing entries; this issue measures whether already-retrieved candidate pools are dominated by one lineage or repeated representation, and whether a conservative deterministic control would improve delivered-context diversity without hurting factual recall.

Hard dependencies

• fix(recall): stabilize deterministic FTS recall across equivalent query forms #1976 must establish the deterministic FTS baseline first. The evaluation may not disguise query-shape regressions as diversity effects.
• feat(memory): add versioned Source evidence declarations and neutral lifecycle projection #1975 must provide versioned Source evidence declarations and a neutral Atomic Memory projection first. The evaluation must report unknown source evidence explicitly, not infer trust or lineage from text.
• Reuse the RFC docs(rfc): define long-horizon workloads #1229 workload/replay boundary. Treat evaluation(memory): assess Jev for duplicate and conflicting Memory candidates #1942's optional Jev relation labels as a separately reported input when available, never as this evaluation's authority.

Implementation and output boundary

• Work in the existing evaluation/replay/report path, starting with evaluation/memory/longmemeval_v2's experiment-arm manifest, replay scoring, and report generation. Add an evaluation-only arm or fixture mode with a versioned identity; do not add a Runtime configuration option or an HTTP/MCP field.
• Instrument only the replay/evaluation process. It may read the current fixed RRF candidate sequence and PreparedContext build result in-process, but it must not emit a Context Receipt, persist a new Runtime event, mutate live records, or expose a per-entry diagnostic surface.
• The deliverable is one reproducible report directory containing a pinned run manifest, content-free aggregate results, command line, backend/capability matrix, fixture/workload version, and a Markdown decision record. Refuse comparisons whose recorded run manifests differ outside the declared evaluation arm.

Evaluation contract

1. Define a versioned, conservative lineage-grouping function over exact existing evidence:
   • exact entry version and predecessor/successor lineage;
   • exact shared Source materialization or Artifact revision where available;
   • explicit unknown when no safe grouping can be established.
     Sharing a Source is an observation only: it is not proof that two distinct facts are duplicates.
2. Build Scope-isolated fixtures and replay workloads containing:
   • byte-identical entries;
   • paraphrases with common evidence;
   • distinct facts that share a Source but must both remain recallable;
   • revised/superseded/historical facts;
   • recurring automated-source material; and
   • the stable CJK/English query forms established by fix(recall): stabilize deterministic FTS recall across equivalent query forms #1976.
3. Measure the baseline post-admission, fixed RRF candidate pool and final in-process PreparedContext build for:
   • same-lineage/top-k concentration;
   • exact and proposed near-duplicate occupancy;
   • unique evidence/fact coverage;
   • relevant-answer recall and false-suppression risk;
   • per-Scope behavior, latency, bytes, candidate count, and model cost where a model is used.
4. Compare only bounded alternatives against baseline:
   • exact-equality collapse for measurement;
   • deterministic lexical-neighbor annotations; and
   • optional advisory relation labels from evaluation(memory): assess Jev for duplicate and conflicting Memory candidates #1942, reported separately.
     No alternative may change the fixed RRF pool membership, make a model call on the production path, or modify live records.
5. Publish a clear recommendation: no product change, a later fixed-pool stable reorder experiment, or a later review-gated relation proposal. State false-suppression and task-regression results separately from duplicate-detection metrics.

Acceptance criteria

• Every evaluated grouping is traceable to exact entry, Source, and Artifact references available under the same Scope authorization; the report exports only aggregate counts/categories and never raw entry/Source content or cross-Scope identifiers.
• The evaluation demonstrates both a concentration case and a legitimate shared-lineage case, so diversity controls cannot be judged solely by duplicate rate.
• Baseline versus alternative reports distinguish retrieval misses, relationship uncertainty, final byte-budget omission, task outcome, latency, and cost.
• SQLite and OceanBase results are reported separately where supported; absent optional capabilities are explicit rather than silently treated as equivalent.
• The evaluation changes no production ranking, entry content, revisions, citations, deactivation state, public API, Runtime event stream, or PreparedContext delivery contract.

Non-goals

• No automatic merge, delete, retirement, supersession inference, importance downgrade, Source trust inference, or top-k diversity reranking in production.
• No replacement for evaluation(memory): assess Jev for duplicate and conflicting Memory candidates #1942, no mandatory Jev/LLM dependency, no broad cross-Scope corpus, and no use of RRF score as temporal or authority evidence.
• No implementation of RFC: explainable PreparedContext receipts and progressive disclosure #1356 Context Receipt or receipt-linked usage observation.

Follow-up boundary

Only a favorable, reproducible result may justify a separate implementation proposal. Any later retrieval behavior must remain inside the existing fixed coarse pool, preserve public RRF score semantics, remain disabled by default, and retain direct access to exact historical evidence as required by RFC #1652.

Validation

Run the evaluation package's focused tests plus the pinned replay/fixture command. Attach or link the resulting report directory, command output, backend/capability matrix, and held-out cases before proposing a production change.

References

• fix(recall): stabilize deterministic FTS recall across equivalent query forms #1976
• feat(memory): add versioned Source evidence declarations and neutral lifecycle projection #1975
• RFC docs: add memory quality,forgetting and lifecycle RFC #1652, fixed-pool quality and near-duplicate boundaries
• RFC docs(rfc): define long-horizon workloads #1229
• evaluation(memory): assess Jev for duplicate and conflicting Memory candidates #1942
• feat: strengthen memory governance, provenance, and diagnostics #1698
• Tracking: stage recall reliability, provenance, diversity, and receipt-linked feedback #1978