Tracking: implement Memory quality and lifecycle RFC #1652

[jiannnnyyyyy]

Feature description

Implement the design approved in PR #1652 and its synchronized English RFC / Chinese RFC.

The implementation must preserve immutable Memory authority and existing behavior when lifecycle options are disabled or neutral. This issue is the parent tracking item; each checked item should land as an independently reviewable PR or be split into a child issue when work starts.

Task breakdown

• P0 — Source evidence declaration and neutral lifecycle projection

  • Add a versioned, reconstructible MemoryEvidenceDeclaration contract on Source/SourceDefinition for authority and verification.
  • Snapshot the declaration for the exact Source materialization used by each Memory entry; missing or unverifiable declarations remain neutral/untrusted.
  • Add the rebuildable lifecycle/quality projection, internal context annotation, rule-version metadata, and SQLite/OceanBase rebuild/conformance fixtures.
  • Do not change entry bodies, content hashes, revisions, citations, or public search schemas.

• P0 — Stage-1 validity filtering and fixed-pool quality ordering

  • Enforce known inactive and explicit superseded validity before RRF and before optional reranking for ordinary current-state recall; preserve exact historical reads and unresolved_conflict.
  • Add deterministic independent dimensions for provenance/verification, importance, novelty, and current-state validity.
  • Add MemoryRankingTrace/bounded statistics for diagnostics without exposing Memory content or changing the public RRF score.
  • Reorder only inside the exact fixed RRF coarse pool, with the RFC's maximum ±2 displacement; never change channel admission or reranker membership.
  • Preserve the RFC 1560/feat(runtime): add a bounded recall sufficiency gate to prepare_context #1596 per-round semantics: rerank before the recall gate receives result.hits; disabled/neutral quality must produce the current behavior.

• P1 — Near-duplicate alignment and admission proposals

  • Implement the bounded sequence normalized hash → lexical candidates → optional semantic neighbors.
  • Emit relationships/proposals and bounded rejection/inventory statistics; never merge, discard, or lower importance solely from similarity.
  • Fail safe without embeddings and keep explicit writes/deterministic adapters able to opt out when they already establish idempotency.

• P1 — L1 reversible maintenance

  • Add lifecycle tiers, protection/pin handling, age/capacity eligibility, dry-run output, and bounded inventory.
  • Implement recoverable deactivation with stable reasons such as auto_decay:v1 and session_end.
  • Make actions auditable, idempotent, inbound-lineage aware, backend-parity tested, and recoverable through reactivate() without creating a new body version.
  • Keep automatic mutation disabled by default and separate logical deactivation from physical retention/erasure.

• P2 — L2 review-gated semantic maintenance

  • Add Memory-specific proposals and operation records for evidence-preserving consolidation, explicit supersession, unresolved conflicts, and recurring working-note promotion.
  • Require approval before creating a Revision or changing lifecycle authority; preserve exact affected entry versions, evidence lineage, successor identity, and compensation behavior.
  • Integrate with the review mechanism without reusing/changing the Experience/Skill recurrence ledger or ordinary direct Memory writes.

• P1 — Evaluation, calibration, and backend conformance

  • Cover retrieval/task/safety/cost metrics, current-vs-historical reads, candidate-density interference, L1 dry-run precision, L2 proposal quality, latency/bytes, and full-context reference.
  • Run FTS/vector/hybrid × reranker on/off × recall-gate on/off checks, including fixed-pool membership and gate-decision invariants.
  • Add longitudinal revision sequences, low-authority/high-frequency source-poisoning cases, and SQLite/OceanBase projection/rebuild parity.
  • Keep numeric thresholds as calibration seeds; do not enable non-neutral defaults without held-out evidence.

• P3 — Physical retention/compaction follow-up (separate RFC, only if justified)

  • Observe active/inactive counts, manifest/tombstone growth, and storage cost first.
  • If inventory demonstrates a storage problem, open a separate RFC covering physical retention, compliance erasure, projection cleanup, and exact-citation behavior. Do not add physical deletion or body compaction to this issue.

Delivery order and dependencies

1. Complete the Source contract and neutral projection first; ranking remains disabled.
2. Land stage-1 validity filtering and fixed-pool ranking with disabled-mode parity.
3. Add near-duplicate alignment and dry-run-first L1 maintenance.
4. Add review-gated L2 after the review integration boundary is stable.
5. Treat physical retention/compaction as a separate proposal; it is not a dependency of retrieval quality.

Each stage should be independently deployable, observable, and reversible.

Guardrails / non-goals

• No rewrite of authoritative Memory bodies, content hashes, Artifact Revisions, evidence citations, or Handoff citation identity.
• No default quality boost, decay, automatic retirement, semantic merge, supersession inference, or model-generated authority score.
• No expansion of the RRF candidate pool, no change to channel admission, no change to public RRF score semantics, and no new HTTP/MCP/CLI trace field in the first stage.
• No compliance deletion, external erasure, manifest compaction, or Topic Memory summarization here; those remain under RFC: define retention, erasure, and lifecycle policies #1425/bug: Memory append storage and latency grow superlinearly with entry history #1321 and follow-up RFCs.

Definition of done

• With all options disabled or neutral, write/search/rerank/Context Pack behavior and public contracts match the current implementation.
• Derived state is rebuildable from authoritative revisions and bounded operation evidence on both supported backends.
• Acceptance criteria in RFC docs: add memory quality,forgetting and lifecycle RFC #1652 are covered by automated tests and a reproducible evaluation report.
• Documentation is updated in both English and Chinese when behavior or configuration becomes user-visible.

Related work

• RFC tracking/design: #1590 (closed by PR docs: add memory quality,forgetting and lifecycle RFC #1652)

• Retention/erasure boundary: #1425

• Memory storage/write amplification: #1321

• Recall sufficiency and gate: #1556, #1596

• Experience review/recurrence boundary: #1586

• Review inbox mechanism: RFC 0050

• I am willing to contribute code, docs, or design feedback.