Repository navigation
Tracking: stage recall reliability, provenance, diversity, and receipt-linked feedback #1978
Description
Purpose
Track the staged path from deterministic recall reliability to provenance-aware retrieval evaluation and, only after an explainable delivery contract exists, a possible host-confirmed aggregate usage observation.
This is a sequencing and acceptance tracker. It does not authorize a new public API, a durable event ledger, query rewriting, read-path mutation, ranking changes, automatic retention, or automatic deactivation by itself.
Work items and gates
| Stage | Issue / prerequisite | Required outcome before the next gated stage |
|---|---|---|
| 1. Deterministic recall baseline | #1912 and implementation task #1976 | The documented equivalent-query forms have stable FTS behavior through both Memory search and prepare_context; backend/capability coverage and negative cases are recorded. |
| 2. Reconstructible source declaration | #1975 | Each Memory entry version can carry a rebuildable, exact Source materialization/declaration snapshot. Missing history is neutral untrusted/unknown; no ranking or lifecycle behavior changes. |
| 3. Candidate-density evidence | #1977, with #1942 reported separately when applicable | A reproducible Scope-local report shows baseline lineage/near-duplicate concentration, legitimate shared-lineage counterexamples, recall/task effects, latency, bytes, and false-suppression risk. A product reorder is not authorized merely by opening this item. |
| 4. Explainable delivery receipt | RFC/design issue #1356, then a separately approved implementation issue | A bounded, optional Context Receipt identifies the bytes actually delivered, exact selected origins, omission reasons, and relevant preparation outcomes without prompt expansion, raw content retention, or default request persistence. |
| 5. Host-confirmed aggregate usage observation | Create a separate implementation issue only after stages 1, 2, and 4 are complete | The host has a stable receipt identity and explicit acknowledgement/use semantics. The first observation remains Scope-local, asynchronous, bounded, idempotent, privacy-safe, and evaluation-only. |
Hard rules for a future usage-observation proposal
A follow-up proposal for stage 5 must meet all of these conditions:
- Observed fact: distinguish
prepared,delivered,host_acknowledged, and any stronger host-specific use signal. A retrieval hit, a rendered citation, or a model response is not automatically proof of use. - Identity: bind an acknowledgement to the exact receipt identity/output digest and exact selected entry versions; reject stale, malformed, unauthorized, cross-Scope, or replayed acknowledgements idempotently.
- Storage: start with bounded, aggregate, Scope-local observations. Do not persist raw query text, prompt/context body, model response, host transcript, vectors, secrets, absolute paths, or an unbounded per-turn event ledger by default.
- Authority: acknowledgement cannot mutate a Memory body, citation, Artifact Revision, Source, or lifecycle state. It is derived evaluation evidence only.
- No immediate product policy: do not create
use_count,last_used_at, ranking boosts, freshness decay, retention, deletion, or automatic deactivation in the first slice. Any later policy must be a separately calibrated RFC/Issue and must use feat(memory): add versioned Source evidence declarations and neutral lifecycle projection #1975's source protection. - Failure semantics: host non-response, receipt loss, privacy refusal, or observation-sink failure must not alter normal context preparation or Memory write/read behavior. They are not negative utility evidence.
- Evaluation: compare enabled and disabled observation paths through replay/workload evidence. Report acknowledgement coverage separately from usefulness, recall quality, task outcome, bad recall, false retention/suppression risk, latency, byte cost, and storage cost.
Dependency semantics
- Stage 1 is a deterministic reliability prerequisite for evaluating later retrieval changes; it must not be bypassed by an LLM query rewrite.
- Stage 2 is a hard safety prerequisite for using frequency or source class in any future ranking/maintenance decision.
- Stage 3 is evidence gathering only. A favorable result may justify a new implementation proposal, not an implicit production reorder.
- Stage 4 is a hard contract prerequisite for host-confirmed observations. RFC: explainable PreparedContext receipts and progressive disclosure #1356 is design-first; its accepted RFC and a concrete implementation are both required.
- feat(runtime): enable privacy-safe hosted-redacted Memory write-gate egress #1973 is related privacy work for hosted DecisionModel egress, but is independent of this sequence and does not authorize receipt or usage tracking.
Non-goals
- No global
MEMORY.md-style memory, session transcript consolidation, mandatory LLM service, cross-host behavioral surveillance, user-global usage profile, or automatic lifecycle policy. - No replacement for RFC docs: add memory quality,forgetting and lifecycle RFC #1652, feat: strengthen memory governance, provenance, and diagnostics #1698, evaluation(memory): assess Jev for duplicate and conflicting Memory candidates #1942, feat(runtime): enable privacy-safe hosted-redacted Memory write-gate egress #1973, or RFC: explainable PreparedContext receipts and progressive disclosure #1356.
- No claim that a high acknowledgement count makes evidence true, current, authoritative, or suitable for automatic retention.
Definition of done
This tracker closes only after every linked implementation/evaluation task has its stated outcome and any proposed stage-5 implementation has separately passed its design, authorization, privacy, backend-parity, and replay gates. Until then, the normal read path remains read-only.
References
- bug(recall): identical fact is returned for one query form and missed for another — retrieval flips on 1-2 character prefixes (CJK) #1912
- fix(recall): stabilize deterministic FTS recall across equivalent query forms #1976
- feat(memory): add versioned Source evidence declarations and neutral lifecycle projection #1975
- evaluation(recall): measure lineage concentration and near-duplicate occupancy #1977
- evaluation(memory): assess Jev for duplicate and conflicting Memory candidates #1942
- RFC: explainable PreparedContext receipts and progressive disclosure #1356
- feat(runtime): enable privacy-safe hosted-redacted Memory write-gate egress #1973
- RFC docs: add memory quality,forgetting and lifecycle RFC #1652
- RFC docs(rfc): propose decision policy governance #1770
- RFC docs(rfc): define long-horizon workloads #1229
Activity