evaluation(workbuddy): measure joined-prompt self-ingestion before adding owned evidence projections #2069

Goal

Measure whether host-assembled prompts cause PowerContext-injected context, historical user turns, or host/Skill scaffolding to be re-ingested as new Atomic Memory evidence. If the risk is reproduced, produce a concrete follow-up contract for a versioned, host-owned current-turn evidence projection while preserving the raw Source for audit.

This issue is evaluation-only. It must not change production capture, Source projections, Atomic Memory extraction, or public APIs until the replay demonstrates a real contamination mode.

Verified data path

On current master, WorkBuddy treats recall and capture differently:

1. integrations/workbuddy/plugins/powercontext/hooks/workbuddy_powercontext_hook.py::_recall_query() validates the host wrapper and prefers the last current <user_query> for recall.
2. The same hook passes the complete host-submitted joined prompt to _capture_prompt().
3. src/powercontext/builtin/sources/content.py::ContentTextEvidenceProjection copies the complete ContentSource.content into powercontext.text-evidence v1.
4. src/powercontext/builtin/artifacts/atomic_memory/extraction.py::project_atomic_memory_evidence() supplies that projection/materialization to Atomic Memory extraction.

Existing WorkBuddy tests intentionally preserve raw capture even when the recall query reduces to the current turn or to nothing. That is correct for audit, but it means previously injected PreparedContext and historical wrapper text can re-enter extraction evidence.

This is a verified data-flow risk, not yet a measured claim that incorrect or duplicate Atomic Memory is produced. The issue must establish the rate and failure modes before changing contracts.

Scope

1. Add a versioned capture-ownership replay

Create evaluation/memory/capture_ownership/ with a committed synthetic manifest and private-run support. The harness must execute the real path:

host payload
  -> /v1/sources/content capture
  -> persisted Source/materialization
  -> powercontext.text-evidence projection
  -> AtomicMemoryGenerationPipeline extraction
  -> candidates + exact evidence IDs

Do not score only _last_user_query() or call the model directly with an ad hoc string; the replay must prove what the runtime actually persists and sends to extraction.

Each fixture must provide exact labeled spans for:

• current user input;
• earlier user turns;
• previously injected PreparedContext/citations;
• host wrapper/system scaffolding;
• Skill or expert instructions;
• quoted or nested fake <user_query> elements;
• optional conversation summary/task notification text.

The committed fixtures must be synthetic and content-safe. A private corpus may be replayed, but raw payloads and generated candidate text remain outside Git and are referenced only by hashes in aggregate artifacts.

2. Compare two arms without changing production behavior

Run the same exact Source identities and model settings through:

• joined-source-v1: current production projection from the full captured prompt;
• host-turn-candidate-v1: an evaluation-only projection produced by the current WorkBuddy host-owned wrapper parser.

The second arm is a hypothesis, not a production design. It must remain inside the evaluation package and must not be registered in SourceDefinitionRegistry, accepted by the public capture API, or used by Runtime processing in this issue.

Both arms must use the same extraction prompt/version, provider/model settings, evidence ordering, and generation budgets. Freeze the dataset manifest and model settings before examining outputs.

3. Measure evidence and candidate ownership

For each case and arm, report:

• UTF-8 bytes and labeled-span occupancy of current user input, earlier turns, prior PreparedContext, host scaffolding, and Skill text in extraction evidence;
• candidate count and kind;
• whether every candidate claim is supported by the current user span, only by an earlier user span, only by PowerContext-injected context, only by host/Skill scaffolding, or by mixed spans;
• false current-user attribution: content originating outside the current user span represented as the user's new fact/preference/decision/constraint;
• duplicate/near-duplicate candidates attributable to repeated joined history;
• retention of required new durable facts from the current user span;
• empty-output and fallback behavior for malformed, nested, forged, or missing wrappers;
• token/latency differences as secondary metrics, not as a quality pass condition.

Candidate support labels must be reviewed against the frozen synthetic ground truth. An LLM judge may assist, but cannot be the only evidence for critical contamination cases. Store exact candidate/evidence text only in private run artifacts; the committed report contains case IDs, categories, counts, rates, hashes, and bounded examples with synthetic placeholders.

4. Decision gate and required follow-up artifact

The report must make one of these decisions:

• no_production_change: no critical candidate contamination was reproduced; record limitations and keep current behavior.
• projection_follow_up_required: at least one reproducible candidate is supported only by injected/scaffold/history content or is falsely attributed as current-user evidence, while the candidate projection arm preserves the fixture's required current facts.
• insufficient_evidence: transcript/parser/provider coverage cannot support either conclusion.

When projection_follow_up_required, emit an implementation-ready design artifact and open a separate production issue. That follow-up must satisfy all of these constraints:

1. Preserve the exact raw host submission as the auditable Source materialization.
2. Store or deterministically rebuild a separate versioned text-evidence projection bound to the exact Source identity/materialization and parser version.
3. The projection is created by the owning host parser; do not add a generic Server regex scrubber for arbitrary prompts.
4. A forged, nested, ambiguous, or missing wrapper cannot be upgraded to current-user asserted evidence. Use a neutral unknown/not-eligible result for automatic extraction, while retaining raw Source auditability.
5. Source declaration/authority remains separate from payload ownership. The projection does not become trusted merely because a host supplied it.
6. Retry/replay of the same host event is idempotent and cannot bind a different projection to the same Source materialization.
7. Existing Artifact bodies/revisions, citation identity, search ranking, and PreparedContext behavior do not change.
8. Codex, Claude Code, WorkBuddy, and other hosts each declare their actual payload contract. Do not assume WorkBuddy's wrapper parser applies to another host.
9. Any public Source/capture contract change is design-first, updates openapi/powercontext.yaml, regenerates clients, and defines compatibility for old captures/projections and projection rebuild.

Required fixture matrix

At minimum:

• one current durable fact after several historical user turns;
• prior PreparedContext containing a durable-looking fact absent from the current turn;
• prior PreparedContext containing instruction-like text and citations;
• host/Skill scaffold containing preferences or procedures that must not become user memory;
• repeated historical user fact across multiple joined messages;
• current input that quotes <user_query> tags;
• nested, truncated, mismatched, and missing wrappers;
• blank current <user_query> with nonblank history;
• multilingual current input and history;
• current factual command/path/config literal that must be retained exactly;
• a negative control with no durable current fact.

Implementation map

• integrations/workbuddy/plugins/powercontext/hooks/workbuddy_powercontext_hook.py: current _last_user_query, _recall_query, and _capture_prompt behavior; production code is read-only for this issue.
• tests/workbuddy_plugin/test_hook.py: wrapper fixtures and raw-capture contract; do not weaken these tests to make the evaluation pass.
• src/powercontext/builtin/sources/content.py: current powercontext.text-evidence v1 projection; production code is read-only for this issue.
• src/powercontext/sources/projections.py: current projection key/schema; possible design seam only after the decision gate.
• src/powercontext/builtin/artifacts/atomic_memory/extraction.py: exact evidence projection consumed by extraction; production code is read-only for this issue.
• evaluation/memory/capture_ownership/: manifest, Source replay adapter, arm assembly, scorer, report, CLI, and tests.
• evaluation/memory/longmemeval_v2/: reuse manifest/report conventions where useful; do not fold this ownership question into retrieval scoring.

Acceptance criteria

• The replay persists and re-reads real Content Sources and invokes the actual Atomic Memory extraction pipeline for both arms.
• Every fixture labels ownership spans and freezes expected required/forbidden claims before generation.
• Both arms use identical Source IDs, evidence ordering, prompt version, model settings, and budgets; only the evaluation projection differs.
• The report separates evidence occupancy, candidate contamination, false attribution, near-duplicate production, and current-fact retention.
• Critical contamination decisions have deterministic human-auditable ground truth and are not based solely on an LLM judge.
• Forged/nested/missing wrapper cases never count joined history as validated current-user evidence in the candidate arm.
• Raw transcripts, Source bodies, prompt text, generated candidate text, credentials, and cross-Scope identifiers are absent from committed aggregate artifacts.
• The result is exactly one of no_production_change, projection_follow_up_required, or insufficient_evidence, with limitations and pinned hashes.
• If a follow-up is required, its generated contract addresses raw Source preservation, projection versioning/rebuild, neutral fallback, idempotency, compatibility, and per-host ownership.
• No production Source, projection, extraction, search, ranking, citation, or API behavior changes in this issue.

Non-goals

• No free-text deletion or regex scrubbing inside the Server.
• No silent replacement of the raw WorkBuddy Source with only the latest user turn.
• No assumption that retrieved or injected context was used by the model.
• No authority/trust upgrade based on an integration metadata string.
• No expansion of feat(memory): add versioned Source evidence declarations and neutral lifecycle projection #1975 into host payload parsing.
• No production parser shared across hosts before their contracts are independently verified.
• No automatic deletion, merge, retirement, or reranking of existing Atomic Memory.

Dependencies and ordering

• The evaluation can run on current master.
• Any production projection implementation must align with feat(memory): add versioned Source evidence declarations and neutral lifecycle projection #1975 and its implementation PR feat(memory): persist versioned Source evidence and neutral Atomic Memory projection #2004 so payload ownership does not bypass immutable evidence declarations or historical rebuild semantics.
• Findings about duplicate candidates may inform evaluation(recall): measure lineage concentration and near-duplicate occupancy #1977, but evaluation(recall): measure lineage concentration and near-duplicate occupancy #1977 does not own host payload parsing.
• WorkBuddy joins the whole session into the hook prompt, so recall returns nothing #1702 fixed WorkBuddy recall-query extraction only; it intentionally did not change raw Source capture.

Validation

At minimum:

uv run pytest tests/workbuddy_plugin/test_hook.py
uv run pytest evaluation/memory/capture_ownership/tests
uv run prek run -a
uv run ty check

Document the opt-in live-generation command and provider requirements. The unit suite validates replay identity, span accounting, scoring, redaction, and report determinism without requiring network credentials.