Build consumer for outcome-records JSONL (efficacy tracking ground truth)

[toejough]

Problem

The memory-loop audit (dev/eval/audit/REPORT-2026-09-02.md, task 5.3) produces draft outcome-record JSONL files (dev/eval/audit/results/outcome-records.jsonl) per CREDIT_SCHEMA.md, encoding per-note usage outcomes for every audited moment: {moment_id, note_ref, situation_text, outcome ∈ {applied-helped, applied-hurt, surfaced-ignored, injected-unused, absent-needed}, ts, evidence}. These records are output-only in the current design (design.md D-F, spec.md): the audit generates them as a data contract for a follow-on capability, but nothing in the memory-loop-audit change reads or acts on them — they are a measurement artifact awaiting a consumer.

The real run (2026-09-02 baseline) produced 168 outcome records across 16 distinct moments, distributed as: applied-helped 140 (92.9% fan-out concentrated in 8 moments with ≥10 co-surfaced notes), surfaced-ignored 1, absent-needed 27, applied-hurt 0, injected-unused 0. CREDIT_SCHEMA.md §7 documents the real-run counts and the known limitation: multiple co-surfaced notes share one verdict, so the records name each note separately but carry the same outcome tag — there is no way to tell which specific note among several drove the moment's outcome.

Why Now

Efficacy tracking (per-note usage-outcome credit) enables ranked memory injection (SPL-style outcome weighting: 0.7×success_rate + 0.3×recency, apply-floor ≥0.4@5 attempts, prune <0.3@5 attempts — optillm reference, with engram's advantage that verdicts are plan-grounded, not self-judged). This is the measurement foundation for the parked watcher/batch-mining layer (learn-rate-skill-only design.md D-F) and the escalation path to full per-note outcome tracking (separate from #718's per-note efficacy metadata, which tracks surfaced-count/verdict, not storage capacity). Establishing the consumer now unblocks the scaling pathway for future recall ranking improvements — and outcome-records.jsonl already has the data, generated deterministically with zero new LLM calls.

What Changes

Build a specialized consumer for the outcome-records JSONL contract:

1. Schema reader: Parse the 6-field schema (moment_id, note_ref, situation_text, outcome, ts, evidence) from the JSONL records; validate against the five outcome enum values (applied-helped, applied-hurt, surfaced-ignored, injected-unused, absent-needed).

2. Credit aggregation (per-note): For each unique note_ref, tally:

   • How many times it appears in the outcome records (appearances / outcomes_total)
   • Outcome distribution: applied-helped, applied-hurt, surfaced-ignored, injected-unused, absent-needed
   • Success rate: applied-helped / (appearances where a verdict was rendered; exclude nulls)
   • Moment types it touched (success, failure, rework, dispatch, correction)
   • Time window (earliest ts, latest ts)

3. Output format: Emit a new JSONL or CSV file per note (or a single aggregated index), one record per note, carrying the tally and the raw outcome records for traceability (or a link back to outcome-records.jsonl, keyed by note_ref).

4. Integration point: Design the output so a future ranker (for ranked memory injection or a watcher layer) can read it and apply outcome weighting — this is the contract. No ranking logic yet; the output is the read-friendly measurement data.

Known Limitations (Inherited from CREDIT_SCHEMA)

• Multi-note fan-out: a moment with 20 co-surfaced notes produces 20 applied-helped records all asserting the same outcome; the real driver among them is unresolved by design.
• Dispatch moments and correction moments are excluded from outcome-records.jsonl entirely (by CREDIT_SCHEMA rules); dispatch-side handoff gap (D1) is separate in the scorecard.

Reference Material

• CREDIT_SCHEMA.md (task 5.3): full schema + real-run counts + known limitations
• outcome-records.jsonl: the raw data (168 records from 2026-09-02 audit)
• learn-rate-skill-only design.md D-F: the parked watcher/batch-mining escalation (blocked on this consumer + re-measure)
• Related open: recall: per-note usage-outcome (efficacy) tracking — record Step-3 verdicts, rank/prune by them #718 (per-note efficacy metadata at vault-write time — distinct from this consumer, orthogonal tracking)