Replay and mining modes for run_audit.py (re-judge and chunk-index harvest)

[toejough]

Problem

The audit instrument (dev/eval/audit/run_audit.py, task 4.3 / 5.1) today runs a fresh pipeline over the live corpus in two optional stages (extract → audit → scorecard) and supports resume across runs. It was built for one-time baseline generation and pre-registered re-measure (design.md D-E) — run the full instrument fresh, collect new data, compare to baseline, measure change. Two complementary use cases are not yet supported:

1. Replay mode: Re-run the judgment stage (audit_moments.audit_transcript) over already-extracted-and-stored transcript-events.jsonl from a prior run, with a different model tier, prompt variant, or recomputed phrases, without re-running the extractor. This unblocks (a) prompt A/B testing (e.g., testing a revised failure-category judgment prompt against the same transcripts to check for prompt-bias), (b) model-tier sensitivity analysis (was the haiku detection or sonnet judgment the bottleneck?), and (c) cheaper iteration during development (extract once, re-judge cheaply after prompt tweaks).

2. Mining mode: Batch-scan the append-only chunk index (~/.local/share/engram/chunks, auto-ingested from subagent transcripts) for moments that would have appeared in the audit sample if their transcripts hadn't been retention-deleted, offering them as candidates for the watcher/batch-mining layer (learn-rate-skill-only D-F escalation). This unblocks the parked watcher's data pipeline: chunk-mined lessons become offers (engram serve) for curation at closing learn time.

Why Now

Replay supports the iterative refinement cycle for the audit tool itself (prompt tuning, model selection). Mining unblocks the parked watcher/batch-mining escalation path (learn-rate-skill-only design.md D-F: if re-measure shows W2 still low, the watcher layer becomes a v2 direction — mining the chunk index for lessons is its evidence production step). Both are prerequisites for scaling the audit from a one-time measurement to a repeatable, refineable system.

What Changes

Extend run_audit.py with two new modes (in addition to the existing --estimate and --config [full-run] modes):

1. Replay mode (--replay):

   • Take a prior run's output directory (containing transcript-events.jsonl) and a config file.
   • Re-run audit_moments.audit_transcript over the same transcripts, skipping extract.
   • Supports changing: model_tiers (e.g., try detect: haiku, judge: haiku vs. baseline detect: haiku, judge: sonnet), and optionally phrase generation (recompute phrases from raw transcripts or use cached phrases from prior run).
   • Output: new moments.jsonl with the same transcript corpus but different judgments; manifest records this as replay with the config deltas (model tier changes).
   • Use case: prompt/model A/B testing without re-extracting; sensitivity analysis.

2. Mining mode (--mine-chunks):

   • Scan ~/.local/share/engram/chunks for transcript records from projects in the configured repo set.
   • Identify candidates: transcript events that resemble recall/dispatch/learn/command patterns (similar detection logic to extract.py's mechanical classifier, but over stripped chunk text).
   • For each candidate, run audit_moments.audit_transcript to judge it (detect moment, answer the scorecard questions).
   • Output: new moments.jsonl + transcript-events.jsonl supplement, labeled with source: 'chunks' and chunk anchor.
   • Optionally, hand mined candidates to the existing pending-offer path (the same notes engram serve lands as pending offers for the curate skill to judge) — no new submission command is assumed; the exact write path is a design question for this issue.
   • Manifest records this as mining with chunk-source provenance.
   • Use case: data recovery (moments whose transcripts were retention-deleted), watcher/batch-mining evidence production, re-measure baseline extension (adding moments from hidden/deleted sessions).

Technical Details

run_audit.py changes:

• Add --replay <prior_run_dir> flag. Load <prior_run_dir>/transcript-events.jsonl, skip extract, run audit stage with the configured model_tiers (or use prior run's if not specified).
• Add --mine-chunks flag. Enumerate chunk index, classify candidates, audit each, output moments + transcript-events supplement.
• Both modes output to --output_dir as usual; manifest records mode, source, and config deltas.

Preserve existing semantics:

• Resume logic is per-run (transcript-events.jsonl as done-marker); replay/mining generate fresh jsonl with judged_from='replay' / judged_from='chunks' provenance labels (don't overwrite originals).
• Scorecard stage is always skipped for replay/mining (prevent silent re-computation; build_scorecard.py's frozen-results check applies).
• --estimate mode works on a config alone (no prior run dir needed).

Reference Material

• run_audit.py docstring (lines 1–39): existing modes and orchestration structure
• extract.py: the mechanical-extractor pattern (how to classify moments from raw text)
• audit_moments.py: the semantic auditor (prompt, runner, model tiers)
• learn-rate-skill-only design.md D-F: watcher/batch-mining escalation (the downstream consumer)
• REPORT-2026-09-02.md §5b: chunk-index recovery precedent (the audit itself used chunks as a fallback; this formalizes that pattern)