Structured oracle artifacts: per-conversation answer key + distractor support

[zaridan]

Background

The deterministic/LLM split already gives us the right foundation; this extends it with a richer evaluation layer.

Phase 1 — Per-conversation answer key artifact

Currently, rforge generate plants issues into conversations and computes accuracy at eval time by reconstructing the oracle on the fly. The oracle is never persisted.

Proposed: emit an answer_key.json alongside each replay envelope at generate time, containing:

• planted issue IDs
• expected scorer findings
• severity

This makes every scorer run comparable against the same seed-level oracle. Regressions become diff-able per conversation rather than visible only in aggregate accuracy stats.

Phase 2 — Distractor planting

Extend corpus generation to plant deliberate distractors — red-herring signals the scorer should not flag. Extend the answer key schema to record distractor IDs, and extend accuracy scoring to report false-positive rate alongside recall.

This distinguishes precision from recall at the seed level — a scorer can have perfect recall and still hallucinate findings on planted noise.

Phase 2 depends on Phase 1 (answer key schema must exist first).

Notes

• Both phases are additive — existing corpus generation and replay workflows are not broken
• Answer key schema will be documented in harness/docs/