Parent: #39
Context
The bundled one-PR seed is useful evidence but cannot support a general superiority or replacement claim. The existing benchmarking protocol already recommends 20–30 representative pull requests.
Acceptance criteria
- Add 20–30 independently adjudicated PR cases spanning frontend, backend, migrations, concurrency, security-sensitive code, small diffs, and large diffs.
- Use blind adjudication and the same head SHA for every compared reviewer.
- Record confirmed defects, false positives, duplicate mappings, cost, latency, reviewer version, model configuration, and failures/skips.
- Validate corpus schema and reviewer completeness in CI.
- Publish aggregate and per-case precision, recall, P0–P2 recall, duplicate rate, cost, and latency.
- Publish named misses and unfavorable cases alongside wins.
- Document selection criteria and known sources of bias.
- Do not promote a general “better than” headline until the preregistered replacement gate is satisfied.
Parent: #39
Context
The bundled one-PR seed is useful evidence but cannot support a general superiority or replacement claim. The existing benchmarking protocol already recommends 20–30 representative pull requests.
Acceptance criteria