Skip to content

Expand the adjudicated benchmark to 20–30 representative PRs #42

Description

@cderinbogaz

Parent: #39

Context

The bundled one-PR seed is useful evidence but cannot support a general superiority or replacement claim. The existing benchmarking protocol already recommends 20–30 representative pull requests.

Acceptance criteria

  • Add 20–30 independently adjudicated PR cases spanning frontend, backend, migrations, concurrency, security-sensitive code, small diffs, and large diffs.
  • Use blind adjudication and the same head SHA for every compared reviewer.
  • Record confirmed defects, false positives, duplicate mappings, cost, latency, reviewer version, model configuration, and failures/skips.
  • Validate corpus schema and reviewer completeness in CI.
  • Publish aggregate and per-case precision, recall, P0–P2 recall, duplicate rate, cost, and latency.
  • Publish named misses and unfavorable cases alongside wins.
  • Document selection criteria and known sources of bias.
  • Do not promote a general “better than” headline until the preregistered replacement gate is satisfied.

Metadata

Metadata

Assignees

Labels

benchmarkCorpus, adjudication, evaluationhelp wantedExtra attention is needed

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions