Port the PR-review benchmark harness as a mock-mode regression suite

.github/workflows/ci.yml

@@ -21,3 +21,16 @@ jobs:
       - run: cargo clippy --quiet --all-targets -- -D warnings
       - run: cargo test --quiet
       - run: cargo build --quiet --release
+
+  bench:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      - uses: oven-sh/setup-bun@v2
+      - run: cargo build --quiet --release
+      - run: bun install --frozen-lockfile
+        working-directory: bench
+      - run: bun run bench
+        working-directory: bench

ROADMAP.md

@@ -61,3 +61,7 @@ Port plan: bring the harness into this repo against the envelope v1 contract,
 keep mock mode as a regression suite, add a live-model mode (real inference,
 same fixtures) to measure detection and silence rate, then run peers on the
 identical fixture set before publishing any comparison.
+
+Status: mock mode is ported and lives in `bench/` (all 30 fixtures, isolation
+and prompt-leakage guardrails kept), runs as the `bench` job in CI against a
+release build. Live-model mode and peer runs remain open.

bench/.gitignore

@@ -0,0 +1,2 @@
+node_modules/
+.runs/

bench/README.md

@@ -0,0 +1,44 @@
+# postil bench
+
+Hermetic PR-review regression suite for the `postil` CLI, ported from the
+previous product line's benchmark harness. Each of the 30 fixtures (seeded
+defects across languages and change classes, plus clean PRs where the correct
+review is silence) runs the release binary against a per-case mock GitHub API
+and a mock OpenAI-compatible model endpoint in an isolated run directory, then
+scores the v1 envelope and the forge interactions against ground truth.
+
+## What it measures
+
+Pipeline fidelity, not detection ability: grounding (no ungrounded findings,
+`counts.ungrounded == 0`), gating (gate fails exactly on error-severity
+findings, exit codes match), statusline correctness (`postil/review` and
+`postil/gate` check-runs created and completed with the right conclusions),
+silence on clean PRs (no review comment posted), and prompt-leakage guardrails
+(fixture metadata such as policy phrasing must never appear in the prompt or
+any pipeline output).
+
+From the roadmap, verbatim — the caveats that gate any public use of these
+numbers:
+
+> - The mock model echoes output generated from the same spec as the ground
+>   truth, so the run measures pipeline fidelity (grounding, gating, statusline
+>   correctness) — not detection ability.
+> - No competitor has been run on the same fixtures, so comparative claims are
+>   not defensible; site comparisons stay qualitative and sourced until peer
+>   runs on identical fixtures exist.
+
+A live-model mode (real inference, same fixtures) is a later slice; see
+ROADMAP.md.
+
+## Running
+
+```sh
+cargo build --quiet --release   # from the repo root
+cd bench
+bun install
+bun run bench                   # add --json or --json-out report.json for machine output
+```
+
+`POSTIL_BIN` points the suite at a different binary;
+`POSTIL_BENCH_KEEP_RUNS=1` keeps per-case run directories under `.runs/`
+(failing cases are always kept).