Repository navigation
Add opt-in live-model mode to the benchmark harness - #16
Conversation
Add a live mode to the bench harness that runs the real release binary against the same fixtures with a real model and no mocked model server, measuring detection ability rather than pipeline fidelity. Each case runs in local diff-file mode (postil review --diff-file <f> --no-post --output-json), which does no forge I/O, so no GitHub server is involved and nothing is written to any repo. It scores detection rate (path match, line within +/-3), severity match among detections, silence on clean PRs, false positives, the confidence distribution of true detections, and per-case duration and token usage, mirroring the earlier manual fixture run. Live mode refuses to run without POSTIL_API_KEY and never reads or prints the key value; REVIEW_MODEL/--model selects the model (default deepseek/deepseek-v4-pro). Selected via --live or BENCH_LIVE=1 (new bench:live script). Every run writes a timestamped JSON report under .runs/ (gitignored). Mock mode and bun run bench are unchanged and remain the default and the CI path.
Document the opt-in live mode: that it needs POSTIL_API_KEY, spends real tokens, is not run in CI, what it scores, and that its numbers are a measured baseline for this CLI (single model, one run per case, diff-only) and not a peer comparison.
![postil-dev[bot]](https://avatars.githubusercontent.com/in/3455612?s=60&v=4){kind=small}
There was a problem hiding this comment.
Live-mode defect scoring uses an unsafe cast for the ground-truth line number; if a fixture supplies null, the tolerance check becomes NaN <= 3 and the case is always scored as a miss, silently corrupting the detection-rate metric.
warn
bench/src/live.ts:238— Ground-truth null line breaks defect scoring · confidence 0.85 · kind: risk
Model: moonshotai/kimi-k2.6
| } | ||
|
|
||
| const match = findings.find( | ||
| (f) => f.path === truth.path && Math.abs(f.line - (truth.line as number)) <= LINE_TOLERANCE, |
There was a problem hiding this comment.
warn · confidence 0.85 · kind: risk
Line 238 computes Math.abs(f.line - (truth.line as number)). If truth.line is null (allowed by the GroundTruth interface), the subtraction produces NaN and the comparison is always false, so the defect is never counted as detected even if the model found the right file. This silently under-reports the detection rate for any fixture lacking an exact line. Fix: add a guard so that when truth.line == null the benchmark either matches on path only or skips the case with an explicit error, and remove the unsafe cast.
Adds a --live / BENCH_LIVE=1 mode to the bench harness that runs the real binary against the real fixtures with a real model (local diff-file mode, no forge I/O), scoring detection rate, silence-on-clean, false positives, confidence distribution, and duration, with a timestamped report under bench/.runs (gitignored). Mock mode stays the default and the CI path; live mode refuses without POSTIL_API_KEY and is documented as a measured baseline (single model, one run each), not a peer comparison. tsc, mock bench (30/30), and cargo test are green.