Repository navigation
Add opt-in live-model mode to the benchmark harness - #16
Merged
Merged
postil-dev / postil/review
succeeded
Jun 13, 2026 in 3m 35s
0 error, 1 warn, 0 info
Live-mode defect scoring uses an unsafe cast for the ground-truth line number; if a fixture supplies null, the tolerance check becomes NaN <= 3 and the case is always scored as a miss, silently corrupting the detection-rate metric.
warn
bench/src/live.ts:238— Ground-truth null line breaks defect scoring · confidence 0.85 · kind: risk
Model: moonshotai/kimi-k2.6
Annotations
Check warning on line 238 in bench/src/live.ts
postil-dev / postil/review
Ground-truth null line breaks defect scoring
Line 238 computes `Math.abs(f.line - (truth.line as number))`. If `truth.line` is `null` (allowed by the `GroundTruth` interface), the subtraction produces `NaN` and the comparison is always false, so the defect is never counted as detected even if the model found the right file. This silently under-reports the detection rate for any fixture lacking an exact line. Fix: add a guard so that when `truth.line == null` the benchmark either matches on path only or skips the case with an explicit error, and remove the unsafe cast.
Loading