Repository navigation
Report severity match exact and within-one-tier in the live benchmark - #17
Merged
Merged
Conversation
The live-mode detection report scored severity by strict equality only, which understated quality: many warn<->error differences are defensible judgment calls on the fixture labels rather than model errors (see the severity-tier calibration investigation), so a strict-equality metric reads as failure on arguable disagreements. Score severity two ways and report both. severityMatch keeps the original strict equality. severityMatchWithinOneTier treats adjacent tiers on the info < warn < error scale (info<->warn, warn<->error) as a match and counts only the two-tier info<->error gap as a real mismatch. Both numbers appear in the report JSON and the summary line (severity match (exact) and (+/-1 tier)), and the per-case detail still shows truth vs found severity, flagging adjacent calls distinctly from true mismatches. Neither figure is a peer-comparison claim. Mock pass/fail behaviour is unaffected.
Explain why live mode reports severity match both exact and within one tier: exact is the strict metric, while the +/-1-tier band reflects that warn<->error is frequently a defensible judgment call (per the severity-calibration finding, the consistent mismatches are reasonable alternative calls and the direction is bidirectional, so it is not a one-way miscalibration). State explicitly that neither metric is a peer-comparison claim and that the per-case detail keeps truth-vs-found severity visible.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a within-one-tier severity metric alongside the existing exact match in the live bench report. The severity-calibration investigation found that warn<->error severity disagreements are usually defensible judgment calls (bidirectional, not a one-way miscalibration), so the exact metric understates review quality; the +/-1-tier metric (adjacent info<->warn and warn<->error count as a match, only info<->error is a true mismatch) reflects that, while the exact metric and full per-case truth->found detail are retained. Neither is a peer-comparison claim. Mock bench stays 30/30; tsc and cargo test green.