You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
negative-result informativeness scorer classifies its own test fixture below the INFORMATIVE threshold #843
Three tests in tests/evidence/test_negative_result_informativeness.py::TestCLI fail on main:
test_cli_entry_flag
test_cli_out_json_creates_file
test_cli_out_md_creates_file
Each invokes scripts/classify_negative_result_informativeness.py on the _make_entry() fixture and expects an INFORMATIVE classification (exit 0). The scorer classifies the fixture below the informative band, so the script exits non-zero and the assertions fail.
Hidden until now because CI's pytest step aborted at collection (fixed in #841). These fail on unmodified main.
Root cause
scripts/classify_negative_result_informativeness.py scores each entry 0.0–7.0 and bands it:
The _make_entry() fixture used by TestCLI does not reach 6.0 under the current dimension scorers (_identity_score, _context_score, _specificity_score, etc.), so it is not classified INFORMATIVE. Either:
(a) the scorer under-credits a legitimately informative entry (scoring bug), or
(b) the fixture is not actually informative enough and the test expectation is wrong (test bug).
Why this needs review (Class C — do not blind-fix)
This is scoring/classification behavior. Per AGENTS.md/CLAUDE.md stop conditions, changes to scoring thresholds, rubric weights, or calibration require human review — they must not be adjusted to make a test pass. Resolving this correctly means deciding whether the rubric or the fixture is wrong, with a decision record if the rubric changes.
Activity