Skip to content

negative-result informativeness scorer classifies its own test fixture below the INFORMATIVE threshold #843

Open

Description

@cschanhniem

Summary

Three tests in tests/evidence/test_negative_result_informativeness.py::TestCLI fail on main:

  • test_cli_entry_flag
  • test_cli_out_json_creates_file
  • test_cli_out_md_creates_file

Each invokes scripts/classify_negative_result_informativeness.py on the _make_entry() fixture and expects an INFORMATIVE classification (exit 0). The scorer classifies the fixture below the informative band, so the script exits non-zero and the assertions fail.

Hidden until now because CI's pytest step aborted at collection (fixed in #841). These fail on unmodified main.

Root cause

scripts/classify_negative_result_informativeness.py scores each entry 0.0–7.0 and bands it:

INFORMATIVE:     6.0–7.0
PARTIAL:         3.5–5.5
NON_INFORMATIVE: 0.0–3.0
INFORMATIVE_SCORE_MIN = 6.0

The _make_entry() fixture used by TestCLI does not reach 6.0 under the current dimension scorers (_identity_score, _context_score, _specificity_score, etc.), so it is not classified INFORMATIVE. Either:

  • (a) the scorer under-credits a legitimately informative entry (scoring bug), or
  • (b) the fixture is not actually informative enough and the test expectation is wrong (test bug).

Why this needs review (Class C — do not blind-fix)

This is scoring/classification behavior. Per AGENTS.md/CLAUDE.md stop conditions, changes to scoring thresholds, rubric weights, or calibration require human review — they must not be adjusted to make a test pass. Resolving this correctly means deciding whether the rubric or the fixture is wrong, with a decision record if the rubric changes.

Repro

pip install -e ".[dev]"
python -m pytest "tests/evidence/test_negative_result_informativeness.py::TestCLI"

Related to the negative-result informativeness feature (Phase L / Loop area).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions