Repository navigation
Add the two tests asked for on #456 (none findings accepted, each phrase refused) - #473
Merged
Merged
Conversation
Asked for in Shubham's review of #456 on 10 November. Two mutants of the check survived the whole suite, and these tests kill them: - A "none" finding explained as an all-clear is accepted ("No issues were found by policy compliance.", and one sentence using two phrases). For "none" the all-clear is the truth, so widening the check to "none" now fails a test. - Each of the seven phrases in _ALL_CLEAR_PATTERN is refused in a found sentence of its own, and a guard test checks that each sentence uses only its own phrase. Dropping any one phrase now fails a test, including "is safe", which survived before. Tests only. Nothing in ai/ or any other non-test file changes.
shubhamkataria2005
approved these changes
Nov 16, 2026
shubhamkataria2005
left a comment
Collaborator
There was a problem hiding this comment.
Approving deffdce. These are exactly the two tests I asked for on #456, and they do their job. I re-ran the mutants that survived there:
- checking
nonefindings too (status in ("found", "none")): killed (2 failures), so the "an all-clear is the truth fornone" exemption is pinned now; - each of the seven phrases dropped from
_ALL_CLEAR_PATTERNin turn: 7 of 7 killed, so the list can't quietly shrink.
test_explain_output_check_gap.py plus the hygiene check: 129 passed, ruff clean on CI's selection. Thank you for closing it so precisely.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #456 (#456). Tests only.
What was asked
@shubhamkataria2005 approved #456 on 10 November and asked for two tests before merge (#456 (review)). They were not added before #456 merged as
ee769b4. His two surviving mutants:nonefindings too (status in ("found", "none")) passed every test.is safefrom_ALL_CLEAR_PATTERNpassed every test.What the tests pin
All in
tests/test_explain_output_check_gap.py, using its existing_accepted()helper and real findings fromanalysis/findings.py.test_a_none_finding_may_be_explained_as_all_clear: anonefinding explained as "No issues were found by policy compliance." is accepted, and so is "The access-control check ran on rtr-us5, found no problems, and the filter is fine.", which uses two phrases. Fornone, an all-clear is the truth.test_each_all_clear_phrase_is_refused_for_a_found_finding: one natural sentence per phrase (no problems, no issues, nothing needs fixing, looks clean, check passed, is fine, is safe), each refused for afoundfinding. "Nothing needs fixing" starts with a capital, so the case-insensitive match is pinned too.test_every_phrase_has_a_sentence_that_uses_only_that_phrase: guards the test above. Each sentence contains exactly one of the seven phrases, so dropping any one phrase fails exactly one case. The two-phrase sentences already in the file are why "is safe" survived before.Mutation check
Each mutant was applied to
ai/explain.pyin a scratch copy, one at a time, and the test file was run:status in ("found", "none")nonetestsno problemsno issuesnothing needs fixinglooks cleancheck passedis fineis safere.IGNORECASE10 of 10 killed. Four were killed only by the new tests: the
nonewidening, and droppingnothing needs fixing,is fineoris safe.Counts
Nothing outside
tests/changes.With #459
#459 touches the same test file, and both PRs add tests at the end of it, so whichever lands second gets one textual conflict. Keeping both sides resolves it. On a merge of this branch with #459's head
d31d8a4, conflict resolved that way, the file gives 32 passed, and the per-phrase tests still pass with #459's widened pattern.Not covered