Skip to content

Add the two tests asked for on #456 (none findings accepted, each phrase refused) - #473

Merged
patelankeet2 merged 1 commit into
mainfrom
test/explain-check-none-and-each-phrase
Nov 19, 2026
Merged

patelankeet2 merged 1 commit into
mainfrom
test/explain-check-none-and-each-phrase

Conversation

@SamikaPerera

Copy link
Copy Markdown
Collaborator

Follow-up to #456 (#456). Tests only.

What was asked

@shubhamkataria2005 approved #456 on 10 November and asked for two tests before merge (#456 (review)). They were not added before #456 merged as ee769b4. His two surviving mutants:

  1. Checking none findings too (status in ("found", "none")) passed every test.
  2. Dropping is safe from _ALL_CLEAR_PATTERN passed every test.

What the tests pin

All in tests/test_explain_output_check_gap.py, using its existing _accepted() helper and real findings from analysis/findings.py.

  • test_a_none_finding_may_be_explained_as_all_clear: a none finding explained as "No issues were found by policy compliance." is accepted, and so is "The access-control check ran on rtr-us5, found no problems, and the filter is fine.", which uses two phrases. For none, an all-clear is the truth.
  • test_each_all_clear_phrase_is_refused_for_a_found_finding: one natural sentence per phrase (no problems, no issues, nothing needs fixing, looks clean, check passed, is fine, is safe), each refused for a found finding. "Nothing needs fixing" starts with a capital, so the case-insensitive match is pinned too.
  • test_every_phrase_has_a_sentence_that_uses_only_that_phrase: guards the test above. Each sentence contains exactly one of the seven phrases, so dropping any one phrase fails exactly one case. The two-phrase sentences already in the file are why "is safe" survived before.

Mutation check

Each mutant was applied to ai/explain.py in a scratch copy, one at a time, and the test file was run:

Mutant Result Failing tests
check applies to status in ("found", "none") killed the 2 new none tests
drop no problems killed 3, including the new one
drop no issues killed 2, including the new one
drop nothing needs fixing killed the new one only
drop looks clean killed 3, including the new one
drop check passed killed 3, including the new one
drop is fine killed the new one only
drop is safe killed the new one only
remove re.IGNORECASE killed 2, including the new one
check applies to nothing killed 17

10 of 10 killed. Four were killed only by the new tests: the none widening, and dropping nothing needs fixing, is fine or is safe.

Counts

tests/test_explain_output_check_gap.py           23 passed (13 before)
full suite, Batfish running                      2334 passed, 13 skipped
ruff check --select E4,E7,E9,F (changed file)    All checks passed

Nothing outside tests/ changes.

With #459

#459 touches the same test file, and both PRs add tests at the end of it, so whichever lands second gets one textual conflict. Keeping both sides resolves it. On a merge of this branch with #459's head d31d8a4, conflict resolved that way, the file gives 32 passed, and the per-phrase tests still pass with #459's widened pattern.

Not covered

Asked for in Shubham's review of #456 on 10 November. Two mutants of the
check survived the whole suite, and these tests kill them:

- A "none" finding explained as an all-clear is accepted ("No issues were
  found by policy compliance.", and one sentence using two phrases). For
  "none" the all-clear is the truth, so widening the check to "none" now
  fails a test.
- Each of the seven phrases in _ALL_CLEAR_PATTERN is refused in a found
  sentence of its own, and a guard test checks that each sentence uses
  only its own phrase. Dropping any one phrase now fails a test, including
  "is safe", which survived before.

Tests only. Nothing in ai/ or any other non-test file changes.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving deffdce. These are exactly the two tests I asked for on #456, and they do their job. I re-ran the mutants that survived there:

  • checking none findings too (status in ("found", "none")): killed (2 failures), so the "an all-clear is the truth for none" exemption is pinned now;
  • each of the seven phrases dropped from _ALL_CLEAR_PATTERN in turn: 7 of 7 killed, so the list can't quietly shrink.

test_explain_output_check_gap.py plus the hygiene check: 129 passed, ruff clean on CI's selection. Thank you for closing it so precisely.

@patelankeet2
patelankeet2 merged commit b6b3865 into main Nov 19, 2026
2 checks passed
@patelankeet2
patelankeet2 deleted the test/explain-check-none-and-each-phrase branch November 19, 2026 01:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants