Skip to content

Refuse an all-clear explanation of a found or error finding - #456

Merged
SamikaPerera merged 5 commits into
mainfrom
test/explain-output-check-gap
Nov 12, 2026
Merged

SamikaPerera merged 5 commits into
mainfrom
test/explain-output-check-gap

Conversation

@SamikaPerera

Copy link
Copy Markdown
Collaborator

This reports a gap in the output check in ai/explain.py and closes it. It does not claim the AI is unsafe: no real model has been shown producing an all-clear for a found finding.

The gap

_is_unacceptable() only rejects a claim about the network for status="error" (ai/explain.py:860 on main), and only by single words (_RESULT_CLAIM_WORDS). Nothing checks a status="found" explanation for an all-clear. The dashboard only sends found findings to the model (web/main.py:1036). So if a model answered "No issues found." about a confirmed problem, the user would see it labelled as model output.

The same list also misses ordinary all-clear wording for error findings, such as "Everything looks clean." or "This check passed." for a check that could not run.

What changed

ai/explain.py, one commit, 15 lines added:

  • Lines 651-662: _ALL_CLEAR_PATTERN, a short list of whole phrases: no problems, no issues, nothing needs fixing, looks clean, check passed, is fine, is safe.
  • Lines 874-876: _is_unacceptable() checks that list for found and error findings. A match is handled like any other rejected answer: one retry, then the deterministic fallback that already exists (_fallback_plain_restatement() for found, _fallback_error_explanation() for error).

none findings are not checked, because there an all-clear is true. The existing error-only word list is unchanged. Reusing it for found would reject honest answers, because they need words like "reach" and "blocked".

Nothing else changed: no prompt change, no input sanitising, nothing in web/.

Commits

The first four commits add tests/test_explain_output_check_gap.py and are red on purpose. They show the gap on main before anything is fixed. The fifth is the fix. Please keep them in that order.

  • 53eaace the test file and helpers
  • e7f2ec3 the control: one faithful explanation per status (found, none, error) must still be accepted, so a fix that rejects everything cannot pass
  • dbe3edc 4 all-clear sentences for an error finding (red)
  • 2890ab7 6 all-clear sentences for a found finding (red)
  • 8c22f2e the fix

False positives, measured

The risk is rejecting an honest explanation of a real problem.

  • Rated holdout set (tests/fixtures/explanation_quality/holdout.json, llama3.2:3b, rated by Ankeet): 4 found explanations, 0 rejected by the new phrases. The old error-only word list would have rejected 3 of the 4, for "reach".

  • Fresh model output: netwise-warden (local Ollama) explained every found finding in seven fixtures (rtr-us5-insecure, rtr-us5-messy, vendor-asa, vendor-arista, vendor-juniper, routing-faults, sample-network). That was 23 findings, one explanation each, 0 matched the new phrases. The old word list would have matched 15. 3 of the 23 are already rejected on main for other reasons (speculation or an invented address), which this PR does not change.

  • Test strings: 37 string literals in tests/ contain one of the phrases. They are docstrings, none summaries ("No issues found by access control") and samples meant to be rejected. The full suite passes.

  • Sentences I wrote to probe the phrases. None appeared in any real output above. These four are honest and would be rejected:

    • "The rule is fine-grained."
    • "Nothing needs fixing elsewhere, but this line does."
    • "The health check passed, but the ACL still permits telnet."
    • "No issues were found with the other lines."

    If a model wrote one, the user would get the deterministic restatement instead of the model's text. These do not match: "There are no other issues on this filter.", "The rule is not safe.", "This is safely ignored.".

Verification

  • tests/test_explain_output_check_gap.py: 13 passed. With ai/explain.py reverted to the parent commit: 10 failed, 3 passed.
  • Full suite on the branch: 2255 passed, 13 skipped.
  • Full suite merged with main (d424e05), on a plain copy: 2250 passed, 13 skipped. tests/test_no_configs_in_git.py (24 tests) was left out because the copy was not a git checkout.
  • ruff check --select E4,E7,E9,F on the changed files: all checks passed.

Not done

  • IPv6 zone ID free text still reaches the subject.
  • The 22-run obedience test was one model only.
  • No real model has been shown producing an all-clear for a found finding. This closes the gap; it does not fix an observed failure.
  • The phrase list is short and phrase-based. Other all-clear wording will still get through.
  • The _is_unacceptable() docstring still says only error findings get the extra result check. I left it to keep the change minimal.

ai/ is Ankeet's (@patelankeet2), so he should review this, including whether the phrase list is the right shape.

The module docstring says what the gap is and why it matters (F-4): the
check that stops a claim about the network runs only for status="error",
and it is a short word list, so "Everything looks clean." passes for a
check that could not run, and "No issues found." passes for a confirmed
problem. Two helpers build a real finding of a given status and report
whether a stubbed model answer would reach the user as model output.

No tests yet, so pytest reports that none ran.
…tatus

One accurate sentence per status (found, none, error), each repeating only
what its finding says. All three pass today. This stops a future fix from
making the gap tests pass by rejecting everything.
A status="error" finding means the check could not run. Four sentences
that tell the user the opposite are accepted by the output check today,
because none of their words is on the error-only result-claim list:
"There are no problems with this rule.", "This is fine and nothing needs
fixing.", "Everything looks clean." and "This check passed." That is F-4's
failure. The test asserts each is rejected, so it fails on main for that
reason, and is left failing on purpose rather than marked xfail.
A status="found" finding is a confirmed problem, and it is the status the
dashboard sends to the model. The output check runs no result-claim test
for it, so all six clean-sounding sentences tried are accepted today,
including "No issues found." and "...the rule is safe.", which are refused
for an error finding. The test asserts each is rejected, so it fails on
main for that reason, and is left failing on purpose rather than marked
xfail.
The output check only rejected claims about the network for status
"error", and only by single words. A model answer such as "No issues
found." for a confirmed problem, or "Everything looks clean." for a check
that could not run, reached the user labelled as model output.

Add a short list of all-clear phrases and check it for "found" and
"error" findings. A match is treated like any other rejected answer: one
retry, then the existing deterministic fallback. "none" findings are not
checked, because there an all-clear is true. The existing error-only word
list is unchanged; reusing it for "found" would have rejected 3 of the 4
rated explanations in the holdout set, which say "reach".

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-verified 8c22f2e in an isolated worktree, with Batfish running.

tests/test_explain_output_check_gap.py: 13 passed, matching the PR body. Mutation-tested by reverting ai/explain.py to the parent commit: 10 failed, 3 passed, matching the PR body exactly (both red-on-purpose counts and the control test still passing). Restored and re-confirmed 13 passed. Full suite on the branch: 2255 passed, 13 skipped, 0 failed, matching your number exactly.

Checked every example in the PR body's own false-positive section live: all four sentences you say would be wrongly rejected do get rejected, and all three you say should not match, do not match. Nothing in that section overstates what the pattern does.

One real gap, since you asked specifically about the phrase list's shape. This is the exact wording from the research question Samika and I looked at together: 'There are no security problems with this configuration.' does NOT match _ALL_CLEAR_PATTERN, because the inserted word 'security' breaks the literal phrase 'no problems'. I checked this live:

'There are no problems with this rule.' MATCH
'There are no security problems with this ...' NO MATCH
'The scan found no real issues of concern.' NO MATCH

So the specific case that started this whole line of research is still not caught by the fix meant to close it. This isn't a surprise regression, your own 'Not done' section already says other all-clear wording will get through, and I'm not asking you to solve open-ended paraphrase detection in this PR. But this particular phrase is not just 'other wording', it's the one that was actually measured getting through in the first place, so I think it's worth naming explicitly rather than folding into the general disclaimer. I'll file a follow-up issue for a version of the pattern that tolerates one inserted word (something like no (\w+\s+)?problems) rather than block this PR on it.

Approving. The fix does exactly what it claims, is proven by a real mutation test, and the false-positive measurement holds up under my own independent check.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 8c22f2e, with two small tests I'd like added before merge. Red tests first, then the fix, plus false positives measured on 23 real findings: this is how a safety-net change should arrive.

Checked:

  • Merged onto current main (2833a1e): 2324 passed, 13 skipped, ruff clean on CI's selection.
  • Removing the found check fails 6. The gap is pinned.
  • One more false-positive sample, a kind your seven fixtures don't stress: 100 real netwise-warden explanations of dead-rule findings, from this week's #401 and #461 measurements. Dead rules are where I'd expect "this line is safe to remove". 0 of 100 match any of the new phrases, and none say "safe to". That supports your measurement from a different angle.

Two mutants survive, both cheap to kill:

  1. Checking none findings too (status in ("found", "none")) passes every test. The description makes a point of not checking none, because there an all-clear is the truth, but nothing pins it. One test would: a none finding explained as "No issues were found by policy compliance." must be accepted.
  2. Dropping is safe from the pattern passes every test. Each of the seven phrases deserves one rejected found sentence of its own, e.g. as a parametrised test, so the list can't quietly shrink.

Neither blocks the idea, which is right. They're just the two places where the code currently has no tests protecting it.

@SamikaPerera
SamikaPerera merged commit ee769b4 into main Nov 12, 2026
2 checks passed
@SamikaPerera
SamikaPerera deleted the test/explain-output-check-gap branch November 12, 2026 22:26
patelankeet2 added a commit that referenced this pull request Nov 19, 2026
…each-phrase

Add the two tests asked for on #456 (none findings accepted, each phrase refused)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants