Repository navigation
A negated reachability question answers the negation, not the plain question (#435) - #472
patelankeet2 wants to merge 1 commit into
Conversation
…uestion (#435) "Can X NOT reach Y" was answered identically to "Can X reach Y" -- the word NOT sat inside source_text, where _find_device() ignores anything that is not a known device name, so it was dropped silently rather than refused. Measured in the issue: grounded true, full confidence, a full inversion of what was asked. Added a closed set of negation words anchored to the end of source_text (_NEGATION_BEFORE_REACH / _is_negated_reachability), matching the same discipline _REACH_KEYWORDS and the back-reference sets already use. question_understood now states the negation back explicitly, and the answer text flips Yes/No to match what was actually asked, over the same Batfish evidence -- the mixed-result case is unaffected since it already has no yes/no word to get backwards. Port and protocol constraints are a separate, still-open part of #435, not touched here on purpose -- one shape at a time, same as #52/#145/#384.
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Approving 6fcd804. This closes the worst of what I found on #407: a negated question answered as the plain one, with full confidence. Showing the negation back in question_understood is the part that matters most, as you say.
Merged onto current main (ee769b4): 2338 passed, 13 skipped, ruff clean on CI's selection. Mutants: the detector always False fails 10, always True fails 18.
Ten phrasings through answer_question(), on Q01's own fake session (rtr-us5 → 218.8.104.58, ACCEPTED):
Can rtr-us5 NOT reach … / not / never -> "Does rtr-us5 NOT reach …?" "No -- traffic … DOES reach …" right
Why does rtr-us5 not reach …? -> negated right
Is it true that rtr-us5 cannot reach …? -> negated right
rtr-us5 can't reach …, right? -> negated right
Doesn't rtr-us5 reach …? -> plain question, "Yes … reaches" right: rhetorical, it asks whether it does
Why can't rtr-us5 reach …? -> plain question, "Yes … reaches" see 1
Can it not reach …? (after a first question) -> refused, no device found see 2
Two gaps left, neither wrong, both worth a line on #435:
- "Why can't rtr-us5 reach X?", from my #407 list. The negation comes before the device, so the end-anchored pattern misses it. The answer is truthful, and
question_understoodshows the reinterpretation, so this isn't a confident wrong answer like before. But it's the exact phrasing #435's negation bullet names, so it shouldn't read as closed. - "Can it not reach X?": with
notafter the pronoun, the back-reference doesn't resolve and the question is refused. Refusing is the safe direction; it's just less helpful than the device-named form.
ARSH871-bot
left a comment
There was a problem hiding this comment.
Reviewed 6fcd804, merged onto today's main. The fix does what it says for the phrasings it lists, and I could not make it worse than main. But five other ways of asking the same question still get the inverted answer, grounded: True, so I'm requesting changes. The fix is small and tested below.
What I checked
- CI-style suite on the merge: green. Ruff clean.
- Live,
rtr-us5-insecure, throughanswer_question(): "Can rtr-us5 NOT reach 10.10.10.5?" now saysDoes rtr-us5 NOT reach 10.10.10.5?and answers "No -- traffic ... DOES reach ...". "never", "cannot" and "can't" directly before the reach word work the same way.
The gap. _NEGATION_BEFORE_REACH only reads a negation directly before the reach word. Anything with a word between them still lands in source_text where _find_device() ignores it, which is exactly the bug #435 describes, and the answer is again the plain question's:
Can rtr-us5 not be able to reach 10.10.10.5? -> Yes. Traffic from rtr-us5 reaches 10.10.10.5. grounded=True
Is rtr-us5 unable to reach 10.10.10.5? -> Yes. ... grounded=True
Does rtr-us5 fail to reach 10.10.10.5? -> Yes. ... grounded=True
Can rtr-us5 no longer reach 10.10.10.5? -> Yes. ... grounded=True
Can rtr-us5 not really reach 10.10.10.5? -> Yes. ... grounded=True
Why can't rtr-us5 reach 10.10.10.5? -> Yes. ... grounded=True
Each question_understood reads "Can rtr-us5 reach 10.10.10.5?", so the person who asked has nothing to notice. This is not a regression: main answers all of them the same way. But CLAUDE.md section 7c says the layer refuses rather than approximates, and a closed set that fails open is an approximation. "Unable to reach" is also the more natural way to ask than "NOT reach".
Suggested fix, tested. A second, closed pattern for a negation word with at most three plain words after it before the reach word, and a refusal when it matches and the first set did not. A comma ends the window, so your own test ("the router that is not in maintenance, rtr-us5, reach ...") still passes untouched.
- The new test fails on
6fcd804for all six phrasings (6 failed) and passes with the change (77 passed in that file). - Live, the six now say "This question looks negated, but not in a form I can read ... Ask it as "Can NOT reach ?", or as the plain question",
grounded=False. - Full suite on main + this PR + the patch: 2473 passed, 63 skipped; ruff clean.
Not covered, and not claimed: a negation after the address ("Can rtr-us5 reach 10.10.10.5? I expect not.") is a statement, not part of the question, and is still answered as the plain question.
Patch (applies on main + #472)
diff --git a/ai/query.py b/ai/query.py
index 934aa38..ccf5e21 100644
--- a/ai/query.py
+++ b/ai/query.py
@@ -242,6 +242,19 @@ _NEGATION_BEFORE_REACH = re.compile(
r"\b(?:not|never|cannot|can't|doesn't|does\s+not|won't|will\s+not)\s*$",
re.IGNORECASE,
)
+# A negation that is NEAR the reach word but not directly before it ("not be able to
+# reach", "unable to reach", "fail to reach", "no longer reach", "not really
+# reach", "why can't rtr-us5 reach"). _NEGATION_BEFORE_REACH does not recognise
+# these, and without this they were answered as the PLAIN question: the same
+# inversion #435 fixes, reached by a phrasing the closed set misses (#472
+# review). A closed set that fails open is a guess, so these are REFUSED. At
+# most three plain words may follow the negation, and a comma ends the window,
+# so "the router that is not in maintenance, rtr-us5, reach X" is untouched.
+_NEGATION_NEAR_REACH = re.compile(
+ r"\b(?:not|never|cannot|unable|fails?|failed|no\s+longer|\w+n't)\b\s+"
+ r"(?:[\w-]+\s+){0,3}$",
+ re.IGNORECASE,
+)
_DEAD_RULE_KEYWORDS = re.compile(
r"\b(?:dead|unreachable|shadow(?:ed)?|never\s+(?:take|takes)\s+effect)\b"
r".*\b(?:rules?|lines?|acls?)\b"
@@ -574,6 +587,14 @@ def _answer_reachability_question(
) -> Dict[str, Any]:
source_text, destination_text = split
negated = _is_negated_reachability(source_text)
+ if not negated and _NEGATION_NEAR_REACH.search(source_text):
+ return _refuse(
+ "This question looks negated, but not in a form I can read: "
+ f"{source_text.strip()!r}. Answering it as the plain question would "
+ "give the opposite of what was asked, so I will not guess. Ask it "
+ "as \"Can <device> NOT reach <address>?\", or as the plain "
+ "question."
+ )
present = snapshot.device_names(bf)
if present is None:
diff --git a/tests/test_query_translation.py b/tests/test_query_translation.py
index 905bba1..0a0ae2e 100644
--- a/tests/test_query_translation.py
+++ b/tests/test_query_translation.py
@@ -425,6 +425,29 @@ def test_an_unrelated_earlier_not_does_not_falsely_trigger_negation():
assert result["question_understood"] == "Can rtr-us5 reach 218.8.104.58?"
+@pytest.mark.parametrize("phrasing", [
+ "Can rtr-us5 not be able to reach 218.8.104.58?",
+ "Is rtr-us5 unable to reach 218.8.104.58?",
+ "Does rtr-us5 fail to reach 218.8.104.58?",
+ "Can rtr-us5 no longer reach 218.8.104.58?",
+ "Can rtr-us5 not really reach 218.8.104.58?",
+ "Why can't rtr-us5 reach 218.8.104.58?",
+])
+def test_a_negation_the_closed_set_does_not_read_is_refused_not_answered_plain(phrasing):
+ """#472 review. The closed set only reads a negation DIRECTLY before the
+ reach word, so each of these was answered as the plain question, grounded
+ and confident: the inversion #435 exists to remove, reached by a phrasing
+ the set misses. A closed set that fails open is a guess, so a negation
+ near the reach word that is not recognised is refused."""
+ frame = _FakeFrame([{"Traces": [FakeTrace("ACCEPTED")]}])
+ session = _session_with_devices({"rtr-us5"}, traceroute_frame=frame)
+ result = answer_question(phrasing, session)
+ assert result["grounded"] is False
+ assert result["question_understood"] is None
+ assert "opposite" in result["answer"]
+ assert not result["answer"].startswith(("Yes", "No"))
+
+
def test_a_negated_mixed_result_still_reports_plainly():
"""The mixed case has no yes/no word to get backwards -- confirm
negation leaves it untouched rather than prefixing somethingPlease re-request me when it is up. Everything else in the PR is fine, including the _describe_traces change for the mixed case.
Closes the negation third of #435 (one of three gaps that issue lists; port and protocol constraints are deliberately untouched here -- see "Not done" below).
What's wrong
ai/query.pysplits a reachability question on the first reach-style keyword ("reach", "access", ...) intosource_textanddestination_text. "Can rtr-us5 NOT reach 218.8.104.58?" splits intosource_text = "Can rtr-us5 NOT "._find_device()only looks for a known device name inside that text and ignores everything else, so the word "NOT" was silently dropped rather than acted on or refused. The question was answered identically to "Can rtr-us5 reach 218.8.104.58?" --grounded: true, full confidence, the exact opposite of what was asked.Measured live against real Batfish (
tests/fixtures/rtr-us5-insecure), before this fix:What changed
ai/query.pyonly:_NEGATION_BEFORE_REACH: a closed set of negation words (not,never,cannot,can't,doesn't,does not,won't,will not), anchored to the END ofsource_text-- the same discipline_REACH_KEYWORDSand the back-reference sets already use, so a question naming something unrelated earlier ("the router that is not in maintenance, rtr-us5, reach X") never falsely triggers it._is_negated_reachability(source_text): the detector itself, one line._answer_reachability_question()computesnegatedonce, right after the split, and threads it through to two places:question_understoodnow says"Does {device} NOT reach {destination}?"when negated, instead of always rendering the affirmative phrasing regardless of what was actually asked. This is the part that matters most: shape C (showing the question back) only does its job of surfacing a mistranslation if what it shows back is honest about the negation._describe_traces()takes anegatedflag and flips the Yes/No framing on the two clean-cut outcomes (all traces succeed / all traces fail) to answer the literal negated question, over the exact same Batfish evidence. The mixed-result case (traced paths disagree) is unchanged -- it already has no yes/no word to get backwards.Live, after this fix
Not done (on purpose)
#435 lists three gaps found in the same review: negation, a dropped port constraint, and a dropped protocol constraint. This PR fixes only the negation case. Matching this project's one-shape-at-a-time discipline for an inversion fix (#52, #145, #384 before it), and #435's own suggested next step, the port/protocol half is left for its own follow-up rather than bundled in here -- it needs a different mechanism (either threading the constraint into
HeaderConstraints, or refusing per the "narrow or refuse, never silently drop" ruleanalysis/policy_conflicts.pyalready applies elsewhere), and bundling it would make this change bigger and harder to review for what it actually fixes.Testing
tests/test_query_translation.py, 14 new tests:question_understoodstates the negation back explicitly.Mutation-tested both new pieces directly:
_NEGATION_BEFORE_REACH(so "not" anywhere in the sentence triggers it) --test_an_unrelated_earlier_not_does_not_falsely_trigger_negationfailed, reproducing the exact false-positive this anchor exists to prevent._is_negated_reachability()to always returnFalse-- all 10 negation-dependent tests failed, reverting to the original (wrong) un-negated behaviour.Both reverted after confirming.
Live-reproduced against real Batfish (
tests/fixtures/rtr-us5-insecure, not a fake session), both directions, shown above under "Live, after this fix".ruff --select E4,E7,E9,Fclean on both changed files.Acceptance criteria (from #435, negation portion only)
question_understoodreflects the negation, so a mistranslation would be visible.Correction, 16 November. The testing section above originally said "18 new tests" and "8 phrasings" -- wrong on both counts. Recounted directly:
test_query_translation.pycollected 57 items onmainbefore this branch and 71 on this branch, a difference of 14, not 18, from 4 standalone tests plus one test parametrized over 7 phrasings (not 8) plus 3 more standalone tests. Fixed above rather than left for a reviewer to catch.