Skip to content

A negated reachability question answers the negation, not the plain question (#435) - #472

Open
patelankeet2 wants to merge 1 commit into
mainfrom
fix/435-negated-reachability
Open

patelankeet2 wants to merge 1 commit into
mainfrom
fix/435-negated-reachability

Conversation

@patelankeet2

@patelankeet2 patelankeet2 commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Closes the negation third of #435 (one of three gaps that issue lists; port and protocol constraints are deliberately untouched here -- see "Not done" below).

What's wrong

ai/query.py splits a reachability question on the first reach-style keyword ("reach", "access", ...) into source_text and destination_text. "Can rtr-us5 NOT reach 218.8.104.58?" splits into source_text = "Can rtr-us5 NOT ". _find_device() only looks for a known device name inside that text and ignores everything else, so the word "NOT" was silently dropped rather than acted on or refused. The question was answered identically to "Can rtr-us5 reach 218.8.104.58?" -- grounded: true, full confidence, the exact opposite of what was asked.

Measured live against real Batfish (tests/fixtures/rtr-us5-insecure), before this fix:

Can rtr-us5 reach 10.10.10.5?       -> Yes. Traffic from rtr-us5 reaches 10.10.10.5.
Can rtr-us5 NOT reach 10.10.10.5?   -> Yes. Traffic from rtr-us5 reaches 10.10.10.5.   (identical, wrong)

What changed

ai/query.py only:

  • _NEGATION_BEFORE_REACH: a closed set of negation words (not, never, cannot, can't, doesn't, does not, won't, will not), anchored to the END of source_text -- the same discipline _REACH_KEYWORDS and the back-reference sets already use, so a question naming something unrelated earlier ("the router that is not in maintenance, rtr-us5, reach X") never falsely triggers it.
  • _is_negated_reachability(source_text): the detector itself, one line.
  • _answer_reachability_question() computes negated once, right after the split, and threads it through to two places:
    • question_understood now says "Does {device} NOT reach {destination}?" when negated, instead of always rendering the affirmative phrasing regardless of what was actually asked. This is the part that matters most: shape C (showing the question back) only does its job of surfacing a mistranslation if what it shows back is honest about the negation.
    • _describe_traces() takes a negated flag and flips the Yes/No framing on the two clean-cut outcomes (all traces succeed / all traces fail) to answer the literal negated question, over the exact same Batfish evidence. The mixed-result case (traced paths disagree) is unchanged -- it already has no yes/no word to get backwards.

Live, after this fix

Can rtr-us5 reach 10.10.10.5?
  question_understood: Can rtr-us5 reach 10.10.10.5?
  answer: Yes. Traffic from rtr-us5 reaches 10.10.10.5.

Can rtr-us5 NOT reach 10.10.10.5?
  question_understood: Does rtr-us5 NOT reach 10.10.10.5?
  answer: No -- traffic from rtr-us5 DOES reach 10.10.10.5, which is the opposite of what you asked (rtr-us5 NOT reaching it).
Can rtr-us5 reach 218.8.104.58?
  answer: No. Traffic from rtr-us5 to 218.8.104.58 ends in NO_ROUTE. Path: rtr-us5.

Can rtr-us5 NOT reach 218.8.104.58?
  question_understood: Does rtr-us5 NOT reach 218.8.104.58?
  answer: Yes -- traffic from rtr-us5 to 218.8.104.58 does NOT reach it; it ends in NO_ROUTE. Path: rtr-us5.

Not done (on purpose)

#435 lists three gaps found in the same review: negation, a dropped port constraint, and a dropped protocol constraint. This PR fixes only the negation case. Matching this project's one-shape-at-a-time discipline for an inversion fix (#52, #145, #384 before it), and #435's own suggested next step, the port/protocol half is left for its own follow-up rather than bundled in here -- it needs a different mechanism (either threading the constraint into HeaderConstraints, or refusing per the "narrow or refuse, never silently drop" rule analysis/policy_conflicts.py already applies elsewhere), and bundling it would make this change bigger and harder to review for what it actually fixes.

Testing

tests/test_query_translation.py, 14 new tests:

  • A negated question that does reach answers "No", DOES-reach framing.
  • A negated question that does not reach answers "Yes", does-NOT-reach framing.
  • question_understood states the negation back explicitly.
  • A plain (non-negated) question renders exactly as before (regression).
  • All 7 phrasings in the closed set recognised (parametrized, 7 cases, deliberately ungrammatical combinations included -- the point is each word in the set is recognised independently).
  • An unrelated earlier "not" in the sentence does not falsely trigger negation (the anchor-to-the-end guard).
  • A negated mixed-result question still reports plainly, unaffected.
  • A negated back-reference ("does it NOT reach X") still refuses rather than guessing -- confirmed as the safe degradation this fix does not need to close, not a gap.
pytest tests/test_query_translation.py -v
  71 passed (57 existing + 14 new)

pytest tests/ -q   (Batfish running)
  2338 passed, 13 skipped, 0 failed

Mutation-tested both new pieces directly:

  • Removed the end-of-string anchor from _NEGATION_BEFORE_REACH (so "not" anywhere in the sentence triggers it) -- test_an_unrelated_earlier_not_does_not_falsely_trigger_negation failed, reproducing the exact false-positive this anchor exists to prevent.
  • Forced _is_negated_reachability() to always return False -- all 10 negation-dependent tests failed, reverting to the original (wrong) un-negated behaviour.

Both reverted after confirming.

Live-reproduced against real Batfish (tests/fixtures/rtr-us5-insecure, not a fake session), both directions, shown above under "Live, after this fix".

ruff --select E4,E7,E9,F clean on both changed files.

Acceptance criteria (from #435, negation portion only)


Correction, 16 November. The testing section above originally said "18 new tests" and "8 phrasings" -- wrong on both counts. Recounted directly: test_query_translation.py collected 57 items on main before this branch and 71 on this branch, a difference of 14, not 18, from 4 standalone tests plus one test parametrized over 7 phrasings (not 8) plus 3 more standalone tests. Fixed above rather than left for a reviewer to catch.

…uestion (#435)

"Can X NOT reach Y" was answered identically to "Can X reach Y" -- the
word NOT sat inside source_text, where _find_device() ignores anything
that is not a known device name, so it was dropped silently rather than
refused. Measured in the issue: grounded true, full confidence, a full
inversion of what was asked.

Added a closed set of negation words anchored to the end of source_text
(_NEGATION_BEFORE_REACH / _is_negated_reachability), matching the same
discipline _REACH_KEYWORDS and the back-reference sets already use.
question_understood now states the negation back explicitly, and the
answer text flips Yes/No to match what was actually asked, over the same
Batfish evidence -- the mixed-result case is unaffected since it already
has no yes/no word to get backwards.

Port and protocol constraints are a separate, still-open part of #435,
not touched here on purpose -- one shape at a time, same as #52/#145/#384.
@patelankeet2 patelankeet2 added this to the Sprint 6 milestone Nov 15, 2026
@patelankeet2 patelankeet2 added bug Something isn't working layer: ai Layer 2 — local LLM explanation and Q&A safety Touches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care labels Nov 15, 2026
@patelankeet2 patelankeet2 self-assigned this Nov 15, 2026

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 6fcd804. This closes the worst of what I found on #407: a negated question answered as the plain one, with full confidence. Showing the negation back in question_understood is the part that matters most, as you say.

Merged onto current main (ee769b4): 2338 passed, 13 skipped, ruff clean on CI's selection. Mutants: the detector always False fails 10, always True fails 18.

Ten phrasings through answer_question(), on Q01's own fake session (rtr-us5 → 218.8.104.58, ACCEPTED):

Can rtr-us5 NOT reach … / not / never            -> "Does rtr-us5 NOT reach …?"  "No -- traffic … DOES reach …"   right
Why does rtr-us5 not reach …?                     -> negated                                                        right
Is it true that rtr-us5 cannot reach …?           -> negated                                                        right
rtr-us5 can't reach …, right?                     -> negated                                                        right
Doesn't rtr-us5 reach …?                          -> plain question, "Yes … reaches"     right: rhetorical, it asks whether it does
Why can't rtr-us5 reach …?                        -> plain question, "Yes … reaches"     see 1
Can it not reach …? (after a first question)      -> refused, no device found            see 2

Two gaps left, neither wrong, both worth a line on #435:

  1. "Why can't rtr-us5 reach X?", from my #407 list. The negation comes before the device, so the end-anchored pattern misses it. The answer is truthful, and question_understood shows the reinterpretation, so this isn't a confident wrong answer like before. But it's the exact phrasing #435's negation bullet names, so it shouldn't read as closed.
  2. "Can it not reach X?": with not after the pronoun, the back-reference doesn't resolve and the question is refused. Refusing is the safe direction; it's just less helpful than the device-named form.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 6fcd804, merged onto today's main. The fix does what it says for the phrasings it lists, and I could not make it worse than main. But five other ways of asking the same question still get the inverted answer, grounded: True, so I'm requesting changes. The fix is small and tested below.

What I checked

  • CI-style suite on the merge: green. Ruff clean.
  • Live, rtr-us5-insecure, through answer_question(): "Can rtr-us5 NOT reach 10.10.10.5?" now says Does rtr-us5 NOT reach 10.10.10.5? and answers "No -- traffic ... DOES reach ...". "never", "cannot" and "can't" directly before the reach word work the same way.

The gap. _NEGATION_BEFORE_REACH only reads a negation directly before the reach word. Anything with a word between them still lands in source_text where _find_device() ignores it, which is exactly the bug #435 describes, and the answer is again the plain question's:

Can rtr-us5 not be able to reach 10.10.10.5?   -> Yes. Traffic from rtr-us5 reaches 10.10.10.5.   grounded=True
Is rtr-us5 unable to reach 10.10.10.5?         -> Yes. ...                                         grounded=True
Does rtr-us5 fail to reach 10.10.10.5?         -> Yes. ...                                         grounded=True
Can rtr-us5 no longer reach 10.10.10.5?        -> Yes. ...                                         grounded=True
Can rtr-us5 not really reach 10.10.10.5?       -> Yes. ...                                         grounded=True
Why can't rtr-us5 reach 10.10.10.5?            -> Yes. ...                                         grounded=True

Each question_understood reads "Can rtr-us5 reach 10.10.10.5?", so the person who asked has nothing to notice. This is not a regression: main answers all of them the same way. But CLAUDE.md section 7c says the layer refuses rather than approximates, and a closed set that fails open is an approximation. "Unable to reach" is also the more natural way to ask than "NOT reach".

Suggested fix, tested. A second, closed pattern for a negation word with at most three plain words after it before the reach word, and a refusal when it matches and the first set did not. A comma ends the window, so your own test ("the router that is not in maintenance, rtr-us5, reach ...") still passes untouched.

  • The new test fails on 6fcd804 for all six phrasings (6 failed) and passes with the change (77 passed in that file).
  • Live, the six now say "This question looks negated, but not in a form I can read ... Ask it as "Can NOT reach ?", or as the plain question", grounded=False.
  • Full suite on main + this PR + the patch: 2473 passed, 63 skipped; ruff clean.

Not covered, and not claimed: a negation after the address ("Can rtr-us5 reach 10.10.10.5? I expect not.") is a statement, not part of the question, and is still answered as the plain question.

Patch (applies on main + #472)
diff --git a/ai/query.py b/ai/query.py
index 934aa38..ccf5e21 100644
--- a/ai/query.py
+++ b/ai/query.py
@@ -242,6 +242,19 @@ _NEGATION_BEFORE_REACH = re.compile(
     r"\b(?:not|never|cannot|can't|doesn't|does\s+not|won't|will\s+not)\s*$",
     re.IGNORECASE,
 )
+# A negation that is NEAR the reach word but not directly before it ("not be able to
+# reach", "unable to reach", "fail to reach", "no longer reach", "not really
+# reach", "why can't rtr-us5 reach"). _NEGATION_BEFORE_REACH does not recognise
+# these, and without this they were answered as the PLAIN question: the same
+# inversion #435 fixes, reached by a phrasing the closed set misses (#472
+# review). A closed set that fails open is a guess, so these are REFUSED. At
+# most three plain words may follow the negation, and a comma ends the window,
+# so "the router that is not in maintenance, rtr-us5, reach X" is untouched.
+_NEGATION_NEAR_REACH = re.compile(
+    r"\b(?:not|never|cannot|unable|fails?|failed|no\s+longer|\w+n't)\b\s+"
+    r"(?:[\w-]+\s+){0,3}$",
+    re.IGNORECASE,
+)
 _DEAD_RULE_KEYWORDS = re.compile(
     r"\b(?:dead|unreachable|shadow(?:ed)?|never\s+(?:take|takes)\s+effect)\b"
     r".*\b(?:rules?|lines?|acls?)\b"
@@ -574,6 +587,14 @@ def _answer_reachability_question(
 ) -> Dict[str, Any]:
     source_text, destination_text = split
     negated = _is_negated_reachability(source_text)
+    if not negated and _NEGATION_NEAR_REACH.search(source_text):
+        return _refuse(
+            "This question looks negated, but not in a form I can read: "
+            f"{source_text.strip()!r}. Answering it as the plain question would "
+            "give the opposite of what was asked, so I will not guess. Ask it "
+            "as \"Can <device> NOT reach <address>?\", or as the plain "
+            "question."
+        )
 
     present = snapshot.device_names(bf)
     if present is None:
diff --git a/tests/test_query_translation.py b/tests/test_query_translation.py
index 905bba1..0a0ae2e 100644
--- a/tests/test_query_translation.py
+++ b/tests/test_query_translation.py
@@ -425,6 +425,29 @@ def test_an_unrelated_earlier_not_does_not_falsely_trigger_negation():
     assert result["question_understood"] == "Can rtr-us5 reach 218.8.104.58?"
 
 
+@pytest.mark.parametrize("phrasing", [
+    "Can rtr-us5 not be able to reach 218.8.104.58?",
+    "Is rtr-us5 unable to reach 218.8.104.58?",
+    "Does rtr-us5 fail to reach 218.8.104.58?",
+    "Can rtr-us5 no longer reach 218.8.104.58?",
+    "Can rtr-us5 not really reach 218.8.104.58?",
+    "Why can't rtr-us5 reach 218.8.104.58?",
+])
+def test_a_negation_the_closed_set_does_not_read_is_refused_not_answered_plain(phrasing):
+    """#472 review. The closed set only reads a negation DIRECTLY before the
+    reach word, so each of these was answered as the plain question, grounded
+    and confident: the inversion #435 exists to remove, reached by a phrasing
+    the set misses. A closed set that fails open is a guess, so a negation
+    near the reach word that is not recognised is refused."""
+    frame = _FakeFrame([{"Traces": [FakeTrace("ACCEPTED")]}])
+    session = _session_with_devices({"rtr-us5"}, traceroute_frame=frame)
+    result = answer_question(phrasing, session)
+    assert result["grounded"] is False
+    assert result["question_understood"] is None
+    assert "opposite" in result["answer"]
+    assert not result["answer"].startswith(("Yes", "No"))
+
+
 def test_a_negated_mixed_result_still_reports_plainly():
     """The mixed case has no yes/no word to get backwards -- confirm
     negation leaves it untouched rather than prefixing something

Please re-request me when it is up. Everything else in the PR is fine, including the _describe_traces change for the mixed case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working layer: ai Layer 2 — local LLM explanation and Q&A safety Touches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants