Skip to content

Dead-rule fact: stop claiming a universal outcome Batfish cannot guarantee (#453) - #461

Open
patelankeet2 wants to merge 3 commits into
feature/384-dead-rule-own-actionfrom
fix/453-dead-rule-fact-completeness
Open

patelankeet2 wants to merge 3 commits into
feature/384-dead-rule-own-actionfrom
fix/453-dead-rule-fact-completeness

Conversation

@patelankeet2

@patelankeet2 patelankeet2 commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #453.

Stacked on #401 (feature/384-dead-rule-own-action) rather than main: #401 adds the contradicted/redundant branching this fix changes, and that shape does not exist on main yet. Basing on #401's branch avoids a second, conflicting change to the same function. Once #401 merges, GitHub retargets this PR onto main automatically.

What's wrong: _compute_dead_rule_outcome()'s contradicted-action branch told the model, as already verified and certain, that "traffic matching it is actually {outcome}" for the dead line's ENTIRE matching pattern. Measured live, same on main db59b00 and #401's 246db8e:

permit tcp host 10.1.1.5 any
deny   tcp 10.1.1.0 0.0.0.255 any
permit tcp 10.1.1.0 0.0.0.255 any eq 443      <- dead

Blocked by:  deny tcp 10.1.1.0 0.0.0.255 any
Fact:        traffic matching it is actually denied
testFilters, tcp/443 to 8.8.8.8:   10.1.1.5 PERMIT    10.1.1.200 DENY

Batfish's Blocking_Lines does not always name every earlier line -- specifically, a line sharing the DEAD line's own action is not always listed, and when one exists it can decide a narrower slice of traffic the opposite way from what the named, opposite-action blocking line alone predicts.

What I checked before fixing it:

  • Confirmed only the contradicted branch is unsafe. The redundant branch (blocking line agrees with the dead line's own action) is unaffected -- A computed dead-rule fact can be false: Batfish can leave a smaller earlier line out of Blocking_Lines #453's own issue quotes Name the dead line's own stated action explicitly, not just the outcome (#384) #401's review testing on this (8 cases built to break it, 6 returned None, 2 had no overlap and the sentence was true). Worked out why: if an omission is always a same-action-as-the-dead-line line, it cannot change the outcome when every alternative, named or not, already agrees.
  • Confirmed the issue's own suggested wording ("name the line Batfish names, X, denies this") is not, by itself, enough. The omitted line in the measured case is earlier than the NAMED blocking line, not narrower than the dead line's pattern -- host .5 matches the named deny's /24 too. Narrowing the subject to "traffic matching the named line" doesn't fix it.

What changed: dropped the universal "actually {outcome}" claim for the contradicted branch entirely. States only the two facts that stay true regardless of what Blocking_Lines could have left out: the dead line's own action never fires (filterLineReachability verifies that exhaustively across the whole match space -- that part of "unreachable" is sound), and the named earlier line genuinely is evaluated first and genuinely does have the opposite action. Example on the measured case: "the unreachable line's own stated action is PERMIT, but that action never takes effect: an earlier line (deny tcp 10.1.1.0 0.0.0.255 any), which is evaluated first, has the opposite action (DENY) instead." No claim about host 10.1.1.5 or any other specific packet.

Testing:

  • Updated the four existing tests that pinned the old universal-outcome wording to the new one, and added a new test reproducing A computed dead-rule fact can be false: Batfish can leave a smaller earlier line out of Blocking_Lines #453's own measured shape directly.
  • pytest tests/test_explain_safety_net.py -v -- 96 passed.
  • Mutation test: reverted to the old wording. All 7 affected tests failed for the right reason; the two redundant-branch tests were unaffected either way, confirming that branch genuinely does not need this fix. Restored.
  • Full suite: pytest tests/ -q -- 2266 passed, 13 skipped, 0 failed.
  • Live-tested against the real model: 15 generations on the measured contradicted-case finding. 0/15 fallbacks, and the wording correctly follows the new, narrower fact in every non-copy generation. (A separate 10/15-ish rate of Modelfile-Example-1 copies showed up in this live check -- that's this worktree not yet having The model can copy Modelfile Example 1 word for word, and the check passes it #455/_ALL_CLEAR_PATTERN still misses the exact phrase that started this (#456 follow-up) #457's separate, not-yet-merged fixes, not something this change caused.)
  • Re-captured and re-rated AC-002 in the held-out evaluation set, since this change moves the exact prompt text fed to the model for that finding (caught by tests/test_explanation_quality.py's own prompt-hash guard). docs/evaluation.md and the holdout fixture both updated, with a request for Shubham and Arsh to rate the new capture.
  • Byte-level check: no new em-dashes in any changed file (confirmed against my own diff, not the whole file -- docs/evaluation.md already has its own pre-existing em-dash convention throughout).

Update, 11 November -- Shubham's review addressed

Shubham measured the fallback path this PR's second commit (e3ec2fd) added -- for a blocking line whose action is known but whose shape _describe_acl_line() can't render in plain English -- and found it causes 10/20 Modelfile Example 1 copies, against 1/20 for #401's main-head wording and 0/20 for no computed fact at all. The fallback's sentence states the named line's own action but no outcome, and that gap got filled with Example 1 more often than either alternative.

Fixed as he suggested: that fallback now returns None instead of naming the line in raw syntax, the same refusal this function already makes a few lines up for unrecognised syntax and disagreeing blocking lines. The explanatory comment about why the dropped universal-outcome claim was unsafe stays (it is still true and still the reason #453 exists), rewritten to also say why a narrower fallback sentence turned out not to be safe either.

Not touched: this doesn't change the primary path (both/all blocking lines parse via _describe_acl_line()) at all -- that's the part #453's own measurement and this PR's existing tests already cover, and it is unaffected by this fallback-only fix.

Testing:

  • Rewrote test_an_unparseable_blocking_line_falls_back_without_a_universal_claim (now ..._returns_none_rather_than_a_weaker_claim) to assert None on the exact unparseable fixture.
  • Added test_one_unparseable_line_among_several_blocking_lines_also_returns_none, a mixed-list case (one parseable blocking line, one not) nothing pinned before.
  • pytest tests/test_explain_safety_net.py -q -- 104 passed.
  • Mutation test: restored the old raw-syntax fallback sentence. Both new/rewritten tests failed, reproducing the exact old wording in the assertion diff. Reverted.
  • Full suite: pytest tests/ -q (Batfish running) -- 2324 passed, 13 skipped, 0 failed.
  • Live-measured, 20 generations on Shubham's exact unparseable fixture: 0/20 Example 1 copies (19/20 from the model, 1/20 fallback-to-Ollama-unreachable on that one run), matching his "no computed fact at all" row exactly.
  • Checked docs/evaluation.md and the holdout fixture: neither needs a re-capture from this change. AC-002's own capture note already says its blocking line always parses via _describe_acl_line(), so it never reaches this fallback, and no other held-out entry hits BLOCKING_LINES with an unparseable line either.
  • Found while reading all 20 live outputs, reported honestly though out of scope here: several non-copy generations still misread the blocking line's source port (1024) as an unusual HTTPS port, or describe the permit and deny as "its own" rule rather than two separate lines. Neither is Example 1 copying and neither is new -- same prompt shape as the rest of this function's primary path -- flagging in case it's worth its own issue.

@shubhamkataria2005 -- re-requesting.

@patelankeet2 patelankeet2 added bug Something isn't working layer: ai Layer 2 — local LLM explanation and Q&A safety Touches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care labels Nov 8, 2026
@patelankeet2 patelankeet2 self-assigned this Nov 8, 2026
…antee (#453)

_compute_dead_rule_outcome()'s contradicted-action branch (own action vs.
blocking line's action disagree) told the model, as already verified and
certain, that 'traffic matching it is actually {outcome}' for the dead
line's ENTIRE matching pattern. Measured live (same on main db59b00 and
-- specifically, a line sharing the DEAD line's own action is not always
listed, and when one exists it can decide a narrower slice of traffic
the opposite way from what the named, opposite-action blocking line
alone predicts. A real config had permit host 10.1.1.5 any before both
the named deny and the dead permit; the fact said the whole /24 is
denied, but testFilters showed host .5 is actually permitted.

Confirmed only the contradicted branch is unsafe. The redundant branch
(blocking line agrees with the dead line's own action) is unaffected --
break it, 6 returned None (Batfish named the overlapping line), 2 had no
such overlap and the sentence was true. Worked out why: if an omission
is always a same-action-as-the-dead-line line, it cannot change the
outcome when every alternative (named or not) already agrees.

Also confirmed naming the blocking line's own text is not, by itself,
enough (the issue's suggested wording): the omitted line in the measured
case is earlier than the NAMED blocking line, not narrower than the dead
line's pattern -- host .5 matches the named deny's /24 too. The fix
drops the universal 'actually {outcome}' claim for this branch entirely,
stating only the two facts that stay true regardless of what
Blocking_Lines could have left out: the dead line's own action never
fires (filterLineReachability verifies that exhaustively), and the
named earlier line genuinely is evaluated first and genuinely does have
the opposite action.

Stacked on #401's branch (feature/384-dead-rule-own-action) rather than
main, since this function does not exist in its post-#401 shape on main
yet.

Tests: updated the four tests pinning the old universal-outcome wording
to the new one; added a new test reproducing #453's own measured shape
directly.

Mutation-tested: reverting to the old wording breaks all 7 affected
tests for the right reason; the two redundant-branch tests are
unaffected either way, confirming that branch genuinely does not need
this fix.

Full suite: 2266 passed, 13 skipped, 0 failed.

Live-tested against the real model: 15 generations on the measured
contradicted-case finding, 0/15 fallbacks, wording correctly follows the
new, narrower fact in every non-copy case.

Re-captured and re-rated AC-002 in the held-out evaluation set (its
prompt hash moved since this changes the exact text fed to the model for
that finding) -- docs/evaluation.md and the holdout fixture both
updated, with a request for Shubham and Arsh to rate the new capture.
)

Rebasing onto #401's plain-English blocking-line fix meant every
existing fixture for the contradicted branch now takes that primary
path, leaving #453's own fix -- the dropped universal-outcome claim in
the raw-syntax fallback -- with no test reaching it at all.

Added one that does, using a blocking line shaped so _describe_acl_line()
refuses it (a source-port clause). Mutation tested: reverting the
fallback to the old 'traffic matching it is actually {outcome}' wording
fails this new test for the right reason; the existing 'omitted' test is
unaffected, since it takes the primary path either way.
@patelankeet2
patelankeet2 force-pushed the fix/453-dead-rule-fact-completeness branch from e46fb9f to e3ec2fd Compare November 10, 2026 00:29
@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Rebased and force-pushed onto #401's new head (e3ec2fd on top of 6549573), since both this PR and #401 touch the exact same return statement in _compute_dead_rule_outcome()'s contradicted branch.

What happened: Shubham reviewed #401 and found the fix there (naming the blocking line in raw ACL syntax) traded one misdescription for another - he measured 16/30 live generations calling the /24 source "a specific host". The fix there replaces raw syntax with a plain-English description (_describe_acl_line()), falling back to raw-syntax naming only when a blocking line's shape isn't recognised.

This PR's own fix (dropping the universal-outcome claim) lives entirely in that fallback path. The two changes compose cleanly: most real findings (including AC-002) now take the new plain-English path, which never had the universal-outcome problem in the first place, since it only ever describes what the blocking line itself does; the fallback path, for the rarer case where a blocking line's shape isn't recognised, keeps this PR's fix intact.

What I did, resolving the rebase:

  • Combined the two return paths correctly in ai/explain.py - plain English first, this PR's safer fallback wording second.
  • Found a real coverage gap: after the rebase, every existing test fixture in this file happens to take the new primary path, so nothing exercised this PR's own fallback fix anymore. Added a new test (test_an_unparseable_blocking_line_falls_back_without_a_universal_claim) using a blocking line shaped so the new parser refuses it, confirming the fallback still avoids the universal-outcome claim. Mutation-tested: reverting the fallback to the old wording fails this new test for the right reason.
  • Reconciled the docs/evaluation.md / held-out fixture conflicts: both branches had independently re-captured AC-002 with different text. Since AC-002's blocking line parses through the new primary path either way, Name the dead line's own stated action explicitly, not just the outcome (#384) #401's capture is what the combined code actually produces - kept that one, noted why in the doc.

Testing on the rebased branch:

No change to this PR's own point beyond the rebase itself - the completeness fix and its original tests are unchanged, just relocated into the combined function.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on e3ec2fd. The over-claim you removed is real, but the replacement wording causes the Example 1 copying you saw. I measured it, and it isn't a missing #455/#457 fix.

What this PR changes now. #401's latest head (bce9017) already describes the blocker in plain English ("an earlier line, evaluated first, denies TCP traffic from 10.1.1.0/24 to any destination instead"), which makes no claim about the dead line's whole pattern. On every blocker it can parse, this branch gives the identical fact. So #461 effectively changes only the fallback, for a blocking line _describe_blocking_line() can't parse. That's where I measured, with your #453 shape and an unparseable blocker (deny tcp 10.1.1.0 0.0.0.255 any log-input object-group WEIRD), 20 live generations each:

                                   copies Example 1   over-claims "all of it is denied"   calls the PERMIT a block
#401 head (fact: "actually denied")       1/20                 1/20                              0/20
#461 (fact: "has the opposite action")   10/20                 0/20                              1/20
a third wording I tried                    0/20                 9/20 (in the model's own words)   0/20
no computed fact at all                    0/20                 0/20                              0/20

So your 10/15 reproduces at 10/20, on this branch's own wording. #455/#458 would only catch those copies after the fact, so with them merged half of these explanations become fallback text. They can't stop the model copying. My reading: a fact with no outcome in it ("has the opposite action instead") leaves a gap, and the model fills it with Example 1. When I added an outcome-shaped sentence, it filled the gap with the over-claim instead (my third row; I read those texts myself, and the pattern alone under-counted them).

Suggested fix: for a blocking line the code can't parse, return None, the same as main already does for unrecognised syntax, and let the model work from the raw evidence. It was the only wording with 0/20 on all three counts. Your docstring explaining why the universal claim is unsafe is good and worth keeping beside that return None.

Caveats, said plainly: 20 runs per row, one finding shape, one unparseable blocker. Enough to see 10/20 against 0/20, not enough to rank the rows that differ by one. Re-measure on whatever you choose and post the counts.

Separately: this branch is based on an older #401 head (it doesn't contain bce9017), and both branches re-capture AC-002 in holdout.json. Once #401 lands, the rebase will need the capture resolved, and #385's prompt-hash guard will say which one is current.

…back claim

Shubham measured the raw-syntax fallback #453 added for a blocking line
_describe_acl_line() can't render in plain English: 10/20 live
generations copied the Modelfile's Example 1 almost verbatim, against
1/20 for main's prior wording and 0/20 for no computed fact at all. A
sentence with no outcome in it still leaves a gap, and the model filled
it with Example 1 more than either alternative.

Return None instead, the same refusal this function already makes for
unrecognised syntax and disagreeing blocking lines. Rewrote the test
pinning the old wording to assert None, and added a mixed-list case
(one parseable blocking line, one not) that nothing pinned before.
@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Fixed, as you suggested: the unparseable-blocking-line fallback now returns None instead of naming it in raw syntax -- the same refusal this function already makes a few lines up for unrecognised syntax and disagreeing blocking lines.

Your two-mutant table is the right read; I didn't re-measure a third wording, just removed the sentence. Live-measured 20 generations on your exact fixture: 0/20 Example 1 copies, matching your "no computed fact at all" row.

Rewrote the test that pinned the old wording to assert None, and added a mixed-list case (one parseable blocking line, one not) that wasn't pinned before -- the all(...) check already requires every line to parse before taking the plain-English path, so a partial list should fall through to the same refusal, not a half-description.

Mutation-tested by restoring the old raw-syntax sentence: both tests fail, reproducing the exact wording in the assertion diff. Reverted.

Checked docs/evaluation.md and the holdout fixture -- neither needs a re-capture. AC-002's own note already says its blocking line always parses, so it never reached this fallback, and nothing else in the held-out set hits an unparseable BLOCKING_LINES line.

Full detail, including two things I found while reading all 20 outputs that are real but out of scope here (a port-1024-as-HTTPS misreading, and "its own rule" phrasing for two separate lines), in the updated PR description.

Re-requesting.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 039986f, which replaces my request for changes. Returning None for a blocker the code can't parse is the version that measured clean, and you took it the simple way.

Checked on #401 + this branch, merged onto current main (ee769b4):

parseable blocker      -> "... an earlier line, evaluated first, denies TCP traffic from 10.1.1.0/24 to any destination instead."
unparseable blocker    -> None
one of each            -> None    (your new mixed-list case: no half-description)
  • Full suite: 2339 passed, 13 skipped, ruff clean on CI's selection.
  • test_explanation_quality.py: 16 passed. #385's prompt-hash guard agrees that AC-002 needed no re-capture, because its blocker always parses.
  • Live: your 0/20 Example 1 copies matches my "no computed fact" row from last week (0/20 copies, 0/20 over-claims, 0/20 inversions) on the same fixture.

The two things you found reading all 20 outputs (port 1024 read as HTTPS, and "its own rule" for two lines) are good catches. They're worth their own issue so they aren't lost in a PR description.

@ARSH871-bot

Copy link
Copy Markdown
Owner

@patelankeet2: #461 is stacked on #401. Its base is feature/384-dead-rule-own-action, #401's branch, not main, and python -m tools.stacked_prs flags it. If #401 merges first, GitHub closes #461, and a closed stacked PR cannot be reopened or retargeted, so the number and the review thread are lost (#179 lost one this way).

Please retarget it before #401 merges:

gh pr edit 461 --base main

Two things to know when you do:

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 039986f (stacked on #401). Commenting rather than approving, because what this PR is meant to close depends on #401's wording, which has changes requested.

This PR's own change is right. The unparseable-blocker path now returns None instead of a weaker raw-syntax claim, and Shubham's 10/20 → 0/20 Example 1 measurement supports it. CI-style on main + this branch: 2468 passed, 63 skipped, ruff clean.

But #453's measured case still gets a misleading fact. Running _compute_dead_rule_outcome() on the exact ACL in #453 (permit tcp host 10.1.1.5 any / deny tcp 10.1.1.0 0.0.0.255 any / dead permit ... eq 443), on this branch and on #401's head bce9017, gives the same sentence:

the unreachable line's own stated action is PERMIT, but it never takes effect: an earlier line, evaluated first, denies TCP traffic from 10.1.1.0/24 to any destination instead.

"Denies ... instead" still tells the model the dead line's traffic is denied, and Batfish says 10.1.1.5 on 443 is permitted (by the line Blocking_Lines leaves out). That sentence comes from #401's plain-English rewording (e22d970), which this branch picked up, so #453 is only closed once #401's next wording handles it. Since #401 is being reworded anyway, the narrower form this PR describes ("the dead line never fires; the named line X is one earlier line that decides some of its traffic") would fit there.

Also, as on #401: please retarget this to main before #401 merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working layer: ai Layer 2 — local LLM explanation and Q&A safety Touches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants