Repository navigation
Dead-rule fact: stop claiming a universal outcome Batfish cannot guarantee (#453) - #461
patelankeet2 wants to merge 3 commits into
Conversation
…antee (#453) _compute_dead_rule_outcome()'s contradicted-action branch (own action vs. blocking line's action disagree) told the model, as already verified and certain, that 'traffic matching it is actually {outcome}' for the dead line's ENTIRE matching pattern. Measured live (same on main db59b00 and -- specifically, a line sharing the DEAD line's own action is not always listed, and when one exists it can decide a narrower slice of traffic the opposite way from what the named, opposite-action blocking line alone predicts. A real config had permit host 10.1.1.5 any before both the named deny and the dead permit; the fact said the whole /24 is denied, but testFilters showed host .5 is actually permitted. Confirmed only the contradicted branch is unsafe. The redundant branch (blocking line agrees with the dead line's own action) is unaffected -- break it, 6 returned None (Batfish named the overlapping line), 2 had no such overlap and the sentence was true. Worked out why: if an omission is always a same-action-as-the-dead-line line, it cannot change the outcome when every alternative (named or not) already agrees. Also confirmed naming the blocking line's own text is not, by itself, enough (the issue's suggested wording): the omitted line in the measured case is earlier than the NAMED blocking line, not narrower than the dead line's pattern -- host .5 matches the named deny's /24 too. The fix drops the universal 'actually {outcome}' claim for this branch entirely, stating only the two facts that stay true regardless of what Blocking_Lines could have left out: the dead line's own action never fires (filterLineReachability verifies that exhaustively), and the named earlier line genuinely is evaluated first and genuinely does have the opposite action. Stacked on #401's branch (feature/384-dead-rule-own-action) rather than main, since this function does not exist in its post-#401 shape on main yet. Tests: updated the four tests pinning the old universal-outcome wording to the new one; added a new test reproducing #453's own measured shape directly. Mutation-tested: reverting to the old wording breaks all 7 affected tests for the right reason; the two redundant-branch tests are unaffected either way, confirming that branch genuinely does not need this fix. Full suite: 2266 passed, 13 skipped, 0 failed. Live-tested against the real model: 15 generations on the measured contradicted-case finding, 0/15 fallbacks, wording correctly follows the new, narrower fact in every non-copy case. Re-captured and re-rated AC-002 in the held-out evaluation set (its prompt hash moved since this changes the exact text fed to the model for that finding) -- docs/evaluation.md and the holdout fixture both updated, with a request for Shubham and Arsh to rate the new capture.
) Rebasing onto #401's plain-English blocking-line fix meant every existing fixture for the contradicted branch now takes that primary path, leaving #453's own fix -- the dropped universal-outcome claim in the raw-syntax fallback -- with no test reaching it at all. Added one that does, using a blocking line shaped so _describe_acl_line() refuses it (a source-port clause). Mutation tested: reverting the fallback to the old 'traffic matching it is actually {outcome}' wording fails this new test for the right reason; the existing 'omitted' test is unaffected, since it takes the primary path either way.
e46fb9f to
e3ec2fd
Compare
|
Rebased and force-pushed onto #401's new head (e3ec2fd on top of 6549573), since both this PR and #401 touch the exact same return statement in What happened: Shubham reviewed #401 and found the fix there (naming the blocking line in raw ACL syntax) traded one misdescription for another - he measured 16/30 live generations calling the /24 source "a specific host". The fix there replaces raw syntax with a plain-English description ( This PR's own fix (dropping the universal-outcome claim) lives entirely in that fallback path. The two changes compose cleanly: most real findings (including AC-002) now take the new plain-English path, which never had the universal-outcome problem in the first place, since it only ever describes what the blocking line itself does; the fallback path, for the rarer case where a blocking line's shape isn't recognised, keeps this PR's fix intact. What I did, resolving the rebase:
Testing on the rebased branch:
No change to this PR's own point beyond the rebase itself - the completeness fix and its original tests are unchanged, just relocated into the combined function. |
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Requesting changes on e3ec2fd. The over-claim you removed is real, but the replacement wording causes the Example 1 copying you saw. I measured it, and it isn't a missing #455/#457 fix.
What this PR changes now. #401's latest head (bce9017) already describes the blocker in plain English ("an earlier line, evaluated first, denies TCP traffic from 10.1.1.0/24 to any destination instead"), which makes no claim about the dead line's whole pattern. On every blocker it can parse, this branch gives the identical fact. So #461 effectively changes only the fallback, for a blocking line _describe_blocking_line() can't parse. That's where I measured, with your #453 shape and an unparseable blocker (deny tcp 10.1.1.0 0.0.0.255 any log-input object-group WEIRD), 20 live generations each:
copies Example 1 over-claims "all of it is denied" calls the PERMIT a block
#401 head (fact: "actually denied") 1/20 1/20 0/20
#461 (fact: "has the opposite action") 10/20 0/20 1/20
a third wording I tried 0/20 9/20 (in the model's own words) 0/20
no computed fact at all 0/20 0/20 0/20
So your 10/15 reproduces at 10/20, on this branch's own wording. #455/#458 would only catch those copies after the fact, so with them merged half of these explanations become fallback text. They can't stop the model copying. My reading: a fact with no outcome in it ("has the opposite action instead") leaves a gap, and the model fills it with Example 1. When I added an outcome-shaped sentence, it filled the gap with the over-claim instead (my third row; I read those texts myself, and the pattern alone under-counted them).
Suggested fix: for a blocking line the code can't parse, return None, the same as main already does for unrecognised syntax, and let the model work from the raw evidence. It was the only wording with 0/20 on all three counts. Your docstring explaining why the universal claim is unsafe is good and worth keeping beside that return None.
Caveats, said plainly: 20 runs per row, one finding shape, one unparseable blocker. Enough to see 10/20 against 0/20, not enough to rank the rows that differ by one. Re-measure on whatever you choose and post the counts.
Separately: this branch is based on an older #401 head (it doesn't contain bce9017), and both branches re-capture AC-002 in holdout.json. Once #401 lands, the rebase will need the capture resolved, and #385's prompt-hash guard will say which one is current.
…back claim Shubham measured the raw-syntax fallback #453 added for a blocking line _describe_acl_line() can't render in plain English: 10/20 live generations copied the Modelfile's Example 1 almost verbatim, against 1/20 for main's prior wording and 0/20 for no computed fact at all. A sentence with no outcome in it still leaves a gap, and the model filled it with Example 1 more than either alternative. Return None instead, the same refusal this function already makes for unrecognised syntax and disagreeing blocking lines. Rewrote the test pinning the old wording to assert None, and added a mixed-list case (one parseable blocking line, one not) that nothing pinned before.
|
Fixed, as you suggested: the unparseable-blocking-line fallback now returns Your two-mutant table is the right read; I didn't re-measure a third wording, just removed the sentence. Live-measured 20 generations on your exact fixture: 0/20 Example 1 copies, matching your "no computed fact at all" row. Rewrote the test that pinned the old wording to assert Mutation-tested by restoring the old raw-syntax sentence: both tests fail, reproducing the exact wording in the assertion diff. Reverted. Checked Full detail, including two things I found while reading all 20 outputs that are real but out of scope here (a port-1024-as-HTTPS misreading, and "its own rule" phrasing for two separate lines), in the updated PR description. Re-requesting. |
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Approving 039986f, which replaces my request for changes. Returning None for a blocker the code can't parse is the version that measured clean, and you took it the simple way.
Checked on #401 + this branch, merged onto current main (ee769b4):
parseable blocker -> "... an earlier line, evaluated first, denies TCP traffic from 10.1.1.0/24 to any destination instead."
unparseable blocker -> None
one of each -> None (your new mixed-list case: no half-description)
- Full suite: 2339 passed, 13 skipped, ruff clean on CI's selection.
test_explanation_quality.py: 16 passed. #385's prompt-hash guard agrees that AC-002 needed no re-capture, because its blocker always parses.- Live: your 0/20 Example 1 copies matches my "no computed fact" row from last week (0/20 copies, 0/20 over-claims, 0/20 inversions) on the same fixture.
The two things you found reading all 20 outputs (port 1024 read as HTTPS, and "its own rule" for two lines) are good catches. They're worth their own issue so they aren't lost in a PR description.
|
@patelankeet2: #461 is stacked on #401. Its base is Please retarget it before #401 merges: Two things to know when you do:
|
ARSH871-bot
left a comment
There was a problem hiding this comment.
Reviewed 039986f (stacked on #401). Commenting rather than approving, because what this PR is meant to close depends on #401's wording, which has changes requested.
This PR's own change is right. The unparseable-blocker path now returns None instead of a weaker raw-syntax claim, and Shubham's 10/20 → 0/20 Example 1 measurement supports it. CI-style on main + this branch: 2468 passed, 63 skipped, ruff clean.
But #453's measured case still gets a misleading fact. Running _compute_dead_rule_outcome() on the exact ACL in #453 (permit tcp host 10.1.1.5 any / deny tcp 10.1.1.0 0.0.0.255 any / dead permit ... eq 443), on this branch and on #401's head bce9017, gives the same sentence:
the unreachable line's own stated action is PERMIT, but it never takes effect: an earlier line, evaluated first, denies TCP traffic from 10.1.1.0/24 to any destination instead.
"Denies ... instead" still tells the model the dead line's traffic is denied, and Batfish says 10.1.1.5 on 443 is permitted (by the line Blocking_Lines leaves out). That sentence comes from #401's plain-English rewording (e22d970), which this branch picked up, so #453 is only closed once #401's next wording handles it. Since #401 is being reworded anyway, the narrower form this PR describes ("the dead line never fires; the named line X is one earlier line that decides some of its traffic") would fit there.
Also, as on #401: please retarget this to main before #401 merges.
Closes #453.
Stacked on #401 (feature/384-dead-rule-own-action) rather than main: #401 adds the contradicted/redundant branching this fix changes, and that shape does not exist on main yet. Basing on #401's branch avoids a second, conflicting change to the same function. Once #401 merges, GitHub retargets this PR onto main automatically.
What's wrong:
_compute_dead_rule_outcome()'s contradicted-action branch told the model, as already verified and certain, that "traffic matching it is actually {outcome}" for the dead line's ENTIRE matching pattern. Measured live, same on maindb59b00and #401's246db8e:Batfish's
Blocking_Linesdoes not always name every earlier line -- specifically, a line sharing the DEAD line's own action is not always listed, and when one exists it can decide a narrower slice of traffic the opposite way from what the named, opposite-action blocking line alone predicts.What I checked before fixing it:
What changed: dropped the universal "actually {outcome}" claim for the contradicted branch entirely. States only the two facts that stay true regardless of what
Blocking_Linescould have left out: the dead line's own action never fires (filterLineReachabilityverifies that exhaustively across the whole match space -- that part of "unreachable" is sound), and the named earlier line genuinely is evaluated first and genuinely does have the opposite action. Example on the measured case: "the unreachable line's own stated action is PERMIT, but that action never takes effect: an earlier line (deny tcp 10.1.1.0 0.0.0.255 any), which is evaluated first, has the opposite action (DENY) instead." No claim about host 10.1.1.5 or any other specific packet.Testing:
pytest tests/test_explain_safety_net.py -v-- 96 passed.pytest tests/ -q-- 2266 passed, 13 skipped, 0 failed.AC-002in the held-out evaluation set, since this change moves the exact prompt text fed to the model for that finding (caught bytests/test_explanation_quality.py's own prompt-hash guard).docs/evaluation.mdand the holdout fixture both updated, with a request for Shubham and Arsh to rate the new capture.docs/evaluation.mdalready has its own pre-existing em-dash convention throughout).Update, 11 November -- Shubham's review addressed
Shubham measured the fallback path this PR's second commit (
e3ec2fd) added -- for a blocking line whose action is known but whose shape_describe_acl_line()can't render in plain English -- and found it causes 10/20 Modelfile Example 1 copies, against 1/20 for #401's main-head wording and 0/20 for no computed fact at all. The fallback's sentence states the named line's own action but no outcome, and that gap got filled with Example 1 more often than either alternative.Fixed as he suggested: that fallback now returns
Noneinstead of naming the line in raw syntax, the same refusal this function already makes a few lines up for unrecognised syntax and disagreeing blocking lines. The explanatory comment about why the dropped universal-outcome claim was unsafe stays (it is still true and still the reason #453 exists), rewritten to also say why a narrower fallback sentence turned out not to be safe either.Not touched: this doesn't change the primary path (both/all blocking lines parse via
_describe_acl_line()) at all -- that's the part #453's own measurement and this PR's existing tests already cover, and it is unaffected by this fallback-only fix.Testing:
test_an_unparseable_blocking_line_falls_back_without_a_universal_claim(now..._returns_none_rather_than_a_weaker_claim) to assertNoneon the exact unparseable fixture.test_one_unparseable_line_among_several_blocking_lines_also_returns_none, a mixed-list case (one parseable blocking line, one not) nothing pinned before.pytest tests/test_explain_safety_net.py -q-- 104 passed.pytest tests/ -q(Batfish running) -- 2324 passed, 13 skipped, 0 failed.docs/evaluation.mdand the holdout fixture: neither needs a re-capture from this change. AC-002's own capture note already says its blocking line always parses via_describe_acl_line(), so it never reaches this fallback, and no other held-out entry hitsBLOCKING_LINESwith an unparseable line either.@shubhamkataria2005 -- re-requesting.