Repository navigation
Do not pass a policy rule whose filter is applied to no interface (closes #430) - #432
Conversation
…oses #430) searchFilters proves what a filter would do to traffic passing through it, not that any traffic does. With the access-group removed from rtr-us5-secure, every rule held and PC-000 gave a green tick while traceroute delivered telnet from the LAN to the server. Each rule's filter must now be an incoming or outgoing filter on one of its node's interfaces (interfaceProperties), or the rule reports could not check. Names compare casefolded, as Batfish resolves them. A specifier filter names no single filter, so it is checked as before. The message says Batfish's model does not apply the filter, not that the config does not: vendor-asa applies its ACL in a line Batfish does not parse. stranger_config output is identical before and after on every fixture.
patelankeet2
left a comment
There was a problem hiding this comment.
Reviewed with real testing against real Batfish, and this one I reproduced from scratch rather than trusting the description -- built the exact before/after fixture myself. Tested merged with current main (b69d591).
git merge origin/main (local only) clean
pytest tests/test_policy_filter_applied.py -q 14 passed
ruff check --select E4,E7,E9,F ... clean
Reproduced the "before" and "after" independently, not from the PR's own output. Copied tests/fixtures/rtr-us5-secure, stripped ip access-group acl_in in from the config, and ran the real pipeline against it:
$ python -m analysis.pipeline <modified fixture>
PC-051 error Could not check policy rule: the lan may reach only the two approved servers
PC-052 error Could not check policy rule: the internal server is reachable only over https
PC-053 error Could not check policy rule: only lan addresses may enter the lan interface
PC-054 error Could not check policy rule: dns to the approved resolver must work
PC-055 error Could not check policy rule: https to the internal server must work
Matches the "after" claim exactly, on a fixture I built myself rather than the one shipped with the PR.
Mutation-tested the actual guard rather than a smaller property of it -- made _not_in_path() always return False, i.e. disabled the whole check, and reran the SAME modified fixture:
PC-000 none No issues found by policy compliance
That is the exact dangerous "before" state the PR describes -- a filter that protects nothing, reported as a clean bill of health. Six tests failed with the guard disabled, including the real-Batfish one (test_real_batfish_no_green_tick_for_an_acl_applied_nowhere). Reverted.
Tried to also confirm the underlying traceroute claim (telnet DELIVERED_TO_SUBNET) directly against my own modified fixture and got NO_ROUTE instead, most likely a startLocation/interface subtlety in how I called traceroute by hand rather than anything about the fix -- recording this honestly rather than silently dropping it, but it doesn't weaken the finding above, which is the one that actually matters: the check now reports error instead of none on a config where the ACL has nowhere to run, confirmed both by direct reproduction and by mutating the guard out.
Full suite, run twice, same shape as #428's review a short while ago. First run: 1 failed, 1914 passed -- test_parse_coverage.py::test_real_vendor_asa_count_reaches_the_dashboard_and_both_reports. Checked it carefully rather than assuming flakiness, because this PR's own description talks about vendor-asa too: read the failing test, and it asserts a parse-line-count message ("did not recognise 5 of its 16 configuration lines") that has no code path through anything this PR touches -- analysis/parse_coverage.py is a separate, already-merged feature. Passed alone, and a full second run came back 1915 passed, 0 failed. Not a regression from this PR. Worth flagging on its own: this is the second unrelated live-Batfish test to flake under a full-suite run today (the other was on #428's review), which might be worth a standalone issue about full-suite Batfish contention, separate from this PR.
Checked the one design choice worth a second look: when the interfaceProperties lookup itself fails, applied stays empty, so every plainly-named filter falls into the "not in path" branch -- but the message correctly distinguishes "we could not find out" from "it is not applied", so a lookup failure still reads honestly rather than as a false specific claim. Confirmed by reading run()'s branch on lookup_error.
No em-dashes, no AI-tool mentions.
Approving.
|
@patelankeet2, thanks for rebuilding it yourself, and for flagging the traceroute rather than dropping it. I've found the cause. It's the fixture, not how you called traceroute. So your On the two live tests that flaked under a full run (#428's and |
Closes #430, found while researching #423 (#431).
Before,
rtr-us5-securewithip access-group acl_in inremoved:After: every rule on that filter reports
PC-05n error Could not check policy rule: …, saying Batfish's model appliesacl_into no interface.How: one
interfacePropertiesquery per run. A rule passes only if its filter is an incoming or outgoing filter on its node. The rules:strip().casefold(), because Batfish resolvesACL_INtoacl_in(the Refuse a policy that forbids what it requires (closes #322) #387 review)./re/,@in(…), a list) is checked as before. It names no single filter.vendor-asareally does applyOUTSIDE_IN, in a line Batfish doesn't parse, so a policy rule on it now says could not check, and "your config doesn't apply it" would be false.What else moves: nothing on our fixtures.
python -m tools.stranger_configoutput is byte-identical tomain(13 / 4 / 21).Tests:
tests/test_policy_filter_applied.py, 14 tests, including two against real Batfish (the measurement above, andrtr-us5-securestill passing). Nine existing tests fake the session, so they now also stub the guard, one commented line each, the same way they already stubdevice_names. Mutation: 8 of 8 killed. One earlier mutant survived, which showed a redundant condition. Removing it also made specifier filters behave consistently when the lookup fails, and a test pins that.1901 passed, 13 skippedwith Batfish running. ruff clean.docs/policy-rules.md§2 saidsearchFilters"cannot be fooled in either direction" and that empty "means exactly one thing: the policy holds". Both are now qualified in the same commit as the behaviour change.