Skip to content

Do not pass a policy rule whose filter is applied to no interface (closes #430) - #432

Merged
patelankeet2 merged 1 commit into
mainfrom
fix/430-unapplied-filter
Nov 2, 2026
Merged

patelankeet2 merged 1 commit into
mainfrom
fix/430-unapplied-filter

Conversation

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator

Closes #430, found while researching #423 (#431).

Before, rtr-us5-secure with ip access-group acl_in in removed:

PC-000  none   No issues found by policy compliance
traceroute LAN -> 10.20.0.5 TCP/23    DELIVERED_TO_SUBNET

After: every rule on that filter reports PC-05n error Could not check policy rule: …, saying Batfish's model applies acl_in to no interface.

How: one interfaceProperties query per run. A rule passes only if its filter is an incoming or outgoing filter on its node. The rules:

  • Names compare strip().casefold(), because Batfish resolves ACL_IN to acl_in (the Refuse a policy that forbids what it requires (closes #322) #387 review).
  • A specifier filter (/re/, @in(…), a list) is checked as before. It names no single filter.
  • If the lookup fails, the rule reports could not check, never a pass.
  • The message blames the model, not the config. vendor-asa really does apply OUTSIDE_IN, in a line Batfish doesn't parse, so a policy rule on it now says could not check, and "your config doesn't apply it" would be false.

What else moves: nothing on our fixtures. python -m tools.stranger_config output is byte-identical to main (13 / 4 / 21).

Tests: tests/test_policy_filter_applied.py, 14 tests, including two against real Batfish (the measurement above, and rtr-us5-secure still passing). Nine existing tests fake the session, so they now also stub the guard, one commented line each, the same way they already stub device_names. Mutation: 8 of 8 killed. One earlier mutant survived, which showed a redundant condition. Removing it also made specifier filters behave consistently when the lookup fails, and a test pins that.

1901 passed, 13 skipped with Batfish running. ruff clean.

docs/policy-rules.md §2 said searchFilters "cannot be fooled in either direction" and that empty "means exactly one thing: the policy holds". Both are now qualified in the same commit as the behaviour change.

…oses #430)

searchFilters proves what a filter would do to traffic passing through it,
not that any traffic does. With the access-group removed from rtr-us5-secure,
every rule held and PC-000 gave a green tick while traceroute delivered telnet
from the LAN to the server.

Each rule's filter must now be an incoming or outgoing filter on one of its
node's interfaces (interfaceProperties), or the rule reports could not check.
Names compare casefolded, as Batfish resolves them. A specifier filter names
no single filter, so it is checked as before. The message says Batfish's
model does not apply the filter, not that the config does not: vendor-asa
applies its ACL in a line Batfish does not parse.

stranger_config output is identical before and after on every fixture.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with real testing against real Batfish, and this one I reproduced from scratch rather than trusting the description -- built the exact before/after fixture myself. Tested merged with current main (b69d591).

git merge origin/main (local only)              clean
pytest tests/test_policy_filter_applied.py -q    14 passed
ruff check --select E4,E7,E9,F ...               clean

Reproduced the "before" and "after" independently, not from the PR's own output. Copied tests/fixtures/rtr-us5-secure, stripped ip access-group acl_in in from the config, and ran the real pipeline against it:

$ python -m analysis.pipeline <modified fixture>
PC-051  error  Could not check policy rule: the lan may reach only the two approved servers
PC-052  error  Could not check policy rule: the internal server is reachable only over https
PC-053  error  Could not check policy rule: only lan addresses may enter the lan interface
PC-054  error  Could not check policy rule: dns to the approved resolver must work
PC-055  error  Could not check policy rule: https to the internal server must work

Matches the "after" claim exactly, on a fixture I built myself rather than the one shipped with the PR.

Mutation-tested the actual guard rather than a smaller property of it -- made _not_in_path() always return False, i.e. disabled the whole check, and reran the SAME modified fixture:

PC-000  none  No issues found by policy compliance

That is the exact dangerous "before" state the PR describes -- a filter that protects nothing, reported as a clean bill of health. Six tests failed with the guard disabled, including the real-Batfish one (test_real_batfish_no_green_tick_for_an_acl_applied_nowhere). Reverted.

Tried to also confirm the underlying traceroute claim (telnet DELIVERED_TO_SUBNET) directly against my own modified fixture and got NO_ROUTE instead, most likely a startLocation/interface subtlety in how I called traceroute by hand rather than anything about the fix -- recording this honestly rather than silently dropping it, but it doesn't weaken the finding above, which is the one that actually matters: the check now reports error instead of none on a config where the ACL has nowhere to run, confirmed both by direct reproduction and by mutating the guard out.

Full suite, run twice, same shape as #428's review a short while ago. First run: 1 failed, 1914 passed -- test_parse_coverage.py::test_real_vendor_asa_count_reaches_the_dashboard_and_both_reports. Checked it carefully rather than assuming flakiness, because this PR's own description talks about vendor-asa too: read the failing test, and it asserts a parse-line-count message ("did not recognise 5 of its 16 configuration lines") that has no code path through anything this PR touches -- analysis/parse_coverage.py is a separate, already-merged feature. Passed alone, and a full second run came back 1915 passed, 0 failed. Not a regression from this PR. Worth flagging on its own: this is the second unrelated live-Batfish test to flake under a full-suite run today (the other was on #428's review), which might be worth a standalone issue about full-suite Batfish contention, separate from this PR.

Checked the one design choice worth a second look: when the interfaceProperties lookup itself fails, applied stays empty, so every plainly-named filter falls into the "not in path" branch -- but the message correctly distinguishes "we could not find out" from "it is not applied", so a lookup failure still reads honestly rather than as a false specific claim. Confirmed by reading run()'s branch on lookup_error.

No em-dashes, no AI-tool mentions.

Approving.

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator Author

@patelankeet2, thanks for rebuilding it yourself, and for flagging the traceroute rather than dropping it. I've found the cause. It's the fixture, not how you called traceroute. rtr-us5-secure has only GigabitEthernet0/0, so 10.20.0.5 isn't a connected network and there's no route to it. Same call, both variants, real Batfish:

access-group removed, one interface (the shipped fixture)       NO_ROUTE
access-group removed, + Gi0/1 10.20.0.1/24 (the #431 recipe)    DELIVERED_TO_SUBNET
traceroute(startLocation="@enter(rtr-us5[GigabitEthernet0/0])", 10.10.10.9 -> 10.20.0.5 TCP/23)

So your NO_ROUTE and my DELIVERED are both right, for their own fixtures. That's also why the new live test only asserts the policy-check result, not the traceroute: the PC-05n behaviour doesn't depend on routing.

On the two live tests that flaked under a full run (#428's and test_parse_coverage's): agreed, two in one day is a pattern. If it happens again I'll open an issue with both test names and the run counts, rather than one quoting a single sighting.

@patelankeet2
patelankeet2 merged commit b010dd6 into main Nov 2, 2026
2 checks passed
@patelankeet2
patelankeet2 deleted the fix/430-unapplied-filter branch November 2, 2026 01:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Policy check says a rule holds when its filter is not applied anywhere

2 participants