Skip to content

Refuse at load a rule whose traffic Batfish cannot be asked about - #468

Merged
patelankeet2 merged 3 commits into
mainfrom
fix/policy-loader-traffic-shape
Nov 19, 2026
Merged

patelankeet2 merged 3 commits into
mainfrom
fix/policy-loader-traffic-shape

Conversation

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator

The follow-up I offered on #442, with the two cases @ARSH871-bot listed under "What this does not do" on #467.

Before, measured on main (2833a1e): each loads, then fails only at scan time

input the scan said
"queries": [] PC-051 "this rule has no queries" (#442's guard)
"queries": "10.10.10.0/24" PC-051 "13 of 13 queries could not run": one query per character
"queries": ["10.10.10.0/24"] PC-051 TypeError from pybatfish
{"srcIp": ...} (typo) PC-051 TypeError: unexpected keyword argument 'srcIp'
{"srcIps": "10.10.10.300"} PC-051 BatfishException: Work terminated abnormally

Never unsafe (each ends in "could not check"), but always found late, with a message the user can't act on.

After

load_policy() refuses each one at upload, naming the rule and the problem, e.g. policy_compliance entry 1 ('x'), query 1: srcIps '10.10.10.300' is not a valid address or prefix. The same check covers access_control's headers, which go to the same HeaderConstraints.

  • Field names are read from HeaderConstraints itself (attr.fields_dict), not copied, so they can't drift.
  • Addresses are checked narrowly, on purpose. Batfish's address language includes ranges (a-b), wildcards (a:b), set difference and named groups, and refusing a valid rule would be worse than the bug. Only an element made purely of digits, dots and a slash is parsed. Each accepted form is run against real Batfish in the new tests, so the loader doesn't refuse anything Batfish can answer.
  • An empty query {} still loads. It means all traffic, and "this filter must permit nothing" is a rule someone can mean.

Tests: tests/test_policy_traffic_shape.py, 23

Every refusal with its message, every accepted form (including POL-1's set difference and the shipped example), and two live tests: the accepted forms really run in Batfish, and the refused address really fails there.

Mutation, 7 of 7 killed: empty list allowed, a string allowed, headers unchecked, unknown fields allowed, no address checked, every element treated as plain (false refusals, fails 7), and the specifier skip removed.

Two existing tests used placeholder traffic ("headers": "x", "queries": []) for things unrelated to traffic, so they now use a real query, as test_policy_loader.py already did for queries. docs/policy-rules.md gets a short section.

2334 passed, 13 skipped with Batfish running; ruff clean on CI's selection. Merges with #467 independently: different functions in policy.py.

Follow-up I offered on #442, with the two cases Arsh listed on #467.
policy_compliance queries and access_control headers go straight to
HeaderConstraints, which accepts almost anything locally, so these loaded
and failed only at scan time with an error the user could not act on:

  queries: []                -> 'this rule has no queries'
  queries: '10.10.10.0/24'   -> 13 queries, one per character
  a query that is a string   -> TypeError from pybatfish
  {'srcIp': ...}             -> TypeError, unexpected keyword
  {'srcIps': '10.10.10.300'} -> 'Work terminated abnormally'

Each is now refused at load, naming the rule. Field names are read from
HeaderConstraints itself. Only an address element that is plainly one
address or prefix is checked; ranges, wildcards, set difference and named
groups are left to Batfish, and each accepted form is run against real
Batfish in the new tests. An empty query still loads: it means all traffic.

Two existing tests used placeholder traffic ('x', []) and now use a real
query, as the loader test already did for queries.

@SamikaPerera SamikaPerera left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the change against main (ee769b4), ran four bad rules through the real loader, and tested the merge with #464.

What it changes. analysis/policy.py adds _check_traffic(), called from _validate_entry() for each policy_compliance query and for access_control headers. It refuses:

  • queries that is not a non-empty list
  • a query that is not a mapping
  • a key that is not a HeaderConstraints field (read with attr.fields_dict)
  • a plain-looking srcIps/dstIps element that ipaddress cannot parse (_PLAIN_ADDRESS = digits, dots, optional /n)

Four bad rules through the real load_policy(). Each was the shipped example rule with node set and only queries changed. On main all four load. Here:

queries []                PolicyError: policy_compliance entry 1 ('The LAN may reach only the two approved servers'): 'queries' must be a non-empty list of traffic descriptions, got []. A rule with no queries asks Batfish nothing
queries null              ... 'queries' must be a non-empty list of traffic descriptions, got None. A rule with no queries asks Batfish nothing
queries "10.10.10.0/24"   ... 'queries' must be a non-empty list of traffic descriptions, got '10.10.10.0/24'. A rule with no queries asks Batfish nothing
queries ["10.10.10.0/24"] ..., query 1: expected a mapping such as {"srcIps": "10.10.10.0/24"}, got str '10.10.10.0/24'

Claims checked:

  • "23 tests": true. On main's code, 10 of them fail (every refusal case) and 13 pass (the accepted forms).
  • "Mutation, 7 of 7 killed": reproduced. I mutated seven guards, one at a time, against tests/test_policy_traffic_shape.py and tests/test_policy_loader.py. All seven were killed: empty list allowed, a string allowed, headers unchecked, unknown fields allowed, no address checked, every element treated as plain (7 failures), and a non-mapping query allowed.
  • "An empty query {} still loads": true. A set-difference srcIps also loads.
  • The two changed existing tests swap placeholder traffic for a real query: true. tests/test_policy_loader.py:374 and tests/test_session_isolation.py:286.
  • "Merges with #467 independently": true. git merge-tree of the two heads is clean.

With #464 (6033b38) and main: git merge-tree is clean, and #468 + #464 + main merged gives 2356 passed, 13 skipped, 0 failed with Batfish running.

Run: this branch merged with current main (ee769b4), Batfish running: 2349 passed, 13 skipped, 0 failed.

One wording point, not blocking. For null or a string, the message ends "A rule with no queries asks Batfish nothing". That reads oddly when the rule does have something under queries; it just isn't a list. The first sentence already says what is wrong, so you could drop the second for those two cases.

Not checked: the "before" table (the scan-time PC-051 messages on main). Reproducing it needs a scan with each bad rule, and I didn't run one. I also didn't run the two live tests separately from the full suite.

shubhamkataria2005 added a commit that referenced this pull request Nov 16, 2026
With either rule's queries empty there are no pairs, any([]) is False, and
the 'could not be compared' warning main gives went away. Nothing proves
such a pair disjoint, so it warns again. #468 refuses an empty list at
load, but policy_conflicts can still be handed one, and this does not
depend on the order the two PRs land in.
For null or a string, the message ended 'A rule with no queries asks Batfish
nothing', which reads oddly when the rule does have something under
'queries'. A non-list and an empty list now get their own messages, each
pinned by a test, with null added to the cases.
@shubhamkataria2005

shubhamkataria2005 commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

@SamikaPerera, agreed, and changed in e99412b. A non-list now says 'queries' must be a list of traffic descriptions, got str '10.10.10.0/24' (or got NoneType None). Only an empty list says 'queries' is empty. A rule with no queries asks Batfish nothing. Each message is pinned, and null is added to the cases. Mutants: allowing an empty list fails 1, allowing a non-list fails 2. Merged with main: 2350 passed, 13 skipped, CI green.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed with real testing, not a read-through (Samika's approval went stale after a follow-up commit).

Live-reproduced the three split error messages directly through load_policy():

empty list    -> 'queries' is empty. A rule with no queries asks Batfish nothing
a string      -> 'queries' must be a list of traffic descriptions, got str '10.10.10.0/24'
None          -> 'queries' must be a list of traffic descriptions, got NoneType None

Matches the review follow-up (e99412b) exactly -- each now names its own problem instead of one message covering all three.

Full suite with Batfish running: 2485 passed, 13 skipped, 0 failed.

Approving.

@patelankeet2
patelankeet2 merged commit 04599e3 into main Nov 19, 2026
2 checks passed
@patelankeet2
patelankeet2 deleted the fix/policy-loader-traffic-shape branch November 19, 2026 01:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants