Repository navigation
Refuse at load a rule whose traffic Batfish cannot be asked about - #468
Conversation
Follow-up I offered on #442, with the two cases Arsh listed on #467. policy_compliance queries and access_control headers go straight to HeaderConstraints, which accepts almost anything locally, so these loaded and failed only at scan time with an error the user could not act on: queries: [] -> 'this rule has no queries' queries: '10.10.10.0/24' -> 13 queries, one per character a query that is a string -> TypeError from pybatfish {'srcIp': ...} -> TypeError, unexpected keyword {'srcIps': '10.10.10.300'} -> 'Work terminated abnormally' Each is now refused at load, naming the rule. Field names are read from HeaderConstraints itself. Only an address element that is plainly one address or prefix is checked; ranges, wildcards, set difference and named groups are left to Batfish, and each accepted form is run against real Batfish in the new tests. An empty query still loads: it means all traffic. Two existing tests used placeholder traffic ('x', []) and now use a real query, as the loader test already did for queries.
SamikaPerera
left a comment
There was a problem hiding this comment.
I checked the change against main (ee769b4), ran four bad rules through the real loader, and tested the merge with #464.
What it changes. analysis/policy.py adds _check_traffic(), called from _validate_entry() for each policy_compliance query and for access_control headers. It refuses:
queriesthat is not a non-empty list- a query that is not a mapping
- a key that is not a
HeaderConstraintsfield (read withattr.fields_dict) - a plain-looking
srcIps/dstIpselement thatipaddresscannot parse (_PLAIN_ADDRESS= digits, dots, optional/n)
Four bad rules through the real load_policy(). Each was the shipped example rule with node set and only queries changed. On main all four load. Here:
queries [] PolicyError: policy_compliance entry 1 ('The LAN may reach only the two approved servers'): 'queries' must be a non-empty list of traffic descriptions, got []. A rule with no queries asks Batfish nothing
queries null ... 'queries' must be a non-empty list of traffic descriptions, got None. A rule with no queries asks Batfish nothing
queries "10.10.10.0/24" ... 'queries' must be a non-empty list of traffic descriptions, got '10.10.10.0/24'. A rule with no queries asks Batfish nothing
queries ["10.10.10.0/24"] ..., query 1: expected a mapping such as {"srcIps": "10.10.10.0/24"}, got str '10.10.10.0/24'
Claims checked:
- "23 tests": true. On main's code, 10 of them fail (every refusal case) and 13 pass (the accepted forms).
- "Mutation, 7 of 7 killed": reproduced. I mutated seven guards, one at a time, against
tests/test_policy_traffic_shape.pyandtests/test_policy_loader.py. All seven were killed: empty list allowed, a string allowed, headers unchecked, unknown fields allowed, no address checked, every element treated as plain (7 failures), and a non-mapping query allowed. - "An empty query
{}still loads": true. A set-differencesrcIpsalso loads. - The two changed existing tests swap placeholder traffic for a real query: true.
tests/test_policy_loader.py:374andtests/test_session_isolation.py:286. - "Merges with #467 independently": true.
git merge-treeof the two heads is clean.
With #464 (6033b38) and main: git merge-tree is clean, and #468 + #464 + main merged gives 2356 passed, 13 skipped, 0 failed with Batfish running.
Run: this branch merged with current main (ee769b4), Batfish running: 2349 passed, 13 skipped, 0 failed.
One wording point, not blocking. For null or a string, the message ends "A rule with no queries asks Batfish nothing". That reads oddly when the rule does have something under queries; it just isn't a list. The first sentence already says what is wrong, so you could drop the second for those two cases.
Not checked: the "before" table (the scan-time PC-051 messages on main). Reproducing it needs a scan with each bad rule, and I didn't run one. I also didn't run the two live tests separately from the full suite.
With either rule's queries empty there are no pairs, any([]) is False, and the 'could not be compared' warning main gives went away. Nothing proves such a pair disjoint, so it warns again. #468 refuses an empty list at load, but policy_conflicts can still be handed one, and this does not depend on the order the two PRs land in.
For null or a string, the message ended 'A rule with no queries asks Batfish nothing', which reads oddly when the rule does have something under 'queries'. A non-list and an empty list now get their own messages, each pinned by a test, with null added to the cases.
|
@SamikaPerera, agreed, and changed in |
patelankeet2
left a comment
There was a problem hiding this comment.
Re-reviewed with real testing, not a read-through (Samika's approval went stale after a follow-up commit).
Live-reproduced the three split error messages directly through load_policy():
empty list -> 'queries' is empty. A rule with no queries asks Batfish nothing
a string -> 'queries' must be a list of traffic descriptions, got str '10.10.10.0/24'
None -> 'queries' must be a list of traffic descriptions, got NoneType None
Matches the review follow-up (e99412b) exactly -- each now names its own problem instead of one message covering all three.
Full suite with Batfish running: 2485 passed, 13 skipped, 0 failed.
Approving.
The follow-up I offered on #442, with the two cases @ARSH871-bot listed under "What this does not do" on #467.
Before, measured on
main(2833a1e): each loads, then fails only at scan time"queries": []"queries": "10.10.10.0/24""queries": ["10.10.10.0/24"]TypeErrorfrom pybatfish{"srcIp": ...}(typo)TypeError: unexpected keyword argument 'srcIp'{"srcIps": "10.10.10.300"}BatfishException: Work terminated abnormallyNever unsafe (each ends in "could not check"), but always found late, with a message the user can't act on.
After
load_policy()refuses each one at upload, naming the rule and the problem, e.g.policy_compliance entry 1 ('x'), query 1: srcIps '10.10.10.300' is not a valid address or prefix. The same check coversaccess_control'sheaders, which go to the sameHeaderConstraints.HeaderConstraintsitself (attr.fields_dict), not copied, so they can't drift.a-b), wildcards (a:b), set difference and named groups, and refusing a valid rule would be worse than the bug. Only an element made purely of digits, dots and a slash is parsed. Each accepted form is run against real Batfish in the new tests, so the loader doesn't refuse anything Batfish can answer.{}still loads. It means all traffic, and "this filter must permit nothing" is a rule someone can mean.Tests:
tests/test_policy_traffic_shape.py, 23Every refusal with its message, every accepted form (including POL-1's set difference and the shipped example), and two live tests: the accepted forms really run in Batfish, and the refused address really fails there.
Mutation, 7 of 7 killed: empty list allowed, a string allowed, headers unchecked, unknown fields allowed, no address checked, every element treated as plain (false refusals, fails 7), and the specifier skip removed.
Two existing tests used placeholder traffic (
"headers": "x","queries": []) for things unrelated to traffic, so they now use a real query, astest_policy_loader.pyalready did forqueries.docs/policy-rules.mdgets a short section.2334 passed, 13 skipped with Batfish running; ruff clean on CI's selection. Merges with #467 independently: different functions in
policy.py.