Repository navigation
Check two policies against each other, not just each against itself (closes #419) - #426
Conversation
…loses #419) #419 (multi-policy conflict check). Stacked on #387, whose comparison code it reuses. load_policy() refuses a policy that contradicts itself, but it reads one file at a time. Two teams can each write a consistent policy for the same firewall that contradict each other once applied together. policy_conflicts.between() compares two policies in both directions -- A's prohibitions against B's requirements, and B's against A's -- and names every rule with its file, because both may have a "rule 1". The pairwise loop is now one function, _compare(), shared by contradictions() and between(), so the single-policy and two-policy checks cannot disagree about what a contradiction is. contradictions() behaves exactly as before: all 45 of its tests pass unchanged. `python -m analysis.policy_conflicts a.json b.json` loads each file the normal way (a file that contradicts itself is refused and named), prints CONFLICT and warning lines, and exits 1 on a conflict so it can gate a script. Mutation-tested: one direction only, rules not named with their policy, the CLI exiting 0 on a conflict, and undecidable pairs dropped each fail the new tests.
…r own regex Answers two of Samika's questions on #387. Mutation-testing _nets_meet and _ranges_meet killed 13 of 14 mutants; the survivor was removing the address de-duplication, which the port side already had a test for. Now 14 of 14. Kept off #387's branch so its approval is not voided for a test-only change.
patelankeet2
left a comment
There was a problem hiding this comment.
Reviewed with real testing, stacked on #387 as described -- tested this branch merged with current main (b69d591), which is the shape it will be in once #387 lands.
git merge origin/main (local only) clean, auto-merged pipeline.py
pytest tests/test_policy_between.py tests/test_policy_contradictions.py -q
59 passed
pytest tests/ -q 1962 passed, 13 skipped
ruff check --select E4,E7,E9,F ... clean
Live-reproduced the PR's own example, with the exact rule text from the description:
$ python -m analysis.policy_conflicts security.json app-owners.json
CONFLICT: security.json rule 1 ('Guests must not reach the finance server')
forbids what app-owners.json rule 1 ('The kiosk subnet must reach the
finance API over HTTPS') requires on rtr-us5 filter acl_in: both describe
src 10.10.10.64/26, dst 10.20.0.5/32, dst ports 443
exit: 1
Ran it again with the two files swapped on the command line -- same conflict, same attribution (security.json still named as the side that forbids). The message is keyed on each rule's own kind, not argument position, which is the right invariant for a two-sided check.
Independently mutation-tested the one I'd worry about most on a two-direction check: silently checking only one direction. Built two small policies where each has exactly one conflict with the other, but in opposite directions (A forbids X / B requires X, and separately B forbids Y / A requires Y). Both show up in one run:
CONFLICT: a.json rule 1 ('A forbids X') forbids what b.json rule 1 ('B requires X') requires ...
CONFLICT: b.json rule 2 ('B forbids Y') forbids what a.json rule 2 ('A requires Y') requires ...
Then commented out the second _compare() call in between() and re-ran the same two files: the second conflict (B forbids Y vs A requires Y) silently disappeared from the live CLI output, and test_both_directions_are_checked failed for exactly that reason. This is the one mutation I'd call safety-critical on this PR -- a two-policy conflict check that only looked one way would report "no conflict" about a real one, which is F-4 through the exact feature meant to prevent it -- and it's caught. Reverted.
Also checked the self-contradictory-file path directly, since the PR claims a distinct exit code for it:
$ python -m analysis.policy_conflicts bad.json ok.json
bad.json: not loaded -- this policy contradicts itself, so no configuration
could satisfy it: rule 1 ('forbid') forbids what rule 2 ('require') requires ...
exit: 2
Matches the table in the description (1 for a cross-policy conflict, 2 for a file that is broken on its own or a usage error).
On the structural change (_compare() shared by both callers): confirmed this is the right call rather than a risk -- contradictions()'s own 45 tests from #387 pass unchanged against the refactored shared function, so the one-policy and two-policy checks are now provably the same logic rather than two implementations that could quietly drift apart. entries_for() is a pre-existing Policy method, not new surface.
No em-dashes, no AI-tool mentions.
Approving.
Closes #419.
Stacked on #387 — the base is #387's branch, so this diff shows only the new commit. When #387 merges, GitHub retargets this to
mainon its own.What it does
load_policy()refuses a policy that contradicts itself (#322), but it reads one file at a time. Two teams can each write a consistent policy for the same firewall that contradict each other once applied together.policy_conflicts.between()compares two policies in both directions:_compare()warning:The one structural change worth reviewing
The pairwise loop from #387 is now one function,
_compare(), called by bothcontradictions()(one policy) andbetween()(two). The only difference between the callers is how a rule is named. So the two checks can never disagree about what a contradiction is.contradictions()behaves exactly as before: all 45 of #387's tests pass unchanged.Tests
tests/test_policy_between.py, 13 tests, including the CLI's three exit codes and a file that contradicts itself being refused by name.1947 passed, 13 skippedwith Batfish running. ruff clean.