Skip to content

Check two policies against each other, not just each against itself (closes #419) - #426

Merged
patelankeet2 merged 2 commits into
mainfrom
feat/multi-policy-conflicts
Nov 2, 2026
Merged

patelankeet2 merged 2 commits into
mainfrom
feat/multi-policy-conflicts

Conversation

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator

Closes #419.

Stacked on #387 — the base is #387's branch, so this diff shows only the new commit. When #387 merges, GitHub retargets this to main on its own.

What it does

load_policy() refuses a policy that contradicts itself (#322), but it reads one file at a time. Two teams can each write a consistent policy for the same firewall that contradict each other once applied together. policy_conflicts.between() compares two policies in both directions:

$ python -m analysis.policy_conflicts security.json app-owners.json
CONFLICT: security.json rule 1 ('Guests must not reach the finance server')
forbids what app-owners.json rule 1 ('The kiosk subnet must reach the finance
API over HTTPS') requires on rtr-us5 filter acl_in: both describe
src 10.10.10.64/26, dst 10.20.0.5/32, dst ports 443, protocol tcp
(exit status 1)
#419 criterion
both directions ✅ A's prohibitions vs B's requirements, and B's vs A's
every rule named with its file ✅ both files can have a "rule 1"
same comparison as within one policy ✅ one shared loop, _compare()
undecidable pairs reported, never dropped ✅ printed as warning:
command line exits 1 on a conflict ✅ and 2 for usage errors or a file that contradicts itself
mutation-tested ✅ below

The one structural change worth reviewing

The pairwise loop from #387 is now one function, _compare(), called by both contradictions() (one policy) and between() (two). The only difference between the callers is how a rule is named. So the two checks can never disagree about what a contradiction is. contradictions() behaves exactly as before: all 45 of #387's tests pass unchanged.

Tests

tests/test_policy_between.py, 13 tests, including the CLI's three exit codes and a file that contradicts itself being refused by name.

only one direction checked              1 failed
rule not named with its policy          5 failed
CLI exits 0 on a conflict               1 failed
undecidable pairs dropped               1 failed

1947 passed, 13 skipped with Batfish running. ruff clean.

…loses #419)

#419 (multi-policy conflict check). Stacked on #387,
whose comparison code it reuses.

load_policy() refuses a policy that contradicts itself, but it reads one file
at a time. Two teams can each write a consistent policy for the same
firewall that contradict each other once applied together.
policy_conflicts.between() compares two policies in both directions -- A's
prohibitions against B's requirements, and B's against A's -- and names
every rule with its file, because both may have a "rule 1".

The pairwise loop is now one function, _compare(), shared by contradictions()
and between(), so the single-policy and two-policy checks cannot disagree
about what a contradiction is. contradictions() behaves exactly as before:
all 45 of its tests pass unchanged.

`python -m analysis.policy_conflicts a.json b.json` loads each file the
normal way (a file that contradicts itself is refused and named), prints
CONFLICT and warning lines, and exits 1 on a conflict so it can gate a
script.

Mutation-tested: one direction only, rules not named with their policy, the
CLI exiting 0 on a conflict, and undecidable pairs dropped each fail the new
tests.
…r own regex

Answers two of Samika's questions on #387. Mutation-testing _nets_meet and
_ranges_meet killed 13 of 14 mutants; the survivor was removing the address
de-duplication, which the port side already had a test for. Now 14 of 14.

Kept off #387's branch so its approval is not voided for a test-only change.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with real testing, stacked on #387 as described -- tested this branch merged with current main (b69d591), which is the shape it will be in once #387 lands.

git merge origin/main (local only)              clean, auto-merged pipeline.py
pytest tests/test_policy_between.py tests/test_policy_contradictions.py -q
                                                  59 passed
pytest tests/ -q                                 1962 passed, 13 skipped
ruff check --select E4,E7,E9,F ...               clean

Live-reproduced the PR's own example, with the exact rule text from the description:

$ python -m analysis.policy_conflicts security.json app-owners.json
CONFLICT: security.json rule 1 ('Guests must not reach the finance server')
forbids what app-owners.json rule 1 ('The kiosk subnet must reach the
finance API over HTTPS') requires on rtr-us5 filter acl_in: both describe
src 10.10.10.64/26, dst 10.20.0.5/32, dst ports 443
exit: 1

Ran it again with the two files swapped on the command line -- same conflict, same attribution (security.json still named as the side that forbids). The message is keyed on each rule's own kind, not argument position, which is the right invariant for a two-sided check.

Independently mutation-tested the one I'd worry about most on a two-direction check: silently checking only one direction. Built two small policies where each has exactly one conflict with the other, but in opposite directions (A forbids X / B requires X, and separately B forbids Y / A requires Y). Both show up in one run:

CONFLICT: a.json rule 1 ('A forbids X') forbids what b.json rule 1 ('B requires X') requires ...
CONFLICT: b.json rule 2 ('B forbids Y') forbids what a.json rule 2 ('A requires Y') requires ...

Then commented out the second _compare() call in between() and re-ran the same two files: the second conflict (B forbids Y vs A requires Y) silently disappeared from the live CLI output, and test_both_directions_are_checked failed for exactly that reason. This is the one mutation I'd call safety-critical on this PR -- a two-policy conflict check that only looked one way would report "no conflict" about a real one, which is F-4 through the exact feature meant to prevent it -- and it's caught. Reverted.

Also checked the self-contradictory-file path directly, since the PR claims a distinct exit code for it:

$ python -m analysis.policy_conflicts bad.json ok.json
bad.json: not loaded -- this policy contradicts itself, so no configuration
could satisfy it: rule 1 ('forbid') forbids what rule 2 ('require') requires ...
exit: 2

Matches the table in the description (1 for a cross-policy conflict, 2 for a file that is broken on its own or a usage error).

On the structural change (_compare() shared by both callers): confirmed this is the right call rather than a risk -- contradictions()'s own 45 tests from #387 pass unchanged against the refactored shared function, so the one-policy and two-policy checks are now provably the same logic rather than two implementations that could quietly drift apart. entries_for() is a pre-existing Policy method, not new surface.

No em-dashes, no AI-tool mentions.

Approving.

Base automatically changed from feat/322-policy-contradictions to main November 2, 2026 01:35
@patelankeet2
patelankeet2 merged commit f569901 into main Nov 2, 2026
2 checks passed
@patelankeet2
patelankeet2 deleted the feat/multi-policy-conflicts branch November 2, 2026 01:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-policy conflict check — two policies that contradict each other

2 participants