Skip to content

Tell me when an edit makes my policy weaker (closes #418) - #427

Merged
patelankeet2 merged 2 commits into
mainfrom
feat/418-policy-drift
Nov 2, 2026
Merged

patelankeet2 merged 2 commits into
mainfrom
feat/418-policy-drift

Conversation

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator

Closes #418.

Stacked on #387 (it uses policy_conflicts' parser and overlap test), so the base is #387's branch and this diff shows only the new work. GitHub retargets it to main when #387 merges.

What it does

Compares two versions of one policy and reports every place the new one protects less:

$ python -m analysis.policy_drift old.json new.json
WEAKER: rule 1 ('...') now forbids less: srcIps 10.10.10.0/24 is no longer fully forbidden
(exit status 1)
change weaker?
rule removed yes
prohibition narrowed — forbids less yes
requirement widened — requires more to be allowed yes
kind changed, or moved to another device or filter yes
rule added, prohibition widened, requirement narrowed no, stricter
#418 criterion
pure logic, no Batfish ✅
names the rule, the traffic and the direction ✅
paired by number; renumbered = removed + added ✅ and the CLI warns when a file relies on assigned numbers
undecidable reported, never "unchanged" ✅
CLI, exit 1 when weaker ✅
mutation-tested ✅

The design point worth reviewing: containment, not overlap

#387 asks whether two traffic descriptions overlap. A /24 and a /25 do, and the /25 still loses half the /24, so overlap alone can't see a narrowing. This needs "is all of the old inside the new":

  • One query against another: exact, field by field. Addresses are collapsed first, so two /25s do contain their /24; port ranges are merged first.
  • Against a rule with several queries: exact when one holds it, when none touch it, or when only one touches it.
  • When several might cover it together: undecidable. I don't compute unions of multi-field boxes. A /24 split into two /25 queries with the same ports is covered, and is reported as undecidable — the safe direction, since it never calls a policy unchanged without proof. A test pins that limit so it is visible, not hidden.

Mutation-tested

addresses not collapsed                  2 failed
port ranges not merged                   1 failed
single-overlap case left undecidable     4 failed
disjoint case left undecidable           1 failed   (first SURVIVED; test added)
requirement direction reversed           2 failed
removed rule ignored                     1 failed
kind change ignored                      1 failed
"any" protocol treated as in the list    1 failed
assigned-number warning removed          1 failed
CLI exits 0 when weaker                  1 failed

The disjoint mutant survived the first run: no test had an old query that touched none of several new ones. One does now.

1962 passed, 13 skipped with Batfish running. ruff clean.

A policy is edited many times and nothing compares an edit with the version
before. A prohibition narrowed from 10.10.10.0/24 to /25 has stopped
protecting half a subnet, and the diff is one character.

analysis/policy_drift.py compares two versions of one policy and reports
every place the new one protects less: a rule removed, a prohibition
narrowed, a requirement widened, a rule that changed kind or moved. Added
rules, wider prohibitions and narrower requirements are stricter and are not
reported.

CONTAINMENT, NOT OVERLAP. #387 asks whether two traffic descriptions overlap;
a /24 and a /25 do, and the /25 still loses half the /24. This needs "is all
of the old inside the new". One query against another is exact, field by
field, with addresses collapsed first so two /25s contain their /24 and port
ranges merged first. Against a rule with several queries it is exact when
one holds it, when none touch it, or when only one touches it; when several
might cover it together it says UNDECIDABLE, never "unchanged".

Rules are paired by number; a renumbered rule reads as removed plus added,
and the CLI warns when a file relies on assigned numbers.

`python -m analysis.policy_drift old.json new.json` exits 1 when weaker.

Builds on #387 (policy_conflicts' parser and overlap), so it is stacked on
that branch.

Mutation-tested: addresses not collapsed, port ranges not merged, the
single-overlap and disjoint cases left undecidable, the requirement direction
reversed, removed rules and kind changes ignored, "any" protocol treated as
inside the closed list, the assigned-number warning removed, and the CLI
exiting 0 -- each fails the new tests. The disjoint mutant first survived;
a test with two non-touching new queries now pins it.

1962 passed, 13 skipped with Batfish running. ruff clean.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with real testing. Stacked on #387 as described, tested merged with current main (b69d591).

git merge origin/main (local only)              clean, auto-merged pipeline.py
pytest tests/test_policy_drift.py -q             28 passed
pytest tests/ -q                                 1976 passed, 13 skipped
ruff check --select E4,E7,E9,F ...               clean

Live-reproduced the PR's own /24-to-/25 example:

$ python -m analysis.policy_drift old.json new.json
WEAKER: rule 1 ('Guests must not reach finance') now forbids less: srcIps
10.10.10.0/24, dstIps 10.20.0.5 is no longer fully forbidden
exit: 1

Checked the one thing I'd most want to see gotten right on a "weaker" check: prohibition and requirement are opposite directions, and reversing them would be the kind of inversion this project has caught more than once elsewhere. Built a requirement widened from /25 to /24 (correctly WEAKER, more traffic now mandated to be let through) and ran the same files reversed (correctly NOT weaker, exit 0, since narrowing a requirement is stricter):

old /25 -> new /24 (widened):    WEAKER: ... now requires more to be allowed ...   exit 1
new /24 -> old /25 (narrowed):   Not weaker: every rule ... at least as wide.       exit 0

Both directions correct. Prohibition narrowing and requirement widening are both reported as weaker, and their opposites are both reported as stricter -- confirmed by direct reproduction, not just by reading the branch in drift().

Independently mutation-tested the guard your own account says survived once already -- the disjoint case in _inside_union. Commented out if not touching: return False, "":

FAILED test_a_prohibition_moved_to_entirely_different_traffic_is_weaker_not_undecidable
1 failed, 27 passed

Same count as reported. Confirms the fix holds, independent of trusting the PR's own table. Reverted.

On the design point you flagged for the hardest look (containment via several queries together is undecidable, never computed): agree this is the right call, for the same reason #387 refuses rather than guesses at Batfish specifier syntax -- computing a union of multi-field boxes to decide containment is a real piece of set algebra this module does not implement, and getting it subtly wrong would silently call a policy unchanged when it had actually narrowed. Undecidable and reported is the safe failure mode here, same as #387 and #426.

One small thing, not blocking: _field_inside's protocol branch (inner is ANY -> _FULL.get(field) -> None -> return False) is correct but easy to misread on a first pass, since every other field in _FULL IS present and this one deliberately isn't. The comment explains why, so this is just a note that a future reader skimming _FULL might expect a fourth entry there.

No em-dashes, no AI-tool mentions.

Approving.

Base automatically changed from feat/322-policy-contradictions to main November 2, 2026 01:35
@patelankeet2

Copy link
Copy Markdown
Collaborator

Merging #387 just now surfaced a real conflict here against current main -- both this PR and #426 inserted their own new ### section into docs/policy-rules.md at the same point. Resolved it, not just taken a side: main's own already-merged structure (from #426) has #### What is not a contradiction and #### Proven, not presumed as genuine subsections of "Two policies that contradict each other", since those describe the shared comparison logic both the single-policy and two-policy checks use. This PR's new section now sits after that whole block instead of interposed inside it, which is where it belongs either way.

No code was touched, only the doc section order. Verified before merging:

pytest tests/test_policy_drift.py tests/test_policy_between.py
       tests/test_policy_contradictions.py -q      87 passed
pytest tests/ -q                                    2181 passed, 13 skipped
ruff check --select E4,E7,E9,F ...                  clean

No new review needed for a conflict resolution with no functional change, but flagging it here for the record rather than merging silently.

@patelankeet2
patelankeet2 merged commit 263ece9 into main Nov 2, 2026
2 checks passed
@patelankeet2
patelankeet2 deleted the feat/418-policy-drift branch November 2, 2026 02:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Policy drift detector — tell me when an edit makes my policy weaker

2 participants