Repository navigation
Tell me when an edit makes my policy weaker (closes #418) - #427
Conversation
A policy is edited many times and nothing compares an edit with the version before. A prohibition narrowed from 10.10.10.0/24 to /25 has stopped protecting half a subnet, and the diff is one character. analysis/policy_drift.py compares two versions of one policy and reports every place the new one protects less: a rule removed, a prohibition narrowed, a requirement widened, a rule that changed kind or moved. Added rules, wider prohibitions and narrower requirements are stricter and are not reported. CONTAINMENT, NOT OVERLAP. #387 asks whether two traffic descriptions overlap; a /24 and a /25 do, and the /25 still loses half the /24. This needs "is all of the old inside the new". One query against another is exact, field by field, with addresses collapsed first so two /25s contain their /24 and port ranges merged first. Against a rule with several queries it is exact when one holds it, when none touch it, or when only one touches it; when several might cover it together it says UNDECIDABLE, never "unchanged". Rules are paired by number; a renumbered rule reads as removed plus added, and the CLI warns when a file relies on assigned numbers. `python -m analysis.policy_drift old.json new.json` exits 1 when weaker. Builds on #387 (policy_conflicts' parser and overlap), so it is stacked on that branch. Mutation-tested: addresses not collapsed, port ranges not merged, the single-overlap and disjoint cases left undecidable, the requirement direction reversed, removed rules and kind changes ignored, "any" protocol treated as inside the closed list, the assigned-number warning removed, and the CLI exiting 0 -- each fails the new tests. The disjoint mutant first survived; a test with two non-touching new queries now pins it. 1962 passed, 13 skipped with Batfish running. ruff clean.
patelankeet2
left a comment
There was a problem hiding this comment.
Reviewed with real testing. Stacked on #387 as described, tested merged with current main (b69d591).
git merge origin/main (local only) clean, auto-merged pipeline.py
pytest tests/test_policy_drift.py -q 28 passed
pytest tests/ -q 1976 passed, 13 skipped
ruff check --select E4,E7,E9,F ... clean
Live-reproduced the PR's own /24-to-/25 example:
$ python -m analysis.policy_drift old.json new.json
WEAKER: rule 1 ('Guests must not reach finance') now forbids less: srcIps
10.10.10.0/24, dstIps 10.20.0.5 is no longer fully forbidden
exit: 1
Checked the one thing I'd most want to see gotten right on a "weaker" check: prohibition and requirement are opposite directions, and reversing them would be the kind of inversion this project has caught more than once elsewhere. Built a requirement widened from /25 to /24 (correctly WEAKER, more traffic now mandated to be let through) and ran the same files reversed (correctly NOT weaker, exit 0, since narrowing a requirement is stricter):
old /25 -> new /24 (widened): WEAKER: ... now requires more to be allowed ... exit 1
new /24 -> old /25 (narrowed): Not weaker: every rule ... at least as wide. exit 0
Both directions correct. Prohibition narrowing and requirement widening are both reported as weaker, and their opposites are both reported as stricter -- confirmed by direct reproduction, not just by reading the branch in drift().
Independently mutation-tested the guard your own account says survived once already -- the disjoint case in _inside_union. Commented out if not touching: return False, "":
FAILED test_a_prohibition_moved_to_entirely_different_traffic_is_weaker_not_undecidable
1 failed, 27 passed
Same count as reported. Confirms the fix holds, independent of trusting the PR's own table. Reverted.
On the design point you flagged for the hardest look (containment via several queries together is undecidable, never computed): agree this is the right call, for the same reason #387 refuses rather than guesses at Batfish specifier syntax -- computing a union of multi-field boxes to decide containment is a real piece of set algebra this module does not implement, and getting it subtly wrong would silently call a policy unchanged when it had actually narrowed. Undecidable and reported is the safe failure mode here, same as #387 and #426.
One small thing, not blocking: _field_inside's protocol branch (inner is ANY -> _FULL.get(field) -> None -> return False) is correct but easy to misread on a first pass, since every other field in _FULL IS present and this one deliberately isn't. The comment explains why, so this is just a note that a future reader skimming _FULL might expect a fourth entry there.
No em-dashes, no AI-tool mentions.
Approving.
# Conflicts: # docs/policy-rules.md
|
Merging #387 just now surfaced a real conflict here against current No code was touched, only the doc section order. Verified before merging: No new review needed for a conflict resolution with no functional change, but flagging it here for the record rather than merging silently. |
Closes #418.
Stacked on #387 (it uses
policy_conflicts' parser and overlap test), so the base is #387's branch and this diff shows only the new work. GitHub retargets it tomainwhen #387 merges.What it does
Compares two versions of one policy and reports every place the new one protects less:
number; renumbered = removed + addedThe design point worth reviewing: containment, not overlap
#387 asks whether two traffic descriptions overlap. A /24 and a /25 do, and the /25 still loses half the /24, so overlap alone can't see a narrowing. This needs "is all of the old inside the new":
Mutation-tested
The disjoint mutant survived the first run: no test had an old query that touched none of several new ones. One does now.
1962 passed, 13 skippedwith Batfish running. ruff clean.