Repository navigation
Show which interfaces and subnets no policy rule talks about (closes #421) - #429
Conversation
…421) PC-049 reports devices no rule covers. That is too coarse: a router can be "covered" by one rule about one subnet while most of its interfaces carry traffic no rule mentions. analysis/policy_coverage.py goes to each interface Batfish reports and asks whether any rule's source or destination overlaps its subnet. Only rules about a device that is in the snapshot count: a rule about a missing device never runs (PC-050), so it covers nothing. Three answers, as in the rest of the policy code: covered, uncovered, and unknown when the only candidate rules use address syntax policy_conflicts does not parse. A device whose interfaces cannot be listed is unknown, never covered. An uncovered subnet is coverage information, not a violation. coverage_map() returns rows for the dashboard to draw; `python -m analysis.policy_coverage <config-dir> [policy.json]` prints them. Live on routing-secure with a policy about the HQ LAN: both LANs covered, the two point-to-point links uncovered. Stacked on #387 for policy_conflicts' address parser. Mutation-tested: rules about absent devices counted, unsure reported as uncovered, the destination ignored, an unconstrained query ignored, and an unreadable device dropped -- each fails the new tests. 1944 passed, 13 skipped with Batfish running. ruff clean.
patelankeet2
left a comment
There was a problem hiding this comment.
Reviewed with real testing against real Batfish, not just the fake-data tests. Stacked on #387, tested merged with current main (b69d591).
git merge origin/main (local only) clean, auto-merged pipeline.py
pytest tests/test_policy_coverage.py -q 10 passed (includes the 1 live test)
pytest tests/ -q 1958 passed, 13 skipped
ruff check --select E4,E7,E9,F ... clean
Live-reproduced both examples in the PR description, against real Batfish on tests/fixtures/routing-secure:
$ python -m analysis.policy_coverage tests/fixtures/routing-secure my-policy.json
rtr-branch GigabitEthernet0/0 10.1.1.0/30 UNCOVERED
rtr-branch GigabitEthernet0/1 10.20.20.0/24 covered rule(s) 1
rtr-hq GigabitEthernet0/0 10.10.10.0/24 covered rule(s) 1
rtr-hq GigabitEthernet0/1 10.1.1.0/30 UNCOVERED
2 of 4 interface subnet(s) are covered by a policy rule
$ python -m analysis.policy_coverage tests/fixtures/routing-secure # built-in rules
(all four UNCOVERED)
0 of 4 interface subnet(s) are covered by a policy rule
Both match the description exactly.
Independently mutation-tested the design point you asked to be checked hardest -- only rules about a device actually in the snapshot should count. Changed live_rules = [r for r in rules if r.get("node") in present] to include every rule regardless of device, and re-ran the exact same live command:
0 of 4 -> 4 of 4 interface subnet(s) are covered by a policy rule
The built-in rules name rtr-us5, which never appears in this snapshot and never runs here, but with the scoping removed they falsely report full coverage where the real answer is none. That's the dangerous direction exactly -- a false "policy says something about every interface" where the truth is "policy says nothing about this network at all." test_a_rule_about_a_device_not_in_the_snapshot_covers_nothing caught it too. This is the one thing on this PR I'd call safety-critical, and it's solid. Reverted both ways.
Checked the other stated guarantee -- an unreadable device reports covered: None (unknown), never False (uncovered is a real answer the policy earned by being silent; unknown is "we couldn't even list its interfaces") and never True. Read the code path (interfaces() raising is caught in main() and reported as "could not list interfaces, so coverage is unknown" rather than silently treated as zero interfaces) and it matches coverage_map()'s own handling of a device present in devices but absent from by_device.
No em-dashes, no AI-tool mentions.
Approving.
# Conflicts: # docs/policy-rules.md
|
Same conflict as #427 hit, against the same point in Verified before merging: |
Closes #421. Stacked on #387 (uses
policy_conflicts' address parser); GitHub retargets tomainwhen #387 merges.PC-049 reports devices no rule covers. This goes one level down, to each interface and its subnet:
interfacePropertiescoverage_map()rows — drawing it is @SamikaPerera's callDesign point worth checking: only rules about a device in the snapshot count. A rule about a missing device never runs (PC-050), so its addresses can't count as coverage. With the built-in rules (all about
rtr-us5) on the routing fixture, the answer is 0 of 4 — correctly.Mutation-tested: rules about absent devices counted, unsure reported as uncovered, destination ignored, unconstrained query ignored, unreadable device dropped — each fails the tests.
1944 passed, 13 skippedwith Batfish running. ruff clean.