Skip to content

Show which interfaces and subnets no policy rule talks about (closes #421) - #429

Merged
patelankeet2 merged 2 commits into
mainfrom
feat/421-policy-coverage-map
Nov 2, 2026
Merged

patelankeet2 merged 2 commits into
mainfrom
feat/421-policy-coverage-map

Conversation

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator

Closes #421. Stacked on #387 (uses policy_conflicts' address parser); GitHub retargets to main when #387 merges.

PC-049 reports devices no rule covers. This goes one level down, to each interface and its subnet:

$ python -m analysis.policy_coverage tests/fixtures/routing-secure my-policy.json
  rtr-branch   GigabitEthernet0/0     10.1.1.0/30        UNCOVERED
  rtr-branch   GigabitEthernet0/1     10.20.20.0/24      covered    rule(s) 1
  rtr-hq       GigabitEthernet0/0     10.10.10.0/24      covered    rule(s) 1
  rtr-hq       GigabitEthernet0/1     10.1.1.0/30        UNCOVERED
2 of 4 interface subnet(s) are covered by a policy rule
#421 criterion
interfaces and subnets from Batfish, not config text ✅ interfaceProperties
each one: does any rule's source or destination overlap it ✅
uncovered is coverage information, never a violation ✅ returns rows, emits no findings
unreadable device is unknown, never covered ✅
data the dashboard can draw ✅ coverage_map() rows — drawing it is @SamikaPerera's call
tested without Batfish, and once live ✅ 9 fake-data tests + 1 live

Design point worth checking: only rules about a device in the snapshot count. A rule about a missing device never runs (PC-050), so its addresses can't count as coverage. With the built-in rules (all about rtr-us5) on the routing fixture, the answer is 0 of 4 — correctly.

Mutation-tested: rules about absent devices counted, unsure reported as uncovered, destination ignored, unconstrained query ignored, unreadable device dropped — each fails the tests.

1944 passed, 13 skipped with Batfish running. ruff clean.

…421)

PC-049 reports devices no rule covers. That is too coarse: a router can be
"covered" by one rule about one subnet while most of its interfaces carry
traffic no rule mentions. analysis/policy_coverage.py goes to each interface
Batfish reports and asks whether any rule's source or destination overlaps
its subnet.

Only rules about a device that is in the snapshot count: a rule about a
missing device never runs (PC-050), so it covers nothing. Three answers, as
in the rest of the policy code: covered, uncovered, and unknown when the only
candidate rules use address syntax policy_conflicts does not parse. A device
whose interfaces cannot be listed is unknown, never covered. An uncovered
subnet is coverage information, not a violation.

coverage_map() returns rows for the dashboard to draw;
`python -m analysis.policy_coverage <config-dir> [policy.json]` prints them.
Live on routing-secure with a policy about the HQ LAN: both LANs covered, the
two point-to-point links uncovered.

Stacked on #387 for policy_conflicts' address parser.

Mutation-tested: rules about absent devices counted, unsure reported as
uncovered, the destination ignored, an unconstrained query ignored, and an
unreadable device dropped -- each fails the new tests.

1944 passed, 13 skipped with Batfish running. ruff clean.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with real testing against real Batfish, not just the fake-data tests. Stacked on #387, tested merged with current main (b69d591).

git merge origin/main (local only)              clean, auto-merged pipeline.py
pytest tests/test_policy_coverage.py -q          10 passed (includes the 1 live test)
pytest tests/ -q                                 1958 passed, 13 skipped
ruff check --select E4,E7,E9,F ...               clean

Live-reproduced both examples in the PR description, against real Batfish on tests/fixtures/routing-secure:

$ python -m analysis.policy_coverage tests/fixtures/routing-secure my-policy.json
  rtr-branch   GigabitEthernet0/0     10.1.1.0/30        UNCOVERED
  rtr-branch   GigabitEthernet0/1     10.20.20.0/24      covered    rule(s) 1
  rtr-hq       GigabitEthernet0/0     10.10.10.0/24      covered    rule(s) 1
  rtr-hq       GigabitEthernet0/1     10.1.1.0/30        UNCOVERED
2 of 4 interface subnet(s) are covered by a policy rule

$ python -m analysis.policy_coverage tests/fixtures/routing-secure   # built-in rules
  (all four UNCOVERED)
0 of 4 interface subnet(s) are covered by a policy rule

Both match the description exactly.

Independently mutation-tested the design point you asked to be checked hardest -- only rules about a device actually in the snapshot should count. Changed live_rules = [r for r in rules if r.get("node") in present] to include every rule regardless of device, and re-ran the exact same live command:

0 of 4 -> 4 of 4 interface subnet(s) are covered by a policy rule

The built-in rules name rtr-us5, which never appears in this snapshot and never runs here, but with the scoping removed they falsely report full coverage where the real answer is none. That's the dangerous direction exactly -- a false "policy says something about every interface" where the truth is "policy says nothing about this network at all." test_a_rule_about_a_device_not_in_the_snapshot_covers_nothing caught it too. This is the one thing on this PR I'd call safety-critical, and it's solid. Reverted both ways.

Checked the other stated guarantee -- an unreadable device reports covered: None (unknown), never False (uncovered is a real answer the policy earned by being silent; unknown is "we couldn't even list its interfaces") and never True. Read the code path (interfaces() raising is caught in main() and reported as "could not list interfaces, so coverage is unknown" rather than silently treated as zero interfaces) and it matches coverage_map()'s own handling of a device present in devices but absent from by_device.

No em-dashes, no AI-tool mentions.

Approving.

Base automatically changed from feat/322-policy-contradictions to main November 2, 2026 01:35
@patelankeet2

Copy link
Copy Markdown
Collaborator

Same conflict as #427 hit, against the same point in docs/policy-rules.md, this time against main after both #426 and #427 had landed. Resolved the same way: this PR's new section now sits after #426's and #427's sections (and #426's own #### What is not a contradiction / #### Proven, not presumed subsections), rather than interposed before them. No code touched, only the doc section order.

Verified before merging:

pytest tests/test_policy_coverage.py tests/test_policy_drift.py
       tests/test_policy_between.py tests/test_policy_contradictions.py -q   97 passed
pytest tests/ -q                                                             2193 passed, 13 skipped
ruff check --select E4,E7,E9,F ...                                           clean

@patelankeet2
patelankeet2 merged commit 79f921b into main Nov 2, 2026
2 checks passed
@patelankeet2
patelankeet2 deleted the feat/421-policy-coverage-map branch November 2, 2026 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Policy coverage map — which interfaces and subnets no policy rule covers

2 participants