Repository navigation
Change impact reports what it did not look at (closes #403, most of #402) - #406
Conversation
) #403. Deleting a whole router reported "No change detected" with a green tick: neither compareFilters nor differentialReachability describes a device that exists on one side only. analyse_change() now compares the device lists of both snapshots and reports each added or removed device as `found`. A side whose devices cannot be listed is an error (CH-052), never "no change". Reproduced live on multi-device-10 minus rtr-dev009. #402, from @ARSH871-bot's post-merge review of #389: 1. A rule about a device in NEITHER config is not applicable -- this change cannot affect it -- and is named as such, not reported as an error. Every built-in rule used to be CH-051 on a network without rtr-us5, so every proposal there was "could not be fully verified". A rule about a device on ONE side only is one CH-051. Reproduced live with rtr-us5 renamed edge-fw1. 3. Only policy_compliance rules are compared, and the result now says so: a sentence beginning "NOT compared:" names the access_control and routing rules it skipped, and a policy with no policy_compliance rules says no rule of the user's was checked. 4. A rule breached before AND after is `found`, never inside the green card. A differential searchFilters tells "breached the same way" (low, not caused by this change) from "breached by different traffic" (the rule's own severity, direction: changed). "Unchanged" is scoped to the rule's filter, because unbinding a filter changes the network without changing that answer. 5. _rule_breached is tested against a fake session, so CI now catches the four mutants only the skipped live test could: the policy passed in replaced by None, only the first query arm asked, the snapshot not switched, and the rule's kind ignored. 6. Every stage runs whatever the others did, so a failed reachability stage no longer silently skips the policy stage. Ordinary findings stop at 48; 49 and up are reserved (CH-049 summarises any overflow, CH-050..054 are this module's own error cards), so a long diff cannot collide with them. The CLI prints renamed and assigned notes. 7. Docs: the module docstring lists what is reported and in what order, CLAUDE.md's change_impact row is current, and the policy-rules section moved out from above section 7's sign-off into its own section 8. Not here: item 2 is #405 (Ankeet's). Shubham's rebase onto it, whichever lands second, absorbs the conflict in _policy_change_findings. Mutation-tested: device changes dropped, absent-device rules made errors, set_active_policy(None), first arm only, reachability failure skipping the policy stage, no cap, still-breached back in the green card, snapshot not switched -- each fails the new tests; the arm, snapshot and kind mutants fail WITHOUT Batfish. 1907 passed, 13 skipped with Batfish running. ruff clean on touched files.
patelankeet2
left a comment
There was a problem hiding this comment.
APPROVE. Tested, not just read.
Independently reproduced both headline fixes, live
#403's own repro (multi-device-10 minus rtr-dev009):
CH-001 found medium rtr-dev009 | Device rtr-dev009 is removed by this change
"No change detected"? absent, confirmed
#402 item 1's repro (rtr-us5 renamed edge-fw1, both sides):
CH-000 none low | No change detected between the two configurations
any status=="error"? False, confirmed
Both match the PR body's own tables exactly.
Read the full diff, not just the summary
Traced the F-4-sensitive part carefully: the old code returned immediately on a filter- or reachability-stage exception, abandoning every later stage. Now each stage appends a reserved-id error card (blind()) and continues, so a failed reachability check can no longer silently skip the policy stage -- and confirmed results = _cap(regular) + reserved means the "no change detected" sentinel is only reachable when BOTH are empty, so an error card can never collapse into a false all-clear. That's the right direction for F-4 and I checked it holds rather than assumed it from the description.
Mutation-tested one of your own claims rather than only trusting the table
Reverted _rule_breached to check only rule["queries"][0] instead of looping every arm -- test_rule_breached_asks_every_arm_not_just_the_first failed for exactly the right reason, nothing else did. Reverted.
Verified independently
python -m pytest tests/ -q 1907 passed, 13 skipped (matches)
python -m ruff check analysis/change_impact.py tests/test_change_impact_coverage.py
tests/test_change_against_policy.py tests/test_change_impact.py all clean
git log -1 --format=%ad --date=short dc93ccb 2026-10-14 -- matches the
CLAUDE.md claim "#323 merged 14 October"
On the #405 merge-order note
Agreed with your plan -- whichever lands second absorbs the conflict. I've also replied on #405's own thread to your direction-ratification question (you're right that my comparison to explanation/explanation_source didn't hold; it's a producer-side key, not a response-layer one, so it's the same class of question as #390's). Not blocking this PR either way, just flagging that I've seen it from both sides so nobody has to repeat the finding a third time.
|
Heads-up for merge order, measured today on current
I'm not pushing it now: that would bring #387's commits into this branch before #387 lands, and void @patelankeet2's approval. Once #387 merges, I'll merge |
# Conflicts: # analysis/change_impact.py
|
@shubhamkataria2005 picked this up since you hadn't gotten to it yet and main had moved a lot since your 31 October note -- but this is your branch and your call, so posting the verification rather than merging it myself. Did exactly what you described: merged main in and resolved the conflict with notes before warnings, matching for note in list(policy.renamed) + list(policy.assigned):
print(f"note: {note}")
for note in policy.undecidable:
print(f"warning: {note}")Live-confirmed the actual order rather than just reading the diff -- built a policy with both an unnumbered rule (triggers a Notes first, warnings second, exactly as planned. Pushed as |
ARSH871-bot
left a comment
There was a problem hiding this comment.
Update, 3 November. Re-checked on 39ecfcf, your branch after Ankeet merged main into it. That merge only resolves the CLI conflict (notes before warnings, as you planned) and is fine. All three blockers below are still in the code at this head: _breach_changed (change_impact.py:421-447) with :564-570 for B1, :539 and :547 with no case-folding for B2, and the hand-built CH-051 at tests/test_change_against_policy.py:57 for B3. #428's score() sits on this branch and inherits B1: a tightening counts as high.
My first pass, on 1503726 (21 October): Live, 1907 passed / 13 skipped, and CI-condition 1869 / 51 both match. #403 is fixed: multi-device-10 minus rtr-dev009 gives CH-001 found "Device rtr-dev009 is removed", where main said "No change detected". #402.1 is fixed: edge-fw1 now proposes verified=True, where main gave CH-051 ×5. Your four item-5 mutants all fail without Batfish. The 49+ reservation holds: 60 removed lines give 48 + CH-049, with no duplicate ids. A raising reachability stage no longer hides the policy stage. Three things need fixing before merge:
- A pure deny now raises the "opens access" banner.
propose_change("block 10.10.10.5 to 10.20.0.5 on tcp/22 on rtr-us5", rtr-us5-insecure): main giveswarning=False, "narrows access"; this branch giveswarning=True, "newly opens access that was previously blocked". Rule 2 is breached on both sides, so_breach_changed(change_impact.py:421) returns True and :570 rates ithigh. The differential frame already tells you the direction: on this caseSnapshot_Action=DENY,Reference_Action=PERMIT. Please use it. A shrinking breach is a tightening, not "changed" at high. With #405 merged it still warns, just with different wording. - Device presence is case-sensitive, and it fails as a tick. Config
hostname RTR-US5, policydevice: RTR-US5, change removes DNS. Main:CH-002 found "This change BREAKS policy rule 1". Here: ✓ "devices in neither configuration, which this change cannot affect: 1 (RTR-US5)". :539 compares the rule's node as written againstsnapshot.device_names(), which holds Batfish's lower-cased names. searchFilters matches case-insensitively, which is why main found the breach. A scan of the same file says PC-050, amber. Please casefold the comparison. Also, a user's own absent rule shouldn't be a ✓ when the scan calls it "could not check". - CH-051 has no test. The
stage()helper builds its own CH-051 (test_change_against_policy.py:57), sotest_the_ids_this_stage_emits_are_uniquechecks an id the test made up. The test that asserted a real error card on main now checksout["not_compared"](:138). Deleting the CH-051 block at :758 passes the whole suite, with and without Batfish. So does droppingone_sidefrom it.
Also, not blocking:
_breach_changedis never exercised: droppingreference_snapshot, or asking only the first arm, both survive.- The filter-failure path is untested. Silencing it turns a failed compareFilters into a green "No change detected".
- The "NOT compared:" sentence sits in a ✓ card titled "Policy rules this change did not move", even when the change breaks that rule. AC-005 (#261) is the precedent for making that an error.
_capfolds the policy cards first. 60 lines plus DNS removed hides "BREAKS rule 4" inside "15 further change(s)".- CLAUDE.md:716 credits "#323 (merged 14 October)". #323 is the issue; the PR was #389.
- The FINDING IDS block at :85 wasn't updated alongside :110.
- Open question 2 in the shapes doc is untouched, although #402 item 7 named it.
Merge order: with #405, once the textual conflict in _policy_change_findings is resolved, #405's new test_direction_field_is_set_regardless_of_severity still unpacks three values from stage(), giving 4 × ValueError. Whoever lands second needs that one line. With #387 there is one conflict in change_impact.main() (both add a note loop; keep both). #390 merges clean.
@patelankeet2, you approved this on 20 October. Items 1 and 2 reproduce through the public entry points named above, so please re-check them before this lands.
Arsh's three blockers on #406: 1. A pure deny raised the "opens access" banner. _breach_changed now returns (grew, shrank) from the differential frame's two verdicts; a breach that only shrinks is "narrows", low, and only a breach that grows keeps the rule's own severity. Batfish returns one example per direction, measured on a change that both adds and removes breaching flows. 2. Device presence was case-sensitive, so policy device RTR-US5 against Batfish's rtr-us5 filed a real breach under a tick. Both sides are now casefolded, and a user's own rule about a device in neither config is part of CH-051, as a scan calls it PC-050. 3. CH-051 was only ever built by the test helper. The helper now returns the stage's own output, and CH-051 is tested through analyse_change(). Also from the review: tests that _breach_changed passes the reference snapshot and asks every arm, a test that a failed compareFilters is CH-050 and never "No change detected", _cap keeps the most severe cards (policy first at equal severity) instead of the first 48, the FINDING IDS block, and the #389 credit in CLAUDE.md. Arsh's two live reproductions are now live tests.
|
@ARSH871-bot, all three blockers are fixed in
Your non-blocking list:
Mutation: 10 mutants across the three fixes and the Your two reproductions are live tests now:
For #405 when it lands second: Under M-2 rule 1 your review and Ankeet's approval are void now, so this needs fresh looks. |
patelankeet2
left a comment
There was a problem hiding this comment.
Re-checked both items you specifically asked me to, through the public entry points, not just the unit tests -- my 20 October approval was against the pre-review-fix state, so this supersedes it.
Item 1, live: propose_change("block 10.10.10.5 to 10.20.0.5 on tcp/22 on rtr-us5", rtr-us5-insecure) with the built-in policy active (your exact repro). Result: warning: False, "narrows access; it does not appear to newly open anything" -- matches main, not the "newly opens access" banner the pre-fix branch gave.
Item 2, live, against real Batfish: ran test_live_a_device_named_in_capitals_still_reports_the_breach directly -- hostname RTR-US5, policy device: RTR-US5, DNS removed. Passes: the breach is reported as BREAKS policy rule 1, high, not filed as not-compared.
Item 3: confirmed the test helper no longer fabricates CH-051 -- stage() returns only the stage's own findings now, and test_rules_not_compared_become_a_real_ch051_error builds the CH-051 card through the real analyse_change(), stubbed one level up. Real code, not a hand-built dict pretending to be its output.
Mutation-tested _breach_changed's new two-value return by collapsing it back to the old single bool: 8 tests failed, including the live Batfish test, with exactly item 1's bug reproduced in the failure message ("newly opens access" where it shouldn't). Restored.
Bonus, not required but done anyway: the _cap() reordering (policy cards kept first, not whichever 48 happened to run first) and a real test for the untested filter-failure-silently-becomes-clean path are both in this commit too, covering two of your explicitly "not blocking" items.
Full suite: 2282 passed, 13 skipped, 0 failed. ruff clean.
Approving, with my 20 October approval now correctly superseded by this one rather than left stale alongside it.
…pact-coverage # Conflicts: # CLAUDE.md
|
Merge-only push ( |
…rds (#405 review, Arsh) The structural direction field already uses opening/closing/changed and is already relied on by opens_access -- not what this proposes changing. What is inconsistent is the human-readable Direction: X phrase in evidence.detail: change_impact.py:488 still says tightening for the case the field calls closing, and #406 separately invented changed for its own case. Proposes one vocabulary for both. Not changing the wording itself here -- that is exactly what this table gates, same as A-4.
ARSH871-bot
left a comment
There was a problem hiding this comment.
Re-review on d4587ca One new blocker, small, with a tested fix. All three of my 3 November blockers are fixed.
My three blockers, re-checked live against Batfish on this head
- Pure deny:
propose_change("block 10.10.10.5 to 10.20.0.5 on tcp/22 on rtr-us5", rtr-us5-insecure)iswarning=False, "narrows access", rated low. Fixed. Direction is read from the differential frame; grew, shrank and both-ways each come out right. - Capitals:
hostname RTR-US5+device: RTR-US5, DNS removed, givesCH-002 BREAKSagain, and a rule about a device in neither config is no longer a tick. Fixed. - CH-051 is a real card now. Deleting the block, dropping
one_sideor droppingnot_comparedeach failtest_rules_not_compared_become_a_real_ch051_error. Fixed.
Mutation run on this head: 16 mutants, 15 caught without Batfish (all of the above plus _breach_changed, the filter-stage failure, casefolding, and the cap ordering). CI-condition 2306 passed / 58 skipped; live 2351 passed / 13 skipped. The minor items (CLAUDE.md credit to #389, the FINDING IDS block) are fixed.
New: _cap can produce a duplicate id (blocking, but a few lines)
_cap now keeps the most important 48, which is right, but each kept finding keeps the number it was found under. With mixed severities the 49th finding found is kept, so it is CH-049, the same id as the "N further change(s)" card. Kept findings numbered 50-54 also sit on the ids this module reserves for CH-050..CH-054.
Repro (60 findings, odd ones high, even ones medium): duplicate ids: ['CH-049'], and 49 ids but 48 distinct. This is a different bug from the old one: on main the 60-line case had no cap and so collided at CH-051; the cap reduced that but does not remove it. The duplicate-id guard in the pipeline will not say a word about it until it fires on someone's screen.
Your existing test_a_long_diff_never_folds_away_a_broken_policy_rule uses 60 findings of one severity, so the cap keeps the first 48 and nothing past 48 is ever kept; it cannot see this. It also did not catch my 16th mutant, "cap: severity ignored", which survived the whole suite.
Suggested fix, tested (patch below): after kept is built in _cap, renumber it in the order found, so CH-001..048 are the ordinary cards and 049 is only ever the summary. Plus two tests:
test_a_capped_list_is_numbered_one_to_forty_eight_whatever_was_kept: mixed severities; fails ond4587cawith the duplicate; fails again if the renumber is removed.test_a_capped_list_keeps_the_worst_severity_not_the_first_found: 55 medium then 5 high; fails if severity is ignored.
On the patched tree: full suite 2353 passed / 13 skipped live, change tests 73 passed. Renumbering does change a kept finding's id relative to when it was found, but nothing outside _cap uses that id (the list is returned as a whole, in order).
Merge order, unchanged from 3 Nov: #463 and #428 are stacked on this; retarget them to main before this merges. #405 will conflict in _policy_change_findings and its test_direction_field_is_set_regardless_of_severity still unpacks three values from stage().
Verdict: request changes for the one item above; otherwise this is ready.
Patch
diff --git a/analysis/change_impact.py b/analysis/change_impact.py
index 875a3d6..e6e3a22 100644
--- a/analysis/change_impact.py
+++ b/analysis/change_impact.py
@@ -667,6 +667,12 @@ def _cap(regular: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
keep = set(sorted(range(len(regular)), key=importance)[: _OVERFLOW_NUMBER - 1])
kept = [f for i, f in enumerate(regular) if i in keep]
rest = [f for i, f in enumerate(regular) if i not in keep]
+ # A kept finding keeps the number it was found under, so the 49th found
+ # can sit here as CH-049 -- the id of the summary card below -- or on a
+ # reserved CH-050+. Renumber in the order found so CH-001..048 are the
+ # ordinary cards and 049 is only ever the summary.
+ for new_number, f in enumerate(kept, start=1):
+ f["id"] = f"{findings.PREFIX_BY_CHECK[CHECK_NAME]}-{new_number:03d}"
worst = min((f["severity"] for f in rest), key=order.get)
kept.append(findings.make_finding(
check=CHECK_NAME, severity=worst, device="unknown",
diff --git a/tests/test_change_against_policy.py b/tests/test_change_against_policy.py
index 303f9f5..68ac1df 100644
--- a/tests/test_change_against_policy.py
+++ b/tests/test_change_against_policy.py
@@ -351,6 +351,40 @@ def test_a_long_diff_never_folds_away_a_broken_policy_rule():
assert ids == sorted(ids), "what is kept stays in the order found"
+def test_a_capped_list_is_numbered_one_to_forty_eight_whatever_was_kept():
+ """The cap keeps the MOST IMPORTANT 48, not the first 48, and a kept
+ finding keeps the number it was given when it was found. With mixed
+ severities the 49th finding found is a high one, is kept, and is CH-049:
+ the same id as the card that summarises the rest. The ids this module
+ reserves (CH-049 and CH-050 to CH-054) must be free whatever was kept, so
+ a capped list is renumbered in the order found. (The test above this one
+ uses 60 findings of ONE severity, where the cap keeps the first 48 and
+ nothing is ever kept past 48, so it cannot see this.)"""
+ mk = change_impact.findings.make_finding
+ found = [mk(check="change_impact", severity="high" if i % 2 == 1 else "medium",
+ device="rtr-us5", summary=f"line {i}", detail="d",
+ source="rtr-us5:acl_in", status="found", number=i) for i in range(1, 61)]
+ kept = change_impact._cap(found)
+ ids = [f["id"] for f in kept]
+ assert len(ids) == len(set(ids)), f"duplicate ids: {sorted(i for i in set(ids) if ids.count(i) > 1)}"
+ assert ids == [f"CH-{n:03d}" for n in range(1, 49)] + ["CH-049"]
+ nums = [int(f["summary"].split()[1]) for f in kept[:-1]]
+ assert nums == sorted(nums), "what is kept stays in the order found"
+
+
+def test_a_capped_list_keeps_the_worst_severity_not_the_first_found():
+ """55 medium lines found first, then 5 high ones: the highs must survive
+ the cap. Keeping the first 48 would fold all five away."""
+ mk = change_impact.findings.make_finding
+ found = [mk(check="change_impact", severity="high" if i > 55 else "medium",
+ device="rtr-us5", summary=f"line {i}", detail="d",
+ source="rtr-us5:acl_in", status="found", number=i) for i in range(1, 61)]
+ kept = change_impact._cap(found)
+ kept_highs = [f for f in kept[:-1] if f["severity"] == "high"]
+ assert len(kept_highs) == 5, "every high finding is kept"
+ assert kept[-1]["severity"] == "medium", "the summary is as bad as the worst it hides"
+
+
def test_the_policy_is_cleared_even_when_the_stage_raises(monkeypatch):
"""A policy outliving the call that supplied it would answer the next
caller's question with someone else's rules."""_cap keeps the most important 48, but each kept finding kept the number it was found under, so with mixed severities the 49th found was kept as CH-049, the id of the summary card, and later ones could sit on CH-050 to CH-054. The kept list is renumbered in the order found. Arsh's two tests: mixed severities give CH-001..048 plus 049, and five highs found after 55 mediums all survive the cap (the 'severity ignored' mutant that used to survive the suite now fails).
|
@ARSH871-bot, fixed in Merge order, done as you advised: #463 and #428 are retargeted to |
ARSH871-bot
left a comment
There was a problem hiding this comment.
Re-reviewed 76cdf8c, your branch with main ee769b4 merged in. The duplicate id is fixed, and my three blockers from 3 November were already fixed. Approving.
What changed since I reviewed d4587ca: f00a3d6 (the _cap renumber) and the merge of main. In analysis/change_impact.py that is +7 lines, and the tests are +34, which are the two I suggested.
Checked
- The change tests pass (73). CI-style suite on this head, with Batfish and Ollama refused: 2323 passed, 58 skipped. GitHub's checks are green on 3.12 and 3.13.
- The renumber is guarded: with the loop removed,
test_a_capped_list_is_numbered_one_to_forty_eight_whatever_was_keptfails with the duplicateCH-049, and with it restored it passes. - The "severity ignored" mutant that survived last time is now caught too. I saw that on my own copy of the same two tests; I did not re-run it on this head.
- Not re-run today: the live suite and the 16-mutant run. Docker is off on my machine, and nothing in
change_impact.pyoutside_caphas changed sinced4587ca, where live was 2351 passed / 13 skipped and 15 of 16 mutants were caught (the 16th is the one above).
Merge order
- #463 and #428 are both on
mainalready. #463's branch contains this one's commits, so merge #406 first and #463's diff shrinks to its own change. - #405 conflicts with this branch in
analysis/change_impact.pyandtests/test_change_against_policy.py(tried today, merging #405's head into76cdf8c). Whichever lands second resolves it, and also needs the one-line fix totest_direction_field_is_set_regardless_of_severity, which unpacks three values fromstage().
Closes #403. Addresses #402 items 1, 3, 4, 5, 6 and 7 — every item except 2, which is #405.
Every item below was reproduced first, mostly with @ARSH871-bot's own repro inputs from the #389 post-merge review.
#403 — a deleted router was "No change detected"
analyse_change()now compares the two snapshots' device lists. Each device added or removed is afoundcard; a side whose devices can't be listed is anerror(CH-052), never "no change". Live, on #403's exact repro:#402
rtr-us5, five built-in rules →CH-051×5 → every proposalverified=FalseCH-051. Live repro withrtr-us5renamededge-fw1: no errorspolicy_compliancerule, silentlyaccess_control/routingrules it skipped; a policy with nopolicy_compliancerules says no rule of yours was checkedfound. A differentialsearchFiltersseparates same traffic (low, not caused by this change) from different traffic (the rule's own severity, directionchanged). "Unchanged" is scoped to the rule's filter_rule_breachedis tested against a fake session. Policy replaced byNone, first arm only, snapshot not switched,kindignored — all four now fail without BatfishCH-051could collide with a 51st filter findingCH-049summarises overflow,CH-050…054are this module's own error cards). CLI printsrenamed/assignedMerge order with #405
#405 restructures
_policy_change_findingstoo (_attach_directionaround everymake_finding). Whichever lands second takes the conflict, and if it's this one, that's me. If #405 lands first I'll rebase onto it and attachdirectionto the new card kinds here (altered→changed, device removed →closing, added →opening), assuming the open question on #405 about the extra key resolves that way.Mutation-tested
1907 passed, 13 skippedwith Batfish running. ruff clean on touched files.