Skip to content

Change impact reports what it did not look at (closes #403, most of #402) - #406

Merged
patelankeet2 merged 6 commits into
mainfrom
fix/403-402-change-impact-coverage
Nov 17, 2026
Merged

patelankeet2 merged 6 commits into
mainfrom
fix/403-402-change-impact-coverage

Conversation

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator

Closes #403. Addresses #402 items 1, 3, 4, 5, 6 and 7 — every item except 2, which is #405.

Every item below was reproduced first, mostly with @ARSH871-bot's own repro inputs from the #389 post-merge review.

#403 — a deleted router was "No change detected"

analyse_change() now compares the two snapshots' device lists. Each device added or removed is a found card; a side whose devices can't be listed is an error (CH-052), never "no change". Live, on #403's exact repro:

before = multi-device-10, after = the same minus rtr-dev009
before this PR:  CH-000 none  "No change detected"
after:           CH-00n found "Device rtr-dev009 is removed by this change"

#402

item what was wrong now
1 on a network without rtr-us5, five built-in rules → CH-051 ×5 → every proposal verified=False a rule about a device in neither config is not applicable, named in the result, not an error. Device on one side only → a single CH-051. Live repro with rtr-us5 renamed edge-fw1: no errors
3 "every policy rule" was every policy_compliance rule, silently a sentence starting "NOT compared:" names the access_control/routing rules it skipped; a policy with no policy_compliance rules says no rule of yours was checked
4 "still breached" sat in the green card; "neither causes nor fixes" overclaimed breached before and after is found. A differential searchFilters separates same traffic (low, not caused by this change) from different traffic (the rule's own severity, direction changed). "Unchanged" is scoped to the rule's filter
5 four mutants only the live test caught, and it skips on CI _rule_breached is tested against a fake session. Policy replaced by None, first arm only, snapshot not switched, kind ignored — all four now fail without Batfish
6 a failed reachability stage silently skipped the policy stage; CH-051 could collide with a 51st filter finding every stage runs whatever the others did. Ordinary findings stop at 48; 49+ are reserved (CH-049 summarises overflow, CH-050…054 are this module's own error cards). CLI prints renamed/assigned
7 stale docs module docstring lists what's reported and in what order; CLAUDE.md row current; the policy-rules section moved out from above §7's sign-off ticks into its own §8

Merge order with #405

#405 restructures _policy_change_findings too (_attach_direction around every make_finding). Whichever lands second takes the conflict, and if it's this one, that's me. If #405 lands first I'll rebase onto it and attach direction to the new card kinds here (altered → changed, device removed → closing, added → opening), assuming the open question on #405 about the extra key resolves that way.

Mutation-tested

device add/remove not reported              3 failed
absent-device rules made errors again       3 failed
set_active_policy(None) instead of (policy) 1 failed
only the first query arm asked              1 failed   (also without Batfish)
reachability failure skips policy stage     1 failed
no cap, ids can collide                     1 failed
still-breached back in the green card       2 failed
snapshot not switched                       2 failed   (also without Batfish)
kind ignored                                1 failed   (without Batfish)

1907 passed, 13 skipped with Batfish running. ruff clean on touched files.

)

#403. Deleting a whole router reported "No change detected" with a green
tick: neither compareFilters nor differentialReachability describes a device
that exists on one side only. analyse_change() now compares the device lists
of both snapshots and reports each added or removed device as `found`. A side
whose devices cannot be listed is an error (CH-052), never "no change".
Reproduced live on multi-device-10 minus rtr-dev009.

#402, from @ARSH871-bot's post-merge review of #389:

1. A rule about a device in NEITHER config is not applicable -- this change
   cannot affect it -- and is named as such, not reported as an error. Every
   built-in rule used to be CH-051 on a network without rtr-us5, so every
   proposal there was "could not be fully verified". A rule about a device on
   ONE side only is one CH-051. Reproduced live with rtr-us5 renamed edge-fw1.

3. Only policy_compliance rules are compared, and the result now says so: a
   sentence beginning "NOT compared:" names the access_control and routing
   rules it skipped, and a policy with no policy_compliance rules says no
   rule of the user's was checked.

4. A rule breached before AND after is `found`, never inside the green card.
   A differential searchFilters tells "breached the same way" (low, not
   caused by this change) from "breached by different traffic" (the rule's
   own severity, direction: changed). "Unchanged" is scoped to the rule's
   filter, because unbinding a filter changes the network without changing
   that answer.

5. _rule_breached is tested against a fake session, so CI now catches the
   four mutants only the skipped live test could: the policy passed in
   replaced by None, only the first query arm asked, the snapshot not
   switched, and the rule's kind ignored.

6. Every stage runs whatever the others did, so a failed reachability stage
   no longer silently skips the policy stage. Ordinary findings stop at 48;
   49 and up are reserved (CH-049 summarises any overflow, CH-050..054 are
   this module's own error cards), so a long diff cannot collide with them.
   The CLI prints renamed and assigned notes.

7. Docs: the module docstring lists what is reported and in what order,
   CLAUDE.md's change_impact row is current, and the policy-rules section
   moved out from above section 7's sign-off into its own section 8.

Not here: item 2 is #405 (Ankeet's). Shubham's rebase onto it, whichever
lands second, absorbs the conflict in _policy_change_findings.

Mutation-tested: device changes dropped, absent-device rules made errors,
set_active_policy(None), first arm only, reachability failure skipping the
policy stage, no cap, still-breached back in the green card, snapshot not
switched -- each fails the new tests; the arm, snapshot and kind mutants fail
WITHOUT Batfish.

1907 passed, 13 skipped with Batfish running. ruff clean on touched files.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE. Tested, not just read.

Independently reproduced both headline fixes, live

#403's own repro (multi-device-10 minus rtr-dev009):
  CH-001 found medium rtr-dev009 | Device rtr-dev009 is removed by this change
  "No change detected"? absent, confirmed

#402 item 1's repro (rtr-us5 renamed edge-fw1, both sides):
  CH-000 none low | No change detected between the two configurations
  any status=="error"? False, confirmed

Both match the PR body's own tables exactly.

Read the full diff, not just the summary

Traced the F-4-sensitive part carefully: the old code returned immediately on a filter- or reachability-stage exception, abandoning every later stage. Now each stage appends a reserved-id error card (blind()) and continues, so a failed reachability check can no longer silently skip the policy stage -- and confirmed results = _cap(regular) + reserved means the "no change detected" sentinel is only reachable when BOTH are empty, so an error card can never collapse into a false all-clear. That's the right direction for F-4 and I checked it holds rather than assumed it from the description.

Mutation-tested one of your own claims rather than only trusting the table

Reverted _rule_breached to check only rule["queries"][0] instead of looping every arm -- test_rule_breached_asks_every_arm_not_just_the_first failed for exactly the right reason, nothing else did. Reverted.

Verified independently

python -m pytest tests/ -q                                             1907 passed, 13 skipped (matches)
python -m ruff check analysis/change_impact.py tests/test_change_impact_coverage.py
                     tests/test_change_against_policy.py tests/test_change_impact.py   all clean
git log -1 --format=%ad --date=short dc93ccb                           2026-10-14 -- matches the
                                                                        CLAUDE.md claim "#323 merged 14 October"

On the #405 merge-order note

Agreed with your plan -- whichever lands second absorbs the conflict. I've also replied on #405's own thread to your direction-ratification question (you're right that my comparison to explanation/explanation_source didn't hold; it's a producer-side key, not a response-layer one, so it's the same class of question as #390's). Not blocking this PR either way, just flagging that I've seen it from both sides so nobody has to repeat the finding a third time.

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator Author

Heads-up for merge order, measured today on current main (b69d591):

I'm not pushing it now: that would bring #387's commits into this branch before #387 lands, and void @patelankeet2's approval. Once #387 merges, I'll merge main into this branch with exactly that resolution. That's a merge-only push, so under M-2 rule 2 I'll post the merged test result here before anyone merges.

@patelankeet2

Copy link
Copy Markdown
Collaborator

@shubhamkataria2005 picked this up since you hadn't gotten to it yet and main had moved a lot since your 31 October note -- but this is your branch and your call, so posting the verification rather than merging it myself.

Did exactly what you described: merged main in and resolved the conflict with notes before warnings, matching pipeline.py's own order.

for note in list(policy.renamed) + list(policy.assigned):
    print(f"note: {note}")
for note in policy.undecidable:
    print(f"warning: {note}")

Live-confirmed the actual order rather than just reading the diff -- built a policy with both an unnumbered rule (triggers a note:) and an unmodelled query field (triggers a warning:) and ran the real CLI:

note: policy_compliance: no entry set 'number', so they were numbered 1-2 in file order ...
warning: rule 1 ('...') and rule 2 ('...') on rtr-us5 filter acl_in could not be compared: ...

Notes first, warnings second, exactly as planned.

pytest tests/test_change_impact.py tests/test_change_against_policy.py -q   33 passed
pytest tests/ -q  (merged onto current main)                                 2262 passed, 13 skipped
ruff check --select E4,E7,E9,F ...                                           clean

Pushed as 39ecfcf. If this matches what you had planned, go ahead and merge -- otherwise let me know what to adjust.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update, 3 November. Re-checked on 39ecfcf, your branch after Ankeet merged main into it. That merge only resolves the CLI conflict (notes before warnings, as you planned) and is fine. All three blockers below are still in the code at this head: _breach_changed (change_impact.py:421-447) with :564-570 for B1, :539 and :547 with no case-folding for B2, and the hand-built CH-051 at tests/test_change_against_policy.py:57 for B3. #428's score() sits on this branch and inherits B1: a tightening counts as high.

My first pass, on 1503726 (21 October): Live, 1907 passed / 13 skipped, and CI-condition 1869 / 51 both match. #403 is fixed: multi-device-10 minus rtr-dev009 gives CH-001 found "Device rtr-dev009 is removed", where main said "No change detected". #402.1 is fixed: edge-fw1 now proposes verified=True, where main gave CH-051 ×5. Your four item-5 mutants all fail without Batfish. The 49+ reservation holds: 60 removed lines give 48 + CH-049, with no duplicate ids. A raising reachability stage no longer hides the policy stage. Three things need fixing before merge:

  1. A pure deny now raises the "opens access" banner. propose_change("block 10.10.10.5 to 10.20.0.5 on tcp/22 on rtr-us5", rtr-us5-insecure): main gives warning=False, "narrows access"; this branch gives warning=True, "newly opens access that was previously blocked". Rule 2 is breached on both sides, so _breach_changed (change_impact.py:421) returns True and :570 rates it high. The differential frame already tells you the direction: on this case Snapshot_Action=DENY, Reference_Action=PERMIT. Please use it. A shrinking breach is a tightening, not "changed" at high. With #405 merged it still warns, just with different wording.
  2. Device presence is case-sensitive, and it fails as a tick. Config hostname RTR-US5, policy device: RTR-US5, change removes DNS. Main: CH-002 found "This change BREAKS policy rule 1". Here: ✓ "devices in neither configuration, which this change cannot affect: 1 (RTR-US5)". :539 compares the rule's node as written against snapshot.device_names(), which holds Batfish's lower-cased names. searchFilters matches case-insensitively, which is why main found the breach. A scan of the same file says PC-050, amber. Please casefold the comparison. Also, a user's own absent rule shouldn't be a ✓ when the scan calls it "could not check".
  3. CH-051 has no test. The stage() helper builds its own CH-051 (test_change_against_policy.py:57), so test_the_ids_this_stage_emits_are_unique checks an id the test made up. The test that asserted a real error card on main now checks out["not_compared"] (:138). Deleting the CH-051 block at :758 passes the whole suite, with and without Batfish. So does dropping one_side from it.

Also, not blocking:

  • _breach_changed is never exercised: dropping reference_snapshot, or asking only the first arm, both survive.
  • The filter-failure path is untested. Silencing it turns a failed compareFilters into a green "No change detected".
  • The "NOT compared:" sentence sits in a ✓ card titled "Policy rules this change did not move", even when the change breaks that rule. AC-005 (#261) is the precedent for making that an error.
  • _cap folds the policy cards first. 60 lines plus DNS removed hides "BREAKS rule 4" inside "15 further change(s)".
  • CLAUDE.md:716 credits "#323 (merged 14 October)". #323 is the issue; the PR was #389.
  • The FINDING IDS block at :85 wasn't updated alongside :110.
  • Open question 2 in the shapes doc is untouched, although #402 item 7 named it.

Merge order: with #405, once the textual conflict in _policy_change_findings is resolved, #405's new test_direction_field_is_set_regardless_of_severity still unpacks three values from stage(), giving 4 × ValueError. Whoever lands second needs that one line. With #387 there is one conflict in change_impact.main() (both add a note loop; keep both). #390 merges clean.

@patelankeet2, you approved this on 20 October. Items 1 and 2 reproduce through the public entry points named above, so please re-check them before this lands.

Arsh's three blockers on #406:

1. A pure deny raised the "opens access" banner. _breach_changed now
   returns (grew, shrank) from the differential frame's two verdicts; a
   breach that only shrinks is "narrows", low, and only a breach that
   grows keeps the rule's own severity. Batfish returns one example per
   direction, measured on a change that both adds and removes breaching
   flows.
2. Device presence was case-sensitive, so policy device RTR-US5 against
   Batfish's rtr-us5 filed a real breach under a tick. Both sides are now
   casefolded, and a user's own rule about a device in neither config is
   part of CH-051, as a scan calls it PC-050.
3. CH-051 was only ever built by the test helper. The helper now returns
   the stage's own output, and CH-051 is tested through analyse_change().

Also from the review: tests that _breach_changed passes the reference
snapshot and asks every arm, a test that a failed compareFilters is CH-050
and never "No change detected", _cap keeps the most severe cards (policy
first at equal severity) instead of the first 48, the FINDING IDS block,
and the #389 credit in CLAUDE.md. Arsh's two live reproductions are now
live tests.
@shubhamkataria2005

Copy link
Copy Markdown
Collaborator Author

@ARSH871-bot, all three blockers are fixed in 641df5a. I reproduced each one through your entry points first. @patelankeet2, thanks for the merge in 39ecfcf; the conflict resolution is exactly what I'd planned.

your finding now
1 pure deny raises "opens access" _breach_changed returns (grew, shrank) from Snapshot_Action/Reference_Action, as you suggested. I measured Batfish first: it returns one example per direction, so a change that both adds and removes breaching flows gives two rows. Grows → "widens", rule's severity. Shrinks → "narrows", low, still breached. Both → "alters", direction changed. Your case: warning=False, "narrows access", rule 2 "narrows how … is breached" at low, no high anywhere
2 RTR-US5 breach filed under a ✓ Casefolded on both sides. Your config now gives CH-002 found high "This change BREAKS policy rule 1". And yes, a user's own rule about a device in neither config is now part of CH-051, like the scan's PC-050. Our built-in example rules stay "not applicable" (#402 item 1)
3 CH-051 built by the test stage() returns the stage's own output only. CH-051 is tested through analyse_change() with both not_compared and one_side. Deleting the block or dropping one_side now fails

Your non-blocking list:

  • _breach_changed exercised: tests on the real call. Dropping reference_snapshot fails 7, asking only the first arm fails 2.
  • Filter-failure path: a raising _filter_change_findings is asserted to be CH-050, never "No change detected". Silencing it fails.
  • _cap: keeps the worst severity first, a policy card ahead of a line card at equal severity, still in the order found. Your 60-lines-plus-DNS case keeps "BREAKS rule 4". Reverting to first-48 fails.
  • CLAUDE.md: now "Check a proposed change against every policy rule (closes #323) #389 (closing US-31 — Check a proposed change against my whole policy #323, merged 14 October)". FINDING IDS block: CH-049 and CH-051+ added.
  • Not done, on purpose: moving the "NOT compared:" sentence to an error card. I agree it's AC-005's shape, but with the propose pane's current verified rule, every proposal for a policy that has an access_control or routing section would become "could not be fully verified". That's the Change impact × policy: follow-ups from #389's post-merge review #402 item 1 complaint again, from the other side. It's a propose-pane decision, so I'd rather settle it with @SamikaPerera on its own issue than slip it in here. Open question 2 on the shapes doc stays open for the same reason: it needs all four.

Mutation: 10 mutants across the three fixes and the _cap change, 10 of 10 killed. One survived at first, casefolding only the device set, because Batfish's names are already lower case. A test with capitals on the device side now kills it.

Your two reproductions are live tests now: test_live_a_pure_deny_narrows_a_breach_and_raises_no_banner and test_live_a_device_named_in_capitals_still_reports_the_breach.

2282 passed, 13 skipped with Batfish running, on current main (db59b00). ruff clean on CI's selection. docs/policy-rules.md has the widens/narrows/alters rows and the device-name rule.

For #405 when it lands second: stage() now takes breach=(grew, shrank) instead of altered= and still returns four values. So test_direction_field_is_set_regardless_of_severity needs that one line, as you said.

Under M-2 rule 1 your review and Ankeet's approval are void now, so this needs fresh looks.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked both items you specifically asked me to, through the public entry points, not just the unit tests -- my 20 October approval was against the pre-review-fix state, so this supersedes it.

Item 1, live: propose_change("block 10.10.10.5 to 10.20.0.5 on tcp/22 on rtr-us5", rtr-us5-insecure) with the built-in policy active (your exact repro). Result: warning: False, "narrows access; it does not appear to newly open anything" -- matches main, not the "newly opens access" banner the pre-fix branch gave.

Item 2, live, against real Batfish: ran test_live_a_device_named_in_capitals_still_reports_the_breach directly -- hostname RTR-US5, policy device: RTR-US5, DNS removed. Passes: the breach is reported as BREAKS policy rule 1, high, not filed as not-compared.

Item 3: confirmed the test helper no longer fabricates CH-051 -- stage() returns only the stage's own findings now, and test_rules_not_compared_become_a_real_ch051_error builds the CH-051 card through the real analyse_change(), stubbed one level up. Real code, not a hand-built dict pretending to be its output.

Mutation-tested _breach_changed's new two-value return by collapsing it back to the old single bool: 8 tests failed, including the live Batfish test, with exactly item 1's bug reproduced in the failure message ("newly opens access" where it shouldn't). Restored.

Bonus, not required but done anyway: the _cap() reordering (policy cards kept first, not whichever 48 happened to run first) and a real test for the untested filter-failure-silently-becomes-clean path are both in this commit too, covering two of your explicitly "not blocking" items.

Full suite: 2282 passed, 13 skipped, 0 failed. ruff clean.

Approving, with my 20 October approval now correctly superseded by this one rather than left stale alongside it.

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator Author

Merge-only push (d4587ca): main had moved (#434, #446, #442) and conflicted in CLAUDE.md. Each side had changed a different row: main the PF Sense row, this branch the change_impact row. Both are kept as each side wrote them. M-2 rule 2, the merged result: 2351 passed, 13 skipped with Batfish running, ruff clean, CI green on both Pythons. No code change on this branch since 641df5a.

patelankeet2 added a commit that referenced this pull request Nov 10, 2026
…rds (#405 review, Arsh)

The structural direction field already uses opening/closing/changed and
is already relied on by opens_access -- not what this proposes changing.
What is inconsistent is the human-readable Direction: X phrase in
evidence.detail: change_impact.py:488 still says tightening for the case
the field calls closing, and #406 separately invented changed for its
own case. Proposes one vocabulary for both. Not changing the wording
itself here -- that is exactly what this table gates, same as A-4.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review on d4587ca One new blocker, small, with a tested fix. All three of my 3 November blockers are fixed.

My three blockers, re-checked live against Batfish on this head

  1. Pure deny: propose_change("block 10.10.10.5 to 10.20.0.5 on tcp/22 on rtr-us5", rtr-us5-insecure) is warning=False, "narrows access", rated low. Fixed. Direction is read from the differential frame; grew, shrank and both-ways each come out right.
  2. Capitals: hostname RTR-US5 + device: RTR-US5, DNS removed, gives CH-002 BREAKS again, and a rule about a device in neither config is no longer a tick. Fixed.
  3. CH-051 is a real card now. Deleting the block, dropping one_side or dropping not_compared each fail test_rules_not_compared_become_a_real_ch051_error. Fixed.

Mutation run on this head: 16 mutants, 15 caught without Batfish (all of the above plus _breach_changed, the filter-stage failure, casefolding, and the cap ordering). CI-condition 2306 passed / 58 skipped; live 2351 passed / 13 skipped. The minor items (CLAUDE.md credit to #389, the FINDING IDS block) are fixed.

New: _cap can produce a duplicate id (blocking, but a few lines)
_cap now keeps the most important 48, which is right, but each kept finding keeps the number it was found under. With mixed severities the 49th finding found is kept, so it is CH-049, the same id as the "N further change(s)" card. Kept findings numbered 50-54 also sit on the ids this module reserves for CH-050..CH-054.

Repro (60 findings, odd ones high, even ones medium): duplicate ids: ['CH-049'], and 49 ids but 48 distinct. This is a different bug from the old one: on main the 60-line case had no cap and so collided at CH-051; the cap reduced that but does not remove it. The duplicate-id guard in the pipeline will not say a word about it until it fires on someone's screen.

Your existing test_a_long_diff_never_folds_away_a_broken_policy_rule uses 60 findings of one severity, so the cap keeps the first 48 and nothing past 48 is ever kept; it cannot see this. It also did not catch my 16th mutant, "cap: severity ignored", which survived the whole suite.

Suggested fix, tested (patch below): after kept is built in _cap, renumber it in the order found, so CH-001..048 are the ordinary cards and 049 is only ever the summary. Plus two tests:

  • test_a_capped_list_is_numbered_one_to_forty_eight_whatever_was_kept: mixed severities; fails on d4587ca with the duplicate; fails again if the renumber is removed.
  • test_a_capped_list_keeps_the_worst_severity_not_the_first_found: 55 medium then 5 high; fails if severity is ignored.

On the patched tree: full suite 2353 passed / 13 skipped live, change tests 73 passed. Renumbering does change a kept finding's id relative to when it was found, but nothing outside _cap uses that id (the list is returned as a whole, in order).

Merge order, unchanged from 3 Nov: #463 and #428 are stacked on this; retarget them to main before this merges. #405 will conflict in _policy_change_findings and its test_direction_field_is_set_regardless_of_severity still unpacks three values from stage().

Verdict: request changes for the one item above; otherwise this is ready.

Patch
diff --git a/analysis/change_impact.py b/analysis/change_impact.py
index 875a3d6..e6e3a22 100644
--- a/analysis/change_impact.py
+++ b/analysis/change_impact.py
@@ -667,6 +667,12 @@ def _cap(regular: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
     keep = set(sorted(range(len(regular)), key=importance)[: _OVERFLOW_NUMBER - 1])
     kept = [f for i, f in enumerate(regular) if i in keep]
     rest = [f for i, f in enumerate(regular) if i not in keep]
+    # A kept finding keeps the number it was found under, so the 49th found
+    # can sit here as CH-049 -- the id of the summary card below -- or on a
+    # reserved CH-050+. Renumber in the order found so CH-001..048 are the
+    # ordinary cards and 049 is only ever the summary.
+    for new_number, f in enumerate(kept, start=1):
+        f["id"] = f"{findings.PREFIX_BY_CHECK[CHECK_NAME]}-{new_number:03d}"
     worst = min((f["severity"] for f in rest), key=order.get)
     kept.append(findings.make_finding(
         check=CHECK_NAME, severity=worst, device="unknown",
diff --git a/tests/test_change_against_policy.py b/tests/test_change_against_policy.py
index 303f9f5..68ac1df 100644
--- a/tests/test_change_against_policy.py
+++ b/tests/test_change_against_policy.py
@@ -351,6 +351,40 @@ def test_a_long_diff_never_folds_away_a_broken_policy_rule():
     assert ids == sorted(ids), "what is kept stays in the order found"
 
 
+def test_a_capped_list_is_numbered_one_to_forty_eight_whatever_was_kept():
+    """The cap keeps the MOST IMPORTANT 48, not the first 48, and a kept
+    finding keeps the number it was given when it was found. With mixed
+    severities the 49th finding found is a high one, is kept, and is CH-049:
+    the same id as the card that summarises the rest. The ids this module
+    reserves (CH-049 and CH-050 to CH-054) must be free whatever was kept, so
+    a capped list is renumbered in the order found. (The test above this one
+    uses 60 findings of ONE severity, where the cap keeps the first 48 and
+    nothing is ever kept past 48, so it cannot see this.)"""
+    mk = change_impact.findings.make_finding
+    found = [mk(check="change_impact", severity="high" if i % 2 == 1 else "medium",
+                device="rtr-us5", summary=f"line {i}", detail="d",
+                source="rtr-us5:acl_in", status="found", number=i) for i in range(1, 61)]
+    kept = change_impact._cap(found)
+    ids = [f["id"] for f in kept]
+    assert len(ids) == len(set(ids)), f"duplicate ids: {sorted(i for i in set(ids) if ids.count(i) > 1)}"
+    assert ids == [f"CH-{n:03d}" for n in range(1, 49)] + ["CH-049"]
+    nums = [int(f["summary"].split()[1]) for f in kept[:-1]]
+    assert nums == sorted(nums), "what is kept stays in the order found"
+
+
+def test_a_capped_list_keeps_the_worst_severity_not_the_first_found():
+    """55 medium lines found first, then 5 high ones: the highs must survive
+    the cap. Keeping the first 48 would fold all five away."""
+    mk = change_impact.findings.make_finding
+    found = [mk(check="change_impact", severity="high" if i > 55 else "medium",
+                device="rtr-us5", summary=f"line {i}", detail="d",
+                source="rtr-us5:acl_in", status="found", number=i) for i in range(1, 61)]
+    kept = change_impact._cap(found)
+    kept_highs = [f for f in kept[:-1] if f["severity"] == "high"]
+    assert len(kept_highs) == 5, "every high finding is kept"
+    assert kept[-1]["severity"] == "medium", "the summary is as bad as the worst it hides"
+
+
 def test_the_policy_is_cleared_even_when_the_stage_raises(monkeypatch):
     """A policy outliving the call that supplied it would answer the next
     caller's question with someone else's rules."""

_cap keeps the most important 48, but each kept finding kept the number it
was found under, so with mixed severities the 49th found was kept as
CH-049, the id of the summary card, and later ones could sit on CH-050 to
CH-054. The kept list is renumbered in the order found. Arsh's two tests:
mixed severities give CH-001..048 plus 049, and five highs found after 55
mediums all survive the cap (the 'severity ignored' mutant that used to
survive the suite now fails).
@shubhamkataria2005

shubhamkataria2005 commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator Author

@ARSH871-bot, fixed in f00a3d6 as you suggested (the head is now 76cdf8c, after a clean merge of current main, which brought in #456). _cap renumbers the kept list in the order found, so CH-001..048 are the ordinary cards and CH-049 is only ever the summary. Both your tests are in, word for word. Mutants: removing the renumber fails 1, and your 16th, "cap: severity ignored", which used to survive the suite, now fails 1. Merged with main: 2368 passed, 13 skipped, ruff clean, CI green.

Merge order, done as you advised: #463 and #428 are retargeted to main now, before this lands, and both are brought up to date with this head (CI green on each). On #405: whichever of it and this lands second takes the _policy_change_findings conflict and the stage() unpack, and if it's this one, that's me.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 76cdf8c, your branch with main ee769b4 merged in. The duplicate id is fixed, and my three blockers from 3 November were already fixed. Approving.

What changed since I reviewed d4587ca: f00a3d6 (the _cap renumber) and the merge of main. In analysis/change_impact.py that is +7 lines, and the tests are +34, which are the two I suggested.

Checked

  • The change tests pass (73). CI-style suite on this head, with Batfish and Ollama refused: 2323 passed, 58 skipped. GitHub's checks are green on 3.12 and 3.13.
  • The renumber is guarded: with the loop removed, test_a_capped_list_is_numbered_one_to_forty_eight_whatever_was_kept fails with the duplicate CH-049, and with it restored it passes.
  • The "severity ignored" mutant that survived last time is now caught too. I saw that on my own copy of the same two tests; I did not re-run it on this head.
  • Not re-run today: the live suite and the 16-mutant run. Docker is off on my machine, and nothing in change_impact.py outside _cap has changed since d4587ca, where live was 2351 passed / 13 skipped and 15 of 16 mutants were caught (the 16th is the one above).

Merge order

  • #463 and #428 are both on main already. #463's branch contains this one's commits, so merge #406 first and #463's diff shrinks to its own change.
  • #405 conflicts with this branch in analysis/change_impact.py and tests/test_change_against_policy.py (tried today, merging #405's head into 76cdf8c). Whichever lands second resolves it, and also needs the one-line fix to test_direction_field_is_set_regardless_of_severity, which unpacks three values from stage().

@patelankeet2
patelankeet2 merged commit 1e2a6c1 into main Nov 17, 2026
2 checks passed
@patelankeet2
patelankeet2 deleted the fix/403-402-change-impact-coverage branch November 17, 2026 22:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

change_impact says "No change detected" when a whole device is removed

3 participants