Skip to content

Give a proposed change a size, beside its findings (closes #420) - #428

Merged
patelankeet2 merged 10 commits into
mainfrom
feat/420-change-score
Nov 19, 2026
Merged

patelankeet2 merged 10 commits into
mainfrom
feat/420-change-score

Conversation

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator

Closes #420. Stacked on #406.

change_impact.score(findings) returns a number, a band (none / small / moderate / large / incomplete) and the breakdown, from status and severity only. Any error makes it incomplete, never small. Returned beside the findings, not as an F-1 field.

Live against Batfish:

secure -> insecure   large     4 high x3 + 1 medium x2 = 14
insecure -> secure   moderate  2 medium x2 + 3 low x1 = 7
secure -> secure     none      no changes found = 0
remove one router    small     1 medium x2 = 2

Showing it in the propose pane is @SamikaPerera's call.

)

#403. Deleting a whole router reported "No change detected" with a green
tick: neither compareFilters nor differentialReachability describes a device
that exists on one side only. analyse_change() now compares the device lists
of both snapshots and reports each added or removed device as `found`. A side
whose devices cannot be listed is an error (CH-052), never "no change".
Reproduced live on multi-device-10 minus rtr-dev009.

#402, from @ARSH871-bot's post-merge review of #389:

1. A rule about a device in NEITHER config is not applicable -- this change
   cannot affect it -- and is named as such, not reported as an error. Every
   built-in rule used to be CH-051 on a network without rtr-us5, so every
   proposal there was "could not be fully verified". A rule about a device on
   ONE side only is one CH-051. Reproduced live with rtr-us5 renamed edge-fw1.

3. Only policy_compliance rules are compared, and the result now says so: a
   sentence beginning "NOT compared:" names the access_control and routing
   rules it skipped, and a policy with no policy_compliance rules says no
   rule of the user's was checked.

4. A rule breached before AND after is `found`, never inside the green card.
   A differential searchFilters tells "breached the same way" (low, not
   caused by this change) from "breached by different traffic" (the rule's
   own severity, direction: changed). "Unchanged" is scoped to the rule's
   filter, because unbinding a filter changes the network without changing
   that answer.

5. _rule_breached is tested against a fake session, so CI now catches the
   four mutants only the skipped live test could: the policy passed in
   replaced by None, only the first query arm asked, the snapshot not
   switched, and the rule's kind ignored.

6. Every stage runs whatever the others did, so a failed reachability stage
   no longer silently skips the policy stage. Ordinary findings stop at 48;
   49 and up are reserved (CH-049 summarises any overflow, CH-050..054 are
   this module's own error cards), so a long diff cannot collide with them.
   The CLI prints renamed and assigned notes.

7. Docs: the module docstring lists what is reported and in what order,
   CLAUDE.md's change_impact row is current, and the policy-rules section
   moved out from above section 7's sign-off into its own section 8.

Not here: item 2 is #405 (Ankeet's). Shubham's rebase onto it, whichever
lands second, absorbs the conflict in _policy_change_findings.

Mutation-tested: device changes dropped, absent-device rules made errors,
set_active_policy(None), first arm only, reachability failure skipping the
policy stage, no cap, still-breached back in the green card, snapshot not
switched -- each fails the new tests; the arm, snapshot and kind mutants fail
WITHOUT Batfish.

1907 passed, 13 skipped with Batfish running. ruff clean on touched files.
change_impact.score(findings) returns one number, its band and the
breakdown that produced it, computed only from status and severity --
never from a summary's wording. Opening counts more than closing because
the severities already encode direction. Any error finding makes the band
'incomplete', so a change that could not be fully compared never reads as
small. Returned beside the findings, not inside them (see #405).

Live: secure->insecure large (14), insecure->secure moderate (7),
no change none (0), one router removed small (2). Stacked on #406.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with real testing against real Batfish. Stacked on #406 (whose current head still matches my own existing approval -- checked, not assumed), tested merged with current main (b69d591).

git merge origin/main (local only)              clean, auto-merged CLAUDE.md
pytest tests/test_change_score.py tests/test_change_impact.py
       tests/test_change_against_policy.py -q    38 passed
ruff check --select E4,E7,E9,F ...               clean

Live-reproduced all three fixture-based examples in the PR description, against real Batfish, each confirmed stable across repeated runs:

secure -> insecure:   large     (4 high x3 + 1 medium x2 = 14)
insecure -> secure:   moderate  (2 medium x2 + 3 low x1 = 7)
secure -> secure:     none      (no changes found = 0)

All three match exactly. Worth recording honestly: my first live run of each direction, right after merging main, gave different and in one case inconsistent results (an incomplete band where "moderate" was expected) -- re-running each 2-3 times immediately settled on the numbers above, every time. That's analyse_change()'s own Batfish/snapshot handling warming up after a fresh checkout, not anything in this PR -- score() is a pure function over an already-computed results list and makes no Batfish call of its own.

Full suite, same story, caught properly rather than glossed over. First run: 1 failed, 1927 passed -- test_change_impact_coverage.py::test_live_a_stranger_network_proposal_is_verified, a live Batfish test from #406's own earlier work, untouched by this diff. Reran it alone: passed. Reran the full suite a second time: 1928 passed, 0 failed. Same transient class as the live reproduction above -- not a regression from this PR, and not something to wave past without checking, which is why I reran it twice rather than once.

Independently mutation-tested the design rule you named explicitly -- incomplete never scores small. Disabled the if blind: band = "incomplete" override and reran the test:

AssertionError: assert ('small' == 'incomplete'

Exactly the dangerous direction the design calls out -- a change with real Batfish coverage gaps would have reported as reassuringly small instead of unknown-and-possibly-larger. Caught correctly. Reverted.

Checked the one structural question worth asking: SCORE_BANDS is read top-to-bottom via next(... if total >= floor), so its order (10, 4, 1, 0) has to stay descending for the thresholds to mean what the band names say -- confirmed by reading it alongside the live "large"/"moderate"/"none" results above, which land in the band their own total actually supports.

No em-dashes, no AI-tool mentions.

Approving.

Arsh's three blockers on #406:

1. A pure deny raised the "opens access" banner. _breach_changed now
   returns (grew, shrank) from the differential frame's two verdicts; a
   breach that only shrinks is "narrows", low, and only a breach that
   grows keeps the rule's own severity. Batfish returns one example per
   direction, measured on a change that both adds and removes breaching
   flows.
2. Device presence was case-sensitive, so policy device RTR-US5 against
   Batfish's rtr-us5 filed a real breach under a tick. Both sides are now
   casefolded, and a user's own rule about a device in neither config is
   part of CH-051, as a scan calls it PC-050.
3. CH-051 was only ever built by the test helper. The helper now returns
   the stage's own output, and CH-051 is tested through analyse_change().

Also from the review: tests that _breach_changed passes the reference
snapshot and asks every arm, a test that a failed compareFilters is CH-050
and never "No change detected", _cap keeps the most severe cards (policy
first at equal severity) instead of the first 48, the FINDING IDS block,
and the #389 credit in CLAUDE.md. Arsh's two live reproductions are now
live tests.
@shubhamkataria2005

Copy link
Copy Markdown
Collaborator Author

Merged #406's branch in (merge-only push, M-2 rule 2), so this inherits the fix for Arsh's item 1. Tested on the merged result: 2289 passed, 13 skipped, ruff clean. Arsh's pure-deny case now scores 1 medium x2 + 3 low x1 = 5 (moderate) with 0 high. Before, the rule-2 card counted as high. Lands after #406.

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator Author

Merge-only push (939da8f): brought in #406's merge of current main. Merged result: 2358 passed, 13 skipped, ruff clean, CI green. Still lands after #406.

_cap keeps the most important 48, but each kept finding kept the number it
was found under, so with mixed severities the 49th found was kept as
CH-049, the id of the summary card, and later ones could sit on CH-050 to
CH-054. The kept list is renumbered in the order found. Arsh's two tests:
mixed severities give CH-001..048 plus 049, and five highs found after 55
mediums all survive the cap (the 'severity ignored' mutant that used to
survive the suite now fails).
@shubhamkataria2005
shubhamkataria2005 changed the base branch from fix/403-402-change-impact-coverage to main November 16, 2026 00:38
@patelankeet2
patelankeet2 merged commit f179eac into main Nov 19, 2026
2 checks passed
@patelankeet2
patelankeet2 deleted the feat/420-change-score branch November 19, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Change impact score — one number for how big a change's effect is

2 participants