Repository navigation
Give a proposed change a size, beside its findings (closes #420) - #428
Conversation
) #403. Deleting a whole router reported "No change detected" with a green tick: neither compareFilters nor differentialReachability describes a device that exists on one side only. analyse_change() now compares the device lists of both snapshots and reports each added or removed device as `found`. A side whose devices cannot be listed is an error (CH-052), never "no change". Reproduced live on multi-device-10 minus rtr-dev009. #402, from @ARSH871-bot's post-merge review of #389: 1. A rule about a device in NEITHER config is not applicable -- this change cannot affect it -- and is named as such, not reported as an error. Every built-in rule used to be CH-051 on a network without rtr-us5, so every proposal there was "could not be fully verified". A rule about a device on ONE side only is one CH-051. Reproduced live with rtr-us5 renamed edge-fw1. 3. Only policy_compliance rules are compared, and the result now says so: a sentence beginning "NOT compared:" names the access_control and routing rules it skipped, and a policy with no policy_compliance rules says no rule of the user's was checked. 4. A rule breached before AND after is `found`, never inside the green card. A differential searchFilters tells "breached the same way" (low, not caused by this change) from "breached by different traffic" (the rule's own severity, direction: changed). "Unchanged" is scoped to the rule's filter, because unbinding a filter changes the network without changing that answer. 5. _rule_breached is tested against a fake session, so CI now catches the four mutants only the skipped live test could: the policy passed in replaced by None, only the first query arm asked, the snapshot not switched, and the rule's kind ignored. 6. Every stage runs whatever the others did, so a failed reachability stage no longer silently skips the policy stage. Ordinary findings stop at 48; 49 and up are reserved (CH-049 summarises any overflow, CH-050..054 are this module's own error cards), so a long diff cannot collide with them. The CLI prints renamed and assigned notes. 7. Docs: the module docstring lists what is reported and in what order, CLAUDE.md's change_impact row is current, and the policy-rules section moved out from above section 7's sign-off into its own section 8. Not here: item 2 is #405 (Ankeet's). Shubham's rebase onto it, whichever lands second, absorbs the conflict in _policy_change_findings. Mutation-tested: device changes dropped, absent-device rules made errors, set_active_policy(None), first arm only, reachability failure skipping the policy stage, no cap, still-breached back in the green card, snapshot not switched -- each fails the new tests; the arm, snapshot and kind mutants fail WITHOUT Batfish. 1907 passed, 13 skipped with Batfish running. ruff clean on touched files.
change_impact.score(findings) returns one number, its band and the breakdown that produced it, computed only from status and severity -- never from a summary's wording. Opening counts more than closing because the severities already encode direction. Any error finding makes the band 'incomplete', so a change that could not be fully compared never reads as small. Returned beside the findings, not inside them (see #405). Live: secure->insecure large (14), insecure->secure moderate (7), no change none (0), one router removed small (2). Stacked on #406.
patelankeet2
left a comment
There was a problem hiding this comment.
Reviewed with real testing against real Batfish. Stacked on #406 (whose current head still matches my own existing approval -- checked, not assumed), tested merged with current main (b69d591).
git merge origin/main (local only) clean, auto-merged CLAUDE.md
pytest tests/test_change_score.py tests/test_change_impact.py
tests/test_change_against_policy.py -q 38 passed
ruff check --select E4,E7,E9,F ... clean
Live-reproduced all three fixture-based examples in the PR description, against real Batfish, each confirmed stable across repeated runs:
secure -> insecure: large (4 high x3 + 1 medium x2 = 14)
insecure -> secure: moderate (2 medium x2 + 3 low x1 = 7)
secure -> secure: none (no changes found = 0)
All three match exactly. Worth recording honestly: my first live run of each direction, right after merging main, gave different and in one case inconsistent results (an incomplete band where "moderate" was expected) -- re-running each 2-3 times immediately settled on the numbers above, every time. That's analyse_change()'s own Batfish/snapshot handling warming up after a fresh checkout, not anything in this PR -- score() is a pure function over an already-computed results list and makes no Batfish call of its own.
Full suite, same story, caught properly rather than glossed over. First run: 1 failed, 1927 passed -- test_change_impact_coverage.py::test_live_a_stranger_network_proposal_is_verified, a live Batfish test from #406's own earlier work, untouched by this diff. Reran it alone: passed. Reran the full suite a second time: 1928 passed, 0 failed. Same transient class as the live reproduction above -- not a regression from this PR, and not something to wave past without checking, which is why I reran it twice rather than once.
Independently mutation-tested the design rule you named explicitly -- incomplete never scores small. Disabled the if blind: band = "incomplete" override and reran the test:
AssertionError: assert ('small' == 'incomplete'
Exactly the dangerous direction the design calls out -- a change with real Batfish coverage gaps would have reported as reassuringly small instead of unknown-and-possibly-larger. Caught correctly. Reverted.
Checked the one structural question worth asking: SCORE_BANDS is read top-to-bottom via next(... if total >= floor), so its order (10, 4, 1, 0) has to stay descending for the thresholds to mean what the band names say -- confirmed by reading it alongside the live "large"/"moderate"/"none" results above, which land in the band their own total actually supports.
No em-dashes, no AI-tool mentions.
Approving.
# Conflicts: # analysis/change_impact.py
Arsh's three blockers on #406: 1. A pure deny raised the "opens access" banner. _breach_changed now returns (grew, shrank) from the differential frame's two verdicts; a breach that only shrinks is "narrows", low, and only a breach that grows keeps the rule's own severity. Batfish returns one example per direction, measured on a change that both adds and removes breaching flows. 2. Device presence was case-sensitive, so policy device RTR-US5 against Batfish's rtr-us5 filed a real breach under a tick. Both sides are now casefolded, and a user's own rule about a device in neither config is part of CH-051, as a scan calls it PC-050. 3. CH-051 was only ever built by the test helper. The helper now returns the stage's own output, and CH-051 is tested through analyse_change(). Also from the review: tests that _breach_changed passes the reference snapshot and asks every arm, a test that a failed compareFilters is CH-050 and never "No change detected", _cap keeps the most severe cards (policy first at equal severity) instead of the first 48, the FINDING IDS block, and the #389 credit in CLAUDE.md. Arsh's two live reproductions are now live tests.
|
Merged #406's branch in (merge-only push, M-2 rule 2), so this inherits the fix for Arsh's item 1. Tested on the merged result: 2289 passed, 13 skipped, ruff clean. Arsh's pure-deny case now scores |
…pact-coverage # Conflicts: # CLAUDE.md
_cap keeps the most important 48, but each kept finding kept the number it was found under, so with mixed severities the 49th found was kept as CH-049, the id of the summary card, and later ones could sit on CH-050 to CH-054. The kept list is renumbered in the order found. Arsh's two tests: mixed severities give CH-001..048 plus 049, and five highs found after 55 mediums all survive the cap (the 'severity ignored' mutant that used to survive the suite now fails).
Closes #420. Stacked on #406.
change_impact.score(findings)returns a number, a band (none / small / moderate / large / incomplete) and the breakdown, fromstatusandseverityonly. Any error makes itincomplete, never small. Returned beside the findings, not as an F-1 field.Live against Batfish:
Showing it in the propose pane is @SamikaPerera's call.