From 1503726a0b5f7fbd655a09a73a3874c72567df39 Mon Sep 17 00:00:00 2001 From: Shubham Date: Wed, 21 Oct 2026 09:25:33 +1300 Subject: [PATCH 1/4] Change impact reports what it did not look at (closes #403, most of #402) #403. Deleting a whole router reported "No change detected" with a green tick: neither compareFilters nor differentialReachability describes a device that exists on one side only. analyse_change() now compares the device lists of both snapshots and reports each added or removed device as `found`. A side whose devices cannot be listed is an error (CH-052), never "no change". Reproduced live on multi-device-10 minus rtr-dev009. #402, from @ARSH871-bot's post-merge review of #389: 1. A rule about a device in NEITHER config is not applicable -- this change cannot affect it -- and is named as such, not reported as an error. Every built-in rule used to be CH-051 on a network without rtr-us5, so every proposal there was "could not be fully verified". A rule about a device on ONE side only is one CH-051. Reproduced live with rtr-us5 renamed edge-fw1. 3. Only policy_compliance rules are compared, and the result now says so: a sentence beginning "NOT compared:" names the access_control and routing rules it skipped, and a policy with no policy_compliance rules says no rule of the user's was checked. 4. A rule breached before AND after is `found`, never inside the green card. A differential searchFilters tells "breached the same way" (low, not caused by this change) from "breached by different traffic" (the rule's own severity, direction: changed). "Unchanged" is scoped to the rule's filter, because unbinding a filter changes the network without changing that answer. 5. _rule_breached is tested against a fake session, so CI now catches the four mutants only the skipped live test could: the policy passed in replaced by None, only the first query arm asked, the snapshot not switched, and the rule's kind ignored. 6. Every stage runs whatever the others did, so a failed reachability stage no longer silently skips the policy stage. Ordinary findings stop at 48; 49 and up are reserved (CH-049 summarises any overflow, CH-050..054 are this module's own error cards), so a long diff cannot collide with them. The CLI prints renamed and assigned notes. 7. Docs: the module docstring lists what is reported and in what order, CLAUDE.md's change_impact row is current, and the policy-rules section moved out from above section 7's sign-off into its own section 8. Not here: item 2 is #405 (Ankeet's). Shubham's rebase onto it, whichever lands second, absorbs the conflict in _policy_change_findings. Mutation-tested: device changes dropped, absent-device rules made errors, set_active_policy(None), first arm only, reachability failure skipping the policy stage, no cap, still-breached back in the green card, snapshot not switched -- each fails the new tests; the arm, snapshot and kind mutants fail WITHOUT Batfish. 1907 passed, 13 skipped with Batfish running. ruff clean on touched files. --- CLAUDE.md | 2 +- analysis/change_impact.py | 402 +++++++++++++++++++-------- docs/policy-rules.md | 31 ++- tests/test_change_against_policy.py | 74 +++-- tests/test_change_impact.py | 5 +- tests/test_change_impact_coverage.py | 268 ++++++++++++++++++ 6 files changed, 626 insertions(+), 156 deletions(-) create mode 100644 tests/test_change_impact_coverage.py diff --git a/CLAUDE.md b/CLAUDE.md index 75e729f..c1d5b81 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -713,7 +713,7 @@ project's recurring failure family arriving through process rather than code. | **`risk` scoring** | Samika | Done (#60) — `POST_PROCESSORS`, ruleset in `docs/severity-rules.md`. Re-rates severity and sorts worst-first; the two limits are enforced in `pipeline.run_post_processors()`, not trusted | | Dashboard + secure upload | Samika | Done — real findings on screen since #39 | | PF Sense conversion | Ankeet | Done — `analysis/pfsense_convert.py`, and **hardened**: refuses config injection and path traversal via free-text fields (#53), an unbound ACL / empty rule set / unvalidated addressing (#54), and ambiguous rule order (#58, closing #47). See §7 | -| **`change_impact`** | Shubham | Done (#140, merged 17 August) — `analysis/change_impact.py`, `analyse_change(before, after)`. Shape C: **not** in `CHECKS`, and both the module docstring and the registry comment say so. Pairs `compareFilters` (which ACL *lines* moved) with `differentialReachability` (which *traffic* changed fate), because a changed line that moves no traffic is noise and moved traffic with no changed line is the case a filter diff alone misses. Direction-aware per `docs/policy-rules.md`: opening `high`, tightening `medium`, both reported | +| **`change_impact`** | Shubham | Done (#140, merged 17 August) — `analysis/change_impact.py`, `analyse_change(before, after)`. Shape C: **not** in `CHECKS`, and both the module docstring and the registry comment say so. Pairs `compareFilters` (which ACL *lines* moved) with `differentialReachability` (which *traffic* changed fate), because a changed line that moves no traffic is noise and moved traffic with no changed line is the case a filter diff alone misses. Direction-aware per `docs/policy-rules.md`: opening `high`, tightening `medium`, both reported. Since #323 (merged 14 October) it also runs every `policy_compliance` rule before and after and reports which the change breaks, fixes or alters; since #403/#402 it reports devices added or removed, treats rules about a device in neither config as not applicable, and says which policy sections it did **not** compare | | **A user policy reaching a check** | Arsh | Done (#181, merged 28 August) — `analysis/policy.py` loads and validates it; `policy_compliance` reads it via `rules_in_use()`, keyed on `active_policy() is None` rather than on emptiness. Measured below: policy-driven detections on a stranger's network go 3 → 8 | | **Draft a policy from your own config** | Arsh | Done (#326) — `analysis/draft_policy.py`, `/api/policy/draft`, a button under the policy picker. One `policy_compliance` rule per line of each applied filter, citing the file and line Batfish records for it. Each rule is checked with the same `searchFilters` question the check will ask, so a line an earlier line partly decides is listed as not drafted, with Batfish's example packet, rather than copied. `kind` and `violation_severity` are left as ``, and `analysis/policy.py` refuses the file until they are decided and its `draft` block is deleted: accepted as it stands, a draft passes on `rtr-us5-insecure` | | **AI: natural-language questions, on screen** | Ankeet + Samika | Done — `ai/query.py` + `/api/ask` (#66), crossing the ACL since #108/#141, and the chat pane is wired (`app.js` `chat-form` → `fetch("/api/ask")`). `question_understood` is rendered, which §7c says is the safety argument, not decoration | diff --git a/analysis/change_impact.py b/analysis/change_impact.py index 7e97d9b..16cf13b 100644 --- a/analysis/change_impact.py +++ b/analysis/change_impact.py @@ -12,7 +12,7 @@ CHECKS registry hands a check one Batfish session with one snapshot loaded. A differential question needs two, so `run(bf)` cannot express it. - def analyse_change(before_dir, after_dir) -> list[dict] + def analyse_change(before_dir, after_dir, host=None, policy=None) -> list[dict] It is called directly and is registered in nothing. DO NOT add it to CHECKS -- the pipeline carries a comment saying so, and this docstring is @@ -94,15 +94,34 @@ def analyse_change(before_dir, after_dir) -> list[dict] rather than two conventions. See #101. WHAT THIS DOES NOT DO YET, STATED RATHER THAN IMPLIED - It diffs. It does not AUDIT the proposed state -- it will not tell you - that the "after" config violates POL-2, only that it changed. Running the + It diffs. Since #323 it also runs every policy_compliance rule before and + after, so it tells you which of those rules the change breaks, fixes, + leaves breached, or alters -- but it still does not run access_control or + routing over `after_dir`, and the result says so rather than implying + otherwise (#402). Running the producer checks over `after_dir` as well is open question 2 on the shapes document and is not mine to settle alone; I have argued for it on #61 and #86. Until then, "no change detected" means exactly that and must not be read as "the proposed config is fine". RUN IT - python -m analysis.change_impact + python -m analysis.change_impact [policy.json] + +WHAT IT REPORTS, IN ORDER (#403, #402) + devices added or removed found -- the largest change, which neither + Batfish comparison describes + filter lines that moved found compareFilters + traffic that changed fate found differentialReachability + policy rules broken / fixed / + altered / still breached found policy_compliance's own query, per side + a stage that could not run error CH-050 filter, CH-052 devices, + CH-053 reachability, CH-054 policy, + CH-051 rules not compared + more than 48 changes found CH-049, the rest summarised + nothing moved none CH-000, naming the rules it did not move + + Each stage runs whatever the others did, so a failure never hides what + another stage proved, and never hides that a stage did not run. """ import json @@ -399,78 +418,206 @@ def _rule_label(rule: Dict[str, Any]) -> str: return f"{head} ({rule['description']})" if rule.get("description") else head -def _policy_change_findings(bf: Any, number: int) -> Tuple[List[Dict[str, Any]], List[str], int]: - """Run every applicable policy rule against both snapshots. +def _breach_changed(bf: Any, rule: Dict[str, Any]) -> bool: + """For a rule breached before AND after: did the change alter WHICH + traffic breaches it? (#402 item 4) - Returns (findings, unchanged, number). `unchanged` is returned rather than - turned into a card here, because analyse_change() already owns the one - `CH-000` this module emits, and two of them would collide on id. + "Breached on both sides" is not "unchanged". Closing LAN->8.8.8.8 while + opening LAN->1.1.1.1 leaves a "no other destinations" rule breached + throughout, and the old wording called that "neither causes nor fixes". + Differential searchFilters answers it directly: rows are flows whose + verdict differs between the two snapshots. """ + from pybatfish.datamodel.flow import HeaderConstraints + from analysis.checks import policy_compliance as pc - rules, policy_label, _ = pc.rules_in_use() - if not rules: - return [], [], number + action = pc.ACTION_BY_KIND[rule["kind"]] + for headers in rule["queries"]: + frame = ( + bf.q.searchFilters( + nodes=rule["node"], filters=rule["filter"], action=action, + headers=HeaderConstraints(**headers), + ) + .answer(snapshot=AFTER_SNAPSHOT, reference_snapshot=BEFORE_SNAPSHOT) + .frame() + ) + if len(frame): + return True + return False - results: List[Dict[str, Any]] = [] - unchanged: List[str] = [] - not_compared: List[str] = [] +def _devices(bf: Any) -> "Tuple[set | None, set | None]": + """The devices in each snapshot, or None for a side we could not read.""" + from analysis import snapshot + + out = [] + for name in (BEFORE_SNAPSHOT, AFTER_SNAPSHOT): + bf.set_snapshot(name) + out.append(snapshot.device_names(bf)) + return out[0], out[1] + + +def _device_change_findings(before: set, after: set, number: int) -> Tuple[List[Dict[str, Any]], int]: + """A device on one side only is a change, and the largest kind (#403). + + Neither compareFilters nor differentialReachability reports it: a filter + that no longer exists has nothing to compare against, so deleting a + whole router used to come back as "No change detected" with a green tick. + """ + results = [] + for name in sorted(before - after): + results.append(findings.make_finding( + check=CHECK_NAME, severity=_CLOSING_SEVERITY, device=name, + summary=f"Device {name} is removed by this change", + detail=(f"{name} is in the before configuration and not in the after one. " + "Every flow it carried changes, and the filter and reachability " + "comparisons below do not describe a device that exists on one " + "side only, so this card is the only record of it."), + source=f"{BEFORE_SNAPSHOT} -> {AFTER_SNAPSHOT}", status="found", number=number)) + number += 1 + for name in sorted(after - before): + results.append(findings.make_finding( + check=CHECK_NAME, severity=_OPENING_SEVERITY, device=name, + summary=f"Device {name} is added by this change", + detail=(f"{name} is in the after configuration and not in the before one. " + "Nothing about its traffic is compared below, because there is " + "nothing to compare it against -- run a scan of the after " + "configuration to check it."), + source=f"{BEFORE_SNAPSHOT} -> {AFTER_SNAPSHOT}", status="found", number=number)) + number += 1 + return results, number + + +def _policy_coverage() -> Tuple[List[Dict[str, Any]], str, bool, str]: + """(policy_compliance rules, label, user_supplied, sentence about the rest). + + US-31 says "every policy rule". This stage compares policy_compliance + rules only, and the sentence says so out loud (#402 item 3) -- a user + whose policy is one access_control rule must not read "unchanged" as + "your rule still holds". + """ + from analysis.checks import policy_compliance as pc + + rules, label, user_supplied = pc.rules_in_use() + note = "" + active = policy_module.active_policy() + if active is not None: + skipped = [f"{len(active.entries_for(s))} {s}" + for s in ("access_control", "routing") if active.entries_for(s)] + if skipped: + note = ("NOT compared: " + " and ".join(skipped) + " rule(s) in your policy. " + "Change impact compares policy_compliance rules only; scan the " + "after configuration to check the others.") + if not rules: + note = ("NOT compared: your policy has no policy_compliance rules, so no " + "rule of yours was checked against this change. " + note).strip() + return rules, label, user_supplied, note + + +def _policy_change_findings(bf: Any, number: int, before: "set | None" = None, + after: "set | None" = None) -> Dict[str, Any]: + """Run every applicable policy_compliance rule against both snapshots. + + Returns a dict, not a finding list, because three of its outputs are + sentences that analyse_change() folds into the ONE card it owns: + + findings created, resolved, altered or still-breached rules + unchanged same verdict on the rule's filter, before and after + absent rules about a device in NEITHER configuration + one_side rules about a device in only ONE configuration + not_compared rules we could not ask -- never "unchanged" (F-4) + coverage what this stage did not compare at all + """ + rules, policy_label, _, coverage = _policy_coverage() + out: Dict[str, Any] = {"findings": [], "unchanged": [], "absent": [], "one_side": [], + "not_compared": [], "coverage": coverage, + "label": policy_label, "number": number} for rule in rules: + node = rule["node"] + if before is not None and after is not None: + if node not in before and node not in after: + # Worded like PC-050 (#402 item 1). This change cannot affect a + # rule about a device in neither configuration: a proven + # "not applicable", not a blind spot, so not an error -- an + # error here made every proposal on a network without rtr-us5 + # "could not be fully verified". + out["absent"].append(f"{rule.get('number', '?')} ({node})") + continue + if (node in before) != (node in after): + side = "before" if node in before else "after" + out["one_side"].append(f"{_rule_label(rule)}: {node} is in the {side} configuration only") + continue try: - before = _rule_breached(bf, BEFORE_SNAPSHOT, rule) - after = _rule_breached(bf, AFTER_SNAPSHOT, rule) + was = _rule_breached(bf, BEFORE_SNAPSHOT, rule) + now = _rule_breached(bf, AFTER_SNAPSHOT, rule) + altered = _breach_changed(bf, rule) if (was and now) else False except Exception as error: - # A rule we could not ask is a blind spot, never "unchanged" (F-4). - not_compared.append(f"{_rule_label(rule)}: {findings.describe_error(error)}") + out["not_compared"].append(f"{_rule_label(rule)}: {findings.describe_error(error)}") continue - if before == after: - state = "still breached" if before else "holds" - unchanged.append(f"{rule.get('number', '?')} ({state})") + if not was and not now: + out["unchanged"].append(str(rule.get("number", "?"))) continue - created = after and not before - direction = _RULE_DIRECTION[(rule["kind"], created)] - if number >= ERROR_NUMBER_OFFSET: - not_compared.append(f"{_rule_label(rule)}: more rule changes than finding numbers") - continue - results.append( - findings.make_finding( - check=CHECK_NAME, - severity=(rule.get("violation_severity", "high") if created else "low"), - device=rule["node"], - summary=( - f"This change BREAKS {_rule_label(rule)}" - if created else - f"This change FIXES {_rule_label(rule)}" - ), - detail=( - f"{rule['kind']} on {rule['node']} filter {rule['filter']}: " - f"{'held before, breached after' if created else 'breached before, holds after'}. " - f"Direction: {direction}. " - + (f"Violation if applied: {rule.get('violation_summary')}." if created - else "The breach this rule reports is gone in the proposed config.") - ), - source=f"{policy_label}; rule {rule.get('number', '?')}", - status="found", - number=number, - ) - ) + where = f"{rule['kind']} on {node} filter {rule['filter']}" + if was and now: + if altered: + summary = f"This change alters how {_rule_label(rule)} is breached" + detail = (f"{where}: breached before and after, but by different " + "traffic -- the change moves which flows break the rule. " + "Direction: changed.") + severity = rule.get("violation_severity", "high") + else: + summary = f"{_rule_label(rule)} is breached before and after this change" + detail = (f"{where}: breached the same way on both sides. This change " + "does not cause it and does not fix it; the violation stays.") + severity = "low" + else: + created = now and not was + direction = _RULE_DIRECTION[(rule["kind"], created)] + summary = (f"This change BREAKS {_rule_label(rule)}" if created + else f"This change FIXES {_rule_label(rule)}") + detail = (f"{where}: {'held before, breached after' if created else 'breached before, holds after'}. " + f"Direction: {direction}. " + + (f"Violation if applied: {rule.get('violation_summary')}." if created + else "The breach this rule reports is gone in the proposed config.")) + severity = rule.get("violation_severity", "high") if created else "low" + out["findings"].append(findings.make_finding( + check=CHECK_NAME, severity=severity, device=node, summary=summary, detail=detail, + source=f"{policy_label}; rule {rule.get('number', '?')}", status="found", + number=number)) number += 1 - - if not_compared: - results.append( - findings.error_finding( - check=CHECK_NAME, - device="unknown", - summary=f"{len(not_compared)} policy rule(s) could not be compared across the change", - detail="; ".join(not_compared), - source=policy_label, - number=SKIPPED_NUMBER + 1, - ) - ) - return results, unchanged, number + out["number"] = number + return out + + +#: Ordinary findings stop here. 49 and above are this module's own cards, so +#: a long diff can never collide with them (#402 item 6) -- the band +#: policy_compliance already uses (PC-049, PC-050). +_OVERFLOW_NUMBER = 49 +_RULES_NOT_COMPARED_NUMBER = SKIPPED_NUMBER + 1 # CH-051 +_DEVICES_UNKNOWN_NUMBER = SKIPPED_NUMBER + 2 # CH-052 +_REACHABILITY_FAILED_NUMBER = SKIPPED_NUMBER + 3 # CH-053 +_POLICY_FAILED_NUMBER = SKIPPED_NUMBER + 4 # CH-054 + + +def _cap(regular: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + """Keep the first 48 ordinary findings; fold the rest into CH-049.""" + if len(regular) < _OVERFLOW_NUMBER: + return regular + kept, rest = regular[: _OVERFLOW_NUMBER - 1], regular[_OVERFLOW_NUMBER - 1:] + order = {"high": 0, "medium": 1, "low": 2} + worst = min((f["severity"] for f in rest), key=order.get) + kept.append(findings.make_finding( + check=CHECK_NAME, severity=worst, device="unknown", + summary=f"{len(rest)} further change(s) not listed individually", + detail=("This change produced more findings than are listed one by one. " + "The first of the rest: " + "; ".join(f["summary"] for f in rest[:5]) + + (" ..." if len(rest) > 5 else "")), + source=f"{BEFORE_SNAPSHOT} -> {AFTER_SNAPSHOT}", status="found", + number=_OVERFLOW_NUMBER)) + return kept def analyse_change( @@ -489,14 +636,15 @@ def analyse_change( The three outcomes are kept distinct, which is the whole of F-4 here: found the change did something, and here is what - none both configs loaded, both questions ran, nothing moved + none both configs loaded, every question ran, nothing moved error we could not compare -- NOT the same as "nothing changed" That middle one is the dangerous claim. "This change is safe" is exactly what a user wants to hear before applying it to a firewall, so it may only - be said when both snapshots parsed and both questions actually ran. + be said when both snapshots parsed and every question actually ran. """ before_dir, after_dir = Path(before_dir), Path(after_dir) + where = f"{before_dir} -> {after_dir}" try: bf = connect(host) @@ -508,7 +656,7 @@ def analyse_change( "docker start batfish. " f"Underlying error: {findings.describe_error(error)}" ), - source=f"{before_dir} -> {after_dir}", + source=where, ) # Load both. Named snapshots in one network so the differential questions @@ -554,62 +702,79 @@ def analyse_change( source=str(folder), ) - results: List[Dict[str, Any]] = [] + regular: List[Dict[str, Any]] = [] + reserved: List[Dict[str, Any]] = [] number = 1 + def blind(n: int, summary: str, detail: str) -> None: + reserved.append(findings.error_finding( + check=CHECK_NAME, summary=summary, detail=detail, source=where, number=n)) + + # Which devices are on each side (#403). Unknown is an error, never "none". + before_devs = after_devs = None + try: + before_devs, after_devs = _devices(bf) + except Exception as error: + blind(_DEVICES_UNKNOWN_NUMBER, "Could not list the devices in each configuration", + findings.describe_error(error)) + if before_devs is None or after_devs is None: + if not any(f["id"].endswith(f"{_DEVICES_UNKNOWN_NUMBER:03d}") for f in reserved): + blind(_DEVICES_UNKNOWN_NUMBER, "Could not list the devices in each configuration", + "Batfish did not say which devices each configuration defines, so a " + "device added or removed by this change would not be reported.") + else: + device_changes, number = _device_change_findings(before_devs, after_devs, number) + regular.extend(device_changes) + + # Each stage runs whatever the others did (#402 item 6). What one proved + # stands when another fails, and the failure is said (#22). try: filter_changes, number = _filter_change_findings(bf, number) + regular.extend(filter_changes) except Exception as error: - return _error( - summary="Change impact could not run: the filter comparison failed", - detail=findings.describe_error(error), - source=f"{before_dir} -> {after_dir}", - ) - results.extend(filter_changes) + blind(SKIPPED_NUMBER, "The filter comparison could not run", findings.describe_error(error)) try: flow_changes, number = _reachability_change_findings(bf, number) + regular.extend(flow_changes) except Exception as error: - # The filter comparison may already have found real changes. Report - # BOTH -- what was proved AND what could not be reached. Discarding a - # proven finding because a second query failed is issue #22, and it - # fails in the reassuring direction. - results.extend( - _error( - summary="The reachability comparison could not run", - detail=findings.describe_error(error), - source=f"{before_dir} -> {after_dir}", - ) - ) - return results - results.extend(flow_changes) + blind(_REACHABILITY_FAILED_NUMBER, "The reachability comparison could not run", + findings.describe_error(error)) # Every policy rule, before and after (#323). Installed the same way # analyse() installs it and cleared in a `finally`, so a policy never # outlives the call that supplied it. policy_module.set_active_policy(policy) + stage: Dict[str, Any] = {} try: - rule_changes, unchanged, number = _policy_change_findings(bf, number) + stage = _policy_change_findings(bf, number, before_devs, after_devs) + regular.extend(stage["findings"]) except Exception as error: - # Same rule as the reachability stage: what was proved above stands, - # and the user is told what could not be checked (#22). - results.extend( - _error( - summary="The policy comparison could not run", - detail=findings.describe_error(error), - source=f"{before_dir} -> {after_dir}", - ) - ) - return results + blind(_POLICY_FAILED_NUMBER, "The policy comparison could not run", + findings.describe_error(error)) finally: policy_module.clear_active_policy() - results.extend(rule_changes) - unchanged_note = ( - " Policy rules with the same verdict before and after, rule (state): " - + ", ".join(unchanged) + "." - if unchanged else "" - ) + not_compared = stage.get("not_compared", []) + stage.get("one_side", []) + if not_compared: + blind(_RULES_NOT_COMPARED_NUMBER, + f"{len(not_compared)} policy rule(s) could not be compared across the change", + "; ".join(not_compared)) + + # The sentences about rules nothing above reported on. + notes = [] + if stage.get("unchanged"): + notes.append("Policy rule(s) with the same verdict on their filter before and " + "after (this says the filter's decision did not move, not that " + "nothing about their traffic did): " + ", ".join(stage["unchanged"]) + ".") + if stage.get("absent"): + notes.append("Policy rule(s) about devices in neither configuration, which this " + "change cannot affect: " + ", ".join(stage["absent"]) + ".") + if stage.get("coverage"): + notes.append(stage["coverage"]) + note_text = (" " + " ".join(notes)) if notes else "" + + results = _cap(regular) + reserved if not results: return [ @@ -618,32 +783,28 @@ def analyse_change( device="unknown", summary="No change detected between the two configurations", detail=( - "No filter line treats the same traffic differently, and no " - "flow reaches a destination it could not reach before. This " - "says the change moved nothing; it does not say the proposed " - "config is free of problems -- run the audit for that." - + unchanged_note + "No device was added or removed, no filter line treats the same " + "traffic differently, and no flow reaches a destination it could " + "not reach before. This says the change moved nothing; it does " + "not say the proposed config is free of problems -- run the " + "audit for that." + note_text ), - source=f"{before_dir} -> {after_dir}", + source=where, number=0, ) ] - if unchanged: - # US-31 asks for a direction per rule, and "unchanged" is one. Only - # emitted here, alongside real changes, so it can never be mistaken - # for the all-clear above -- and it takes CH-000 only because that - # all-clear is, by construction, not in this list. + if notes: + # Alongside real changes, so it can never be mistaken for the + # all-clear above -- and it takes CH-000 only because that all-clear + # is, by construction, not in this list. results.append( findings.no_issues_finding( check=CHECK_NAME, device="unknown", - summary=f"{len(unchanged)} policy rule(s) unchanged by this edit", - detail=( - "Same verdict before and after, so this change neither " - "causes nor fixes them." + unchanged_note - ), - source=f"{before_dir} -> {after_dir}", + summary="Policy rules this change did not move", + detail=note_text.strip(), + source=where, number=0, ) ) @@ -660,6 +821,9 @@ def main() -> None: policy = None if len(sys.argv) == 4: policy = policy_module.load_policy_file(sys.argv[3]) + # Every channel, like `python -m analysis.pipeline --policy` (#402 item 6). + for note in list(policy.renamed) + list(policy.assigned): + print(f"note: {note}") results = analyse_change(sys.argv[1], sys.argv[2], policy=policy) print(json.dumps(results, indent=2)) print() diff --git a/docs/policy-rules.md b/docs/policy-rules.md index a05e08f..81ff264 100644 --- a/docs/policy-rules.md +++ b/docs/policy-rules.md @@ -604,7 +604,16 @@ unapproved destination, and POL-2 arm A catches DNS to `10.20.0.5`. Verified: --- -### Change impact now checks every policy rule (#323, US-31) +## 7. Sign-off + +| Member | Role in this decision | Agreed | +|---|---|---| +| Shubham | Author; owns the check | ✅ | +| Arsh | Registry, pipeline, `findings.py` impact | ✅ rules, IDs, and the §4 `findings.py` change | +| Samika | Severity values are his | ✅ | +| Ankeet | Consumes findings in the AI layer | ⬜ | + +## 8. Change impact checks every policy rule (#323, #402, #403) `analyse_change(before, after)` used to answer one question: *what traffic moved*. It now also answers the one an engineer actually asks before applying @@ -616,8 +625,9 @@ breach is), once against each snapshot: | before | after | reported as | |---|---|---| -| holds | holds | unchanged | -| breached | breached | unchanged — still broken, not caused by this edit | +| holds | holds | unchanged — *on the rule's filter*, which is all that is claimed | +| breached | breached, by the same traffic | **still breached** — `found`, `low`: not caused by this edit, but never a green tick (#402) | +| breached | breached, by different traffic | **ALTERS** the rule — at the rule's own severity, from a differential `searchFilters` (#402) | | breached | holds | **FIXES** the rule, `low` | | holds | breached | **BREAKS** the rule, at the rule's own severity | @@ -653,11 +663,12 @@ built-in rules, like a scan does. The CLI takes it as a third argument: python -m analysis.change_impact before/ after/ my-policy.json ``` -## 7. Sign-off -| Member | Role in this decision | Agreed | -|---|---|---| -| Shubham | Author; owns the check | ✅ | -| Arsh | Registry, pipeline, `findings.py` impact | ✅ rules, IDs, and the §4 `findings.py` change | -| Samika | Severity values are his | ✅ | -| Ankeet | Consumes findings in the AI layer | ⬜ | +### What it does not look at, said out loud (#402, #403) + +- **Devices added or removed** are their own `found` cards. Before #403, deleting a whole router came back as "No change detected" with a green tick, because neither Batfish comparison describes a device that exists on one side only. +- **A rule about a device in neither configuration** is *not applicable* — this change cannot affect it — and is named in the result, not reported as an error. Reporting it as an error made every proposal on a network without `rtr-us5` "could not be fully verified". +- **A rule about a device on one side only** cannot be compared and is a single `CH-051` error. +- **Only `policy_compliance` rules are compared.** A policy with `access_control` or `routing` rules gets a sentence beginning "NOT compared:" naming how many, and a policy with no `policy_compliance` rules says no rule of the user's was checked. +- **"Unchanged" means the rule's filter decided its traffic the same way.** Unbinding the filter from its interface changes the network without changing that answer, so the wording claims no more than that. + diff --git a/tests/test_change_against_policy.py b/tests/test_change_against_policy.py index 71a9a00..842fb07 100644 --- a/tests/test_change_against_policy.py +++ b/tests/test_change_against_policy.py @@ -32,11 +32,12 @@ def rule(number, kind, severity="high"): } -def stage(monkeypatch, rules, verdicts): +def stage(monkeypatch, rules, verdicts, altered=False, devices=None): """Run the policy stage with fixed rules and fixed per-snapshot verdicts. `verdicts` maps rule number -> (breached_before, breached_after), or to an - Exception to raise when that rule is asked. + Exception to raise when that rule is asked. Returns (findings, unchanged, + number), plus the whole stage dict as a fourth value for newer tests. """ monkeypatch.setattr(pc, "rules_in_use", lambda: (rules, "test policy", True)) @@ -47,13 +48,22 @@ def fake(bf, snapshot, r): return v[0] if snapshot == change_impact.BEFORE_SNAPSHOT else v[1] monkeypatch.setattr(change_impact, "_rule_breached", fake) - return change_impact._policy_change_findings(object(), 1) + monkeypatch.setattr(change_impact, "_breach_changed", lambda bf, r: altered) + before, after = devices if devices else (None, None) + out = change_impact._policy_change_findings(object(), 1, before, after) + errors = [f"{n}" for n in out["not_compared"]] + found = list(out["findings"]) + if errors: + found.append({"status": "error", "id": "CH-051", + "evidence": {"detail": "; ".join(out["not_compared"])}}) + unchanged = [f"{n} (holds)" for n in out["unchanged"]] + return found, unchanged, out["number"], out # --- the four outcomes ----------------------------------------------------- def test_a_change_that_causes_a_breach_says_it_breaks_the_rule(monkeypatch): - found, unchanged, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (False, True)}) + found, unchanged, _, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (False, True)}) assert len(found) == 1 f = found[0] assert f["status"] == "found" @@ -63,7 +73,7 @@ def test_a_change_that_causes_a_breach_says_it_breaks_the_rule(monkeypatch): def test_a_change_that_repairs_a_breach_says_it_fixes_the_rule(monkeypatch): - found, _, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (True, False)}) + found, _, _, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (True, False)}) assert len(found) == 1 assert "FIXES" in found[0]["summary"] assert found[0]["severity"] == "low", "a fix is good news, not a high finding" @@ -71,7 +81,7 @@ def test_a_change_that_repairs_a_breach_says_it_fixes_the_rule(monkeypatch): def test_a_change_that_fixes_one_and_breaks_another_reports_both(monkeypatch): """US-31's own criterion, and the reason this stage exists.""" - found, _, _ = stage( + found, _, _, _ = stage( monkeypatch, [rule(1, "prohibition"), rule(4, "requirement", severity="medium")], {1: (True, False), 4: (False, True)}, @@ -81,15 +91,30 @@ def test_a_change_that_fixes_one_and_breaks_another_reports_both(monkeypatch): assert any("BREAKS" in s and "rule 4" in s for s in summaries) -def test_a_rule_with_the_same_verdict_either_side_is_unchanged_not_a_finding(monkeypatch): - found, unchanged, _ = stage( - monkeypatch, [rule(1, "prohibition"), rule(2, "prohibition")], - {1: (False, False), 2: (True, True)}, - ) +def test_a_rule_that_holds_either_side_is_unchanged_not_a_finding(monkeypatch): + found, unchanged, _, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (False, False)}) assert found == [] - assert unchanged == ["1 (holds)", "2 (still breached)"], ( - "a rule broken before AND after is unchanged -- this edit did not cause it" - ) + assert unchanged == ["1 (holds)"] + + +def test_a_rule_breached_on_both_sides_is_never_a_green_tick(monkeypatch): + """#402 item 4. "Still breached" sat inside the `none` card, which the + propose pane draws as a tick. A violation that survives the change is + still a violation.""" + found, unchanged, _, _ = stage(monkeypatch, [rule(2, "prohibition")], {2: (True, True)}) + assert unchanged == [] + assert len(found) == 1 and found[0]["status"] == "found" + assert "breached before and after" in found[0]["summary"] + assert found[0]["severity"] == "low", "not caused by this change" + + +def test_a_breach_that_moves_to_other_traffic_is_reported_as_altered(monkeypatch): + """Closing LAN->8.8.8.8 while opening LAN->1.1.1.1 keeps a rule breached + throughout; the differential query says the traffic moved.""" + found, _, _, _ = stage(monkeypatch, [rule(2, "prohibition")], {2: (True, True)}, altered=True) + assert "alters how" in found[0]["summary"] + assert found[0]["severity"] == "high", "it is the rule's own severity" + assert "Direction: changed" in found[0]["evidence"]["detail"] @pytest.mark.parametrize("kind,before,after,direction", [ @@ -99,25 +124,23 @@ def test_a_rule_with_the_same_verdict_either_side_is_unchanged_not_a_finding(mon ("requirement", True, False, "opening"), ]) def test_direction_is_stated_in_this_modules_own_terms(monkeypatch, kind, before, after, direction): - found, _, _ = stage(monkeypatch, [rule(1, kind)], {1: (before, after)}) + found, _, _, _ = stage(monkeypatch, [rule(1, kind)], {1: (before, after)}) assert f"Direction: {direction}" in found[0]["evidence"]["detail"] # --- F-4: could-not-ask is never unchanged ---------------------------------- def test_a_rule_that_cannot_be_asked_is_an_error_never_unchanged(monkeypatch): - found, unchanged, _ = stage( + found, unchanged, _, out = stage( monkeypatch, [rule(1, "prohibition"), rule(2, "prohibition")], {1: RuntimeError("searchFilters failed"), 2: (False, False)}, ) - errors = [f for f in found if f["status"] == "error"] - assert len(errors) == 1 - assert "rule 1" in errors[0]["evidence"]["detail"] + assert len(out["not_compared"]) == 1 and "rule 1" in out["not_compared"][0] assert unchanged == ["2 (holds)"], "the rule we could not ask must not be listed as holding" def test_the_ids_this_stage_emits_are_unique(monkeypatch): - found, _, _ = stage( + found, _, _, _ = stage( monkeypatch, [rule(n, "prohibition") for n in (1, 2, 3)] + [rule(9, "prohibition")], {1: (False, True), 2: (True, False), 3: (False, True), 9: RuntimeError("x")}, ) @@ -137,6 +160,7 @@ def _stub_pipeline(monkeypatch, *, filters=(), policy_stage=None): monkeypatch.setattr(change_impact, "_filter_change_findings", lambda bf, n: (list(filters), n + len(filters))) monkeypatch.setattr(change_impact, "_reachability_change_findings", lambda bf, n: ([], n)) + monkeypatch.setattr(change_impact, "_devices", lambda bf: ({"rtr-us5"}, {"rtr-us5"})) if policy_stage is not None: monkeypatch.setattr(change_impact, "_policy_change_findings", policy_stage) @@ -144,11 +168,11 @@ def _stub_pipeline(monkeypatch, *, filters=(), policy_stage=None): def test_unchanged_rules_join_the_all_clear_rather_than_a_second_ch000(monkeypatch): """Two cards both numbered 0 would collide on id. When nothing moved, the unchanged rules are named inside the single all-clear.""" - _stub_pipeline(monkeypatch, policy_stage=lambda bf, n: ([], ["1 (holds)", "2 (holds)"], n)) + _stub_pipeline(monkeypatch, policy_stage=lambda bf, n, b=None, a=None: {"findings": [], "unchanged": ["1", "2"], "number": n}) results = change_impact.analyse_change("before", "after") assert len(results) == 1 assert results[0]["status"] == "none" - assert "1 (holds)" in results[0]["evidence"]["detail"] + assert "1, 2" in results[0]["evidence"]["detail"] def test_with_real_changes_the_unchanged_rules_still_get_reported(monkeypatch): @@ -156,17 +180,17 @@ def test_with_real_changes_the_unchanged_rules_still_get_reported(monkeypatch): check="change_impact", severity="high", device="rtr-us5", summary="moved", detail="d", source="s", status="found", number=1) _stub_pipeline(monkeypatch, filters=[moved], - policy_stage=lambda bf, n: ([], ["3 (holds)"], n)) + policy_stage=lambda bf, n, b=None, a=None: {"findings": [], "unchanged": ["3"], "number": n}) results = change_impact.analyse_change("before", "after") ids = [f["id"] for f in results] assert len(ids) == len(set(ids)) - assert any(f["status"] == "none" and "3 (holds)" in f["evidence"]["detail"] for f in results) + assert any(f["status"] == "none" and "3." in f["evidence"]["detail"] for f in results) def test_the_policy_is_cleared_even_when_the_stage_raises(monkeypatch): """A policy outliving the call that supplied it would answer the next caller's question with someone else's rules.""" - def boom(bf, n): + def boom(bf, n, b=None, a=None): raise RuntimeError("policy stage failed") _stub_pipeline(monkeypatch, policy_stage=boom) sentinel = policy_module.load_policy({"device": "rtr-us5", "policy_compliance": []}) diff --git a/tests/test_change_impact.py b/tests/test_change_impact.py index d6bde60..82246bf 100644 --- a/tests/test_change_impact.py +++ b/tests/test_change_impact.py @@ -121,7 +121,10 @@ def fake_flows(bf, number): # tests keep meaning what they meant before it existed. Its own tests are # in tests/test_change_against_policy.py. monkeypatch.setattr(change_impact, "_policy_change_findings", - lambda bf, number: ([], [], number)) + lambda bf, number, before=None, after=None: {"findings": [], "number": number}) + # The same device on both sides, so #403's device check finds no change + # and these tests keep meaning what they meant before it existed. + monkeypatch.setattr(change_impact, "_devices", lambda bf: ({"rtr-us5"}, {"rtr-us5"})) def _a_change(number=1): diff --git a/tests/test_change_impact_coverage.py b/tests/test_change_impact_coverage.py new file mode 100644 index 0000000..b5f5cac --- /dev/null +++ b/tests/test_change_impact_coverage.py @@ -0,0 +1,268 @@ +"""What change impact looked at, and what it says it looked at (#403, #402). + +Every test here is a case where the old answer was confident and wrong: + + #403 a router deleted between before and after -> "No change detected" + #402 item 1 a network without rtr-us5 -> every proposal "could not be verified" + #402 item 3 a policy of access_control rules -> silence about them + #402 item 5 four mutants that only a live test (skipped on CI) could catch + #402 item 6 a failed reachability stage silently skipping the policy stage, + and ids that a long diff could collide with +""" + +import shutil +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from analysis import change_impact, findings, policy as policy_module +from analysis.checks import policy_compliance as pc +from conftest import needs_batfish + +FIXTURES = Path(__file__).parent / "fixtures" + + +def _pipeline(monkeypatch, *, devices=({"rtr-us5"}, {"rtr-us5"}), filters=(), + flows_raise=False, stage=None): + monkeypatch.setattr(change_impact, "connect", lambda host=None: SimpleNamespace( + set_network=lambda n: None, set_snapshot=lambda n: None)) + monkeypatch.setattr(change_impact, "load_snapshot", lambda *a, **k: None) + monkeypatch.setattr(change_impact, "find_parse_problems", lambda bf: []) + monkeypatch.setattr(change_impact, "_devices", lambda bf: devices) + monkeypatch.setattr(change_impact, "_filter_change_findings", + lambda bf, n: (list(filters), n + len(filters))) + + def flows(bf, n): + if flows_raise: + raise RuntimeError("differentialReachability exploded") + return [], n + monkeypatch.setattr(change_impact, "_reachability_change_findings", flows) + if stage is not None: + monkeypatch.setattr(change_impact, "_policy_change_findings", stage) + else: + monkeypatch.setattr(change_impact, "_policy_change_findings", + lambda bf, n, b=None, a=None: {"findings": [], "number": n}) + + +def _found(n, severity="high"): + return findings.make_finding(check="change_impact", severity=severity, device="r", + summary=f"change {n}", detail="d", source="s", + status="found", number=n) + + +# --- #403: devices on one side only ------------------------------------------ + +def test_a_removed_device_is_a_change_never_no_change_detected(monkeypatch): + _pipeline(monkeypatch, devices=({"rtr-a", "rtr-b"}, {"rtr-a"})) + results = change_impact.analyse_change("before", "after") + assert not any("No change detected" in f["summary"] for f in results) + removed = [f for f in results if "removed" in f["summary"]] + assert len(removed) == 1 and removed[0]["status"] == "found" + assert removed[0]["device"] == "rtr-b" + + +def test_an_added_device_is_a_change(monkeypatch): + _pipeline(monkeypatch, devices=({"rtr-a"}, {"rtr-a", "rtr-c"})) + results = change_impact.analyse_change("before", "after") + added = [f for f in results if "added" in f["summary"]] + assert len(added) == 1 and added[0]["status"] == "found" and added[0]["device"] == "rtr-c" + + +def test_unknown_device_lists_are_an_error_never_no_change(monkeypatch): + """None means "we could not find out", which is not "nothing changed".""" + _pipeline(monkeypatch, devices=({"rtr-a"}, None)) + results = change_impact.analyse_change("before", "after") + assert [f["status"] for f in results] == ["error"] + assert results[0]["id"] == "CH-052" + + +# --- #402 item 1: rules about devices that are not there ---------------------- + +def _rule(n, node="rtr-us5", kind="prohibition"): + return {"number": n, "kind": kind, "node": node, "filter": "acl_in", + "description": f"rule {n}", "violation_severity": "high", + "violation_summary": "x", "queries": [{"dstIps": "10.20.0.5"}]} + + +def test_a_rule_about_a_device_in_neither_config_is_not_applicable_not_an_error(monkeypatch): + monkeypatch.setattr(pc, "rules_in_use", lambda: ([_rule(1)], "built-in", False)) + monkeypatch.setattr(change_impact, "_rule_breached", + lambda *a: pytest.fail("must not ask Batfish about an absent device")) + out = change_impact._policy_change_findings(object(), 1, {"edge-fw1"}, {"edge-fw1"}) + assert out["absent"] == ["1 (rtr-us5)"] + assert out["not_compared"] == [] and out["findings"] == [] + + +def test_a_stranger_network_is_verified_rather_than_could_not_compare(monkeypatch): + """Arsh's repro, end to end through analyse_change: five built-in rules + about rtr-us5 on a network called edge-fw1 produced CH-051 x5 errors, so + propose_change() said verified=False on every request.""" + monkeypatch.setattr(pc, "rules_in_use", lambda: ([_rule(n) for n in range(1, 6)], "built-in", False)) + _pipeline(monkeypatch, devices=({"edge-fw1"}, {"edge-fw1"}), + stage=change_impact._policy_change_findings) + results = change_impact.analyse_change("before", "after") + assert all(f["status"] != "error" for f in results), results + assert "neither configuration" in results[0]["evidence"]["detail"] + + +def test_a_rule_about_a_device_on_one_side_only_cannot_be_compared(monkeypatch): + monkeypatch.setattr(pc, "rules_in_use", lambda: ([_rule(1, node="rtr-b")], "built-in", False)) + out = change_impact._policy_change_findings(object(), 1, {"rtr-a", "rtr-b"}, {"rtr-a"}) + assert len(out["one_side"]) == 1 and "before configuration only" in out["one_side"][0] + + +# --- #402 item 3: say what was not compared ------------------------------------ + +def _policy(**sections): + return policy_module.load_policy({"device": "rtr-us5", **sections}) + + +AC_RULE = {"node": "rtr-us5", "description": "8443 must be denied", "filter": "acl_in", + "headers": {"dstPorts": "8443"}, "expected": "DENY", + "violation_severity": "high", "violation_summary": "8443 open"} + + +def test_rules_in_sections_change_impact_does_not_compare_are_named(monkeypatch): + policy_module.set_active_policy(_policy(access_control=[AC_RULE], + policy_compliance=[_rule(1)])) + try: + _, _, _, note = change_impact._policy_coverage() + finally: + policy_module.clear_active_policy() + assert note.startswith("NOT compared") + assert "1 access_control" in note + + +def test_a_policy_with_no_policy_compliance_rules_says_so_loudly(): + policy_module.set_active_policy(_policy(access_control=[AC_RULE])) + try: + rules, _, _, note = change_impact._policy_coverage() + finally: + policy_module.clear_active_policy() + assert rules == [] + assert "no policy_compliance rules" in note and "no rule of yours was checked" in note + + +# --- #402 item 5: the mutants a fake session can catch ------------------------- + +class _Session: + """Records every question _rule_breached asks, answers from a table.""" + + def __init__(self, breached_arms): + self.snapshots, self.asked, self.breached = [], [], breached_arms + self.current = None + self.q = SimpleNamespace(searchFilters=self._search) + + def set_snapshot(self, name): + self.current = name + self.snapshots.append(name) + + def _search(self, nodes, filters, action, headers): + self.asked.append((self.current, action, str(headers))) + rows = 1 if (self.current, str(headers)) in self.breached else 0 + return SimpleNamespace(answer=lambda **k: SimpleNamespace( + frame=lambda: SimpleNamespace(__len__=lambda s: rows, iloc=[None] * rows))) + + +def test_rule_breached_switches_to_the_snapshot_it_is_asked_about(monkeypatch): + calls = [] + monkeypatch.setattr(pc, "_search", lambda bf, n, f, a, h: calls.append((bf.current, a)) or []) + s = _Session(set()) + change_impact._rule_breached(s, change_impact.AFTER_SNAPSHOT, _rule(1)) + assert s.snapshots == [change_impact.AFTER_SNAPSHOT] + assert calls[0][0] == change_impact.AFTER_SNAPSHOT + + +@pytest.mark.parametrize("kind", ["prohibition", "requirement"]) +def test_rule_breached_asks_with_the_checks_own_action_for_the_kind(monkeypatch, kind): + seen = [] + monkeypatch.setattr(pc, "_search", lambda bf, n, f, a, h: seen.append(a) or []) + change_impact._rule_breached(_Session(set()), change_impact.BEFORE_SNAPSHOT, _rule(1, kind=kind)) + assert seen == [pc.ACTION_BY_KIND[kind]] + + +def test_rule_breached_asks_every_arm_not_just_the_first(monkeypatch): + """Survived without Batfish before: only the second arm breaches.""" + r = _rule(1) + r["queries"] = [{"dstIps": "10.20.0.5"}, {"dstIps": "10.20.0.6"}] + monkeypatch.setattr(pc, "_search", lambda bf, n, f, a, h: [1] if h["dstIps"] == "10.20.0.6" else []) + assert change_impact._rule_breached(_Session(set()), change_impact.BEFORE_SNAPSHOT, r) is True + + +def test_the_policy_passed_in_is_the_policy_evaluated(monkeypatch): + """Survived both with and without Batfish: set_active_policy(None) in + place of set_active_policy(policy).""" + mine = _policy(policy_compliance=[_rule(7)]) + seen = [] + + def stage(bf, n, b=None, a=None): + seen.append(policy_module.active_policy()) + return {"findings": [], "number": n} + _pipeline(monkeypatch, stage=stage) + change_impact.analyse_change("before", "after", policy=mine) + assert seen == [mine] + assert policy_module.active_policy() is None, "and it is cleared afterwards" + + +# --- #402 item 6: stages are independent, ids cannot collide ------------------- + +def test_the_policy_stage_runs_when_reachability_fails(monkeypatch): + ran = [] + + def stage(bf, n, b=None, a=None): + ran.append(True) + return {"findings": [_found(n)], "number": n + 1} + _pipeline(monkeypatch, flows_raise=True, stage=stage) + results = change_impact.analyse_change("before", "after") + assert ran == [True] + assert {f["status"] for f in results} == {"found", "error"} + + +def test_a_long_diff_cannot_collide_with_the_reserved_ids(monkeypatch): + """60 filter findings used to be numbered CH-001..CH-060, two of which + are this module's own error cards.""" + _pipeline(monkeypatch, filters=[_found(n) for n in range(1, 61)], flows_raise=True) + results = change_impact.analyse_change("before", "after") + ids = [f["id"] for f in results] + assert len(ids) == len(set(ids)), ids + assert "CH-049" in ids and "further change" in next(f for f in results if f["id"] == "CH-049")["summary"] + assert max(int(i[3:]) for i in ids if i != "CH-049" and not i.startswith("CH-05")) <= 48 + + +def test_the_change_impact_cli_prints_every_policy_note(monkeypatch, capsys, tmp_path): + p = tmp_path / "p.json" + p.write_text('{"device": "rtr-us5", "policy_compliance": [{"description": "d", ' + '"kind": "prohibition", "filter": "acl_in", "violation_severity": "high", ' + '"violation_summary": "s", "queries": [{"dstIps": "10.20.0.5"}]}]}') + monkeypatch.setattr(change_impact, "analyse_change", lambda *a, **k: []) + monkeypatch.setattr("sys.argv", ["change_impact", "b", "a", str(p)]) + change_impact.main() + assert any(line.startswith("note:") for line in capsys.readouterr().out.splitlines()), \ + "the assigned rule number is a note the CLI used to drop" + + +# --- live, against real Batfish --------------------------------------------- + +@needs_batfish +def test_live_removing_a_device_is_reported(tmp_path): + """#403's own repro: multi-device-10 minus rtr-dev009.""" + after = tmp_path / "after" + shutil.copytree(FIXTURES / "multi-device-10", after) + (after / "configs" / "rtr-dev009.cfg").unlink() + results = change_impact.analyse_change(FIXTURES / "multi-device-10", after) + assert not any("No change detected" in f["summary"] for f in results) + assert any(f["device"] == "rtr-dev009" and "removed" in f["summary"] for f in results) + + +@needs_batfish +def test_live_a_stranger_network_proposal_is_verified(tmp_path): + """#402 item 1's repro: rtr-us5 renamed edge-fw1. Before this, every + built-in rule came back as CH-051 and verified was False.""" + for side in ("before", "after"): + d = tmp_path / side / "configs" + d.mkdir(parents=True) + text = (FIXTURES / "rtr-us5-secure" / "configs" / "rtr-us5.cfg").read_text() + d.joinpath("edge-fw1.cfg").write_text(text.replace("hostname rtr-us5", "hostname edge-fw1")) + results = change_impact.analyse_change(tmp_path / "before", tmp_path / "after") + assert all(f["status"] != "error" for f in results), [f["summary"] for f in results] From c636ee2d43d8684322caf8bf4ba6db8110998b67 Mon Sep 17 00:00:00 2001 From: Shubham Date: Thu, 29 Oct 2026 18:17:16 +1300 Subject: [PATCH 2/4] Give a proposed change a size, beside its findings (closes #420) change_impact.score(findings) returns one number, its band and the breakdown that produced it, computed only from status and severity -- never from a summary's wording. Opening counts more than closing because the severities already encode direction. Any error finding makes the band 'incomplete', so a change that could not be fully compared never reads as small. Returned beside the findings, not inside them (see #405). Live: secure->insecure large (14), insecure->secure moderate (7), no change none (0), one router removed small (2). Stacked on #406. --- analysis/change_impact.py | 51 ++++++++++++++++++++++++++++++++++++++ tests/test_change_score.py | 35 ++++++++++++++++++++++++++ 2 files changed, 86 insertions(+) create mode 100644 tests/test_change_score.py diff --git a/analysis/change_impact.py b/analysis/change_impact.py index 16cf13b..6f3b21a 100644 --- a/analysis/change_impact.py +++ b/analysis/change_impact.py @@ -811,6 +811,54 @@ def blind(n: int, summary: str, detail: str) -> None: return results +# --- How big is this change? (#420) ------------------------------------------- +# +# One number beside the findings, so a reviewer gets the size of a change +# before reading every card. Three rules, each from #420's criteria: +# +# COMPUTED, AND SHOWN. The score is a weighted count of the findings above, +# and the breakdown is returned with it, so it can always be explained. It +# reads only `status` and `severity` -- structured fields -- never a +# summary's wording, which this project has been burnt by twice (#145, +# #205). +# +# OPENING COUNTS MORE THAN CLOSING. Not a second model: the severities +# above already encode it (_OPENING_SEVERITY high, _CLOSING_SEVERITY +# medium; a broken policy rule carries the rule's own severity), so +# weighting by severity carries it through. +# +# INCOMPLETE NEVER SCORES SMALL. A change that could not be fully compared +# has an unknown size, and a low number would read as reassurance. Any +# `error` finding makes the band "incomplete", whatever the number. +# +# Returned beside the findings, never inside them: it is not an F-1 field +# (see the open question on #405 about adding keys to findings). + +SCORE_WEIGHTS = {"high": 3, "medium": 2, "low": 1} + +#: (lowest score, band). A score of 0 is "none" only when nothing was blind. +SCORE_BANDS = ((10, "large"), (4, "moderate"), (1, "small"), (0, "none")) + + +def score(results: List[Dict[str, Any]]) -> Dict[str, Any]: + """The size of a change, from analyse_change()'s findings.""" + found = [f for f in results if f.get("status") == "found"] + blind = [f for f in results if f.get("status") == "error"] + breakdown = {sev: sum(1 for f in found if f.get("severity") == sev) for sev in SCORE_WEIGHTS} + total = sum(SCORE_WEIGHTS[sev] * n for sev, n in breakdown.items()) + band = next(name for floor, name in SCORE_BANDS if total >= floor) + if blind: + band = "incomplete" + parts = [f"{n} {sev} x{SCORE_WEIGHTS[sev]}" for sev, n in breakdown.items() if n] + explanation = (" + ".join(parts) + f" = {total}") if parts else "no changes found = 0" + if blind: + explanation += (f"; {len(blind)} part(s) could not be compared, so the real size is " + "unknown and may be larger") + return {"score": total, "band": band, "complete": not blind, + "breakdown": dict(breakdown, could_not_compare=len(blind)), + "explanation": explanation} + + def main() -> None: if len(sys.argv) not in (3, 4): sys.exit( @@ -826,6 +874,9 @@ def main() -> None: print(f"note: {note}") results = analyse_change(sys.argv[1], sys.argv[2], policy=policy) print(json.dumps(results, indent=2)) + size = score(results) + print() + print(f" size {size['band']} ({size['explanation']})") print() for status in ("found", "none", "error"): print(f" {status:<6} {sum(1 for f in results if f['status'] == status)}") diff --git a/tests/test_change_score.py b/tests/test_change_score.py new file mode 100644 index 0000000..0e6aab1 --- /dev/null +++ b/tests/test_change_score.py @@ -0,0 +1,35 @@ +"""How big is a change? (#420)""" +from analysis import change_impact, findings + + +def f(sev, status="found", n=1): + if status == "error": + return findings.error_finding(check="change_impact", summary="s", detail="d", source="x", number=50) + return findings.make_finding(check="change_impact", severity=sev, device="r", summary="s", + detail="d", source="x", status=status, number=n) + + +def test_opening_counts_more_than_closing(): + assert change_impact.score([f("high")])["score"] > change_impact.score([f("medium")])["score"] + + +def test_the_breakdown_explains_the_number(): + s = change_impact.score([f("high"), f("medium", n=2), f("low", n=3)]) + assert s["score"] == 6 and s["explanation"] == "1 high x3 + 1 medium x2 + 1 low x1 = 6" + + +def test_incomplete_never_scores_small(): + s = change_impact.score([f("low"), f(None, "error")]) + assert s["band"] == "incomplete" and s["complete"] is False + assert "may be larger" in s["explanation"] + + +def test_no_change_is_zero_and_none(): + s = change_impact.score([f("low", "none", 0)]) + assert s["score"] == 0 and s["band"] == "none" and s["complete"] + + +def test_bands(): + assert change_impact.score([f("low")])["band"] == "small" + assert change_impact.score([f("high"), f("high", n=2)])["band"] == "moderate" + assert change_impact.score([f("high", n=i) for i in range(1, 5)])["band"] == "large" From 641df5a54875eceec27858dea9422753be3fd769 Mon Sep 17 00:00:00 2001 From: Shubham Date: Wed, 4 Nov 2026 11:09:15 +1300 Subject: [PATCH 3/4] Rate a still-breached rule by which way its breach moved (#406 review) Arsh's three blockers on #406: 1. A pure deny raised the "opens access" banner. _breach_changed now returns (grew, shrank) from the differential frame's two verdicts; a breach that only shrinks is "narrows", low, and only a breach that grows keeps the rule's own severity. Batfish returns one example per direction, measured on a change that both adds and removes breaching flows. 2. Device presence was case-sensitive, so policy device RTR-US5 against Batfish's rtr-us5 filed a real breach under a tick. Both sides are now casefolded, and a user's own rule about a device in neither config is part of CH-051, as a scan calls it PC-050. 3. CH-051 was only ever built by the test helper. The helper now returns the stage's own output, and CH-051 is tested through analyse_change(). Also from the review: tests that _breach_changed passes the reference snapshot and asks every arm, a test that a failed compareFilters is CH-050 and never "No change detected", _cap keeps the most severe cards (policy first at equal severity) instead of the first 48, the FINDING IDS block, and the #389 credit in CLAUDE.md. Arsh's two live reproductions are now live tests. --- CLAUDE.md | 2 +- analysis/change_impact.py | 113 ++++++++++---- docs/policy-rules.md | 15 +- tests/test_change_against_policy.py | 225 ++++++++++++++++++++++++++-- 4 files changed, 312 insertions(+), 43 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 2516080..97f52be 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -723,7 +723,7 @@ project's recurring failure family arriving through process rather than code. | **`risk` scoring** | Samika | Done (#60) — `POST_PROCESSORS`, ruleset in `docs/severity-rules.md`. Re-rates severity and sorts worst-first; the two limits are enforced in `pipeline.run_post_processors()`, not trusted | | Dashboard + secure upload | Samika | Done — real findings on screen since #39 | | PF Sense conversion | Ankeet | Done — `analysis/pfsense_convert.py`, and **hardened**: refuses config injection and path traversal via free-text fields (#53), an unbound ACL / empty rule set / unvalidated addressing (#54), and ambiguous rule order (#58, closing #47). See §7 | -| **`change_impact`** | Shubham | Done (#140, merged 17 August) — `analysis/change_impact.py`, `analyse_change(before, after)`. Shape C: **not** in `CHECKS`, and both the module docstring and the registry comment say so. Pairs `compareFilters` (which ACL *lines* moved) with `differentialReachability` (which *traffic* changed fate), because a changed line that moves no traffic is noise and moved traffic with no changed line is the case a filter diff alone misses. Direction-aware per `docs/policy-rules.md`: opening `high`, tightening `medium`, both reported. Since #323 (merged 14 October) it also runs every `policy_compliance` rule before and after and reports which the change breaks, fixes or alters; since #403/#402 it reports devices added or removed, treats rules about a device in neither config as not applicable, and says which policy sections it did **not** compare | +| **`change_impact`** | Shubham | Done (#140, merged 17 August) — `analysis/change_impact.py`, `analyse_change(before, after)`. Shape C: **not** in `CHECKS`, and both the module docstring and the registry comment say so. Pairs `compareFilters` (which ACL *lines* moved) with `differentialReachability` (which *traffic* changed fate), because a changed line that moves no traffic is noise and moved traffic with no changed line is the case a filter diff alone misses. Direction-aware per `docs/policy-rules.md`: opening `high`, tightening `medium`, both reported. Since #389 (closing #323, merged 14 October) it also runs every `policy_compliance` rule before and after and reports which the change breaks, fixes or alters; since #403/#402 it reports devices added or removed, treats rules about a device in neither config as not applicable, and says which policy sections it did **not** compare | | **A user policy reaching a check** | Arsh | Done (#181, merged 28 August) — `analysis/policy.py` loads and validates it; `policy_compliance` reads it via `rules_in_use()`, keyed on `active_policy() is None` rather than on emptiness. Measured below: policy-driven detections on a stranger's network go 3 → 8 | | **Draft a policy from your own config** | Arsh | Done (#326) — `analysis/draft_policy.py`, `/api/policy/draft`, a button under the policy picker. One `policy_compliance` rule per line of each applied filter, citing the file and line Batfish records for it. Each rule is checked with the same `searchFilters` question the check will ask, so a line an earlier line partly decides is listed as not drafted, with Batfish's example packet, rather than copied. `kind` and `violation_severity` are left as ``, and `analysis/policy.py` refuses the file until they are decided and its `draft` block is deleted: accepted as it stands, a draft passes on `rtr-us5-insecure` | | **What Netwise could not read** | Arsh | Done (#317) — `analysis/parse_coverage.py`. A partly parsed file's "Results may be incomplete" finding now says how many configuration lines Batfish did not recognise, by kind and line number, from Batfish's own `parseWarning`. On `vendor-asa`: 5 of 16, including the `access-group` line behind #373. Comment and blank lines are left out of the count only where the format's marker is known; "31 of 36" would have overstated it. Lives in the finding's detail, so it reaches every surface without an F-1 change | diff --git a/analysis/change_impact.py b/analysis/change_impact.py index b859c38..875a3d6 100644 --- a/analysis/change_impact.py +++ b/analysis/change_impact.py @@ -86,8 +86,10 @@ def analyse_change(before_dir, after_dir, host=None, policy=None) -> list[dict] change_impact owns the `CH-` prefix outright since amendment A-2 (#102). CH-000 nothing changed -- the clean sentinel - CH-001+ one change, numbered in the order found + CH-001+ one change, numbered in the order found, at most 48 + CH-049 the rest, summarised -- when there are more than 48 CH-050 could not compare -- the check-level error sentinel + CH-051+ one stage could not run -- see WHAT IT REPORTS below CH-050 mirrors policy_compliance's PC-050 deliberately: SENTINEL_NUMBER + ERROR_NUMBER_OFFSET, so there is one pattern across both of my features @@ -418,21 +420,30 @@ def _rule_label(rule: Dict[str, Any]) -> str: return f"{head} ({rule['description']})" if rule.get("description") else head -def _breach_changed(bf: Any, rule: Dict[str, Any]) -> bool: - """For a rule breached before AND after: did the change alter WHICH - traffic breaches it? (#402 item 4) +def _breach_changed(bf: Any, rule: Dict[str, Any]) -> Tuple[bool, bool]: + """For a rule breached before AND after: (did the breach grow, did it shrink)? - "Breached on both sides" is not "unchanged". Closing LAN->8.8.8.8 while - opening LAN->1.1.1.1 leaves a "no other destinations" rule breached - throughout, and the old wording called that "neither causes nor fixes". - Differential searchFilters answers it directly: rows are flows whose - verdict differs between the two snapshots. + "Breached on both sides" is not "unchanged" (#402 item 4). Closing + LAN->8.8.8.8 while opening LAN->1.1.1.1 leaves a "no other destinations" + rule breached throughout. Differential searchFilters answers it: its rows + are flows whose verdict differs between the two snapshots. + + WHICH WAY, NOT ONLY WHETHER (#406 review, @ARSH871-bot). The first version + returned one boolean, so a pure deny -- a breach that only SHRANK -- was + rated at the rule's own severity and raised the "opens access" banner. + Each row carries both verdicts, and Batfish returns one example per + direction (measured: a change that both adds and removes breaching + traffic gives two rows, one each way). A flow now taking the breaching + action, which it did not take before, grows the breach; the reverse + shrinks it. """ from pybatfish.datamodel.flow import HeaderConstraints from analysis.checks import policy_compliance as pc action = pc.ACTION_BY_KIND[rule["kind"]] + breaching = action.upper() + grew = shrank = False for headers in rule["queries"]: frame = ( bf.q.searchFilters( @@ -442,9 +453,11 @@ def _breach_changed(bf: Any, rule: Dict[str, Any]) -> bool: .answer(snapshot=AFTER_SNAPSHOT, reference_snapshot=BEFORE_SNAPSHOT) .frame() ) - if len(frame): - return True - return False + for _, row in frame.iterrows(): + now, then = str(row["Snapshot_Action"]).upper(), str(row["Reference_Action"]).upper() + grew = grew or (now == breaching and then != breaching) + shrank = shrank or (then == breaching and now != breaching) + return grew, shrank def _devices(bf: Any) -> "Tuple[set | None, set | None]": @@ -529,29 +542,43 @@ def _policy_change_findings(bf: Any, number: int, before: "set | None" = None, not_compared rules we could not ask -- never "unchanged" (F-4) coverage what this stage did not compare at all """ - rules, policy_label, _, coverage = _policy_coverage() + rules, policy_label, user_supplied, coverage = _policy_coverage() out: Dict[str, Any] = {"findings": [], "unchanged": [], "absent": [], "one_side": [], "not_compared": [], "coverage": coverage, "label": policy_label, "number": number} + # Casefolded on both sides (#406 review, item 2). Batfish names nodes in + # lower case and resolves a policy's `RTR-US5` to `rtr-us5`, so a + # case-sensitive test called a present device absent -- and filed a + # breach the change really caused under a tick. + before = {d.casefold() for d in before} if before is not None else None + after = {d.casefold() for d in after} if after is not None else None for rule in rules: node = rule["node"] + key = str(node).casefold() if before is not None and after is not None: - if node not in before and node not in after: - # Worded like PC-050 (#402 item 1). This change cannot affect a - # rule about a device in neither configuration: a proven - # "not applicable", not a blind spot, so not an error -- an - # error here made every proposal on a network without rtr-us5 - # "could not be fully verified". + if key not in before and key not in after: + if user_supplied: + # The user's own rule. A scan of the same files calls it + # PC-050, "could not check", so a tick here would + # disagree with the scan (#406 review, item 2). + out["not_compared"].append( + f"{_rule_label(rule)}: {node} is in neither configuration") + continue + # Our built-in example rules, worded like PC-050 (#402 item 1). + # A rule we wrote about a device the user does not have is a + # proven "not applicable", not a blind spot: an error here + # made every proposal on a network without rtr-us5 "could not + # be fully verified". out["absent"].append(f"{rule.get('number', '?')} ({node})") continue - if (node in before) != (node in after): - side = "before" if node in before else "after" + if (key in before) != (key in after): + side = "before" if key in before else "after" out["one_side"].append(f"{_rule_label(rule)}: {node} is in the {side} configuration only") continue try: was = _rule_breached(bf, BEFORE_SNAPSHOT, rule) now = _rule_breached(bf, AFTER_SNAPSHOT, rule) - altered = _breach_changed(bf, rule) if (was and now) else False + grew, shrank = _breach_changed(bf, rule) if (was and now) else (False, False) except Exception as error: out["not_compared"].append(f"{_rule_label(rule)}: {findings.describe_error(error)}") continue @@ -562,12 +589,27 @@ def _policy_change_findings(bf: Any, number: int, before: "set | None" = None, where = f"{rule['kind']} on {node} filter {rule['filter']}" if was and now: - if altered: + if grew and shrank: summary = f"This change alters how {_rule_label(rule)} is breached" detail = (f"{where}: breached before and after, but by different " - "traffic -- the change moves which flows break the rule. " - "Direction: changed.") + "traffic -- the change adds some breaching flows and " + "removes others. Direction: changed.") + severity = rule.get("violation_severity", "high") + elif grew: + summary = f"This change widens how {_rule_label(rule)} is breached" + detail = (f"{where}: breached before, and by more traffic after. " + f"Direction: {_RULE_DIRECTION[(rule['kind'], True)]}. " + f"Violation if applied: {rule.get('violation_summary')}.") severity = rule.get("violation_severity", "high") + elif shrank: + # A shrinking breach is a tightening of the breach, not a new + # problem: the rule's own severity here raised "opens access" + # for a pure deny (#406 review, item 1). + summary = f"This change narrows how {_rule_label(rule)} is breached" + detail = (f"{where}: breached before, and by less traffic after, but " + "still breached. Direction: " + f"{_RULE_DIRECTION[(rule['kind'], False)]}.") + severity = "low" else: summary = f"{_rule_label(rule)} is breached before and after this change" detail = (f"{where}: breached the same way on both sides. This change " @@ -603,11 +645,28 @@ def _policy_change_findings(bf: Any, number: int, before: "set | None" = None, def _cap(regular: List[Dict[str, Any]]) -> List[Dict[str, Any]]: - """Keep the first 48 ordinary findings; fold the rest into CH-049.""" + """Keep 48 ordinary findings, the most important ones; fold the rest into CH-049. + + WHICH 48 (#406 review). Keeping the first 48 kept them in the order the + stages ran, so the policy cards -- which run last and are the ones about + the user's own rules -- were the first to be folded away: 60 moved lines + plus a removed DNS line hid "BREAKS rule 4" inside "15 further + change(s)". Now the worst severity is kept first, a policy card ahead of + a filter or flow card of the same severity, and what is kept is still + shown in the order found. + """ if len(regular) < _OVERFLOW_NUMBER: return regular - kept, rest = regular[: _OVERFLOW_NUMBER - 1], regular[_OVERFLOW_NUMBER - 1:] order = {"high": 0, "medium": 1, "low": 2} + + def importance(index: int) -> Tuple[int, int, int]: + f = regular[index] + about_a_rule = "; rule " in str(f["evidence"]["source"]) + return order.get(f["severity"], 3), 0 if about_a_rule else 1, index + + keep = set(sorted(range(len(regular)), key=importance)[: _OVERFLOW_NUMBER - 1]) + kept = [f for i, f in enumerate(regular) if i in keep] + rest = [f for i, f in enumerate(regular) if i not in keep] worst = min((f["severity"] for f in rest), key=order.get) kept.append(findings.make_finding( check=CHECK_NAME, severity=worst, device="unknown", diff --git a/docs/policy-rules.md b/docs/policy-rules.md index a1dec9d..ddc5851 100644 --- a/docs/policy-rules.md +++ b/docs/policy-rules.md @@ -822,10 +822,19 @@ breach is), once against each snapshot: |---|---|---| | holds | holds | unchanged — *on the rule's filter*, which is all that is claimed | | breached | breached, by the same traffic | **still breached** — `found`, `low`: not caused by this edit, but never a green tick (#402) | -| breached | breached, by different traffic | **ALTERS** the rule — at the rule's own severity, from a differential `searchFilters` (#402) | +| breached | breached, by **more** traffic | **WIDENS** the breach — at the rule's own severity | +| breached | breached, by **less** traffic | **NARROWS** the breach — `low`, still breached, never the "opens access" banner (#406 review) | +| breached | breached, some flows added and some removed | **ALTERS** the rule — at the rule's own severity, direction *changed* | | breached | holds | **FIXES** the rule, `low` | | holds | breached | **BREAKS** the rule, at the rule's own severity | +The last three rows come from one differential `searchFilters`. Each row it +returns carries both verdicts, and Batfish returns one example per direction, +so a flow now taking the breaching action that did not before widens the +breach, and the reverse narrows it. The first version asked only *whether* +the breach moved and rated every move at the rule's own severity, so a pure +deny raised the "opens access" warning. + Direction uses change impact's existing terms: a prohibition breach created is *opening*, a requirement breach created is *tightening*, and the reverse for fixes. @@ -862,7 +871,9 @@ python -m analysis.change_impact before/ after/ my-policy.json ### What it does not look at, said out loud (#402, #403) - **Devices added or removed** are their own `found` cards. Before #403, deleting a whole router came back as "No change detected" with a green tick, because neither Batfish comparison describes a device that exists on one side only. -- **A rule about a device in neither configuration** is *not applicable* — this change cannot affect it — and is named in the result, not reported as an error. Reporting it as an error made every proposal on a network without `rtr-us5` "could not be fully verified". +- **One of our built-in rules about a device in neither configuration** is *not applicable* — this change cannot affect it — and is named in the result, not reported as an error. Reporting it as an error made every proposal on a network without `rtr-us5` "could not be fully verified". +- **One of the user's own rules about a device in neither configuration** is part of the `CH-051` error instead. A scan of the same files calls it `PC-050`, could not check, and the two must not disagree (#406 review). +- **Device names are compared as Batfish compares them**, case-free. A policy naming `RTR-US5` is about the device Batfish calls `rtr-us5`; a case-sensitive test once filed a real breach on it under a tick. - **A rule about a device on one side only** cannot be compared and is a single `CH-051` error. - **Only `policy_compliance` rules are compared.** A policy with `access_control` or `routing` rules gets a sentence beginning "NOT compared:" naming how many, and a policy with no `policy_compliance` rules says no rule of the user's was checked. - **"Unchanged" means the rule's filter decided its traffic the same way.** Unbinding the filter from its interface changes the network without changing that answer, so the wording claims no more than that. diff --git a/tests/test_change_against_policy.py b/tests/test_change_against_policy.py index 842fb07..303f9f5 100644 --- a/tests/test_change_against_policy.py +++ b/tests/test_change_against_policy.py @@ -32,14 +32,20 @@ def rule(number, kind, severity="high"): } -def stage(monkeypatch, rules, verdicts, altered=False, devices=None): +def stage(monkeypatch, rules, verdicts, breach=(False, False), devices=None, + user_supplied=True): """Run the policy stage with fixed rules and fixed per-snapshot verdicts. `verdicts` maps rule number -> (breached_before, breached_after), or to an - Exception to raise when that rule is asked. Returns (findings, unchanged, - number), plus the whole stage dict as a fourth value for newer tests. + Exception to raise when that rule is asked. `breach` is what + `_breach_changed` reports for a rule breached on both sides: (grew, + shrank). Returns (findings, unchanged, number), plus the whole stage dict. + + It returns the stage's own findings and nothing it makes up: CH-051 is + built by analyse_change(), so it is tested there, against the real code + (#406 review, item 3). """ - monkeypatch.setattr(pc, "rules_in_use", lambda: (rules, "test policy", True)) + monkeypatch.setattr(pc, "rules_in_use", lambda: (rules, "test policy", user_supplied)) def fake(bf, snapshot, r): v = verdicts[r["number"]] @@ -48,16 +54,11 @@ def fake(bf, snapshot, r): return v[0] if snapshot == change_impact.BEFORE_SNAPSHOT else v[1] monkeypatch.setattr(change_impact, "_rule_breached", fake) - monkeypatch.setattr(change_impact, "_breach_changed", lambda bf, r: altered) + monkeypatch.setattr(change_impact, "_breach_changed", lambda bf, r: breach) before, after = devices if devices else (None, None) out = change_impact._policy_change_findings(object(), 1, before, after) - errors = [f"{n}" for n in out["not_compared"]] - found = list(out["findings"]) - if errors: - found.append({"status": "error", "id": "CH-051", - "evidence": {"detail": "; ".join(out["not_compared"])}}) unchanged = [f"{n} (holds)" for n in out["unchanged"]] - return found, unchanged, out["number"], out + return list(out["findings"]), unchanged, out["number"], out # --- the four outcomes ----------------------------------------------------- @@ -110,13 +111,37 @@ def test_a_rule_breached_on_both_sides_is_never_a_green_tick(monkeypatch): def test_a_breach_that_moves_to_other_traffic_is_reported_as_altered(monkeypatch): """Closing LAN->8.8.8.8 while opening LAN->1.1.1.1 keeps a rule breached - throughout; the differential query says the traffic moved.""" - found, _, _, _ = stage(monkeypatch, [rule(2, "prohibition")], {2: (True, True)}, altered=True) + throughout; the differential query says traffic moved both ways.""" + found, _, _, _ = stage(monkeypatch, [rule(2, "prohibition")], {2: (True, True)}, + breach=(True, True)) assert "alters how" in found[0]["summary"] assert found[0]["severity"] == "high", "it is the rule's own severity" assert "Direction: changed" in found[0]["evidence"]["detail"] +@pytest.mark.parametrize("kind,direction", [("prohibition", "opening"), + ("requirement", "tightening")]) +def test_a_breach_that_only_grows_is_the_rules_own_severity(monkeypatch, kind, direction): + found, _, _, _ = stage(monkeypatch, [rule(2, kind)], {2: (True, True)}, + breach=(True, False)) + assert "widens how" in found[0]["summary"] + assert found[0]["severity"] == "high" + assert f"Direction: {direction}" in found[0]["evidence"]["detail"] + + +@pytest.mark.parametrize("kind,direction", [("prohibition", "tightening"), + ("requirement", "opening")]) +def test_a_breach_that_only_shrinks_is_low_never_high(monkeypatch, kind, direction): + """#406 review, item 1. A pure deny shrinks a prohibition's breach. Rated + at the rule's own severity, it raised the "opens access" banner.""" + found, _, _, _ = stage(monkeypatch, [rule(2, kind)], {2: (True, True)}, + breach=(False, True)) + assert "narrows how" in found[0]["summary"] + assert found[0]["severity"] == "low" + assert f"Direction: {direction}" in found[0]["evidence"]["detail"] + assert "still breached" in found[0]["evidence"]["detail"] + + @pytest.mark.parametrize("kind,before,after,direction", [ ("prohibition", False, True, "opening"), # forbidden traffic now permitted ("prohibition", True, False, "tightening"), @@ -148,6 +173,98 @@ def test_the_ids_this_stage_emits_are_unique(monkeypatch): assert len(ids) == len(set(ids)) +# --- device names: compared as Batfish compares them (#406 review, item 2) --- + +def test_a_device_named_in_capitals_is_still_the_device_batfish_found(monkeypatch): + r = dict(rule(1, "prohibition"), node="RTR-US5") + found, _, _, out = stage(monkeypatch, [r], {1: (False, True)}, + devices=({"rtr-us5"}, {"rtr-us5"})) + assert out["absent"] == [] and out["not_compared"] == [] + assert "BREAKS" in found[0]["summary"], "a breach the change caused, not a tick" + + +def test_the_comparison_is_case_free_from_either_side(monkeypatch): + """Batfish's names are lower case today; the rule is "compare as Batfish + resolves", not "trust that the set happens to be lower case".""" + found, _, _, out = stage(monkeypatch, [rule(1, "prohibition")], {1: (False, True)}, + devices=({"RTR-US5"}, {"RTR-US5"})) + assert out["absent"] == [] and out["not_compared"] == [] + assert "BREAKS" in found[0]["summary"] + + +def test_the_users_rule_about_a_device_in_neither_config_is_not_compared(monkeypatch): + """A scan calls this PC-050, could not check. A tick here would disagree.""" + r = dict(rule(1, "prohibition"), node="edge-fw9") + _, _, _, out = stage(monkeypatch, [r], {1: (False, False)}, + devices=({"rtr-us5"}, {"rtr-us5"}), user_supplied=True) + assert out["absent"] == [] + assert len(out["not_compared"]) == 1 and "edge-fw9" in out["not_compared"][0] + + +def test_our_example_rule_about_a_device_in_neither_config_is_not_applicable(monkeypatch): + """#402 item 1 still holds for the built-in rules: an error there made + every proposal on a network without rtr-us5 unverified.""" + r = dict(rule(1, "prohibition"), node="edge-fw9") + _, _, _, out = stage(monkeypatch, [r], {1: (False, False)}, + devices=({"rtr-us5"}, {"rtr-us5"}), user_supplied=False) + assert out["absent"] == ["1 (edge-fw9)"] and out["not_compared"] == [] + + +# --- _breach_changed reads both verdicts, from the real differential call --- + +class _Frame: + def __init__(self, rows): + self._rows = rows + + def iterrows(self): + return enumerate(self._rows) + + +class _DiffSession: + """Records every differential searchFilters call; answers per query.""" + + def __init__(self, rows_per_query): + self.calls = [] + rows = iter(rows_per_query) + session = self + + class _Q: + def searchFilters(self, **kwargs): + class _A: + def answer(self, **answer_kwargs): + session.calls.append(answer_kwargs) + frame = _Frame(next(rows)) + return type("R", (), {"frame": lambda self: frame})() + return _A() + self.q = _Q() + + +def _row(now, then): + return {"Snapshot_Action": now, "Reference_Action": then} + + +@pytest.mark.parametrize("kind,rows,expected", [ + ("prohibition", [_row("PERMIT", "DENY")], (True, False)), # newly permitted: grew + ("prohibition", [_row("DENY", "PERMIT")], (False, True)), # newly denied: shrank + ("prohibition", [_row("DENY", "PERMIT"), _row("PERMIT", "DENY")], (True, True)), + ("requirement", [_row("DENY", "PERMIT")], (True, False)), # newly denied: grew + ("requirement", [_row("PERMIT", "DENY")], (False, True)), + ("prohibition", [], (False, False)), +]) +def test_breach_changed_reads_which_way_each_flow_moved(kind, rows, expected): + bf = _DiffSession([rows]) + assert change_impact._breach_changed(bf, rule(1, kind)) == expected + assert bf.calls == [{"snapshot": change_impact.AFTER_SNAPSHOT, + "reference_snapshot": change_impact.BEFORE_SNAPSHOT}] + + +def test_breach_changed_asks_every_query_arm(): + r = dict(rule(1, "prohibition"), queries=[{"dstIps": "10.20.0.5"}, {"dstIps": "10.20.0.6"}]) + bf = _DiffSession([[], [_row("PERMIT", "DENY")]]) + assert change_impact._breach_changed(bf, r) == (True, False) + assert len(bf.calls) == 2 + + # --- how analyse_change() folds it in -------------------------------------- def _stub_pipeline(monkeypatch, *, filters=(), policy_stage=None): @@ -187,6 +304,53 @@ def test_with_real_changes_the_unchanged_rules_still_get_reported(monkeypatch): assert any(f["status"] == "none" and "3." in f["evidence"]["detail"] for f in results) +def test_rules_not_compared_become_a_real_ch051_error(monkeypatch): + """#406 review, item 3: the CH-051 card analyse_change() builds, not one + a test made up. Both lists reach it.""" + _stub_pipeline(monkeypatch, policy_stage=lambda bf, n, b=None, a=None: { + "findings": [], "unchanged": [], "number": n, + "not_compared": ["rule 1: searchFilters failed"], + "one_side": ["rule 2: rtr-b is in the after configuration only"]}) + results = change_impact.analyse_change("before", "after") + card = [f for f in results if f["id"] == "CH-051"] + assert len(card) == 1 and card[0]["status"] == "error" + assert "2 policy rule(s)" in card[0]["summary"] + assert "searchFilters failed" in card[0]["evidence"]["detail"] + assert "after configuration only" in card[0]["evidence"]["detail"] + assert not any(f["status"] == "none" and "No change detected" in f["summary"] for f in results) + + +def test_a_failed_filter_comparison_is_ch050_never_no_change(monkeypatch): + """#406 review: silencing this path turned a failed compareFilters into + a green "No change detected".""" + _stub_pipeline(monkeypatch, policy_stage=lambda bf, n, b=None, a=None: { + "findings": [], "unchanged": [], "number": n}) + + def broken(bf, n): + raise RuntimeError("compareFilters failed") + monkeypatch.setattr(change_impact, "_filter_change_findings", broken) + results = change_impact.analyse_change("before", "after") + assert [f["id"] for f in results if f["status"] == "error"] == ["CH-050"] + assert not any(f["status"] == "none" for f in results) + + +def test_a_long_diff_never_folds_away_a_broken_policy_rule(): + """#406 review: 60 moved lines plus a removed DNS line hid "BREAKS rule 4" + inside "15 further change(s)".""" + mk = change_impact.findings.make_finding + lines = [mk(check="change_impact", severity="high", device="rtr-us5", + summary=f"line {i}", detail="d", source="rtr-us5:acl_in", + status="found", number=i) for i in range(1, 61)] + breaks = mk(check="change_impact", severity="high", device="rtr-us5", + summary="This change BREAKS policy rule 4", detail="d", + source="your policy; rule 4", status="found", number=61) + kept = change_impact._cap(lines + [breaks]) + assert len(kept) == 49 and kept[-1]["id"] == "CH-049", "48 kept, plus the summary" + assert any("BREAKS policy rule 4" in f["summary"] for f in kept[:-1]) + ids = [f["id"] for f in kept[:-1]] + assert ids == sorted(ids), "what is kept stays in the order found" + + def test_the_policy_is_cleared_even_when_the_stage_raises(monkeypatch): """A policy outliving the call that supplied it would answer the next caller's question with someone else's rules.""" @@ -217,3 +381,38 @@ def test_live_a_swap_is_reported_as_both_a_fix_and_a_break(tmp_path): assert any("BREAKS policy rule 4" in s for s in summaries), summaries ids = [f["id"] for f in results] assert len(ids) == len(set(ids)) + + +@needs_batfish +def test_live_a_pure_deny_narrows_a_breach_and_raises_no_banner(): + """#406 review, item 1, through the public entry point named there.""" + from ai.propose import propose_change + out = propose_change("block 10.10.10.5 to 10.20.0.5 on tcp/22 on rtr-us5", + FIXTURES / "rtr-us5-insecure") + assert out["warning"] is False, out["answer"] + assert any("narrows how policy rule 2" in f["summary"] and f["severity"] == "low" + for f in out["impact"]) + assert not any(f["severity"] == "high" for f in out["impact"]) + + +@needs_batfish +def test_live_a_device_named_in_capitals_still_reports_the_breach(tmp_path): + """#406 review, item 2: `hostname RTR-US5`, policy `device: RTR-US5`, the + change removes DNS. Main found the breach; the first version of this + branch filed it under a tick.""" + secure = (FIXTURES / "rtr-us5-secure" / "configs" / "rtr-us5.cfg").read_text() + before = secure.replace("hostname rtr-us5", "hostname RTR-US5") + after = "\n".join(line for line in before.splitlines() if "eq domain" not in line) + "\n" + for name, text in (("before", before), ("after", after)): + (tmp_path / name / "configs").mkdir(parents=True) + (tmp_path / name / "configs" / "rtr-us5.cfg").write_text(text) + policy = policy_module.load_policy({"device": "RTR-US5", "policy_compliance": [{ + "number": 1, "description": "DNS to the approved resolver must work", + "kind": "requirement", "filter": "acl_in", "violation_severity": "high", + "violation_summary": "DNS blocked", + "queries": [{"srcIps": "10.10.10.0/24", "dstIps": "218.8.104.58", + "ipProtocols": ["udp"], "dstPorts": "53"}]}]}) + results = change_impact.analyse_change(tmp_path / "before", tmp_path / "after", + policy=policy) + assert any("BREAKS policy rule 1" in f["summary"] and f["severity"] == "high" + for f in results), [f["summary"] for f in results] From f00a3d6331f5f93c62e823cb6105f8b687b6b00a Mon Sep 17 00:00:00 2001 From: Shubham Date: Mon, 16 Nov 2026 13:28:39 +1300 Subject: [PATCH 4/4] Renumber a capped list so CH-049 and above stay reserved (#406 review) _cap keeps the most important 48, but each kept finding kept the number it was found under, so with mixed severities the 49th found was kept as CH-049, the id of the summary card, and later ones could sit on CH-050 to CH-054. The kept list is renumbered in the order found. Arsh's two tests: mixed severities give CH-001..048 plus 049, and five highs found after 55 mediums all survive the cap (the 'severity ignored' mutant that used to survive the suite now fails). --- analysis/change_impact.py | 7 ++++++ tests/test_change_against_policy.py | 34 +++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/analysis/change_impact.py b/analysis/change_impact.py index 875a3d6..3dd3ed2 100644 --- a/analysis/change_impact.py +++ b/analysis/change_impact.py @@ -667,6 +667,13 @@ def importance(index: int) -> Tuple[int, int, int]: keep = set(sorted(range(len(regular)), key=importance)[: _OVERFLOW_NUMBER - 1]) kept = [f for i, f in enumerate(regular) if i in keep] rest = [f for i, f in enumerate(regular) if i not in keep] + # A kept finding keeps the number it was found under, so the 49th found + # can sit here as CH-049 -- the id of the summary card below -- or on a + # reserved CH-050+ (#406 review, @ARSH871-bot). Renumber in the order + # found, so CH-001..048 are the ordinary cards and 049 is only ever the + # summary. Nothing outside this function relies on a kept card's id. + for new_number, f in enumerate(kept, start=1): + f["id"] = f"{findings.PREFIX_BY_CHECK[CHECK_NAME]}-{new_number:03d}" worst = min((f["severity"] for f in rest), key=order.get) kept.append(findings.make_finding( check=CHECK_NAME, severity=worst, device="unknown", diff --git a/tests/test_change_against_policy.py b/tests/test_change_against_policy.py index 303f9f5..90aa05c 100644 --- a/tests/test_change_against_policy.py +++ b/tests/test_change_against_policy.py @@ -351,6 +351,40 @@ def test_a_long_diff_never_folds_away_a_broken_policy_rule(): assert ids == sorted(ids), "what is kept stays in the order found" +def test_a_capped_list_is_numbered_one_to_forty_eight_whatever_was_kept(): + """The cap keeps the MOST IMPORTANT 48, not the first 48, and a kept + finding kept the number it was given when it was found. With mixed + severities the 49th finding found is a high one, is kept, and was CH-049: + the same id as the card that summarises the rest. The ids this module + reserves (CH-049 and CH-050 to CH-054) must be free whatever was kept, so + a capped list is renumbered in the order found. (#406 review: the test + above uses one severity, where the cap keeps the first 48, so it cannot + see this.)""" + mk = change_impact.findings.make_finding + found = [mk(check="change_impact", severity="high" if i % 2 == 1 else "medium", + device="rtr-us5", summary=f"line {i}", detail="d", + source="rtr-us5:acl_in", status="found", number=i) for i in range(1, 61)] + kept = change_impact._cap(found) + ids = [f["id"] for f in kept] + assert len(ids) == len(set(ids)), f"duplicate ids: {sorted(i for i in set(ids) if ids.count(i) > 1)}" + assert ids == [f"CH-{n:03d}" for n in range(1, 49)] + ["CH-049"] + nums = [int(f["summary"].split()[1]) for f in kept[:-1]] + assert nums == sorted(nums), "what is kept stays in the order found" + + +def test_a_capped_list_keeps_the_worst_severity_not_the_first_found(): + """55 medium lines found first, then 5 high ones: the highs must survive + the cap. Keeping the first 48 would fold all five away.""" + mk = change_impact.findings.make_finding + found = [mk(check="change_impact", severity="high" if i > 55 else "medium", + device="rtr-us5", summary=f"line {i}", detail="d", + source="rtr-us5:acl_in", status="found", number=i) for i in range(1, 61)] + kept = change_impact._cap(found) + kept_highs = [f for f in kept[:-1] if f["severity"] == "high"] + assert len(kept_highs) == 5, "every high finding is kept" + assert kept[-1]["severity"] == "medium", "the summary is as bad as the worst it hides" + + def test_the_policy_is_cleared_even_when_the_stage_raises(monkeypatch): """A policy outliving the call that supplied it would answer the next caller's question with someone else's rules."""