diff --git a/CLAUDE.md b/CLAUDE.md index 89eb1b1..a5db1bc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -725,7 +725,7 @@ project's recurring failure family arriving through process rather than code. | **`risk` scoring** | Samika | Done (#60) — `POST_PROCESSORS`, ruleset in `docs/severity-rules.md`. Re-rates severity and sorts worst-first; the two limits are enforced in `pipeline.run_post_processors()`, not trusted | | Dashboard + secure upload | Samika | Done — real findings on screen since #39 | | PF Sense conversion | Ankeet | Done — `analysis/pfsense_convert.py`, and **hardened**: refuses config injection and path traversal via free-text fields (#53), an unbound ACL / empty rule set / unvalidated addressing (#54), and ambiguous rule order (#58, closing #47). Also refuses an interface tag that is not a plain name (a namespaced tag injected config lines), and, per rule, ``, IPv6-only and ICMP-type rules; disabled rules are left out, as pfSense does. See §7 | -| **`change_impact`** | Shubham | Done (#140, merged 17 August) — `analysis/change_impact.py`, `analyse_change(before, after)`. Shape C: **not** in `CHECKS`, and both the module docstring and the registry comment say so. Pairs `compareFilters` (which ACL *lines* moved) with `differentialReachability` (which *traffic* changed fate), because a changed line that moves no traffic is noise and moved traffic with no changed line is the case a filter diff alone misses. Direction-aware per `docs/policy-rules.md`: opening `high`, tightening `medium`, both reported | +| **`change_impact`** | Shubham | Done (#140, merged 17 August) — `analysis/change_impact.py`, `analyse_change(before, after)`. Shape C: **not** in `CHECKS`, and both the module docstring and the registry comment say so. Pairs `compareFilters` (which ACL *lines* moved) with `differentialReachability` (which *traffic* changed fate), because a changed line that moves no traffic is noise and moved traffic with no changed line is the case a filter diff alone misses. Direction-aware per `docs/policy-rules.md`: opening `high`, tightening `medium`, both reported. Since #389 (closing #323, merged 14 October) it also runs every `policy_compliance` rule before and after and reports which the change breaks, fixes or alters; since #403/#402 it reports devices added or removed, treats rules about a device in neither config as not applicable, and says which policy sections it did **not** compare | | **A user policy reaching a check** | Arsh | Done (#181, merged 28 August) — `analysis/policy.py` loads and validates it; `policy_compliance` reads it via `rules_in_use()`, keyed on `active_policy() is None` rather than on emptiness. Measured below: policy-driven detections on a stranger's network go 3 → 8 | | **Draft a policy from your own config** | Arsh | Done (#326) — `analysis/draft_policy.py`, `/api/policy/draft`, a button under the policy picker. One `policy_compliance` rule per line of each applied filter, citing the file and line Batfish records for it. Each rule is checked with the same `searchFilters` question the check will ask, so a line an earlier line partly decides is listed as not drafted, with Batfish's example packet, rather than copied. `kind` and `violation_severity` are left as ``, and `analysis/policy.py` refuses the file until they are decided and its `draft` block is deleted: accepted as it stands, a draft passes on `rtr-us5-insecure` | | **What Netwise could not read** | Arsh | Done (#317) — `analysis/parse_coverage.py`. A partly parsed file's "Results may be incomplete" finding now says how many configuration lines Batfish did not recognise, by kind and line number, from Batfish's own `parseWarning`. On `vendor-asa`: 5 of 16, including the `access-group` line behind #373. Comment and blank lines are left out of the count only where the format's marker is known; "31 of 36" would have overstated it. Lives in the finding's detail, so it reaches every surface without an F-1 change | diff --git a/analysis/change_impact.py b/analysis/change_impact.py index 0f38339..3dd3ed2 100644 --- a/analysis/change_impact.py +++ b/analysis/change_impact.py @@ -12,7 +12,7 @@ CHECKS registry hands a check one Batfish session with one snapshot loaded. A differential question needs two, so `run(bf)` cannot express it. - def analyse_change(before_dir, after_dir) -> list[dict] + def analyse_change(before_dir, after_dir, host=None, policy=None) -> list[dict] It is called directly and is registered in nothing. DO NOT add it to CHECKS -- the pipeline carries a comment saying so, and this docstring is @@ -86,23 +86,44 @@ def analyse_change(before_dir, after_dir) -> list[dict] change_impact owns the `CH-` prefix outright since amendment A-2 (#102). CH-000 nothing changed -- the clean sentinel - CH-001+ one change, numbered in the order found + CH-001+ one change, numbered in the order found, at most 48 + CH-049 the rest, summarised -- when there are more than 48 CH-050 could not compare -- the check-level error sentinel + CH-051+ one stage could not run -- see WHAT IT REPORTS below CH-050 mirrors policy_compliance's PC-050 deliberately: SENTINEL_NUMBER + ERROR_NUMBER_OFFSET, so there is one pattern across both of my features rather than two conventions. See #101. WHAT THIS DOES NOT DO YET, STATED RATHER THAN IMPLIED - It diffs. It does not AUDIT the proposed state -- it will not tell you - that the "after" config violates POL-2, only that it changed. Running the + It diffs. Since #323 it also runs every policy_compliance rule before and + after, so it tells you which of those rules the change breaks, fixes, + leaves breached, or alters -- but it still does not run access_control or + routing over `after_dir`, and the result says so rather than implying + otherwise (#402). Running the producer checks over `after_dir` as well is open question 2 on the shapes document and is not mine to settle alone; I have argued for it on #61 and #86. Until then, "no change detected" means exactly that and must not be read as "the proposed config is fine". RUN IT - python -m analysis.change_impact + python -m analysis.change_impact [policy.json] + +WHAT IT REPORTS, IN ORDER (#403, #402) + devices added or removed found -- the largest change, which neither + Batfish comparison describes + filter lines that moved found compareFilters + traffic that changed fate found differentialReachability + policy rules broken / fixed / + altered / still breached found policy_compliance's own query, per side + a stage that could not run error CH-050 filter, CH-052 devices, + CH-053 reachability, CH-054 policy, + CH-051 rules not compared + more than 48 changes found CH-049, the rest summarised + nothing moved none CH-000, naming the rules it did not move + + Each stage runs whatever the others did, so a failure never hides what + another stage proved, and never hides that a stage did not run. """ import json @@ -399,78 +420,270 @@ def _rule_label(rule: Dict[str, Any]) -> str: return f"{head} ({rule['description']})" if rule.get("description") else head -def _policy_change_findings(bf: Any, number: int) -> Tuple[List[Dict[str, Any]], List[str], int]: - """Run every applicable policy rule against both snapshots. +def _breach_changed(bf: Any, rule: Dict[str, Any]) -> Tuple[bool, bool]: + """For a rule breached before AND after: (did the breach grow, did it shrink)? - Returns (findings, unchanged, number). `unchanged` is returned rather than - turned into a card here, because analyse_change() already owns the one - `CH-000` this module emits, and two of them would collide on id. + "Breached on both sides" is not "unchanged" (#402 item 4). Closing + LAN->8.8.8.8 while opening LAN->1.1.1.1 leaves a "no other destinations" + rule breached throughout. Differential searchFilters answers it: its rows + are flows whose verdict differs between the two snapshots. + + WHICH WAY, NOT ONLY WHETHER (#406 review, @ARSH871-bot). The first version + returned one boolean, so a pure deny -- a breach that only SHRANK -- was + rated at the rule's own severity and raised the "opens access" banner. + Each row carries both verdicts, and Batfish returns one example per + direction (measured: a change that both adds and removes breaching + traffic gives two rows, one each way). A flow now taking the breaching + action, which it did not take before, grows the breach; the reverse + shrinks it. """ + from pybatfish.datamodel.flow import HeaderConstraints + from analysis.checks import policy_compliance as pc - rules, policy_label, _ = pc.rules_in_use() - if not rules: - return [], [], number + action = pc.ACTION_BY_KIND[rule["kind"]] + breaching = action.upper() + grew = shrank = False + for headers in rule["queries"]: + frame = ( + bf.q.searchFilters( + nodes=rule["node"], filters=rule["filter"], action=action, + headers=HeaderConstraints(**headers), + ) + .answer(snapshot=AFTER_SNAPSHOT, reference_snapshot=BEFORE_SNAPSHOT) + .frame() + ) + for _, row in frame.iterrows(): + now, then = str(row["Snapshot_Action"]).upper(), str(row["Reference_Action"]).upper() + grew = grew or (now == breaching and then != breaching) + shrank = shrank or (then == breaching and now != breaching) + return grew, shrank + - results: List[Dict[str, Any]] = [] - unchanged: List[str] = [] - not_compared: List[str] = [] +def _devices(bf: Any) -> "Tuple[set | None, set | None]": + """The devices in each snapshot, or None for a side we could not read.""" + from analysis import snapshot + + out = [] + for name in (BEFORE_SNAPSHOT, AFTER_SNAPSHOT): + bf.set_snapshot(name) + out.append(snapshot.device_names(bf)) + return out[0], out[1] + + +def _device_change_findings(before: set, after: set, number: int) -> Tuple[List[Dict[str, Any]], int]: + """A device on one side only is a change, and the largest kind (#403). + + Neither compareFilters nor differentialReachability reports it: a filter + that no longer exists has nothing to compare against, so deleting a + whole router used to come back as "No change detected" with a green tick. + """ + results = [] + for name in sorted(before - after): + results.append(findings.make_finding( + check=CHECK_NAME, severity=_CLOSING_SEVERITY, device=name, + summary=f"Device {name} is removed by this change", + detail=(f"{name} is in the before configuration and not in the after one. " + "Every flow it carried changes, and the filter and reachability " + "comparisons below do not describe a device that exists on one " + "side only, so this card is the only record of it."), + source=f"{BEFORE_SNAPSHOT} -> {AFTER_SNAPSHOT}", status="found", number=number)) + number += 1 + for name in sorted(after - before): + results.append(findings.make_finding( + check=CHECK_NAME, severity=_OPENING_SEVERITY, device=name, + summary=f"Device {name} is added by this change", + detail=(f"{name} is in the after configuration and not in the before one. " + "Nothing about its traffic is compared below, because there is " + "nothing to compare it against -- run a scan of the after " + "configuration to check it."), + source=f"{BEFORE_SNAPSHOT} -> {AFTER_SNAPSHOT}", status="found", number=number)) + number += 1 + return results, number + + +def _policy_coverage() -> Tuple[List[Dict[str, Any]], str, bool, str]: + """(policy_compliance rules, label, user_supplied, sentence about the rest). + US-31 says "every policy rule". This stage compares policy_compliance + rules only, and the sentence says so out loud (#402 item 3) -- a user + whose policy is one access_control rule must not read "unchanged" as + "your rule still holds". + """ + from analysis.checks import policy_compliance as pc + + rules, label, user_supplied = pc.rules_in_use() + note = "" + active = policy_module.active_policy() + if active is not None: + skipped = [f"{len(active.entries_for(s))} {s}" + for s in ("access_control", "routing") if active.entries_for(s)] + if skipped: + note = ("NOT compared: " + " and ".join(skipped) + " rule(s) in your policy. " + "Change impact compares policy_compliance rules only; scan the " + "after configuration to check the others.") + if not rules: + note = ("NOT compared: your policy has no policy_compliance rules, so no " + "rule of yours was checked against this change. " + note).strip() + return rules, label, user_supplied, note + + +def _policy_change_findings(bf: Any, number: int, before: "set | None" = None, + after: "set | None" = None) -> Dict[str, Any]: + """Run every applicable policy_compliance rule against both snapshots. + + Returns a dict, not a finding list, because three of its outputs are + sentences that analyse_change() folds into the ONE card it owns: + + findings created, resolved, altered or still-breached rules + unchanged same verdict on the rule's filter, before and after + absent rules about a device in NEITHER configuration + one_side rules about a device in only ONE configuration + not_compared rules we could not ask -- never "unchanged" (F-4) + coverage what this stage did not compare at all + """ + rules, policy_label, user_supplied, coverage = _policy_coverage() + out: Dict[str, Any] = {"findings": [], "unchanged": [], "absent": [], "one_side": [], + "not_compared": [], "coverage": coverage, + "label": policy_label, "number": number} + # Casefolded on both sides (#406 review, item 2). Batfish names nodes in + # lower case and resolves a policy's `RTR-US5` to `rtr-us5`, so a + # case-sensitive test called a present device absent -- and filed a + # breach the change really caused under a tick. + before = {d.casefold() for d in before} if before is not None else None + after = {d.casefold() for d in after} if after is not None else None for rule in rules: + node = rule["node"] + key = str(node).casefold() + if before is not None and after is not None: + if key not in before and key not in after: + if user_supplied: + # The user's own rule. A scan of the same files calls it + # PC-050, "could not check", so a tick here would + # disagree with the scan (#406 review, item 2). + out["not_compared"].append( + f"{_rule_label(rule)}: {node} is in neither configuration") + continue + # Our built-in example rules, worded like PC-050 (#402 item 1). + # A rule we wrote about a device the user does not have is a + # proven "not applicable", not a blind spot: an error here + # made every proposal on a network without rtr-us5 "could not + # be fully verified". + out["absent"].append(f"{rule.get('number', '?')} ({node})") + continue + if (key in before) != (key in after): + side = "before" if key in before else "after" + out["one_side"].append(f"{_rule_label(rule)}: {node} is in the {side} configuration only") + continue try: - before = _rule_breached(bf, BEFORE_SNAPSHOT, rule) - after = _rule_breached(bf, AFTER_SNAPSHOT, rule) + was = _rule_breached(bf, BEFORE_SNAPSHOT, rule) + now = _rule_breached(bf, AFTER_SNAPSHOT, rule) + grew, shrank = _breach_changed(bf, rule) if (was and now) else (False, False) except Exception as error: - # A rule we could not ask is a blind spot, never "unchanged" (F-4). - not_compared.append(f"{_rule_label(rule)}: {findings.describe_error(error)}") + out["not_compared"].append(f"{_rule_label(rule)}: {findings.describe_error(error)}") continue - if before == after: - state = "still breached" if before else "holds" - unchanged.append(f"{rule.get('number', '?')} ({state})") + if not was and not now: + out["unchanged"].append(str(rule.get("number", "?"))) continue - created = after and not before - direction = _RULE_DIRECTION[(rule["kind"], created)] - if number >= ERROR_NUMBER_OFFSET: - not_compared.append(f"{_rule_label(rule)}: more rule changes than finding numbers") - continue - results.append( - findings.make_finding( - check=CHECK_NAME, - severity=(rule.get("violation_severity", "high") if created else "low"), - device=rule["node"], - summary=( - f"This change BREAKS {_rule_label(rule)}" - if created else - f"This change FIXES {_rule_label(rule)}" - ), - detail=( - f"{rule['kind']} on {rule['node']} filter {rule['filter']}: " - f"{'held before, breached after' if created else 'breached before, holds after'}. " - f"Direction: {direction}. " - + (f"Violation if applied: {rule.get('violation_summary')}." if created - else "The breach this rule reports is gone in the proposed config.") - ), - source=f"{policy_label}; rule {rule.get('number', '?')}", - status="found", - number=number, - ) - ) + where = f"{rule['kind']} on {node} filter {rule['filter']}" + if was and now: + if grew and shrank: + summary = f"This change alters how {_rule_label(rule)} is breached" + detail = (f"{where}: breached before and after, but by different " + "traffic -- the change adds some breaching flows and " + "removes others. Direction: changed.") + severity = rule.get("violation_severity", "high") + elif grew: + summary = f"This change widens how {_rule_label(rule)} is breached" + detail = (f"{where}: breached before, and by more traffic after. " + f"Direction: {_RULE_DIRECTION[(rule['kind'], True)]}. " + f"Violation if applied: {rule.get('violation_summary')}.") + severity = rule.get("violation_severity", "high") + elif shrank: + # A shrinking breach is a tightening of the breach, not a new + # problem: the rule's own severity here raised "opens access" + # for a pure deny (#406 review, item 1). + summary = f"This change narrows how {_rule_label(rule)} is breached" + detail = (f"{where}: breached before, and by less traffic after, but " + "still breached. Direction: " + f"{_RULE_DIRECTION[(rule['kind'], False)]}.") + severity = "low" + else: + summary = f"{_rule_label(rule)} is breached before and after this change" + detail = (f"{where}: breached the same way on both sides. This change " + "does not cause it and does not fix it; the violation stays.") + severity = "low" + else: + created = now and not was + direction = _RULE_DIRECTION[(rule["kind"], created)] + summary = (f"This change BREAKS {_rule_label(rule)}" if created + else f"This change FIXES {_rule_label(rule)}") + detail = (f"{where}: {'held before, breached after' if created else 'breached before, holds after'}. " + f"Direction: {direction}. " + + (f"Violation if applied: {rule.get('violation_summary')}." if created + else "The breach this rule reports is gone in the proposed config.")) + severity = rule.get("violation_severity", "high") if created else "low" + out["findings"].append(findings.make_finding( + check=CHECK_NAME, severity=severity, device=node, summary=summary, detail=detail, + source=f"{policy_label}; rule {rule.get('number', '?')}", status="found", + number=number)) number += 1 - - if not_compared: - results.append( - findings.error_finding( - check=CHECK_NAME, - device="unknown", - summary=f"{len(not_compared)} policy rule(s) could not be compared across the change", - detail="; ".join(not_compared), - source=policy_label, - number=SKIPPED_NUMBER + 1, - ) - ) - return results, unchanged, number + out["number"] = number + return out + + +#: Ordinary findings stop here. 49 and above are this module's own cards, so +#: a long diff can never collide with them (#402 item 6) -- the band +#: policy_compliance already uses (PC-049, PC-050). +_OVERFLOW_NUMBER = 49 +_RULES_NOT_COMPARED_NUMBER = SKIPPED_NUMBER + 1 # CH-051 +_DEVICES_UNKNOWN_NUMBER = SKIPPED_NUMBER + 2 # CH-052 +_REACHABILITY_FAILED_NUMBER = SKIPPED_NUMBER + 3 # CH-053 +_POLICY_FAILED_NUMBER = SKIPPED_NUMBER + 4 # CH-054 + + +def _cap(regular: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + """Keep 48 ordinary findings, the most important ones; fold the rest into CH-049. + + WHICH 48 (#406 review). Keeping the first 48 kept them in the order the + stages ran, so the policy cards -- which run last and are the ones about + the user's own rules -- were the first to be folded away: 60 moved lines + plus a removed DNS line hid "BREAKS rule 4" inside "15 further + change(s)". Now the worst severity is kept first, a policy card ahead of + a filter or flow card of the same severity, and what is kept is still + shown in the order found. + """ + if len(regular) < _OVERFLOW_NUMBER: + return regular + order = {"high": 0, "medium": 1, "low": 2} + + def importance(index: int) -> Tuple[int, int, int]: + f = regular[index] + about_a_rule = "; rule " in str(f["evidence"]["source"]) + return order.get(f["severity"], 3), 0 if about_a_rule else 1, index + + keep = set(sorted(range(len(regular)), key=importance)[: _OVERFLOW_NUMBER - 1]) + kept = [f for i, f in enumerate(regular) if i in keep] + rest = [f for i, f in enumerate(regular) if i not in keep] + # A kept finding keeps the number it was found under, so the 49th found + # can sit here as CH-049 -- the id of the summary card below -- or on a + # reserved CH-050+ (#406 review, @ARSH871-bot). Renumber in the order + # found, so CH-001..048 are the ordinary cards and 049 is only ever the + # summary. Nothing outside this function relies on a kept card's id. + for new_number, f in enumerate(kept, start=1): + f["id"] = f"{findings.PREFIX_BY_CHECK[CHECK_NAME]}-{new_number:03d}" + worst = min((f["severity"] for f in rest), key=order.get) + kept.append(findings.make_finding( + check=CHECK_NAME, severity=worst, device="unknown", + summary=f"{len(rest)} further change(s) not listed individually", + detail=("This change produced more findings than are listed one by one. " + "The first of the rest: " + "; ".join(f["summary"] for f in rest[:5]) + + (" ..." if len(rest) > 5 else "")), + source=f"{BEFORE_SNAPSHOT} -> {AFTER_SNAPSHOT}", status="found", + number=_OVERFLOW_NUMBER)) + return kept def analyse_change( @@ -489,14 +702,15 @@ def analyse_change( The three outcomes are kept distinct, which is the whole of F-4 here: found the change did something, and here is what - none both configs loaded, both questions ran, nothing moved + none both configs loaded, every question ran, nothing moved error we could not compare -- NOT the same as "nothing changed" That middle one is the dangerous claim. "This change is safe" is exactly what a user wants to hear before applying it to a firewall, so it may only - be said when both snapshots parsed and both questions actually ran. + be said when both snapshots parsed and every question actually ran. """ before_dir, after_dir = Path(before_dir), Path(after_dir) + where = f"{before_dir} -> {after_dir}" try: bf = connect(host) @@ -508,7 +722,7 @@ def analyse_change( "docker start batfish. " f"Underlying error: {findings.describe_error(error)}" ), - source=f"{before_dir} -> {after_dir}", + source=where, ) # Load both. Named snapshots in one network so the differential questions @@ -554,62 +768,79 @@ def analyse_change( source=str(folder), ) - results: List[Dict[str, Any]] = [] + regular: List[Dict[str, Any]] = [] + reserved: List[Dict[str, Any]] = [] number = 1 + def blind(n: int, summary: str, detail: str) -> None: + reserved.append(findings.error_finding( + check=CHECK_NAME, summary=summary, detail=detail, source=where, number=n)) + + # Which devices are on each side (#403). Unknown is an error, never "none". + before_devs = after_devs = None + try: + before_devs, after_devs = _devices(bf) + except Exception as error: + blind(_DEVICES_UNKNOWN_NUMBER, "Could not list the devices in each configuration", + findings.describe_error(error)) + if before_devs is None or after_devs is None: + if not any(f["id"].endswith(f"{_DEVICES_UNKNOWN_NUMBER:03d}") for f in reserved): + blind(_DEVICES_UNKNOWN_NUMBER, "Could not list the devices in each configuration", + "Batfish did not say which devices each configuration defines, so a " + "device added or removed by this change would not be reported.") + else: + device_changes, number = _device_change_findings(before_devs, after_devs, number) + regular.extend(device_changes) + + # Each stage runs whatever the others did (#402 item 6). What one proved + # stands when another fails, and the failure is said (#22). try: filter_changes, number = _filter_change_findings(bf, number) + regular.extend(filter_changes) except Exception as error: - return _error( - summary="Change impact could not run: the filter comparison failed", - detail=findings.describe_error(error), - source=f"{before_dir} -> {after_dir}", - ) - results.extend(filter_changes) + blind(SKIPPED_NUMBER, "The filter comparison could not run", findings.describe_error(error)) try: flow_changes, number = _reachability_change_findings(bf, number) + regular.extend(flow_changes) except Exception as error: - # The filter comparison may already have found real changes. Report - # BOTH -- what was proved AND what could not be reached. Discarding a - # proven finding because a second query failed is issue #22, and it - # fails in the reassuring direction. - results.extend( - _error( - summary="The reachability comparison could not run", - detail=findings.describe_error(error), - source=f"{before_dir} -> {after_dir}", - ) - ) - return results - results.extend(flow_changes) + blind(_REACHABILITY_FAILED_NUMBER, "The reachability comparison could not run", + findings.describe_error(error)) # Every policy rule, before and after (#323). Installed the same way # analyse() installs it and cleared in a `finally`, so a policy never # outlives the call that supplied it. policy_module.set_active_policy(policy) + stage: Dict[str, Any] = {} try: - rule_changes, unchanged, number = _policy_change_findings(bf, number) + stage = _policy_change_findings(bf, number, before_devs, after_devs) + regular.extend(stage["findings"]) except Exception as error: - # Same rule as the reachability stage: what was proved above stands, - # and the user is told what could not be checked (#22). - results.extend( - _error( - summary="The policy comparison could not run", - detail=findings.describe_error(error), - source=f"{before_dir} -> {after_dir}", - ) - ) - return results + blind(_POLICY_FAILED_NUMBER, "The policy comparison could not run", + findings.describe_error(error)) finally: policy_module.clear_active_policy() - results.extend(rule_changes) - unchanged_note = ( - " Policy rules with the same verdict before and after, rule (state): " - + ", ".join(unchanged) + "." - if unchanged else "" - ) + not_compared = stage.get("not_compared", []) + stage.get("one_side", []) + if not_compared: + blind(_RULES_NOT_COMPARED_NUMBER, + f"{len(not_compared)} policy rule(s) could not be compared across the change", + "; ".join(not_compared)) + + # The sentences about rules nothing above reported on. + notes = [] + if stage.get("unchanged"): + notes.append("Policy rule(s) with the same verdict on their filter before and " + "after (this says the filter's decision did not move, not that " + "nothing about their traffic did): " + ", ".join(stage["unchanged"]) + ".") + if stage.get("absent"): + notes.append("Policy rule(s) about devices in neither configuration, which this " + "change cannot affect: " + ", ".join(stage["absent"]) + ".") + if stage.get("coverage"): + notes.append(stage["coverage"]) + note_text = (" " + " ".join(notes)) if notes else "" + + results = _cap(regular) + reserved if not results: return [ @@ -618,32 +849,28 @@ def analyse_change( device="unknown", summary="No change detected between the two configurations", detail=( - "No filter line treats the same traffic differently, and no " - "flow reaches a destination it could not reach before. This " - "says the change moved nothing; it does not say the proposed " - "config is free of problems -- run the audit for that." - + unchanged_note + "No device was added or removed, no filter line treats the same " + "traffic differently, and no flow reaches a destination it could " + "not reach before. This says the change moved nothing; it does " + "not say the proposed config is free of problems -- run the " + "audit for that." + note_text ), - source=f"{before_dir} -> {after_dir}", + source=where, number=0, ) ] - if unchanged: - # US-31 asks for a direction per rule, and "unchanged" is one. Only - # emitted here, alongside real changes, so it can never be mistaken - # for the all-clear above -- and it takes CH-000 only because that - # all-clear is, by construction, not in this list. + if notes: + # Alongside real changes, so it can never be mistaken for the + # all-clear above -- and it takes CH-000 only because that all-clear + # is, by construction, not in this list. results.append( findings.no_issues_finding( check=CHECK_NAME, device="unknown", - summary=f"{len(unchanged)} policy rule(s) unchanged by this edit", - detail=( - "Same verdict before and after, so this change neither " - "causes nor fixes them." + unchanged_note - ), - source=f"{before_dir} -> {after_dir}", + summary="Policy rules this change did not move", + detail=note_text.strip(), + source=where, number=0, ) ) @@ -660,9 +887,13 @@ def main() -> None: policy = None if len(sys.argv) == 4: policy = policy_module.load_policy_file(sys.argv[3]) + # Every channel, like `python -m analysis.pipeline --policy` (#402 item 6). + for note in list(policy.renamed) + list(policy.assigned): + print(f"note: {note}") # Same contract as `python -m analysis.pipeline --policy` (#387 # review): a pair the contradiction check could not compare is said, - # not dropped. + # not dropped. Notes before warnings, matching pipeline.py's own + # established order. for note in policy.undecidable: print(f"warning: {note}") results = analyse_change(sys.argv[1], sys.argv[2], policy=policy) diff --git a/docs/policy-rules.md b/docs/policy-rules.md index 9ae8930..ddc5851 100644 --- a/docs/policy-rules.md +++ b/docs/policy-rules.md @@ -799,7 +799,16 @@ unapproved destination, and POL-2 arm A catches DNS to `10.20.0.5`. Verified: --- -### Change impact now checks every policy rule (#323, US-31) +## 7. Sign-off + +| Member | Role in this decision | Agreed | +|---|---|---| +| Shubham | Author; owns the check | ✅ | +| Arsh | Registry, pipeline, `findings.py` impact | ✅ rules, IDs, and the §4 `findings.py` change | +| Samika | Severity values are his | ✅ | +| Ankeet | Consumes findings in the AI layer | ⬜ | + +## 8. Change impact checks every policy rule (#323, #402, #403) `analyse_change(before, after)` used to answer one question: *what traffic moved*. It now also answers the one an engineer actually asks before applying @@ -811,11 +820,21 @@ breach is), once against each snapshot: | before | after | reported as | |---|---|---| -| holds | holds | unchanged | -| breached | breached | unchanged — still broken, not caused by this edit | +| holds | holds | unchanged — *on the rule's filter*, which is all that is claimed | +| breached | breached, by the same traffic | **still breached** — `found`, `low`: not caused by this edit, but never a green tick (#402) | +| breached | breached, by **more** traffic | **WIDENS** the breach — at the rule's own severity | +| breached | breached, by **less** traffic | **NARROWS** the breach — `low`, still breached, never the "opens access" banner (#406 review) | +| breached | breached, some flows added and some removed | **ALTERS** the rule — at the rule's own severity, direction *changed* | | breached | holds | **FIXES** the rule, `low` | | holds | breached | **BREAKS** the rule, at the rule's own severity | +The last three rows come from one differential `searchFilters`. Each row it +returns carries both verdicts, and Batfish returns one example per direction, +so a flow now taking the breaching action that did not before widens the +breach, and the reverse narrows it. The first version asked only *whether* +the breach moved and rated every move at the rule's own severity, so a pure +deny raised the "opens access" warning. + Direction uses change impact's existing terms: a prohibition breach created is *opening*, a requirement breach created is *tightening*, and the reverse for fixes. @@ -848,11 +867,14 @@ built-in rules, like a scan does. The CLI takes it as a third argument: python -m analysis.change_impact before/ after/ my-policy.json ``` -## 7. Sign-off -| Member | Role in this decision | Agreed | -|---|---|---| -| Shubham | Author; owns the check | ✅ | -| Arsh | Registry, pipeline, `findings.py` impact | ✅ rules, IDs, and the §4 `findings.py` change | -| Samika | Severity values are his | ✅ | -| Ankeet | Consumes findings in the AI layer | ⬜ | +### What it does not look at, said out loud (#402, #403) + +- **Devices added or removed** are their own `found` cards. Before #403, deleting a whole router came back as "No change detected" with a green tick, because neither Batfish comparison describes a device that exists on one side only. +- **One of our built-in rules about a device in neither configuration** is *not applicable* — this change cannot affect it — and is named in the result, not reported as an error. Reporting it as an error made every proposal on a network without `rtr-us5` "could not be fully verified". +- **One of the user's own rules about a device in neither configuration** is part of the `CH-051` error instead. A scan of the same files calls it `PC-050`, could not check, and the two must not disagree (#406 review). +- **Device names are compared as Batfish compares them**, case-free. A policy naming `RTR-US5` is about the device Batfish calls `rtr-us5`; a case-sensitive test once filed a real breach on it under a tick. +- **A rule about a device on one side only** cannot be compared and is a single `CH-051` error. +- **Only `policy_compliance` rules are compared.** A policy with `access_control` or `routing` rules gets a sentence beginning "NOT compared:" naming how many, and a policy with no `policy_compliance` rules says no rule of the user's was checked. +- **"Unchanged" means the rule's filter decided its traffic the same way.** Unbinding the filter from its interface changes the network without changing that answer, so the wording claims no more than that. + diff --git a/tests/test_change_against_policy.py b/tests/test_change_against_policy.py index 71a9a00..90aa05c 100644 --- a/tests/test_change_against_policy.py +++ b/tests/test_change_against_policy.py @@ -32,13 +32,20 @@ def rule(number, kind, severity="high"): } -def stage(monkeypatch, rules, verdicts): +def stage(monkeypatch, rules, verdicts, breach=(False, False), devices=None, + user_supplied=True): """Run the policy stage with fixed rules and fixed per-snapshot verdicts. `verdicts` maps rule number -> (breached_before, breached_after), or to an - Exception to raise when that rule is asked. + Exception to raise when that rule is asked. `breach` is what + `_breach_changed` reports for a rule breached on both sides: (grew, + shrank). Returns (findings, unchanged, number), plus the whole stage dict. + + It returns the stage's own findings and nothing it makes up: CH-051 is + built by analyse_change(), so it is tested there, against the real code + (#406 review, item 3). """ - monkeypatch.setattr(pc, "rules_in_use", lambda: (rules, "test policy", True)) + monkeypatch.setattr(pc, "rules_in_use", lambda: (rules, "test policy", user_supplied)) def fake(bf, snapshot, r): v = verdicts[r["number"]] @@ -47,13 +54,17 @@ def fake(bf, snapshot, r): return v[0] if snapshot == change_impact.BEFORE_SNAPSHOT else v[1] monkeypatch.setattr(change_impact, "_rule_breached", fake) - return change_impact._policy_change_findings(object(), 1) + monkeypatch.setattr(change_impact, "_breach_changed", lambda bf, r: breach) + before, after = devices if devices else (None, None) + out = change_impact._policy_change_findings(object(), 1, before, after) + unchanged = [f"{n} (holds)" for n in out["unchanged"]] + return list(out["findings"]), unchanged, out["number"], out # --- the four outcomes ----------------------------------------------------- def test_a_change_that_causes_a_breach_says_it_breaks_the_rule(monkeypatch): - found, unchanged, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (False, True)}) + found, unchanged, _, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (False, True)}) assert len(found) == 1 f = found[0] assert f["status"] == "found" @@ -63,7 +74,7 @@ def test_a_change_that_causes_a_breach_says_it_breaks_the_rule(monkeypatch): def test_a_change_that_repairs_a_breach_says_it_fixes_the_rule(monkeypatch): - found, _, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (True, False)}) + found, _, _, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (True, False)}) assert len(found) == 1 assert "FIXES" in found[0]["summary"] assert found[0]["severity"] == "low", "a fix is good news, not a high finding" @@ -71,7 +82,7 @@ def test_a_change_that_repairs_a_breach_says_it_fixes_the_rule(monkeypatch): def test_a_change_that_fixes_one_and_breaks_another_reports_both(monkeypatch): """US-31's own criterion, and the reason this stage exists.""" - found, _, _ = stage( + found, _, _, _ = stage( monkeypatch, [rule(1, "prohibition"), rule(4, "requirement", severity="medium")], {1: (True, False), 4: (False, True)}, @@ -81,15 +92,54 @@ def test_a_change_that_fixes_one_and_breaks_another_reports_both(monkeypatch): assert any("BREAKS" in s and "rule 4" in s for s in summaries) -def test_a_rule_with_the_same_verdict_either_side_is_unchanged_not_a_finding(monkeypatch): - found, unchanged, _ = stage( - monkeypatch, [rule(1, "prohibition"), rule(2, "prohibition")], - {1: (False, False), 2: (True, True)}, - ) +def test_a_rule_that_holds_either_side_is_unchanged_not_a_finding(monkeypatch): + found, unchanged, _, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (False, False)}) assert found == [] - assert unchanged == ["1 (holds)", "2 (still breached)"], ( - "a rule broken before AND after is unchanged -- this edit did not cause it" - ) + assert unchanged == ["1 (holds)"] + + +def test_a_rule_breached_on_both_sides_is_never_a_green_tick(monkeypatch): + """#402 item 4. "Still breached" sat inside the `none` card, which the + propose pane draws as a tick. A violation that survives the change is + still a violation.""" + found, unchanged, _, _ = stage(monkeypatch, [rule(2, "prohibition")], {2: (True, True)}) + assert unchanged == [] + assert len(found) == 1 and found[0]["status"] == "found" + assert "breached before and after" in found[0]["summary"] + assert found[0]["severity"] == "low", "not caused by this change" + + +def test_a_breach_that_moves_to_other_traffic_is_reported_as_altered(monkeypatch): + """Closing LAN->8.8.8.8 while opening LAN->1.1.1.1 keeps a rule breached + throughout; the differential query says traffic moved both ways.""" + found, _, _, _ = stage(monkeypatch, [rule(2, "prohibition")], {2: (True, True)}, + breach=(True, True)) + assert "alters how" in found[0]["summary"] + assert found[0]["severity"] == "high", "it is the rule's own severity" + assert "Direction: changed" in found[0]["evidence"]["detail"] + + +@pytest.mark.parametrize("kind,direction", [("prohibition", "opening"), + ("requirement", "tightening")]) +def test_a_breach_that_only_grows_is_the_rules_own_severity(monkeypatch, kind, direction): + found, _, _, _ = stage(monkeypatch, [rule(2, kind)], {2: (True, True)}, + breach=(True, False)) + assert "widens how" in found[0]["summary"] + assert found[0]["severity"] == "high" + assert f"Direction: {direction}" in found[0]["evidence"]["detail"] + + +@pytest.mark.parametrize("kind,direction", [("prohibition", "tightening"), + ("requirement", "opening")]) +def test_a_breach_that_only_shrinks_is_low_never_high(monkeypatch, kind, direction): + """#406 review, item 1. A pure deny shrinks a prohibition's breach. Rated + at the rule's own severity, it raised the "opens access" banner.""" + found, _, _, _ = stage(monkeypatch, [rule(2, kind)], {2: (True, True)}, + breach=(False, True)) + assert "narrows how" in found[0]["summary"] + assert found[0]["severity"] == "low" + assert f"Direction: {direction}" in found[0]["evidence"]["detail"] + assert "still breached" in found[0]["evidence"]["detail"] @pytest.mark.parametrize("kind,before,after,direction", [ @@ -99,25 +149,23 @@ def test_a_rule_with_the_same_verdict_either_side_is_unchanged_not_a_finding(mon ("requirement", True, False, "opening"), ]) def test_direction_is_stated_in_this_modules_own_terms(monkeypatch, kind, before, after, direction): - found, _, _ = stage(monkeypatch, [rule(1, kind)], {1: (before, after)}) + found, _, _, _ = stage(monkeypatch, [rule(1, kind)], {1: (before, after)}) assert f"Direction: {direction}" in found[0]["evidence"]["detail"] # --- F-4: could-not-ask is never unchanged ---------------------------------- def test_a_rule_that_cannot_be_asked_is_an_error_never_unchanged(monkeypatch): - found, unchanged, _ = stage( + found, unchanged, _, out = stage( monkeypatch, [rule(1, "prohibition"), rule(2, "prohibition")], {1: RuntimeError("searchFilters failed"), 2: (False, False)}, ) - errors = [f for f in found if f["status"] == "error"] - assert len(errors) == 1 - assert "rule 1" in errors[0]["evidence"]["detail"] + assert len(out["not_compared"]) == 1 and "rule 1" in out["not_compared"][0] assert unchanged == ["2 (holds)"], "the rule we could not ask must not be listed as holding" def test_the_ids_this_stage_emits_are_unique(monkeypatch): - found, _, _ = stage( + found, _, _, _ = stage( monkeypatch, [rule(n, "prohibition") for n in (1, 2, 3)] + [rule(9, "prohibition")], {1: (False, True), 2: (True, False), 3: (False, True), 9: RuntimeError("x")}, ) @@ -125,6 +173,98 @@ def test_the_ids_this_stage_emits_are_unique(monkeypatch): assert len(ids) == len(set(ids)) +# --- device names: compared as Batfish compares them (#406 review, item 2) --- + +def test_a_device_named_in_capitals_is_still_the_device_batfish_found(monkeypatch): + r = dict(rule(1, "prohibition"), node="RTR-US5") + found, _, _, out = stage(monkeypatch, [r], {1: (False, True)}, + devices=({"rtr-us5"}, {"rtr-us5"})) + assert out["absent"] == [] and out["not_compared"] == [] + assert "BREAKS" in found[0]["summary"], "a breach the change caused, not a tick" + + +def test_the_comparison_is_case_free_from_either_side(monkeypatch): + """Batfish's names are lower case today; the rule is "compare as Batfish + resolves", not "trust that the set happens to be lower case".""" + found, _, _, out = stage(monkeypatch, [rule(1, "prohibition")], {1: (False, True)}, + devices=({"RTR-US5"}, {"RTR-US5"})) + assert out["absent"] == [] and out["not_compared"] == [] + assert "BREAKS" in found[0]["summary"] + + +def test_the_users_rule_about_a_device_in_neither_config_is_not_compared(monkeypatch): + """A scan calls this PC-050, could not check. A tick here would disagree.""" + r = dict(rule(1, "prohibition"), node="edge-fw9") + _, _, _, out = stage(monkeypatch, [r], {1: (False, False)}, + devices=({"rtr-us5"}, {"rtr-us5"}), user_supplied=True) + assert out["absent"] == [] + assert len(out["not_compared"]) == 1 and "edge-fw9" in out["not_compared"][0] + + +def test_our_example_rule_about_a_device_in_neither_config_is_not_applicable(monkeypatch): + """#402 item 1 still holds for the built-in rules: an error there made + every proposal on a network without rtr-us5 unverified.""" + r = dict(rule(1, "prohibition"), node="edge-fw9") + _, _, _, out = stage(monkeypatch, [r], {1: (False, False)}, + devices=({"rtr-us5"}, {"rtr-us5"}), user_supplied=False) + assert out["absent"] == ["1 (edge-fw9)"] and out["not_compared"] == [] + + +# --- _breach_changed reads both verdicts, from the real differential call --- + +class _Frame: + def __init__(self, rows): + self._rows = rows + + def iterrows(self): + return enumerate(self._rows) + + +class _DiffSession: + """Records every differential searchFilters call; answers per query.""" + + def __init__(self, rows_per_query): + self.calls = [] + rows = iter(rows_per_query) + session = self + + class _Q: + def searchFilters(self, **kwargs): + class _A: + def answer(self, **answer_kwargs): + session.calls.append(answer_kwargs) + frame = _Frame(next(rows)) + return type("R", (), {"frame": lambda self: frame})() + return _A() + self.q = _Q() + + +def _row(now, then): + return {"Snapshot_Action": now, "Reference_Action": then} + + +@pytest.mark.parametrize("kind,rows,expected", [ + ("prohibition", [_row("PERMIT", "DENY")], (True, False)), # newly permitted: grew + ("prohibition", [_row("DENY", "PERMIT")], (False, True)), # newly denied: shrank + ("prohibition", [_row("DENY", "PERMIT"), _row("PERMIT", "DENY")], (True, True)), + ("requirement", [_row("DENY", "PERMIT")], (True, False)), # newly denied: grew + ("requirement", [_row("PERMIT", "DENY")], (False, True)), + ("prohibition", [], (False, False)), +]) +def test_breach_changed_reads_which_way_each_flow_moved(kind, rows, expected): + bf = _DiffSession([rows]) + assert change_impact._breach_changed(bf, rule(1, kind)) == expected + assert bf.calls == [{"snapshot": change_impact.AFTER_SNAPSHOT, + "reference_snapshot": change_impact.BEFORE_SNAPSHOT}] + + +def test_breach_changed_asks_every_query_arm(): + r = dict(rule(1, "prohibition"), queries=[{"dstIps": "10.20.0.5"}, {"dstIps": "10.20.0.6"}]) + bf = _DiffSession([[], [_row("PERMIT", "DENY")]]) + assert change_impact._breach_changed(bf, r) == (True, False) + assert len(bf.calls) == 2 + + # --- how analyse_change() folds it in -------------------------------------- def _stub_pipeline(monkeypatch, *, filters=(), policy_stage=None): @@ -137,6 +277,7 @@ def _stub_pipeline(monkeypatch, *, filters=(), policy_stage=None): monkeypatch.setattr(change_impact, "_filter_change_findings", lambda bf, n: (list(filters), n + len(filters))) monkeypatch.setattr(change_impact, "_reachability_change_findings", lambda bf, n: ([], n)) + monkeypatch.setattr(change_impact, "_devices", lambda bf: ({"rtr-us5"}, {"rtr-us5"})) if policy_stage is not None: monkeypatch.setattr(change_impact, "_policy_change_findings", policy_stage) @@ -144,11 +285,11 @@ def _stub_pipeline(monkeypatch, *, filters=(), policy_stage=None): def test_unchanged_rules_join_the_all_clear_rather_than_a_second_ch000(monkeypatch): """Two cards both numbered 0 would collide on id. When nothing moved, the unchanged rules are named inside the single all-clear.""" - _stub_pipeline(monkeypatch, policy_stage=lambda bf, n: ([], ["1 (holds)", "2 (holds)"], n)) + _stub_pipeline(monkeypatch, policy_stage=lambda bf, n, b=None, a=None: {"findings": [], "unchanged": ["1", "2"], "number": n}) results = change_impact.analyse_change("before", "after") assert len(results) == 1 assert results[0]["status"] == "none" - assert "1 (holds)" in results[0]["evidence"]["detail"] + assert "1, 2" in results[0]["evidence"]["detail"] def test_with_real_changes_the_unchanged_rules_still_get_reported(monkeypatch): @@ -156,17 +297,98 @@ def test_with_real_changes_the_unchanged_rules_still_get_reported(monkeypatch): check="change_impact", severity="high", device="rtr-us5", summary="moved", detail="d", source="s", status="found", number=1) _stub_pipeline(monkeypatch, filters=[moved], - policy_stage=lambda bf, n: ([], ["3 (holds)"], n)) + policy_stage=lambda bf, n, b=None, a=None: {"findings": [], "unchanged": ["3"], "number": n}) results = change_impact.analyse_change("before", "after") ids = [f["id"] for f in results] assert len(ids) == len(set(ids)) - assert any(f["status"] == "none" and "3 (holds)" in f["evidence"]["detail"] for f in results) + assert any(f["status"] == "none" and "3." in f["evidence"]["detail"] for f in results) + + +def test_rules_not_compared_become_a_real_ch051_error(monkeypatch): + """#406 review, item 3: the CH-051 card analyse_change() builds, not one + a test made up. Both lists reach it.""" + _stub_pipeline(monkeypatch, policy_stage=lambda bf, n, b=None, a=None: { + "findings": [], "unchanged": [], "number": n, + "not_compared": ["rule 1: searchFilters failed"], + "one_side": ["rule 2: rtr-b is in the after configuration only"]}) + results = change_impact.analyse_change("before", "after") + card = [f for f in results if f["id"] == "CH-051"] + assert len(card) == 1 and card[0]["status"] == "error" + assert "2 policy rule(s)" in card[0]["summary"] + assert "searchFilters failed" in card[0]["evidence"]["detail"] + assert "after configuration only" in card[0]["evidence"]["detail"] + assert not any(f["status"] == "none" and "No change detected" in f["summary"] for f in results) + + +def test_a_failed_filter_comparison_is_ch050_never_no_change(monkeypatch): + """#406 review: silencing this path turned a failed compareFilters into + a green "No change detected".""" + _stub_pipeline(monkeypatch, policy_stage=lambda bf, n, b=None, a=None: { + "findings": [], "unchanged": [], "number": n}) + + def broken(bf, n): + raise RuntimeError("compareFilters failed") + monkeypatch.setattr(change_impact, "_filter_change_findings", broken) + results = change_impact.analyse_change("before", "after") + assert [f["id"] for f in results if f["status"] == "error"] == ["CH-050"] + assert not any(f["status"] == "none" for f in results) + + +def test_a_long_diff_never_folds_away_a_broken_policy_rule(): + """#406 review: 60 moved lines plus a removed DNS line hid "BREAKS rule 4" + inside "15 further change(s)".""" + mk = change_impact.findings.make_finding + lines = [mk(check="change_impact", severity="high", device="rtr-us5", + summary=f"line {i}", detail="d", source="rtr-us5:acl_in", + status="found", number=i) for i in range(1, 61)] + breaks = mk(check="change_impact", severity="high", device="rtr-us5", + summary="This change BREAKS policy rule 4", detail="d", + source="your policy; rule 4", status="found", number=61) + kept = change_impact._cap(lines + [breaks]) + assert len(kept) == 49 and kept[-1]["id"] == "CH-049", "48 kept, plus the summary" + assert any("BREAKS policy rule 4" in f["summary"] for f in kept[:-1]) + ids = [f["id"] for f in kept[:-1]] + assert ids == sorted(ids), "what is kept stays in the order found" + + +def test_a_capped_list_is_numbered_one_to_forty_eight_whatever_was_kept(): + """The cap keeps the MOST IMPORTANT 48, not the first 48, and a kept + finding kept the number it was given when it was found. With mixed + severities the 49th finding found is a high one, is kept, and was CH-049: + the same id as the card that summarises the rest. The ids this module + reserves (CH-049 and CH-050 to CH-054) must be free whatever was kept, so + a capped list is renumbered in the order found. (#406 review: the test + above uses one severity, where the cap keeps the first 48, so it cannot + see this.)""" + mk = change_impact.findings.make_finding + found = [mk(check="change_impact", severity="high" if i % 2 == 1 else "medium", + device="rtr-us5", summary=f"line {i}", detail="d", + source="rtr-us5:acl_in", status="found", number=i) for i in range(1, 61)] + kept = change_impact._cap(found) + ids = [f["id"] for f in kept] + assert len(ids) == len(set(ids)), f"duplicate ids: {sorted(i for i in set(ids) if ids.count(i) > 1)}" + assert ids == [f"CH-{n:03d}" for n in range(1, 49)] + ["CH-049"] + nums = [int(f["summary"].split()[1]) for f in kept[:-1]] + assert nums == sorted(nums), "what is kept stays in the order found" + + +def test_a_capped_list_keeps_the_worst_severity_not_the_first_found(): + """55 medium lines found first, then 5 high ones: the highs must survive + the cap. Keeping the first 48 would fold all five away.""" + mk = change_impact.findings.make_finding + found = [mk(check="change_impact", severity="high" if i > 55 else "medium", + device="rtr-us5", summary=f"line {i}", detail="d", + source="rtr-us5:acl_in", status="found", number=i) for i in range(1, 61)] + kept = change_impact._cap(found) + kept_highs = [f for f in kept[:-1] if f["severity"] == "high"] + assert len(kept_highs) == 5, "every high finding is kept" + assert kept[-1]["severity"] == "medium", "the summary is as bad as the worst it hides" def test_the_policy_is_cleared_even_when_the_stage_raises(monkeypatch): """A policy outliving the call that supplied it would answer the next caller's question with someone else's rules.""" - def boom(bf, n): + def boom(bf, n, b=None, a=None): raise RuntimeError("policy stage failed") _stub_pipeline(monkeypatch, policy_stage=boom) sentinel = policy_module.load_policy({"device": "rtr-us5", "policy_compliance": []}) @@ -193,3 +415,38 @@ def test_live_a_swap_is_reported_as_both_a_fix_and_a_break(tmp_path): assert any("BREAKS policy rule 4" in s for s in summaries), summaries ids = [f["id"] for f in results] assert len(ids) == len(set(ids)) + + +@needs_batfish +def test_live_a_pure_deny_narrows_a_breach_and_raises_no_banner(): + """#406 review, item 1, through the public entry point named there.""" + from ai.propose import propose_change + out = propose_change("block 10.10.10.5 to 10.20.0.5 on tcp/22 on rtr-us5", + FIXTURES / "rtr-us5-insecure") + assert out["warning"] is False, out["answer"] + assert any("narrows how policy rule 2" in f["summary"] and f["severity"] == "low" + for f in out["impact"]) + assert not any(f["severity"] == "high" for f in out["impact"]) + + +@needs_batfish +def test_live_a_device_named_in_capitals_still_reports_the_breach(tmp_path): + """#406 review, item 2: `hostname RTR-US5`, policy `device: RTR-US5`, the + change removes DNS. Main found the breach; the first version of this + branch filed it under a tick.""" + secure = (FIXTURES / "rtr-us5-secure" / "configs" / "rtr-us5.cfg").read_text() + before = secure.replace("hostname rtr-us5", "hostname RTR-US5") + after = "\n".join(line for line in before.splitlines() if "eq domain" not in line) + "\n" + for name, text in (("before", before), ("after", after)): + (tmp_path / name / "configs").mkdir(parents=True) + (tmp_path / name / "configs" / "rtr-us5.cfg").write_text(text) + policy = policy_module.load_policy({"device": "RTR-US5", "policy_compliance": [{ + "number": 1, "description": "DNS to the approved resolver must work", + "kind": "requirement", "filter": "acl_in", "violation_severity": "high", + "violation_summary": "DNS blocked", + "queries": [{"srcIps": "10.10.10.0/24", "dstIps": "218.8.104.58", + "ipProtocols": ["udp"], "dstPorts": "53"}]}]}) + results = change_impact.analyse_change(tmp_path / "before", tmp_path / "after", + policy=policy) + assert any("BREAKS policy rule 1" in f["summary"] and f["severity"] == "high" + for f in results), [f["summary"] for f in results] diff --git a/tests/test_change_impact.py b/tests/test_change_impact.py index d6bde60..82246bf 100644 --- a/tests/test_change_impact.py +++ b/tests/test_change_impact.py @@ -121,7 +121,10 @@ def fake_flows(bf, number): # tests keep meaning what they meant before it existed. Its own tests are # in tests/test_change_against_policy.py. monkeypatch.setattr(change_impact, "_policy_change_findings", - lambda bf, number: ([], [], number)) + lambda bf, number, before=None, after=None: {"findings": [], "number": number}) + # The same device on both sides, so #403's device check finds no change + # and these tests keep meaning what they meant before it existed. + monkeypatch.setattr(change_impact, "_devices", lambda bf: ({"rtr-us5"}, {"rtr-us5"})) def _a_change(number=1): diff --git a/tests/test_change_impact_coverage.py b/tests/test_change_impact_coverage.py new file mode 100644 index 0000000..b5f5cac --- /dev/null +++ b/tests/test_change_impact_coverage.py @@ -0,0 +1,268 @@ +"""What change impact looked at, and what it says it looked at (#403, #402). + +Every test here is a case where the old answer was confident and wrong: + + #403 a router deleted between before and after -> "No change detected" + #402 item 1 a network without rtr-us5 -> every proposal "could not be verified" + #402 item 3 a policy of access_control rules -> silence about them + #402 item 5 four mutants that only a live test (skipped on CI) could catch + #402 item 6 a failed reachability stage silently skipping the policy stage, + and ids that a long diff could collide with +""" + +import shutil +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from analysis import change_impact, findings, policy as policy_module +from analysis.checks import policy_compliance as pc +from conftest import needs_batfish + +FIXTURES = Path(__file__).parent / "fixtures" + + +def _pipeline(monkeypatch, *, devices=({"rtr-us5"}, {"rtr-us5"}), filters=(), + flows_raise=False, stage=None): + monkeypatch.setattr(change_impact, "connect", lambda host=None: SimpleNamespace( + set_network=lambda n: None, set_snapshot=lambda n: None)) + monkeypatch.setattr(change_impact, "load_snapshot", lambda *a, **k: None) + monkeypatch.setattr(change_impact, "find_parse_problems", lambda bf: []) + monkeypatch.setattr(change_impact, "_devices", lambda bf: devices) + monkeypatch.setattr(change_impact, "_filter_change_findings", + lambda bf, n: (list(filters), n + len(filters))) + + def flows(bf, n): + if flows_raise: + raise RuntimeError("differentialReachability exploded") + return [], n + monkeypatch.setattr(change_impact, "_reachability_change_findings", flows) + if stage is not None: + monkeypatch.setattr(change_impact, "_policy_change_findings", stage) + else: + monkeypatch.setattr(change_impact, "_policy_change_findings", + lambda bf, n, b=None, a=None: {"findings": [], "number": n}) + + +def _found(n, severity="high"): + return findings.make_finding(check="change_impact", severity=severity, device="r", + summary=f"change {n}", detail="d", source="s", + status="found", number=n) + + +# --- #403: devices on one side only ------------------------------------------ + +def test_a_removed_device_is_a_change_never_no_change_detected(monkeypatch): + _pipeline(monkeypatch, devices=({"rtr-a", "rtr-b"}, {"rtr-a"})) + results = change_impact.analyse_change("before", "after") + assert not any("No change detected" in f["summary"] for f in results) + removed = [f for f in results if "removed" in f["summary"]] + assert len(removed) == 1 and removed[0]["status"] == "found" + assert removed[0]["device"] == "rtr-b" + + +def test_an_added_device_is_a_change(monkeypatch): + _pipeline(monkeypatch, devices=({"rtr-a"}, {"rtr-a", "rtr-c"})) + results = change_impact.analyse_change("before", "after") + added = [f for f in results if "added" in f["summary"]] + assert len(added) == 1 and added[0]["status"] == "found" and added[0]["device"] == "rtr-c" + + +def test_unknown_device_lists_are_an_error_never_no_change(monkeypatch): + """None means "we could not find out", which is not "nothing changed".""" + _pipeline(monkeypatch, devices=({"rtr-a"}, None)) + results = change_impact.analyse_change("before", "after") + assert [f["status"] for f in results] == ["error"] + assert results[0]["id"] == "CH-052" + + +# --- #402 item 1: rules about devices that are not there ---------------------- + +def _rule(n, node="rtr-us5", kind="prohibition"): + return {"number": n, "kind": kind, "node": node, "filter": "acl_in", + "description": f"rule {n}", "violation_severity": "high", + "violation_summary": "x", "queries": [{"dstIps": "10.20.0.5"}]} + + +def test_a_rule_about_a_device_in_neither_config_is_not_applicable_not_an_error(monkeypatch): + monkeypatch.setattr(pc, "rules_in_use", lambda: ([_rule(1)], "built-in", False)) + monkeypatch.setattr(change_impact, "_rule_breached", + lambda *a: pytest.fail("must not ask Batfish about an absent device")) + out = change_impact._policy_change_findings(object(), 1, {"edge-fw1"}, {"edge-fw1"}) + assert out["absent"] == ["1 (rtr-us5)"] + assert out["not_compared"] == [] and out["findings"] == [] + + +def test_a_stranger_network_is_verified_rather_than_could_not_compare(monkeypatch): + """Arsh's repro, end to end through analyse_change: five built-in rules + about rtr-us5 on a network called edge-fw1 produced CH-051 x5 errors, so + propose_change() said verified=False on every request.""" + monkeypatch.setattr(pc, "rules_in_use", lambda: ([_rule(n) for n in range(1, 6)], "built-in", False)) + _pipeline(monkeypatch, devices=({"edge-fw1"}, {"edge-fw1"}), + stage=change_impact._policy_change_findings) + results = change_impact.analyse_change("before", "after") + assert all(f["status"] != "error" for f in results), results + assert "neither configuration" in results[0]["evidence"]["detail"] + + +def test_a_rule_about_a_device_on_one_side_only_cannot_be_compared(monkeypatch): + monkeypatch.setattr(pc, "rules_in_use", lambda: ([_rule(1, node="rtr-b")], "built-in", False)) + out = change_impact._policy_change_findings(object(), 1, {"rtr-a", "rtr-b"}, {"rtr-a"}) + assert len(out["one_side"]) == 1 and "before configuration only" in out["one_side"][0] + + +# --- #402 item 3: say what was not compared ------------------------------------ + +def _policy(**sections): + return policy_module.load_policy({"device": "rtr-us5", **sections}) + + +AC_RULE = {"node": "rtr-us5", "description": "8443 must be denied", "filter": "acl_in", + "headers": {"dstPorts": "8443"}, "expected": "DENY", + "violation_severity": "high", "violation_summary": "8443 open"} + + +def test_rules_in_sections_change_impact_does_not_compare_are_named(monkeypatch): + policy_module.set_active_policy(_policy(access_control=[AC_RULE], + policy_compliance=[_rule(1)])) + try: + _, _, _, note = change_impact._policy_coverage() + finally: + policy_module.clear_active_policy() + assert note.startswith("NOT compared") + assert "1 access_control" in note + + +def test_a_policy_with_no_policy_compliance_rules_says_so_loudly(): + policy_module.set_active_policy(_policy(access_control=[AC_RULE])) + try: + rules, _, _, note = change_impact._policy_coverage() + finally: + policy_module.clear_active_policy() + assert rules == [] + assert "no policy_compliance rules" in note and "no rule of yours was checked" in note + + +# --- #402 item 5: the mutants a fake session can catch ------------------------- + +class _Session: + """Records every question _rule_breached asks, answers from a table.""" + + def __init__(self, breached_arms): + self.snapshots, self.asked, self.breached = [], [], breached_arms + self.current = None + self.q = SimpleNamespace(searchFilters=self._search) + + def set_snapshot(self, name): + self.current = name + self.snapshots.append(name) + + def _search(self, nodes, filters, action, headers): + self.asked.append((self.current, action, str(headers))) + rows = 1 if (self.current, str(headers)) in self.breached else 0 + return SimpleNamespace(answer=lambda **k: SimpleNamespace( + frame=lambda: SimpleNamespace(__len__=lambda s: rows, iloc=[None] * rows))) + + +def test_rule_breached_switches_to_the_snapshot_it_is_asked_about(monkeypatch): + calls = [] + monkeypatch.setattr(pc, "_search", lambda bf, n, f, a, h: calls.append((bf.current, a)) or []) + s = _Session(set()) + change_impact._rule_breached(s, change_impact.AFTER_SNAPSHOT, _rule(1)) + assert s.snapshots == [change_impact.AFTER_SNAPSHOT] + assert calls[0][0] == change_impact.AFTER_SNAPSHOT + + +@pytest.mark.parametrize("kind", ["prohibition", "requirement"]) +def test_rule_breached_asks_with_the_checks_own_action_for_the_kind(monkeypatch, kind): + seen = [] + monkeypatch.setattr(pc, "_search", lambda bf, n, f, a, h: seen.append(a) or []) + change_impact._rule_breached(_Session(set()), change_impact.BEFORE_SNAPSHOT, _rule(1, kind=kind)) + assert seen == [pc.ACTION_BY_KIND[kind]] + + +def test_rule_breached_asks_every_arm_not_just_the_first(monkeypatch): + """Survived without Batfish before: only the second arm breaches.""" + r = _rule(1) + r["queries"] = [{"dstIps": "10.20.0.5"}, {"dstIps": "10.20.0.6"}] + monkeypatch.setattr(pc, "_search", lambda bf, n, f, a, h: [1] if h["dstIps"] == "10.20.0.6" else []) + assert change_impact._rule_breached(_Session(set()), change_impact.BEFORE_SNAPSHOT, r) is True + + +def test_the_policy_passed_in_is_the_policy_evaluated(monkeypatch): + """Survived both with and without Batfish: set_active_policy(None) in + place of set_active_policy(policy).""" + mine = _policy(policy_compliance=[_rule(7)]) + seen = [] + + def stage(bf, n, b=None, a=None): + seen.append(policy_module.active_policy()) + return {"findings": [], "number": n} + _pipeline(monkeypatch, stage=stage) + change_impact.analyse_change("before", "after", policy=mine) + assert seen == [mine] + assert policy_module.active_policy() is None, "and it is cleared afterwards" + + +# --- #402 item 6: stages are independent, ids cannot collide ------------------- + +def test_the_policy_stage_runs_when_reachability_fails(monkeypatch): + ran = [] + + def stage(bf, n, b=None, a=None): + ran.append(True) + return {"findings": [_found(n)], "number": n + 1} + _pipeline(monkeypatch, flows_raise=True, stage=stage) + results = change_impact.analyse_change("before", "after") + assert ran == [True] + assert {f["status"] for f in results} == {"found", "error"} + + +def test_a_long_diff_cannot_collide_with_the_reserved_ids(monkeypatch): + """60 filter findings used to be numbered CH-001..CH-060, two of which + are this module's own error cards.""" + _pipeline(monkeypatch, filters=[_found(n) for n in range(1, 61)], flows_raise=True) + results = change_impact.analyse_change("before", "after") + ids = [f["id"] for f in results] + assert len(ids) == len(set(ids)), ids + assert "CH-049" in ids and "further change" in next(f for f in results if f["id"] == "CH-049")["summary"] + assert max(int(i[3:]) for i in ids if i != "CH-049" and not i.startswith("CH-05")) <= 48 + + +def test_the_change_impact_cli_prints_every_policy_note(monkeypatch, capsys, tmp_path): + p = tmp_path / "p.json" + p.write_text('{"device": "rtr-us5", "policy_compliance": [{"description": "d", ' + '"kind": "prohibition", "filter": "acl_in", "violation_severity": "high", ' + '"violation_summary": "s", "queries": [{"dstIps": "10.20.0.5"}]}]}') + monkeypatch.setattr(change_impact, "analyse_change", lambda *a, **k: []) + monkeypatch.setattr("sys.argv", ["change_impact", "b", "a", str(p)]) + change_impact.main() + assert any(line.startswith("note:") for line in capsys.readouterr().out.splitlines()), \ + "the assigned rule number is a note the CLI used to drop" + + +# --- live, against real Batfish --------------------------------------------- + +@needs_batfish +def test_live_removing_a_device_is_reported(tmp_path): + """#403's own repro: multi-device-10 minus rtr-dev009.""" + after = tmp_path / "after" + shutil.copytree(FIXTURES / "multi-device-10", after) + (after / "configs" / "rtr-dev009.cfg").unlink() + results = change_impact.analyse_change(FIXTURES / "multi-device-10", after) + assert not any("No change detected" in f["summary"] for f in results) + assert any(f["device"] == "rtr-dev009" and "removed" in f["summary"] for f in results) + + +@needs_batfish +def test_live_a_stranger_network_proposal_is_verified(tmp_path): + """#402 item 1's repro: rtr-us5 renamed edge-fw1. Before this, every + built-in rule came back as CH-051 and verified was False.""" + for side in ("before", "after"): + d = tmp_path / side / "configs" + d.mkdir(parents=True) + text = (FIXTURES / "rtr-us5-secure" / "configs" / "rtr-us5.cfg").read_text() + d.joinpath("edge-fw1.cfg").write_text(text.replace("hostname rtr-us5", "hostname edge-fw1")) + results = change_impact.analyse_change(tmp_path / "before", tmp_path / "after") + assert all(f["status"] != "error" for f in results), [f["summary"] for f in results]