diff --git a/CLAUDE.md b/CLAUDE.md
index 89eb1b1..a5db1bc 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -725,7 +725,7 @@ project's recurring failure family arriving through process rather than code.
| **`risk` scoring** | Samika | Done (#60) — `POST_PROCESSORS`, ruleset in `docs/severity-rules.md`. Re-rates severity and sorts worst-first; the two limits are enforced in `pipeline.run_post_processors()`, not trusted |
| Dashboard + secure upload | Samika | Done — real findings on screen since #39 |
| PF Sense conversion | Ankeet | Done — `analysis/pfsense_convert.py`, and **hardened**: refuses config injection and path traversal via free-text fields (#53), an unbound ACL / empty rule set / unvalidated addressing (#54), and ambiguous rule order (#58, closing #47). Also refuses an interface tag that is not a plain name (a namespaced tag injected config lines), and, per rule, ``, IPv6-only and ICMP-type rules; disabled rules are left out, as pfSense does. See §7 |
-| **`change_impact`** | Shubham | Done (#140, merged 17 August) — `analysis/change_impact.py`, `analyse_change(before, after)`. Shape C: **not** in `CHECKS`, and both the module docstring and the registry comment say so. Pairs `compareFilters` (which ACL *lines* moved) with `differentialReachability` (which *traffic* changed fate), because a changed line that moves no traffic is noise and moved traffic with no changed line is the case a filter diff alone misses. Direction-aware per `docs/policy-rules.md`: opening `high`, tightening `medium`, both reported |
+| **`change_impact`** | Shubham | Done (#140, merged 17 August) — `analysis/change_impact.py`, `analyse_change(before, after)`. Shape C: **not** in `CHECKS`, and both the module docstring and the registry comment say so. Pairs `compareFilters` (which ACL *lines* moved) with `differentialReachability` (which *traffic* changed fate), because a changed line that moves no traffic is noise and moved traffic with no changed line is the case a filter diff alone misses. Direction-aware per `docs/policy-rules.md`: opening `high`, tightening `medium`, both reported. Since #389 (closing #323, merged 14 October) it also runs every `policy_compliance` rule before and after and reports which the change breaks, fixes or alters; since #403/#402 it reports devices added or removed, treats rules about a device in neither config as not applicable, and says which policy sections it did **not** compare |
| **A user policy reaching a check** | Arsh | Done (#181, merged 28 August) — `analysis/policy.py` loads and validates it; `policy_compliance` reads it via `rules_in_use()`, keyed on `active_policy() is None` rather than on emptiness. Measured below: policy-driven detections on a stranger's network go 3 → 8 |
| **Draft a policy from your own config** | Arsh | Done (#326) — `analysis/draft_policy.py`, `/api/policy/draft`, a button under the policy picker. One `policy_compliance` rule per line of each applied filter, citing the file and line Batfish records for it. Each rule is checked with the same `searchFilters` question the check will ask, so a line an earlier line partly decides is listed as not drafted, with Batfish's example packet, rather than copied. `kind` and `violation_severity` are left as ``, and `analysis/policy.py` refuses the file until they are decided and its `draft` block is deleted: accepted as it stands, a draft passes on `rtr-us5-insecure` |
| **What Netwise could not read** | Arsh | Done (#317) — `analysis/parse_coverage.py`. A partly parsed file's "Results may be incomplete" finding now says how many configuration lines Batfish did not recognise, by kind and line number, from Batfish's own `parseWarning`. On `vendor-asa`: 5 of 16, including the `access-group` line behind #373. Comment and blank lines are left out of the count only where the format's marker is known; "31 of 36" would have overstated it. Lives in the finding's detail, so it reaches every surface without an F-1 change |
diff --git a/analysis/change_impact.py b/analysis/change_impact.py
index 0f38339..4f7e5f6 100644
--- a/analysis/change_impact.py
+++ b/analysis/change_impact.py
@@ -12,7 +12,7 @@
CHECKS registry hands a check one Batfish session with one snapshot
loaded. A differential question needs two, so `run(bf)` cannot express it.
- def analyse_change(before_dir, after_dir) -> list[dict]
+ def analyse_change(before_dir, after_dir, host=None, policy=None) -> list[dict]
It is called directly and is registered in nothing. DO NOT add it to
CHECKS -- the pipeline carries a comment saying so, and this docstring is
@@ -86,23 +86,44 @@ def analyse_change(before_dir, after_dir) -> list[dict]
change_impact owns the `CH-` prefix outright since amendment A-2 (#102).
CH-000 nothing changed -- the clean sentinel
- CH-001+ one change, numbered in the order found
+ CH-001+ one change, numbered in the order found, at most 48
+ CH-049 the rest, summarised -- when there are more than 48
CH-050 could not compare -- the check-level error sentinel
+ CH-051+ one stage could not run -- see WHAT IT REPORTS below
CH-050 mirrors policy_compliance's PC-050 deliberately: SENTINEL_NUMBER +
ERROR_NUMBER_OFFSET, so there is one pattern across both of my features
rather than two conventions. See #101.
WHAT THIS DOES NOT DO YET, STATED RATHER THAN IMPLIED
- It diffs. It does not AUDIT the proposed state -- it will not tell you
- that the "after" config violates POL-2, only that it changed. Running the
+ It diffs. Since #323 it also runs every policy_compliance rule before and
+ after, so it tells you which of those rules the change breaks, fixes,
+ leaves breached, or alters -- but it still does not run access_control or
+ routing over `after_dir`, and the result says so rather than implying
+ otherwise (#402). Running the
producer checks over `after_dir` as well is open question 2 on the shapes
document and is not mine to settle alone; I have argued for it on #61 and
#86. Until then, "no change detected" means exactly that and must not be
read as "the proposed config is fine".
RUN IT
- python -m analysis.change_impact
+ python -m analysis.change_impact [policy.json]
+
+WHAT IT REPORTS, IN ORDER (#403, #402)
+ devices added or removed found -- the largest change, which neither
+ Batfish comparison describes
+ filter lines that moved found compareFilters
+ traffic that changed fate found differentialReachability
+ policy rules broken / fixed /
+ altered / still breached found policy_compliance's own query, per side
+ a stage that could not run error CH-050 filter, CH-052 devices,
+ CH-053 reachability, CH-054 policy,
+ CH-051 rules not compared
+ more than 48 changes found CH-049, the rest summarised
+ nothing moved none CH-000, naming the rules it did not move
+
+ Each stage runs whatever the others did, so a failure never hides what
+ another stage proved, and never hides that a stage did not run.
"""
import json
@@ -399,78 +420,270 @@ def _rule_label(rule: Dict[str, Any]) -> str:
return f"{head} ({rule['description']})" if rule.get("description") else head
-def _policy_change_findings(bf: Any, number: int) -> Tuple[List[Dict[str, Any]], List[str], int]:
- """Run every applicable policy rule against both snapshots.
+def _breach_changed(bf: Any, rule: Dict[str, Any]) -> Tuple[bool, bool]:
+ """For a rule breached before AND after: (did the breach grow, did it shrink)?
- Returns (findings, unchanged, number). `unchanged` is returned rather than
- turned into a card here, because analyse_change() already owns the one
- `CH-000` this module emits, and two of them would collide on id.
+ "Breached on both sides" is not "unchanged" (#402 item 4). Closing
+ LAN->8.8.8.8 while opening LAN->1.1.1.1 leaves a "no other destinations"
+ rule breached throughout. Differential searchFilters answers it: its rows
+ are flows whose verdict differs between the two snapshots.
+
+ WHICH WAY, NOT ONLY WHETHER (#406 review, @ARSH871-bot). The first version
+ returned one boolean, so a pure deny -- a breach that only SHRANK -- was
+ rated at the rule's own severity and raised the "opens access" banner.
+ Each row carries both verdicts, and Batfish returns one example per
+ direction (measured: a change that both adds and removes breaching
+ traffic gives two rows, one each way). A flow now taking the breaching
+ action, which it did not take before, grows the breach; the reverse
+ shrinks it.
"""
+ from pybatfish.datamodel.flow import HeaderConstraints
+
from analysis.checks import policy_compliance as pc
- rules, policy_label, _ = pc.rules_in_use()
- if not rules:
- return [], [], number
+ action = pc.ACTION_BY_KIND[rule["kind"]]
+ breaching = action.upper()
+ grew = shrank = False
+ for headers in rule["queries"]:
+ frame = (
+ bf.q.searchFilters(
+ nodes=rule["node"], filters=rule["filter"], action=action,
+ headers=HeaderConstraints(**headers),
+ )
+ .answer(snapshot=AFTER_SNAPSHOT, reference_snapshot=BEFORE_SNAPSHOT)
+ .frame()
+ )
+ for _, row in frame.iterrows():
+ now, then = str(row["Snapshot_Action"]).upper(), str(row["Reference_Action"]).upper()
+ grew = grew or (now == breaching and then != breaching)
+ shrank = shrank or (then == breaching and now != breaching)
+ return grew, shrank
- results: List[Dict[str, Any]] = []
- unchanged: List[str] = []
- not_compared: List[str] = []
+def _devices(bf: Any) -> "Tuple[set | None, set | None]":
+ """The devices in each snapshot, or None for a side we could not read."""
+ from analysis import snapshot
+
+ out = []
+ for name in (BEFORE_SNAPSHOT, AFTER_SNAPSHOT):
+ bf.set_snapshot(name)
+ out.append(snapshot.device_names(bf))
+ return out[0], out[1]
+
+
+def _device_change_findings(before: set, after: set, number: int) -> Tuple[List[Dict[str, Any]], int]:
+ """A device on one side only is a change, and the largest kind (#403).
+
+ Neither compareFilters nor differentialReachability reports it: a filter
+ that no longer exists has nothing to compare against, so deleting a
+ whole router used to come back as "No change detected" with a green tick.
+ """
+ results = []
+ for name in sorted(before - after):
+ results.append(findings.make_finding(
+ check=CHECK_NAME, severity=_CLOSING_SEVERITY, device=name,
+ summary=f"Device {name} is removed by this change",
+ detail=(f"{name} is in the before configuration and not in the after one. "
+ "Every flow it carried changes, and the filter and reachability "
+ "comparisons below do not describe a device that exists on one "
+ "side only, so this card is the only record of it."),
+ source=f"{BEFORE_SNAPSHOT} -> {AFTER_SNAPSHOT}", status="found", number=number))
+ number += 1
+ for name in sorted(after - before):
+ results.append(findings.make_finding(
+ check=CHECK_NAME, severity=_OPENING_SEVERITY, device=name,
+ summary=f"Device {name} is added by this change",
+ detail=(f"{name} is in the after configuration and not in the before one. "
+ "Nothing about its traffic is compared below, because there is "
+ "nothing to compare it against -- run a scan of the after "
+ "configuration to check it."),
+ source=f"{BEFORE_SNAPSHOT} -> {AFTER_SNAPSHOT}", status="found", number=number))
+ number += 1
+ return results, number
+
+
+def _policy_coverage() -> Tuple[List[Dict[str, Any]], str, bool, str]:
+ """(policy_compliance rules, label, user_supplied, sentence about the rest).
+
+ US-31 says "every policy rule". This stage compares policy_compliance
+ rules only, and the sentence says so out loud (#402 item 3) -- a user
+ whose policy is one access_control rule must not read "unchanged" as
+ "your rule still holds".
+ """
+ from analysis.checks import policy_compliance as pc
+
+ rules, label, user_supplied = pc.rules_in_use()
+ note = ""
+ active = policy_module.active_policy()
+ if active is not None:
+ skipped = [f"{len(active.entries_for(s))} {s}"
+ for s in ("access_control", "routing") if active.entries_for(s)]
+ if skipped:
+ note = ("NOT compared: " + " and ".join(skipped) + " rule(s) in your policy. "
+ "Change impact compares policy_compliance rules only; scan the "
+ "after configuration to check the others.")
+ if not rules:
+ note = ("NOT compared: your policy has no policy_compliance rules, so no "
+ "rule of yours was checked against this change. " + note).strip()
+ return rules, label, user_supplied, note
+
+
+def _policy_change_findings(bf: Any, number: int, before: "set | None" = None,
+ after: "set | None" = None) -> Dict[str, Any]:
+ """Run every applicable policy_compliance rule against both snapshots.
+
+ Returns a dict, not a finding list, because three of its outputs are
+ sentences that analyse_change() folds into the ONE card it owns:
+
+ findings created, resolved, altered or still-breached rules
+ unchanged same verdict on the rule's filter, before and after
+ absent rules about a device in NEITHER configuration
+ one_side rules about a device in only ONE configuration
+ not_compared rules we could not ask -- never "unchanged" (F-4)
+ coverage what this stage did not compare at all
+ """
+ rules, policy_label, user_supplied, coverage = _policy_coverage()
+ out: Dict[str, Any] = {"findings": [], "unchanged": [], "absent": [], "one_side": [],
+ "not_compared": [], "coverage": coverage,
+ "label": policy_label, "number": number}
+ # Casefolded on both sides (#406 review, item 2). Batfish names nodes in
+ # lower case and resolves a policy's `RTR-US5` to `rtr-us5`, so a
+ # case-sensitive test called a present device absent -- and filed a
+ # breach the change really caused under a tick.
+ before = {d.casefold() for d in before} if before is not None else None
+ after = {d.casefold() for d in after} if after is not None else None
for rule in rules:
+ node = rule["node"]
+ key = str(node).casefold()
+ if before is not None and after is not None:
+ if key not in before and key not in after:
+ if user_supplied:
+ # The user's own rule. A scan of the same files calls it
+ # PC-050, "could not check", so a tick here would
+ # disagree with the scan (#406 review, item 2).
+ out["not_compared"].append(
+ f"{_rule_label(rule)}: {node} is in neither configuration")
+ continue
+ # Our built-in example rules, worded like PC-050 (#402 item 1).
+ # A rule we wrote about a device the user does not have is a
+ # proven "not applicable", not a blind spot: an error here
+ # made every proposal on a network without rtr-us5 "could not
+ # be fully verified".
+ out["absent"].append(f"{rule.get('number', '?')} ({node})")
+ continue
+ if (key in before) != (key in after):
+ side = "before" if key in before else "after"
+ out["one_side"].append(f"{_rule_label(rule)}: {node} is in the {side} configuration only")
+ continue
try:
- before = _rule_breached(bf, BEFORE_SNAPSHOT, rule)
- after = _rule_breached(bf, AFTER_SNAPSHOT, rule)
+ was = _rule_breached(bf, BEFORE_SNAPSHOT, rule)
+ now = _rule_breached(bf, AFTER_SNAPSHOT, rule)
+ grew, shrank = _breach_changed(bf, rule) if (was and now) else (False, False)
except Exception as error:
- # A rule we could not ask is a blind spot, never "unchanged" (F-4).
- not_compared.append(f"{_rule_label(rule)}: {findings.describe_error(error)}")
+ out["not_compared"].append(f"{_rule_label(rule)}: {findings.describe_error(error)}")
continue
- if before == after:
- state = "still breached" if before else "holds"
- unchanged.append(f"{rule.get('number', '?')} ({state})")
+ if not was and not now:
+ out["unchanged"].append(str(rule.get("number", "?")))
continue
- created = after and not before
- direction = _RULE_DIRECTION[(rule["kind"], created)]
- if number >= ERROR_NUMBER_OFFSET:
- not_compared.append(f"{_rule_label(rule)}: more rule changes than finding numbers")
- continue
- results.append(
- findings.make_finding(
- check=CHECK_NAME,
- severity=(rule.get("violation_severity", "high") if created else "low"),
- device=rule["node"],
- summary=(
- f"This change BREAKS {_rule_label(rule)}"
- if created else
- f"This change FIXES {_rule_label(rule)}"
- ),
- detail=(
- f"{rule['kind']} on {rule['node']} filter {rule['filter']}: "
- f"{'held before, breached after' if created else 'breached before, holds after'}. "
- f"Direction: {direction}. "
- + (f"Violation if applied: {rule.get('violation_summary')}." if created
- else "The breach this rule reports is gone in the proposed config.")
- ),
- source=f"{policy_label}; rule {rule.get('number', '?')}",
- status="found",
- number=number,
- )
- )
+ where = f"{rule['kind']} on {node} filter {rule['filter']}"
+ if was and now:
+ if grew and shrank:
+ summary = f"This change alters how {_rule_label(rule)} is breached"
+ detail = (f"{where}: breached before and after, but by different "
+ "traffic -- the change adds some breaching flows and "
+ "removes others. Direction: changed.")
+ severity = rule.get("violation_severity", "high")
+ elif grew:
+ summary = f"This change widens how {_rule_label(rule)} is breached"
+ detail = (f"{where}: breached before, and by more traffic after. "
+ f"Direction: {_RULE_DIRECTION[(rule['kind'], True)]}. "
+ f"Violation if applied: {rule.get('violation_summary')}.")
+ severity = rule.get("violation_severity", "high")
+ elif shrank:
+ # A shrinking breach is a tightening of the breach, not a new
+ # problem: the rule's own severity here raised "opens access"
+ # for a pure deny (#406 review, item 1).
+ summary = f"This change narrows how {_rule_label(rule)} is breached"
+ detail = (f"{where}: breached before, and by less traffic after, but "
+ "still breached. Direction: "
+ f"{_RULE_DIRECTION[(rule['kind'], False)]}.")
+ severity = "low"
+ else:
+ summary = f"{_rule_label(rule)} is breached before and after this change"
+ detail = (f"{where}: breached the same way on both sides. This change "
+ "does not cause it and does not fix it; the violation stays.")
+ severity = "low"
+ else:
+ created = now and not was
+ direction = _RULE_DIRECTION[(rule["kind"], created)]
+ summary = (f"This change BREAKS {_rule_label(rule)}" if created
+ else f"This change FIXES {_rule_label(rule)}")
+ detail = (f"{where}: {'held before, breached after' if created else 'breached before, holds after'}. "
+ f"Direction: {direction}. "
+ + (f"Violation if applied: {rule.get('violation_summary')}." if created
+ else "The breach this rule reports is gone in the proposed config."))
+ severity = rule.get("violation_severity", "high") if created else "low"
+ out["findings"].append(findings.make_finding(
+ check=CHECK_NAME, severity=severity, device=node, summary=summary, detail=detail,
+ source=f"{policy_label}; rule {rule.get('number', '?')}", status="found",
+ number=number))
number += 1
-
- if not_compared:
- results.append(
- findings.error_finding(
- check=CHECK_NAME,
- device="unknown",
- summary=f"{len(not_compared)} policy rule(s) could not be compared across the change",
- detail="; ".join(not_compared),
- source=policy_label,
- number=SKIPPED_NUMBER + 1,
- )
- )
- return results, unchanged, number
+ out["number"] = number
+ return out
+
+
+#: Ordinary findings stop here. 49 and above are this module's own cards, so
+#: a long diff can never collide with them (#402 item 6) -- the band
+#: policy_compliance already uses (PC-049, PC-050).
+_OVERFLOW_NUMBER = 49
+_RULES_NOT_COMPARED_NUMBER = SKIPPED_NUMBER + 1 # CH-051
+_DEVICES_UNKNOWN_NUMBER = SKIPPED_NUMBER + 2 # CH-052
+_REACHABILITY_FAILED_NUMBER = SKIPPED_NUMBER + 3 # CH-053
+_POLICY_FAILED_NUMBER = SKIPPED_NUMBER + 4 # CH-054
+
+
+def _cap(regular: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+ """Keep 48 ordinary findings, the most important ones; fold the rest into CH-049.
+
+ WHICH 48 (#406 review). Keeping the first 48 kept them in the order the
+ stages ran, so the policy cards -- which run last and are the ones about
+ the user's own rules -- were the first to be folded away: 60 moved lines
+ plus a removed DNS line hid "BREAKS rule 4" inside "15 further
+ change(s)". Now the worst severity is kept first, a policy card ahead of
+ a filter or flow card of the same severity, and what is kept is still
+ shown in the order found.
+ """
+ if len(regular) < _OVERFLOW_NUMBER:
+ return regular
+ order = {"high": 0, "medium": 1, "low": 2}
+
+ def importance(index: int) -> Tuple[int, int, int]:
+ f = regular[index]
+ about_a_rule = "; rule " in str(f["evidence"]["source"])
+ return order.get(f["severity"], 3), 0 if about_a_rule else 1, index
+
+ keep = set(sorted(range(len(regular)), key=importance)[: _OVERFLOW_NUMBER - 1])
+ kept = [f for i, f in enumerate(regular) if i in keep]
+ rest = [f for i, f in enumerate(regular) if i not in keep]
+ # A kept finding keeps the number it was found under, so the 49th found
+ # can sit here as CH-049 -- the id of the summary card below -- or on a
+ # reserved CH-050+ (#406 review, @ARSH871-bot). Renumber in the order
+ # found, so CH-001..048 are the ordinary cards and 049 is only ever the
+ # summary. Nothing outside this function relies on a kept card's id.
+ for new_number, f in enumerate(kept, start=1):
+ f["id"] = f"{findings.PREFIX_BY_CHECK[CHECK_NAME]}-{new_number:03d}"
+ worst = min((f["severity"] for f in rest), key=order.get)
+ kept.append(findings.make_finding(
+ check=CHECK_NAME, severity=worst, device="unknown",
+ summary=f"{len(rest)} further change(s) not listed individually",
+ detail=("This change produced more findings than are listed one by one. "
+ "The first of the rest: " + "; ".join(f["summary"] for f in rest[:5])
+ + (" ..." if len(rest) > 5 else "")),
+ source=f"{BEFORE_SNAPSHOT} -> {AFTER_SNAPSHOT}", status="found",
+ number=_OVERFLOW_NUMBER))
+ return kept
def analyse_change(
@@ -489,14 +702,15 @@ def analyse_change(
The three outcomes are kept distinct, which is the whole of F-4 here:
found the change did something, and here is what
- none both configs loaded, both questions ran, nothing moved
+ none both configs loaded, every question ran, nothing moved
error we could not compare -- NOT the same as "nothing changed"
That middle one is the dangerous claim. "This change is safe" is exactly
what a user wants to hear before applying it to a firewall, so it may only
- be said when both snapshots parsed and both questions actually ran.
+ be said when both snapshots parsed and every question actually ran.
"""
before_dir, after_dir = Path(before_dir), Path(after_dir)
+ where = f"{before_dir} -> {after_dir}"
try:
bf = connect(host)
@@ -508,7 +722,7 @@ def analyse_change(
"docker start batfish. "
f"Underlying error: {findings.describe_error(error)}"
),
- source=f"{before_dir} -> {after_dir}",
+ source=where,
)
# Load both. Named snapshots in one network so the differential questions
@@ -554,62 +768,79 @@ def analyse_change(
source=str(folder),
)
- results: List[Dict[str, Any]] = []
+ regular: List[Dict[str, Any]] = []
+ reserved: List[Dict[str, Any]] = []
number = 1
+ def blind(n: int, summary: str, detail: str) -> None:
+ reserved.append(findings.error_finding(
+ check=CHECK_NAME, summary=summary, detail=detail, source=where, number=n))
+
+ # Which devices are on each side (#403). Unknown is an error, never "none".
+ before_devs = after_devs = None
+ try:
+ before_devs, after_devs = _devices(bf)
+ except Exception as error:
+ blind(_DEVICES_UNKNOWN_NUMBER, "Could not list the devices in each configuration",
+ findings.describe_error(error))
+ if before_devs is None or after_devs is None:
+ if not any(f["id"].endswith(f"{_DEVICES_UNKNOWN_NUMBER:03d}") for f in reserved):
+ blind(_DEVICES_UNKNOWN_NUMBER, "Could not list the devices in each configuration",
+ "Batfish did not say which devices each configuration defines, so a "
+ "device added or removed by this change would not be reported.")
+ else:
+ device_changes, number = _device_change_findings(before_devs, after_devs, number)
+ regular.extend(device_changes)
+
+ # Each stage runs whatever the others did (#402 item 6). What one proved
+ # stands when another fails, and the failure is said (#22).
try:
filter_changes, number = _filter_change_findings(bf, number)
+ regular.extend(filter_changes)
except Exception as error:
- return _error(
- summary="Change impact could not run: the filter comparison failed",
- detail=findings.describe_error(error),
- source=f"{before_dir} -> {after_dir}",
- )
- results.extend(filter_changes)
+ blind(SKIPPED_NUMBER, "The filter comparison could not run", findings.describe_error(error))
try:
flow_changes, number = _reachability_change_findings(bf, number)
+ regular.extend(flow_changes)
except Exception as error:
- # The filter comparison may already have found real changes. Report
- # BOTH -- what was proved AND what could not be reached. Discarding a
- # proven finding because a second query failed is issue #22, and it
- # fails in the reassuring direction.
- results.extend(
- _error(
- summary="The reachability comparison could not run",
- detail=findings.describe_error(error),
- source=f"{before_dir} -> {after_dir}",
- )
- )
- return results
- results.extend(flow_changes)
+ blind(_REACHABILITY_FAILED_NUMBER, "The reachability comparison could not run",
+ findings.describe_error(error))
# Every policy rule, before and after (#323). Installed the same way
# analyse() installs it and cleared in a `finally`, so a policy never
# outlives the call that supplied it.
policy_module.set_active_policy(policy)
+ stage: Dict[str, Any] = {}
try:
- rule_changes, unchanged, number = _policy_change_findings(bf, number)
+ stage = _policy_change_findings(bf, number, before_devs, after_devs)
+ regular.extend(stage["findings"])
except Exception as error:
- # Same rule as the reachability stage: what was proved above stands,
- # and the user is told what could not be checked (#22).
- results.extend(
- _error(
- summary="The policy comparison could not run",
- detail=findings.describe_error(error),
- source=f"{before_dir} -> {after_dir}",
- )
- )
- return results
+ blind(_POLICY_FAILED_NUMBER, "The policy comparison could not run",
+ findings.describe_error(error))
finally:
policy_module.clear_active_policy()
- results.extend(rule_changes)
- unchanged_note = (
- " Policy rules with the same verdict before and after, rule (state): "
- + ", ".join(unchanged) + "."
- if unchanged else ""
- )
+ not_compared = stage.get("not_compared", []) + stage.get("one_side", [])
+ if not_compared:
+ blind(_RULES_NOT_COMPARED_NUMBER,
+ f"{len(not_compared)} policy rule(s) could not be compared across the change",
+ "; ".join(not_compared))
+
+ # The sentences about rules nothing above reported on.
+ notes = []
+ if stage.get("unchanged"):
+ notes.append("Policy rule(s) with the same verdict on their filter before and "
+ "after (this says the filter's decision did not move, not that "
+ "nothing about their traffic did): " + ", ".join(stage["unchanged"]) + ".")
+ if stage.get("absent"):
+ notes.append("Policy rule(s) about devices in neither configuration, which this "
+ "change cannot affect: " + ", ".join(stage["absent"]) + ".")
+ if stage.get("coverage"):
+ notes.append(stage["coverage"])
+ note_text = (" " + " ".join(notes)) if notes else ""
+
+ results = _cap(regular) + reserved
if not results:
return [
@@ -618,38 +849,82 @@ def analyse_change(
device="unknown",
summary="No change detected between the two configurations",
detail=(
- "No filter line treats the same traffic differently, and no "
- "flow reaches a destination it could not reach before. This "
- "says the change moved nothing; it does not say the proposed "
- "config is free of problems -- run the audit for that."
- + unchanged_note
+ "No device was added or removed, no filter line treats the same "
+ "traffic differently, and no flow reaches a destination it could "
+ "not reach before. This says the change moved nothing; it does "
+ "not say the proposed config is free of problems -- run the "
+ "audit for that." + note_text
),
- source=f"{before_dir} -> {after_dir}",
+ source=where,
number=0,
)
]
- if unchanged:
- # US-31 asks for a direction per rule, and "unchanged" is one. Only
- # emitted here, alongside real changes, so it can never be mistaken
- # for the all-clear above -- and it takes CH-000 only because that
- # all-clear is, by construction, not in this list.
+ if notes:
+ # Alongside real changes, so it can never be mistaken for the
+ # all-clear above -- and it takes CH-000 only because that all-clear
+ # is, by construction, not in this list.
results.append(
findings.no_issues_finding(
check=CHECK_NAME,
device="unknown",
- summary=f"{len(unchanged)} policy rule(s) unchanged by this edit",
- detail=(
- "Same verdict before and after, so this change neither "
- "causes nor fixes them." + unchanged_note
- ),
- source=f"{before_dir} -> {after_dir}",
+ summary="Policy rules this change did not move",
+ detail=note_text.strip(),
+ source=where,
number=0,
)
)
return results
+# --- How big is this change? (#420) -------------------------------------------
+#
+# One number beside the findings, so a reviewer gets the size of a change
+# before reading every card. Three rules, each from #420's criteria:
+#
+# COMPUTED, AND SHOWN. The score is a weighted count of the findings above,
+# and the breakdown is returned with it, so it can always be explained. It
+# reads only `status` and `severity` -- structured fields -- never a
+# summary's wording, which this project has been burnt by twice (#145,
+# #205).
+#
+# OPENING COUNTS MORE THAN CLOSING. Not a second model: the severities
+# above already encode it (_OPENING_SEVERITY high, _CLOSING_SEVERITY
+# medium; a broken policy rule carries the rule's own severity), so
+# weighting by severity carries it through.
+#
+# INCOMPLETE NEVER SCORES SMALL. A change that could not be fully compared
+# has an unknown size, and a low number would read as reassurance. Any
+# `error` finding makes the band "incomplete", whatever the number.
+#
+# Returned beside the findings, never inside them: it is not an F-1 field
+# (see the open question on #405 about adding keys to findings).
+
+SCORE_WEIGHTS = {"high": 3, "medium": 2, "low": 1}
+
+#: (lowest score, band). A score of 0 is "none" only when nothing was blind.
+SCORE_BANDS = ((10, "large"), (4, "moderate"), (1, "small"), (0, "none"))
+
+
+def score(results: List[Dict[str, Any]]) -> Dict[str, Any]:
+ """The size of a change, from analyse_change()'s findings."""
+ found = [f for f in results if f.get("status") == "found"]
+ blind = [f for f in results if f.get("status") == "error"]
+ breakdown = {sev: sum(1 for f in found if f.get("severity") == sev) for sev in SCORE_WEIGHTS}
+ total = sum(SCORE_WEIGHTS[sev] * n for sev, n in breakdown.items())
+ band = next(name for floor, name in SCORE_BANDS if total >= floor)
+ if blind:
+ band = "incomplete"
+ parts = [f"{n} {sev} x{SCORE_WEIGHTS[sev]}" for sev, n in breakdown.items() if n]
+ explanation = (" + ".join(parts) + f" = {total}") if parts else "no changes found = 0"
+ if blind:
+ explanation += (f"; {len(blind)} part(s) could not be compared, so the real size is "
+ "unknown and may be larger")
+ return {"score": total, "band": band, "complete": not blind,
+ "breakdown": dict(breakdown, could_not_compare=len(blind)),
+ "explanation": explanation}
+
+
def main() -> None:
if len(sys.argv) not in (3, 4):
sys.exit(
@@ -660,13 +935,20 @@ def main() -> None:
policy = None
if len(sys.argv) == 4:
policy = policy_module.load_policy_file(sys.argv[3])
+ # Every channel, like `python -m analysis.pipeline --policy` (#402 item 6).
+ for note in list(policy.renamed) + list(policy.assigned):
+ print(f"note: {note}")
# Same contract as `python -m analysis.pipeline --policy` (#387
# review): a pair the contradiction check could not compare is said,
- # not dropped.
+ # not dropped. Notes before warnings, matching pipeline.py's own
+ # established order.
for note in policy.undecidable:
print(f"warning: {note}")
results = analyse_change(sys.argv[1], sys.argv[2], policy=policy)
print(json.dumps(results, indent=2))
+ size = score(results)
+ print()
+ print(f" size {size['band']} ({size['explanation']})")
print()
for status in ("found", "none", "error"):
print(f" {status:<6} {sum(1 for f in results if f['status'] == status)}")
diff --git a/docs/policy-rules.md b/docs/policy-rules.md
index 9ae8930..ddc5851 100644
--- a/docs/policy-rules.md
+++ b/docs/policy-rules.md
@@ -799,7 +799,16 @@ unapproved destination, and POL-2 arm A catches DNS to `10.20.0.5`. Verified:
---
-### Change impact now checks every policy rule (#323, US-31)
+## 7. Sign-off
+
+| Member | Role in this decision | Agreed |
+|---|---|---|
+| Shubham | Author; owns the check | ✅ |
+| Arsh | Registry, pipeline, `findings.py` impact | ✅ rules, IDs, and the §4 `findings.py` change |
+| Samika | Severity values are his | ✅ |
+| Ankeet | Consumes findings in the AI layer | ⬜ |
+
+## 8. Change impact checks every policy rule (#323, #402, #403)
`analyse_change(before, after)` used to answer one question: *what traffic
moved*. It now also answers the one an engineer actually asks before applying
@@ -811,11 +820,21 @@ breach is), once against each snapshot:
| before | after | reported as |
|---|---|---|
-| holds | holds | unchanged |
-| breached | breached | unchanged — still broken, not caused by this edit |
+| holds | holds | unchanged — *on the rule's filter*, which is all that is claimed |
+| breached | breached, by the same traffic | **still breached** — `found`, `low`: not caused by this edit, but never a green tick (#402) |
+| breached | breached, by **more** traffic | **WIDENS** the breach — at the rule's own severity |
+| breached | breached, by **less** traffic | **NARROWS** the breach — `low`, still breached, never the "opens access" banner (#406 review) |
+| breached | breached, some flows added and some removed | **ALTERS** the rule — at the rule's own severity, direction *changed* |
| breached | holds | **FIXES** the rule, `low` |
| holds | breached | **BREAKS** the rule, at the rule's own severity |
+The last three rows come from one differential `searchFilters`. Each row it
+returns carries both verdicts, and Batfish returns one example per direction,
+so a flow now taking the breaching action that did not before widens the
+breach, and the reverse narrows it. The first version asked only *whether*
+the breach moved and rated every move at the rule's own severity, so a pure
+deny raised the "opens access" warning.
+
Direction uses change impact's existing terms: a prohibition breach created is
*opening*, a requirement breach created is *tightening*, and the reverse for
fixes.
@@ -848,11 +867,14 @@ built-in rules, like a scan does. The CLI takes it as a third argument:
python -m analysis.change_impact before/ after/ my-policy.json
```
-## 7. Sign-off
-| Member | Role in this decision | Agreed |
-|---|---|---|
-| Shubham | Author; owns the check | ✅ |
-| Arsh | Registry, pipeline, `findings.py` impact | ✅ rules, IDs, and the §4 `findings.py` change |
-| Samika | Severity values are his | ✅ |
-| Ankeet | Consumes findings in the AI layer | ⬜ |
+### What it does not look at, said out loud (#402, #403)
+
+- **Devices added or removed** are their own `found` cards. Before #403, deleting a whole router came back as "No change detected" with a green tick, because neither Batfish comparison describes a device that exists on one side only.
+- **One of our built-in rules about a device in neither configuration** is *not applicable* — this change cannot affect it — and is named in the result, not reported as an error. Reporting it as an error made every proposal on a network without `rtr-us5` "could not be fully verified".
+- **One of the user's own rules about a device in neither configuration** is part of the `CH-051` error instead. A scan of the same files calls it `PC-050`, could not check, and the two must not disagree (#406 review).
+- **Device names are compared as Batfish compares them**, case-free. A policy naming `RTR-US5` is about the device Batfish calls `rtr-us5`; a case-sensitive test once filed a real breach on it under a tick.
+- **A rule about a device on one side only** cannot be compared and is a single `CH-051` error.
+- **Only `policy_compliance` rules are compared.** A policy with `access_control` or `routing` rules gets a sentence beginning "NOT compared:" naming how many, and a policy with no `policy_compliance` rules says no rule of the user's was checked.
+- **"Unchanged" means the rule's filter decided its traffic the same way.** Unbinding the filter from its interface changes the network without changing that answer, so the wording claims no more than that.
+
diff --git a/tests/test_change_against_policy.py b/tests/test_change_against_policy.py
index 71a9a00..90aa05c 100644
--- a/tests/test_change_against_policy.py
+++ b/tests/test_change_against_policy.py
@@ -32,13 +32,20 @@ def rule(number, kind, severity="high"):
}
-def stage(monkeypatch, rules, verdicts):
+def stage(monkeypatch, rules, verdicts, breach=(False, False), devices=None,
+ user_supplied=True):
"""Run the policy stage with fixed rules and fixed per-snapshot verdicts.
`verdicts` maps rule number -> (breached_before, breached_after), or to an
- Exception to raise when that rule is asked.
+ Exception to raise when that rule is asked. `breach` is what
+ `_breach_changed` reports for a rule breached on both sides: (grew,
+ shrank). Returns (findings, unchanged, number), plus the whole stage dict.
+
+ It returns the stage's own findings and nothing it makes up: CH-051 is
+ built by analyse_change(), so it is tested there, against the real code
+ (#406 review, item 3).
"""
- monkeypatch.setattr(pc, "rules_in_use", lambda: (rules, "test policy", True))
+ monkeypatch.setattr(pc, "rules_in_use", lambda: (rules, "test policy", user_supplied))
def fake(bf, snapshot, r):
v = verdicts[r["number"]]
@@ -47,13 +54,17 @@ def fake(bf, snapshot, r):
return v[0] if snapshot == change_impact.BEFORE_SNAPSHOT else v[1]
monkeypatch.setattr(change_impact, "_rule_breached", fake)
- return change_impact._policy_change_findings(object(), 1)
+ monkeypatch.setattr(change_impact, "_breach_changed", lambda bf, r: breach)
+ before, after = devices if devices else (None, None)
+ out = change_impact._policy_change_findings(object(), 1, before, after)
+ unchanged = [f"{n} (holds)" for n in out["unchanged"]]
+ return list(out["findings"]), unchanged, out["number"], out
# --- the four outcomes -----------------------------------------------------
def test_a_change_that_causes_a_breach_says_it_breaks_the_rule(monkeypatch):
- found, unchanged, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (False, True)})
+ found, unchanged, _, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (False, True)})
assert len(found) == 1
f = found[0]
assert f["status"] == "found"
@@ -63,7 +74,7 @@ def test_a_change_that_causes_a_breach_says_it_breaks_the_rule(monkeypatch):
def test_a_change_that_repairs_a_breach_says_it_fixes_the_rule(monkeypatch):
- found, _, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (True, False)})
+ found, _, _, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (True, False)})
assert len(found) == 1
assert "FIXES" in found[0]["summary"]
assert found[0]["severity"] == "low", "a fix is good news, not a high finding"
@@ -71,7 +82,7 @@ def test_a_change_that_repairs_a_breach_says_it_fixes_the_rule(monkeypatch):
def test_a_change_that_fixes_one_and_breaks_another_reports_both(monkeypatch):
"""US-31's own criterion, and the reason this stage exists."""
- found, _, _ = stage(
+ found, _, _, _ = stage(
monkeypatch,
[rule(1, "prohibition"), rule(4, "requirement", severity="medium")],
{1: (True, False), 4: (False, True)},
@@ -81,15 +92,54 @@ def test_a_change_that_fixes_one_and_breaks_another_reports_both(monkeypatch):
assert any("BREAKS" in s and "rule 4" in s for s in summaries)
-def test_a_rule_with_the_same_verdict_either_side_is_unchanged_not_a_finding(monkeypatch):
- found, unchanged, _ = stage(
- monkeypatch, [rule(1, "prohibition"), rule(2, "prohibition")],
- {1: (False, False), 2: (True, True)},
- )
+def test_a_rule_that_holds_either_side_is_unchanged_not_a_finding(monkeypatch):
+ found, unchanged, _, _ = stage(monkeypatch, [rule(1, "prohibition")], {1: (False, False)})
assert found == []
- assert unchanged == ["1 (holds)", "2 (still breached)"], (
- "a rule broken before AND after is unchanged -- this edit did not cause it"
- )
+ assert unchanged == ["1 (holds)"]
+
+
+def test_a_rule_breached_on_both_sides_is_never_a_green_tick(monkeypatch):
+ """#402 item 4. "Still breached" sat inside the `none` card, which the
+ propose pane draws as a tick. A violation that survives the change is
+ still a violation."""
+ found, unchanged, _, _ = stage(monkeypatch, [rule(2, "prohibition")], {2: (True, True)})
+ assert unchanged == []
+ assert len(found) == 1 and found[0]["status"] == "found"
+ assert "breached before and after" in found[0]["summary"]
+ assert found[0]["severity"] == "low", "not caused by this change"
+
+
+def test_a_breach_that_moves_to_other_traffic_is_reported_as_altered(monkeypatch):
+ """Closing LAN->8.8.8.8 while opening LAN->1.1.1.1 keeps a rule breached
+ throughout; the differential query says traffic moved both ways."""
+ found, _, _, _ = stage(monkeypatch, [rule(2, "prohibition")], {2: (True, True)},
+ breach=(True, True))
+ assert "alters how" in found[0]["summary"]
+ assert found[0]["severity"] == "high", "it is the rule's own severity"
+ assert "Direction: changed" in found[0]["evidence"]["detail"]
+
+
+@pytest.mark.parametrize("kind,direction", [("prohibition", "opening"),
+ ("requirement", "tightening")])
+def test_a_breach_that_only_grows_is_the_rules_own_severity(monkeypatch, kind, direction):
+ found, _, _, _ = stage(monkeypatch, [rule(2, kind)], {2: (True, True)},
+ breach=(True, False))
+ assert "widens how" in found[0]["summary"]
+ assert found[0]["severity"] == "high"
+ assert f"Direction: {direction}" in found[0]["evidence"]["detail"]
+
+
+@pytest.mark.parametrize("kind,direction", [("prohibition", "tightening"),
+ ("requirement", "opening")])
+def test_a_breach_that_only_shrinks_is_low_never_high(monkeypatch, kind, direction):
+ """#406 review, item 1. A pure deny shrinks a prohibition's breach. Rated
+ at the rule's own severity, it raised the "opens access" banner."""
+ found, _, _, _ = stage(monkeypatch, [rule(2, kind)], {2: (True, True)},
+ breach=(False, True))
+ assert "narrows how" in found[0]["summary"]
+ assert found[0]["severity"] == "low"
+ assert f"Direction: {direction}" in found[0]["evidence"]["detail"]
+ assert "still breached" in found[0]["evidence"]["detail"]
@pytest.mark.parametrize("kind,before,after,direction", [
@@ -99,25 +149,23 @@ def test_a_rule_with_the_same_verdict_either_side_is_unchanged_not_a_finding(mon
("requirement", True, False, "opening"),
])
def test_direction_is_stated_in_this_modules_own_terms(monkeypatch, kind, before, after, direction):
- found, _, _ = stage(monkeypatch, [rule(1, kind)], {1: (before, after)})
+ found, _, _, _ = stage(monkeypatch, [rule(1, kind)], {1: (before, after)})
assert f"Direction: {direction}" in found[0]["evidence"]["detail"]
# --- F-4: could-not-ask is never unchanged ----------------------------------
def test_a_rule_that_cannot_be_asked_is_an_error_never_unchanged(monkeypatch):
- found, unchanged, _ = stage(
+ found, unchanged, _, out = stage(
monkeypatch, [rule(1, "prohibition"), rule(2, "prohibition")],
{1: RuntimeError("searchFilters failed"), 2: (False, False)},
)
- errors = [f for f in found if f["status"] == "error"]
- assert len(errors) == 1
- assert "rule 1" in errors[0]["evidence"]["detail"]
+ assert len(out["not_compared"]) == 1 and "rule 1" in out["not_compared"][0]
assert unchanged == ["2 (holds)"], "the rule we could not ask must not be listed as holding"
def test_the_ids_this_stage_emits_are_unique(monkeypatch):
- found, _, _ = stage(
+ found, _, _, _ = stage(
monkeypatch, [rule(n, "prohibition") for n in (1, 2, 3)] + [rule(9, "prohibition")],
{1: (False, True), 2: (True, False), 3: (False, True), 9: RuntimeError("x")},
)
@@ -125,6 +173,98 @@ def test_the_ids_this_stage_emits_are_unique(monkeypatch):
assert len(ids) == len(set(ids))
+# --- device names: compared as Batfish compares them (#406 review, item 2) ---
+
+def test_a_device_named_in_capitals_is_still_the_device_batfish_found(monkeypatch):
+ r = dict(rule(1, "prohibition"), node="RTR-US5")
+ found, _, _, out = stage(monkeypatch, [r], {1: (False, True)},
+ devices=({"rtr-us5"}, {"rtr-us5"}))
+ assert out["absent"] == [] and out["not_compared"] == []
+ assert "BREAKS" in found[0]["summary"], "a breach the change caused, not a tick"
+
+
+def test_the_comparison_is_case_free_from_either_side(monkeypatch):
+ """Batfish's names are lower case today; the rule is "compare as Batfish
+ resolves", not "trust that the set happens to be lower case"."""
+ found, _, _, out = stage(monkeypatch, [rule(1, "prohibition")], {1: (False, True)},
+ devices=({"RTR-US5"}, {"RTR-US5"}))
+ assert out["absent"] == [] and out["not_compared"] == []
+ assert "BREAKS" in found[0]["summary"]
+
+
+def test_the_users_rule_about_a_device_in_neither_config_is_not_compared(monkeypatch):
+ """A scan calls this PC-050, could not check. A tick here would disagree."""
+ r = dict(rule(1, "prohibition"), node="edge-fw9")
+ _, _, _, out = stage(monkeypatch, [r], {1: (False, False)},
+ devices=({"rtr-us5"}, {"rtr-us5"}), user_supplied=True)
+ assert out["absent"] == []
+ assert len(out["not_compared"]) == 1 and "edge-fw9" in out["not_compared"][0]
+
+
+def test_our_example_rule_about_a_device_in_neither_config_is_not_applicable(monkeypatch):
+ """#402 item 1 still holds for the built-in rules: an error there made
+ every proposal on a network without rtr-us5 unverified."""
+ r = dict(rule(1, "prohibition"), node="edge-fw9")
+ _, _, _, out = stage(monkeypatch, [r], {1: (False, False)},
+ devices=({"rtr-us5"}, {"rtr-us5"}), user_supplied=False)
+ assert out["absent"] == ["1 (edge-fw9)"] and out["not_compared"] == []
+
+
+# --- _breach_changed reads both verdicts, from the real differential call ---
+
+class _Frame:
+ def __init__(self, rows):
+ self._rows = rows
+
+ def iterrows(self):
+ return enumerate(self._rows)
+
+
+class _DiffSession:
+ """Records every differential searchFilters call; answers per query."""
+
+ def __init__(self, rows_per_query):
+ self.calls = []
+ rows = iter(rows_per_query)
+ session = self
+
+ class _Q:
+ def searchFilters(self, **kwargs):
+ class _A:
+ def answer(self, **answer_kwargs):
+ session.calls.append(answer_kwargs)
+ frame = _Frame(next(rows))
+ return type("R", (), {"frame": lambda self: frame})()
+ return _A()
+ self.q = _Q()
+
+
+def _row(now, then):
+ return {"Snapshot_Action": now, "Reference_Action": then}
+
+
+@pytest.mark.parametrize("kind,rows,expected", [
+ ("prohibition", [_row("PERMIT", "DENY")], (True, False)), # newly permitted: grew
+ ("prohibition", [_row("DENY", "PERMIT")], (False, True)), # newly denied: shrank
+ ("prohibition", [_row("DENY", "PERMIT"), _row("PERMIT", "DENY")], (True, True)),
+ ("requirement", [_row("DENY", "PERMIT")], (True, False)), # newly denied: grew
+ ("requirement", [_row("PERMIT", "DENY")], (False, True)),
+ ("prohibition", [], (False, False)),
+])
+def test_breach_changed_reads_which_way_each_flow_moved(kind, rows, expected):
+ bf = _DiffSession([rows])
+ assert change_impact._breach_changed(bf, rule(1, kind)) == expected
+ assert bf.calls == [{"snapshot": change_impact.AFTER_SNAPSHOT,
+ "reference_snapshot": change_impact.BEFORE_SNAPSHOT}]
+
+
+def test_breach_changed_asks_every_query_arm():
+ r = dict(rule(1, "prohibition"), queries=[{"dstIps": "10.20.0.5"}, {"dstIps": "10.20.0.6"}])
+ bf = _DiffSession([[], [_row("PERMIT", "DENY")]])
+ assert change_impact._breach_changed(bf, r) == (True, False)
+ assert len(bf.calls) == 2
+
+
# --- how analyse_change() folds it in --------------------------------------
def _stub_pipeline(monkeypatch, *, filters=(), policy_stage=None):
@@ -137,6 +277,7 @@ def _stub_pipeline(monkeypatch, *, filters=(), policy_stage=None):
monkeypatch.setattr(change_impact, "_filter_change_findings",
lambda bf, n: (list(filters), n + len(filters)))
monkeypatch.setattr(change_impact, "_reachability_change_findings", lambda bf, n: ([], n))
+ monkeypatch.setattr(change_impact, "_devices", lambda bf: ({"rtr-us5"}, {"rtr-us5"}))
if policy_stage is not None:
monkeypatch.setattr(change_impact, "_policy_change_findings", policy_stage)
@@ -144,11 +285,11 @@ def _stub_pipeline(monkeypatch, *, filters=(), policy_stage=None):
def test_unchanged_rules_join_the_all_clear_rather_than_a_second_ch000(monkeypatch):
"""Two cards both numbered 0 would collide on id. When nothing moved,
the unchanged rules are named inside the single all-clear."""
- _stub_pipeline(monkeypatch, policy_stage=lambda bf, n: ([], ["1 (holds)", "2 (holds)"], n))
+ _stub_pipeline(monkeypatch, policy_stage=lambda bf, n, b=None, a=None: {"findings": [], "unchanged": ["1", "2"], "number": n})
results = change_impact.analyse_change("before", "after")
assert len(results) == 1
assert results[0]["status"] == "none"
- assert "1 (holds)" in results[0]["evidence"]["detail"]
+ assert "1, 2" in results[0]["evidence"]["detail"]
def test_with_real_changes_the_unchanged_rules_still_get_reported(monkeypatch):
@@ -156,17 +297,98 @@ def test_with_real_changes_the_unchanged_rules_still_get_reported(monkeypatch):
check="change_impact", severity="high", device="rtr-us5", summary="moved",
detail="d", source="s", status="found", number=1)
_stub_pipeline(monkeypatch, filters=[moved],
- policy_stage=lambda bf, n: ([], ["3 (holds)"], n))
+ policy_stage=lambda bf, n, b=None, a=None: {"findings": [], "unchanged": ["3"], "number": n})
results = change_impact.analyse_change("before", "after")
ids = [f["id"] for f in results]
assert len(ids) == len(set(ids))
- assert any(f["status"] == "none" and "3 (holds)" in f["evidence"]["detail"] for f in results)
+ assert any(f["status"] == "none" and "3." in f["evidence"]["detail"] for f in results)
+
+
+def test_rules_not_compared_become_a_real_ch051_error(monkeypatch):
+ """#406 review, item 3: the CH-051 card analyse_change() builds, not one
+ a test made up. Both lists reach it."""
+ _stub_pipeline(monkeypatch, policy_stage=lambda bf, n, b=None, a=None: {
+ "findings": [], "unchanged": [], "number": n,
+ "not_compared": ["rule 1: searchFilters failed"],
+ "one_side": ["rule 2: rtr-b is in the after configuration only"]})
+ results = change_impact.analyse_change("before", "after")
+ card = [f for f in results if f["id"] == "CH-051"]
+ assert len(card) == 1 and card[0]["status"] == "error"
+ assert "2 policy rule(s)" in card[0]["summary"]
+ assert "searchFilters failed" in card[0]["evidence"]["detail"]
+ assert "after configuration only" in card[0]["evidence"]["detail"]
+ assert not any(f["status"] == "none" and "No change detected" in f["summary"] for f in results)
+
+
+def test_a_failed_filter_comparison_is_ch050_never_no_change(monkeypatch):
+ """#406 review: silencing this path turned a failed compareFilters into
+ a green "No change detected"."""
+ _stub_pipeline(monkeypatch, policy_stage=lambda bf, n, b=None, a=None: {
+ "findings": [], "unchanged": [], "number": n})
+
+ def broken(bf, n):
+ raise RuntimeError("compareFilters failed")
+ monkeypatch.setattr(change_impact, "_filter_change_findings", broken)
+ results = change_impact.analyse_change("before", "after")
+ assert [f["id"] for f in results if f["status"] == "error"] == ["CH-050"]
+ assert not any(f["status"] == "none" for f in results)
+
+
+def test_a_long_diff_never_folds_away_a_broken_policy_rule():
+ """#406 review: 60 moved lines plus a removed DNS line hid "BREAKS rule 4"
+ inside "15 further change(s)"."""
+ mk = change_impact.findings.make_finding
+ lines = [mk(check="change_impact", severity="high", device="rtr-us5",
+ summary=f"line {i}", detail="d", source="rtr-us5:acl_in",
+ status="found", number=i) for i in range(1, 61)]
+ breaks = mk(check="change_impact", severity="high", device="rtr-us5",
+ summary="This change BREAKS policy rule 4", detail="d",
+ source="your policy; rule 4", status="found", number=61)
+ kept = change_impact._cap(lines + [breaks])
+ assert len(kept) == 49 and kept[-1]["id"] == "CH-049", "48 kept, plus the summary"
+ assert any("BREAKS policy rule 4" in f["summary"] for f in kept[:-1])
+ ids = [f["id"] for f in kept[:-1]]
+ assert ids == sorted(ids), "what is kept stays in the order found"
+
+
+def test_a_capped_list_is_numbered_one_to_forty_eight_whatever_was_kept():
+ """The cap keeps the MOST IMPORTANT 48, not the first 48, and a kept
+ finding kept the number it was given when it was found. With mixed
+ severities the 49th finding found is a high one, is kept, and was CH-049:
+ the same id as the card that summarises the rest. The ids this module
+ reserves (CH-049 and CH-050 to CH-054) must be free whatever was kept, so
+ a capped list is renumbered in the order found. (#406 review: the test
+ above uses one severity, where the cap keeps the first 48, so it cannot
+ see this.)"""
+ mk = change_impact.findings.make_finding
+ found = [mk(check="change_impact", severity="high" if i % 2 == 1 else "medium",
+ device="rtr-us5", summary=f"line {i}", detail="d",
+ source="rtr-us5:acl_in", status="found", number=i) for i in range(1, 61)]
+ kept = change_impact._cap(found)
+ ids = [f["id"] for f in kept]
+ assert len(ids) == len(set(ids)), f"duplicate ids: {sorted(i for i in set(ids) if ids.count(i) > 1)}"
+ assert ids == [f"CH-{n:03d}" for n in range(1, 49)] + ["CH-049"]
+ nums = [int(f["summary"].split()[1]) for f in kept[:-1]]
+ assert nums == sorted(nums), "what is kept stays in the order found"
+
+
+def test_a_capped_list_keeps_the_worst_severity_not_the_first_found():
+ """55 medium lines found first, then 5 high ones: the highs must survive
+ the cap. Keeping the first 48 would fold all five away."""
+ mk = change_impact.findings.make_finding
+ found = [mk(check="change_impact", severity="high" if i > 55 else "medium",
+ device="rtr-us5", summary=f"line {i}", detail="d",
+ source="rtr-us5:acl_in", status="found", number=i) for i in range(1, 61)]
+ kept = change_impact._cap(found)
+ kept_highs = [f for f in kept[:-1] if f["severity"] == "high"]
+ assert len(kept_highs) == 5, "every high finding is kept"
+ assert kept[-1]["severity"] == "medium", "the summary is as bad as the worst it hides"
def test_the_policy_is_cleared_even_when_the_stage_raises(monkeypatch):
"""A policy outliving the call that supplied it would answer the next
caller's question with someone else's rules."""
- def boom(bf, n):
+ def boom(bf, n, b=None, a=None):
raise RuntimeError("policy stage failed")
_stub_pipeline(monkeypatch, policy_stage=boom)
sentinel = policy_module.load_policy({"device": "rtr-us5", "policy_compliance": []})
@@ -193,3 +415,38 @@ def test_live_a_swap_is_reported_as_both_a_fix_and_a_break(tmp_path):
assert any("BREAKS policy rule 4" in s for s in summaries), summaries
ids = [f["id"] for f in results]
assert len(ids) == len(set(ids))
+
+
+@needs_batfish
+def test_live_a_pure_deny_narrows_a_breach_and_raises_no_banner():
+ """#406 review, item 1, through the public entry point named there."""
+ from ai.propose import propose_change
+ out = propose_change("block 10.10.10.5 to 10.20.0.5 on tcp/22 on rtr-us5",
+ FIXTURES / "rtr-us5-insecure")
+ assert out["warning"] is False, out["answer"]
+ assert any("narrows how policy rule 2" in f["summary"] and f["severity"] == "low"
+ for f in out["impact"])
+ assert not any(f["severity"] == "high" for f in out["impact"])
+
+
+@needs_batfish
+def test_live_a_device_named_in_capitals_still_reports_the_breach(tmp_path):
+ """#406 review, item 2: `hostname RTR-US5`, policy `device: RTR-US5`, the
+ change removes DNS. Main found the breach; the first version of this
+ branch filed it under a tick."""
+ secure = (FIXTURES / "rtr-us5-secure" / "configs" / "rtr-us5.cfg").read_text()
+ before = secure.replace("hostname rtr-us5", "hostname RTR-US5")
+ after = "\n".join(line for line in before.splitlines() if "eq domain" not in line) + "\n"
+ for name, text in (("before", before), ("after", after)):
+ (tmp_path / name / "configs").mkdir(parents=True)
+ (tmp_path / name / "configs" / "rtr-us5.cfg").write_text(text)
+ policy = policy_module.load_policy({"device": "RTR-US5", "policy_compliance": [{
+ "number": 1, "description": "DNS to the approved resolver must work",
+ "kind": "requirement", "filter": "acl_in", "violation_severity": "high",
+ "violation_summary": "DNS blocked",
+ "queries": [{"srcIps": "10.10.10.0/24", "dstIps": "218.8.104.58",
+ "ipProtocols": ["udp"], "dstPorts": "53"}]}]})
+ results = change_impact.analyse_change(tmp_path / "before", tmp_path / "after",
+ policy=policy)
+ assert any("BREAKS policy rule 1" in f["summary"] and f["severity"] == "high"
+ for f in results), [f["summary"] for f in results]
diff --git a/tests/test_change_impact.py b/tests/test_change_impact.py
index d6bde60..82246bf 100644
--- a/tests/test_change_impact.py
+++ b/tests/test_change_impact.py
@@ -121,7 +121,10 @@ def fake_flows(bf, number):
# tests keep meaning what they meant before it existed. Its own tests are
# in tests/test_change_against_policy.py.
monkeypatch.setattr(change_impact, "_policy_change_findings",
- lambda bf, number: ([], [], number))
+ lambda bf, number, before=None, after=None: {"findings": [], "number": number})
+ # The same device on both sides, so #403's device check finds no change
+ # and these tests keep meaning what they meant before it existed.
+ monkeypatch.setattr(change_impact, "_devices", lambda bf: ({"rtr-us5"}, {"rtr-us5"}))
def _a_change(number=1):
diff --git a/tests/test_change_impact_coverage.py b/tests/test_change_impact_coverage.py
new file mode 100644
index 0000000..b5f5cac
--- /dev/null
+++ b/tests/test_change_impact_coverage.py
@@ -0,0 +1,268 @@
+"""What change impact looked at, and what it says it looked at (#403, #402).
+
+Every test here is a case where the old answer was confident and wrong:
+
+ #403 a router deleted between before and after -> "No change detected"
+ #402 item 1 a network without rtr-us5 -> every proposal "could not be verified"
+ #402 item 3 a policy of access_control rules -> silence about them
+ #402 item 5 four mutants that only a live test (skipped on CI) could catch
+ #402 item 6 a failed reachability stage silently skipping the policy stage,
+ and ids that a long diff could collide with
+"""
+
+import shutil
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+
+from analysis import change_impact, findings, policy as policy_module
+from analysis.checks import policy_compliance as pc
+from conftest import needs_batfish
+
+FIXTURES = Path(__file__).parent / "fixtures"
+
+
+def _pipeline(monkeypatch, *, devices=({"rtr-us5"}, {"rtr-us5"}), filters=(),
+ flows_raise=False, stage=None):
+ monkeypatch.setattr(change_impact, "connect", lambda host=None: SimpleNamespace(
+ set_network=lambda n: None, set_snapshot=lambda n: None))
+ monkeypatch.setattr(change_impact, "load_snapshot", lambda *a, **k: None)
+ monkeypatch.setattr(change_impact, "find_parse_problems", lambda bf: [])
+ monkeypatch.setattr(change_impact, "_devices", lambda bf: devices)
+ monkeypatch.setattr(change_impact, "_filter_change_findings",
+ lambda bf, n: (list(filters), n + len(filters)))
+
+ def flows(bf, n):
+ if flows_raise:
+ raise RuntimeError("differentialReachability exploded")
+ return [], n
+ monkeypatch.setattr(change_impact, "_reachability_change_findings", flows)
+ if stage is not None:
+ monkeypatch.setattr(change_impact, "_policy_change_findings", stage)
+ else:
+ monkeypatch.setattr(change_impact, "_policy_change_findings",
+ lambda bf, n, b=None, a=None: {"findings": [], "number": n})
+
+
+def _found(n, severity="high"):
+ return findings.make_finding(check="change_impact", severity=severity, device="r",
+ summary=f"change {n}", detail="d", source="s",
+ status="found", number=n)
+
+
+# --- #403: devices on one side only ------------------------------------------
+
+def test_a_removed_device_is_a_change_never_no_change_detected(monkeypatch):
+ _pipeline(monkeypatch, devices=({"rtr-a", "rtr-b"}, {"rtr-a"}))
+ results = change_impact.analyse_change("before", "after")
+ assert not any("No change detected" in f["summary"] for f in results)
+ removed = [f for f in results if "removed" in f["summary"]]
+ assert len(removed) == 1 and removed[0]["status"] == "found"
+ assert removed[0]["device"] == "rtr-b"
+
+
+def test_an_added_device_is_a_change(monkeypatch):
+ _pipeline(monkeypatch, devices=({"rtr-a"}, {"rtr-a", "rtr-c"}))
+ results = change_impact.analyse_change("before", "after")
+ added = [f for f in results if "added" in f["summary"]]
+ assert len(added) == 1 and added[0]["status"] == "found" and added[0]["device"] == "rtr-c"
+
+
+def test_unknown_device_lists_are_an_error_never_no_change(monkeypatch):
+ """None means "we could not find out", which is not "nothing changed"."""
+ _pipeline(monkeypatch, devices=({"rtr-a"}, None))
+ results = change_impact.analyse_change("before", "after")
+ assert [f["status"] for f in results] == ["error"]
+ assert results[0]["id"] == "CH-052"
+
+
+# --- #402 item 1: rules about devices that are not there ----------------------
+
+def _rule(n, node="rtr-us5", kind="prohibition"):
+ return {"number": n, "kind": kind, "node": node, "filter": "acl_in",
+ "description": f"rule {n}", "violation_severity": "high",
+ "violation_summary": "x", "queries": [{"dstIps": "10.20.0.5"}]}
+
+
+def test_a_rule_about_a_device_in_neither_config_is_not_applicable_not_an_error(monkeypatch):
+ monkeypatch.setattr(pc, "rules_in_use", lambda: ([_rule(1)], "built-in", False))
+ monkeypatch.setattr(change_impact, "_rule_breached",
+ lambda *a: pytest.fail("must not ask Batfish about an absent device"))
+ out = change_impact._policy_change_findings(object(), 1, {"edge-fw1"}, {"edge-fw1"})
+ assert out["absent"] == ["1 (rtr-us5)"]
+ assert out["not_compared"] == [] and out["findings"] == []
+
+
+def test_a_stranger_network_is_verified_rather_than_could_not_compare(monkeypatch):
+ """Arsh's repro, end to end through analyse_change: five built-in rules
+ about rtr-us5 on a network called edge-fw1 produced CH-051 x5 errors, so
+ propose_change() said verified=False on every request."""
+ monkeypatch.setattr(pc, "rules_in_use", lambda: ([_rule(n) for n in range(1, 6)], "built-in", False))
+ _pipeline(monkeypatch, devices=({"edge-fw1"}, {"edge-fw1"}),
+ stage=change_impact._policy_change_findings)
+ results = change_impact.analyse_change("before", "after")
+ assert all(f["status"] != "error" for f in results), results
+ assert "neither configuration" in results[0]["evidence"]["detail"]
+
+
+def test_a_rule_about_a_device_on_one_side_only_cannot_be_compared(monkeypatch):
+ monkeypatch.setattr(pc, "rules_in_use", lambda: ([_rule(1, node="rtr-b")], "built-in", False))
+ out = change_impact._policy_change_findings(object(), 1, {"rtr-a", "rtr-b"}, {"rtr-a"})
+ assert len(out["one_side"]) == 1 and "before configuration only" in out["one_side"][0]
+
+
+# --- #402 item 3: say what was not compared ------------------------------------
+
+def _policy(**sections):
+ return policy_module.load_policy({"device": "rtr-us5", **sections})
+
+
+AC_RULE = {"node": "rtr-us5", "description": "8443 must be denied", "filter": "acl_in",
+ "headers": {"dstPorts": "8443"}, "expected": "DENY",
+ "violation_severity": "high", "violation_summary": "8443 open"}
+
+
+def test_rules_in_sections_change_impact_does_not_compare_are_named(monkeypatch):
+ policy_module.set_active_policy(_policy(access_control=[AC_RULE],
+ policy_compliance=[_rule(1)]))
+ try:
+ _, _, _, note = change_impact._policy_coverage()
+ finally:
+ policy_module.clear_active_policy()
+ assert note.startswith("NOT compared")
+ assert "1 access_control" in note
+
+
+def test_a_policy_with_no_policy_compliance_rules_says_so_loudly():
+ policy_module.set_active_policy(_policy(access_control=[AC_RULE]))
+ try:
+ rules, _, _, note = change_impact._policy_coverage()
+ finally:
+ policy_module.clear_active_policy()
+ assert rules == []
+ assert "no policy_compliance rules" in note and "no rule of yours was checked" in note
+
+
+# --- #402 item 5: the mutants a fake session can catch -------------------------
+
+class _Session:
+ """Records every question _rule_breached asks, answers from a table."""
+
+ def __init__(self, breached_arms):
+ self.snapshots, self.asked, self.breached = [], [], breached_arms
+ self.current = None
+ self.q = SimpleNamespace(searchFilters=self._search)
+
+ def set_snapshot(self, name):
+ self.current = name
+ self.snapshots.append(name)
+
+ def _search(self, nodes, filters, action, headers):
+ self.asked.append((self.current, action, str(headers)))
+ rows = 1 if (self.current, str(headers)) in self.breached else 0
+ return SimpleNamespace(answer=lambda **k: SimpleNamespace(
+ frame=lambda: SimpleNamespace(__len__=lambda s: rows, iloc=[None] * rows)))
+
+
+def test_rule_breached_switches_to_the_snapshot_it_is_asked_about(monkeypatch):
+ calls = []
+ monkeypatch.setattr(pc, "_search", lambda bf, n, f, a, h: calls.append((bf.current, a)) or [])
+ s = _Session(set())
+ change_impact._rule_breached(s, change_impact.AFTER_SNAPSHOT, _rule(1))
+ assert s.snapshots == [change_impact.AFTER_SNAPSHOT]
+ assert calls[0][0] == change_impact.AFTER_SNAPSHOT
+
+
+@pytest.mark.parametrize("kind", ["prohibition", "requirement"])
+def test_rule_breached_asks_with_the_checks_own_action_for_the_kind(monkeypatch, kind):
+ seen = []
+ monkeypatch.setattr(pc, "_search", lambda bf, n, f, a, h: seen.append(a) or [])
+ change_impact._rule_breached(_Session(set()), change_impact.BEFORE_SNAPSHOT, _rule(1, kind=kind))
+ assert seen == [pc.ACTION_BY_KIND[kind]]
+
+
+def test_rule_breached_asks_every_arm_not_just_the_first(monkeypatch):
+ """Survived without Batfish before: only the second arm breaches."""
+ r = _rule(1)
+ r["queries"] = [{"dstIps": "10.20.0.5"}, {"dstIps": "10.20.0.6"}]
+ monkeypatch.setattr(pc, "_search", lambda bf, n, f, a, h: [1] if h["dstIps"] == "10.20.0.6" else [])
+ assert change_impact._rule_breached(_Session(set()), change_impact.BEFORE_SNAPSHOT, r) is True
+
+
+def test_the_policy_passed_in_is_the_policy_evaluated(monkeypatch):
+ """Survived both with and without Batfish: set_active_policy(None) in
+ place of set_active_policy(policy)."""
+ mine = _policy(policy_compliance=[_rule(7)])
+ seen = []
+
+ def stage(bf, n, b=None, a=None):
+ seen.append(policy_module.active_policy())
+ return {"findings": [], "number": n}
+ _pipeline(monkeypatch, stage=stage)
+ change_impact.analyse_change("before", "after", policy=mine)
+ assert seen == [mine]
+ assert policy_module.active_policy() is None, "and it is cleared afterwards"
+
+
+# --- #402 item 6: stages are independent, ids cannot collide -------------------
+
+def test_the_policy_stage_runs_when_reachability_fails(monkeypatch):
+ ran = []
+
+ def stage(bf, n, b=None, a=None):
+ ran.append(True)
+ return {"findings": [_found(n)], "number": n + 1}
+ _pipeline(monkeypatch, flows_raise=True, stage=stage)
+ results = change_impact.analyse_change("before", "after")
+ assert ran == [True]
+ assert {f["status"] for f in results} == {"found", "error"}
+
+
+def test_a_long_diff_cannot_collide_with_the_reserved_ids(monkeypatch):
+ """60 filter findings used to be numbered CH-001..CH-060, two of which
+ are this module's own error cards."""
+ _pipeline(monkeypatch, filters=[_found(n) for n in range(1, 61)], flows_raise=True)
+ results = change_impact.analyse_change("before", "after")
+ ids = [f["id"] for f in results]
+ assert len(ids) == len(set(ids)), ids
+ assert "CH-049" in ids and "further change" in next(f for f in results if f["id"] == "CH-049")["summary"]
+ assert max(int(i[3:]) for i in ids if i != "CH-049" and not i.startswith("CH-05")) <= 48
+
+
+def test_the_change_impact_cli_prints_every_policy_note(monkeypatch, capsys, tmp_path):
+ p = tmp_path / "p.json"
+ p.write_text('{"device": "rtr-us5", "policy_compliance": [{"description": "d", '
+ '"kind": "prohibition", "filter": "acl_in", "violation_severity": "high", '
+ '"violation_summary": "s", "queries": [{"dstIps": "10.20.0.5"}]}]}')
+ monkeypatch.setattr(change_impact, "analyse_change", lambda *a, **k: [])
+ monkeypatch.setattr("sys.argv", ["change_impact", "b", "a", str(p)])
+ change_impact.main()
+ assert any(line.startswith("note:") for line in capsys.readouterr().out.splitlines()), \
+ "the assigned rule number is a note the CLI used to drop"
+
+
+# --- live, against real Batfish ---------------------------------------------
+
+@needs_batfish
+def test_live_removing_a_device_is_reported(tmp_path):
+ """#403's own repro: multi-device-10 minus rtr-dev009."""
+ after = tmp_path / "after"
+ shutil.copytree(FIXTURES / "multi-device-10", after)
+ (after / "configs" / "rtr-dev009.cfg").unlink()
+ results = change_impact.analyse_change(FIXTURES / "multi-device-10", after)
+ assert not any("No change detected" in f["summary"] for f in results)
+ assert any(f["device"] == "rtr-dev009" and "removed" in f["summary"] for f in results)
+
+
+@needs_batfish
+def test_live_a_stranger_network_proposal_is_verified(tmp_path):
+ """#402 item 1's repro: rtr-us5 renamed edge-fw1. Before this, every
+ built-in rule came back as CH-051 and verified was False."""
+ for side in ("before", "after"):
+ d = tmp_path / side / "configs"
+ d.mkdir(parents=True)
+ text = (FIXTURES / "rtr-us5-secure" / "configs" / "rtr-us5.cfg").read_text()
+ d.joinpath("edge-fw1.cfg").write_text(text.replace("hostname rtr-us5", "hostname edge-fw1"))
+ results = change_impact.analyse_change(tmp_path / "before", tmp_path / "after")
+ assert all(f["status"] != "error" for f in results), [f["summary"] for f in results]
diff --git a/tests/test_change_score.py b/tests/test_change_score.py
new file mode 100644
index 0000000..0e6aab1
--- /dev/null
+++ b/tests/test_change_score.py
@@ -0,0 +1,35 @@
+"""How big is a change? (#420)"""
+from analysis import change_impact, findings
+
+
+def f(sev, status="found", n=1):
+ if status == "error":
+ return findings.error_finding(check="change_impact", summary="s", detail="d", source="x", number=50)
+ return findings.make_finding(check="change_impact", severity=sev, device="r", summary="s",
+ detail="d", source="x", status=status, number=n)
+
+
+def test_opening_counts_more_than_closing():
+ assert change_impact.score([f("high")])["score"] > change_impact.score([f("medium")])["score"]
+
+
+def test_the_breakdown_explains_the_number():
+ s = change_impact.score([f("high"), f("medium", n=2), f("low", n=3)])
+ assert s["score"] == 6 and s["explanation"] == "1 high x3 + 1 medium x2 + 1 low x1 = 6"
+
+
+def test_incomplete_never_scores_small():
+ s = change_impact.score([f("low"), f(None, "error")])
+ assert s["band"] == "incomplete" and s["complete"] is False
+ assert "may be larger" in s["explanation"]
+
+
+def test_no_change_is_zero_and_none():
+ s = change_impact.score([f("low", "none", 0)])
+ assert s["score"] == 0 and s["band"] == "none" and s["complete"]
+
+
+def test_bands():
+ assert change_impact.score([f("low")])["band"] == "small"
+ assert change_impact.score([f("high"), f("high", n=2)])["band"] == "moderate"
+ assert change_impact.score([f("high", n=i) for i in range(1, 5)])["band"] == "large"