Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
1b2a665
change_impact: carry a structured 'direction' on every finding (#402)
patelankeet2 Oct 18, 2026
933118e
propose: accept and thread a staged policy through, fix the warning w…
patelankeet2 Oct 18, 2026
d005831
propose tests: pin that the real policy value actually reaches analys…
patelankeet2 Oct 18, 2026
f4668d0
web: pass the staged policy to propose-a-change (#402)
patelankeet2 Oct 18, 2026
273a496
Merge main into fix/402-propose-policy-and-direction
patelankeet2 Nov 5, 2026
1f91574
Pin that analyse_change's own install actually uses the supplied policy
patelankeet2 Nov 5, 2026
0ed8ece
Key the propose-pane banner on the real direction, not on warning alone
patelankeet2 Nov 5, 2026
de42070
Merge remote-tracking branch 'origin/main' into fix/402-propose-polic…
patelankeet2 Nov 10, 2026
edf4929
Add opens_access to every place the response shape is documented or b…
patelankeet2 Nov 10, 2026
af51f87
Correct the docstring: a closing is not always medium (#405 review, A…
patelankeet2 Nov 10, 2026
c99f53e
Fix the propose_and_scan comment and test docstring: no impact list i…
patelankeet2 Nov 10, 2026
12be902
Add Arsh's two regression tests for an unclassified direction (#405 r…
patelankeet2 Nov 10, 2026
7ea7c16
Strengthen the direction wording test to also assert the field (#405 …
patelankeet2 Nov 10, 2026
8a6cd01
Mention the serious-closing warning in the user guide too (#405 revie…
patelankeet2 Nov 10, 2026
08733c9
Propose A-5: one vocabulary for direction, in the field and in the wo…
patelankeet2 Nov 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 70 additions & 12 deletions ai/propose.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
"impact": list[dict], # F-1 findings from change_impact, [] if none
"verified": bool,
"warning": bool,
"opens_access": bool, # see the warning/opens_access split below
"grounded": bool,
"answer": str}

Expand Down Expand Up @@ -49,11 +50,15 @@
compareFilters/differentialReachability comparison Shubham's feature
(#30) already runs for a given before/after pair. That module already
grades severity by direction: a change that newly OPENS traffic is
`high` (silent -- nothing breaks, so nobody looks), a change that
CLOSES traffic is `medium` (loud -- someone notices). `warning=True`
here means exactly "a high-severity finding is present in that diff" --
running the tool the project already trusts, twice, and reading its
answer. No new judgement is introduced.
`high` (silent -- nothing breaks, so nobody looks), a filter or
reachability change that CLOSES traffic is `medium` (loud -- someone
notices). A CLOSING that breaches a written policy requirement is the
one exception (#402): it carries that rule's own `violation_severity`
instead, which can be `high` -- this PR exists because that case was
reported with the wrong banner. `warning=True` here means exactly "a
high-severity finding is present in that diff" -- running the tool
the project already trusts, twice, and reading its answer. No new
judgement is introduced.

This module does NOT audit the proposed state the way change_impact's
own docstring flags as unbuilt ("it will not tell you that the after
Expand Down Expand Up @@ -311,18 +316,31 @@ def _refuse(reason: str) -> Dict[str, Any]:
"impact": [],
"verified": False,
"warning": False,
"opens_access": False,
"grounded": False,
"answer": reason,
}


def propose_change(
request: str, before_dir: "str | Path", host: "str | None" = None
request: str,
before_dir: "str | Path",
host: "str | None" = None,
policy: Any = None,
) -> Dict[str, Any]:
"""Propose ONE config change from plain English, and simulate it against
a scratch copy of `before_dir` -- see the module docstring for the shape,
the scope limit, and why neither step here calls a model.

`policy` (#402): the user's staged policy, or None to compare against
Netwise's built-in example rules. Passed straight through to
`analyse_change()`, which already installs and clears it -- this
function does no policy handling of its own. Before this parameter
existed, propose always used the built-in rules regardless of what the
caller had staged, and `policy_compliance`'s own evidence then correctly
-- but misleadingly -- reported "no policy file was supplied", because
nothing here had ever told it otherwise.

Never raises. Every failure -- an unparseable request, an unknown
device, an unreachable Batfish -- is a refusal, not an exception, the
same convention `ai/query.py`'s `answer_question()` and
Expand Down Expand Up @@ -455,7 +473,7 @@ def propose_change(
after_dir = Path(tmp) / "after"
shutil.copytree(before_dir, after_dir)
(after_dir / "configs" / cfg_path.name).write_text(new_cfg_text, encoding="utf-8")
impact = change_impact.analyse_change(before_dir, after_dir, host=host)
impact = change_impact.analyse_change(before_dir, after_dir, host=host, policy=policy)

# Shown on the dashboard as a before/after diff of the affected filter,
# not just the one generated line -- a line in isolation does not say
Expand Down Expand Up @@ -489,15 +507,48 @@ def propose_change(
# know" reads as LESS alarming than the truth. Gating warning on
# verified silently swallowed the proof in that direction.
verified = all(f["status"] != "error" for f in impact)
warning = any(
f["status"] == "found" and f["severity"] == "high" for f in impact
high_severity_found = [
f for f in impact if f["status"] == "found" and f["severity"] == "high"
]
warning = bool(high_severity_found)
# Returned alongside `warning` rather than left for the caller to
# re-derive from `impact` (#405 review, Arsh + Samika). `web/static/
# app.js` used to render every `warning=True` response with "This change
# opens access that is currently blocked", unconditionally -- correct
# when `warning` meant only an opening, wrong the moment it could also
# mean a closing breach that is still a serious effect. The same
# direction check `warning_text` below already makes, hoisted into the
# response so the one place that decides it is this function, not every
# renderer that reads the response.
opens_access = warning and all(
f.get("direction") == "opening" for f in high_severity_found
)

if warning:
# The trigger stays severity-high regardless of direction (#402) --
# under-warning on a genuinely serious change is the dangerous
# direction, same reasoning as the verified/warning split above. What
# changes is the WORDING: "opens access" is only true when every
# triggering finding actually opens something. A high-severity
# POLICY finding can instead be a requirement breached, which CLOSES
# access that should stay open -- the opposite claim, found by
# @ARSH871-bot reviewing #389. Checking the real `direction` field
# (analysis/change_impact.py) rather than assuming severity implies
# it is what avoids repeating that mistake here.
if opens_access:
warning_text = (
"Warning: simulating this change shows it newly opens access "
"that was previously blocked -- see the impact list."
)
else:
warning_text = (
"Warning: simulating this change has a serious effect that "
"is not simply opening access -- see the impact list for "
"exactly what changed."
)
answer = (
f"Generated: {rule_line}, on {device}'s {acl_name!r} filter. "
"Warning: simulating this change shows it newly opens access "
"that was previously blocked -- see the impact list."
f"{warning_text}"
)
if not verified:
answer += (
Expand Down Expand Up @@ -529,6 +580,7 @@ def propose_change(
"impact": impact,
"verified": verified,
"warning": warning,
"opens_access": opens_access,
"grounded": True,
"answer": answer,
}
Expand Down Expand Up @@ -602,7 +654,13 @@ def propose_and_scan(
"findings": [] and "grounded": False -- there is nothing to scan when
there was nothing to propose.
"""
base = propose_change(request, before_dir, host=host)
# Every call to propose_change() should see the same policy regardless
# of caller (#405 review, Samika). Only base["proposed_change"] and
# base["answer"] are read below -- neither depends on policy -- so this
# has no visible effect on what this function returns today. Kept
# anyway: passing the wrong policy into a shared helper is the wrong
# default even when nothing currently reads the result.
base = propose_change(request, before_dir, host=host, policy=policy)
change = base["proposed_change"]
if change is None:
return {
Expand Down
131 changes: 85 additions & 46 deletions analysis/change_impact.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,36 @@ def _direction(line_action: str) -> str:
return "opened" if str(line_action).upper() == "PERMIT" else "closed"


#: One vocabulary for "direction" across all three finding producers here,
#: carried on the finding as an extra key (see _attach_direction()) rather
#: than left for a caller to infer from severity or parse out of `detail`.
#:
#: WHY THIS EXISTS (#402, raised by @ARSH871-bot reviewing #389)
#: For the filter and reachability producers, severity already IS a
#: faithful proxy for direction (_OPENING_SEVERITY/_CLOSING_SEVERITY are
#: assigned directly from "opened"/"closed"). For the POLICY producer it
#: is not: severity there tracks whether a breach was newly created, and
#: a newly-created breach can be a prohibition violated (access opens)
#: OR a requirement violated (access closes -- something that must flow
#: no longer does). A caller that assumes "severity high means opened"
#: -- ai/propose.py did -- gets the direction backwards for exactly the
#: second case. A real field removes the need to infer it at all.
_DIRECTION_NORMAL_FORM = {
"opened": "opening",
"closed": "closing",
"changed": "changed",
"opening": "opening",
"tightening": "closing",
}


def _attach_direction(finding: Dict[str, Any], raw_direction: str) -> Dict[str, Any]:
"""Set the normalised `direction` key on `finding` and return it, so
every append site reads as one line rather than three."""
finding["direction"] = _DIRECTION_NORMAL_FORM[raw_direction]
return finding


# Which trace dispositions count as the flow having got through.
#
# DELIBERATELY WIDER THAN analysis/checks/routing.py'S SET, which excludes
Expand Down Expand Up @@ -239,23 +269,26 @@ def _filter_change_findings(
severity = _CLOSING_SEVERITY

results.append(
findings.make_finding(
check=CHECK_NAME,
severity=severity,
device=node,
summary=summary,
# Both lines, verbatim from Batfish. The reader needs the pair
# to see the change; one line alone is not a diff.
detail=(
f"After: {row['Line_Content']} (line {row['Line_Index']}, "
f"{row['Line_Action']}). "
f"Before: {row['Reference_Line_Content']} "
f"(line {row['Reference_Line_Index']}). "
"Both match at least one common flow and treat it differently."
_attach_direction(
findings.make_finding(
check=CHECK_NAME,
severity=severity,
device=node,
summary=summary,
# Both lines, verbatim from Batfish. The reader needs the
# pair to see the change; one line alone is not a diff.
detail=(
f"After: {row['Line_Content']} (line {row['Line_Index']}, "
f"{row['Line_Action']}). "
f"Before: {row['Reference_Line_Content']} "
f"(line {row['Reference_Line_Index']}). "
"Both match at least one common flow and treat it differently."
),
source=f"{node}:{filter_name}",
status="found",
number=number,
),
source=f"{node}:{filter_name}",
status="found",
number=number,
direction,
)
)
number += 1
Expand Down Expand Up @@ -297,19 +330,22 @@ def _reachability_change_findings(
severity = _OPENING_SEVERITY

results.append(
findings.make_finding(
check=CHECK_NAME,
severity=severity,
device=_device_of(row["Flow"]),
summary=summary,
detail=(
f"Flow {row['Flow']}. "
f"After: {_first_trace(row['Snapshot_Traces'])}. "
f"Before: {_first_trace(row['Reference_Traces'])}."
_attach_direction(
findings.make_finding(
check=CHECK_NAME,
severity=severity,
device=_device_of(row["Flow"]),
summary=summary,
detail=(
f"Flow {row['Flow']}. "
f"After: {_first_trace(row['Snapshot_Traces'])}. "
f"Before: {_first_trace(row['Reference_Traces'])}."
),
source="differentialReachability",
status="found",
number=number,
),
source="differentialReachability",
status="found",
number=number,
direction,
)
)
number += 1
Expand Down Expand Up @@ -436,25 +472,28 @@ def _policy_change_findings(bf: Any, number: int) -> Tuple[List[Dict[str, Any]],
not_compared.append(f"{_rule_label(rule)}: more rule changes than finding numbers")
continue
results.append(
findings.make_finding(
check=CHECK_NAME,
severity=(rule.get("violation_severity", "high") if created else "low"),
device=rule["node"],
summary=(
f"This change BREAKS {_rule_label(rule)}"
if created else
f"This change FIXES {_rule_label(rule)}"
),
detail=(
f"{rule['kind']} on {rule['node']} filter {rule['filter']}: "
f"{'held before, breached after' if created else 'breached before, holds after'}. "
f"Direction: {direction}. "
+ (f"Violation if applied: {rule.get('violation_summary')}." if created
else "The breach this rule reports is gone in the proposed config.")
_attach_direction(
findings.make_finding(
check=CHECK_NAME,
severity=(rule.get("violation_severity", "high") if created else "low"),
device=rule["node"],
summary=(
f"This change BREAKS {_rule_label(rule)}"
if created else
f"This change FIXES {_rule_label(rule)}"
),
detail=(
f"{rule['kind']} on {rule['node']} filter {rule['filter']}: "
f"{'held before, breached after' if created else 'breached before, holds after'}. "
f"Direction: {direction}. "
+ (f"Violation if applied: {rule.get('violation_summary')}." if created
else "The breach this rule reports is gone in the proposed config.")
),
source=f"{policy_label}; rule {rule.get('number', '?')}",
status="found",
number=number,
),
source=f"{policy_label}; rule {rule.get('number', '?')}",
status="found",
number=number,
direction,
)
)
number += 1
Expand Down
46 changes: 46 additions & 0 deletions docs/finding-format.md
Original file line number Diff line number Diff line change
Expand Up @@ -512,3 +512,49 @@ somewhere later reports it as agreed, sourced from nothing.
>
> **A ⬜ here is not a disagreement.** It is the absence of a recorded answer,
> which is the only state this file is able to distinguish from agreement.

### A-5 - one vocabulary for `direction`, in the field and in the words

**PROPOSED, not ratified.**

The structural `direction` field already uses three values in code -
`"opening"`, `"closing"`, `"changed"` - via `_DIRECTION_NORMAL_FORM` in
`analysis/change_impact.py`, and `opens_access` in `ai/propose.py` already
relies on exactly those three. That part is a de facto standard: nothing
here proposes changing it, only recording that it is agreed.

What is not settled is the human-readable `"Direction: X"` phrase inside
`evidence.detail`, which still uses a second vocabulary for the same fact.
`analysis/change_impact.py:488` writes `"Direction: tightening"` for the
same case the structural field calls `"closing"`: `_RULE_DIRECTION` maps a
policy requirement newly breached to the raw value `"tightening"`, which
`_attach_direction()` then normalises to `"closing"` on the field, while the
sentence a person reads still says "tightening". PR #406 (not yet merged)
separately writes `"Direction: changed."` for its own case, a third,
independently chosen word for a fact the field already has a name for.

This proposes ratifying `opening` / `closing` / `changed` as the single
vocabulary for BOTH the structural field and the `"Direction: X"` phrasing,
one set of words rather than two that happen to describe the same thing.

**Not changing `analysis/change_impact.py:488`'s actual wording in this
PR.** That edit, replacing `tightening` with `closing` in the sentence
itself, is exactly the code change this table is meant to gate, the same
discipline A-4 uses for `cve_mapping`: the agreement is recorded before the
behaviour changes, not after.

| Member | Why it touches them | Agreed |
|---|---|---|
| **Arsh** | Raised the inconsistency on #405's review; owns the shared pipeline this field flows through | ⬜ |
| **Ankeet** | `ai/explain.py` reads finding text when building an explanation, and a sentence using one word while the field uses another is exactly the kind of mismatch that module is built to not repeat | ⬜ |
| **Shubham** | Owns `analysis/change_impact.py`, where `_RULE_DIRECTION` and the `"Direction: tightening"` wording live | ⬜ |
| **Samika** | `opens_access` in `ai/propose.py` already reads the structural field; the dashboard banner is downstream of whichever wording this settles on | ⬜ |

> **NOT RATIFIED - zero of four, as of 11 November.** The structural field's
> three values are already relied on and are not what this table gates.
> **`"Direction: tightening"` must not be changed to `"Direction: closing"`
> in `analysis/change_impact.py`, and #406's `"Direction: changed"` must not
> be treated as settled, until this table is full.**
>
> **A ⬜ here is not a disagreement.** It is the absence of a recorded answer,
> which is the only state this file is able to distinguish from agreement.
3 changes: 2 additions & 1 deletion docs/user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,8 @@ Netwise will:
2. Generate a candidate config line.
3. Apply it to a **disposable copy** of your network model and re-analyse.
4. Report what actually changed — and **warn** if the change opens access
that was previously blocked.
that was previously blocked, or closes access your policy says must
stay open.

**Nothing is ever applied to your device.** The generated line is text for
you to review. Your uploaded config is never modified; a test asserts it is
Expand Down
Loading
Loading