Skip to content

Propose-a-change reads the staged policy, and the warning names the real direction (#402) - #405

Open
patelankeet2 wants to merge 15 commits into
mainfrom
fix/402-propose-policy-and-direction
Open

patelankeet2 wants to merge 15 commits into
mainfrom
fix/402-propose-policy-and-direction

Conversation

@patelankeet2

@patelankeet2 patelankeet2 commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Closes item 2 on #402 (Arsh's post-merge review of #389), assigned to me and Samika. The other items on that checklist (device-presence filtering, coverage wording, differential searchFilters, docs) are Shubham's or joint follow-ups and are not touched here.

What was wrong

1. Propose-a-change never used the staged policy. propose_change() had no policy parameter at all, so /api/propose always compared a proposed change against Netwise's built-in example rules -- even when the user had uploaded their own via /api/policy. Worse, the finding's own evidence then said "no policy file was supplied", which was true of that specific call and false about the app as a whole.

2. The warning banner could claim the wrong direction. ai/propose.py's warning text was keyed on severity == "high" and unconditionally said "newly opens access". For a policy finding that assumption breaks: severity there tracks whether a breach was newly created, and a created breach can be a requirement violated -- which CLOSES access that should stay open, the opposite of what the text claimed.

Live reproduction, before touching anything

Staged a policy with a high-severity requirement rule (DNS to the approved resolver must work), then proposed blocking that exact flow:

BEFORE this PR, direct call:
  source: "Netwise's built-in example policy (analysis/checks/policy_compliance.py)
           -- no policy file was supplied; rule 4"
  severity: medium   (the BUILT-IN rule's own violation_severity, so the wording bug
                       doesn't even surface with it -- only with a user's OWN high-severity
                       requirement, exactly Arsh's scenario)

Constructed a high-severity version directly against change_impact.analyse_change(policy=...):
  CH-002  high  "This change BREAKS policy rule 4"  Direction: tightening
  current propose.py logic: warning=True, wording would say "newly opens access"
  ACTUAL effect: the change BLOCKS required DNS traffic -- it closes access, not opens it

What changed

  1. analysis/change_impact.py -- every finding now carries a structured direction key (opening/closing/changed), attached after make_finding() returns (same place explanation/explanation_source are attached -- F-1's frozen schema is untouched, no four-way amendment needed). For the filter and reachability producers this was already implicit in severity; for the policy producer it was not, which is the actual bug.
  2. ai/propose.py -- propose_change() accepts policy=None, threaded into the one analyse_change() call it already makes. The warning's trigger stays severity == "high" unchanged (under-warning a serious change is the dangerous direction, same reasoning as the verified/warning split from #13/#14: propose a config change from plain English, and simulate its impact #183) -- only the wording changes, checking the real direction field instead of assuming severity implies it.
  3. web/main.py -- /api/propose now passes policy=_staged_policy(), the exact same accessor /api/findings already uses.

Verified end to end, not just unit tests

After the fix, same live sequence through the real running server (real session, real uploaded sample, real staged policy file):

"source": "the policy file you supplied; rule 4",
"direction": "closing",
"warning": true,
"answer": "...Warning: simulating this change has a serious effect that is not simply
            opening access -- see the impact list for exactly what changed."

Regression check: a request with no policy staged produces byte-identical output to before this PR.

A real mutation-testing gap, found and closed in its own commit

First version of the ai/propose.py commit shipped with every existing test accepting a policy keyword in its stub but none of them checking its value -- reverting the policy=policy line back to dropping it entirely left the whole suite green. Caught by mutation-testing my own commit, fixed in a follow-up commit (d005831) that captures the real call and pins both the supplied-policy and omitted-policy cases.

Acceptance criteria (against #402 item 2)

  • A proposal checked against a staged policy actually uses it, not the built-in example
  • The finding's evidence names the real policy source
  • The warning banner never claims "opens access" for a change that closes required access
  • The warning still fires for every high-severity finding (no narrowing of the safety trigger)
  • No-policy behaviour is unchanged (regression-tested)

Testing

python -m pytest tests/test_change_impact.py tests/test_change_against_policy.py tests/test_propose.py tests/test_web_propose_endpoint.py -v
  166 passed

python -m pytest tests/ -q
  1896 passed, 13 skipped   (main: 1885 passed, 13 skipped -- +11 accounted for by new tests)

Mutation-tested each commit independently (direction field, policy threading at both the ai/propose.py and web/main.py layers) -- each targeted revert killed exactly the test(s) it should have, nothing else.

@SamikaPerera -- requesting your review since these are files #402 names as ours jointly. @ARSH871-bot -- flagging you too since you raised both findings on #389's review; happy to take a second look from you as well if you have time.

Update, 6 November -- the banner bug and the missing test, both fixed

Arsh and Samika each independently reproduced the same two problems live against real Batfish and confirmed they still held on the merged tree:

1. The banner still said "opens access" for a closing breach. The backend's own answer text was already direction-aware since the original version of this PR, but app.js rendered every warning === true response as "This change opens access that is currently blocked", unconditionally -- a leftover assumption from before warning could mean anything else. Fixed by returning a new opens_access field alongside warning, computed with the same direction check the answer text already used, and keying the banner on it. A malformed opens_access defaults to the generic wording, the same cautious-default discipline warning/verified already use. Added a closing-direction scenario and a malformed-flag scenario to the render harness, matching its own existing rigor for the other two flags.

2. The last link was unpinned. Samika found that changing set_active_policy(policy) to set_active_policy(None) inside analyse_change() left the whole suite green -- every existing test stubs the policy stage itself rather than exercising the real install. New test drives the real pc.rules_in_use() path with a policy the built-in rules would never produce, and is detectable by the supplied rule's own text being absent from the result, not just a call count.

Also fixed, smaller: propose_and_scan() threaded the staged policy into its full-scan pipeline call but not into its own inner call to propose_change(), so the "impact" list it also returns stayed evaluated against the built-in rules regardless of what the caller staged (Samika's find, explicitly non-blocking, fixed anyway since it's the same file and the same cause).

Mutation-tested every change against the exact revert that caused each bug in the first place; each failure reproduces that bug's own symptom rather than a generic assertion failure.

python -m pytest tests/test_propose.py tests/test_propose_rendering.py tests/test_change_impact.py -q
  188 passed

python -m pytest tests/ -q
  2215 passed, 54 skipped (Batfish unreachable this run -- CI-condition baseline)
ruff check --select E4,E7,E9,F ai/ analysis/ tests/ -> clean

The direction/A-5 ratification question is still open and untouched by this update -- it needs the team, not more code. Still happy to draft the table's wording if that's the preferred path once the team lands on it.

Update, 11 November -- Arsh's full review addressed

Arsh ran an extensive live/mutation review and confirmed both fixes above hold. Addressed everything in his list:

A wrong claim in two places, corrected, not just one. The "Update, 6 November" section above says propose_and_scan()'s inner propose_change() call "builds the 'impact' list this function also returns". That is not what happens: only base["proposed_change"] and base["answer"] are ever read from that call's result, and this function's own Returns: docstring already said so correctly, three lines above the comment that got it wrong. Fixed the comment and the matching test docstring (test_full_scans_own_impact_list_also_gets_the_policy) to the real reason the policy threads through: consistency of a shared helper across callers, not because this function's return value depends on it.

opens_access was missing from three places that document or build the response shape. ai/propose.py's module docstring, _refuse(), and web/main.py's /api/propose no-upload response and its own docstring, plus app.js's shape comment. All now list/set it.

The "closing is medium" docstring claim is now false, and was corrected. A policy-breaching closing carries its own rule's violation_severity, which this PR's own live demonstration already shows can be high -- the docstring still said every closing is medium.

Two regression tests added, as given, pinning that a "changed" direction (neither side of a reachability comparison clearly succeeded) is not treated as an opening by opens_access, and is not dropped by _DIRECTION_NORMAL_FORM. Both already held; nothing was broken, but nothing pinned it either.

test_direction_is_stated_in_this_modules_own_terms now actually checks normalised, which it accepted as a parametrize value but never read.

docs/user-guide.md step 4 now also mentions the serious-closing warning, not only an opening.

Proposed A-5 in docs/finding-format.md: one vocabulary for direction, ratifying opening/closing/changed for both the structural field (already de facto standard) and the human-readable "Direction: X" phrasing in evidence.detail (still inconsistent -- change_impact.py:488 says "tightening" where the field says "closing", and #406 separately says "changed"). Not ratified, zero of four; not changing the wording itself in this PR, same discipline A-4 uses.

Verification:

pytest tests/test_propose.py tests/test_change_impact.py tests/test_change_against_policy.py -v
  all pass, including the two new tests and the strengthened one

pytest tests/ -q (Batfish running)
  2327 passed, 13 skipped

Mutation-tested both mutants Arsh named directly: treating "changed" as an opening in opens_access's all() check, and removing "changed" from _DIRECTION_NORMAL_FORM. Both caught, for the right reason (a wrong boolean and a KeyError respectively), then reverted.

Live spot-check against real Batfish, reproducing Arsh's exact scenario (a staged high-severity requirement rule for DNS, then proposing to block that exact flow): warning: True, opens_access: False, answer text says "not simply opening access" -- unchanged by these edits, as expected, since none of them touch the logic itself.

Arsh's banner-wording suggestion (from his review) is non-blocking and is not implemented here -- flagging it for him and Samika to discuss, since it's a product-wording choice rather than a correctness fix.

@ARSH871-bot @SamikaPerera -- re-requesting review.

Severity alone cannot tell a caller which way a change moved traffic.
For the filter and reachability producers it happens to be a faithful
proxy (_OPENING_SEVERITY/_CLOSING_SEVERITY are assigned directly from
the computed direction), but for the policy producer it is not:
severity there tracks whether a breach was newly created, and a
created breach can be a prohibition violated (access opens) or a
requirement violated (access closes). ai/propose.py's warning banner
assumed severity high means opened, which is wrong for the second
case -- found by @ARSH871-bot reviewing #389, tracked as item 2 on
#402.

Added _attach_direction(), a one-line wrapper around make_finding()
that sets a normalised 'direction' key (opening/closing/changed) on
the finding dict AFTER it returns from validation -- the same place
explanation/explanation_source are already attached downstream, so
F-1's frozen schema is untouched and this needs no four-way amendment.

Reproduced live against real Batfish before writing anything: staged
a policy with a high-severity requirement rule, proposed a change
that blocks the required flow, confirmed the resulting finding had
severity=high and (before this commit) no way to tell it was actually
closing access, not opening it.

Tests: extended tests/test_change_impact.py (filter and reachability
producers, two new direct unit tests using the existing fake-session
pattern) and tests/test_change_against_policy.py (policy producer,
parametrized across all four kind/before/after combinations against
the existing rule() fixture).

python -m pytest tests/test_change_impact.py tests/test_change_against_policy.py -v
  37 passed
…ording (#402)

propose_change() never accepted a policy at all, so every proposal was
compared against Netwise's built-in example rules even when the user
had staged their own -- and the resulting finding's own evidence then
said 'no policy file was supplied', which was simply false once a
policy had been staged somewhere else in the app. Reproduced live
before fixing: staged a policy, proposed a change that broke it,
confirmed the source named the built-in label regardless.

Added a policy=None parameter, threaded straight into the one
analyse_change() call this function already makes -- analyse_change()
has accepted policy= since #389 and already installs/clears it, so
nothing on the receiving end needed to change.

Second, related fix: the warning banner was keyed on severity=='high'
and always said 'newly opens access'. For a policy finding that is
wrong whenever the high-severity breach is a requirement, not a
prohibition -- that closes access that should stay open, the opposite
claim. Kept the trigger condition unchanged (under-warning a serious
change is the dangerous direction) and fixed only the wording, using
the real 'direction' field change_impact now attaches (previous
commit) rather than inferring it from severity.

Reproduced both fixes live against real Batfish: a staged policy with
a high-severity requirement rule now correctly names the user's
policy in the finding source, and the warning text no longer claims
'opens access' for a change that closes required traffic. Confirmed
the no-policy path is byte-for-byte unchanged (regression check).

Tests: extended tests/test_propose.py -- _found()'s new direction
parameter (defaulting to 'opening', matching every existing caller's
actual intent), a strengthened assertion on the existing opening test,
and two new tests for the closing and mixed-direction cases. Fixed
_stub_change_impact()'s fake signature to accept the new policy
keyword. tests/test_propose_rendering.py is unaffected -- its fixtures
are hand-written canned responses, never a call into propose_change().

python -m pytest tests/test_propose.py tests/test_propose_rendering.py -v
  130 passed
…e_change

Mutation-tested the previous commit and found a real gap: every test
that stubs change_impact.analyse_change() accepted a policy keyword
but none of them captured or asserted its VALUE, so the policy=policy
argument could be silently dropped (reverted to a hardcoded call with
no policy at all) and the whole suite stayed green. Same shape as the
wiring gap found and fixed during #395's work earlier this block --
a parameter accepted by a fake is not the same claim as a parameter
actually threaded through.

Added _stub_change_impact_recording(), which records each call's
kwargs, and two tests: one pinning that a caller-supplied policy
object reaches analyse_change unchanged, one pinning that omitting it
still passes None (the pre-existing behaviour).

python -m pytest tests/test_propose.py -v
  73 passed

Mutation-tested: reverted analyse_change's call back to dropping
policy entirely, confirmed test_the_caller_supplied_policy_actually_
reaches_analyse_change fails for the right reason and nothing else
does, reverted.
propose_change_endpoint() never passed the user's staged policy, so
every proposal through the dashboard was compared against Netwise's
built-in example rules regardless of what the user had uploaded via
/api/policy. _staged_policy() is the exact same accessor /api/findings
already calls -- no second staging mechanism needed, just the one
call site that had never used it.

Tests: extended tests/test_web_propose_endpoint.py -- all five existing
fake_propose_change() stubs updated to accept the new keyword, the
main delegation test now asserts the no-policy case passes None
through unchanged, and a new test mocks _staged_policy() and confirms
its exact return value reaches propose_change().

python -m pytest tests/test_web_propose_endpoint.py -v
  8 passed
@patelankeet2 patelankeet2 added bug Something isn't working layer: analysis Layer 1 — Batfish orchestration, checks, the pipeline phase: 3 Market readiness -- what stands between us and a product someone could buy. labels Oct 18, 2026
@patelankeet2 patelankeet2 self-assigned this Oct 18, 2026

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified rather than read, merged onto current main (dc93ccb): 1896 passed, 13 skipped with Batfish running.

Both bugs from #402 item 2 are fixed. I staged a user policy with one high-severity requirement (DNS to 218.8.104.58 must work), then proposed blocking exactly that flow:

verified: True   warning: True
CH-001  found  medium  direction=closing  A rule change blocks traffic that acl_in used to allow
CH-002  found  high    direction=closing  This change BREAKS policy rule 4 (DNS to the approved resolver must work)
answer: ... Warning: simulating this change has a serious effect that is not simply opening access

The user's own rule is the one evaluated, it keeps its own severity, and the banner no longer says "opens access" for a change that closes it. That was the scenario Arsh described, and it's gone.

One question, and it's for @ARSH871-bot, not a code problem

I'm leaving this as a comment rather than an approval because of one claim in the code comment:

attached after make_finding() returns (same place explanation/explanation_source are attached -- F-1's frozen schema is untouched, no four-way amendment needed)

That comparison doesn't hold. explanation is attached in web/main.py:1012, at the response layer, downstream of analysis. direction is attached inside analysis/change_impact.py, so the analysis producer itself now returns findings with an eighth key.

Whether that needs an amendment is genuinely open. But it's the same question Arsh pressed on my #390 last week: there the extra key was optional and populated nowhere, and it still got a sign-off table because §7a puts field vocabulary under all four. This one is populated on every change_impact finding. I'd rather the team decide it once than have #390 and #405 answer it in opposite ways a week apart.

Two cheap ways through, if you'd rather not amend:

  • analyse_change() returns the direction beside the findings, as a mapping from id to direction, instead of inside them. That's my function, and I'm happy to change its signature with you.
  • Or ratify direction the same way as #390's key set: one table, values opening / closing / changed.

Heads-up on merge order

I'm about to fix #403 and #402 items 1, 3, 5 and 6, all in _policy_change_findings, which this PR also restructures. I'll build on main and rebase onto yours once it lands, so you're not blocked on me. Whatever lands second absorbs the conflict, and that will be me.

@patelankeet2

Copy link
Copy Markdown
Collaborator Author

You're right, and I should have caught it myself rather than reaching for a comparison that doesn't hold. explanation/explanation_source are attached in web/main.py, at the response layer, strictly downstream of analyse_change() returning. direction is attached inside analysis/change_impact.py itself, on every finding the producer makes -- that's a different claim than "a key some caller adds afterward", and I wrote the comment as if it were the same one.

Agreed this is the same question as #390's, not a smaller version of it. If anything it's a stronger case for a table: A-3's subject was optional and populated nowhere when it needed four signatures; direction is unconditional and populated on every change_impact finding already on main (once this merges). I'd rather not have #390 and #405 answer "does an unconditional producer-side key need ratification" two different ways a week apart either.

Of your two options, I'd lean toward ratifying it the way #390 did -- a table in docs/finding-format.md, values opening/closing/changed -- over moving it outside the finding, mostly because a caller reading impact for any OTHER reason (a future dashboard view, say) would have to know to go look in a second mapping rather than finding it on the finding itself. But that's a preference, not a unilateral call, and I'll go whichever way the four of us land on it.

@ARSH871-bot -- happy to write the table's wording if you'd rather not, same way I did for #390's.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed f4668d0 on today's main (db59b00), since #298 rewrote much of ai/propose.py around this. The code is ready, and I'd merge it today, except for one thing we both said needs the team: the direction key.

Still needed. Both bugs are still live on main: propose_change() has no policy parameter, and the warning text still says "newly opens access".

Still works after #298. It auto-merges clean: 2251 passed, 13 skipped, ruff clean on CI's selection, CI green. My 20 October scenario, re-run through /api/sample, /api/policy (one high requirement: DNS to 218.8.104.58 must work) and /api/propose (block 10.10.10.0/24 to 218.8.104.58 on udp/53 on rtr-us5):

verified True   warning True
CH-001  found  medium  direction=closing  A rule change blocks traffic that acl_in used to allow
CH-002  found  high    direction=closing  This change BREAKS policy rule 4 ...   source: the policy file you supplied; rule 4
answer: ... Warning: simulating this change has a serious effect that is not simply opening access

Why I'm not merging it. On 20 October we agreed that direction is a key the producer puts on every change_impact finding, and that it needs ratifying like #390's key set. You leaned towards a table in docs/finding-format.md with values opening / closing / changed. There's still no table, and docs/finding-format.md on main has no direction entry. I don't want this to land the way #390 did on Monday, merged with its key-set table at 2 of 4. Doing that twice makes it how we work.

Quickest way through: push the table onto this branch (you offered to write it). I'll tick my row the same day, then ask Arsh and Samika for theirs. As soon as it reads 4 of 4, I'll re-test and merge. If we'd rather not amend F-1, the other option still stands: analyse_change() returns directions beside the findings. That's my function, and I'll change it with you.

Merge order with my #406: both PRs touch _policy_change_findings. Whichever lands second absorbs the conflict, and if that's mine I'll handle it, as promised.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update, 3 November. Still on f4668d0. Shubham's 2 November comment holds this for a direction ratification table. I agree with that, and I'd sign a table that also settles #406's evidence-text form. Separately, the banner point below still stands at this head: app.js:1429-1437 here, and the same lines are app.js:1838-1846 on main now that #298 is in.

Measured on f4668d0 against real Batfish through /api/upload → /api/policy → /api/propose, plus the suite with Batfish refused. Live: 1896 passed / 13 skipped, matching your number. CI condition: 1860 / 49.

The staged-policy half works. With a one-rule policy staged and "block … udp/53": on dc93ccb we evaluate our rules 1–5 and report "BREAKS policy rule 4 … no policy file was supplied". On your branch only the user's rule is evaluated. I also checked two cookie jars (no leakage either way), a policy replaced between proposals (the new one is used), and a re-upload (the policy is dropped and the label is correct). All nine mutants I aimed at your changes are caught without Batfish. Pinning the real policy value in d005831 was the right instinct.

Requesting changes for one thing: the banner. app.js:1429-1437 still prints "This change opens access that is currently blocked" whenever warning === true, and :1396 still documents warning as "a high-severity OPENING". Stage a requirement for UDP/53 to 218.8.104.58 with violation_severity: "high" and propose "block 10.10.10.0/24 to 218.8.104.58 on udp/53 on rtr-us5". The generated line is deny udp … eq 53, and the real app.js (run through tests/js/propose_render_harness.js) shows the amber banner "This change opens access…" directly above your new "not simply opening access" answer. On dc93ccb that request gives warning=False, so this PR is what makes the false banner reachable. The PR's acceptance box says the opposite. I agree with keeping the trigger on severity. Please return the direction fact (for example opens_access = every high finding is opening), key the banner text on it, and add a closing case to the render harness.

@shubhamkataria2005, your comment says the banner no longer says "opens access". Your own output shows why it still does: warning: True next to direction=closing is exactly the response app.js:1429-1437 renders as "This change opens access that is currently blocked". The answer text you quoted is right. The banner above it is not, and this PR doesn't touch app.js.

Before merge, also:

  1. direction lives on the finding inside analysis/. The explanation precedent you cite is attached in web/main.py:1012, at the response layer, explicitly so that analyse() "still returns… the F-1 shape unchanged" (:929). A-3 took four signatures for an optional key. This one changes analyse_change()'s output, the CLI JSON, and the model prompt (ai/explain.py:671 dumps the whole finding). Either raise it as A-5 or keep the findings to F-1 and return direction alongside them. Shubham raised the same point on 20 October and offered to change analyse_change()'s signature with you. His #406 already answers it the other way: it writes direction into the evidence text ("Direction: changed.", analysis/change_impact.py:569 on #406) and adds no key. So the two open PRs disagree, and I'd like the team to settle it once, for both. That's a team call, not a reason to stop.
  2. One test for the last link. Replace set_active_policy(policy) with (None) in analyse_change and the whole suite still passes, so this PR's feature could silently revert. A stub-level test that analyse_change(policy=P) asks P's rules would close it.

Worth knowing, not yours to fix here:

  • A policy naming a device that isn't in the upload now gives verified=False with "BatfishException: Work terminated abnormally" (dc93ccb: verified). An empty policy now produces no policy statement at all, although /api/policy promises "every policy-driven check will say so". Those are #402 items 1 and 3, which Shubham's #406 now claims to fix. I'm reviewing that separately.
  • D1 is unchanged for users without a policy, as you said. It is fixed in practice for anyone who stages a policy naming their own device.
  • #298 merges cleanly with this but propose_and_scan still calls propose_change(request, before_dir, host=host) without policy (ai/propose.py:638 in the merged tree) while its full scan gets it, and the download path at web/main.py:2481 does the same. Whichever lands second needs to thread it through.

Nits: ai/propose.py:47-54 still argues that closing is always medium. The normalised parameter added to test_direction_is_stated_in_this_modules_own_terms is never used. The body's "byte-identical" isn't quite true: every found impact finding now carries direction.

@SamikaPerera SamikaPerera left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed f4668d0 merged onto today's main (db59b00). It merges cleanly. Requesting changes for the same two things Arsh raised on 3 November. I checked both again and they still hold. The web/main.py part is right.

What works

  • /api/propose now passes policy=_staged_policy(), the same accessor /api/findings uses. Live against Batfish: I loaded the sample, staged a one-rule policy (DNS to 218.8.104.58 must work, violation_severity: "high"), and proposed block 10.10.10.0/24 to 218.8.104.58 on udp/53 on rtr-us5. Only my rule was evaluated, and its source reads "the policy file you supplied; rule 1".
  • test_a_staged_policy_reaches_propose_change pins that line. With policy=_staged_policy() removed from web/main.py, that test fails (1 failed, 16 passed).

1. The banner still says the opposite of the answer. The same live request returns warning: True, with both impact findings direction=closing, and the answer "…has a serious effect that is not simply opening access". app.js:1838-1846 on the merged tree shows "This change opens access that is currently blocked" whenever warning === true. I rendered that response through the real app.js (a copy of tests/js/propose_render_harness.js with one added case). The banner reads "This change opens access that is currently blocked." directly above the answer saying it isn't. The comment at app.js:1805 still describes warning as "a high-severity OPENING". app.js is in web/, which CODEOWNERS gives me, so I'm fine with this PR changing it. Arsh's suggestion works for me: a direction fact in the response, the banner text keyed on it, and a closing case in the render harness.

2. The last link is still unpinned. With set_active_policy(policy) changed to set_active_policy(None) at analysis/change_impact.py:629, the whole suite still passes on the merged tree (2227 passed, 13 skipped, with test_no_configs_in_git.py left out because my copy was not a git checkout). So the feature could silently revert.

On direction: I have no position yet. I'll answer on the ratification table when there is one.

Smaller, not blocking: propose_and_scan() still calls propose_change(request, before_dir, host=host) without policy (ai/propose.py:638 merged). Live, /api/propose/full-scan returns only proposed_change, findings, grounded and answer, and its policy finding did use my rule (PC-001 found high). So I found no visible effect, but the impact it computes there ignores the staged policy.

What I ran

  • The full suite on a plain copy of main and of the merge. Main: 2220 passed, 13 skipped. Merge: 2231 passed, 13 skipped. Both had the same 20 failures, all in tests/test_no_configs_in_git.py, because the copies are not git checkouts. No test fails on only one side.
  • ruff check --select E4,E7,E9,F analysis/ ai/ web/ tools/ tests/ on the merge: all checks passed.
  • The live sequence above, through the merged app's endpoints against local Batfish, with Ollama unreachable.

Not checked: the dashboard in a browser, a policy naming a device that isn't in the upload, and #406's interaction with this PR.

Every existing test here stubs _policy_change_findings() itself, so none
of them exercise the line that installs `policy` via
policy_module.set_active_policy(). Found in review (#405, Samika):
changing that one argument to None left the whole suite green.

Drives the real pc.rules_in_use() path instead, with a policy naming a
device and a violation summary the built-in rules never use, so a
silently-dropped policy argument is detectable by its absence from the
result rather than by a call count alone.

Mutation-tested against the exact change Samika made: the supplied
rule's own text never reaches the comparison, and the failure output
shows the built-in rules being used instead.
app.js rendered every warning=true response as "This change opens access
that is currently blocked", because that used to be the only thing
warning could mean. It no longer is: a high-severity POLICY finding can
be a requirement breached, which CLOSES access, and the banner kept
claiming the opposite (#405 review, Arsh + Samika, both reproduced live
against real Batfish).

ai/propose.py now returns opens_access alongside warning, computed with
the same direction check the answer text already used -- one place
decides the fact, not every renderer that reads the response. app.js
keys the banner text on it, defaulting to the generic "serious effect"
wording on anything but opens_access === true, the same cautious-default
discipline warning/verified already use.

Also fixes a related gap Samika found: propose_and_scan() threaded the
staged policy into the full-scan pipeline but not into its own call to
propose_change(), so the "impact" list it also returns stayed evaluated
against the built-in example rules regardless of what the caller staged.

Render harness: added a closing-direction scenario and a malformed-
opens_access scenario, matching the file's own existing rigor for
malformed warning/verified flags. Mutation-tested both backend changes
and the frontend banner logic; each failure reproduces the exact bug it
guards against.
@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Fixed both items both of you independently confirmed, plus the smaller one Samika found.

The banner (Arsh's #1, Samika's #1): returns a new opens_access field alongside warning, computed with the exact direction check the answer text already used. app.js keys the banner on it instead of assuming warning always means an opening. Malformed opens_access falls back to the generic wording rather than the specific claim, matching how warning/verified already handle a junk flag. Added a closing-direction scenario and a malformed-flag scenario to the render harness.

The missing test (Samika's #2): every existing test stubbed _policy_change_findings() itself, so the actual line that installs the policy was never exercised by anything. New test drives the real rules_in_use() path with a rule the built-in set would never produce. Mutation-tested against your exact change -- the supplied rule's text goes missing from the result and the failure output shows the built-in rules being used instead.

Also fixed: propose_and_scan()'s own inner call to propose_change() wasn't getting the staged policy, even though the full-scan pipeline call right next to it was. Samika's find, not blocking, fixed anyway since it's the same cause in the same file.

Full detail and the live-against-real-Batfish verification I ran before this is in the updated PR description. Samika, the propose_and_scan policy fact (not the one you asked about) was a genuinely separate call site from the one you checked, worth a second look if you have a minute. The direction ratification question is untouched, still waiting on the team.

@SamikaPerera SamikaPerera left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 0ed8ece. I checked it on a scratch merge with main (d424e05), against real Batfish.

Both of my points are fixed.

  1. The banner. I ran my closing request through /api/propose: verified True, warning True, opens_access False, with CH-001 and CH-002 both closing. Through the real app.js (the render harness) it now shows the generic wording, not "opens access". An opens_access that is not exactly true also gets the generic wording.

  2. The policy link. With set_active_policy(policy) changed to set_active_policy(None) at analysis/change_impact.py:629, test_the_supplied_policy_is_what_gets_installed_not_dropped fails (1 failed, 2274 passed).

I also checked the other two things this PR fixes by breaking them. With the banner ignoring opens_access, 2 tests fail. With propose_and_scan()'s inner propose_change() losing the policy, 1 test fails. So none of the three can silently regress.

Full suite on the merge: 2275 passed, 13 skipped. ruff (CI's rule set) is clean. Thanks for taking the propose_and_scan call site as well.

Not checked: the banner in a browser (harness and one live API call only), and the MIXED_IMPACT case live.

Two facts, not blocking: /api/propose/download still calls propose_change without the policy (it only uses the result to build the file), and /api/propose/full-scan does not return warning or opens_access. I did not re-check that second one at this head.

Direction: I still have no position and will answer on the ratification table when it exists. This approval does not depend on it.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 0ed8ece, merged onto today's main (d424e05). Both things I asked for on 3 November are fixed, and the logic held up against everything I tried. I'm still requesting changes, for two small test and comment fixes and the direction table. None of them changes behaviour, and I'll re-review the same day.

Before and after, reproduced. Live Batfish, through /api/upload, /api/policy and /api/propose, with each real response rendered through that tree's real app.js. Policy: one high requirement, DNS to 218.8.104.58 must work. Request: block 10.10.10.0/24 to 218.8.104.58 on udp/53 on rtr-us5.

main d424e05   warning False. Evaluated "Netwise's built-in example policy". My rule was ignored.
f4668d0        warning True. Banner: "This change opens access that is currently blocked",
               above an answer saying it is not simply opening access.
0ed8ece        warning True, opens_access False, source "the policy file you supplied; rule 1".
               Banner: "This change has a serious effect that is not simply opening access".

A real opening still says so: allow any to 10.20.0.5 on tcp/23 gives opens_access True and the "opens access" banner.

What I tried to break, and couldn't:

  • A real opening downgraded to the generic banner. I suggested all() on 3 November and wanted to be sure it was right, since a high changed finding beside a real opening would weaken the banner. Routed, connected, unrouted and null-routed destinations all stayed opening. The last two produce no reachability row at all, so changed didn't appear. all() holds.
  • No policy staged. 7 scenarios on main and on this branch are identical apart from direction and opens_access.
  • The download path I flagged on 3 November. /api/propose/download returns the same bytes with and without a staged policy, because it only uses the config text. Please ignore that note.
  • A policy leaking between requests. The install is per-thread (analysis/policy.py:798), so a proposal can't reach a scan running beside it. I read this one and didn't run it.

Also checked:

  • CI-style suite on the merge: 2234 passed, 54 skipped (main 2218, so +16).
  • Live suite on the merge: 2275 passed, 13 skipped.
  • Your 188 reproduce, and ruff is clean.
  • 21 of my 23 mutants are caught, with Batfish and Ollama running and with them off, including Samika's set_active_policy(None) and all three banner mutants.

1. changed is never exercised, and it's about to go into a contract. Two mutants survive in both environments: treating changed as an opening in opens_access, and removing "changed" from _DIRECTION_NORMAL_FORM, which makes a real changed row raise KeyError. It's rare, but it's one of the three values in the table below. These two tests close it. I've run them: the real code passes and both mutants fail.

# tests/test_change_impact.py
def test_reachability_findings_say_changed_when_the_direction_is_unknown():
    bf = _FakeSessionWithFlows([_flow_row("NO_ROUTE", "DENIED_IN")])
    results, _ = change_impact._reachability_change_findings(bf, 1)

    assert len(results) == 1
    assert results[0]["severity"] == "high"
    assert results[0]["direction"] == "changed"

# tests/test_propose.py
def test_an_unclassified_high_finding_warns_without_claiming_an_opening(monkeypatch, tmp_path):
    before_dir = _write_snapshot(tmp_path)
    _stub_batfish(monkeypatch)
    _stub_change_impact(monkeypatch, [_found(severity="high", direction="changed")])

    result = propose.propose_change(GOOD_REQUEST, before_dir)

    assert result["warning"] is True
    assert result["opens_access"] is False

2. The propose_and_scan claim isn't true. ai/propose.py:651-654, the docstring at tests/test_propose.py:910 and the PR description all say this function "also returns" an impact list. It returns proposed_change, findings, grounded and answer, and live, /api/propose/full-scan has no impact anywhere. base's impact is computed and thrown away, so this threading has no visible effect, as Samika found on 3 November. My own 3 November note pointed you here, and I hadn't noticed that either. Keeping the argument is harmless, but please correct all three so they say what's true, or the test pins a fix for something that was never wrong.

3. The direction table. The description says this needs the team, not more code, but I think we're only waiting for the table to exist. You and Shubham both lean towards a table, I said I'd sign one, and Samika will answer once there is one. Please push it here as A-5 (A-4 is #381). I'll tick my row the same day, and Shubham offered the same on 2 November. It should also settle the wording, because one finding currently uses two words for the same thing. Live on this request, CH-002 has the key direction: closing and the evidence "Direction: tightening" (analysis/change_impact.py:488), and #406 writes "Direction: changed." into evidence too.

Smaller, worth folding in since there's a new commit anyway:

  • opens_access is missing from _refuse() (ai/propose.py:313) and from the no-upload response (web/main.py:2451). app.js copes, but it's a different shape from a success. The shape comment at app.js:1769 doesn't list it either.
  • Still open from 3 November: ai/propose.py:53 says closing is medium, and this PR exists because a breached requirement can be high. normalised is still unused at tests/test_change_against_policy.py:95.
  • docs/user-guide.md:260 says we warn "if the change opens access". We now warn on a serious closing too.

A suggestion for you and Samika, not blocking. "Not simply opening access" on a change that only closes reads as if it might open something too. A high closing can only be a breached requirement, because filter and reachability closings are medium. So when every high finding is closing, the banner could say it plainly: "This change blocks traffic your policy says must be allowed."

Not this PR, but it changes what lands:

  • A medium requirement still gets no warning and "narrows access". That's #451, which sits on top of this.
  • Once this merges, a policy naming a device not in the upload shows "BatfishException: Work terminated abnormally", and an empty policy produces no policy statement at all. Both are #402 items in Shubham's #406. This PR is what makes them reachable from the propose pane, so they should land close together. The two PRs currently conflict in analysis/change_impact.py and tests/test_change_against_policy.py.
  • CH-000's source is {before_dir} -> {after_dir}, so the card shows folder paths on this machine. It's on main too, and I'll file it.

A pushed fix notifies nobody, so please re-request Samika and me when it's up. Samika approved this head on 8 November, and the new commit means that approval will need repeating too.

…uilt (#405 review, Arsh)

_refuse(), the module docstring, /api/propose's no-upload response and
its own docstring, and the app.js shape comment all listed the six
original keys and never picked up opens_access when #405 added it.
app.js copes with a missing key, but a refusal's shape is now a
different one from a success, which the comment should say.
…rsh)

A policy-breaching closing carries its rule's own violation_severity,
which can be high -- the exact case this PR fixes the banner for. The
docstring still said every closing is medium, which this PR's own live
demonstration already disproves.
…s returned (#405 review, Arsh)

The comment and the test's own docstring both claimed this function's
call to propose_change() 'builds the impact list this function also
returns'. It returns proposed_change and answer only -- the Returns:
block a few lines above already said so correctly. Corrected both to
the real reason for threading the policy through: every caller of
propose_change() should see the same policy, not because this
function's own return value depends on it.
…eview)

A 'changed' reachability direction (neither side clearly succeeded) is
real, already produced by _reachability_change_findings(), and already
handled correctly by opens_access's all() check and by
_DIRECTION_NORMAL_FORM. Nothing was broken, but nothing pinned it
either -- these two guard the behaviour so a future change to either
site cannot silently start treating 'changed' as an opening.
…review, Arsh)

test_direction_is_stated_in_this_modules_own_terms accepted 'normalised'
as a parametrize value but never read it in the body -- only the sibling
test two functions down did. The table already carries the right value
for every case; now both tests actually check it.
…w, Arsh)

Step 4 only described warning on an opening. The warning also fires on
a policy-breaching closing (#402), and the guide should say so.
…rds (#405 review, Arsh)

The structural direction field already uses opening/closing/changed and
is already relied on by opens_access -- not what this proposes changing.
What is inconsistent is the human-readable Direction: X phrase in
evidence.detail: change_impact.py:488 still says tightening for the case
the field calls closing, and #406 separately invented changed for its
own case. Proposes one vocabulary for both. Not changing the wording
itself here -- that is exactly what this table gates, same as A-4.
@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Addressed everything in the 8 November review, in seven commits:

1. changed is never exercised. Added both tests exactly as given, in their target files. Confirmed both pass against the current code and both mutants you named fail for the reason you describe: treating changed as an opening in opens_access's all() check returns a wrong True, and removing "changed" from _DIRECTION_NORMAL_FORM raises KeyError rather than silently mislabeling. Reverted both after confirming.

2. The propose_and_scan claim. Fixed all three places you named. The comment at ai/propose.py now says the real reason the policy threads through: every caller of propose_change() should see the same policy, not because this function's own return value depends on it (only proposed_change and answer are ever read from base). Same correction in test_full_scans_own_impact_list_also_gets_the_policy's docstring, and in the PR description below (see the 11 November update).

3. The direction table. Pushed as A-5 in docs/finding-format.md, same shape as A-4: not ratified, zero of four, your row and Shubham's and Samika's and mine all open. It proposes ratifying opening/closing/changed for both the structural field (already de facto standard) and the "Direction: X" phrasing in evidence text, which is where the real inconsistency is (change_impact.py:488 still says "tightening", #406 separately says "changed"). Not changing that wording in this PR, deliberately, same as A-4's discipline for cve_mapping.

Smaller items, all folded in:

  • opens_access added to _refuse(), the no-upload response in web/main.py, and both shape comments (ai/propose.py's module docstring and app.js:1769).
  • ai/propose.py:53's docstring corrected: a closing is medium unless it breaches a policy requirement, in which case it carries that rule's own violation_severity.
  • normalised is now actually asserted in test_direction_is_stated_in_this_modules_own_terms.
  • docs/user-guide.md:260 now also mentions warning on a serious closing, not only an opening.

Verification:

pytest tests/test_propose.py tests/test_change_impact.py tests/test_change_against_policy.py -v
  all pass, including the two new tests and the strengthened one

pytest tests/ -q (Batfish running)
  2327 passed, 13 skipped

Live spot-check reproducing your exact scenario (staged high-severity DNS requirement, proposing to block that flow, against real Batfish): warning: True, opens_access: False, answer text "not simply opening access" -- unchanged, as expected, since none of these commits touch that logic.

Your banner-wording suggestion ("This change blocks traffic your policy says must be allowed" when every high finding is closing) is not implemented here. Flagging it for you and Samika to decide, since it's a product-wording call rather than a correctness fix.

Not touched, as agreed: #451 (medium requirement warning), the two #406 items (device-not-in-upload exception, empty-policy statement), and CH-000's source showing local paths. All tracked separately.

Re-requesting both of you now that this is pushed.

@SamikaPerera SamikaPerera left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 08733c9. In a clean checkout of that commit with Batfish up (run on 11 Nov), the PR's three test files give 128 passed and the full suite gives 2327 passed, 13 skipped.

Arsh's three points from 9 Nov are addressed. The two changed tests are in (12be902). The propose_and_scan comment and the test docstring now say only proposed_change and answer are read (c99f53e). A-5 is in docs/finding-format.md, with zero of four ticked.

Two small things to fix before merge. Neither changes my approval.

  1. The PR body still says, at line 81: "so the "impact" list it also returns stayed evaluated against the built-in rules". The 11 November update says that sentence is wrong, but it is still there. The test name test_full_scans_own_impact_list_also_gets_the_policy (tests/test_propose.py:920) says the same thing. Could you fix the sentence and rename the test?

  2. Two lines in A-5 do not match the code. I will not tick my row until they are corrected.
    a. My row says the dashboard banner is downstream of the wording A-5 settles. The banner reads result.opens_access (web/static/app.js:1877), which comes from the structural field (ai/propose.py:523-525), not from the "Direction: X" sentence. Please change my row to say that.
    b. The block says opens_access relies on all three values. It only compares with "opening" (ai/propose.py:524); anything else counts as not opening.

On the banner wording you asked about: my call is to keep the current wording in this PR. I will open a small follow-up for the wording on a change that only closes access.

Not checked: the banner in a browser.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 08733c9, merged onto today's main (ee769b4). Everything I asked for on 10 November is done. Approving, with two small leftovers that don't block.

Checked

  • CI-style suite on the merge: 2301 passed, 54 skipped. Live suite on the merge: 2342 passed, 13 skipped. Ruff reports 8 findings, all in docs/document-builds, identical on main.
  • Mutation run: 23 of 23 caught, with Batfish and Ollama running and with them off. The two that survived last time are now caught: changed counted as an opening, and "changed" removed from _DIRECTION_NORMAL_FORM.

My points from the last review

  1. changed tests: both are in, and they are the two I wrote.
  2. propose_and_scan: the comment at ai/propose.py:657 and the test docstring now say what is true. The PR description corrects itself too (the paragraph beginning "A wrong claim in two places").
  3. The direction table: pushed as A-5, marked PROPOSED and 0 of 4. Correctly not changing the "Direction: tightening" wording until it is full.
  4. Smaller items, all done: opens_access is in _refuse(), the no-upload response and the app.js shape comment; the docstring no longer says a closing is always medium; the unused normalised is now asserted; docs/user-guide.md mentions the serious-closing warning.

Two leftovers, not blocking

  • The PR description still has the old sentence at "Also fixed, smaller" ("the 'impact' list it also returns"), and the test is still named test_full_scans_own_impact_list_also_gets_the_policy. Both say the thing the corrected comment now says isn't true. A rename and one edited sentence.
  • The banner suggestion (when every high finding is closing, say "This change blocks traffic your policy says must be allowed") wasn't taken. It was optional; mentioning it only so it isn't lost.

A-5
I agree with the table as written. Please tick my row in your next push, so the head only moves once.

Before this merges

  • Samika has already approved this head (08733c9, 12 November), so that approval does not need repeating.
  • This conflicts with #406 in analysis/change_impact.py and tests/test_change_against_policy.py (tried today, merging this head into #406's 76cdf8c). Whichever lands second resolves it, and also needs one line in test_direction_field_is_set_regardless_of_severity, which unpacks three values from stage(). I approved #406 today, so I would expect it to land first.

@ARSH871-bot

ARSH871-bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

@patelankeet2, a quick nudge on this one, since it is close. It's approved (Samika and me), and two things stand between it and main:

  1. It now conflicts with main, in four files: ai/propose.py, analysis/change_impact.py, tests/test_change_against_policy.py and tests/test_propose.py (tried today, merging 08733c9 into main cceffec). Change impact reports what it did not look at (closes #403, most of #402) #406 merged on 18 November and Never show the caller's folders on change impact cards (closes #462) #463 and Give a proposed change a size, beside its findings (closes #420) #428 after it. While resolving, test_direction_field_is_set_regardless_of_severity needs one line: stage() on main now returns four values, and the test unpacks three (found, _, _ = stage(...)).
  2. The A-5 table. As agreed in my review, please tick my row in the same push, so the head only moves once.

A merge-only push keeps both approvals under M-2. If the conflict resolution changes behaviour, say so and I'll re-review the same day. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working layer: analysis Layer 1 — Batfish orchestration, checks, the pipeline phase: 3 Market readiness -- what stands between us and a product someone could buy.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants