Repository navigation
Propose-a-change reads the staged policy, and the warning names the real direction (#402) - #405
patelankeet2 wants to merge 15 commits into
Conversation
Severity alone cannot tell a caller which way a change moved traffic. For the filter and reachability producers it happens to be a faithful proxy (_OPENING_SEVERITY/_CLOSING_SEVERITY are assigned directly from the computed direction), but for the policy producer it is not: severity there tracks whether a breach was newly created, and a created breach can be a prohibition violated (access opens) or a requirement violated (access closes). ai/propose.py's warning banner assumed severity high means opened, which is wrong for the second case -- found by @ARSH871-bot reviewing #389, tracked as item 2 on #402. Added _attach_direction(), a one-line wrapper around make_finding() that sets a normalised 'direction' key (opening/closing/changed) on the finding dict AFTER it returns from validation -- the same place explanation/explanation_source are already attached downstream, so F-1's frozen schema is untouched and this needs no four-way amendment. Reproduced live against real Batfish before writing anything: staged a policy with a high-severity requirement rule, proposed a change that blocks the required flow, confirmed the resulting finding had severity=high and (before this commit) no way to tell it was actually closing access, not opening it. Tests: extended tests/test_change_impact.py (filter and reachability producers, two new direct unit tests using the existing fake-session pattern) and tests/test_change_against_policy.py (policy producer, parametrized across all four kind/before/after combinations against the existing rule() fixture). python -m pytest tests/test_change_impact.py tests/test_change_against_policy.py -v 37 passed
…ording (#402) propose_change() never accepted a policy at all, so every proposal was compared against Netwise's built-in example rules even when the user had staged their own -- and the resulting finding's own evidence then said 'no policy file was supplied', which was simply false once a policy had been staged somewhere else in the app. Reproduced live before fixing: staged a policy, proposed a change that broke it, confirmed the source named the built-in label regardless. Added a policy=None parameter, threaded straight into the one analyse_change() call this function already makes -- analyse_change() has accepted policy= since #389 and already installs/clears it, so nothing on the receiving end needed to change. Second, related fix: the warning banner was keyed on severity=='high' and always said 'newly opens access'. For a policy finding that is wrong whenever the high-severity breach is a requirement, not a prohibition -- that closes access that should stay open, the opposite claim. Kept the trigger condition unchanged (under-warning a serious change is the dangerous direction) and fixed only the wording, using the real 'direction' field change_impact now attaches (previous commit) rather than inferring it from severity. Reproduced both fixes live against real Batfish: a staged policy with a high-severity requirement rule now correctly names the user's policy in the finding source, and the warning text no longer claims 'opens access' for a change that closes required traffic. Confirmed the no-policy path is byte-for-byte unchanged (regression check). Tests: extended tests/test_propose.py -- _found()'s new direction parameter (defaulting to 'opening', matching every existing caller's actual intent), a strengthened assertion on the existing opening test, and two new tests for the closing and mixed-direction cases. Fixed _stub_change_impact()'s fake signature to accept the new policy keyword. tests/test_propose_rendering.py is unaffected -- its fixtures are hand-written canned responses, never a call into propose_change(). python -m pytest tests/test_propose.py tests/test_propose_rendering.py -v 130 passed
…e_change Mutation-tested the previous commit and found a real gap: every test that stubs change_impact.analyse_change() accepted a policy keyword but none of them captured or asserted its VALUE, so the policy=policy argument could be silently dropped (reverted to a hardcoded call with no policy at all) and the whole suite stayed green. Same shape as the wiring gap found and fixed during #395's work earlier this block -- a parameter accepted by a fake is not the same claim as a parameter actually threaded through. Added _stub_change_impact_recording(), which records each call's kwargs, and two tests: one pinning that a caller-supplied policy object reaches analyse_change unchanged, one pinning that omitting it still passes None (the pre-existing behaviour). python -m pytest tests/test_propose.py -v 73 passed Mutation-tested: reverted analyse_change's call back to dropping policy entirely, confirmed test_the_caller_supplied_policy_actually_ reaches_analyse_change fails for the right reason and nothing else does, reverted.
propose_change_endpoint() never passed the user's staged policy, so every proposal through the dashboard was compared against Netwise's built-in example rules regardless of what the user had uploaded via /api/policy. _staged_policy() is the exact same accessor /api/findings already calls -- no second staging mechanism needed, just the one call site that had never used it. Tests: extended tests/test_web_propose_endpoint.py -- all five existing fake_propose_change() stubs updated to accept the new keyword, the main delegation test now asserts the no-policy case passes None through unchanged, and a new test mocks _staged_policy() and confirms its exact return value reaches propose_change(). python -m pytest tests/test_web_propose_endpoint.py -v 8 passed
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Verified rather than read, merged onto current main (dc93ccb): 1896 passed, 13 skipped with Batfish running.
Both bugs from #402 item 2 are fixed. I staged a user policy with one high-severity requirement (DNS to 218.8.104.58 must work), then proposed blocking exactly that flow:
verified: True warning: True
CH-001 found medium direction=closing A rule change blocks traffic that acl_in used to allow
CH-002 found high direction=closing This change BREAKS policy rule 4 (DNS to the approved resolver must work)
answer: ... Warning: simulating this change has a serious effect that is not simply opening access
The user's own rule is the one evaluated, it keeps its own severity, and the banner no longer says "opens access" for a change that closes it. That was the scenario Arsh described, and it's gone.
One question, and it's for @ARSH871-bot, not a code problem
I'm leaving this as a comment rather than an approval because of one claim in the code comment:
attached after
make_finding()returns (same placeexplanation/explanation_sourceare attached -- F-1's frozen schema is untouched, no four-way amendment needed)
That comparison doesn't hold. explanation is attached in web/main.py:1012, at the response layer, downstream of analysis. direction is attached inside analysis/change_impact.py, so the analysis producer itself now returns findings with an eighth key.
Whether that needs an amendment is genuinely open. But it's the same question Arsh pressed on my #390 last week: there the extra key was optional and populated nowhere, and it still got a sign-off table because §7a puts field vocabulary under all four. This one is populated on every change_impact finding. I'd rather the team decide it once than have #390 and #405 answer it in opposite ways a week apart.
Two cheap ways through, if you'd rather not amend:
analyse_change()returns the direction beside the findings, as a mapping from id to direction, instead of inside them. That's my function, and I'm happy to change its signature with you.- Or ratify
directionthe same way as #390's key set: one table, valuesopening/closing/changed.
Heads-up on merge order
I'm about to fix #403 and #402 items 1, 3, 5 and 6, all in _policy_change_findings, which this PR also restructures. I'll build on main and rebase onto yours once it lands, so you're not blocked on me. Whatever lands second absorbs the conflict, and that will be me.
|
You're right, and I should have caught it myself rather than reaching for a comparison that doesn't hold. Agreed this is the same question as #390's, not a smaller version of it. If anything it's a stronger case for a table: A-3's Of your two options, I'd lean toward ratifying it the way #390 did -- a table in @ARSH871-bot -- happy to write the table's wording if you'd rather not, same way I did for #390's. |
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Re-reviewed f4668d0 on today's main (db59b00), since #298 rewrote much of ai/propose.py around this. The code is ready, and I'd merge it today, except for one thing we both said needs the team: the direction key.
Still needed. Both bugs are still live on main: propose_change() has no policy parameter, and the warning text still says "newly opens access".
Still works after #298. It auto-merges clean: 2251 passed, 13 skipped, ruff clean on CI's selection, CI green. My 20 October scenario, re-run through /api/sample, /api/policy (one high requirement: DNS to 218.8.104.58 must work) and /api/propose (block 10.10.10.0/24 to 218.8.104.58 on udp/53 on rtr-us5):
verified True warning True
CH-001 found medium direction=closing A rule change blocks traffic that acl_in used to allow
CH-002 found high direction=closing This change BREAKS policy rule 4 ... source: the policy file you supplied; rule 4
answer: ... Warning: simulating this change has a serious effect that is not simply opening access
Why I'm not merging it. On 20 October we agreed that direction is a key the producer puts on every change_impact finding, and that it needs ratifying like #390's key set. You leaned towards a table in docs/finding-format.md with values opening / closing / changed. There's still no table, and docs/finding-format.md on main has no direction entry. I don't want this to land the way #390 did on Monday, merged with its key-set table at 2 of 4. Doing that twice makes it how we work.
Quickest way through: push the table onto this branch (you offered to write it). I'll tick my row the same day, then ask Arsh and Samika for theirs. As soon as it reads 4 of 4, I'll re-test and merge. If we'd rather not amend F-1, the other option still stands: analyse_change() returns directions beside the findings. That's my function, and I'll change it with you.
Merge order with my #406: both PRs touch _policy_change_findings. Whichever lands second absorbs the conflict, and if that's mine I'll handle it, as promised.
ARSH871-bot
left a comment
There was a problem hiding this comment.
Update, 3 November. Still on f4668d0. Shubham's 2 November comment holds this for a direction ratification table. I agree with that, and I'd sign a table that also settles #406's evidence-text form. Separately, the banner point below still stands at this head: app.js:1429-1437 here, and the same lines are app.js:1838-1846 on main now that #298 is in.
Measured on f4668d0 against real Batfish through /api/upload → /api/policy → /api/propose, plus the suite with Batfish refused. Live: 1896 passed / 13 skipped, matching your number. CI condition: 1860 / 49.
The staged-policy half works. With a one-rule policy staged and "block … udp/53": on dc93ccb we evaluate our rules 1–5 and report "BREAKS policy rule 4 … no policy file was supplied". On your branch only the user's rule is evaluated. I also checked two cookie jars (no leakage either way), a policy replaced between proposals (the new one is used), and a re-upload (the policy is dropped and the label is correct). All nine mutants I aimed at your changes are caught without Batfish. Pinning the real policy value in d005831 was the right instinct.
Requesting changes for one thing: the banner. app.js:1429-1437 still prints "This change opens access that is currently blocked" whenever warning === true, and :1396 still documents warning as "a high-severity OPENING". Stage a requirement for UDP/53 to 218.8.104.58 with violation_severity: "high" and propose "block 10.10.10.0/24 to 218.8.104.58 on udp/53 on rtr-us5". The generated line is deny udp … eq 53, and the real app.js (run through tests/js/propose_render_harness.js) shows the amber banner "This change opens access…" directly above your new "not simply opening access" answer. On dc93ccb that request gives warning=False, so this PR is what makes the false banner reachable. The PR's acceptance box says the opposite. I agree with keeping the trigger on severity. Please return the direction fact (for example opens_access = every high finding is opening), key the banner text on it, and add a closing case to the render harness.
@shubhamkataria2005, your comment says the banner no longer says "opens access". Your own output shows why it still does: warning: True next to direction=closing is exactly the response app.js:1429-1437 renders as "This change opens access that is currently blocked". The answer text you quoted is right. The banner above it is not, and this PR doesn't touch app.js.
Before merge, also:
directionlives on the finding insideanalysis/. Theexplanationprecedent you cite is attached inweb/main.py:1012, at the response layer, explicitly so thatanalyse()"still returns… the F-1 shape unchanged" (:929). A-3 took four signatures for an optional key. This one changesanalyse_change()'s output, the CLI JSON, and the model prompt (ai/explain.py:671dumps the whole finding). Either raise it as A-5 or keep the findings to F-1 and return direction alongside them. Shubham raised the same point on 20 October and offered to changeanalyse_change()'s signature with you. His #406 already answers it the other way: it writes direction into the evidence text ("Direction: changed.",analysis/change_impact.py:569on #406) and adds no key. So the two open PRs disagree, and I'd like the team to settle it once, for both. That's a team call, not a reason to stop.- One test for the last link. Replace
set_active_policy(policy)with(None)inanalyse_changeand the whole suite still passes, so this PR's feature could silently revert. A stub-level test thatanalyse_change(policy=P)asks P's rules would close it.
Worth knowing, not yours to fix here:
- A policy naming a device that isn't in the upload now gives
verified=Falsewith "BatfishException: Work terminated abnormally" (dc93ccb: verified). An empty policy now produces no policy statement at all, although/api/policypromises "every policy-driven check will say so". Those are #402 items 1 and 3, which Shubham's #406 now claims to fix. I'm reviewing that separately. - D1 is unchanged for users without a policy, as you said. It is fixed in practice for anyone who stages a policy naming their own device.
- #298 merges cleanly with this but
propose_and_scanstill callspropose_change(request, before_dir, host=host)withoutpolicy(ai/propose.py:638in the merged tree) while its full scan gets it, and the download path atweb/main.py:2481does the same. Whichever lands second needs to thread it through.
Nits: ai/propose.py:47-54 still argues that closing is always medium. The normalised parameter added to test_direction_is_stated_in_this_modules_own_terms is never used. The body's "byte-identical" isn't quite true: every found impact finding now carries direction.
SamikaPerera
left a comment
There was a problem hiding this comment.
Reviewed f4668d0 merged onto today's main (db59b00). It merges cleanly. Requesting changes for the same two things Arsh raised on 3 November. I checked both again and they still hold. The web/main.py part is right.
What works
/api/proposenow passespolicy=_staged_policy(), the same accessor/api/findingsuses. Live against Batfish: I loaded the sample, staged a one-rule policy (DNS to 218.8.104.58 must work,violation_severity: "high"), and proposedblock 10.10.10.0/24 to 218.8.104.58 on udp/53 on rtr-us5. Only my rule was evaluated, and its source reads "the policy file you supplied; rule 1".test_a_staged_policy_reaches_propose_changepins that line. Withpolicy=_staged_policy()removed fromweb/main.py, that test fails (1 failed, 16 passed).
1. The banner still says the opposite of the answer. The same live request returns warning: True, with both impact findings direction=closing, and the answer "…has a serious effect that is not simply opening access". app.js:1838-1846 on the merged tree shows "This change opens access that is currently blocked" whenever warning === true. I rendered that response through the real app.js (a copy of tests/js/propose_render_harness.js with one added case). The banner reads "This change opens access that is currently blocked." directly above the answer saying it isn't. The comment at app.js:1805 still describes warning as "a high-severity OPENING". app.js is in web/, which CODEOWNERS gives me, so I'm fine with this PR changing it. Arsh's suggestion works for me: a direction fact in the response, the banner text keyed on it, and a closing case in the render harness.
2. The last link is still unpinned. With set_active_policy(policy) changed to set_active_policy(None) at analysis/change_impact.py:629, the whole suite still passes on the merged tree (2227 passed, 13 skipped, with test_no_configs_in_git.py left out because my copy was not a git checkout). So the feature could silently revert.
On direction: I have no position yet. I'll answer on the ratification table when there is one.
Smaller, not blocking: propose_and_scan() still calls propose_change(request, before_dir, host=host) without policy (ai/propose.py:638 merged). Live, /api/propose/full-scan returns only proposed_change, findings, grounded and answer, and its policy finding did use my rule (PC-001 found high). So I found no visible effect, but the impact it computes there ignores the staged policy.
What I ran
- The full suite on a plain copy of
mainand of the merge. Main: 2220 passed, 13 skipped. Merge: 2231 passed, 13 skipped. Both had the same 20 failures, all intests/test_no_configs_in_git.py, because the copies are not git checkouts. No test fails on only one side. ruff check --select E4,E7,E9,F analysis/ ai/ web/ tools/ tests/on the merge: all checks passed.- The live sequence above, through the merged app's endpoints against local Batfish, with Ollama unreachable.
Not checked: the dashboard in a browser, a policy naming a device that isn't in the upload, and #406's interaction with this PR.
Every existing test here stubs _policy_change_findings() itself, so none of them exercise the line that installs `policy` via policy_module.set_active_policy(). Found in review (#405, Samika): changing that one argument to None left the whole suite green. Drives the real pc.rules_in_use() path instead, with a policy naming a device and a violation summary the built-in rules never use, so a silently-dropped policy argument is detectable by its absence from the result rather than by a call count alone. Mutation-tested against the exact change Samika made: the supplied rule's own text never reaches the comparison, and the failure output shows the built-in rules being used instead.
app.js rendered every warning=true response as "This change opens access that is currently blocked", because that used to be the only thing warning could mean. It no longer is: a high-severity POLICY finding can be a requirement breached, which CLOSES access, and the banner kept claiming the opposite (#405 review, Arsh + Samika, both reproduced live against real Batfish). ai/propose.py now returns opens_access alongside warning, computed with the same direction check the answer text already used -- one place decides the fact, not every renderer that reads the response. app.js keys the banner text on it, defaulting to the generic "serious effect" wording on anything but opens_access === true, the same cautious-default discipline warning/verified already use. Also fixes a related gap Samika found: propose_and_scan() threaded the staged policy into the full-scan pipeline but not into its own call to propose_change(), so the "impact" list it also returns stayed evaluated against the built-in example rules regardless of what the caller staged. Render harness: added a closing-direction scenario and a malformed- opens_access scenario, matching the file's own existing rigor for malformed warning/verified flags. Mutation-tested both backend changes and the frontend banner logic; each failure reproduces the exact bug it guards against.
|
Fixed both items both of you independently confirmed, plus the smaller one Samika found. The banner (Arsh's #1, Samika's #1): returns a new The missing test (Samika's #2): every existing test stubbed Also fixed: Full detail and the live-against-real-Batfish verification I ran before this is in the updated PR description. Samika, the |
SamikaPerera
left a comment
There was a problem hiding this comment.
Approving 0ed8ece. I checked it on a scratch merge with main (d424e05), against real Batfish.
Both of my points are fixed.
-
The banner. I ran my closing request through /api/propose: verified True, warning True, opens_access False, with CH-001 and CH-002 both closing. Through the real app.js (the render harness) it now shows the generic wording, not "opens access". An opens_access that is not exactly true also gets the generic wording.
-
The policy link. With set_active_policy(policy) changed to set_active_policy(None) at analysis/change_impact.py:629, test_the_supplied_policy_is_what_gets_installed_not_dropped fails (1 failed, 2274 passed).
I also checked the other two things this PR fixes by breaking them. With the banner ignoring opens_access, 2 tests fail. With propose_and_scan()'s inner propose_change() losing the policy, 1 test fails. So none of the three can silently regress.
Full suite on the merge: 2275 passed, 13 skipped. ruff (CI's rule set) is clean. Thanks for taking the propose_and_scan call site as well.
Not checked: the banner in a browser (harness and one live API call only), and the MIXED_IMPACT case live.
Two facts, not blocking: /api/propose/download still calls propose_change without the policy (it only uses the result to build the file), and /api/propose/full-scan does not return warning or opens_access. I did not re-check that second one at this head.
Direction: I still have no position and will answer on the ratification table when it exists. This approval does not depend on it.
ARSH871-bot
left a comment
There was a problem hiding this comment.
Reviewed 0ed8ece, merged onto today's main (d424e05). Both things I asked for on 3 November are fixed, and the logic held up against everything I tried. I'm still requesting changes, for two small test and comment fixes and the direction table. None of them changes behaviour, and I'll re-review the same day.
Before and after, reproduced. Live Batfish, through /api/upload, /api/policy and /api/propose, with each real response rendered through that tree's real app.js. Policy: one high requirement, DNS to 218.8.104.58 must work. Request: block 10.10.10.0/24 to 218.8.104.58 on udp/53 on rtr-us5.
main d424e05 warning False. Evaluated "Netwise's built-in example policy". My rule was ignored.
f4668d0 warning True. Banner: "This change opens access that is currently blocked",
above an answer saying it is not simply opening access.
0ed8ece warning True, opens_access False, source "the policy file you supplied; rule 1".
Banner: "This change has a serious effect that is not simply opening access".
A real opening still says so: allow any to 10.20.0.5 on tcp/23 gives opens_access True and the "opens access" banner.
What I tried to break, and couldn't:
- A real opening downgraded to the generic banner. I suggested
all()on 3 November and wanted to be sure it was right, since a highchangedfinding beside a real opening would weaken the banner. Routed, connected, unrouted and null-routed destinations all stayedopening. The last two produce no reachability row at all, sochangeddidn't appear.all()holds. - No policy staged. 7 scenarios on main and on this branch are identical apart from
directionandopens_access. - The download path I flagged on 3 November.
/api/propose/downloadreturns the same bytes with and without a staged policy, because it only uses the config text. Please ignore that note. - A policy leaking between requests. The install is per-thread (
analysis/policy.py:798), so a proposal can't reach a scan running beside it. I read this one and didn't run it.
Also checked:
- CI-style suite on the merge: 2234 passed, 54 skipped (main 2218, so +16).
- Live suite on the merge: 2275 passed, 13 skipped.
- Your 188 reproduce, and ruff is clean.
- 21 of my 23 mutants are caught, with Batfish and Ollama running and with them off, including Samika's
set_active_policy(None)and all three banner mutants.
1. changed is never exercised, and it's about to go into a contract. Two mutants survive in both environments: treating changed as an opening in opens_access, and removing "changed" from _DIRECTION_NORMAL_FORM, which makes a real changed row raise KeyError. It's rare, but it's one of the three values in the table below. These two tests close it. I've run them: the real code passes and both mutants fail.
# tests/test_change_impact.py
def test_reachability_findings_say_changed_when_the_direction_is_unknown():
bf = _FakeSessionWithFlows([_flow_row("NO_ROUTE", "DENIED_IN")])
results, _ = change_impact._reachability_change_findings(bf, 1)
assert len(results) == 1
assert results[0]["severity"] == "high"
assert results[0]["direction"] == "changed"
# tests/test_propose.py
def test_an_unclassified_high_finding_warns_without_claiming_an_opening(monkeypatch, tmp_path):
before_dir = _write_snapshot(tmp_path)
_stub_batfish(monkeypatch)
_stub_change_impact(monkeypatch, [_found(severity="high", direction="changed")])
result = propose.propose_change(GOOD_REQUEST, before_dir)
assert result["warning"] is True
assert result["opens_access"] is False2. The propose_and_scan claim isn't true. ai/propose.py:651-654, the docstring at tests/test_propose.py:910 and the PR description all say this function "also returns" an impact list. It returns proposed_change, findings, grounded and answer, and live, /api/propose/full-scan has no impact anywhere. base's impact is computed and thrown away, so this threading has no visible effect, as Samika found on 3 November. My own 3 November note pointed you here, and I hadn't noticed that either. Keeping the argument is harmless, but please correct all three so they say what's true, or the test pins a fix for something that was never wrong.
3. The direction table. The description says this needs the team, not more code, but I think we're only waiting for the table to exist. You and Shubham both lean towards a table, I said I'd sign one, and Samika will answer once there is one. Please push it here as A-5 (A-4 is #381). I'll tick my row the same day, and Shubham offered the same on 2 November. It should also settle the wording, because one finding currently uses two words for the same thing. Live on this request, CH-002 has the key direction: closing and the evidence "Direction: tightening" (analysis/change_impact.py:488), and #406 writes "Direction: changed." into evidence too.
Smaller, worth folding in since there's a new commit anyway:
opens_accessis missing from_refuse()(ai/propose.py:313) and from the no-upload response (web/main.py:2451).app.jscopes, but it's a different shape from a success. The shape comment atapp.js:1769doesn't list it either.- Still open from 3 November:
ai/propose.py:53says closing ismedium, and this PR exists because a breached requirement can behigh.normalisedis still unused attests/test_change_against_policy.py:95. docs/user-guide.md:260says we warn "if the change opens access". We now warn on a serious closing too.
A suggestion for you and Samika, not blocking. "Not simply opening access" on a change that only closes reads as if it might open something too. A high closing can only be a breached requirement, because filter and reachability closings are medium. So when every high finding is closing, the banner could say it plainly: "This change blocks traffic your policy says must be allowed."
Not this PR, but it changes what lands:
- A medium requirement still gets no warning and "narrows access". That's #451, which sits on top of this.
- Once this merges, a policy naming a device not in the upload shows "BatfishException: Work terminated abnormally", and an empty policy produces no policy statement at all. Both are #402 items in Shubham's #406. This PR is what makes them reachable from the propose pane, so they should land close together. The two PRs currently conflict in
analysis/change_impact.pyandtests/test_change_against_policy.py. CH-000'ssourceis{before_dir} -> {after_dir}, so the card shows folder paths on this machine. It's on main too, and I'll file it.
A pushed fix notifies nobody, so please re-request Samika and me when it's up. Samika approved this head on 8 November, and the new commit means that approval will need repeating too.
…uilt (#405 review, Arsh) _refuse(), the module docstring, /api/propose's no-upload response and its own docstring, and the app.js shape comment all listed the six original keys and never picked up opens_access when #405 added it. app.js copes with a missing key, but a refusal's shape is now a different one from a success, which the comment should say.
…rsh) A policy-breaching closing carries its rule's own violation_severity, which can be high -- the exact case this PR fixes the banner for. The docstring still said every closing is medium, which this PR's own live demonstration already disproves.
…s returned (#405 review, Arsh) The comment and the test's own docstring both claimed this function's call to propose_change() 'builds the impact list this function also returns'. It returns proposed_change and answer only -- the Returns: block a few lines above already said so correctly. Corrected both to the real reason for threading the policy through: every caller of propose_change() should see the same policy, not because this function's own return value depends on it.
…eview) A 'changed' reachability direction (neither side clearly succeeded) is real, already produced by _reachability_change_findings(), and already handled correctly by opens_access's all() check and by _DIRECTION_NORMAL_FORM. Nothing was broken, but nothing pinned it either -- these two guard the behaviour so a future change to either site cannot silently start treating 'changed' as an opening.
…review, Arsh) test_direction_is_stated_in_this_modules_own_terms accepted 'normalised' as a parametrize value but never read it in the body -- only the sibling test two functions down did. The table already carries the right value for every case; now both tests actually check it.
…w, Arsh) Step 4 only described warning on an opening. The warning also fires on a policy-breaching closing (#402), and the guide should say so.
…rds (#405 review, Arsh) The structural direction field already uses opening/closing/changed and is already relied on by opens_access -- not what this proposes changing. What is inconsistent is the human-readable Direction: X phrase in evidence.detail: change_impact.py:488 still says tightening for the case the field calls closing, and #406 separately invented changed for its own case. Proposes one vocabulary for both. Not changing the wording itself here -- that is exactly what this table gates, same as A-4.
|
Addressed everything in the 8 November review, in seven commits: 1. 2. The 3. The Smaller items, all folded in:
Verification: Live spot-check reproducing your exact scenario (staged high-severity DNS requirement, proposing to block that flow, against real Batfish): Your banner-wording suggestion ("This change blocks traffic your policy says must be allowed" when every high finding is Not touched, as agreed: #451 (medium requirement warning), the two #406 items (device-not-in-upload exception, empty-policy statement), and Re-requesting both of you now that this is pushed. |
SamikaPerera
left a comment
There was a problem hiding this comment.
Approving 08733c9. In a clean checkout of that commit with Batfish up (run on 11 Nov), the PR's three test files give 128 passed and the full suite gives 2327 passed, 13 skipped.
Arsh's three points from 9 Nov are addressed. The two changed tests are in (12be902). The propose_and_scan comment and the test docstring now say only proposed_change and answer are read (c99f53e). A-5 is in docs/finding-format.md, with zero of four ticked.
Two small things to fix before merge. Neither changes my approval.
-
The PR body still says, at line 81: "so the "impact" list it also returns stayed evaluated against the built-in rules". The 11 November update says that sentence is wrong, but it is still there. The test name test_full_scans_own_impact_list_also_gets_the_policy (tests/test_propose.py:920) says the same thing. Could you fix the sentence and rename the test?
-
Two lines in A-5 do not match the code. I will not tick my row until they are corrected.
a. My row says the dashboard banner is downstream of the wording A-5 settles. The banner reads result.opens_access (web/static/app.js:1877), which comes from the structural field (ai/propose.py:523-525), not from the "Direction: X" sentence. Please change my row to say that.
b. The block says opens_access relies on all three values. It only compares with "opening" (ai/propose.py:524); anything else counts as not opening.
On the banner wording you asked about: my call is to keep the current wording in this PR. I will open a small follow-up for the wording on a change that only closes access.
Not checked: the banner in a browser.
ARSH871-bot
left a comment
There was a problem hiding this comment.
Re-reviewed 08733c9, merged onto today's main (ee769b4). Everything I asked for on 10 November is done. Approving, with two small leftovers that don't block.
Checked
- CI-style suite on the merge: 2301 passed, 54 skipped. Live suite on the merge: 2342 passed, 13 skipped. Ruff reports 8 findings, all in
docs/document-builds, identical onmain. - Mutation run: 23 of 23 caught, with Batfish and Ollama running and with them off. The two that survived last time are now caught:
changedcounted as an opening, and"changed"removed from_DIRECTION_NORMAL_FORM.
My points from the last review
changedtests: both are in, and they are the two I wrote.propose_and_scan: the comment atai/propose.py:657and the test docstring now say what is true. The PR description corrects itself too (the paragraph beginning "A wrong claim in two places").- The
directiontable: pushed as A-5, marked PROPOSED and 0 of 4. Correctly not changing the "Direction: tightening" wording until it is full. - Smaller items, all done:
opens_accessis in_refuse(), the no-upload response and theapp.jsshape comment; the docstring no longer says a closing is always medium; the unusednormalisedis now asserted;docs/user-guide.mdmentions the serious-closing warning.
Two leftovers, not blocking
- The PR description still has the old sentence at "Also fixed, smaller" ("the 'impact' list it also returns"), and the test is still named
test_full_scans_own_impact_list_also_gets_the_policy. Both say the thing the corrected comment now says isn't true. A rename and one edited sentence. - The banner suggestion (when every high finding is
closing, say "This change blocks traffic your policy says must be allowed") wasn't taken. It was optional; mentioning it only so it isn't lost.
A-5
I agree with the table as written. Please tick my row in your next push, so the head only moves once.
Before this merges
- Samika has already approved this head (
08733c9, 12 November), so that approval does not need repeating. - This conflicts with #406 in
analysis/change_impact.pyandtests/test_change_against_policy.py(tried today, merging this head into #406's76cdf8c). Whichever lands second resolves it, and also needs one line intest_direction_field_is_set_regardless_of_severity, which unpacks three values fromstage(). I approved #406 today, so I would expect it to land first.
|
@patelankeet2, a quick nudge on this one, since it is close. It's approved (Samika and me), and two things stand between it and
A merge-only push keeps both approvals under M-2. If the conflict resolution changes behaviour, say so and I'll re-review the same day. Thanks! |
Closes item 2 on #402 (Arsh's post-merge review of #389), assigned to me and Samika. The other items on that checklist (device-presence filtering, coverage wording, differential searchFilters, docs) are Shubham's or joint follow-ups and are not touched here.
What was wrong
1. Propose-a-change never used the staged policy.
propose_change()had nopolicyparameter at all, so/api/proposealways compared a proposed change against Netwise's built-in example rules -- even when the user had uploaded their own via/api/policy. Worse, the finding's own evidence then said "no policy file was supplied", which was true of that specific call and false about the app as a whole.2. The warning banner could claim the wrong direction.
ai/propose.py's warning text was keyed onseverity == "high"and unconditionally said "newly opens access". For a policy finding that assumption breaks: severity there tracks whether a breach was newly created, and a created breach can be a requirement violated -- which CLOSES access that should stay open, the opposite of what the text claimed.Live reproduction, before touching anything
Staged a policy with a high-severity requirement rule (DNS to the approved resolver must work), then proposed blocking that exact flow:
What changed
analysis/change_impact.py-- every finding now carries a structureddirectionkey (opening/closing/changed), attached aftermake_finding()returns (same placeexplanation/explanation_sourceare attached -- F-1's frozen schema is untouched, no four-way amendment needed). For the filter and reachability producers this was already implicit in severity; for the policy producer it was not, which is the actual bug.ai/propose.py--propose_change()acceptspolicy=None, threaded into the oneanalyse_change()call it already makes. The warning's trigger staysseverity == "high"unchanged (under-warning a serious change is the dangerous direction, same reasoning as the verified/warning split from #13/#14: propose a config change from plain English, and simulate its impact #183) -- only the wording changes, checking the realdirectionfield instead of assuming severity implies it.web/main.py--/api/proposenow passespolicy=_staged_policy(), the exact same accessor/api/findingsalready uses.Verified end to end, not just unit tests
After the fix, same live sequence through the real running server (real session, real uploaded sample, real staged policy file):
Regression check: a request with no policy staged produces byte-identical output to before this PR.
A real mutation-testing gap, found and closed in its own commit
First version of the
ai/propose.pycommit shipped with every existing test accepting apolicykeyword in its stub but none of them checking its value -- reverting thepolicy=policyline back to dropping it entirely left the whole suite green. Caught by mutation-testing my own commit, fixed in a follow-up commit (d005831) that captures the real call and pins both the supplied-policy and omitted-policy cases.Acceptance criteria (against #402 item 2)
Testing
Mutation-tested each commit independently (direction field, policy threading at both the
ai/propose.pyandweb/main.pylayers) -- each targeted revert killed exactly the test(s) it should have, nothing else.@SamikaPerera -- requesting your review since these are files #402 names as ours jointly. @ARSH871-bot -- flagging you too since you raised both findings on #389's review; happy to take a second look from you as well if you have time.
Update, 6 November -- the banner bug and the missing test, both fixed
Arsh and Samika each independently reproduced the same two problems live against real Batfish and confirmed they still held on the merged tree:
1. The banner still said "opens access" for a closing breach. The backend's own
answertext was already direction-aware since the original version of this PR, butapp.jsrendered everywarning === trueresponse as "This change opens access that is currently blocked", unconditionally -- a leftover assumption from beforewarningcould mean anything else. Fixed by returning a newopens_accessfield alongsidewarning, computed with the same direction check the answer text already used, and keying the banner on it. A malformedopens_accessdefaults to the generic wording, the same cautious-default disciplinewarning/verifiedalready use. Added a closing-direction scenario and a malformed-flag scenario to the render harness, matching its own existing rigor for the other two flags.2. The last link was unpinned. Samika found that changing
set_active_policy(policy)toset_active_policy(None)insideanalyse_change()left the whole suite green -- every existing test stubs the policy stage itself rather than exercising the real install. New test drives the realpc.rules_in_use()path with a policy the built-in rules would never produce, and is detectable by the supplied rule's own text being absent from the result, not just a call count.Also fixed, smaller:
propose_and_scan()threaded the staged policy into its full-scan pipeline call but not into its own inner call topropose_change(), so the "impact" list it also returns stayed evaluated against the built-in rules regardless of what the caller staged (Samika's find, explicitly non-blocking, fixed anyway since it's the same file and the same cause).Mutation-tested every change against the exact revert that caused each bug in the first place; each failure reproduces that bug's own symptom rather than a generic assertion failure.
The
direction/A-5 ratification question is still open and untouched by this update -- it needs the team, not more code. Still happy to draft the table's wording if that's the preferred path once the team lands on it.Update, 11 November -- Arsh's full review addressed
Arsh ran an extensive live/mutation review and confirmed both fixes above hold. Addressed everything in his list:
A wrong claim in two places, corrected, not just one. The "Update, 6 November" section above says
propose_and_scan()'s innerpropose_change()call "builds the 'impact' list this function also returns". That is not what happens: onlybase["proposed_change"]andbase["answer"]are ever read from that call's result, and this function's ownReturns:docstring already said so correctly, three lines above the comment that got it wrong. Fixed the comment and the matching test docstring (test_full_scans_own_impact_list_also_gets_the_policy) to the real reason the policy threads through: consistency of a shared helper across callers, not because this function's return value depends on it.opens_accesswas missing from three places that document or build the response shape.ai/propose.py's module docstring,_refuse(), andweb/main.py's/api/proposeno-upload response and its own docstring, plusapp.js's shape comment. All now list/set it.The "closing is medium" docstring claim is now false, and was corrected. A policy-breaching closing carries its own rule's
violation_severity, which this PR's own live demonstration already shows can behigh-- the docstring still said every closing ismedium.Two regression tests added, as given, pinning that a
"changed"direction (neither side of a reachability comparison clearly succeeded) is not treated as an opening byopens_access, and is not dropped by_DIRECTION_NORMAL_FORM. Both already held; nothing was broken, but nothing pinned it either.test_direction_is_stated_in_this_modules_own_termsnow actually checksnormalised, which it accepted as a parametrize value but never read.docs/user-guide.mdstep 4 now also mentions the serious-closing warning, not only an opening.Proposed A-5 in
docs/finding-format.md: one vocabulary fordirection, ratifyingopening/closing/changedfor both the structural field (already de facto standard) and the human-readable"Direction: X"phrasing inevidence.detail(still inconsistent --change_impact.py:488says "tightening" where the field says "closing", and #406 separately says "changed"). Not ratified, zero of four; not changing the wording itself in this PR, same discipline A-4 uses.Verification:
Mutation-tested both mutants Arsh named directly: treating
"changed"as an opening inopens_access'sall()check, and removing"changed"from_DIRECTION_NORMAL_FORM. Both caught, for the right reason (a wrong boolean and aKeyErrorrespectively), then reverted.Live spot-check against real Batfish, reproducing Arsh's exact scenario (a staged high-severity requirement rule for DNS, then proposing to block that exact flow):
warning: True,opens_access: False, answer text says "not simply opening access" -- unchanged by these edits, as expected, since none of them touch the logic itself.Arsh's banner-wording suggestion (from his review) is non-blocking and is not implemented here -- flagging it for him and Samika to discuss, since it's a product-wording choice rather than a correctness fix.
@ARSH871-bot @SamikaPerera -- re-requesting review.