Repository navigation
Never show the caller's folders on change impact cards (closes #462) - #463
Conversation
) #403. Deleting a whole router reported "No change detected" with a green tick: neither compareFilters nor differentialReachability describes a device that exists on one side only. analyse_change() now compares the device lists of both snapshots and reports each added or removed device as `found`. A side whose devices cannot be listed is an error (CH-052), never "no change". Reproduced live on multi-device-10 minus rtr-dev009. #402, from @ARSH871-bot's post-merge review of #389: 1. A rule about a device in NEITHER config is not applicable -- this change cannot affect it -- and is named as such, not reported as an error. Every built-in rule used to be CH-051 on a network without rtr-us5, so every proposal there was "could not be fully verified". A rule about a device on ONE side only is one CH-051. Reproduced live with rtr-us5 renamed edge-fw1. 3. Only policy_compliance rules are compared, and the result now says so: a sentence beginning "NOT compared:" names the access_control and routing rules it skipped, and a policy with no policy_compliance rules says no rule of the user's was checked. 4. A rule breached before AND after is `found`, never inside the green card. A differential searchFilters tells "breached the same way" (low, not caused by this change) from "breached by different traffic" (the rule's own severity, direction: changed). "Unchanged" is scoped to the rule's filter, because unbinding a filter changes the network without changing that answer. 5. _rule_breached is tested against a fake session, so CI now catches the four mutants only the skipped live test could: the policy passed in replaced by None, only the first query arm asked, the snapshot not switched, and the rule's kind ignored. 6. Every stage runs whatever the others did, so a failed reachability stage no longer silently skips the policy stage. Ordinary findings stop at 48; 49 and up are reserved (CH-049 summarises any overflow, CH-050..054 are this module's own error cards), so a long diff cannot collide with them. The CLI prints renamed and assigned notes. 7. Docs: the module docstring lists what is reported and in what order, CLAUDE.md's change_impact row is current, and the policy-rules section moved out from above section 7's sign-off into its own section 8. Not here: item 2 is #405 (Ankeet's). Shubham's rebase onto it, whichever lands second, absorbs the conflict in _policy_change_findings. Mutation-tested: device changes dropped, absent-device rules made errors, set_active_policy(None), first arm only, reachability failure skipping the policy stage, no cap, still-breached back in the green card, snapshot not switched -- each fails the new tests; the arm, snapshot and kind mutants fail WITHOUT Batfish. 1907 passed, 13 skipped with Batfish running. ruff clean on touched files.
# Conflicts: # analysis/change_impact.py
Arsh's three blockers on #406: 1. A pure deny raised the "opens access" banner. _breach_changed now returns (grew, shrank) from the differential frame's two verdicts; a breach that only shrinks is "narrows", low, and only a breach that grows keeps the rule's own severity. Batfish returns one example per direction, measured on a change that both adds and removes breaching flows. 2. Device presence was case-sensitive, so policy device RTR-US5 against Batfish's rtr-us5 filed a real breach under a tick. Both sides are now casefolded, and a user's own rule about a device in neither config is part of CH-051, as a scan calls it PC-050. 3. CH-051 was only ever built by the test helper. The helper now returns the stage's own output, and CH-051 is tested through analyse_change(). Also from the review: tests that _breach_changed passes the reference snapshot and asks every arm, a test that a failed compareFilters is CH-050 and never "No change detected", _cap keeps the most severe cards (policy first at equal severity) instead of the first 48, the FINDING IDS block, and the #389 credit in CLAUDE.md. Arsh's two live reproductions are now live tests.
Propose-a-change passes the session's upload folder, which is named after the netwise_session cookie, and every card carried it in evidence.source. The dashboard printed the cookie httponly exists to hide, the user's own folders and a deleted temp directory. analyse_change() takes labels for the two sides. With them, every string in every finding passes through one replacement: source, summary, detail, and any error message quoting a folder. Propose passes 'your uploaded config' and 'the proposed config'; the command line, given no labels, still shows the paths the user typed. A second copy at each source= was tried and was never load-bearing, so there is one place only. Also #440 N5: the change_impact CLI reports a policy it cannot load as a message and runs nothing, like pipeline.py, instead of a traceback.
patelankeet2
left a comment
There was a problem hiding this comment.
Tested, not just read. Full trail below; one real, reproducible issue found.
The feature itself is solid. Re-ran your own mutations independently rather than trust the PR body's counts:
- Propose no longer passing
labels(Arsh's exact leak): 2 tests fail - matches. - Skipping the replacement pass entirely: 6 tests fail - stronger than the claimed 5, still a clean kill either way.
- Not cleaning
summary, onlyevidence: 1 test fails - matches. - Removing the N5 handler: 1 test fails (a real traceback where a message is expected) - matches.
All four restored cleanly afterward, git diff --stat empty. Full suite with real Batfish running: 2293 passed, 13 skipped - matches your own number exactly.
CI is actually red on this branch, for a real reason - not noise.
pytest (3.12) and pytest (3.13) both fail on 9f8fcae:
FAILED tests/test_change_impact_no_paths.py::test_propose_passes_labels - KeyError: 'labels'
Reproduced locally with no Batfish running (matching what CI actually has): same failure, same KeyError. With Batfish running, it passes. That's the diagnosis: propose_change() calls connect(), load_snapshot() and find_parse_problems() (imported straight into ai/propose.py's own namespace from analysis.pipeline) before it ever reaches change_impact.analyse_change(). Against no Batfish, connect() fails and propose_change() returns a refusal right there - the fake() your test monkeypatches onto analyse_change never gets called, so seen["labels"] is never set.
Every other test in this file either calls change_impact.analyse_change() directly, or goes through _quiet_stages(), which stubs change_impact.connect/load_snapshot/find_parse_problems - but test_propose_passes_labels goes through propose_change() instead, which has its own separate, unstubbed copies of those three. tests.yml says the suite needs neither Batfish nor Ollama, on purpose, so this test needs the same stubbing _quiet_stages() already does, just aimed at the propose module instead of change_impact:
def test_propose_passes_labels(monkeypatch, tmp_path):
seen = {}
def fake(before_dir, after_dir, host=None, labels=None):
seen["labels"] = labels
return []
monkeypatch.setattr(propose.change_impact, "analyse_change", fake)
monkeypatch.setattr(propose, "connect", lambda host=None: object())
monkeypatch.setattr(propose, "load_snapshot", lambda *a, **k: None)
monkeypatch.setattr(propose, "find_parse_problems", lambda bf: [])
propose.propose_change("block 10.10.10.5 to 10.20.0.5 on tcp/22 on rtr-us5",
FIXTURES / "rtr-us5-secure")
assert seen["labels"] == ("your uploaded config", "the proposed config")I'd rather name the exact fix than just the symptom - requesting changes for this one, everything else here is good. The underlying feature is correct (confirmed by the live test passing and the full suite matching your claim with Batfish up), this is purely a test-isolation gap in one new test.
One more thing worth a line, not blocking: this PR is stacked on #406, whose own base (db59b002) is 9 commits behind current main. #406 is already fully approved and just sitting - might be worth merging that first so this one rebases onto something current rather than stacking further behind.
propose_change() connects, loads, reads the parse status and the device names through its own imports before it reaches analyse_change(). With no Batfish, as on CI, it refused early, the fake was never called, and the test failed with KeyError. All four are stubbed now; the review's fix had three, and device_names was the fourth.
…pact-coverage # Conflicts: # CLAUDE.md
…hs-in-change-impact
|
@patelankeet2, thank you. You were right, and it was a real isolation gap, not CI noise. Fixed in The red CI had a second cause: CI merges Agreed on merge order: #406 is green and approved again, so landing it first will let this and #428 retarget to |
patelankeet2
left a comment
There was a problem hiding this comment.
Re-reviewed bd73b65b, the fix for the CHANGES_REQUESTED above.
My original fix stubbed three of propose_change()'s pre-flight calls (connect, load_snapshot, find_parse_problems). You found the real fourth one: snapshot.device_names(bf), called right after find_parse_problems and before analyse_change() is ever reached. Confirmed by reading ai/propose.py directly: connect -> load_snapshot -> find_parse_problems -> snapshot.device_names, in that order, all through propose's own imported names.
Mutation-tested your fix to be sure the fourth stub is actually load-bearing and not redundant: removed only the device_names stub and reran the test. It fails with the exact same KeyError: 'labels' the original CI failure showed, which confirms it was the real remaining gap, not a second unrelated issue. Restored it after confirming.
Verified clean:
- tests/test_change_impact_no_paths.py: 8 passed, 1 skipped, without Batfish running (matches CI's own stated condition)
- Full suite without Batfish: 2316 passed, 59 skipped
- Full suite with Batfish running: 2362 passed, 13 skipped
- CI on GitHub (commit 40743b3): pytest 3.12 and 3.13 both green
The only change in this update is the 7-line addition to the test file; the rest of the diff since my last review is the base branch merge bringing in unrelated work already on fix/403-402-change-impact-coverage.
Approving.
ARSH871-bot
left a comment
There was a problem hiding this comment.
Reviewed 40743b3, merged onto main 2833a1e. The fix does what it says on the path that matters, and I found one leak it still lets through. It's the same leak as #462, reached through an error card on Windows, so I'm requesting changes for that and one smaller thing. Both have a tested fix below.
I reviewed #463's own change (4 files, +266 −3, the diff against #406's head). #406's 1,100 lines are in the diff against main, and they have their own reviews.
Before and after, reproduced. Live Batfish, through the real endpoints, searching every byte of each response for the session id, the session folder in every spelling, AppData, the temp folder name and the Windows user name:
main 2833a1e #463 40743b3
/api/propose leaks: session id, the full folder, leaks: none
AppData, user name (7 matches)
/api/propose/full-scan none none
On main the session id came out in evidence.source exactly as in #462. Both responses are clean after the fix.
Also checked:
- Live suite on
main+ #463: 2362 passed, 13 skipped, the same as your number. CI-style (Batfish and Ollama refused): 2316 passed, 59 skipped, ruff clean, CI green on both Pythons. - Ankeet's red-CI finding is fixed:
test_propose_passes_labelsnow passes with every connection forced to fail, which is the CI condition. - Five real failure modes (missing before folder, a before folder with no
configs, a binary config, a before folder that is really a file, a missing after folder). I checked that the raw message quotes the folder before cleaning, so the replacement is genuinely exercised, and that none of the five survives it. - #440 N5:
python -m analysis.change_impact <before> <after> <policy-with-a-duplicate-key>gives a traceback onmainandPolicy not loaded, so nothing was analysed.with exit code 1 here. - Mutation, 11 mutants, with Batfish and Ollama running and refused. The four you and Ankeet reported are all caught: propose no longer passing
labels, skipping the replacement pass, not cleaning the summary, and removing the N5 handler. Also caught: cleaning onlyevidence.source, labelling only one side, and swapping the two labels. Four survive, and I don't think they're separate defects. Dropping theas_posix(),resolve()or as-givenstr(folder)spelling changes nothing on Linux, where for aPaththey are the same string asstr(path), so CI can't tell them apart. On Windowsas_posix()differs, so a message quoting a forward-slash path would still leak: the same blind spot as finding 1, which I'm not asking you to write extra tests for. Replacing shortest-first only matters when one folder sits inside the other, whichpropose_change()never does.
1. A Windows OSError still prints the folder. Python formats an OSError's filename with repr(), so on Windows every backslash in the path is doubled: [Errno 13] Permission denied: 'C:\\Users\\vhora\\…\\sessions\\<id>\\configs\\rtr.cfg'. The replacement looks for the path as written, so it misses that form. I made load_snapshot() raise exactly that error for a file under the before folder, with labels set:
detail: PermissionError: [Errno 13] Permission denied: 'C:\\Users\\vhora\\AppData\\Local\\Temp\\sessions-hCf7lpCUZ2e753zhNbblAA-wqumidmf\\configs\\rtr.cfg'
That is the full path, session folder included, with labels in force. A config file that can't be read (a lock, a permission) is plausible on the machines this runs on. The suite can't see it, because _leaks() in tests/test_change_impact_no_paths.py also only searches for str(folder) and as_posix().
The fix is three lines in the spelling set:
spellings = {str(folder), str(path), path.as_posix(), str(path.resolve())}
spellings |= {spelling.replace("\\", "\\\\") for spelling in spellings}
for spelling in spellings:
swaps.append((spelling, label))And a test that fails on 40743b3 and passes with the fix. The folder name contains a backslash so the doubled form differs from the plain one on any OS, CI's Linux included:
def test_an_oserror_quoting_a_file_in_the_folder_is_cleaned_too(tmp_path, monkeypatch):
before = tmp_path / ("sessions\\" + SESSION)
before.mkdir(parents=True)
after = tmp_path / "netwise-proposal-xyz" / "after"
after.mkdir(parents=True)
locked = PermissionError(13, "Permission denied", str(before / "rtr.cfg"))
assert SESSION in str(locked)
_quiet_stages(monkeypatch)
def refuse(bf, folder, *a, **k):
raise locked
monkeypatch.setattr(change_impact, "load_snapshot", refuse)
results = change_impact.analyse_change(before, after, labels=LABELS)
text = json.dumps(results)
assert SESSION not in text
assert "Permission denied" in text
assert "your uploaded config" in text2. A relative folder rewrites ordinary text. With labels and a relative folder such as ., the folder is itself a spelling to replace, and . is everywhere:
analyse_change(".", "../rtr-us5-insecure", labels=...) -> "10your uploaded config10your uploaded config10..."
detail: "After: permit ip any any (line 1, PERMIT)your uploaded config Before: ..."
The web endpoint always passes absolute folders, so it isn't reachable from the UI today. But propose_change() accepts any path and always passes labels, so a relative one corrupts the evidence. I tried refusing relative folders first and rejected it: propose_change() promises never to raise, so that would turn a relative path into a crash. Resolving first is better, and it's one line at the top of analyse_change():
if labels:
before_dir, after_dir = Path(before_dir).resolve(), Path(after_dir).resolve()With the test:
def test_a_relative_folder_does_not_rewrite_ordinary_text(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
(tmp_path / "after").mkdir()
finding = {"summary": "Traffic from 10.10.10.0 changed.",
"evidence": {"detail": "Flow 10.10.10.0->8.8.8.8 (line 1). Before: deny ip any any.",
"source": "differentialReachability"}}
monkeypatch.setattr(change_impact, "_analyse_change",
lambda *a, **k: [dict(finding, evidence=dict(finding["evidence"]))])
results = change_impact.analyse_change(".", "after", labels=LABELS)
assert results[0]["summary"] == "Traffic from 10.10.10.0 changed."
assert results[0]["evidence"]["detail"] == finding["evidence"]["detail"]Both tests fail on 40743b3 for the right reasons (the session folder in the output, and the mangled text in the assertion). With both changes, the 181 tests in the six files that exercise this code pass.
3. Merge order, and this is the blocker for landing, not for the code. The PR is stacked on #406, and its base is #406's branch, not main. python -m tools.stacked_prs says the same: FIX FIRST: gh pr edit 463 --base main. The PR description says GitHub retargets it when #406 merges. #179 lost a PR that way, so I'd retarget first, and #428 needs the same.
#406 still shows changes requested, and that is mine, from 3 November. I owe it a re-review and I'll do it next, because #463 and #428 can't land until it does.
4. #405 collides with this. #405 and #406 already conflict in analysis/change_impact.py and tests/test_change_against_policy.py, and this PR adds a conflict in tests/test_propose.py. Past the textual conflicts: #405's stub analyse_change(before_dir, after_dir, host="localhost", policy=None) doesn't accept labels, and #463's propose_change() now passes it. Whichever of the two lands second has to pass both policy= and labels= at the one call in ai/propose.py, and fix those stubs. That's from reading both diffs; I haven't run the combination.
Smaller, not blocking:
- The description still says
2293 passed. It's 2362 now. #462should close when this lands onmain. After the retarget, check that it does.
A pushed fix notifies nobody, so please re-request me and @patelankeet2 when it's up. Ankeet's changes-requested is on 9f8fcae, and bd73b65 fixed his point.
_cap keeps the most important 48, but each kept finding kept the number it was found under, so with mixed severities the 49th found was kept as CH-049, the id of the summary card, and later ones could sit on CH-050 to CH-054. The kept list is renumbered in the order found. Arsh's two tests: mixed severities give CH-001..048 plus 049, and five highs found after 55 mediums all survive the cap (the 'severity ignored' mutant that used to survive the suite now fails).
…hs-in-change-impact
…463 review) An OSError formats its filename with repr(), which doubles every backslash, so on Windows the session folder reached the card past a replacement that looked for the path as written: the doubled spelling is now replaced too. And a relative folder such as '.' was itself a spelling to replace, which rewrote ordinary text ('10.10.10.0'): with labels, both folders are resolved first. Never refused, because propose_change() promises not to raise. Arsh's two tests, each failing without its fix.
|
@ARSH871-bot, both fixed in
Each fix's removal fails its test (2 of 2). This branch also has #406's |
ARSH871-bot
left a comment
There was a problem hiding this comment.
Re-reviewed ed86d0b, merged with main ee769b4. Both points from my review are fixed and I reproduced each fix. Approving, with two small test gaps you can take or leave.
Before and after, reproduced (no Batfish needed: the analysis is stubbed, and the load is made to fail the way Windows does):
40743b3 ed86d0b
OSError quoting the session folder still in the card, gone: 'your uploaded config\configs\rtr.cfg'
folder (doubled backslashes) as 'C:\Users\...\sessions-...'
relative folder "." with labels "Traffic from 10your uploaded "Traffic from 10.10.10.0 changed."
config10your uploaded config0"
absolute folders unchanged unchanged
Checked
- CI-style suite on the PR as pushed: 2335 passed, 59 skipped. Ruff is clean. GitHub's checks are green on 3.12 and 3.13. I did not run the live suite (Docker is off on my machine).
- Mutation, 7 mutants, Batfish and Ollama refused. Caught: removing the doubled spellings, making the doubling a no-op, removing the resolve, resolving only the after folder.
- Three survived:
- "Only
str(folder)is doubled": equivalent now. Once the folders are resolved first,str(folder),str(path)andstr(path.resolve())are the same string, and the posix spelling has no backslashes to double. - "Only the before folder is resolved": a real gap.
"."only proves the before side, and a relative after folder namedafterrewrites the word "after" in a card. - "Resolve even without labels": a real gap. Nothing checks that the command line still shows a relative folder as typed.
- "Only
Two tests close the gaps. I ran them: they pass on ed86d0b, and each fails against its mutant.
def test_a_relative_before_or_after_folder_does_not_rewrite_the_words(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
(tmp_path / "before").mkdir()
(tmp_path / "after").mkdir()
detail = "The filter was a deny before the edit and a permit after it."
monkeypatch.setattr(change_impact, "_analyse_change", lambda *a, **k: [
{"summary": "A line moved.",
"evidence": {"detail": detail, "source": "compareFilters"}}])
results = change_impact.analyse_change("before", "after", labels=LABELS)
assert results[0]["evidence"]["detail"] == detail
def test_without_labels_a_relative_folder_is_shown_as_typed(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
(tmp_path / "before").mkdir()
(tmp_path / "after").mkdir()
def down(host=None):
raise ConnectionError("down")
monkeypatch.setattr(change_impact, "connect", down)
results = change_impact.analyse_change("before", "after")
assert results[0]["evidence"]["source"] == "before -> after"Merge order: this branch contains #406's commits, so land #406 first. I'm approving that one separately.
Closes #462, and #440's N5. Stacked on #406, which rewrites
change_impact.py; GitHub retargets this tomainwhen #406 merges. Thanks @ARSH871-bot for finding it while reviewing #405.The bug
Propose-a-change passes the session's upload folder to
analyse_change(). That folder is named after thenetwise_sessioncookie, and every card carried it inevidence.source, so the dashboard printed the cookie thathttponlyexists to hide, plus the user's Windows folders and a temp directory that no longer existed.The fix
analyse_change(..., labels=(before, after)). With labels, every string in every finding passes through one replacement:source, summary, detail, and any error message that quotes a folder. That last one matters because a load error can contain the path.ai/propose.pypasses("your uploaded config", "the proposed config").source=, but mutation showed that copy was never load-bearing, because the replacement pass covered it. So it's gone, and the pass is the single decision point.#440 N5:
python -m analysis.change_impact … policy.jsonwith a contradictory or invalid policy now printsPolicy not loaded, so nothing was analysed.and runs nothing, exactly likeanalysis.pipeline, instead of a traceback.Tests:
tests/test_change_impact_no_paths.py, 9propose_change()run from a copy ofrtr-us5-secureundersessions/hCf7lpCUZ2e753zhNbblAA/current; the session id, the temp folder and thenetwise-proposal-prefix appear nowhere in the response.Mutation, 4 of 4 killed:
Suite:
2381 passed, 13 skippedon this branch (updated 16 November), with Batfish running. ruff clean on CI's selection. One existing test stub intests/test_propose.pyneeded the newlabelskeyword.#440's N1 and N2 (
policy_conflicts.pyonly) come in a separate PR frommain.