Skip to content

Never show the caller's folders on change impact cards (closes #462) - #463

Merged
patelankeet2 merged 11 commits into
mainfrom
fix/462-no-paths-in-change-impact
Nov 19, 2026
Merged

patelankeet2 merged 11 commits into
mainfrom
fix/462-no-paths-in-change-impact

Conversation

@shubhamkataria2005

@shubhamkataria2005 shubhamkataria2005 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #462, and #440's N5. Stacked on #406, which rewrites change_impact.py; GitHub retargets this to main when #406 merges. Thanks @ARSH871-bot for finding it while reviewing #405.

The bug

Propose-a-change passes the session's upload folder to analyse_change(). That folder is named after the netwise_session cookie, and every card carried it in evidence.source, so the dashboard printed the cookie that httponly exists to hide, plus the user's Windows folders and a temp directory that no longer existed.

The fix

  • analyse_change(..., labels=(before, after)). With labels, every string in every finding passes through one replacement: source, summary, detail, and any error message that quotes a folder. That last one matters because a load error can contain the path.
  • ai/propose.py passes ("your uploaded config", "the proposed config").
  • With no labels, as from the command line, the paths the user typed are shown as typed: useful there, and not secret.
  • One place only. I first also put the labels at each source=, but mutation showed that copy was never load-bearing, because the replacement pass covered it. So it's gone, and the pass is the single decision point.

#440 N5: python -m analysis.change_impact … policy.json with a contradictory or invalid policy now prints Policy not loaded, so nothing was analysed. and runs nothing, exactly like analysis.pipeline, instead of a traceback.

Tests: tests/test_change_impact_no_paths.py, 9

  • Each path a folder can take: unreachable Batfish, a load error quoting the folder, a parse failure, an ordinary stage card, and a stage that names a folder in its summary and detail.
  • Live, real Batfish: propose_change() run from a copy of rtr-us5-secure under sessions/hCf7lpCUZ2e753zhNbblAA/current; the session id, the temp folder and the netwise-proposal- prefix appear nowhere in the response.
  • Without labels, the CLI paths are unchanged. Propose passes the labels. N5 is run as a real subprocess.

Mutation, 4 of 4 killed:

  • propose no longer passing labels (Arsh's exact leak) fails 2;
  • skipping the replacement pass fails 5;
  • not cleaning summaries fails 1;
  • removing the N5 handler fails 1.

Suite: 2381 passed, 13 skipped on this branch (updated 16 November), with Batfish running. ruff clean on CI's selection. One existing test stub in tests/test_propose.py needed the new labels keyword.

#440's N1 and N2 (policy_conflicts.py only) come in a separate PR from main.

shubhamkataria2005 and others added 4 commits October 21, 2026 09:25
)

#403. Deleting a whole router reported "No change detected" with a green
tick: neither compareFilters nor differentialReachability describes a device
that exists on one side only. analyse_change() now compares the device lists
of both snapshots and reports each added or removed device as `found`. A side
whose devices cannot be listed is an error (CH-052), never "no change".
Reproduced live on multi-device-10 minus rtr-dev009.

#402, from @ARSH871-bot's post-merge review of #389:

1. A rule about a device in NEITHER config is not applicable -- this change
   cannot affect it -- and is named as such, not reported as an error. Every
   built-in rule used to be CH-051 on a network without rtr-us5, so every
   proposal there was "could not be fully verified". A rule about a device on
   ONE side only is one CH-051. Reproduced live with rtr-us5 renamed edge-fw1.

3. Only policy_compliance rules are compared, and the result now says so: a
   sentence beginning "NOT compared:" names the access_control and routing
   rules it skipped, and a policy with no policy_compliance rules says no
   rule of the user's was checked.

4. A rule breached before AND after is `found`, never inside the green card.
   A differential searchFilters tells "breached the same way" (low, not
   caused by this change) from "breached by different traffic" (the rule's
   own severity, direction: changed). "Unchanged" is scoped to the rule's
   filter, because unbinding a filter changes the network without changing
   that answer.

5. _rule_breached is tested against a fake session, so CI now catches the
   four mutants only the skipped live test could: the policy passed in
   replaced by None, only the first query arm asked, the snapshot not
   switched, and the rule's kind ignored.

6. Every stage runs whatever the others did, so a failed reachability stage
   no longer silently skips the policy stage. Ordinary findings stop at 48;
   49 and up are reserved (CH-049 summarises any overflow, CH-050..054 are
   this module's own error cards), so a long diff cannot collide with them.
   The CLI prints renamed and assigned notes.

7. Docs: the module docstring lists what is reported and in what order,
   CLAUDE.md's change_impact row is current, and the policy-rules section
   moved out from above section 7's sign-off into its own section 8.

Not here: item 2 is #405 (Ankeet's). Shubham's rebase onto it, whichever
lands second, absorbs the conflict in _policy_change_findings.

Mutation-tested: device changes dropped, absent-device rules made errors,
set_active_policy(None), first arm only, reachability failure skipping the
policy stage, no cap, still-breached back in the green card, snapshot not
switched -- each fails the new tests; the arm, snapshot and kind mutants fail
WITHOUT Batfish.

1907 passed, 13 skipped with Batfish running. ruff clean on touched files.
Arsh's three blockers on #406:

1. A pure deny raised the "opens access" banner. _breach_changed now
   returns (grew, shrank) from the differential frame's two verdicts; a
   breach that only shrinks is "narrows", low, and only a breach that
   grows keeps the rule's own severity. Batfish returns one example per
   direction, measured on a change that both adds and removes breaching
   flows.
2. Device presence was case-sensitive, so policy device RTR-US5 against
   Batfish's rtr-us5 filed a real breach under a tick. Both sides are now
   casefolded, and a user's own rule about a device in neither config is
   part of CH-051, as a scan calls it PC-050.
3. CH-051 was only ever built by the test helper. The helper now returns
   the stage's own output, and CH-051 is tested through analyse_change().

Also from the review: tests that _breach_changed passes the reference
snapshot and asks every arm, a test that a failed compareFilters is CH-050
and never "No change detected", _cap keeps the most severe cards (policy
first at equal severity) instead of the first 48, the FINDING IDS block,
and the #389 credit in CLAUDE.md. Arsh's two live reproductions are now
live tests.
Propose-a-change passes the session's upload folder, which is named after
the netwise_session cookie, and every card carried it in evidence.source.
The dashboard printed the cookie httponly exists to hide, the user's own
folders and a deleted temp directory.

analyse_change() takes labels for the two sides. With them, every string in
every finding passes through one replacement: source, summary, detail, and
any error message quoting a folder. Propose passes 'your uploaded config'
and 'the proposed config'; the command line, given no labels, still shows
the paths the user typed. A second copy at each source= was tried and was
never load-bearing, so there is one place only.

Also #440 N5: the change_impact CLI reports a policy it cannot load as a
message and runs nothing, like pipeline.py, instead of a traceback.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested, not just read. Full trail below; one real, reproducible issue found.

The feature itself is solid. Re-ran your own mutations independently rather than trust the PR body's counts:

  • Propose no longer passing labels (Arsh's exact leak): 2 tests fail - matches.
  • Skipping the replacement pass entirely: 6 tests fail - stronger than the claimed 5, still a clean kill either way.
  • Not cleaning summary, only evidence: 1 test fails - matches.
  • Removing the N5 handler: 1 test fails (a real traceback where a message is expected) - matches.

All four restored cleanly afterward, git diff --stat empty. Full suite with real Batfish running: 2293 passed, 13 skipped - matches your own number exactly.

CI is actually red on this branch, for a real reason - not noise.

pytest (3.12) and pytest (3.13) both fail on 9f8fcae:

FAILED tests/test_change_impact_no_paths.py::test_propose_passes_labels - KeyError: 'labels'

Reproduced locally with no Batfish running (matching what CI actually has): same failure, same KeyError. With Batfish running, it passes. That's the diagnosis: propose_change() calls connect(), load_snapshot() and find_parse_problems() (imported straight into ai/propose.py's own namespace from analysis.pipeline) before it ever reaches change_impact.analyse_change(). Against no Batfish, connect() fails and propose_change() returns a refusal right there - the fake() your test monkeypatches onto analyse_change never gets called, so seen["labels"] is never set.

Every other test in this file either calls change_impact.analyse_change() directly, or goes through _quiet_stages(), which stubs change_impact.connect/load_snapshot/find_parse_problems - but test_propose_passes_labels goes through propose_change() instead, which has its own separate, unstubbed copies of those three. tests.yml says the suite needs neither Batfish nor Ollama, on purpose, so this test needs the same stubbing _quiet_stages() already does, just aimed at the propose module instead of change_impact:

def test_propose_passes_labels(monkeypatch, tmp_path):
    seen = {}

    def fake(before_dir, after_dir, host=None, labels=None):
        seen["labels"] = labels
        return []
    monkeypatch.setattr(propose.change_impact, "analyse_change", fake)
    monkeypatch.setattr(propose, "connect", lambda host=None: object())
    monkeypatch.setattr(propose, "load_snapshot", lambda *a, **k: None)
    monkeypatch.setattr(propose, "find_parse_problems", lambda bf: [])

    propose.propose_change("block 10.10.10.5 to 10.20.0.5 on tcp/22 on rtr-us5",
                           FIXTURES / "rtr-us5-secure")

    assert seen["labels"] == ("your uploaded config", "the proposed config")

I'd rather name the exact fix than just the symptom - requesting changes for this one, everything else here is good. The underlying feature is correct (confirmed by the live test passing and the full suite matching your claim with Batfish up), this is purely a test-isolation gap in one new test.

One more thing worth a line, not blocking: this PR is stacked on #406, whose own base (db59b002) is 9 commits behind current main. #406 is already fully approved and just sitting - might be worth merging that first so this one rebases onto something current rather than stacking further behind.

propose_change() connects, loads, reads the parse status and the device
names through its own imports before it reaches analyse_change(). With no
Batfish, as on CI, it refused early, the fake was never called, and the
test failed with KeyError. All four are stubbed now; the review's fix had
three, and device_names was the fourth.
@shubhamkataria2005

Copy link
Copy Markdown
Collaborator Author

@patelankeet2, thank you. You were right, and it was a real isolation gap, not CI noise. Fixed in bd73b65, with one addition: your three stubs weren't quite enough. After connect/load_snapshot/find_parse_problems, propose_change() also asks snapshot.device_names(bf), and a fake session returns None there, so it still refused before reaching analyse_change() (I reproduced that with your version). With the fourth stub, the test passes with every connection forced to fail, as on CI.

The red CI had a second cause: CI merges main first, and #406 had started conflicting with main in CLAUDE.md. That's resolved on #406 (d4587ca) and merged up here (40743b3). Merged result: 2362 passed, 13 skipped, ruff clean, CI green on both Pythons.

Agreed on merge order: #406 is green and approved again, so landing it first will let this and #428 retarget to main.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed bd73b65b, the fix for the CHANGES_REQUESTED above.

My original fix stubbed three of propose_change()'s pre-flight calls (connect, load_snapshot, find_parse_problems). You found the real fourth one: snapshot.device_names(bf), called right after find_parse_problems and before analyse_change() is ever reached. Confirmed by reading ai/propose.py directly: connect -> load_snapshot -> find_parse_problems -> snapshot.device_names, in that order, all through propose's own imported names.

Mutation-tested your fix to be sure the fourth stub is actually load-bearing and not redundant: removed only the device_names stub and reran the test. It fails with the exact same KeyError: 'labels' the original CI failure showed, which confirms it was the real remaining gap, not a second unrelated issue. Restored it after confirming.

Verified clean:

  • tests/test_change_impact_no_paths.py: 8 passed, 1 skipped, without Batfish running (matches CI's own stated condition)
  • Full suite without Batfish: 2316 passed, 59 skipped
  • Full suite with Batfish running: 2362 passed, 13 skipped
  • CI on GitHub (commit 40743b3): pytest 3.12 and 3.13 both green

The only change in this update is the 7-line addition to the test file; the rest of the diff since my last review is the base branch merge bringing in unrelated work already on fix/403-402-change-impact-coverage.

Approving.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 40743b3, merged onto main 2833a1e. The fix does what it says on the path that matters, and I found one leak it still lets through. It's the same leak as #462, reached through an error card on Windows, so I'm requesting changes for that and one smaller thing. Both have a tested fix below.

I reviewed #463's own change (4 files, +266 −3, the diff against #406's head). #406's 1,100 lines are in the diff against main, and they have their own reviews.

Before and after, reproduced. Live Batfish, through the real endpoints, searching every byte of each response for the session id, the session folder in every spelling, AppData, the temp folder name and the Windows user name:

                          main 2833a1e                          #463 40743b3
/api/propose              leaks: session id, the full folder,   leaks: none
                          AppData, user name  (7 matches)
/api/propose/full-scan    none                                  none

On main the session id came out in evidence.source exactly as in #462. Both responses are clean after the fix.

Also checked:

  • Live suite on main + #463: 2362 passed, 13 skipped, the same as your number. CI-style (Batfish and Ollama refused): 2316 passed, 59 skipped, ruff clean, CI green on both Pythons.
  • Ankeet's red-CI finding is fixed: test_propose_passes_labels now passes with every connection forced to fail, which is the CI condition.
  • Five real failure modes (missing before folder, a before folder with no configs, a binary config, a before folder that is really a file, a missing after folder). I checked that the raw message quotes the folder before cleaning, so the replacement is genuinely exercised, and that none of the five survives it.
  • #440 N5: python -m analysis.change_impact <before> <after> <policy-with-a-duplicate-key> gives a traceback on main and Policy not loaded, so nothing was analysed. with exit code 1 here.
  • Mutation, 11 mutants, with Batfish and Ollama running and refused. The four you and Ankeet reported are all caught: propose no longer passing labels, skipping the replacement pass, not cleaning the summary, and removing the N5 handler. Also caught: cleaning only evidence.source, labelling only one side, and swapping the two labels. Four survive, and I don't think they're separate defects. Dropping the as_posix(), resolve() or as-given str(folder) spelling changes nothing on Linux, where for a Path they are the same string as str(path), so CI can't tell them apart. On Windows as_posix() differs, so a message quoting a forward-slash path would still leak: the same blind spot as finding 1, which I'm not asking you to write extra tests for. Replacing shortest-first only matters when one folder sits inside the other, which propose_change() never does.

1. A Windows OSError still prints the folder. Python formats an OSError's filename with repr(), so on Windows every backslash in the path is doubled: [Errno 13] Permission denied: 'C:\\Users\\vhora\\…\\sessions\\<id>\\configs\\rtr.cfg'. The replacement looks for the path as written, so it misses that form. I made load_snapshot() raise exactly that error for a file under the before folder, with labels set:

detail: PermissionError: [Errno 13] Permission denied: 'C:\\Users\\vhora\\AppData\\Local\\Temp\\sessions-hCf7lpCUZ2e753zhNbblAA-wqumidmf\\configs\\rtr.cfg'

That is the full path, session folder included, with labels in force. A config file that can't be read (a lock, a permission) is plausible on the machines this runs on. The suite can't see it, because _leaks() in tests/test_change_impact_no_paths.py also only searches for str(folder) and as_posix().

The fix is three lines in the spelling set:

spellings = {str(folder), str(path), path.as_posix(), str(path.resolve())}
spellings |= {spelling.replace("\\", "\\\\") for spelling in spellings}
for spelling in spellings:
    swaps.append((spelling, label))

And a test that fails on 40743b3 and passes with the fix. The folder name contains a backslash so the doubled form differs from the plain one on any OS, CI's Linux included:

def test_an_oserror_quoting_a_file_in_the_folder_is_cleaned_too(tmp_path, monkeypatch):
    before = tmp_path / ("sessions\\" + SESSION)
    before.mkdir(parents=True)
    after = tmp_path / "netwise-proposal-xyz" / "after"
    after.mkdir(parents=True)
    locked = PermissionError(13, "Permission denied", str(before / "rtr.cfg"))
    assert SESSION in str(locked)
    _quiet_stages(monkeypatch)

    def refuse(bf, folder, *a, **k):
        raise locked
    monkeypatch.setattr(change_impact, "load_snapshot", refuse)

    results = change_impact.analyse_change(before, after, labels=LABELS)

    text = json.dumps(results)
    assert SESSION not in text
    assert "Permission denied" in text
    assert "your uploaded config" in text

2. A relative folder rewrites ordinary text. With labels and a relative folder such as ., the folder is itself a spelling to replace, and . is everywhere:

analyse_change(".", "../rtr-us5-insecure", labels=...)  ->  "10your uploaded config10your uploaded config10..."
detail: "After:  permit ip any any (line 1, PERMIT)your uploaded config Before: ..."

The web endpoint always passes absolute folders, so it isn't reachable from the UI today. But propose_change() accepts any path and always passes labels, so a relative one corrupts the evidence. I tried refusing relative folders first and rejected it: propose_change() promises never to raise, so that would turn a relative path into a crash. Resolving first is better, and it's one line at the top of analyse_change():

if labels:
    before_dir, after_dir = Path(before_dir).resolve(), Path(after_dir).resolve()

With the test:

def test_a_relative_folder_does_not_rewrite_ordinary_text(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    (tmp_path / "after").mkdir()
    finding = {"summary": "Traffic from 10.10.10.0 changed.",
               "evidence": {"detail": "Flow 10.10.10.0->8.8.8.8 (line 1). Before: deny ip any any.",
                            "source": "differentialReachability"}}
    monkeypatch.setattr(change_impact, "_analyse_change",
                        lambda *a, **k: [dict(finding, evidence=dict(finding["evidence"]))])

    results = change_impact.analyse_change(".", "after", labels=LABELS)

    assert results[0]["summary"] == "Traffic from 10.10.10.0 changed."
    assert results[0]["evidence"]["detail"] == finding["evidence"]["detail"]

Both tests fail on 40743b3 for the right reasons (the session folder in the output, and the mangled text in the assertion). With both changes, the 181 tests in the six files that exercise this code pass.

3. Merge order, and this is the blocker for landing, not for the code. The PR is stacked on #406, and its base is #406's branch, not main. python -m tools.stacked_prs says the same: FIX FIRST: gh pr edit 463 --base main. The PR description says GitHub retargets it when #406 merges. #179 lost a PR that way, so I'd retarget first, and #428 needs the same.

#406 still shows changes requested, and that is mine, from 3 November. I owe it a re-review and I'll do it next, because #463 and #428 can't land until it does.

4. #405 collides with this. #405 and #406 already conflict in analysis/change_impact.py and tests/test_change_against_policy.py, and this PR adds a conflict in tests/test_propose.py. Past the textual conflicts: #405's stub analyse_change(before_dir, after_dir, host="localhost", policy=None) doesn't accept labels, and #463's propose_change() now passes it. Whichever of the two lands second has to pass both policy= and labels= at the one call in ai/propose.py, and fix those stubs. That's from reading both diffs; I haven't run the combination.

Smaller, not blocking:

  • The description still says 2293 passed. It's 2362 now.
  • #462 should close when this lands on main. After the retarget, check that it does.

A pushed fix notifies nobody, so please re-request me and @patelankeet2 when it's up. Ankeet's changes-requested is on 9f8fcae, and bd73b65 fixed his point.

_cap keeps the most important 48, but each kept finding kept the number it
was found under, so with mixed severities the 49th found was kept as
CH-049, the id of the summary card, and later ones could sit on CH-050 to
CH-054. The kept list is renumbered in the order found. Arsh's two tests:
mixed severities give CH-001..048 plus 049, and five highs found after 55
mediums all survive the cap (the 'severity ignored' mutant that used to
survive the suite now fails).
…463 review)

An OSError formats its filename with repr(), which doubles every backslash,
so on Windows the session folder reached the card past a replacement that
looked for the path as written: the doubled spelling is now replaced too.
And a relative folder such as '.' was itself a spelling to replace, which
rewrote ordinary text ('10.10.10.0'): with labels, both folders are
resolved first. Never refused, because propose_change() promises not to
raise. Arsh's two tests, each failing without its fix.
@shubhamkataria2005
shubhamkataria2005 changed the base branch from fix/403-402-change-impact-coverage to main November 16, 2026 00:38
@shubhamkataria2005

Copy link
Copy Markdown
Collaborator Author

@ARSH871-bot, both fixed in ed86d0b, with your tests word for word:

  1. Doubled backslashes: the spelling set now includes each path with its backslashes doubled, which is repr()'s form in an OSError. Your PermissionError test fails without it and passes with it, on Linux too, thanks to the backslash in the folder name.
  2. Relative folders: with labels, both folders are resolved first, never refused, as you said, since propose_change() promises not to raise. Your test with . fails without it.

Each fix's removal fails its test (2 of 2). This branch also has #406's _cap renumbering now. Retargeted to main, as you advised after #179. Merged: 2381 passed, 13 skipped, ruff clean, CI green. The description's test count is updated too. @patelankeet2, your point was fixed earlier (bd73b65), so a fresh look would be welcome.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed ed86d0b, merged with main ee769b4. Both points from my review are fixed and I reproduced each fix. Approving, with two small test gaps you can take or leave.

Before and after, reproduced (no Batfish needed: the analysis is stubbed, and the load is made to fail the way Windows does):

                                   40743b3                          ed86d0b
OSError quoting the session        folder still in the card,        gone: 'your uploaded config\configs\rtr.cfg'
folder (doubled backslashes)       as 'C:\Users\...\sessions-...'
relative folder "." with labels    "Traffic from 10your uploaded    "Traffic from 10.10.10.0 changed."
                                   config10your uploaded config0"
absolute folders                   unchanged                        unchanged

Checked

  • CI-style suite on the PR as pushed: 2335 passed, 59 skipped. Ruff is clean. GitHub's checks are green on 3.12 and 3.13. I did not run the live suite (Docker is off on my machine).
  • Mutation, 7 mutants, Batfish and Ollama refused. Caught: removing the doubled spellings, making the doubling a no-op, removing the resolve, resolving only the after folder.
  • Three survived:
    • "Only str(folder) is doubled": equivalent now. Once the folders are resolved first, str(folder), str(path) and str(path.resolve()) are the same string, and the posix spelling has no backslashes to double.
    • "Only the before folder is resolved": a real gap. "." only proves the before side, and a relative after folder named after rewrites the word "after" in a card.
    • "Resolve even without labels": a real gap. Nothing checks that the command line still shows a relative folder as typed.

Two tests close the gaps. I ran them: they pass on ed86d0b, and each fails against its mutant.

def test_a_relative_before_or_after_folder_does_not_rewrite_the_words(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    (tmp_path / "before").mkdir()
    (tmp_path / "after").mkdir()
    detail = "The filter was a deny before the edit and a permit after it."
    monkeypatch.setattr(change_impact, "_analyse_change", lambda *a, **k: [
        {"summary": "A line moved.",
         "evidence": {"detail": detail, "source": "compareFilters"}}])

    results = change_impact.analyse_change("before", "after", labels=LABELS)

    assert results[0]["evidence"]["detail"] == detail


def test_without_labels_a_relative_folder_is_shown_as_typed(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    (tmp_path / "before").mkdir()
    (tmp_path / "after").mkdir()

    def down(host=None):
        raise ConnectionError("down")
    monkeypatch.setattr(change_impact, "connect", down)

    results = change_impact.analyse_change("before", "after")

    assert results[0]["evidence"]["source"] == "before -> after"

Merge order: this branch contains #406's commits, so land #406 first. I'm approving that one separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Change impact cards show the session cookie and local folder paths in their evidence

3 participants