Repository navigation
Close five paths to a false all-clear - #442
Conversation
Each of these let Netwise report "checked and clean" (status none, or grounded True) when nothing was checked -- the F-4 failure CLAUDE.md 7a calls the worst this project can have. Found in a stability audit (21-30 Oct) and re-measured on main db59b00. - policy_compliance: a user rule with no queries asked Batfish nothing and was reported as holding. It is now "could not check". - ai/query.py: a config Batfish could not read at all answered "No, checked the whole snapshot", grounded. It now refuses and names the file; a partly read config answers with a caveat, as analyse() does. Refusals keep the four-key shape from #318 (resolved_entities None). - pipeline: a post-processor could change a finding's status unnoticed. The status is now restored and the stage named, like the existing drop and downgrade guards. - routing: a traceroute with zero traces counted as "holds". It is now refused, so the check reports could not check. - tests: nine F-4 guards no test protected are pinned; deleting any one now fails its test. tools/query_quality.py's fake fileParseStatus now returns the three columns real Batfish returns, which the new parse check reads; #407's score is unchanged.
patelankeet2
left a comment
There was a problem hiding this comment.
Re-tested in an isolated worktree at the branch's real head (331e489). R1, R2, R4, and R6 all check out, but R5 has a real regression I found through the full suite, not through reading the diff.
R1 (policy_compliance.py): straightforward, appends to the existing failures list rather than inventing a new error path. Fine as-is.
R2 (ai/query.py): mutation-tested by disabling both call sites of _what_was_not_read() while leaving the helper defined but unused, matching the actual shape of the bug on main. Result: 4 failed, 53 passed, matching the PR body's claimed 4 failed, 52 passed exactly (the one extra pass is from #361 landing since). Checked the two unchanged branches of _answer_reachability_question (the except and frame.empty cases) still return grounded: False without the caveat appended, which is correct, a caveat only belongs on a confident answer.
R4 (pipeline.py): mutation-tested by deleting the new status-restore block entirely. test_changing_a_finding_status_is_undone_and_reported fails for the right reason, an error finding silently becomes none. Restored, clean.
R6: all nine guard tests pass; didn't re-derive the mutation-per-guard claim myself given time, but the shape (run() through a fake session) matches the file's own established convention.
R5 (routing.py) has a real blast-radius bug. Full suite on this branch: 1 failed, 2263 passed, 13 skipped -- test_policy_numbers.py::test_routing_at_its_limit_produces_no_shared_id. Unlike the test_draft_policy.py flakiness #442's own body already discloses (and that I independently hit too, see #448), this one is NOT a flake: it fails the same way every time, in isolation, deterministically.
Bisected it properly rather than guessing: swapped main's original routing.py into this branch and the test passes; swapped this branch's routing.py back in and it fails. So this specific change causes it.
The mechanism: _evaluate()'s call site at routing.py:474 (violation = _evaluate(route["expected"], row["Traces"])) has no try/except around it, unlike its two neighbours in the same loop (the "could not ask the question" and "frame.empty" branches immediately above it, both of which already catch their own failures and convert them to a per-assertion error finding). Now that _evaluate() can raise ValueError for the zero-traces case, that exception escapes uncaught from this one call site, and it is not isolated to the one route assertion that triggered it -- it kills the rest of run(), including the hygiene findings (RT-100+) computed separately afterward. Confirmed directly: wrapping just this one call site in the same try/except pattern as its neighbours (converting the ValueError to a findings.error_finding() for that route, then continue) makes the failing test pass again, with no other change.
This is the same shape of bug R4 in this very PR fixes elsewhere (a stage's own failure taking down more than it should), just one layer down, in the one call site this PR's own R5 change didn't wrap.
Requesting changes: wrap _evaluate()'s call at routing.py:474 in a try/except that converts ValueError into a per-route findings.error_finding(), the same pattern the two branches directly above it already use, then re-run test_policy_numbers.py and the full suite. Everything else in this PR is solid and I'd approve on sight once that one call site is fixed.
Found by Ankeet reviewing this PR. _evaluate() now raises when a route has
no traces, and its call site in run() was the one place in the loop with no
handling of its own. So one such route ended run(): the routes after it were
never asked, the policy-free analyses never ran, and the user saw a single
"failed to run" card.
That route now reports could-not-check on its own, the same way as the two
cases above it, and the rest of the check carries on. A bad "expected" value
goes the same way, so its comment changes too.
Measured with three routes where the first has no traces:
before 1 of 3 routes asked, policy-free analyses not run, one RT-000 error
after 3 of 3 asked, policy-free analyses run, RT-001 error on that route,
the real violation still found, and the all-clear does not name
the route nobody checked
Removing the guard fails all three new tests. Full suite live: 2267 passed,
13 skipped, 0 failed.
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Approving 883f9af for the parts you asked me to look at, policy_compliance.py (R1) and pipeline.py (R4). @patelankeet2's R5 point is his to re-check; your fix in 883f9af looks right to me, and the full suite agrees.
Merged onto current main (d424e05): 2286 passed, 13 skipped, ruff clean on CI's selection.
R1, my check. Routing an empty queries list into the existing failures path is the right place: it reuses PC-05n and the partial-check wording rather than inventing a new card. It also sits after my #432 "filter applied nowhere" guard, so the two can't both fire on one rule. Removing the guard fails 1 test (with the whole suite run), so it's pinned.
R4, the status guard. This is the one docs/severity-rules.md §2 has claimed for weeks without the code doing it, so thank you for closing that gap. Mutants: removing the check fails 1, and keeping the error card but not restoring the status fails 1. So both halves (report it and undo it) are pinned separately, which is what makes it a guard rather than a log line.
One suggestion, not a blocker. analysis/policy.py still accepts a rule with "queries": [] (checked: load_policy loads it). R1 now makes the scan say "could not check", which is correct. But the user would learn sooner, at upload, if the loader refused a rule that asks nothing, the same way it refuses other rules it can't use. Happy to do that in a follow-up if you'd rather keep this PR to the five audit fixes.
patelankeet2
left a comment
There was a problem hiding this comment.
Re-verified 883f9af in an isolated worktree, with Batfish running.
R5, the fix I asked for: _evaluate()'s call at routing.py:474 is now wrapped in the same try/except pattern as its two neighbours above it, converting the ValueError into a per-route error_finding() and continuing. Confirmed by reading it, not just the diff summary.
Mutation-tested it myself: removed the try/except, leaving the bare call. The full suite (2267 non-skipped at this head) caught it in tests/test_routing_classification.py -- three tests failed for the right reason, each showing only rtr-a asked instead of all three routes, which is exactly the original blast-radius bug. Restored and confirmed clean. Worth noting for anyone else checking this: test_policy_numbers.py and test_error_status_is_reported.py alone do not catch this mutation on their own, only test_routing_classification.py does -- so a partial run of just those two files would miss a regression here.
Your specific ask: test_policy_numbers.py::test_routing_at_its_limit_produces_no_shared_id passes on my machine too, in isolation, same pybatfish version (2025.07.07.2423). It did fail once during a full live suite run just now, then passed again on an immediate re-run in isolation -- matching the intermittent-only-under-a-full-run pattern the PR body already discloses, and the same pattern #448 tracks. Not a regression from this PR.
R1, R2, R4, R6: Shubham already mutation-tested R1 and R4 independently and both held. I did not re-derive those or R6 myself given time, consistent with what I said in my first pass -- the shape matches the file's own convention and I have no reason to doubt it.
Approving. All five audit fixes check out, and the one thing I asked for is fixed and proven, not just claimed.
|
Thanks, and agreed. Checked on Please do take it, as you offered. It's
|
Five ways Netwise could say "checked and clean" when nothing was checked. Each one is the F-4 failure CLAUDE.md §7a calls the worst this project can have:
none(orgrounded: True) standing in for "could not check". All five were found in a stability audit (21–30 Oct) and re-measured on currentmain(db59b00, after the 2 November merges).What changes
mainanalysis/checks/policy_compliance.py"queries": []asks Batfish nothing, then reportsPC-000 none"All 1 policy rule(s) … hold"PC-051), through the existing error pathai/query.pyUNKNOWN, zero devices): "are there any dead rules?" got "No, checked the whole snapshot and found none.", withgrounded: Truegrounded: False, naming the unread file. A partly read config answers with a caveat, asanalyse()does (#217)analysis/pipeline.pystatus(error→none,found→none) and nothing noticed. docs/severity-rules.md §2 says this is enforcedanalysis/checks/routing.pyexpectedvalue. That one route reports "could not check", and the other routes and the policy-free analyses still runAC-000 none "1 policy statement(s) hold, 1 guarantee(s) proven"after a swallowed searchFilters failuretests/test_error_status_is_reported.py(wholerun()calls through a fake session) plus one test intest_partial_parse.pyTwo follow-on changes, because #361 and #407 landed after R2 was written.
resolved_entities: R2's refusals now carry"resolved_entities": None, keepinganswer_question()'s four-key shape from A reachability follow-up can reuse what the last answer already resolved (#318) #361. A follow-up question must never reuse anything from a config Batfish could not read. Pinned by two assertions; removing the key fails both.tools/query_quality.py(A repeatable score for the question layer, run in CI (#393) #407): its fakefileParseStatusreturned onlyNodes. Real Batfish also returnsFile_NameandStatus, which R2's check reads, so without them R2 refused every question and 7 of A repeatable score for the question layer, run in CI (#393) #407's tests failed. The fake now returns the same three columns astests/test_query_translation.py's fake. A repeatable score for the question layer, run in CI (#393) #407's score is unchanged: 0 answered wrong, 0 refused that should answer.Evidence (measured on
db59b00)Each fix's tests, run on
mainwithout the fix, then with it:Full suite at
883f9af: 2226 passed, 54 skipped, 0 failed with Batfish down, on CI, on both Python versions. Live against real Batfish: 2267 passed, 13 skipped, 0 failed.Before that fix, at
331e489, a live run gave 2262 passed, 13 skipped, 2 failed:test_draft_policy.py::test_flipping_one_decision_produces_a_real_findingandtest_parse_coverage.py::test_real_vendor_asa_caveat_states_how_much_was_read. Neither file is touched here. Both passed on re-run, twice each, on this branch and onmain, and both passed in a live run of all sixteen audit fixes combined. So they're intermittent under a full live run, not caused by this PR. Worth an issue if they recur. Lint: CI's exact command (ruff check --select E4,E7,E9,F analysis/ ai/ web/ tools/ tests/, ruff 0.14.14) reports all checks passed.Update after @patelankeet2's review (
883f9af)R5 made
_evaluate()raise for zero traces, and its call inrun()was the one place in the loop with no handling of its own. So one such route ended the whole routing check. Fixed as requested: that route now gets its ownerror_finding(), the same as the two cases above it, and the check carries on.A bad
expectedvalue now takes the same path, so it reports on that route alone rather than through the pipeline's crash handling. Its comment changed to match.One thing I could not reproduce.
test_policy_numbers.py::test_routing_at_its_limit_produces_no_shared_idpasses here at331e489and at883f9af, three runs each way. On this machine that route returns oneNO_ROUTEtrace and not zero, on Batfish2025.07.07.2423. So the new tests pin the mechanism directly with a fake session. @patelankeet2, please re-run that test at883f9af, as your environment hits a case mine does not.Notes for review
ai/query.pynow importsclassify_parse_statusfromanalysis.pipeline, the same classificationanalyse()uses, so the two cannot disagree./api/askreadsresolved_entitieswith.pop(..., None), so the missing key never crashed anything. The fix is for the documented contract, and for the next caller that reads it directly.@patelankeet2:
ai/query.py,routing.pyand the explain guard. @shubhamkataria2005:policy_compliance.pyandpipeline.py.