Skip to content

Close five paths to a false all-clear - #442

Merged
ARSH871-bot merged 2 commits into
mainfrom
fix/stability-false-all-clears
Nov 9, 2026
Merged

ARSH871-bot merged 2 commits into
mainfrom
fix/stability-false-all-clears

Conversation

@ARSH871-bot

@ARSH871-bot ARSH871-bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Five ways Netwise could say "checked and clean" when nothing was checked. Each one is the F-4 failure CLAUDE.md §7a calls the worst this project can have: none (or grounded: True) standing in for "could not check". All five were found in a stability audit (21–30 Oct) and re-measured on current main (db59b00, after the 2 November merges).

What changes

Where Before, on main After
R1 analysis/checks/policy_compliance.py A user rule with "queries": [] asks Batfish nothing, then reports PC-000 none "All 1 policy rule(s) … hold" That rule is a "could not check" error (PC-051), through the existing error path
R2 ai/query.py A config Batfish could not read at all (status UNKNOWN, zero devices): "are there any dead rules?" got "No, checked the whole snapshot and found none.", with grounded: True Refused with grounded: False, naming the unread file. A partly read config answers with a caveat, as analyse() does (#217)
R4 analysis/pipeline.py A post-processor could change a finding's status (error → none, found → none) and nothing noticed. docs/severity-rules.md §2 says this is enforced The status is restored, and an error card names the stage, the same way the existing drop and downgrade guards work
R5 analysis/checks/routing.py A traceroute row with zero traces counted as "holds" for both REACHABLE and UNREACHABLE Refused like a bad expected value. That one route reports "could not check", and the other routes and the policy-free analyses still run
R6 tests only Nine F-4 guards that no test protected (access_control ×5, routing ×2, pipeline, explain). Deleting one gave, for example, AC-000 none "1 policy statement(s) hold, 1 guarantee(s) proven" after a swallowed searchFilters failure Pinned in tests/test_error_status_is_reported.py (whole run() calls through a fake session) plus one test in test_partial_parse.py

Two follow-on changes, because #361 and #407 landed after R2 was written.

Evidence (measured on db59b00)

Each fix's tests, run on main without the fix, then with it:

R1  1 failed, 6 passed    ->  7 passed
R2  4 failed, 52 passed   ->  all pass (plus the two new shape checks)
R4  1 failed, 26 passed   ->  27 passed
R5  3 failed, 11 passed   ->  14 passed
    plus 3 new tests for the fix below: all 3 fail with the new guard removed
R6  24 passed on main by design: it pins guards that work today. Its proof is
    mutation: each of the nine guards deleted makes its test fail (measured in
    the audit, 30 Oct).

Full suite at 883f9af: 2226 passed, 54 skipped, 0 failed with Batfish down, on CI, on both Python versions. Live against real Batfish: 2267 passed, 13 skipped, 0 failed.

Before that fix, at 331e489, a live run gave 2262 passed, 13 skipped, 2 failed: test_draft_policy.py::test_flipping_one_decision_produces_a_real_finding and test_parse_coverage.py::test_real_vendor_asa_caveat_states_how_much_was_read. Neither file is touched here. Both passed on re-run, twice each, on this branch and on main, and both passed in a live run of all sixteen audit fixes combined. So they're intermittent under a full live run, not caused by this PR. Worth an issue if they recur. Lint: CI's exact command (ruff check --select E4,E7,E9,F analysis/ ai/ web/ tools/ tests/, ruff 0.14.14) reports all checks passed.

Update after @patelankeet2's review (883f9af)

R5 made _evaluate() raise for zero traces, and its call in run() was the one place in the loop with no handling of its own. So one such route ended the whole routing check. Fixed as requested: that route now gets its own error_finding(), the same as the two cases above it, and the check carries on.

Three routes, the first with no traces
  331e489   1 of 3 routes asked, policy-free analyses not run, one RT-000 error
  883f9af   3 of 3 asked, policy-free analyses run, RT-001 error on that route,
            the real violation still found, all-clear does not name the unchecked route

A bad expected value now takes the same path, so it reports on that route alone rather than through the pipeline's crash handling. Its comment changed to match.

One thing I could not reproduce. test_policy_numbers.py::test_routing_at_its_limit_produces_no_shared_id passes here at 331e489 and at 883f9af, three runs each way. On this machine that route returns one NO_ROUTE trace and not zero, on Batfish 2025.07.07.2423. So the new tests pin the mechanism directly with a fake session. @patelankeet2, please re-run that test at 883f9af, as your environment hits a case mine does not.

Notes for review

  • ai/query.py now imports classify_parse_status from analysis.pipeline, the same classification analyse() uses, so the two cannot disagree.
  • /api/ask reads resolved_entities with .pop(..., None), so the missing key never crashed anything. The fix is for the documented contract, and for the next caller that reads it directly.
  • No F-1 change: no new keys, no new statuses.

@patelankeet2: ai/query.py, routing.py and the explain guard. @shubhamkataria2005: policy_compliance.py and pipeline.py.

Each of these let Netwise report "checked and clean" (status none, or
grounded True) when nothing was checked -- the F-4 failure CLAUDE.md 7a
calls the worst this project can have. Found in a stability audit
(21-30 Oct) and re-measured on main db59b00.

- policy_compliance: a user rule with no queries asked Batfish nothing
  and was reported as holding. It is now "could not check".
- ai/query.py: a config Batfish could not read at all answered "No,
  checked the whole snapshot", grounded. It now refuses and names the
  file; a partly read config answers with a caveat, as analyse() does.
  Refusals keep the four-key shape from #318 (resolved_entities None).
- pipeline: a post-processor could change a finding's status unnoticed.
  The status is now restored and the stage named, like the existing
  drop and downgrade guards.
- routing: a traceroute with zero traces counted as "holds". It is now
  refused, so the check reports could not check.
- tests: nine F-4 guards no test protected are pinned; deleting any one
  now fails its test.

tools/query_quality.py's fake fileParseStatus now returns the three
columns real Batfish returns, which the new parse check reads; #407's
score is unchanged.
@ARSH871-bot ARSH871-bot added bug Something isn't working layer: analysis Layer 1 — Batfish orchestration, checks, the pipeline layer: ai Layer 2 — local LLM explanation and Q&A safety Touches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care labels Nov 3, 2026

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-tested in an isolated worktree at the branch's real head (331e489). R1, R2, R4, and R6 all check out, but R5 has a real regression I found through the full suite, not through reading the diff.

R1 (policy_compliance.py): straightforward, appends to the existing failures list rather than inventing a new error path. Fine as-is.

R2 (ai/query.py): mutation-tested by disabling both call sites of _what_was_not_read() while leaving the helper defined but unused, matching the actual shape of the bug on main. Result: 4 failed, 53 passed, matching the PR body's claimed 4 failed, 52 passed exactly (the one extra pass is from #361 landing since). Checked the two unchanged branches of _answer_reachability_question (the except and frame.empty cases) still return grounded: False without the caveat appended, which is correct, a caveat only belongs on a confident answer.

R4 (pipeline.py): mutation-tested by deleting the new status-restore block entirely. test_changing_a_finding_status_is_undone_and_reported fails for the right reason, an error finding silently becomes none. Restored, clean.

R6: all nine guard tests pass; didn't re-derive the mutation-per-guard claim myself given time, but the shape (run() through a fake session) matches the file's own established convention.

R5 (routing.py) has a real blast-radius bug. Full suite on this branch: 1 failed, 2263 passed, 13 skipped -- test_policy_numbers.py::test_routing_at_its_limit_produces_no_shared_id. Unlike the test_draft_policy.py flakiness #442's own body already discloses (and that I independently hit too, see #448), this one is NOT a flake: it fails the same way every time, in isolation, deterministically.

Bisected it properly rather than guessing: swapped main's original routing.py into this branch and the test passes; swapped this branch's routing.py back in and it fails. So this specific change causes it.

The mechanism: _evaluate()'s call site at routing.py:474 (violation = _evaluate(route["expected"], row["Traces"])) has no try/except around it, unlike its two neighbours in the same loop (the "could not ask the question" and "frame.empty" branches immediately above it, both of which already catch their own failures and convert them to a per-assertion error finding). Now that _evaluate() can raise ValueError for the zero-traces case, that exception escapes uncaught from this one call site, and it is not isolated to the one route assertion that triggered it -- it kills the rest of run(), including the hygiene findings (RT-100+) computed separately afterward. Confirmed directly: wrapping just this one call site in the same try/except pattern as its neighbours (converting the ValueError to a findings.error_finding() for that route, then continue) makes the failing test pass again, with no other change.

This is the same shape of bug R4 in this very PR fixes elsewhere (a stage's own failure taking down more than it should), just one layer down, in the one call site this PR's own R5 change didn't wrap.

Requesting changes: wrap _evaluate()'s call at routing.py:474 in a try/except that converts ValueError into a per-route findings.error_finding(), the same pattern the two branches directly above it already use, then re-run test_policy_numbers.py and the full suite. Everything else in this PR is solid and I'd approve on sight once that one call site is fixed.

Found by Ankeet reviewing this PR. _evaluate() now raises when a route has
no traces, and its call site in run() was the one place in the loop with no
handling of its own. So one such route ended run(): the routes after it were
never asked, the policy-free analyses never ran, and the user saw a single
"failed to run" card.

That route now reports could-not-check on its own, the same way as the two
cases above it, and the rest of the check carries on. A bad "expected" value
goes the same way, so its comment changes too.

Measured with three routes where the first has no traces:
  before  1 of 3 routes asked, policy-free analyses not run, one RT-000 error
  after   3 of 3 asked, policy-free analyses run, RT-001 error on that route,
          the real violation still found, and the all-clear does not name
          the route nobody checked

Removing the guard fails all three new tests. Full suite live: 2267 passed,
13 skipped, 0 failed.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 883f9af for the parts you asked me to look at, policy_compliance.py (R1) and pipeline.py (R4). @patelankeet2's R5 point is his to re-check; your fix in 883f9af looks right to me, and the full suite agrees.

Merged onto current main (d424e05): 2286 passed, 13 skipped, ruff clean on CI's selection.

R1, my check. Routing an empty queries list into the existing failures path is the right place: it reuses PC-05n and the partial-check wording rather than inventing a new card. It also sits after my #432 "filter applied nowhere" guard, so the two can't both fire on one rule. Removing the guard fails 1 test (with the whole suite run), so it's pinned.

R4, the status guard. This is the one docs/severity-rules.md §2 has claimed for weeks without the code doing it, so thank you for closing that gap. Mutants: removing the check fails 1, and keeping the error card but not restoring the status fails 1. So both halves (report it and undo it) are pinned separately, which is what makes it a guard rather than a log line.

One suggestion, not a blocker. analysis/policy.py still accepts a rule with "queries": [] (checked: load_policy loads it). R1 now makes the scan say "could not check", which is correct. But the user would learn sooner, at upload, if the loader refused a rule that asks nothing, the same way it refuses other rules it can't use. Happy to do that in a follow-up if you'd rather keep this PR to the five audit fixes.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-verified 883f9af in an isolated worktree, with Batfish running.

R5, the fix I asked for: _evaluate()'s call at routing.py:474 is now wrapped in the same try/except pattern as its two neighbours above it, converting the ValueError into a per-route error_finding() and continuing. Confirmed by reading it, not just the diff summary.

Mutation-tested it myself: removed the try/except, leaving the bare call. The full suite (2267 non-skipped at this head) caught it in tests/test_routing_classification.py -- three tests failed for the right reason, each showing only rtr-a asked instead of all three routes, which is exactly the original blast-radius bug. Restored and confirmed clean. Worth noting for anyone else checking this: test_policy_numbers.py and test_error_status_is_reported.py alone do not catch this mutation on their own, only test_routing_classification.py does -- so a partial run of just those two files would miss a regression here.

Your specific ask: test_policy_numbers.py::test_routing_at_its_limit_produces_no_shared_id passes on my machine too, in isolation, same pybatfish version (2025.07.07.2423). It did fail once during a full live suite run just now, then passed again on an immediate re-run in isolation -- matching the intermittent-only-under-a-full-run pattern the PR body already discloses, and the same pattern #448 tracks. Not a regression from this PR.

R1, R2, R4, R6: Shubham already mutation-tested R1 and R4 independently and both held. I did not re-derive those or R6 myself given time, consistent with what I said in my first pass -- the shape matches the file's own convention and I have no reason to doubt it.

Approving. All five audit fixes check out, and the one thing I asked for is fixed and proven, not just claimed.

@ARSH871-bot
ARSH871-bot merged commit 2833a1e into main Nov 9, 2026
4 checks passed
@ARSH871-bot
ARSH871-bot deleted the fix/stability-false-all-clears branch November 9, 2026 02:39
@ARSH871-bot

Copy link
Copy Markdown
Owner Author

Thanks, and agreed. Checked on main after the merge: load_policy() still accepts a policy_compliance rule with "queries": [], so the user only finds out at scan time, from the "could not check" card #442 added. Refusing it at upload is better, because that's when they can still fix the file.

Please do take it, as you offered. It's analysis/policy.py and mine, but you know the rule shape better. Two things I'd want from it:

  • Keep Close five paths to a false all-clear #442's scan-time guard. A policy can still reach policy_compliance without going through the loader (tests, the CLI and rules_in_use()'s built-in rules), so the loader check adds to that guard rather than replacing it.
  • Say which rule. For example, "rule 3 has an empty queries list, so it asks Batfish nothing and could never fail". The same goes for a rule whose only query is {}, if that's also possible. I haven't checked that one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working layer: ai Layer 2 — local LLM explanation and Q&A layer: analysis Layer 1 — Batfish orchestration, checks, the pipeline safety Touches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants