Skip to content

Routing fact: name the real cause, not always 'routing' (#439) - #460

Merged
ARSH871-bot merged 3 commits into
mainfrom
fix/439-routing-fact-cause-attribution
Nov 19, 2026
Merged

ARSH871-bot merged 3 commits into
mainfrom
fix/439-routing-fact-cause-attribution

Conversation

@patelankeet2

Copy link
Copy Markdown
Collaborator

Closes #439.

What's wrong: _compute_routing_outcome() in ai/explain.py told the model, as an already verified fact, that "the network's current routing is what disagrees with that requirement" for EVERY failed or unexpectedly-succeeded routing assertion. Measured still present on main (d424e05): analysis/checks/routing.py's own module docstring says DENIED_IN/DENIED_OUT are filter decisions, not routing, and EXITS_NETWORK most often means the destination device named in the statement is not in this snapshot (measured there by deleting a config and re-running). Both were blamed on "routing" anyway, in a sentence the prompt labels "already verified by code and certain" - the same shape of risk #453 raised about the sibling dead-rule function.

What changed:

  • Two small disposition sets, _FILTER_DISPOSITIONS and _MISSING_DEVICE_DISPOSITIONS, declared locally in ai/explain.py rather than imported from routing.py - matching the exact boundary ai/query.py's own _SUCCESS_DISPOSITIONS already holds (re-declared there "so ai/ does not depend on a specific check module for a general Batfish-interpretation fact"). Everything else in this project's known disposition vocabulary (NO_ROUTE, NULL_ROUTED, LOOP, INSUFFICIENT_INFO, NEIGHBOR_UNREACHABLE) really is a routing outcome, so that wording is unchanged for the remaining case - not a guessed default, since these are the only two non-routing causes this project has observed or documented.
  • The mixed-result wording "(one ended in X)" named only the first of possibly several failing dispositions (routing._evaluate()'s failures[0]), so it could read as if only one path failed when the count already said otherwise. Reworded to "(at least one of them ended in X)".
  • Factored the literal "Expected ... for a flow from ..." detail-prefix out of routing.py's one call site into _violation_detail(), so the test file's routing fixtures call the real producer instead of retyping it by hand - closing the producer-drift risk the issue named, without the larger lift of building a full fake Session to drive run() end-to-end.

Explicit scope limit, documented in _compute_routing_outcome()'s own docstring rather than left implicit: routing._evaluate() itself only ever names ONE representative disposition even in a unanimous multi-path failure, so a finding where different paths fail for genuinely different reasons (one DENIED_IN, one NO_ROUTE) is still categorized by only the first. That is a limit of what routing.py reports, not something this fix can see past.

No duplication: confirmed via the GitHub search API and the open-PR list that no PR exists for #439, and it does not overlap my two other currently open PRs (#458 touches _compute_dead_rule_outcome(), #459 touches _ALL_CLEAR_PATTERN; neither touches this function or routing.py).

Testing:

  • New tests for both disposition sets (filter, missing-device) plus a control pinning that a genuine routing disposition (NO_ROUTE) still gets the original wording - without it, a fix that removed the causal claim entirely would make every new test pass for the wrong reason.
  • Updated the two existing mixed-result tests with assertions pinning the reworded "(at least one of them ended in X)" phrase.
  • A direct unit test for _violation_detail() against the real _evaluate() output.
  • Full file: pytest tests/test_explain_safety_net.py tests/test_routing_classification.py -v - 105 passed.
  • Mutation test: removed the disposition branching (reverted to always returning the "routing" clause). All 3 new cause-attribution tests failed for the right reason (fell back to the old blanket wording), while the NO_ROUTE control still passed. Restored.
  • Full suite: pytest tests/ -q - 2230 passed, 46 skipped, 0 failed.
  • Live-tested against the real model (Ollama + Batfish): 10 generations each for a DENIED_IN, an EXITS_NETWORK, and a NO_ROUTE finding. 0/30 fallbacks, and every generation correctly followed the new disposition-specific fact - filter language for all 10 DENIED_IN runs, destination-device language for all 10 EXITS_NETWORK runs, routing language only for the NO_ROUTE control.
  • Byte-level check: no em-dashes in any changed file.

)

Pure refactor, no behavior change. run() built the finding's detail
string inline, so tests/test_explain_safety_net.py's
_real_routing_detail_multi() had to retype the same literal prefix by
hand to construct a realistic detail for testing ai/explain.py against.
A reword of the real prefix could silently stop matching what that
hand-copy produces.

Pulled the prefix into its own function so the test can call the real
producer instead -- same discipline the file already applies to
_evaluate() one line above, rather than testing a copy of its output.
…ting (#439)

_compute_routing_outcome() told the model, as an already verified fact,
that 'the network's current routing is what disagrees' for every failed
or unexpectedly-succeeded routing assertion. Measured still present on
main (d424e05): analysis/checks/routing.py's own module docstring says
DENIED_IN/DENIED_OUT are filter decisions, not routing, and EXITS_NETWORK
most often means the destination device is missing from the snapshot
(measured there by deleting a config and re-running). Both were blamed
on routing anyway.

Adds two small disposition sets (filter-caused, missing-device-caused),
declared locally in ai/explain.py rather than imported from routing.py --
same boundary ai/query.py's own _SUCCESS_DISPOSITIONS already holds, and
for the same documented reason. Everything else in this project's known
disposition vocabulary (NO_ROUTE, NULL_ROUTED, LOOP, INSUFFICIENT_INFO,
NEIGHBOR_UNREACHABLE) really is a routing outcome, so that wording is
unchanged for the remaining case.

Also fixes the mixed-result wording '(one ended in X)', which names only
the first of possibly several failing dispositions and so can read as if
only one path failed when the count already says otherwise. Reworded to
'(at least one of them ended in X)'.

Scope limit, documented in the function's own docstring rather than left
implicit: routing._evaluate() itself only ever names ONE representative
disposition even in a unanimous multi-path failure, so a finding where
different paths fail for genuinely different reasons is still
categorized by only the first. That is a limit of what routing.py
reports, not something this fix can see past.

Tests: new cases for both disposition sets, a control pinning that a
genuine routing disposition (NO_ROUTE) still gets the original wording,
and updated assertions for the reworded mixed-result phrase.

Mutation-tested: removing the disposition branching breaks all 3 new
cause-attribution tests for the right reason (fall back to the old
blanket 'routing' wording), while the control still passes; restored.

Full suite: 2230 passed, 46 skipped, 0 failed.

Live-tested against the real model: 10 generations each for a
DENIED_IN, an EXITS_NETWORK, and a NO_ROUTE finding. 0/30 fallbacks, and
every generation correctly followed the new disposition-specific fact --
filter language for DENIED_IN, destination-device language for
EXITS_NETWORK, routing language only for NO_ROUTE.
Direct unit test for the function extracted in 781ba8e, so a future
reword of the detail-prefix format is caught here as well as through
tests/test_explain_safety_net.py's routing fixtures, which already call
the same function.
@patelankeet2 patelankeet2 added bug Something isn't working layer: ai Layer 2 — local LLM explanation and Q&A safety Touches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care labels Nov 8, 2026
@patelankeet2 patelankeet2 self-assigned this Nov 8, 2026

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 3f415ef. The cause attribution is right, and it matches routing.py's own documented meaning of each disposition.

Checked:

  • Merged onto current main (2833a1e): 2314 passed, 13 skipped, ruff clean on CI's selection.
  • Mutants: dropping DENIED_OUT from the filter set fails 1, dropping EXITS_NETWORK fails 1, and the old "(one ended in X)" wording fails 1. With your own branch-removal mutant, each part is pinned.
  • _violation_detail() is a good extraction: the routing tests now call the real producer.

One case this leaves as it was, worth doing here or as a follow-up. For an UNREACHABLE assertion that's violated because the traffic gets through, the fact still blames routing:

Expected UNREACHABLE … but traceroute unexpectedly succeeded: ACCEPTED
→ "…The statement requires this flow to be UNREACHABLE; the network's current routing
   is what disagrees with that requirement, not the requirement itself."

When unwanted traffic is delivered, a filter that should have blocked it is at least as likely a cause as routing, and the prompt labels this sentence "already verified". It's the same over-claim #439 fixes for failures, just in the success direction. _routing_cause_clause() gets ACCEPTED/DELIVERED_TO_SUBNET here and falls through to "routing". A neutral clause for the success dispositions ("the network's current configuration lets it through, whether by its routes or its filters") would close it. Not blocking: it was there before this PR.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 3f415ef, merged onto main cceffec. Approving.

The fix is right and the scope note is honest. Checked:

  • CI-style suite on the merge: 2458 passed, 63 skipped. Ruff clean.
  • The computed fact, called directly with routing._violation_detail() (good to have the real producer there now): DENIED_IN/DENIED_OUT name a filter, EXITS_NETWORK names a device missing from the snapshot, NO_ROUTE still names routing.

Two cases still say "routing" without evidence, for a follow-up rather than this PR:

  • INSUFFICIENT_INFO gets "the network's current routing is what disagrees". Batfish uses that disposition when it does not have enough of the model to decide, which is not a routing fault. The comment above _FILTER_DISPOSITIONS lists it with the routing ones; I'd move it to its own neutral clause ("Batfish could not determine the outcome from the configs it has").
  • An unexpected success gets the same routing clause. Expected UNREACHABLE ... traceroute unexpectedly succeeded: ACCEPTED gives "the network's current routing is what disagrees with that requirement". When a flow that should be blocked gets through, the usual cause is a missing or too-wide filter, not routing, and Batfish does not tell us which. A neutral "the network currently lets this flow through" would claim only what was observed.

Both are the same shape as #439 on the dispositions it did not cover. Happy to file them as one issue if you agree.

@ARSH871-bot

Copy link
Copy Markdown
Owner

Filed the two follow-up cases from my review as #481 (INSUFFICIENT_INFO and an unexpected success), so they don't hold this PR up.

@ARSH871-bot
ARSH871-bot merged commit 8e14cc6 into main Nov 19, 2026
2 checks passed
@ARSH871-bot
ARSH871-bot deleted the fix/439-routing-fact-cause-attribution branch November 19, 2026 21:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working layer: ai Layer 2 — local LLM explanation and Q&A safety Touches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Routing computed fact blames routing for filter denials and missing devices (#399 follow-up)

3 participants