Repository navigation
Routing fact: name the real cause, not always 'routing' (#439) - #460
Conversation
) Pure refactor, no behavior change. run() built the finding's detail string inline, so tests/test_explain_safety_net.py's _real_routing_detail_multi() had to retype the same literal prefix by hand to construct a realistic detail for testing ai/explain.py against. A reword of the real prefix could silently stop matching what that hand-copy produces. Pulled the prefix into its own function so the test can call the real producer instead -- same discipline the file already applies to _evaluate() one line above, rather than testing a copy of its output.
…ting (#439) _compute_routing_outcome() told the model, as an already verified fact, that 'the network's current routing is what disagrees' for every failed or unexpectedly-succeeded routing assertion. Measured still present on main (d424e05): analysis/checks/routing.py's own module docstring says DENIED_IN/DENIED_OUT are filter decisions, not routing, and EXITS_NETWORK most often means the destination device is missing from the snapshot (measured there by deleting a config and re-running). Both were blamed on routing anyway. Adds two small disposition sets (filter-caused, missing-device-caused), declared locally in ai/explain.py rather than imported from routing.py -- same boundary ai/query.py's own _SUCCESS_DISPOSITIONS already holds, and for the same documented reason. Everything else in this project's known disposition vocabulary (NO_ROUTE, NULL_ROUTED, LOOP, INSUFFICIENT_INFO, NEIGHBOR_UNREACHABLE) really is a routing outcome, so that wording is unchanged for the remaining case. Also fixes the mixed-result wording '(one ended in X)', which names only the first of possibly several failing dispositions and so can read as if only one path failed when the count already says otherwise. Reworded to '(at least one of them ended in X)'. Scope limit, documented in the function's own docstring rather than left implicit: routing._evaluate() itself only ever names ONE representative disposition even in a unanimous multi-path failure, so a finding where different paths fail for genuinely different reasons is still categorized by only the first. That is a limit of what routing.py reports, not something this fix can see past. Tests: new cases for both disposition sets, a control pinning that a genuine routing disposition (NO_ROUTE) still gets the original wording, and updated assertions for the reworded mixed-result phrase. Mutation-tested: removing the disposition branching breaks all 3 new cause-attribution tests for the right reason (fall back to the old blanket 'routing' wording), while the control still passes; restored. Full suite: 2230 passed, 46 skipped, 0 failed. Live-tested against the real model: 10 generations each for a DENIED_IN, an EXITS_NETWORK, and a NO_ROUTE finding. 0/30 fallbacks, and every generation correctly followed the new disposition-specific fact -- filter language for DENIED_IN, destination-device language for EXITS_NETWORK, routing language only for NO_ROUTE.
Direct unit test for the function extracted in 781ba8e, so a future reword of the detail-prefix format is caught here as well as through tests/test_explain_safety_net.py's routing fixtures, which already call the same function.
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Approving 3f415ef. The cause attribution is right, and it matches routing.py's own documented meaning of each disposition.
Checked:
- Merged onto current
main(2833a1e): 2314 passed, 13 skipped, ruff clean on CI's selection. - Mutants: dropping
DENIED_OUTfrom the filter set fails 1, droppingEXITS_NETWORKfails 1, and the old "(one ended in X)" wording fails 1. With your own branch-removal mutant, each part is pinned. _violation_detail()is a good extraction: the routing tests now call the real producer.
One case this leaves as it was, worth doing here or as a follow-up. For an UNREACHABLE assertion that's violated because the traffic gets through, the fact still blames routing:
Expected UNREACHABLE … but traceroute unexpectedly succeeded: ACCEPTED
→ "…The statement requires this flow to be UNREACHABLE; the network's current routing
is what disagrees with that requirement, not the requirement itself."
When unwanted traffic is delivered, a filter that should have blocked it is at least as likely a cause as routing, and the prompt labels this sentence "already verified". It's the same over-claim #439 fixes for failures, just in the success direction. _routing_cause_clause() gets ACCEPTED/DELIVERED_TO_SUBNET here and falls through to "routing". A neutral clause for the success dispositions ("the network's current configuration lets it through, whether by its routes or its filters") would close it. Not blocking: it was there before this PR.
ARSH871-bot
left a comment
There was a problem hiding this comment.
Reviewed 3f415ef, merged onto main cceffec. Approving.
The fix is right and the scope note is honest. Checked:
- CI-style suite on the merge: 2458 passed, 63 skipped. Ruff clean.
- The computed fact, called directly with
routing._violation_detail()(good to have the real producer there now):DENIED_IN/DENIED_OUTname a filter,EXITS_NETWORKnames a device missing from the snapshot,NO_ROUTEstill names routing.
Two cases still say "routing" without evidence, for a follow-up rather than this PR:
INSUFFICIENT_INFOgets "the network's current routing is what disagrees". Batfish uses that disposition when it does not have enough of the model to decide, which is not a routing fault. The comment above_FILTER_DISPOSITIONSlists it with the routing ones; I'd move it to its own neutral clause ("Batfish could not determine the outcome from the configs it has").- An unexpected success gets the same routing clause.
Expected UNREACHABLE ... traceroute unexpectedly succeeded: ACCEPTEDgives "the network's current routing is what disagrees with that requirement". When a flow that should be blocked gets through, the usual cause is a missing or too-wide filter, not routing, and Batfish does not tell us which. A neutral "the network currently lets this flow through" would claim only what was observed.
Both are the same shape as #439 on the dispositions it did not cover. Happy to file them as one issue if you agree.
|
Filed the two follow-up cases from my review as #481 ( |
Closes #439.
What's wrong:
_compute_routing_outcome()in ai/explain.py told the model, as an already verified fact, that "the network's current routing is what disagrees with that requirement" for EVERY failed or unexpectedly-succeeded routing assertion. Measured still present on main (d424e05):analysis/checks/routing.py's own module docstring says DENIED_IN/DENIED_OUT are filter decisions, not routing, and EXITS_NETWORK most often means the destination device named in the statement is not in this snapshot (measured there by deleting a config and re-running). Both were blamed on "routing" anyway, in a sentence the prompt labels "already verified by code and certain" - the same shape of risk #453 raised about the sibling dead-rule function.What changed:
_FILTER_DISPOSITIONSand_MISSING_DEVICE_DISPOSITIONS, declared locally in ai/explain.py rather than imported from routing.py - matching the exact boundaryai/query.py's own_SUCCESS_DISPOSITIONSalready holds (re-declared there "so ai/ does not depend on a specific check module for a general Batfish-interpretation fact"). Everything else in this project's known disposition vocabulary (NO_ROUTE, NULL_ROUTED, LOOP, INSUFFICIENT_INFO, NEIGHBOR_UNREACHABLE) really is a routing outcome, so that wording is unchanged for the remaining case - not a guessed default, since these are the only two non-routing causes this project has observed or documented.routing._evaluate()'sfailures[0]), so it could read as if only one path failed when the count already said otherwise. Reworded to "(at least one of them ended in X)"."Expected ... for a flow from ..."detail-prefix out of routing.py's one call site into_violation_detail(), so the test file's routing fixtures call the real producer instead of retyping it by hand - closing the producer-drift risk the issue named, without the larger lift of building a full fake Session to driverun()end-to-end.Explicit scope limit, documented in
_compute_routing_outcome()'s own docstring rather than left implicit:routing._evaluate()itself only ever names ONE representative disposition even in a unanimous multi-path failure, so a finding where different paths fail for genuinely different reasons (one DENIED_IN, one NO_ROUTE) is still categorized by only the first. That is a limit of what routing.py reports, not something this fix can see past.No duplication: confirmed via the GitHub search API and the open-PR list that no PR exists for #439, and it does not overlap my two other currently open PRs (#458 touches
_compute_dead_rule_outcome(), #459 touches_ALL_CLEAR_PATTERN; neither touches this function or routing.py).Testing:
_violation_detail()against the real_evaluate()output.pytest tests/test_explain_safety_net.py tests/test_routing_classification.py -v- 105 passed.pytest tests/ -q- 2230 passed, 46 skipped, 0 failed.