Skip to content

A fourth computed-fact guard, for routing route assertions (#395) - #417

Closed
patelankeet2 wants to merge 1 commit into
mainfrom
feature/395-route-assertion-computed-fact
Closed

patelankeet2 wants to merge 1 commit into
mainfrom
feature/395-route-assertion-computed-fact

Conversation

@patelankeet2

Copy link
Copy Markdown
Collaborator

What this does

Adds a fourth deterministic computed-fact guard to ai/explain.py, for routing route assertions. ai/explain.py already has three of these (_compute_dead_rule_outcome, _compute_policy_outcome, _compute_expected_actual_outcome), each existing because the local model, measured directly on real findings, has gotten a "which of two opposite states is actually true" attribution backwards (#52, #145, and a third case checked deliberately before it reproduced by accident). Each computes the real answer in Python from evidence.detail and hands it to the model as an already-verified fact to state, never a question to reason about.

This is the first increment against #395 (US-62), whose acceptance criteria ask for computed-fact coverage across more finding types than the original three.

Design decisions (named alternatives, not just the choice made)

  1. Scope: one new finding shape, not "every finding type" in one PR. The three existing guards were each added one at a time, each only after being confident of the exact shape (and for the third, a live test before shipping). Matching that discipline rather than a wide sweep across several checks with thinner verification on each. change_impact.py's CH- findings and other policy_compliance.py shapes remain real candidates for a follow-up issue, named here rather than silently dropped.
  2. Which shape: routing's route assertions, not a teammate's module. Chosen over the change_impact.py candidate specifically because routing.py is my own owned module - no cross-team coordination needed, and I know its real output shape in detail from reviewing change_impact.py closely this week for an unrelated PR.
  3. Follow the exact existing pattern. _compute_route_assertion_outcome() matches the same section-1 style as the three existing functions: a narrow compiled regex matching routing.py's exact two sub-shapes (REACHABLE-but-failed, UNREACHABLE-but-succeeded), returning None for anything that doesn't match exactly. Wired into _build_prompt()'s or chain alongside the other three.
  4. Verify before shipping, the same way guard three was. Live-tested against the real model on a real routing finding before trusting it, not just unit-tested.

The actual risk this closes

routing.py's route assertions use the identical two-sided "Expected X ... but Y" phrasing the other three guarded shapes needed a fix for:

Expected REACHABLE for a flow from 10.10.10.5 to 10.20.20.5, but traceroute ended in NO_ROUTE. Path: rtr-hq

A model restating this could plausibly invert which side is true - say the flow succeeded when it did not, the same class of mistake #52 and #145 were each found making on a different producer.

Testing

  • tests/test_explain_safety_net.py: 6 new tests, following the file's own established conventions exactly - a _real_route_detail() helper calling the real routing._evaluate() producer (not a hardcoded literal, the Generated explanations can invert which side (policy vs config) requires what #145/#145: an AI-layer safeguard, not the evidence-format fix the issue calls for #169/Name the required action in policy evidence, not a pronoun (closes #145) #194 lesson this file's own comments name), mutual-exclusion tests against the other three shapes, and a wiring test confirming the computed fact actually reaches the generated prompt text (not just that the function exists).
  • Directly affected files: pytest tests/test_explain_safety_net.py tests/test_routing_classification.py -q - 95 passed.
  • Full suite: 1891 passed, 13 skipped. Checked the exact current main baseline in a disposable worktree: 1885 passed, 13 skipped - exactly the 6 new tests, no regressions.
  • Mutation-tested the wiring itself: temporarily removed _compute_route_assertion_outcome(detail) from _build_prompt()'s or chain. The wiring test failed correctly (the computed fact no longer reached the prompt). Reverted.
  • Live-tested against the real model (Ollama + Batfish running) on a real finding (tests/fixtures/routing-missing-route, RT-001, an Expected-REACHABLE assertion resolving to NO_ROUTE): 6 of 6 runs correctly stated the flow was blocked, no inversion reproduced. Recorded honestly as a clean but limited result on this one real finding, not proof the model can never get this backwards - the same standard the third guard's own docstring already holds itself to, and the same wording now in ai/explain.py's module docstring for this fourth shape.
  • Byte-level check for em-dashes and AI-attribution language across the full diff: zero of either.

What this is not

Not the whole of #395. The issue's acceptance criteria read as covering every finding type the explanation layer handles; this closes one more of them with real verification behind it, and names the remaining candidates (change_impact.py, policy_compliance.py's other shapes) as follow-up work rather than claiming the issue closed.

_compute_route_assertion_outcome() follows the exact pattern of the
three existing computed-fact functions: routing.py's route assertions
use the same two-sided "Expected X ... but Y" phrasing that #52 and
#145 already found the model getting backwards on real findings, so the
real outcome is computed in Python and handed to the model as an
already-verified fact rather than something to re-derive.

Wired into _build_prompt()'s or-chain alongside the other three, same
ordering discipline (each check produces its own exact evidence.detail
shape, so trying the next only when an earlier one returns None cannot
mask one with another).

Live-tested six times against a real finding (routing-missing-route's
RT-001, Ollama + Batfish running) before adding the guard: every run
correctly stated the flow was blocked, no inversion reproduced. Recorded
as a clean but limited result, not proof of safety, the same honesty the
third guard's own docstring already holds itself to.

New tests follow test_explain_safety_net.py's own established
conventions exactly: a _real_route_detail() helper building evidence
text from the real routing._evaluate() producer (not a hardcoded
literal, the #145/#169/#194 lesson), mutual-exclusion tests against the
other three shapes, and a wiring test confirming the computed fact
actually reaches the generated prompt text.

6 new tests, 95/95 passing in the directly affected files.
@patelankeet2 patelankeet2 added enhancement New feature or request layer: ai Layer 2 — local LLM explanation and Q&A phase: 3 Market readiness -- what stands between us and a product someone could buy. labels Oct 27, 2026
@patelankeet2 patelankeet2 self-assigned this Oct 27, 2026
@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Closing as a duplicate of #399.

Both PRs implement the same computed-fact guard in ai/explain.py for routing.py's assertion findings - same evidence shape, same section of the file. #399 is the earlier of the two (opened 13 October) and is strictly the more complete guard: while live-testing for the state-inversion bug the other three existing guards catch, it found a different, more serious one instead - 3 of 6 real runs swapped the flow's source and destination ("HQ cannot reach branch" came back as "HQ cannot be reached FROM branch", describing the reverse flow). It fixes that by stating direction explicitly, on top of the disposition fact. This PR's _compute_route_assertion_outcome() only covers the disposition fact, not direction, so it does not guard the bug #399 already found and fixed.

Re-verified #399 independently before closing this one, rather than trusting its own two-week-old numbers:

git merge origin/main into feature/395-computed-facts-every-finding-type (local only)
  clean merge, no conflicts, nothing overlapping ai/explain.py or its test file

pytest tests/test_explain_safety_net.py tests/test_routing_classification.py -q
  97 passed

pytest tests/ -q
  1907 passed, 13 skipped

mutation test (independent, not just trusting the PR's own claim): removed
_compute_routing_outcome() from _build_prompt()'s chain -- the dedicated
wiring test failed for the right reason. Reverted.

No action needed on this one beyond closing it. Keeping this PR's scope note for the record - it also named two change_impact.py shapes (compareFilters, differentialReachability) as real follow-up candidates against #395, same as #399's own body does.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request layer: ai Layer 2 — local LLM explanation and Q&A phase: 3 Market readiness -- what stands between us and a product someone could buy.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant