Repository navigation
A fourth computed-fact guard, for routing route assertions (#395) - #417
patelankeet2 wants to merge 1 commit into
Conversation
_compute_route_assertion_outcome() follows the exact pattern of the three existing computed-fact functions: routing.py's route assertions use the same two-sided "Expected X ... but Y" phrasing that #52 and #145 already found the model getting backwards on real findings, so the real outcome is computed in Python and handed to the model as an already-verified fact rather than something to re-derive. Wired into _build_prompt()'s or-chain alongside the other three, same ordering discipline (each check produces its own exact evidence.detail shape, so trying the next only when an earlier one returns None cannot mask one with another). Live-tested six times against a real finding (routing-missing-route's RT-001, Ollama + Batfish running) before adding the guard: every run correctly stated the flow was blocked, no inversion reproduced. Recorded as a clean but limited result, not proof of safety, the same honesty the third guard's own docstring already holds itself to. New tests follow test_explain_safety_net.py's own established conventions exactly: a _real_route_detail() helper building evidence text from the real routing._evaluate() producer (not a hardcoded literal, the #145/#169/#194 lesson), mutual-exclusion tests against the other three shapes, and a wiring test confirming the computed fact actually reaches the generated prompt text. 6 new tests, 95/95 passing in the directly affected files.
|
Closing as a duplicate of #399. Both PRs implement the same computed-fact guard in Re-verified #399 independently before closing this one, rather than trusting its own two-week-old numbers: No action needed on this one beyond closing it. Keeping this PR's scope note for the record - it also named two |
What this does
Adds a fourth deterministic computed-fact guard to
ai/explain.py, for routing route assertions.ai/explain.pyalready has three of these (_compute_dead_rule_outcome,_compute_policy_outcome,_compute_expected_actual_outcome), each existing because the local model, measured directly on real findings, has gotten a "which of two opposite states is actually true" attribution backwards (#52, #145, and a third case checked deliberately before it reproduced by accident). Each computes the real answer in Python fromevidence.detailand hands it to the model as an already-verified fact to state, never a question to reason about.This is the first increment against #395 (US-62), whose acceptance criteria ask for computed-fact coverage across more finding types than the original three.
Design decisions (named alternatives, not just the choice made)
change_impact.py's CH- findings and otherpolicy_compliance.pyshapes remain real candidates for a follow-up issue, named here rather than silently dropped.change_impact.pycandidate specifically becauserouting.pyis my own owned module - no cross-team coordination needed, and I know its real output shape in detail from reviewingchange_impact.pyclosely this week for an unrelated PR._compute_route_assertion_outcome()matches the same section-1 style as the three existing functions: a narrow compiled regex matching routing.py's exact two sub-shapes (REACHABLE-but-failed, UNREACHABLE-but-succeeded), returningNonefor anything that doesn't match exactly. Wired into_build_prompt()'sorchain alongside the other three.The actual risk this closes
routing.py's route assertions use the identical two-sided "Expected X ... but Y" phrasing the other three guarded shapes needed a fix for:A model restating this could plausibly invert which side is true - say the flow succeeded when it did not, the same class of mistake #52 and #145 were each found making on a different producer.
Testing
tests/test_explain_safety_net.py: 6 new tests, following the file's own established conventions exactly - a_real_route_detail()helper calling the realrouting._evaluate()producer (not a hardcoded literal, the Generated explanations can invert which side (policy vs config) requires what #145/#145: an AI-layer safeguard, not the evidence-format fix the issue calls for #169/Name the required action in policy evidence, not a pronoun (closes #145) #194 lesson this file's own comments name), mutual-exclusion tests against the other three shapes, and a wiring test confirming the computed fact actually reaches the generated prompt text (not just that the function exists).pytest tests/test_explain_safety_net.py tests/test_routing_classification.py -q- 95 passed.mainbaseline in a disposable worktree: 1885 passed, 13 skipped - exactly the 6 new tests, no regressions._compute_route_assertion_outcome(detail)from_build_prompt()'sorchain. The wiring test failed correctly (the computed fact no longer reached the prompt). Reverted.tests/fixtures/routing-missing-route, RT-001, an Expected-REACHABLE assertion resolving to NO_ROUTE): 6 of 6 runs correctly stated the flow was blocked, no inversion reproduced. Recorded honestly as a clean but limited result on this one real finding, not proof the model can never get this backwards - the same standard the third guard's own docstring already holds itself to, and the same wording now inai/explain.py's module docstring for this fourth shape.What this is not
Not the whole of #395. The issue's acceptance criteria read as covering every finding type the explanation layer handles; this closes one more of them with real verification behind it, and names the remaining candidates (
change_impact.py,policy_compliance.py's other shapes) as follow-up work rather than claiming the issue closed.