You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
US-62 — Computed facts for every finding type #395
As a maintainer, I want every finding type's key fact computed in code before the model sees it, so the model never has to reason out a technical fact for itself.
Acceptance criteria
Every finding type the explanation layer handles has a computed fact, not only the three handled today (_compute_dead_rule_outcome, _compute_policy_outcome, _compute_expected_actual_outcome)
Tested without needing Batfish or Ollama
Shape
Not a pipeline feature: AI layer.
Why this is in the milestone
ai/explain.py has exactly three hand-coded computed-fact functions, so generalising them is real work. It was proposed as "retrieval-grounded" explanations; nothing is retrieved, so it is named here for what it does.
Epic: Specialist pick (explainable and trustworthy AI for network security) | Size: estimated at sprint planning | Assigned:@patelankeet2
Specialist pick agreed by all four on 13 October 2026 (slate row 5). Acceptance criteria are drafted from the slate's wording; the owner refines them at sprint planning on 19 October.
Status update on the two remaining shapes from the original inventory, plus a new one that landed mid-task.
compareFilters -- investigated, no computed-fact function needed
Before building anything, live-tested change_impact.py's compareFilters shape against the real model, the same discipline PR #399 used for routing -- expecting the same kind of before/after inversion risk. Tested both directions on real findings, 12 generations total, not just one sample:
"opens" direction (CH-001, rtr-us5-secure -> insecure) 5/6 clearly correct, 1/6 vague but not wrong
"closes" direction (CH-001, rtr-us5-insecure -> secure) 6/6 clearly correct
Zero inversions across both directions. Looked at why: unlike the four shapes that already have a computed-fact function, compareFilters' summary already states the direction deterministically (_direction() decides "opens"/"blocks" in Python before the model runs at all), and the detail text's After: X. Before: Y. pairing is a single action token each side -- simple enough that the model kept it straight every time. Structurally this is a temporal before/after diff, not two opposing claims about one current state, which is the actual shape the existing four guards protect against. Building a function here would be guarding against a risk the evidence says is not real, the same reasoning #395's own notes already give for excluding the three flat-fact shapes (GUARANTEES, undefined reference, unused structure).
Asking for a second pair of eyes before calling this settled.@ARSH871-bot, would you re-run this live yourself (or spot-check my method) before we treat compareFilters as closed with no guard? Happy to share the exact repro commands.
#389 (merged since #399 was opened) added change_impact.py's policy-rule-change findings, with their own before/after text: "held before, breached after" / "breached before, holds after". Structurally similar to compareFilters (a temporal diff, summary already states BREAKS/FIXES deterministically) but genuinely not tested yet -- found it while merging main into this branch, didn't want to silently fold it into a round that was scoped to compareFilters specifically.
@shubhamkataria2005, since this is your code from #389, would be good to get your read on whether you'd expect the same "summary already anchors the direction" effect here, before I either live-test it the same way or we agree it's out of scope for this round.
differentialReachability -- still deferred, unchanged reasoning
Its evidence text is an uncontrolled str() of a pybatfish trace object (confirmed messy and inconsistently shaped when I looked at a real one for #399), not something safe to build a regex against -- a wrong extraction would be worse than no computed fact at all. Unchanged from the original scoping note.
Activity
patelankeet2 commentedon Oct 13, 2026
Status update on the two remaining shapes from the original inventory, plus a new one that landed mid-task.
compareFilters -- investigated, no computed-fact function needed
Before building anything, live-tested
change_impact.py's compareFilters shape against the real model, the same discipline PR #399 used for routing -- expecting the same kind of before/after inversion risk. Tested both directions on real findings, 12 generations total, not just one sample:Zero inversions across both directions. Looked at why: unlike the four shapes that already have a computed-fact function, compareFilters'
summaryalready states the direction deterministically (_direction()decides "opens"/"blocks" in Python before the model runs at all), and the detail text'sAfter: X. Before: Y.pairing is a single action token each side -- simple enough that the model kept it straight every time. Structurally this is a temporal before/after diff, not two opposing claims about one current state, which is the actual shape the existing four guards protect against. Building a function here would be guarding against a risk the evidence says is not real, the same reasoning #395's own notes already give for excluding the three flat-fact shapes (GUARANTEES, undefined reference, unused structure).Asking for a second pair of eyes before calling this settled. @ARSH871-bot, would you re-run this live yourself (or spot-check my method) before we treat compareFilters as closed with no guard? Happy to share the exact repro commands.
A new, untested shape from #389
#389 (merged since #399 was opened) added
change_impact.py's policy-rule-change findings, with their own before/after text:"held before, breached after"/"breached before, holds after". Structurally similar to compareFilters (a temporal diff, summary already states BREAKS/FIXES deterministically) but genuinely not tested yet -- found it while mergingmaininto this branch, didn't want to silently fold it into a round that was scoped to compareFilters specifically.@shubhamkataria2005, since this is your code from #389, would be good to get your read on whether you'd expect the same "summary already anchors the direction" effect here, before I either live-test it the same way or we agree it's out of scope for this round.
differentialReachability -- still deferred, unchanged reasoning
Its evidence text is an uncontrolled
str()of a pybatfish trace object (confirmed messy and inconsistently shaped when I looked at a real one for #399), not something safe to build a regex against -- a wrong extraction would be worse than no computed fact at all. Unchanged from the original scoping note.Merge pull request #399 from ARSH871-bot/feature/395-computed-facts-e…