Skip to content

US-62 — Computed facts for every finding type #395

Open

Description

@ARSH871-bot

As a maintainer, I want every finding type's key fact computed in code before the model sees it, so the model never has to reason out a technical fact for itself.

Acceptance criteria

  • Every finding type the explanation layer handles has a computed fact, not only the three handled today (_compute_dead_rule_outcome, _compute_policy_outcome, _compute_expected_actual_outcome)
  • Tested without needing Batfish or Ollama

Shape

Not a pipeline feature: AI layer.

Why this is in the milestone

ai/explain.py has exactly three hand-coded computed-fact functions, so generalising them is real work. It was proposed as "retrieval-grounded" explanations; nothing is retrieved, so it is named here for what it does.


Epic: Specialist pick (explainable and trustworthy AI for network security) | Size: estimated at sprint planning | Assigned: @patelankeet2

Specialist pick agreed by all four on 13 October 2026 (slate row 5). Acceptance criteria are drafted from the slate's wording; the owner refines them at sprint planning on 19 October.

Part of Phase 3 - market readiness.

Activity

  1. added
    enhancementNew feature or request
    layer: aiLayer 2 — local LLM explanation and Q&A
    phase: 3Market readiness -- what stands between us and a product someone could buy.
    on Oct 13, 2026
  2. patelankeet2 commented on Oct 13, 2026

    @patelankeet2
    Collaborator

    Status update on the two remaining shapes from the original inventory, plus a new one that landed mid-task.

    compareFilters -- investigated, no computed-fact function needed

    Before building anything, live-tested change_impact.py's compareFilters shape against the real model, the same discipline PR #399 used for routing -- expecting the same kind of before/after inversion risk. Tested both directions on real findings, 12 generations total, not just one sample:

    "opens" direction (CH-001, rtr-us5-secure -> insecure)   5/6 clearly correct, 1/6 vague but not wrong
    "closes" direction (CH-001, rtr-us5-insecure -> secure)   6/6 clearly correct
    

    Zero inversions across both directions. Looked at why: unlike the four shapes that already have a computed-fact function, compareFilters' summary already states the direction deterministically (_direction() decides "opens"/"blocks" in Python before the model runs at all), and the detail text's After: X. Before: Y. pairing is a single action token each side -- simple enough that the model kept it straight every time. Structurally this is a temporal before/after diff, not two opposing claims about one current state, which is the actual shape the existing four guards protect against. Building a function here would be guarding against a risk the evidence says is not real, the same reasoning #395's own notes already give for excluding the three flat-fact shapes (GUARANTEES, undefined reference, unused structure).

    Asking for a second pair of eyes before calling this settled. @ARSH871-bot, would you re-run this live yourself (or spot-check my method) before we treat compareFilters as closed with no guard? Happy to share the exact repro commands.

    A new, untested shape from #389

    #389 (merged since #399 was opened) added change_impact.py's policy-rule-change findings, with their own before/after text: "held before, breached after" / "breached before, holds after". Structurally similar to compareFilters (a temporal diff, summary already states BREAKS/FIXES deterministically) but genuinely not tested yet -- found it while merging main into this branch, didn't want to silently fold it into a round that was scoped to compareFilters specifically.

    @shubhamkataria2005, since this is your code from #389, would be good to get your read on whether you'd expect the same "summary already anchors the direction" effect here, before I either live-test it the same way or we agree it's out of scope for this round.

    differentialReachability -- still deferred, unchanged reasoning

    Its evidence text is an uncontrolled str() of a pybatfish trace object (confirmed messy and inconsistently shaped when I looked at a real one for #399), not something safe to build a regex against -- a wrong extraction would be worse than no computed fact at all. Unchanged from the original scoping note.

  3. added a commit that references this issue on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestlayer: aiLayer 2 — local LLM explanation and Q&Aphase: 3Market readiness -- what stands between us and a product someone could buy.

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions