Skip to content

Name the dead line's own stated action explicitly, not just the outcome (#384) - #401

Open
patelankeet2 wants to merge 10 commits into
mainfrom
feature/384-dead-rule-own-action
Open

patelankeet2 wants to merge 10 commits into
mainfrom
feature/384-dead-rule-own-action

Conversation

@patelankeet2

@patelankeet2 patelankeet2 commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #384.

The bug

_compute_dead_rule_outcome()'s regex already captured the dead line's own action ((?P<dead_action>PERMIT|DENY)), but the function never used it. The computed fact stated only the real OUTCOME (what happens to matching traffic), never what the dead line itself says its own action is -- on the reasoning that the action "never takes effect, regardless of what it says", so stating it seemed unnecessary.

Found live during #346's held-out set work: the model sometimes mischaracterised the dead line's own action anyway, even while correctly stating the outcome -- "which is supposed to block traffic" for a line whose real action is PERMIT. Correct about the outcome, backwards about a different, separate fact.

Re-measured before fixing

Same real finding (AC-002, rtr-us5-messy), 8 fresh live generations against the code as it stood before this PR:

run 2: "...which is supposed to BLOCK traffic from a specific host..."              inverted
run 5: "...supposed to BLOCK traffic..." / "...ACL rule allows certain traffic..."   internally inconsistent
run 8: "...supposed to BLOCK traffic..." / "...allowing traffic from a certain host" internally inconsistent
runs 1,3,4,6,7: correct

3 of 8 -- worse than the 1-in-8 originally reported in #384, and good evidence this is a real, recurring failure mode worth fixing now rather than a one-off.

The fix (original version)

State both facts explicitly, the same discipline the other two computed-fact functions already use for their own two-sided facts: the dead line's own stated action, and the real outcome, named as two separate things.

Review found a second, more serious problem -- fixed on top (@ARSH871-bot)

The original single-template wording assumed the two facts always disagree. That is true for _compute_policy_outcome() and _compute_expected_actual_outcome() by construction -- their findings only exist when the two sides differ. A dead-ACL-rule finding carries no such guarantee: it exists whenever a line is unreachable, whether the shadowing line agrees with the dead line's own action (redundant) or disagrees (the real exposure this check exists to catch).

Measured live, 60 generations each: a redundant dead DENY (shadowed by an earlier DENY) went from main's ~6/60 wrong to ~23/60 wrong under the single-template wording -- the exact #52 inversion this function exists to prevent, reintroduced by assuming "different" when the real relationship was "the same, coincidentally". Separately, on AC-002 itself (a genuine opposite-action case), the single template's own repetition measurably made the model more likely to give up and copy the Modelfile's Example 1 almost verbatim: 9/60, against 0/60 before.

Fixed by branching on whether the dead line's own action matches the real outcome. The differing-action branch is restated once, plainly, instead of three times with "not what happens... what actually happens is different... regardless of what it says". The agreeing-action (redundant) branch states agreement explicitly and never uses the word "different" about the outcome.

Two new tests exercise the redundant case through the real producer (access_control._check_dead_rules), not a literal, per the review.

The docstring also now explains why this function needs the branch the other two don't -- their two sides are guaranteed to differ by construction of the finding; this one's two facts are independent.

Re-measured after the full fix

16 live generations (8 redundant-deny, 8 AC-002-shaped): 0 of 16 claimed the outcome was "different" when it agreed, 0 of 16 mischaracterised the dead line's own action, 0 of 16 copied Modelfile Example 1.

Merged main in and re-captured AC-002 (#385's guard)

#385 landed on main after this branch started, with a per-finding prompt hash that correctly flagged AC-002 as stale the moment this fix's wording changed the prompt it was captured against. Re-captured one live generation under the new prompt, rated by hand: accuracy 5, usefulness 4 (was 2/2). docs/evaluation.md Part 4 updated with a dated note recording the re-capture rather than rewriting the row silently; the two tests pinning the published numbers updated to match (mean accuracy 4.25 -> 5.0, mean usefulness 3.25 -> 3.75).

Tests

Five new tests total: the three original (own-action named explicitly, both PERMIT and DENY, and that the two facts can differ) plus two for the redundant case, through the real producer.

Mutation-tested each change against its own tests. Reverting the branch to the single-template wording: the two redundant-case tests fail for the right reason (the old wording's "what actually happens is different" is false for that case).

Verified

python -m pytest tests/test_explain_safety_net.py tests/test_explanation_quality.py -q -> 110 passed
python -m pytest tests/ -q -> 2245 passed, 13 skipped, 0 failed
ruff check --select E4,E7,E9,F -> clean on every changed file

No em-dashes in any changed file.

What this does not do

Does not touch _compute_policy_outcome() or _compute_expected_actual_outcome() -- both already state both sides of their own two-state fact and don't share this function's "two facts can independently agree" shape. Checked _compute_routing_outcome() (added by #399) for the same shape of bug: it does not have it, since it states a flow's direction and disposition rather than comparing two facts that could agree or disagree. Does not touch the compareFilters investigation currently on #395 -- unrelated shape, unrelated files beyond ai/explain.py and its own tests.

…me (#384)

_compute_dead_rule_outcome() already captured the dead line's own
action via its regex ('(?P<dead_action>PERMIT|DENY)') but never used
it -- the computed fact stated only the real outcome, never what the
dead line itself says, on the reasoning that the action 'never takes
effect, regardless of what it says' made stating it unnecessary.

Found live, not assumed (#384): the model sometimes mischaracterised
the dead line's own action anyway, even while correctly stating the
outcome -- 'which is supposed to block traffic' for a line whose real
action is PERMIT. Correct about the outcome, backwards about a
different fact.

Re-measured on the same real finding (AC-002, rtr-us5-messy)
immediately before fixing: 3 of 8 live generations mischaracterised
or were internally inconsistent about the dead line's own action,
worse than the 1-in-8 originally reported in #384. Re-measured again
with the fix in place: 0 of 8.

Both facts are now stated explicitly, the same discipline the other
three computed-fact functions already use for their own two-sided
facts -- the dead line's own stated action, and the real outcome,
named as two separate things rather than leaving the model to read
one of them off the raw evidence and risk inverting it.

python -m pytest tests/test_explain_safety_net.py -q -> 81 passed (was 78).
@patelankeet2 patelankeet2 added bug Something isn't working layer: ai Layer 2 — local LLM explanation and Q&A safety Touches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care labels Oct 13, 2026
@patelankeet2 patelankeet2 self-assigned this Oct 13, 2026

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving c7cf47e. I re-ran the live measurement myself rather than taking the description's numbers.

Live, 8 generations each, real AC-002 from rtr-us5-messy, netwise-warden:

main (b69d591)   1 of 8 wrong about the dead line's own action
                 run 2: "The device's ACL rule, which is supposed to block traffic ...
                         resulting in no traffic being allowed despite the unreachable line's permit action"
this PR          0 of 8. Every run names it as a permit/allow rule, e.g.
                 "The unreachable line itself states that it should permit this traffic,
                  but its action is overridden by the blocking rule"

I got 1 of 8 on main where you got 3 of 8, so the rate varies between runs, but the failure is real and reproduces on the same finding. With the fix it didn't happen once.

Other checks:

  • Merged onto current main (b69d591, which has #386 and #398): 1902 passed, 13 skipped, ruff clean on ai/ and tests/.
  • Mutation-tested _compute_dead_rule_outcome: action hard-coded to PERMIT, action taken from the outcome instead of the regex group, and the new sentence dropped. 3 of 3 killed.
  • ai/query.py only mentions this function in a comment, so the reachability answers don't change.
  • Merges cleanly with #399, which also edits ai/explain.py and tests/test_explain_safety_net.py.

Merge order with #385: #385's held-out set rates this exact AC-002 capture 2/2, so once this lands that rating describes a prompt the product no longer sends. Detail is on #385.

No changes requested. One nit, no action needed: the description's 1888 passed is from before #386 and #398 landed.

patelankeet2 added a commit that referenced this pull request Nov 1, 2026
The Modelfile hash cannot see a change to _build_prompt() itself.
Measured: with #401 merged on top of this held-out set, every test still
passed, because #401 rewrites the computed fact for AC-002 (this set's
own #384 capture, rated 2/2) without touching ai/Modelfile at all. The
published score would stay green while describing a prompt the product
no longer sends.

Added current_prompt_sha256(finding), recorded one hash per model-written
entry in holdout.json, and a test that rebuilds each prompt from the
stored finding dict and compares. One hash per finding rather than one
for the whole set, so a change to one computed-fact function marks only
the findings whose own prompt actually moved.

Also closes the nit that current_modelfile_sha256() was never shown to
hash file content rather than return a constant, and adds the
__package__ guard the new ai.explain import requires for the script
invocation form (tests/test_tools_invocation.py caught this immediately).

Addresses the change requested on PR #385 (review by Shubham).
@shubhamkataria2005

Copy link
Copy Markdown
Collaborator

Re-checked on current main (d4535ef, which now has #298): merges clean, 2005 passed, 13 skipped. My approval of c7cf47e stands.

Merge order with #385. #385's new per-finding prompt hash fails the moment this lands, and names AC-002. Measured: #385 + this PR gives captured against a prompt _build_prompt() no longer builds: ['AC-002']. That's correct, because this PR changes exactly that prompt. So whichever lands second needs AC-002 re-captured under the new prompt and re-rated by hand. I'd merge #385 first, then add the re-capture and new rating here, so the PR that changed the prompt also carries the new rating. Details on #385.

@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Approved on the current head and everything's verified. Please merge when you have a moment.

@shubhamkataria2005

Copy link
Copy Markdown
Collaborator

@patelankeet2, not merging this one yet, on purpose. #385 is in (5db6498), and its prompt-hash guard does what it was built for. On current main + this PR:

1 failed, 15 passed   tests/test_explanation_quality.py
these held-out entries were captured against a prompt _build_prompt() no longer builds: ['AC-002']

So merging it as it stands would turn main red. To land it:

  1. merge main into this branch;
  2. re-capture AC-002's explanation under this PR's prompt, store it with its new captured_prompt_sha256, and re-rate it by hand, which is the step a test can't do for you;
  3. update the published numbers in docs/evaluation.md if the mean moves.

It was 2/2 because of the exact inversion this PR fixes, so I'd expect it to go up. It's the first real use of #385's guard, and worth recording as such. Push it and I'll re-check and merge the same day. My approval of the code stands.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update, 3 November. Still on c7cf47e. I'm posting this before AC-002 is re-captured and re-rated for #385's prompt-hash guard (Shubham's note of 2 November), because a re-rating on this wording would pin the redundant-rule case in as correct.

Requesting changes. Stating the dead line's own action is the right idea, and on AC-002 it does cut "the line is supposed to block" from 13/60 to 6/60 on my machine. But the sentence it adds is false for a whole class of dead rules, and on AC-002 itself it brings in a worse failure than the one it fixes.

1. The fact says "different" for every dead rule, including redundant ones. ai/explain.py:213-220 always says "that is what it says, not what happens. What actually happens is different", whatever the blocker's action is. A deny tcp … eq 22 under an earlier deny ip … any comes back from filterLineReachability with Reason: BLOCKING_LINES like any other dead line. Through the real producer (_check_dead_rules with a fake frame, as in tests/test_found_status_is_reported.py:162) you get:

Unreachable line: deny   tcp 10.10.10.0 0.0.0.255 host 10.20.0.5 eq 22 (action DENY). Blocked by: deny   ip 10.10.10.0 0.0.0.255 any. Reason: BLOCKING_LINES
-> "...own stated action is DENY -- that is what it says, not what happens. What actually happens is different: ... actually denied"

That reaches the model labelled "already verified by code and certain … not something to … contradict" (:684), and the model believes it. 60 live runs each:

main   ~6/60 wrong
#401  ~23/60 wrong   "no traffic is actually denied", "it allows traffic to pass through",
                     "the actual action ... is different from what its own stated action claims,
                      and this discrepancy has been verified by code"

That is #52 brought back by the guard meant to stop it. The two functions you modelled this on can say both sides differ because their findings only exist when they do. A dead line's own action and its outcome are independent.

2. On AC-002, the new wording makes the model copy Modelfile Example 1 word for word. 0/60 on main against 9/60 on this branch, and 3/30 through explain_with_source(), all with source="model": "The internal server can be reached over unencrypted web traffic that should have been blocked. The device's rule set currently allows all traffic through with no restriction…", for a dead DNS permit. _is_unacceptable passes it. Counting those, AC-002 goes from 13/60 wrong to 15/60. Your 0 of 8 fits these rates (about 27% likely by chance), so this isn't a missed step on your side. 8 runs is too few to catch a 1-in-7 failure.

What fixed both for me: branch on whether the two actions match:

own = "permitted" if dead_action == "PERMIT" else "denied"
if own == outcome:
    return (f"the unreachable line's own stated action is {dead_action}, and traffic matching it is "
            f"{outcome} anyway, by an earlier line that also {'permits' if outcome == 'permitted' else 'denies'} "
            f"it and is evaluated first. The outcome is the same as the unreachable line says; only the line "
            f"that decides is different.")
return (f"the unreachable line's own stated action is {dead_action}, but it never takes effect: traffic "
        f"matching it is actually {outcome}, because an earlier line that matches the same traffic is "
        f"evaluated first and decides instead.")

30 runs per case: AC-002 had no Example 1 copies and own action right except one muddled run; the redundant DENY was 1/30 wrong; 0/120 Example 1 copies across the four cases. It passes your three tests and the full suite (1852). I'm not saying this wording is the best one. It shows the shape of the sentence is what matters.

3. Tests. tests/test_explain_safety_net.py:216-260 only cover opposite-action pairs, so nothing pins the same-action case either way. They also use literal details, where the #145 tests at :880-906 drive pc._describe(). Please add a test through _check_dead_rules for deny-under-deny and permit-under-permit, asserting the fact doesn't say "not what happens". Mine fails on c7cf47e and passes on the version above.

Nits. The PR body says "the other three computed-fact functions". There are two on main. The docstring at :175 should say why this fact differs from the other two.

Checked and fine: CI green; 81 vs 78 tests in the safety-net file; full suite 1852 passed / 49 skipped with Batfish down (same 1901 total as your 1888 + 13); your mutation reproduces, plus five more of mine that the tests catch; merges cleanly with #399 and #385.

A dead ACL line's own stated action and the real outcome are independent
facts, unlike the other two computed-fact functions, whose two sides are
guaranteed to differ by construction of when their finding even exists.
The single template that always said "what actually happens is different"
was false whenever the dead line was redundant rather than contradicted (a
DENY shadowed by an earlier DENY, say) -- reintroducing the exact #52
inversion this function exists to prevent, measured live going from ~6/60
wrong to ~23/60 wrong on that case.

The old template's own repetition also measurably made the model more
likely to give up and copy the Modelfile's Example 1 verbatim on AC-002
(a genuine opposite-action case): 9/60 against 0/60 before. The simpler,
single-statement wording for the differing-action branch removes that
without changing the fact it states.

Two new tests exercise the redundant case through the real producer
(access_control._check_dead_rules), not a literal, asserting the output
never claims the outcome is different when it agrees.

Found and specified by review before this PR merged.
#385's per-finding prompt hash correctly flagged AC-002 as stale the
moment main (with #385 and #399) merged onto this branch -- the prompt
_build_prompt() builds for it moved when the previous commit fixed #384.

Re-captured one live generation under the new prompt and rated it by
hand (Ankeet): accuracy 5, usefulness 4, up from 2/2. The old rating's
defect is gone -- the dead line's own PERMIT action is now named
correctly, and the real blocked outcome is stated without inverting
which rule is which.

Updated docs/evaluation.md Part 4 with a dated note recording the
re-capture rather than silently rewriting the AC-002 row, and updated
the two tests that pin the published numbers (mean accuracy 4.25 -> 5.0,
mean usefulness 3.25 -> 3.75; the breakdown's low-scorer demonstration
now points at PC-005's usefulness, the new minimum, since AC-002 no
longer has an outlier to show).

Full suite: 2245 passed, 13 skipped, 0 failed.
@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Fixed all three points, plus the nit. Full detail in the updated PR description.

  1. The redundant-dead-rule mis-rating: branched the computed fact on whether the dead line's own action matches the real outcome, instead of always assuming they differ. Two new tests through access_control._check_dead_rules, per your request.
  2. The AC-002 Example-1-copying regression: the differing-action branch is now a single plain statement instead of the old three-times-repeated phrasing. Live-checked 16 generations (8 redundant, 8 AC-002-shaped): 0/16 wrong either way.
  3. Added the test you asked for, in both directions (deny-under-deny and permit-under-permit).

Nit fixed: "other two" computed-fact functions, not three.

Also merged main in and handled the #385 merge-order item Shubham flagged: AC-002 was correctly caught as stale by #385's prompt-hash guard the moment the wording changed, re-captured and re-rated by hand (2/2 -> 5/4), and docs/evaluation.md Part 4 updated with a dated note rather than a silent edit.

Full suite: 2245 passed, 13 skipped, 0 failed. Ready for another look whenever you have a moment.

@shubhamkataria2005 shubhamkataria2005 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 246db8e. My approval of c7cf47e didn't count once these commits landed, and Arsh's review found what my 8-run check missed, so I re-ran this at his scale rather than mine.

Live, 30 generations per case, netwise-warden, main (db59b00) against this branch:

                                              main      this PR
AC-002: PERMIT line called a block            10/30     0/30
redundant DENY-under-DENY: says traffic gets
  through, or that the actions differ         (fine)    0/30
copies of Modelfile Example 1, both cases     2/60      0/60

So both of Arsh's findings are fixed: the redundant case states agreement, never difference, and the Example 1 copying is gone. I read every one of the 60 PR-branch texts myself rather than trusting a regex. Mine over-flagged correct texts on main.

Other checks:

  • Merged onto current main: 2245 passed, 13 skipped, ruff clean on CI's selection.
  • #385's prompt-hash guard is satisfied by the re-capture, which is the guard's first real use.
  • Mutants: dropping the same-action branch fails 2, inverting the comparison fails 4, taking the own action from the outcome fails 2. The two new tests go through the real _check_dead_rules, as Arsh asked.

Not blocking, both about the re-rating:

  1. I'd rate the new AC-002 capture 4, not 5, for accuracy. It says the earlier rule "blocks all traffic with the same destination IP address", but the blocker is deny ip 10.10.10.0 0.0.0.255 any, all LAN traffic to any destination. The live runs do it too: 5/30 on this branch describe the blocker as destination- or port-specific, against 2/30 on main. Small numbers, but the direction is worth knowing.
  2. The capture was rated once, by the person who wrote the fix. That's no criticism of you; it's how the set works today. But the published accuracy mean is now 5.00 with a minimum of 5, and a second rater would make that number much stronger. Arsh offered back in October, and I'm happy to rate too. Added as a second rating, my 4 would make the accuracy mean 4.8 and the minimum 4 (the tool pools every rating).

@ARSH871-bot, your 3 November review is on c7cf47e. The redundant case and the Example 1 copying were your findings, so I'd rather you confirm them before this merges than merge on my numbers alone.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 246db8e on main db59b00. The fix is right, and it does more good than the description says. I'm asking for three small changes, mostly to the tests and the published score, and I'll re-review the same day.

What I confirmed. 30 live generations per case through explain_with_source(), on main and on this branch. Counted by pattern, then I read every text flagged on this branch, and all of main's for the two redundant cases, because a pattern alone over-flags.

                                                       main     #401
AC-002: calls the PERMIT line a block                  4/30     0/29
Redundant PERMIT under PERMIT: calls it a block       25/30     1/30
Redundant DENY under DENY: says the traffic gets through  2/29     0/30
Redundant DENY under DENY: says the outcome differs     0/29     1/30
DENY under PERMIT: says the traffic gets through      29/30    30/30

The second row is the case nobody has measured, and it is the strongest argument for this PR. On main, almost every text says a permit line "is supposed to block traffic", and that another rule "blocks the same traffic". It's worth adding to the description.

Both of my 3 November findings are fixed: the redundant case no longer says "different" about the outcome, and the 9/60 Example 1 copying from the earlier wording is gone (one caveat below).

I also tried to make the new "the outcome is the same" sentence lie, because it's a stronger claim than main makes. Eight invented ACLs on live Batfish, with small lines of either action hidden above the dead one. It held every time. In six, Batfish named the overlapping opposite-action line, so the function returned None. In the other two no such line overlapped, and testFilters agreed with the sentence.

Also checked:

  • CI-style suite: 2204 passed, 54 skipped (main 2199, so +5, your five new tests).
  • Live suite: 2245 passed, 13 skipped, as you said.
  • python -m tools.explanation_quality gives your 5.00 / 3.75.
  • main moved while I was checking: #388 and #437 landed (d424e05). On that, with this PR merged: 2223 passed, 54 skipped (main 2218, so still +5).
  • Merges cleanly with #405, #406, #428, #434, #442, #443, #445, #446 and #449.
  • Your original mutants, and Shubham's, are all caught, with Batfish and Ollama running and with them off.

1. The new tests don't pin the sentence they are named for. test_a_redundant_dead_deny_states_agreement_not_difference only rejects the two old phrases, so these all pass, in both environments:

"The outcome differs from what the unreachable line says; the line that decides is earlier."
(the agreement sentence removed entirely)
"permitted anyway, by an earlier line that also denies it"   (verb inverted)
"by an earlier line that also denies it", always              (verb fixed to "denies")

Four lines close it. I've run them: the real code passes (94 passed in that file) and all four mutants fail.

# in the deny test
assert "also denies it" in outcome
assert "the same as the unreachable line says" in outcome
# in the permit test
assert "also permits it" in outcome
assert "the same as the unreachable line says" in outcome

2. AC-002 isn't a 5 for accuracy, and the score pins it. The capture says an earlier rule "blocks all traffic with the same destination IP address". The blocker is deny ip 10.10.10.0 0.0.0.255 any: one source subnet, to any destination. Part 4's scale asks whether the text claims only what the evidence supports, so I agree with Shubham's 4. It matters because this rating produces "mean 5.00, min 5", and test_the_published_score_matches_the_committed_ratings now holds that number in place.

It isn't a one-off. I measured the same misdescription on 6 of 29 runs here, against 0 of 30 on main (Shubham had 5/30 against 2/30). "An earlier line that matches the same traffic" seems to invite it.

Naming the blocking line fixes that, and it means AC-002 gets a fresh capture anyway:

named = " ".join(match.group("blocking").split())   # Batfish pads its spaces
lines, verb = ("earlier lines", "are") if len(blocking_lines) > 1 else ("an earlier line", "is")
...
f"because {lines} ({named}) {verb} evaluated first and decides instead."

30 live runs on AC-002 with one blocking line, patched in memory: 0/30 called the permit a block, 0/30 copied Example 1, 1/30 called the blocker destination-specific. I haven't measured the two-line wording live. In those three test files the only test that objects is the prompt-hash guard, which is its job. Not the only way to fix it, so use different wording if you prefer.

Either way, please post the new capture here. Shubham and I will each rate it and you can add the ratings, so the published number stops resting on one rating by the person who wrote the fix. If you keep the current wording, I'd add my 4 to this capture.

3. Two stale references. ai/explain.py:248 says the redundant case is "#444's exact shape", but #444 is R-1's word match. The docstring has it right as #52. And tools/explanation_quality.py:61-62 still calls 4.25/3.25 "the current measurement".

Separate from this PR. None of these is caused by it, and all are on main too. I'll file them as issues.

  • "Actually denied" can be false. With permit host .5, deny /24, permit /24 eq 443, Batfish names only the deny, so the fact says the traffic is denied, and testFilters says 10.1.1.5 is permitted.
  • Arista and Juniper dead rules get no fact, because their lines don't start with permit or deny. That's 2 of the 5 dead-rule findings across our fixtures.
  • The Example 1 copying isn't gone, only back to main's rate. I got 2/60 here and 0/60 on main across AC-002 and the redundant DENY. Shubham got the reverse, 0/60 and 2/60. So about 2 in 120 each, and _is_unacceptable passes it. A dead DNS rule explained as "the internal server can be reached over unencrypted web traffic" still reaches the screen labelled as AI.
  • A hand edit to captured_explanation passes #385's guard, which hashes the prompt and not the text. Noted for whenever that set is next touched.

A pushed fix notifies nobody, so please re-request both of us. The new commit voids Shubham's approval too.

Arsh's 5 November review: the existing asserts only rejected the two old
false phrases, so a rewrite that drops the agreement sentence entirely, or
inverts its verb, still passed. Added the two asserts he gave, mutation
tested against both of his exact adversarial wordings (sentence removed,
verb inverted) -- both now fail for the right reason.

Also fixed a stale reference found in the same review: the redundant-case
comment said this was #444's shape; #444 is R-1's word match, this is the
#52 inversion. The docstring already had it right, only this comment did
not.
…claim

Arsh's 5 November review: the wording said an earlier line blocks 'the
same traffic', which the real AC-002 finding's blocking line (deny ip
10.10.10.0 0.0.0.255 any) does not actually support -- it is one source
subnet to ANY destination, not a destination match, and 6 of 29 live
generations under the old wording said so anyway ('same destination IP
address'). Now names the real blocking line text, singular and plural
grammar both pinned by new tests.

Found and fixed a real subject-verb mismatch in the plural branch while
writing it (Arsh's own suggested wording for that branch was not live
tested by either of us) -- 'lines ... are ... decides' corrected to
'... decide'.

Re-captured AC-002 in the held-out set under the new wording and rated it
myself (30 live generations: 1/30 still said same destination, 0/30
called the permit line a block, 1/30 copied Example 1 -- normal model
variance, not hidden by the new wording). docs/evaluation.md and the
holdout fixture both updated, with an explicit request for Shubham and
Arsh to rate the new capture since their earlier 4s rated the old one.

Also fixed the second stale reference from the same review:
tools/explanation_quality.py called the 4.25/3.25 threshold basis 'the
current measurement' when a later re-run already reads higher -- reworded
to say when it was taken and to point at docs/evaluation.md for the real
current number instead of repeating one here.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed 148379e (merged onto main d424e05, full suite 2265 passed / 13 skipped / 0 failed). All three points addressed.

  1. The two tests now pin the actual agreement sentence, with your exact asserts. Mutation tested against both your adversarial wordings (sentence removed entirely, verb inverted) -- both fail for the right reason now.

  2. AC-002's wording now names the real blocking line (deny ip 10.10.10.0 0.0.0.255 any) instead of the vague "same traffic" claim. While writing the plural branch (the one you flagged you hadn't live tested) I found a real subject-verb mismatch in my own first draft -- fixed before it shipped, pinned by a new test. Re-captured AC-002 in the held-out set and rated it myself: 30 live generations, 1/30 still said "same destination" (down from 6/29), 0/30 called the permit line a block, 1/30 copied Example 1. Posted the new capture in docs/evaluation.md -- your and Shubham's earlier 4s rated the old text, so both of you rating this one fresh would be appreciated.

  3. Both stale references fixed: the redundant-case comment now says #52, not #444. tools/explanation_quality.py no longer calls 4.25/3.25 "the current measurement" -- reworded to say when it was taken and point at docs/evaluation.md for the real current number.

Re-requesting both of you since the push voids the existing approvals.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 148379e (it already contains main d424e05). Points 1 and 3 are done, and the tests are much stronger. But the wording that fixes point 2 brings in a new misdescription at a higher rate than the one it removes, so I'm requesting one more change. It's measured below, not guessed.

What holds:

  • Full suite at the head: 2265 passed, 13 skipped, ruff clean on CI's selection.
  • Point 1: Arsh's four mutants (the outcome said to "differ", the agreement sentence removed, the verb inverted, the verb fixed to "denies") are all killed now, and so are two of mine on the new code (singular and plural swapped, the blocking line not named). 6 of 6.
  • Point 3: both references are corrected.

The new problem: 30 live generations on AC-002, netwise-warden, same finding on main and on this head. I read every text, not just the pattern counts:

                                                     main     148379e
calls the dead PERMIT line a block (the #384 bug)    12/30     0/30
calls the source "a specific host"                    0/30    16/30   <- new
copies Modelfile Example 1                             0/30     1/30

16 of 30 now say "set to permit UDP traffic from a specific host". The dead line is permit udp 10.10.10.0 0.0.0.255 host 218.8.104.58: from a subnet, to a host. The action is right every time, so this is milder than the #384 inversion. But it's a confident misstatement in more than half of the explanations, and neither main nor 246db8e produced it. On 246db8e the same finding read "allows traffic to reach host 218.8.104.58". The difference is 08345b3, which puts the blocker into the fact in raw ACL syntax ((deny ip 10.10.10.0 0.0.0.255 any)). My guess is that two ACL lines side by side lead the model to misread the dead line's host keyword, but that's a guess. The measurement isn't.

Meanwhile the error 08345b3 was meant to fix has mostly gone anyway. On this head, "blocks all traffic with that same IP address range" is now the common wording, and it's accurate for deny ip 10.10.10.0/24 any.

Asked for:

  1. Describe the blocker in plain words rather than raw syntax, e.g. "an earlier line that denies all traffic from 10.10.10.0/24, to any destination". Or state the dead line's own source and destination in the fact too, so the model has nothing to infer. Either way, please re-run 30 on AC-002 and post the counts for these three rows.
  2. A test can't catch a phrasing the model invents, so the 30-run counts in the PR description are the evidence. Please include the "source called a specific host" row.

Rating the re-captured AC-002 (you asked Arsh and me to rate it fresh): accuracy 4, usefulness 4. The action and the source subnet are right. Two claims aren't in the evidence: "unencrypted" (the dead line is DNS; nothing says unencrypted) and "blocks all traffic through that same interface" (the blocker covers LAN-sourced traffic, not everything on the interface). If the wording changes under point 1, that capture will need re-taking anyway, so I'd rate the new one rather than add mine to this.

Arsh's separate list (Arista and Juniper dead rules getting no fact, "actually denied" that can be false, the ~2 in 120 Example 1 copies) is unchanged by this PR, and I agree they belong in their own issues.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 148379e. Points 1 and 3 are done, and I agree with Shubham's request on point 2. The wording I suggested causes a second misdescription that Shubham's review didn't cover, so this adds to his request rather than repeating it.

Confirmed:

  • CI-style suite: 2224 passed, 54 skipped (main 2218, so +6). Live: 2265 passed, 13 skipped, matching yours. Ruff is clean.
  • My four mutants from 5 November, plus the two on the new wording, are all caught. Same 6 of 6 as Shubham.
  • Shubham's "from a specific host" result reproduces in my own 30 AC-002 runs, at a lower rate: 3/30 on 246db8e, 10/30 here.

The two-line case gets worse, and that's on my suggestion. I gave you the plural branch without measuring it live, and nobody had. Two blocking lines that only together cover the dead one (deny tcp 10.1.1.0 0.0.0.127 any, deny tcp 10.1.1.128 0.0.0.127 any, then permit tcp 10.1.1.0 0.0.0.255 any eq 443). Batfish reports this shape, and it's the collective shadowing Yuan et al. describe. 30 live runs each, read as well as counted:

                                                    246db8e   148379e   148379e, plural branch reverted
says the blockers deny traffic TO that range           1/30     24/30     3/30
"unencrypted web traffic on port 443" (443 is HTTPS)   2/30     11/30     2/30
"internal network"                                     3/30     13/30     5/30
calls the dead PERMIT a block                          0/30      0/30     0/30

With two lines in raw syntax side by side, the model reads the source range as a destination, the same misreading Shubham found with host. The right column keeps the named line for one blocker and goes back to the 246db8e sentence for several, patched in memory only. It shows the plural case recovers, but it doesn't touch Shubham's single-line finding.

So, for point 2: whichever way you go, from Shubham's two options or another, please measure it on three shapes, not only AC-002: AC-002, this two-line case, and a redundant same-action pair. Post the counts for Shubham's three rows plus "denies traffic to that range". Every fix in this PR so far has moved an error from a measured shape into an unmeasured one, mine included.

Rating the 8 November capture, as you asked: accuracy 4, usefulness 4, for the same two reasons Shubham gives. "Unencrypted" isn't in the evidence, and "all traffic through that same interface" is wider than a blocker that only matches LAN-sourced traffic. As Shubham says, if point 2 changes the wording again, that capture gets re-taken, so I'll rate the new one then.

One correction to the record: your comment and docs/evaluation.md:810 say both of us gave the 3 November capture a 4. I didn't rate it. I said I'd add a 4 if the wording stayed, and it changed. Only Shubham's 4 exists, so please make that line say "one".

Nit: holdout.json lost its final newline.

A pushed fix notifies nobody, so please re-request both of us when it's up.

Shubham's review (9 November): naming the blocking line in raw ACL
syntax (the previous fix, 5 November) measurably traded that
misdescription for a different one. 30 live generations on AC-002:

  calls the dead PERMIT line a block (the #384 bug)   12/30 -> 0/30
  calls the source a specific host (new)                0/30 -> 16/30

The dead line's source is the subnet 10.10.10.0 0.0.0.255 (a /24); the
raw blocking-line text sitting next to the dead line's own raw text in
evidence.detail (which contains the literal word 'host' for its
destination) appears to make the model misattribute which address plays
which role.

Adds _describe_acl_line(), a narrow Cisco-ACL-line-to-plain-English
translator reusing this project's own established ipaddress-based
wildcard/netmask conversion (same technique ai/propose.py's
_cisco_endpoint() already uses in the opposite direction, same
contiguous-mask validation analysis/draft_policy.py already applies).
Recognises exactly the three endpoint forms this project's own checks
and converters ever produce (any, host A.B.C.D, A.B.C.D wildcard) and
refuses -- returns None -- on anything else: a source-port clause, an
unrecognised shape, a non-contiguous wildcard mask. The contradicted
branch of _compute_dead_rule_outcome() uses it when every named
blocking line parses; if any one does not, falls back to the current
raw-syntax naming for all of them, so this never blocks a finding from
being explained.

Tests: six new unit tests for _describe_acl_line() directly (both real
fixtures, a host destination, and three refusal cases); updated the
four existing tests that pinned the old wording.

Mutation-tested: forcing the helper to always return None breaks all 9
dependent tests for the right reason (back to raw-syntax wording); the
three refusal-path tests are unaffected either way, confirming they
test real refusal behaviour, not a side effect of the mutation.

Full suite: 2271 passed, 13 skipped, 0 failed.

Live-tested against the real model: reproduced Shubham's exact 30-run
measurement on the same AC-002 finding, read every output by hand
rather than trust a keyword heuristic (an early pattern-matching
attempt wildly overcounted, since 'a specific host' correctly
describes the real destination in most outputs). Real result:

  calls the dead PERMIT line a block: 0/30
  calls the source a specific host: 1/30 clear, 1/30 ambiguous (down from 16/30)
  copies Modelfile Example 1 verbatim: 0/30

Two things found while reading, reported but out of scope for this fix:
2/30 invert the flow direction (source and destination swapped), 2/30
echo Example 1's 'unencrypted' vocabulary without copying the phrase
exactly.

Re-captured and re-rated AC-002 in the held-out evaluation set a third
time (its prompt hash moved again), with a request for Shubham and Arsh
to rate the new capture.
@patelankeet2

Copy link
Copy Markdown
Collaborator Author

Pushed 6549573 (merges onto current main, full suite 2321 passed / 13 skipped / 0 failed). Fixed.

Raw ACL syntax is gone from the fact. Added _describe_acl_line(): a narrow Cisco-ACL-line-to-plain-English translator, using the same ipaddress-based wildcard/netmask technique ai/propose.py's _cisco_endpoint() already uses in the opposite direction (and the same contiguous-mask validation analysis/draft_policy.py already applies). It recognises exactly the three endpoint forms this project's own checks and converters ever produce (any, host A.B.C.D, A.B.C.D + wildcard) and refuses - returns None - on anything else (a source-port clause, an unrecognised shape, a non-contiguous wildcard mask). If every named blocking line parses, the fact describes them in plain English; if any one does not, it falls back to the current raw-syntax naming for all of them, so this never blocks a finding from being explained.

Re-ran your exact 30-generation measurement on the same AC-002 finding, and read every output by hand rather than trust a keyword count - my first pass at a pattern-matching script wildly overcounted, since "a specific host" correctly describes the real destination (218.8.104.58) in most outputs and a naive count can't tell that apart from it wrongly describing the source. Real result, read one at a time:

calls the dead PERMIT line a block (#384 bug):        0/30
calls the source a specific host (your finding):       1/30 clear, 1/30 ambiguous   (was 16/30)
copies Modelfile Example 1 verbatim:                    0/30

Two things I found while reading all 30 that neither of us had flagged before, reported honestly rather than left out since they're real, but out of scope for this fix:

  • 2/30 invert the flow direction - say traffic runs from 218.8.104.58 to the internal network, backwards from the real permit udp 10.10.10.0/24 host 218.8.104.58.
  • 2/30 describe the traffic as "unencrypted", which doesn't fit - the dead line is DNS (port domain), and nothing in the evidence says anything about encryption. Reads like an echo of Example 1's own vocabulary without copying the phrase exactly.

Neither is what you asked about, and I haven't touched either here - flagging them in case they're worth their own issue, same as Arsh's list.

Re-captured and re-rated AC-002 a third time (its prompt hash moved again) - docs/evaluation.md and the holdout fixture both updated. @ARSH871-bot @shubhamkataria2005, fresh capture against a changed fact, please rate again when you have a moment.

Testing:

  • pytest tests/test_explain_safety_net.py -v - 101 passed, including 6 new unit tests for _describe_acl_line() directly (both real fixtures, a host destination, and the three refusal cases).
  • Mutation test: forced the helper to always return None. All 9 dependent tests failed for the right reason (back to raw-syntax wording); the three refusal-path tests were unaffected either way, confirming they test real refusal behaviour, not a side effect of the mutation.
  • Full suite on the branch, merged onto current main: 2321 passed, 13 skipped, 0 failed.
  • Live-tested as above: 0/30 fallbacks, numbers reported above.
  • Byte-level check: no new em-dashes in any changed file.

patelankeet2 added a commit that referenced this pull request Nov 10, 2026
…antee (#453)

_compute_dead_rule_outcome()'s contradicted-action branch (own action vs.
blocking line's action disagree) told the model, as already verified and
certain, that 'traffic matching it is actually {outcome}' for the dead
line's ENTIRE matching pattern. Measured live (same on main db59b00 and
-- specifically, a line sharing the DEAD line's own action is not always
listed, and when one exists it can decide a narrower slice of traffic
the opposite way from what the named, opposite-action blocking line
alone predicts. A real config had permit host 10.1.1.5 any before both
the named deny and the dead permit; the fact said the whole /24 is
denied, but testFilters showed host .5 is actually permitted.

Confirmed only the contradicted branch is unsafe. The redundant branch
(blocking line agrees with the dead line's own action) is unaffected --
break it, 6 returned None (Batfish named the overlapping line), 2 had no
such overlap and the sentence was true. Worked out why: if an omission
is always a same-action-as-the-dead-line line, it cannot change the
outcome when every alternative (named or not) already agrees.

Also confirmed naming the blocking line's own text is not, by itself,
enough (the issue's suggested wording): the omitted line in the measured
case is earlier than the NAMED blocking line, not narrower than the dead
line's pattern -- host .5 matches the named deny's /24 too. The fix
drops the universal 'actually {outcome}' claim for this branch entirely,
stating only the two facts that stay true regardless of what
Blocking_Lines could have left out: the dead line's own action never
fires (filterLineReachability verifies that exhaustively), and the
named earlier line genuinely is evaluated first and genuinely does have
the opposite action.

Stacked on #401's branch (feature/384-dead-rule-own-action) rather than
main, since this function does not exist in its post-#401 shape on main
yet.

Tests: updated the four tests pinning the old universal-outcome wording
to the new one; added a new test reproducing #453's own measured shape
directly.

Mutation-tested: reverting to the old wording breaks all 7 affected
tests for the right reason; the two redundant-branch tests are
unaffected either way, confirming that branch genuinely does not need
this fix.

Full suite: 2266 passed, 13 skipped, 0 failed.

Live-tested against the real model: 15 generations on the measured
contradicted-case finding, 0/15 fallbacks, wording correctly follows the
new, narrower fact in every non-copy case.

Re-captured and re-rated AC-002 in the held-out evaluation set (its
prompt hash moved since this changes the exact text fed to the model for
that finding) -- docs/evaluation.md and the holdout fixture both
updated, with a request for Shubham and Arsh to rate the new capture.
patelankeet2 added a commit that referenced this pull request Nov 10, 2026
)

Rebasing onto #401's plain-English blocking-line fix meant every
existing fixture for the contradicted branch now takes that primary
path, leaving #453's own fix -- the dropped universal-outcome claim in
the raw-syntax fallback -- with no test reaching it at all.

Added one that does, using a blocking line shaped so _describe_acl_line()
refuses it (a source-port clause). Mutation tested: reverting the
fallback to the old 'traffic matching it is actually {outcome}' wording
fails this new test for the right reason; the existing 'omitted' test is
unaffected, since it takes the primary path either way.
A small git-hygiene nit named in review -- the file's last save dropped
the final newline after the closing brace.
@patelankeet2

Copy link
Copy Markdown
Collaborator Author

@ARSH871-bot the 3-shape measurement, read by hand (90 generations, not pattern-matched) against the already-pushed fix (6549573):

                                              AC-002    two-line case    redundant pair (control)
calls the dead PERMIT line a block            0/29*         0/30              0/30
"denies traffic TO that range" (your find)      n/a         0/30              n/a
calls the source a specific host (Shubham's)  0/29*          n/a               n/a
copies Modelfile Example 1 verbatim           0/29*         1/30              0/30

* 1/30 fell back on this run, not a wrong answer -- 29 model-generated texts read.

Your own finding, the two-line inversion, is gone: 0/30, down from your measured 24/30. Makes sense once I checked why: _describe_acl_line() never puts raw syntax in the fact at all, singular or plural, so the collision you found (two raw snippets, model loses track of which address is source and which is destination) has nothing to collide with anymore. I hadn't verified that against your exact two-line shape before pushing -- should have measured before claiming it, not after. It's measured now.

Fixtures used, so you can re-run them: the two-line case is your own (deny tcp 10.1.1.0 0.0.0.127 any, deny tcp 10.1.1.128 0.0.0.127 any, dead permit tcp 10.1.1.0 0.0.0.255 any eq 443). The redundant pair mirrors it with both blockers set to permit instead of deny, so own == outcome and it exercises the OTHER branch -- the one neither my fix nor Shubham's finding touches, included as your control.

Found while reading, not part of what either of you asked about: the redundant-pair control shows the model inventing an encryption-status claim with no basis in the evidence -- 6/30 say "encrypted web traffic" or "unencrypted web traffic" for a TCP/443 dead line that carries no claim about encryption either way. Two-line shows the same thing at 3/30, separate from its one Example-1 copy. This branch's wording is unchanged by this PR (confirmed: it's the same template that was already on main), so it's pre-existing, not something I introduced -- flagging it as a possible issue of its own, same as the direction-inversion and "unencrypted" items I reported earlier.

Fixed, both nits:

  • holdout.json's trailing newline, pushed as bce9017.
  • The "your earlier 4s" line: already gone. That was the 8 November row; the 9 November rewrite (already up before your review landed) says "your earlier ratings were of the 8 November text" instead, no longer claiming a plural you didn't give. Checked the rest of the doc for the same mistake and didn't find it repeated anywhere else.

Re-requested you and Shubham again since the newline fix is one more push.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving bce9017. The plain-English blocker fixed what I asked about, and your hand counts across three shapes are the right evidence for it.

My own check, the same 30-run AC-002 setup as my last review, on this head:

                                           main    148379e   bce9017
calls the dead PERMIT line a block        12/30     0/30      0/30
copies Modelfile Example 1                  0/30     1/30      0/30
"from a specific host" (pattern)            0/30    16/30      4/30

On the last row you're right that the plain phrase "a specific host" over-counts, because it's often the destination. I matched on "from a specific host", which is narrower, but I didn't read these 4 by hand this time, so take your hand count (1 clear, 1 ambiguous) over my pattern. Either way it's down from 16, and the PERMIT-called-a-block bug this PR exists for is 0.

  • Merged onto current main (2833a1e): 2321 passed, 13 skipped, ruff clean on CI's selection.
  • _describe_acl_line() refusing rather than guessing (a source-port clause, an unrecognised shape, a non-contiguous wildcard) is the right shape. Falling back for the whole set when any line fails keeps the fact consistent.

My rating of the new AC-002 capture: accuracy 5, usefulness 4. Every claim is true: source subnet, destination host, the permit never enforced, and the blocker correctly "to any destination". The one imprecision is "the same source IP address" for a /24 range, which doesn't change the meaning. Usefulness 4 because it doesn't say it's the DNS line or what to do about it. Please add it as my rating in holdout.json, beside yours.

Merge order: #461 is stacked on an older head of this branch and re-captures AC-002 too. I've requested changes there, because its fallback wording measures 10/20 Example 1 copies. Land this one first.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed bce9017. The "to that range" misreading is fixed. The source misreading is not, and the plural case has a new one. I'm still requesting changes, because the PR claims a measurement I can't reproduce.

Fixed, and I agree

  • holdout.json has its final newline back, and the "both gave it a 4" line is gone.
  • Two blockers, "the blockers deny traffic TO that range": 24/30 on 148379e, 0/30 now. The plain-English translation did that.
  • The dead PERMIT called a block: 0/30 on AC-002 and on two blockers. Modelfile Example 1 copied verbatim: 0 of 90.

Not reproduced: AC-002 "source is a specific host"
Same finding, real explain_with_source(), 30 generations, each hit read in context:

  • 11/30 describe the dead line's /24 source as one host or address: 7 say "from a specific host", and 4 say "from a specific IP address to reach the server at that address".
  • That is not the "1/30 clear, 1/30 ambiguous, down from 16/30" in the commit message and the held-out row. On 148379e I counted 10/30, so on my count it did not move.
  • I did not count "to a specific host (218.8.104.58)", which is correct: the destination is a host.
  • You may be counting more strictly. If so, please post the literal phrases you counted, so the two of us can compare lists rather than numbers.

New: the destination of the two-blocker case is called "the internal network"
The destination in that case is any. 18/30 say "permitting TCP traffic on port 443 to the internal network". It fixes one misreading of the destination and puts another in its place: the same pattern as the last round.

The redundant pair, which I asked you to measure and which wasn't
permit tcp 10.1.1.0 0.0.0.255 any eq 443 shadowed by an earlier permit tcp 10.1.1.0 0.0.0.255 any. 15/30 describe a permit line as blocking or blocked ("is blocked by an earlier rule that also permits traffic", "never actually blocks traffic, despite its stated action of PERMIT"). On main (ee769b4) it is 14/30, so this is not a regression from #401. It is a real misdescription that was already there, and I'd file it rather than fix it here.

                                      148379e      bce9017
AC-002  source called one host         10/30        11/30
AC-002  dead PERMIT called a block      0/30         0/30
2 blockers  "traffic TO that range"    24/30         0/30
2 blockers  destination "internal net"   --         18/30
redundant pair  permit called blocking   --         15/30   (main: 14/30)
Example 1 verbatim                       --          0/90

What I'd try, untested: the fact sentence is now plain English, but evidence.detail still carries both lines in raw ACL syntax (permit udp 10.10.10.0 0.0.0.255 host 218.8.104.58 ... and the deny line). If the model is still reading roles off the raw text, leaving the raw lines out of what it sees for this finding would test that. I haven't run it.

Rating the capture, as you asked: accuracy 4, usefulness 4. It says traffic "from the same source IP address" for a /24, and calls 10.10.10.0/24 "the internal network", which isn't in the evidence. Everything else is right and it names the real cause. The row currently shows your own 5/5, which counts as your rating, not as the second one.

Please:

  1. Re-measure all three shapes the same way, post the literal phrases counted for the AC-002 source and the two-blocker destination, and correct the commit message and docs/evaluation.md if the AC-002 number moves.
  2. File the redundant-pair finding as its own issue.
  3. Re-request me when it is up. A pushed fix notifies nobody. @shubhamkataria2005 approved this head on 10 November; please look at the 11/30 and 18/30 figures above before it merges, since they came after your approval.
How I measured

Same stubbed filterLineReachability frame as my 5 November script, through access_control._check_dead_rules and the real explain_with_source() against local Ollama, 30 runs per shape, bce9017 checked out.

"""30 live generations on three shapes through the real explain_with_source(), at #401's bce9017. Run from the wt-401 tree. Counts are a first pass; samples are read by hand."""
import json, re, sys
from pathlib import Path
sys.path.insert(0, ".")
sys.stdout.reconfigure(encoding="utf-8")
src = open(r"C:/Users/vhora/Netwise/scratchpad/reviews/r401/live_401b.py", encoding="utf-8").read().split("EX1 =")[0]
exec(src)
from ai.explain import explain_with_source  # noqa
CASES["redundant pair"] = ("permit tcp 10.1.1.0 0.0.0.255 any eq 443", "PERMIT", ["permit tcp 10.1.1.0 0.0.0.255 any"])
EX1 = "internal server can be reached over unencrypted web traffic"
BLOCKISH = re.compile(r"supposed to (block|deny)|intended to (block|deny)|would (normally )?(block|deny)|should (block|deny)|permit line (blocks|denies)", re.I)
TORANGE = re.compile(r"(deny|denies|denying|block|blocks|blocking)\w*[^.]{0,40}(traffic )?(to|destined for|headed to) (that|the same|this) (range|subnet|network)|same destination", re.I)
HOST = re.compile(r"(specific|single|one) (host|machine|device|address)", re.I)
UNENC = re.compile(r"unencrypted", re.I)
INTERNAL = re.compile(r"internal network", re.I)
out = {}
for name, args in CASES.items():
    f = finding(*args)
    runs = [dict(zip(("text", "source"), explain_with_source(f))) for _ in range(30)]
    model = [r["text"] for r in runs if r["source"] == "model"]
    out[name] = runs
    c = lambda rx: sum(bool(rx.search(t)) for t in model)
    print(f"{name}: model {len(model)}/30 | permit called a block {c(BLOCKISH)} | Example1 verbatim {sum(EX1 in t for t in model)} | denies traffic TO the range {c(TORANGE)} | specific host {c(HOST)} | 'unencrypted' {c(UNENC)} | 'internal network' {c(INTERNAL)}", flush=True)
    Path(r"C:/Users/vhora/Netwise/scratchpad/reviews/r401/live-bce9017.json").write_text(json.dumps(out, indent=1, ensure_ascii=False), encoding="utf-8")
print("DONE")

@shubhamkataria2005 shubhamkataria2005 left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing my 10 November approval of bce9017 to request changes. @ARSH871-bot asked me to look at his figures before this merges. I re-measured them, and they hold.

My approval deferred to the hand count in the commit message ("1/30 clear, 1/30 ambiguous") over my own pattern count. That was the wrong call: the count doesn't reproduce.

Same head, 30 live generations per shape through the real explain_with_source(), all 60 model-written. I read the matching sentence in each run.

                                                         Arsh     mine
AC-002: the /24 SOURCE described as one host/address    11/30    13/30
2 blockers: destination `any` called "the internal net"  18/30    20/30

AC-002, what I counted (runs 1, 2, 4, 7, 13, 16, 18, 20, 21, 23, 24, 29, 30). All describe the source:

  • "allow traffic from a specific IP address to reach the server at that address" (runs 1, 21)
  • "UDP traffic from a specific host to reach the internal network" (2, 16, 29); this one also gets the destination wrong
  • "intended to allow UDP traffic from a specific host" (4, 13, 23)
  • "allow traffic from a specific IP address" (7, 18, 30)
  • "allow UDP traffic from a specific host on the internal network" (20, 24)

Not counted: "to a specific host (218.8.104.58)", which is the destination and correct; "from a specific IP address range" (run 12); and "from a specific internal network (10.10.10.0/24)" (run 6).

Two blockers: 20 of 30 open with "permitting TCP traffic on port 443 to the internal network". The dead line is permit tcp 10.1.1.0 0.0.0.255 any eq 443, so the destination is any.

So the PR fixes what it set out to fix: 0/30 call the PERMIT line a block, and 0/30 now say "traffic TO that range". But both remaining misreadings are at the old rates or higher, and the commit message and docs/evaluation.md report a number for the first that a third reader can't reproduce. Please correct those, and try Arsh's untested idea (leave the raw ACL lines out of what the model sees for this finding), since both errors look like the model reading roles off the raw syntax rather than the plain-English fact. Re-request me with the counts and the literal phrases, and I'll re-measure the same day.

My rating of the AC-002 capture, corrected: accuracy 4, usefulness 4 (I gave it 5/4 on 10 November). Arsh's point is right: the capture calls the source "the internal network", which isn't in the evidence, and says "the same source IP address" for a /24. So it overstates what the evidence supports, which is what a 4 means on Part 4's scale.

@ARSH871-bot

Copy link
Copy Markdown
Owner

@patelankeet2, checking in on this one. Nothing has moved since my review on 16 November, and Shubham re-measured on 17 November and now requests changes too. His counts are close to mine (13/30 and 11/30 for the /24 source described as one host), so I think the numbers are settled.

What would unblock it, from both reviews:

  1. A wording change, then 30 runs on each of the three shapes (AC-002, the two-blocker case, the redundant pair), posting the literal phrases you counted.
  2. The commit message and docs/evaluation.md brought in line with whatever you measure.
  3. The redundant-pair misdescription filed as its own issue. It is on main too (14/30), so it doesn't need to be fixed here.
  4. Re-request both of us when it's up.

#461 is stacked on this branch too, so it is waiting on this as well. If you're short on time this week, say so and we can split the work or park it. Happy to pair on the measurement if that helps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working layer: ai Layer 2 — local LLM explanation and Q&A safety Touches F-4, grounding, or the refuse-rather-than-guess boundary. Review with care

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A dead-rule explanation can invert the unreachable line's own stated action

3 participants