Repository navigation
Description
Found while building the held-out evaluation set for #346 (US-55), on a fresh live regeneration of AC-002 (tests/fixtures/rtr-us5-messy) -- not from reading the diff, from the real pipeline plus the real model.
The bug
detail: Unreachable line: permit udp 10.10.10.0 0.0.0.255 host 218.8.104.58
eq domain (action PERMIT). Blocked by: deny ip 10.10.10.0
0.0.0.255 any. Reason: BLOCKING_LINES
explanation:
"The device's ACL rule, which is supposed to block traffic from a
specific IP address, never actually takes effect because it is
overridden by another rule that blocks all traffic with the same
destination IP address. As a result, even though the unreachable
line in the ACL rule would normally deny traffic matching its
pattern, the blocking rule always wins and prevents any traffic
from being allowed."
The unreachable line's own action is explicitly PERMIT in evidence.detail. The explanation describes it twice as if it were a block/deny rule ("which is supposed to block traffic", "would normally deny traffic matching its pattern"). The final-outcome claim (traffic ends up denied, because the blocking rule wins) is correct -- this is not the #52 mistake repeating. What is backwards is the characterisation of the DEAD LINE'S OWN action, a different fact than the one #52's fix already protects.
Why this is a new category, not #52 repeating
_compute_dead_rule_outcome() already feeds the model an already-verified fact about the OUTCOME: "traffic matching the unreachable line's pattern is actually {outcome}... The unreachable line's own action never takes effect, regardless of what it says." That sentence deliberately does not state what the line's action actually IS -- it only says it doesn't matter. The model is left to read (action PERMIT) off the raw evidence text itself, and on this occasion it inverted that instead.
How often, checked rather than assumed
Same AC-002 finding, same evidence, six more live generations immediately after:
run 1: "...the unreachable line itself would have allowed it if not blocked..." correct
run 2: "...the permitting rule itself would normally allow it to pass through." correct
run 3: "...even though the ACL rule allows certain traffic to pass..." correct
run 4: "...even though the permitting rule is present..." correct
run 5: "...the unreachable line itself does not take any action." correct (vague, not backwards)
run 6: "...the unreachable line in the ACL would normally permit such traffic..." correct
0 of 6 repeated it. Combined with the two earlier captures (1 bad, 1 clean), that is 1 inversion in 8 live generations of the same finding -- infrequent, not reproducible on demand, but real and not invented: this project's own two prior guards (_compute_dead_rule_outcome, _compute_policy_outcome) were each added after exactly one observed real failure, and the third guard (_compute_expected_actual_outcome) was added proactively on structural risk alone, with zero failures observed in six tries. This sits between those two bars, not below them.
Suggested next step, not applied here
Extend _compute_dead_rule_outcome()'s computed-fact string to also name the unreachable line's own action explicitly ("The unreachable line's own stated action is {PERMIT|DENY}, but..."), the same discipline _compute_policy_outcome() and _compute_expected_actual_outcome() already use for naming both sides of their own two-state facts. Proposal only -- this is ai/explain.py, not filing a PR against it from here.
Why this is filed separately rather than fixed under #346
#346's own held-out set is a measurement, and the exact discipline #145 already established: fixing the code that generated the held-out sample mid-measurement means rating text that no longer exists. The held-out fixture for #346 keeps this exact generation, rated honestly (low), rather than re-rolling it for a cleaner sample or silently fixing the cause first. This issue is tracked so the finding is not lost in a PR description.
Activity
US-55: hash the built prompt per finding, not just ai/Modelfile
Re-capture and re-rate AC-002 under the fixed prompt (#385's guard)
ARSH871-bot commentedon Nov 5, 2026
Reopening. #385's merge closed this because its description lists it as closed, but the fix is #401, which hasn't merged, so main still has the defect. #401's "Closes #384" will close it properly when it lands.
Describe the blocking line in plain English, not raw ACL syntax (#401)