Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
217 changes: 210 additions & 7 deletions ai/explain.py
Original file line number Diff line number Diff line change
Expand Up @@ -146,16 +146,175 @@
)


def _parse_cisco_endpoint(tokens, i):
"""Consume one Cisco ACL endpoint (source or destination) starting at
tokens[i]: "any", "host A.B.C.D", or "A.B.C.D W.X.Y.Z" (wildcard) --
the exact three forms ai/propose.py's own _cisco_endpoint() ever
emits in the other direction. Returns (kind, value, next_index), or
None for anything else -- refuse rather than guess at an
unfamiliar shape (#401 review, @shubhamkataria2005, 9 November).
"""
if i >= len(tokens):
return None
if tokens[i] == "any":
return "any", None, i + 1
if tokens[i] == "host":
if i + 1 >= len(tokens):
return None
return "host", tokens[i + 1], i + 2
if i + 1 >= len(tokens):
return None
addr, wildcard = tokens[i], tokens[i + 1]
octets = wildcard.split(".")
if len(octets) != 4:
return None
try:
netmask = ".".join(str(255 - int(b)) for b in octets)
network = ipaddress.ip_network(f"{addr}/{netmask}", strict=False)
except ValueError:
return None # not a valid address, or a non-contiguous wildcard
# mask (the inverted bytes do not form a legal netmask) -- same
# reason analysis/draft_policy.py refuses one of these as
# "cannot be written as one" prefix.
return "network", f"{network.network_address}/{network.prefixlen}", i + 2


def _format_endpoint(kind, value, role):
if kind == "any":
return "anywhere" if role == "source" else "any destination"
if kind == "host":
return f"host {value}"
return value # bare CIDR, e.g. "10.10.10.0/24"


def _describe_acl_line(line):
"""A plain-English description of one Cisco ACL line's own criteria,
not a restatement of its raw syntax.

WHY THIS EXISTS (#401 review, @shubhamkataria2005, 9 November)
Naming the blocking line in raw ACL text (the previous fix for a
different misdescription, found in review 5 November) measurably
traded one misdescription for another: 30 live generations on
AC-002 went from 12/30 calling the dead PERMIT line a block (the
original #384 bug, now 0/30) to 16/30 calling the dead line's own
/24 source "a specific host" -- a confusion that did not happen
on main or on the commit before the raw-syntax fix. The raw
blocking-line text sits next to the dead line's own raw text
(also shown to the model, in evidence.detail), and the two
addresses of different specificity next to each other appear to
get their roles crossed. Removing the raw ACL syntax from the
computed fact removes that collision.

Recognises exactly "permit|deny PROTOCOL SRC [SRC] DST [DST] [eq
PORT]" -- the only shape this project's own checks and converters
ever produce. Returns None for anything else (an unrecognised
protocol-keyword shape, a source-port clause, trailing tokens, a
non-contiguous wildcard mask) rather than guess at it; the caller
falls back to the current raw-syntax naming in that case, so this
never blocks a finding from being explained, only changes how
confidently the blocking line's own criteria can be phrased.
"""
tokens = line.split()
if len(tokens) < 4:
return None
action = tokens[0].lower()
if action not in ("permit", "deny"):
return None
verb = "permits" if action == "permit" else "denies"
protocol = tokens[1].lower()

src = _parse_cisco_endpoint(tokens, 2)
if src is None:
return None
src_kind, src_value, i = src
dst = _parse_cisco_endpoint(tokens, i)
if dst is None:
return None
dst_kind, dst_value, i = dst

port = None
if i < len(tokens):
if tokens[i].lower() == "eq" and i + 1 < len(tokens):
port = tokens[i + 1]
i += 2
else:
return None # a shape this project does not recognise
if i != len(tokens):
return None # trailing tokens left over -- refuse rather than guess

proto_phrase = "all traffic" if protocol == "ip" else f"{protocol.upper()} traffic"
sentence = (
f"{verb} {proto_phrase} from {_format_endpoint(src_kind, src_value, 'source')} "
f"to {_format_endpoint(dst_kind, dst_value, 'destination')}"
)
if port is not None:
sentence += f" on port {port}"
return sentence


def _compute_dead_rule_outcome(detail: str) -> Optional[str]:
"""For a dead-ACL-rule finding, compute what actually happens to
traffic matching the dead line -- rather than asking the model to
infer it from the raw text, which measurably goes wrong sometimes.
traffic matching the dead line, AND what the dead line's own action
actually is -- rather than asking the model to infer either from the
raw text, which measurably goes wrong sometimes.

The rule: whichever line SHADOWS the dead one decides the outcome for
matching traffic, because it is evaluated first. A dead line's own
action never takes effect, regardless of what it says.

Returns a plain-English statement of the real outcome, or None if:
THE DEAD LINE'S OWN ACTION IS A SEPARATE FACT FROM THE OUTCOME (#384)
The original version of this function stated only the OUTCOME
(what happens to matching traffic) and deliberately did not state
what the dead line's own action IS -- on the reasoning that it
"never takes effect, regardless of what it says", so stating it
seemed unnecessary. Found live, not assumed: `(action PERMIT)` is
shown in the raw evidence text either way, and sometimes the model
mischaracterised a PERMIT line as a block/deny rule anyway, even
though it correctly stated the OUTCOME was denied -- e.g. "the
device's ACL rule, which is supposed to block traffic... never
actually takes effect". Correct about the outcome, backwards about
what the dead line itself says.

Re-measured on the same real finding immediately before this fix:
3 of 8 live generations mischaracterised or were internally
inconsistent about the dead line's own action (worse than the 1-in-8
originally reported). Fixed the same way the other computed-fact
functions already handle a two-sided fact: state both sides
explicitly, not just the one judged most important.

WHY THIS FUNCTION ALSO NEEDS A BRANCH THE OTHER TWO DON'T (found in
review, before this merged)
_compute_policy_outcome() and _compute_expected_actual_outcome() can
always say their two sides differ, because the finding that feeds
them only EXISTS when they do -- a policy_compliance violation and a
testFilters mismatch are both, by construction, "expected one thing,
got the other". A dead-ACL-rule finding carries no such guarantee:
it exists whenever a line is unreachable, whether the line shadowing
it agrees with the dead line's own action (redundant -- a tidiness
problem) or disagrees (the real exposure this check exists to catch).
The two facts are independent, so a single template that always
says "different" is false for the redundant case.

Measured live, 60 generations each, on a real redundant-deny finding
(a DENY shadowed by an earlier DENY): the single-template wording
that shipped before this branch existed produced "no traffic is
actually denied" and "the actual action... is different from what
its own stated action claims" -- the exact #52 inversion this whole
function exists to prevent, reintroduced by assuming "different"
when the real relationship is "the same, coincidentally". Separately
measured on AC-002 (a real PERMIT-vs-DENY finding, where the two
sides genuinely do differ): the previous single template's own
repetition ("not what happens... what actually happens is
different... regardless of what it says") measurably made the model
more likely to give up and copy the Modelfile's Example 1 almost
verbatim instead of using the computed fact -- 0/60 on the
single-template wording's predecessor against 9/60 with the
single-template version. The two-branch wording below removes both:
each branch states its own case once, plainly, and never claims
"different" when the lines happen to agree.

Returns a plain-English statement of the real outcome AND the dead
line's own stated action, or None if:
- `detail` is not in the exact shape access_control.py's dead-rule
check produces (nothing to compute from), or
- more than one blocking line is named and they do not all agree on
Expand Down Expand Up @@ -187,11 +346,55 @@ def _compute_dead_rule_outcome(detail: str) -> Optional[str]:
return None # blocking lines disagree -- not confident enough to state one outcome

outcome = outcomes.pop()
dead_action = match.group("dead_action")
own = "permitted" if dead_action == "PERMIT" else "denied"

if own == outcome:
# Redundant, not contradicted -- the two facts happen to agree. Never
# say "different" or "not what happens" here; that is the #52
# inversion, reintroduced.
verb = "permits" if outcome == "permitted" else "denies"
return (
f"the unreachable line's own stated action is {dead_action}, "
f"and traffic matching it is {outcome} anyway, by an earlier "
f"line that also {verb} it and is evaluated first. The outcome "
f"is the same as the unreachable line says; only the line that "
f"decides is different."
)

# Describe the actual blocking line(s) in plain English rather than a
# vague "same traffic" claim (found in review, 5 November) OR raw ACL
# syntax (found in review, 9 November -- raw syntax measurably traded
# that misdescription for a different one, confusing the model about
# which address was "a specific host"; see _describe_acl_line()'s own
# docstring). If any blocking line does not parse, fall back to
# naming them in raw syntax rather than withholding the fact.
descriptions = [_describe_acl_line(line) for line in blocking_lines]
if all(d is not None for d in descriptions):
if len(descriptions) == 1:
return (
f"the unreachable line's own stated action is {dead_action}, "
f"but it never takes effect: an earlier line, evaluated "
f"first, {descriptions[0]} instead."
)
first, rest = descriptions[0], descriptions[1:]
cases = f"one {first}" + "".join(f", and another {d}" for d in rest)
return (
f"the unreachable line's own stated action is {dead_action}, "
f"but it never takes effect: earlier lines, evaluated first, "
f"decide instead -- {cases}."
)

named = " ".join(match.group("blocking").split()) # Batfish pads its spaces
lines, verb, decide_verb = (
("earlier lines", "are", "decide") if len(blocking_lines) > 1
else ("an earlier line", "is", "decides")
)
return (
f"traffic matching the unreachable line's pattern is actually "
f"{outcome}, because the blocking rule is evaluated first and "
f"decides instead. The unreachable line's own action never takes "
f"effect, regardless of what it says."
f"the unreachable line's own stated action is {dead_action}, but it "
f"never takes effect: traffic matching it is actually {outcome}, "
f"because {lines} ({named}) {verb} evaluated first and {decide_verb} "
f"instead."
)


Expand Down
55 changes: 42 additions & 13 deletions docs/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -774,9 +774,30 @@ functions already name both sides of their fact), and the same discipline
#145 itself set: **not fixed here.** Fixing it now would mean rating text
that no longer exists by the time anyone reads this table.

> **RE-CAPTURED, after #401 fixed #384 (reviewed by @ARSH871-bot before
> merge).** #401 extended `_compute_dead_rule_outcome()` to name the dead
> line's own action explicitly, which is exactly the fix this section asked
> for -- and review found a second problem while measuring it: the
> single-template wording also assumed a dead line's own action and the
> real outcome always disagree, which is false whenever the line is
> redundant rather than contradicted (a DENY shadowed by an earlier DENY,
> say). Both are fixed in #401, with a branch on whether the two sides
> agree. #385's own prompt-hash guard caught AC-002 as stale the moment
> #401 merged onto this held-out set, which is exactly what that guard
> exists to catch -- re-captured under the fixed prompt rather than editing
> the old rating in place, same discipline this section used the first
> time. The table and the repeatable score below are both updated; the
> `AC-002` row and this paragraph are left as the honest record of what was
> found and when, not rewritten to read as if it were always correct.

### The held-out set and its ratings

One rater so far (Ankeet) - see
Four rounds so far (Ankeet, 10 October; AC-002 re-captured and re-rated by
Ankeet, 3 November, after #401; AC-002 re-captured and re-rated by Ankeet
again, 8 November, after review found the 3 November capture's own wording
was itself a misdescription; AC-002 re-captured and re-rated a third time,
9 November, after review found the 8 November wording traded that
misdescription for a different one) - see
[`tests/fixtures/explanation_quality/holdout.json`](../tests/fixtures/explanation_quality/holdout.json)
for the full captured findings, explanations and reasoning behind each
score. Same 1-5 scale as Part 3 (accuracy: does the text claim only what the
Expand All @@ -788,7 +809,7 @@ read `evidence.detail` what is wrong).
| RT-050 | fallback | 5 | 4 | Deterministic, cannot be wrong by construction - **reported separately below, not part of the model score.** |
| AC-001 | model | 5 | 4 | Correct, no inversion. |
| PC-001 | model | 5 | 4 | Regenerated post-#145: inversion gone. |
| AC-002 | model | **2** | **2** | **Real defect, kept rather than re-rolled - see #384.** |
| AC-002 | model | 5 | 5 | **Re-captured a third time, 9 November, after review found that naming the blocking line in raw ACL syntax (the 8 November fix) traded one misdescription for another.** 30 live generations on the 8 November wording: 0/30 called the permit line a block, but 16/30 called the dead line's own /24 source "a specific host" - a confusion neither the pre-fix wording nor `main` produced, caused by two raw ACL snippets sitting next to each other in the prompt. Fixed by describing the blocking line in plain English instead of raw syntax, refusing rather than guessing on any shape it does not recognise. Re-measured, same finding, 30 generations: 0/30 call the permit line a block, 1/30 clearly calls the source a specific host (down from 16/30), 1/30 ambiguous, 0/30 copy Modelfile Example 1. This capture states both source and destination explicitly and correctly. Two things found along the way, reported but out of scope here: 2/30 invert the flow direction, 2/30 echo Example 1's "unencrypted" vocabulary without copying the phrase exactly. **@ARSH871-bot @shubhamkataria2005, fresh capture against a changed fact - your earlier ratings were of the 8 November text. Please rate again.** |
| PC-005 | model | 5 | 3 | Regenerated post-#145: inversion gone. First-sentence phrasing still asks the reader to resolve an apparent contradiction before it resolves itself - a readability risk, not a false claim this time. |

@ARSH871-bot @shubhamkataria2005 @SamikaPerera - a second independent rating
Expand All @@ -804,14 +825,20 @@ python -m tools.explanation_quality

```
4 model-written findings, 4 ratings
accuracy: mean 4.25 min 2
usefulness: mean 3.25 min 2
accuracy: mean 5.00 min 5
usefulness: mean 4.00 min 3

fallback-sourced (not scored against the threshold - cannot be wrong by
construction):
RT-050 accuracy 5.0 usefulness 4.0 (1 rating(s))
```

The minimum moved from AC-002's accuracy=2 to PC-005's usefulness=3 - a
readability nit already noted above, not a factual inversion. The
per-finding breakdown is still what surfaces it rather than the mean
hiding it; see `tools/explanation_quality.py`'s own docstring for why mean
alone was rejected as the published number in the first place.

**Published, and pinned in `tests/test_explanation_quality.py`** so CI
catches it if the committed ratings ever drift from what this document
claims - the same "generated artefact can go stale between the commit and
Expand Down Expand Up @@ -853,15 +880,17 @@ mean accuracy 4.0, mean usefulness 3.0 - both below the measurement above,
so the threshold is a real tripwire rather than a number chosen to always
pass.

**Deliberately based on the mean, not the minimum.** `AC-002`'s accuracy=2 is
a correctness bug in one evidence shape, the same class of thing #52 and
#145 already were, and both of those were fixed for their one shape rather
than by turning the model off for every finding. Tying this threshold to the
minimum would mean one narrow, already-understood, 1-in-8 defect forces every
explanation in the product onto the deterministic fallback - a far bigger
change than the finding justifies. The mean is what actually answers the
question this threshold exists for: has quality collapsed broadly, not did
one shape have one bad day.
**Deliberately based on the mean, not the minimum.** `AC-002`'s accuracy=2
(now fixed by #401, re-captured at 5/4 above) was a correctness bug in one
evidence shape, the same class of thing #52 and #145 already were, and all
three were fixed for their one shape rather than by turning the model off
for every finding. Tying this threshold to the minimum would have meant one
narrow, already-understood, 1-in-8 defect forcing every explanation in the
product onto the deterministic fallback - a far bigger change than the
finding justified. The mean is what actually answers the question this
threshold exists for: has quality collapsed broadly, not did one shape have
one bad day. The reasoning stands regardless of which specific finding is
currently lowest.

**Why this is a documented number, not a runtime check.** There is no honest
way to score one live generation's quality automatically without another
Expand Down
Loading
Loading