Skip to content

R-1 matches a dead ACL line, not the word "unreachable" (#444) - #445

Open
SamikaPerera wants to merge 6 commits into
mainfrom
fix/444-r1-matches-dead-acl-lines-only
Open

SamikaPerera wants to merge 6 commits into
mainfrom
fix/444-r1-matches-dead-acl-lines-only

Conversation

@SamikaPerera

Copy link
Copy Markdown
Collaborator

Based on #443, merge that first. This PR's base is #443's branch, fix/r1-dead-deny-keeps-severity. Before #443 is merged, this PR needs gh pr edit <this PR> --base main, as CONTRIBUTING.md says for a stacked PR.

Fixes #444.

What changes

R-1 in analysis/checks/risk.py now fires only when a finding's evidence starts with Unreachable line: . That is the opening access_control writes for a dead ACL line (access_control.py:608-610). Before, R-1 fired on the word "unreachable" anywhere in the evidence.

Why

R-1 is meant for dead ACL lines. Matching the word also lowered findings that have nothing to do with one:

finding before after
routing, "Expected UNREACHABLE ..." and the path succeeded low medium (R-3)
routing, path ended in NEIGHBOR_UNREACHABLE low medium (R-3)
a finding quoting an ICMP unreachable, host-unreachable or port-unreachable line low the check's own severity
an undefined or unused filter whose name contains "UNREACHABLE" low the check's own severity
dead permit line low low (unchanged)
dead deny line high on #443 high (unchanged)

Low sorts to the bottom of the list, so these sat below every medium.

The match is on the start of the evidence, not on Unreachable line: anywhere. A quoted ACL line or a name is config text, and every other check opens its evidence with its own fixed words.

Three commits

  1. Tests, red on purpose. Seven new cases in tests/test_severity_rules.py. The two routing details are built from routing._evaluate() itself. One existing test, test_dead_rule_beats_the_blanket_permit_it_quotes, used wording no check writes ("Unreachable: ... -- blocked by: ..."). It now uses access_control's real wording, with a dead permit ip any any line, so it still tests why R-1 runs before R-2.
  2. The change in risk.py. The docstring's "first on purpose" paragraph said R-2 would match the quoted blocker. It would not, because the blocker follows "Blocked by:", which R-2 treats as a citation. The paragraph now names the case that needs R-1 first: a dead line that is itself a blanket permit.
  3. docs/severity-rules.md: R-1's "Matches" line now says what the code matches, and the "first" paragraph gets the same correction. No other rule is touched.

status is never written. Only severity changes, as before.

What I had Claude Code run

  • tests/test_severity_rules.py on R-1: a dead deny keeps its check's severity #443's head with only commit 1: 7 failed, 26 passed. Every failure was R-1 returning low.
  • The same file after commit 2: 33 passed.
  • Commit 2's risk.py change reverted on its own: the 7 new tests failed, and the dead permit and dead deny tests passed.
  • R-2 moved ahead of R-1: only test_dead_rule_beats_the_blanket_permit_it_quotes failed (1 failed, 32 passed).
  • The other test files that import risk (business context scoring and wiring, CVE mapping) plus test_explain_safety_net.py: 198 passed.
  • Full suite at this branch's head: 2253 passed, 13 skipped.
  • ruff with CI's rule set (E4, E7, E9, F) on the two Python files: clean.
  • R-1 lowers any finding whose evidence contains the word "unreachable", not only dead ACL lines #444's first row, end to end on real Batfish before this change: routing-secure with one user UNREACHABLE route assertion came out RT-003 found low.

Not tested

  • The end-to-end Batfish run was not repeated after this change. The after-state is shown by the unit tests only.
  • The ICMP, name and NEIGHBOR_UNREACHABLE cases were built as findings in tests, not produced by Batfish from a config.
  • ai/explain.py:144 reads the same dead-line marker with its own pattern. It is not changed here, and I did not check whether it has the same looseness.
  • The routing hygiene findings from RT-100 quote Batfish status values. I did not list every value Batfish can return.
  • How business-context escalation treats a finding whose severity is now different.

R-1 rates every dead ACL line low. For a dead deny that hides an
exposure: traffic someone meant to block gets through, and when the
blocker is a specific permit R-2 does not fire either. Measured in
docs/research/cvss-comparison.md section 4 (#434).

Three of these tests fail on main for that reason, 'low' == 'high':
a dead deny behind a specific permit, a dead deny behind a blanket
permit, and the finding the real access_control check builds for a
dead deny, run through refine(). The dead permit test and the case of
a missing or odd action marker pass today and must keep passing.
R-1 now lowers a dead PERMIT only. A dead DENY means traffic someone
meant to block is getting through, and access_control already grades
it high for that reason, so R-1 leaves it alone.

The finding has no structured field for the dead line's action, so
the rule reads the fixed "(action DENY). Blocked by:" marker that
access_control writes into evidence.detail, the same shape
ai/explain.py reads. Anything that says "unreachable" without that
marker keeps the old answer, low.

R-2 still does not promote a dead deny behind a blanket permit: the
blocker follows "Blocked by:", so it is a citation and R-2 strips it.
The ruleset is the doc, so it now says what the code does: R-1 lowers a
dead permit only, and a dead deny keeps the high access_control gives
it. It records why, the measured case from #434 (section 4 of the CVSS
comparison), that the old "reported separately by R-2" reasoning is
kept as history, and how the action is read: from the fixed
"(action DENY)" marker our own check writes, with anything unclear
staying low.

Also adds the one line agreed on #434: business context is our
equivalent of CVSS Environmental metrics.
…n purpose

R-1 matches "unreachable" anywhere in evidence.detail. These tests pin
what docs/severity-rules.md says it matches instead: a dead ACL line.

Seven new cases, each failing on this commit because R-1 returns "low":

  routing, expected UNREACHABLE, path succeeded    want medium (R-3)
  routing, path ended in NEIGHBOR_UNREACHABLE      want medium (R-3)
  testFilters decided by an ICMP "unreachable"     want high (kept)
  searchFilters allowed by "host-unreachable"      want high (kept)
  policy_compliance decided by "port-unreachable"  want high (kept)
  undefined reference named DROP-UNREACHABLE       want high (kept)
  unused filter named DROP-UNREACHABLE             want medium (kept)

The routing details are built from routing._evaluate() itself, so a
change to that wording reaches these tests.

One existing test is reworded, not changed in intent.
test_dead_rule_beats_the_blanket_permit_it_quotes used
"Unreachable: ... -- blocked by: ...", which no check writes. It now uses
the wording access_control.py:608-610 writes, with the dead line itself a
blanket permit, so R-2 would still match it if it ran first. It passes
before and after the fix.

The dead permit (low) and dead deny (keeps high) tests from #443 are
unchanged and pass.
R-1 now fires only when evidence.detail starts with "Unreachable line: ",
the fixed opening access_control writes for a dead line
(access_control.py:608-610). Before, it fired on the word anywhere.

What that lowered to "low", measured on #444:

  routing "Expected UNREACHABLE ..."      now medium, by R-3
  routing path ending NEIGHBOR_UNREACHABLE now medium, by R-3
  ICMP unreachable / host- / port- lines  now keep their check's severity
  names containing UNREACHABLE            now keep their check's severity

Dead lines are unchanged: a dead permit is still low, a dead deny still
keeps its check's severity, as #443 decides.

Matching the start rather than "Unreachable line:" anywhere, because a
quoted ACL line or name is config text and every other producer opens
with its own fixed words.

The docstring's FIRST ON PURPOSE paragraph said R-2 would match the
quoted BLOCKER. It would not: the blocker follows "Blocked by:", which
R-2 treats as a citation. The case that needs R-1 first is a dead line
that is itself a blanket permit, so the paragraph now says that.

tests/test_severity_rules.py: 33 passed (7 failed before this commit).
docs/severity-rules.md said R-1 matches when "the evidence describes an
unreachable / shadowed ACL line". The code matched the word anywhere,
and now matches evidence that starts with "Unreachable line: ". The
Matches line says that now, and a short paragraph records what the old
match lowered.

The "deliberately first" paragraph said R-2 would match the quoted
blocker. It would not, since the blocker follows "Blocked by:", which R-2
treats as a citation. It now names the case that needs R-1 first: a dead
line that is itself a blanket permit.

No other rule is touched.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-measured on 34e7300. This is right, and it's strictly safer than main, because R-1's match set only shrinks. #444's first row reproduces on real Batfish: RT-001 is low on main and medium here. I also ran your untested ICMP rows through Batfish, which keeps the keyword word for word. AC-001 and PC-001 are low on main and #443, and high here.

What I ran:

  • red-first is 7 failed / 26 passed
  • the suite is 2253 passed / 13 skipped live and 2212 / 54 with Batfish down
  • reverting to matching the word fails all 7

Good catch on the old ordering test: it pinned nothing on main either. Business context and the error-card backstop both behave, including with my #442 merged.

Should-fix, fine in this PR: R-1 now depends on the exact opening access_control writes, and nothing tests that for a dead permit. Changing Unreachable line: to Dead line: at access_control.py:608 passes the whole suite, with and without Batfish. A copy of #443's real-check test for a dead PERMIT, asserting low, closes it.

Nits:

  • Matching the prefix anywhere instead of at the start fails no test.
  • risk.py:114-117 cite old line numbers; they are now 458, 545, 609 and 389.
  • "Not promoted by R-2" holds only when the blanket permit is the first blocker.

Merge order matters: this PR's base is #443's branch, and the repo deletes merged branches. Please retarget this to main (gh pr edit 445 --base main) before #443 merges, or merging #443 closes this PR. Then #443, then this one straight after.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-tested in an isolated worktree at the branch's real head (34e7300), since this is stacked on #443 and I wanted the combined behaviour, not just the diff.

Confirmed #444 is genuinely fixed: test_severity_rules.py goes from 33 passed to 7 failed / 26 passed when R-1's check reverts to the old "contains 'unreachable' anywhere" match, every failure a routing/ICMP/name false positive, matching the PR body's own figures exactly.

Independently re-verified the ordering-paragraph fix, since this directly answers point 2 of Arsh's CHANGES_REQUESTED on #443. Swapped _rule_blanket_permit_is_high ahead of _rule_dead_rule_is_low: exactly 1 failed, 32 passed, and the one failure is test_dead_rule_beats_the_blanket_permit_it_quotes, the dead-line-that-is-itself-a-permit case this PR's rewritten docstring now names as the real reason for the ordering. That matches what the PR claims and genuinely answers Arsh's point, rather than just asserting it does.

Checked one thing the PR's own "Not tested" section flags but doesn't resolve: whether ai/explain.py:144's own dead-line pattern shares this same looseness. It does not -- _DEAD_RULE_DETAIL_PATTERN already requires the full Unreachable line: ... (action PERMIT|DENY). Blocked by: ... Reason: shape, not a bare substring match, so none of the false-positive cases #444 found (routing assertions, ICMP keywords, filter names) could trigger it. Worth saying plainly in case anyone was holding that open as a question.

ruff clean on both changed files.

One thing to flag rather than hold against this PR: the first full-suite run had one failure, test_draft_policy.py::test_a_draft_decided_as_it_is_today_passes_the_real_check[rtr-us5-insecure]. It passes cleanly alone, as the whole file (46/46), and a second full-suite rerun came back completely clean (2253 passed, 13 skipped, 0 failed). #445's diff doesn't touch policy drafting or Batfish setup at all, so this isn't caused by this change. It also matches, almost exactly, what #442's own PR body independently reports: two different test_draft_policy.py-adjacent tests failing intermittently under a full live run, on a different PR, confirmed not caused by that PR either after repeated re-runs. Two independent sightings of the same file being flaky under full-suite load is enough that I think it's worth its own issue now rather than waiting for a third. I'll file it.

Approving. Still blocked on #443 landing first, as the PR body says, but the diff on its own merits is correct and verified.

shubhamkataria2005 added a commit that referenced this pull request Nov 5, 2026
Option A was implemented in #443 (with #445). The table row, section 4 and
the two recommendations now say so, re-measured on the three-line ACL, with
the known deny-behind-a-deny over-rating (#447). Raised on #443's review: the
old wording would have gone stale when it lands.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 34e7300, with the one end-to-end check your "Not tested" list asked for.

#444's first row, re-run on real Batfish after the change. routing-secure, plus a user routing assertion that HQ must NOT reach the branch LAN (expected: UNREACHABLE, the path succeeds):

main                      RT-003 low     HQ can reach the branch LAN
main + #443 + #445        RT-003 medium  HQ can reach the branch LAN   (R-3, as intended)

So the end-to-end after-state matches your unit tests, not just the tests themselves.

Also checked: merged onto main with #443, resolving the one comment-block conflict as on #443, the full suite gives 2273 passed, 13 skipped and test_severity_rules.py gives 34 passed. The dead permit and dead deny cases are unchanged.

The table in your description is the clearest statement of a rule's blast radius I've seen on this project. Matching on the start of the evidence, not anywhere, is the right call for the reason you give: quoted config text can contain anything.

@ARSH871-bot

Copy link
Copy Markdown
Owner

@SamikaPerera: #445 is stacked on #443. Its base is fix/r1-dead-deny-keeps-severity, #443's branch, not main. python -m tools.stacked_prs flags it: if #443 merges first, GitHub closes #445, and a closed stacked PR cannot be reopened or retargeted, so the number and the review thread are lost (#179 lost one this way).

Please retarget it before #443 merges:

gh pr edit 445 --base main

#443 still has changes requested (mine on 4 November, Ankeet's on 4 and 5 November), so nothing is urgent, but it is a ten-second change and it is cheaper now than after the merge. Once it is on main, #445's diff will include #443's commits until #443 lands, which is expected.

If you would rather merge #445 into #443's branch first so they land as one, that also works. Just say which.

@SamikaPerera
SamikaPerera changed the base branch from fix/r1-dead-deny-keeps-severity to main November 19, 2026 11:39
@SamikaPerera

Copy link
Copy Markdown
Collaborator Author

Retargeted to main, as @ARSH871-bot asked. Its diff will include #443's commits until #443 lands. When #443 merges, the one conflict will be in tests/test_severity_rules.py: keep the comment from #443 (the corrected 'copied by hand' one), drop the old test_dead_rule_beats_the_blanket_permit_it_quotes because this PR re-adds it with the real detail wording, and drop this PR's one-line comment; I checked that this gives 34 passed. I will resolve it then and ask for a fresh look, as M-2 says.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

R-1 lowers any finding whose evidence contains the word "unreachable", not only dead ACL lines

4 participants