Repository navigation
R-1 matches a dead ACL line, not the word "unreachable" (#444) - #445
SamikaPerera wants to merge 6 commits into
Conversation
R-1 rates every dead ACL line low. For a dead deny that hides an exposure: traffic someone meant to block gets through, and when the blocker is a specific permit R-2 does not fire either. Measured in docs/research/cvss-comparison.md section 4 (#434). Three of these tests fail on main for that reason, 'low' == 'high': a dead deny behind a specific permit, a dead deny behind a blanket permit, and the finding the real access_control check builds for a dead deny, run through refine(). The dead permit test and the case of a missing or odd action marker pass today and must keep passing.
R-1 now lowers a dead PERMIT only. A dead DENY means traffic someone meant to block is getting through, and access_control already grades it high for that reason, so R-1 leaves it alone. The finding has no structured field for the dead line's action, so the rule reads the fixed "(action DENY). Blocked by:" marker that access_control writes into evidence.detail, the same shape ai/explain.py reads. Anything that says "unreachable" without that marker keeps the old answer, low. R-2 still does not promote a dead deny behind a blanket permit: the blocker follows "Blocked by:", so it is a citation and R-2 strips it.
The ruleset is the doc, so it now says what the code does: R-1 lowers a dead permit only, and a dead deny keeps the high access_control gives it. It records why, the measured case from #434 (section 4 of the CVSS comparison), that the old "reported separately by R-2" reasoning is kept as history, and how the action is read: from the fixed "(action DENY)" marker our own check writes, with anything unclear staying low. Also adds the one line agreed on #434: business context is our equivalent of CVSS Environmental metrics.
…n purpose R-1 matches "unreachable" anywhere in evidence.detail. These tests pin what docs/severity-rules.md says it matches instead: a dead ACL line. Seven new cases, each failing on this commit because R-1 returns "low": routing, expected UNREACHABLE, path succeeded want medium (R-3) routing, path ended in NEIGHBOR_UNREACHABLE want medium (R-3) testFilters decided by an ICMP "unreachable" want high (kept) searchFilters allowed by "host-unreachable" want high (kept) policy_compliance decided by "port-unreachable" want high (kept) undefined reference named DROP-UNREACHABLE want high (kept) unused filter named DROP-UNREACHABLE want medium (kept) The routing details are built from routing._evaluate() itself, so a change to that wording reaches these tests. One existing test is reworded, not changed in intent. test_dead_rule_beats_the_blanket_permit_it_quotes used "Unreachable: ... -- blocked by: ...", which no check writes. It now uses the wording access_control.py:608-610 writes, with the dead line itself a blanket permit, so R-2 would still match it if it ran first. It passes before and after the fix. The dead permit (low) and dead deny (keeps high) tests from #443 are unchanged and pass.
R-1 now fires only when evidence.detail starts with "Unreachable line: ", the fixed opening access_control writes for a dead line (access_control.py:608-610). Before, it fired on the word anywhere. What that lowered to "low", measured on #444: routing "Expected UNREACHABLE ..." now medium, by R-3 routing path ending NEIGHBOR_UNREACHABLE now medium, by R-3 ICMP unreachable / host- / port- lines now keep their check's severity names containing UNREACHABLE now keep their check's severity Dead lines are unchanged: a dead permit is still low, a dead deny still keeps its check's severity, as #443 decides. Matching the start rather than "Unreachable line:" anywhere, because a quoted ACL line or name is config text and every other producer opens with its own fixed words. The docstring's FIRST ON PURPOSE paragraph said R-2 would match the quoted BLOCKER. It would not: the blocker follows "Blocked by:", which R-2 treats as a citation. The case that needs R-1 first is a dead line that is itself a blanket permit, so the paragraph now says that. tests/test_severity_rules.py: 33 passed (7 failed before this commit).
docs/severity-rules.md said R-1 matches when "the evidence describes an unreachable / shadowed ACL line". The code matched the word anywhere, and now matches evidence that starts with "Unreachable line: ". The Matches line says that now, and a short paragraph records what the old match lowered. The "deliberately first" paragraph said R-2 would match the quoted blocker. It would not, since the blocker follows "Blocked by:", which R-2 treats as a citation. It now names the case that needs R-1 first: a dead line that is itself a blanket permit. No other rule is touched.
ARSH871-bot
left a comment
There was a problem hiding this comment.
I re-measured on 34e7300. This is right, and it's strictly safer than main, because R-1's match set only shrinks. #444's first row reproduces on real Batfish: RT-001 is low on main and medium here. I also ran your untested ICMP rows through Batfish, which keeps the keyword word for word. AC-001 and PC-001 are low on main and #443, and high here.
What I ran:
- red-first is 7 failed / 26 passed
- the suite is 2253 passed / 13 skipped live and 2212 / 54 with Batfish down
- reverting to matching the word fails all 7
Good catch on the old ordering test: it pinned nothing on main either. Business context and the error-card backstop both behave, including with my #442 merged.
Should-fix, fine in this PR: R-1 now depends on the exact opening access_control writes, and nothing tests that for a dead permit. Changing Unreachable line: to Dead line: at access_control.py:608 passes the whole suite, with and without Batfish. A copy of #443's real-check test for a dead PERMIT, asserting low, closes it.
Nits:
- Matching the prefix anywhere instead of at the start fails no test.
risk.py:114-117cite old line numbers; they are now 458, 545, 609 and 389.- "Not promoted by R-2" holds only when the blanket permit is the first blocker.
Merge order matters: this PR's base is #443's branch, and the repo deletes merged branches. Please retarget this to main (gh pr edit 445 --base main) before #443 merges, or merging #443 closes this PR. Then #443, then this one straight after.
patelankeet2
left a comment
There was a problem hiding this comment.
Re-tested in an isolated worktree at the branch's real head (34e7300), since this is stacked on #443 and I wanted the combined behaviour, not just the diff.
Confirmed #444 is genuinely fixed: test_severity_rules.py goes from 33 passed to 7 failed / 26 passed when R-1's check reverts to the old "contains 'unreachable' anywhere" match, every failure a routing/ICMP/name false positive, matching the PR body's own figures exactly.
Independently re-verified the ordering-paragraph fix, since this directly answers point 2 of Arsh's CHANGES_REQUESTED on #443. Swapped _rule_blanket_permit_is_high ahead of _rule_dead_rule_is_low: exactly 1 failed, 32 passed, and the one failure is test_dead_rule_beats_the_blanket_permit_it_quotes, the dead-line-that-is-itself-a-permit case this PR's rewritten docstring now names as the real reason for the ordering. That matches what the PR claims and genuinely answers Arsh's point, rather than just asserting it does.
Checked one thing the PR's own "Not tested" section flags but doesn't resolve: whether ai/explain.py:144's own dead-line pattern shares this same looseness. It does not -- _DEAD_RULE_DETAIL_PATTERN already requires the full Unreachable line: ... (action PERMIT|DENY). Blocked by: ... Reason: shape, not a bare substring match, so none of the false-positive cases #444 found (routing assertions, ICMP keywords, filter names) could trigger it. Worth saying plainly in case anyone was holding that open as a question.
ruff clean on both changed files.
One thing to flag rather than hold against this PR: the first full-suite run had one failure, test_draft_policy.py::test_a_draft_decided_as_it_is_today_passes_the_real_check[rtr-us5-insecure]. It passes cleanly alone, as the whole file (46/46), and a second full-suite rerun came back completely clean (2253 passed, 13 skipped, 0 failed). #445's diff doesn't touch policy drafting or Batfish setup at all, so this isn't caused by this change. It also matches, almost exactly, what #442's own PR body independently reports: two different test_draft_policy.py-adjacent tests failing intermittently under a full live run, on a different PR, confirmed not caused by that PR either after repeated re-runs. Two independent sightings of the same file being flaky under full-suite load is enough that I think it's worth its own issue now rather than waiting for a third. I'll file it.
Approving. Still blocked on #443 landing first, as the PR body says, but the diff on its own merits is correct and verified.
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Approving 34e7300, with the one end-to-end check your "Not tested" list asked for.
#444's first row, re-run on real Batfish after the change. routing-secure, plus a user routing assertion that HQ must NOT reach the branch LAN (expected: UNREACHABLE, the path succeeds):
main RT-003 low HQ can reach the branch LAN
main + #443 + #445 RT-003 medium HQ can reach the branch LAN (R-3, as intended)
So the end-to-end after-state matches your unit tests, not just the tests themselves.
Also checked: merged onto main with #443, resolving the one comment-block conflict as on #443, the full suite gives 2273 passed, 13 skipped and test_severity_rules.py gives 34 passed. The dead permit and dead deny cases are unchanged.
The table in your description is the clearest statement of a rule's blast radius I've seen on this project. Matching on the start of the evidence, not anywhere, is the right call for the reason you give: quoted config text can contain anything.
|
@SamikaPerera: #445 is stacked on #443. Its base is Please retarget it before #443 merges: #443 still has changes requested (mine on 4 November, Ankeet's on 4 and 5 November), so nothing is urgent, but it is a ten-second change and it is cheaper now than after the merge. Once it is on If you would rather merge #445 into #443's branch first so they land as one, that also works. Just say which. |
|
Retargeted to main, as @ARSH871-bot asked. Its diff will include #443's commits until #443 lands. When #443 merges, the one conflict will be in tests/test_severity_rules.py: keep the comment from #443 (the corrected 'copied by hand' one), drop the old test_dead_rule_beats_the_blanket_permit_it_quotes because this PR re-adds it with the real detail wording, and drop this PR's one-line comment; I checked that this gives 34 passed. I will resolve it then and ask for a fresh look, as M-2 says. |
Based on #443, merge that first. This PR's base is #443's branch,
fix/r1-dead-deny-keeps-severity. Before #443 is merged, this PR needsgh pr edit <this PR> --base main, asCONTRIBUTING.mdsays for a stacked PR.Fixes #444.
What changes
R-1 in
analysis/checks/risk.pynow fires only when a finding's evidence starts withUnreachable line:. That is the openingaccess_controlwrites for a dead ACL line (access_control.py:608-610). Before, R-1 fired on the word "unreachable" anywhere in the evidence.Why
R-1 is meant for dead ACL lines. Matching the word also lowered findings that have nothing to do with one:
NEIGHBOR_UNREACHABLEunreachable,host-unreachableorport-unreachablelineLow sorts to the bottom of the list, so these sat below every medium.
The match is on the start of the evidence, not on
Unreachable line:anywhere. A quoted ACL line or a name is config text, and every other check opens its evidence with its own fixed words.Three commits
tests/test_severity_rules.py. The two routing details are built fromrouting._evaluate()itself. One existing test,test_dead_rule_beats_the_blanket_permit_it_quotes, used wording no check writes ("Unreachable: ... -- blocked by: ..."). It now usesaccess_control's real wording, with a deadpermit ip any anyline, so it still tests why R-1 runs before R-2.risk.py. The docstring's "first on purpose" paragraph said R-2 would match the quoted blocker. It would not, because the blocker follows "Blocked by:", which R-2 treats as a citation. The paragraph now names the case that needs R-1 first: a dead line that is itself a blanket permit.docs/severity-rules.md: R-1's "Matches" line now says what the code matches, and the "first" paragraph gets the same correction. No other rule is touched.statusis never written. Onlyseveritychanges, as before.What I had Claude Code run
tests/test_severity_rules.pyon R-1: a dead deny keeps its check's severity #443's head with only commit 1: 7 failed, 26 passed. Every failure was R-1 returninglow.risk.pychange reverted on its own: the 7 new tests failed, and the dead permit and dead deny tests passed.test_dead_rule_beats_the_blanket_permit_it_quotesfailed (1 failed, 32 passed).risk(business context scoring and wiring, CVE mapping) plustest_explain_safety_net.py: 198 passed.routing-securewith one userUNREACHABLEroute assertion came outRT-003 found low.Not tested
NEIGHBOR_UNREACHABLEcases were built as findings in tests, not produced by Batfish from a config.ai/explain.py:144reads the same dead-line marker with its own pattern. It is not changed here, and I did not check whether it has the same looseness.