Skip to content

R-1: a dead deny keeps its check's severity - #443

Open
SamikaPerera wants to merge 8 commits into
mainfrom
fix/r1-dead-deny-keeps-severity
Open

SamikaPerera wants to merge 8 commits into
mainfrom
fix/r1-dead-deny-keeps-severity

Conversation

@SamikaPerera

@SamikaPerera SamikaPerera commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Answers the R-1 question raised in #434 (option A).

What changes

R-1 used to rate every dead ACL line low. Now it lowers a dead permit only. A dead deny keeps the severity its check gives it, which for access_control is high.

Why: a dead deny means traffic someone meant to block is getting through. R-1's old reason was that the exposure "is reported separately by R-2". That only holds when the line shadowing the deny is a blanket permit. Section 4 of the CVSS comparison in PR #434 measured a telnet deny shadowed by a specific permit: R-1 lowered it to low, R-2 did not fire, and the only finding about that traffic sat at the bottom of the list. docs/severity-rules.md section 6 item 2 also records that R-2 has no observed effect on any fixture we have.

How the action is read

A finding has no structured field for the dead line's action, and adding one would be an F-1 change. So R-1 reads the fixed marker our own access_control check writes into evidence.detail:

Unreachable line: <line> (action DENY). Blocked by: <lines>. Reason: <reason>

That is our wording, not config text, and it is the same shape ai/explain.py already reads. A dead-rule finding without a clear (action DENY) marker keeps the old answer, low.

A dead deny behind a blanket permit is not promoted by R-2 either: the blocker follows "Blocked by:", so R-2 treats it as a citation. It stays at the check's high.

Files

  • analysis/checks/risk.py: _rule_dead_rule_is_low, and its docstring.
  • docs/severity-rules.md: R-1 rewritten to match the code, with the reason and the old reasoning kept as history. Also the one line agreed on Research: our severity rules compared with CVSS (closes #425) #434: business context is our equivalent of CVSS Environmental metrics.
  • tests/test_severity_rules.py: new tests for a dead deny behind a specific permit, a dead deny behind a blanket permit, a dead permit, a missing or odd action marker, and the finding the real access_control check builds, run through refine().

No existing test changed its expectation. The two old R-1 tests both use dead permits, which are still low.

Not tested

  • CI skips the Batfish tests, so the three-line ACL run above was local only. That config is not added to the repo as a fixture.
  • No other dead-deny case was run against Batfish.

R-1 rates every dead ACL line low. For a dead deny that hides an
exposure: traffic someone meant to block gets through, and when the
blocker is a specific permit R-2 does not fire either. Measured in
docs/research/cvss-comparison.md section 4 (#434).

Three of these tests fail on main for that reason, 'low' == 'high':
a dead deny behind a specific permit, a dead deny behind a blanket
permit, and the finding the real access_control check builds for a
dead deny, run through refine(). The dead permit test and the case of
a missing or odd action marker pass today and must keep passing.
R-1 now lowers a dead PERMIT only. A dead DENY means traffic someone
meant to block is getting through, and access_control already grades
it high for that reason, so R-1 leaves it alone.

The finding has no structured field for the dead line's action, so
the rule reads the fixed "(action DENY). Blocked by:" marker that
access_control writes into evidence.detail, the same shape
ai/explain.py reads. Anything that says "unreachable" without that
marker keeps the old answer, low.

R-2 still does not promote a dead deny behind a blanket permit: the
blocker follows "Blocked by:", so it is a citation and R-2 strips it.
The ruleset is the doc, so it now says what the code does: R-1 lowers a
dead permit only, and a dead deny keeps the high access_control gives
it. It records why, the measured case from #434 (section 4 of the CVSS
comparison), that the old "reported separately by R-2" reasoning is
kept as history, and how the action is read: from the fixed
"(action DENY)" marker our own check writes, with anything unclear
staying low.

Also adds the one line agreed on #434: business context is our
equivalent of CVSS Environmental metrics.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I re-measured the body on db59b00 and on your head. The core change is right, and the case for it is stronger than the body says. The #434 ACL reproduces on real Batfish: low on main, high here. It isn't only that one case, either. vendor-asa (deny 3389 behind permit ip any any), vendor-arista and vendor-juniper each have a dead deny that main rates low and this branch rates high. On vendor-asa R-2 still fired on nothing, so "reported separately by R-2" was false even with a blanket-permit blocker. Your regex works on all four vendors' wording.

What I ran: red-first is 3 failed / 5 passed; the suite is 2246 passed / 13 skipped live and 2205 / 54 with Batfish down; removing the deny pass fails 3 tests.

What I need changed:

  1. A dead deny behind a deny isn't an exposure, and this PR rates it high. Repro:
    deny tcp 10.10.10.0 0.0.0.255 host 10.20.0.5 / deny tcp host 10.10.10.66 host 10.20.0.5 eq telnet / permit … eq 443 / deny ip any any.

    • Batfish reports Different_Action: False, and telnet from .66 is DENIED by line 1.
    • main gives low; here it's high, while ai/explain.py:166-177 on the same card says the traffic is "actually denied".
    • severity-rules.md:60-61 and risk.py:77-80 say every dead deny lets traffic through.

    Options:

    • (a) I add Batfish's Different_Action to the detail at access_control.py:607-611 (my file), and R-1 keeps the check's severity only when the blocker's action differs.
    • (b) R-1 reads the blockers the way explain.py does: all-deny gives low, and anything unrecognised keeps high.
    • (c) Keep the code, say in the doc and docstring that this case is over-rated, and open an issue with this repro.

    (c) is the minimum, because the doc currently says something false. I'm happy to do (a) on my side if you'd rather.

  2. The ordering paragraph contradicts itself. risk.py:85-89 and severity-rules.md:76-80 say R-2 "would match the quoted blocker", then say it treats the blocker as a citation. The second is right. Moving R-2 ahead of R-1 fails nothing, here or on main. #445 fixes this, so the two should land together.

  3. A dangling citation. risk.py:78 and severity-rules.md:64 cite docs/research/cvss-comparison.md, which isn't on main until #434 merges. Either #434 lands first, or cite #434 itself.

  4. One fact, several places. docs/document-builds/build_group_report.py:408 still says a dead rule "cannot expose anything", and #434's table (line 48) and §4 (line 86) will go stale when this lands.

Nits:

  • The comment at tests/test_severity_rules.py:43 overclaims: only the real-producer test catches wording drift.
  • Deleting the \. blocked by: anchor fails nothing.
  • Consider re.DOTALL.

A dead deny is only an exposure when the line shadowing it is a permit.
Behind another deny the traffic is still blocked. R-1 reads only the dead
line's action, so it keeps high for both. The docstring and the rules
said every dead deny lets traffic through; they now name the case and
link #447. No code change.
risk.py, severity-rules.md and one test docstring pointed at
docs/research/cvss-comparison.md, which only exists once #434 merges.
The helper copies access_control's wording by hand, so tests built on it
do not notice if the real wording changes. Only the test that runs the
real producer does. The comment claimed all of them did.
R-1 keeps a dead deny's severity only when "(action DENY)" is followed by
". Blocked by:", the way access_control writes it. Deleting that second
half of the pattern failed no test. This one fails if it is deleted.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified this independently rather than just adding a second signature to Arsh's review, in an isolated worktree at the branch's real head (2563f1e).

The core change is right and I can confirm it the same way the PR body does: with _rule_dead_rule_is_low reverted to the old unconditional-low behaviour, test_severity_rules.py goes from 26 passed to 3 failed / 23 passed, and all three failures are dead-deny cases going from high back to low.

I also independently checked all four of Arsh's CHANGES_REQUESTED points against the code rather than taking his review on trust, since I'm a requested reviewer here too:

  1. Confirmed the dead-deny-behind-a-deny mis-rating is real. Called _rule_dead_rule_is_low() directly with a constructed detail string matching his repro shape (a dead deny whose only blocker is itself a deny). It returns None (keep the check's high) exactly the same as a dead deny behind a specific permit (the real exposure case). The function only looks at the DEAD line's own action, never the blocker's action, so it cannot tell "already denied earlier, no real exposure" apart from "a specific permit actually lets it through." Confirmed the root cause is upstream in access_control.py:598 (severity = "high" if dead_action == "DENY" else "medium", independent of what blocks it), which this PR's change stops masking for the one case that needed unmasking, while leaving this other case exposed too.
  2. Confirmed the ordering paragraph's stated reason doesn't hold. Swapped _rule_blanket_permit_is_high ahead of _rule_dead_rule_is_low in _RULES and reran: all 26 tests still passed. So "FIRST ON PURPOSE... if R-2 ran first it would match the quoted BLOCKER" isn't actually load-bearing against this branch's own test suite, matching what Arsh found. I also looked at #445: it rewrites this exact paragraph to name the real case (a dead line that IS itself a blanket permit), and re-running the same swap on #445's branch fails exactly one test, the right one. So point 2 is correctly deferred to #445 rather than needing separate work here, as long as they land together.
  3. Confirmed the citation is dangling right now. docs/research/cvss-comparison.md does not exist on this branch (ls fails). #434 is still open, unmerged. If this merges before #434, main would carry a citation to a file that isn't there.
  4. Confirmed the stale claim elsewhere. docs/document-builds/build_group_report.py:408 still reads "...rate it low, because a dead rule cannot expose anything," which this PR's own change contradicts for a dead deny.

Full suite on this branch: 2246 passed, 13 skipped, matching the PR body exactly. ruff clean.

Agreeing with CHANGES_REQUESTED. Everything above is confirmed real and still unaddressed (no commits landed after Arsh's review). My vote on his option for point 1: (c) as the immediate minimum (fix the doc's false claim, open the follow-up issue), since (a) and (b) both change behaviour and deserve their own look rather than riding in on this fix.

@SamikaPerera

Copy link
Copy Markdown
Collaborator Author

Thanks Arsh. Four new commits, 72af109 to 139333f. Point by point:

1. Dead deny behind a deny. I went with (c). The code is unchanged. The docstring in risk.py and R-1 in severity-rules.md now say a dead deny is an exposure only when the line shadowing it is a permit, that R-1 cannot tell from the evidence what the blocker does, and that it over-rates a deny behind a deny. Both link #447, which I filed with your repro. I reproduced it on local Batfish: line 2 has Different_Action: False, telnet from .66 is DENIED by line 1, main's R-1 gives low and this branch gives high. Your option (a) is welcome. #447 lists it, with (b) as the other way.

2. The ordering paragraph. Not changed here. #445 already rewrites both paragraphs (risk.py and severity-rules.md) with the corrected wording, so I left them alone to avoid a conflict. As you said, the two should land together.

3. The dangling citation. risk.py, severity-rules.md and one test docstring now cite section 4 of the CVSS comparison in PR #434, not docs/research/cvss-comparison.md.

4. Other places. I did not edit docs/document-builds/build_group_report.py:408. CODEOWNERS gives it to you (the * line), and its recent commits are all yours. The sentence "rate it low, because a dead rule cannot expose anything" no longer matches R-1 on this branch. I did not touch anything from #434.

Nits.

  • The comment at tests/test_severity_rules.py:43 is corrected. It now says the helper copies the wording by hand, and only the test that runs the real producer catches drift.
  • New test: (action DENY) without . Blocked by: stays low. With the \. blocked by: anchor deleted from the pattern, that test fails (1 failed, 26 passed) and no other does.
  • re.DOTALL: not added. I ran filterLineReachability on all 13 fixtures on local Batfish. There were 5 dead lines and none had a newline in Unreachable_Line, Blocking_Lines or Reason. Juniper's comes back as the term name (block-smb). If a newline did get in, the pattern would miss and R-1 would give its old low.

What I ran

Not tested

  • The suite with Batfish down, as on CI.
  • Any dead deny behind a deny other than your four-line ACL.

This push adds new work, so it needs a fresh review.

@patelankeet2 patelankeet2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correcting my own 3 November review -- it was done against a stale checkout. I fetched origin at the start of a longer review session and built this PR's worktree from that cached ref without re-fetching; four commits (72af109..139333f) had already landed between Arsh's review and mine, and I reviewed the state before them without realising it. Re-verified at the real current head (139333f) just now.

Three of Arsh's four points are genuinely resolved there:

  1. The dead-deny-behind-a-deny mis-rating: disclosed via docstring and docs/severity-rules.md, filed as #447 (confirmed open), which is exactly the minimum Arsh's own review said was acceptable ("(c) is the minimum"). Not behaviourally fixed, and doesn't need to be for this PR -- correctly scoped as a known, named limitation rather than silently accepted.
  2. The ordering paragraph: still correctly deferred to #445, which I independently verified rewrites it to name the real case. No separate action needed here as long as the two land together, per Arsh's own note.
  3. The dangling citation: fixed. Both risk.py and severity-rules.md now cite "PR #434" instead of a file path that doesn't exist on main.

One thing is still outstanding that none of the four new commits touched: docs/document-builds/build_group_report.py:408 still reads "...rate it low, because a dead rule cannot expose anything," which this PR's own change contradicts for a dead deny. Small, but it's exactly the "one fact in two places, only one updated" failure this project names repeatedly as its own recurring pattern, so worth closing before merge rather than leaving for someone to notice later.

Re-tested at the real head: test_severity_rules.py 27 passed (the new marker-completeness test, mutation-tested by loosening the regex to drop the "Blocked by:" half, confirmed it catches the right failure). Full suite: 2247 passed, 13 skipped, 0 failed. ruff clean.

Requesting changes for the one remaining line. Everything else checks out against the current head, not the stale one I reviewed before.

shubhamkataria2005 added a commit that referenced this pull request Nov 5, 2026
Option A was implemented in #443 (with #445). The table row, section 4 and
the two recommendations now say so, re-measured on the three-line ACL, with
the known deny-behind-a-deny over-rating (#447). Raised on #443's review: the
old wording would have gone stale when it lands.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 139333f. This is the change my CVSS comparison asked for, so I re-measured it rather than just agreeing.

  • Live, the #434 case (permit LAN → server, dead telnet deny, deny ip any any): AC-002 low on main, AC-002 high here. That's option A doing what we agreed.
  • Merged onto current main (d424e05): 2266 passed, 13 skipped, ruff clean on CI's selection.
  • Together with #445, since they land together: only tests/test_severity_rules.py conflicts, in exactly the comment block you described. Keeping #445's move of the old ordering test and your corrected comment gives 2273 passed, with 34 in test_severity_rules.py.
  • Arsh's points: (1) disclosed in the docstring and severity-rules.md and tracked as #447, the minimum he said was acceptable; (2) is #445's; (3) the citation now points at #434, not a missing file.

On Ankeet's remaining point, build_group_report.py:408: I agree it has to change, and you were right not to edit a file CODEOWNERS gives to Arsh. I don't think it should hold this PR. It's one sentence in a report generator, and leaving it doesn't make the product wrong. @ARSH871-bot, could you fix that line in your next PR? If you'd rather it land with this one, say so and I'll stay out of the way.

On my side, #434's table and section 4 now record your answer (116c97f), so that part of point 4 is done.

…eny keeps its severity

Changes one line, build_group_report.py:408, which said a dead rule
cannot expose anything. That no longer matches R-1 on this branch: a
dead deny shadowed by a permit lets through the traffic it was meant to
block, and keeps its check's severity. Ankeet and Arsh flagged the line
in their reviews of this PR. CODEOWNERS gives the file to Arsh.
@SamikaPerera

Copy link
Copy Markdown
Collaborator Author

Pushed 6b33709: it changes only build_group_report.py:408, the line @patelankeet2 asked about. It said a dead rule cannot expose anything. That is true for a dead permit and for a dead deny behind a deny (#447), but not for a dead deny behind a permit, where the permit lets through the traffic the deny was meant to block. The line now says a dead permit is low and a dead deny keeps its check's severity. Measured on this head with the real functions: dead permit behind a deny or a permit goes medium to low; dead deny behind a permit or a deny stays high. @shubhamkataria2005 this is a new commit, so your approval of 139333f no longer covers the head, as M-2 says; it is a one-line docs change. @ARSH871-bot the file is yours under CODEOWNERS, so please say if you want the wording changed.

@ARSH871-bot ARSH871-bot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 6b33709, merged onto main cceffec. My four points from 3 November are answered. Approving.

  1. A dead deny behind a deny. Option (c): the over-rating is stated in risk.py:80 and severity-rules.md:63-66, with #447 open for the repro. That is what I asked for as the minimum.
  2. The ordering paragraph. Still reads "R-2 would match the quoted blocker" before saying it is a citation (risk.py:93-95, severity-rules.md:84-86). That is the fix in #445, so the two need to land together. #445 is still based on this branch: please retarget it to main before this merges (gh pr edit 445 --base main), or it closes when this lands.
  3. The citation now points at PR #434.
  4. The group report says what R-1 does now (6b33709). #434's table already reads "high once #443 lands", so it stays true.

Nits handled too (the test comment, the anchor test). re.DOTALL wasn't taken; fine.

CI-style suite on the merge: 2460 passed, 63 skipped. Ruff clean.

@patelankeet2, your changes-requested from 5 November is still the PR's status. Worth a look before this merges.

@shubhamkataria2005 shubhamkataria2005 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving at 6b33709, as M-2 needs after a new commit.

The new line matches the code. On this head, _rule_dead_rule_is_low returns low only when the evidence lacks the (action DENY) marker. A dead deny keeps its check's severity whatever shadows it. The new line says exactly that, and gives the permit-shadowing case as the reason. That case is the one that matters. The deny-behind-a-deny over-rating stays where it belongs, in the docstring and #447.

Merged result (main f163125 + 6b33709): 2515 passed, 13 skipped. build_group_report.py still compiles. ruff check . over the whole repo reports 8 issues, but main alone reports the same 8. All of them are in docs/document-builds/ scripts that CI does not lint, and none is on line 408, so this commit adds nothing.

#445 is already based on main, so Arsh's point 2 is covered. The two can land together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants