Repository navigation
R-1: a dead deny keeps its check's severity - #443
SamikaPerera wants to merge 8 commits into
Conversation
R-1 rates every dead ACL line low. For a dead deny that hides an exposure: traffic someone meant to block gets through, and when the blocker is a specific permit R-2 does not fire either. Measured in docs/research/cvss-comparison.md section 4 (#434). Three of these tests fail on main for that reason, 'low' == 'high': a dead deny behind a specific permit, a dead deny behind a blanket permit, and the finding the real access_control check builds for a dead deny, run through refine(). The dead permit test and the case of a missing or odd action marker pass today and must keep passing.
R-1 now lowers a dead PERMIT only. A dead DENY means traffic someone meant to block is getting through, and access_control already grades it high for that reason, so R-1 leaves it alone. The finding has no structured field for the dead line's action, so the rule reads the fixed "(action DENY). Blocked by:" marker that access_control writes into evidence.detail, the same shape ai/explain.py reads. Anything that says "unreachable" without that marker keeps the old answer, low. R-2 still does not promote a dead deny behind a blanket permit: the blocker follows "Blocked by:", so it is a citation and R-2 strips it.
The ruleset is the doc, so it now says what the code does: R-1 lowers a dead permit only, and a dead deny keeps the high access_control gives it. It records why, the measured case from #434 (section 4 of the CVSS comparison), that the old "reported separately by R-2" reasoning is kept as history, and how the action is read: from the fixed "(action DENY)" marker our own check writes, with anything unclear staying low. Also adds the one line agreed on #434: business context is our equivalent of CVSS Environmental metrics.
ARSH871-bot
left a comment
There was a problem hiding this comment.
I re-measured the body on db59b00 and on your head. The core change is right, and the case for it is stronger than the body says. The #434 ACL reproduces on real Batfish: low on main, high here. It isn't only that one case, either. vendor-asa (deny 3389 behind permit ip any any), vendor-arista and vendor-juniper each have a dead deny that main rates low and this branch rates high. On vendor-asa R-2 still fired on nothing, so "reported separately by R-2" was false even with a blanket-permit blocker. Your regex works on all four vendors' wording.
What I ran: red-first is 3 failed / 5 passed; the suite is 2246 passed / 13 skipped live and 2205 / 54 with Batfish down; removing the deny pass fails 3 tests.
What I need changed:
-
A dead deny behind a deny isn't an exposure, and this PR rates it
high. Repro:
deny tcp 10.10.10.0 0.0.0.255 host 10.20.0.5/deny tcp host 10.10.10.66 host 10.20.0.5 eq telnet/permit … eq 443/deny ip any any.- Batfish reports
Different_Action: False, and telnet from .66 is DENIED by line 1. - main gives
low; here it'shigh, whileai/explain.py:166-177on the same card says the traffic is "actually denied". severity-rules.md:60-61andrisk.py:77-80say every dead deny lets traffic through.
Options:
- (a) I add Batfish's
Different_Actionto the detail ataccess_control.py:607-611(my file), and R-1 keeps the check's severity only when the blocker's action differs. - (b) R-1 reads the blockers the way explain.py does: all-deny gives
low, and anything unrecognised keepshigh. - (c) Keep the code, say in the doc and docstring that this case is over-rated, and open an issue with this repro.
(c) is the minimum, because the doc currently says something false. I'm happy to do (a) on my side if you'd rather.
- Batfish reports
-
The ordering paragraph contradicts itself.
risk.py:85-89andseverity-rules.md:76-80say R-2 "would match the quoted blocker", then say it treats the blocker as a citation. The second is right. Moving R-2 ahead of R-1 fails nothing, here or on main. #445 fixes this, so the two should land together. -
A dangling citation.
risk.py:78andseverity-rules.md:64citedocs/research/cvss-comparison.md, which isn't on main until #434 merges. Either #434 lands first, or cite #434 itself. -
One fact, several places.
docs/document-builds/build_group_report.py:408still says a dead rule "cannot expose anything", and #434's table (line 48) and §4 (line 86) will go stale when this lands.
Nits:
- The comment at
tests/test_severity_rules.py:43overclaims: only the real-producer test catches wording drift. - Deleting the
\. blocked by:anchor fails nothing. - Consider
re.DOTALL.
A dead deny is only an exposure when the line shadowing it is a permit. Behind another deny the traffic is still blocked. R-1 reads only the dead line's action, so it keeps high for both. The docstring and the rules said every dead deny lets traffic through; they now name the case and link #447. No code change.
risk.py, severity-rules.md and one test docstring pointed at docs/research/cvss-comparison.md, which only exists once #434 merges.
The helper copies access_control's wording by hand, so tests built on it do not notice if the real wording changes. Only the test that runs the real producer does. The comment claimed all of them did.
R-1 keeps a dead deny's severity only when "(action DENY)" is followed by ". Blocked by:", the way access_control writes it. Deleting that second half of the pattern failed no test. This one fails if it is deleted.
patelankeet2
left a comment
There was a problem hiding this comment.
Verified this independently rather than just adding a second signature to Arsh's review, in an isolated worktree at the branch's real head (2563f1e).
The core change is right and I can confirm it the same way the PR body does: with _rule_dead_rule_is_low reverted to the old unconditional-low behaviour, test_severity_rules.py goes from 26 passed to 3 failed / 23 passed, and all three failures are dead-deny cases going from high back to low.
I also independently checked all four of Arsh's CHANGES_REQUESTED points against the code rather than taking his review on trust, since I'm a requested reviewer here too:
- Confirmed the dead-deny-behind-a-deny mis-rating is real. Called
_rule_dead_rule_is_low()directly with a constructed detail string matching his repro shape (a dead deny whose only blocker is itself a deny). It returnsNone(keep the check'shigh) exactly the same as a dead deny behind a specific permit (the real exposure case). The function only looks at the DEAD line's own action, never the blocker's action, so it cannot tell "already denied earlier, no real exposure" apart from "a specific permit actually lets it through." Confirmed the root cause is upstream inaccess_control.py:598(severity = "high" if dead_action == "DENY" else "medium", independent of what blocks it), which this PR's change stops masking for the one case that needed unmasking, while leaving this other case exposed too. - Confirmed the ordering paragraph's stated reason doesn't hold. Swapped
_rule_blanket_permit_is_highahead of_rule_dead_rule_is_lowin_RULESand reran: all 26 tests still passed. So "FIRST ON PURPOSE... if R-2 ran first it would match the quoted BLOCKER" isn't actually load-bearing against this branch's own test suite, matching what Arsh found. I also looked at #445: it rewrites this exact paragraph to name the real case (a dead line that IS itself a blanket permit), and re-running the same swap on #445's branch fails exactly one test, the right one. So point 2 is correctly deferred to #445 rather than needing separate work here, as long as they land together. - Confirmed the citation is dangling right now.
docs/research/cvss-comparison.mddoes not exist on this branch (lsfails). #434 is still open, unmerged. If this merges before #434,mainwould carry a citation to a file that isn't there. - Confirmed the stale claim elsewhere.
docs/document-builds/build_group_report.py:408still reads "...rate it low, because a dead rule cannot expose anything," which this PR's own change contradicts for a dead deny.
Full suite on this branch: 2246 passed, 13 skipped, matching the PR body exactly. ruff clean.
Agreeing with CHANGES_REQUESTED. Everything above is confirmed real and still unaddressed (no commits landed after Arsh's review). My vote on his option for point 1: (c) as the immediate minimum (fix the doc's false claim, open the follow-up issue), since (a) and (b) both change behaviour and deserve their own look rather than riding in on this fix.
|
Thanks Arsh. Four new commits, 1. Dead deny behind a deny. I went with (c). The code is unchanged. The docstring in 2. The ordering paragraph. Not changed here. #445 already rewrites both paragraphs ( 3. The dangling citation. 4. Other places. I did not edit Nits.
What I ran
Not tested
This push adds new work, so it needs a fresh review. |
patelankeet2
left a comment
There was a problem hiding this comment.
Correcting my own 3 November review -- it was done against a stale checkout. I fetched origin at the start of a longer review session and built this PR's worktree from that cached ref without re-fetching; four commits (72af109..139333f) had already landed between Arsh's review and mine, and I reviewed the state before them without realising it. Re-verified at the real current head (139333f) just now.
Three of Arsh's four points are genuinely resolved there:
- The dead-deny-behind-a-deny mis-rating: disclosed via docstring and docs/severity-rules.md, filed as #447 (confirmed open), which is exactly the minimum Arsh's own review said was acceptable ("(c) is the minimum"). Not behaviourally fixed, and doesn't need to be for this PR -- correctly scoped as a known, named limitation rather than silently accepted.
- The ordering paragraph: still correctly deferred to #445, which I independently verified rewrites it to name the real case. No separate action needed here as long as the two land together, per Arsh's own note.
- The dangling citation: fixed. Both risk.py and severity-rules.md now cite "PR #434" instead of a file path that doesn't exist on main.
One thing is still outstanding that none of the four new commits touched: docs/document-builds/build_group_report.py:408 still reads "...rate it low, because a dead rule cannot expose anything," which this PR's own change contradicts for a dead deny. Small, but it's exactly the "one fact in two places, only one updated" failure this project names repeatedly as its own recurring pattern, so worth closing before merge rather than leaving for someone to notice later.
Re-tested at the real head: test_severity_rules.py 27 passed (the new marker-completeness test, mutation-tested by loosening the regex to drop the "Blocked by:" half, confirmed it catches the right failure). Full suite: 2247 passed, 13 skipped, 0 failed. ruff clean.
Requesting changes for the one remaining line. Everything else checks out against the current head, not the stale one I reviewed before.
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Approving 139333f. This is the change my CVSS comparison asked for, so I re-measured it rather than just agreeing.
- Live, the #434 case (permit LAN → server, dead telnet deny,
deny ip any any):AC-002 lowon main,AC-002 highhere. That's option A doing what we agreed. - Merged onto current
main(d424e05): 2266 passed, 13 skipped, ruff clean on CI's selection. - Together with #445, since they land together: only
tests/test_severity_rules.pyconflicts, in exactly the comment block you described. Keeping #445's move of the old ordering test and your corrected comment gives 2273 passed, with 34 intest_severity_rules.py. - Arsh's points: (1) disclosed in the docstring and
severity-rules.mdand tracked as #447, the minimum he said was acceptable; (2) is #445's; (3) the citation now points at #434, not a missing file.
On Ankeet's remaining point, build_group_report.py:408: I agree it has to change, and you were right not to edit a file CODEOWNERS gives to Arsh. I don't think it should hold this PR. It's one sentence in a report generator, and leaving it doesn't make the product wrong. @ARSH871-bot, could you fix that line in your next PR? If you'd rather it land with this one, say so and I'll stay out of the way.
On my side, #434's table and section 4 now record your answer (116c97f), so that part of point 4 is done.
…eny keeps its severity Changes one line, build_group_report.py:408, which said a dead rule cannot expose anything. That no longer matches R-1 on this branch: a dead deny shadowed by a permit lets through the traffic it was meant to block, and keeps its check's severity. Ankeet and Arsh flagged the line in their reviews of this PR. CODEOWNERS gives the file to Arsh.
|
Pushed 6b33709: it changes only build_group_report.py:408, the line @patelankeet2 asked about. It said a dead rule cannot expose anything. That is true for a dead permit and for a dead deny behind a deny (#447), but not for a dead deny behind a permit, where the permit lets through the traffic the deny was meant to block. The line now says a dead permit is low and a dead deny keeps its check's severity. Measured on this head with the real functions: dead permit behind a deny or a permit goes medium to low; dead deny behind a permit or a deny stays high. @shubhamkataria2005 this is a new commit, so your approval of 139333f no longer covers the head, as M-2 says; it is a one-line docs change. @ARSH871-bot the file is yours under CODEOWNERS, so please say if you want the wording changed. |
ARSH871-bot
left a comment
There was a problem hiding this comment.
Re-reviewed 6b33709, merged onto main cceffec. My four points from 3 November are answered. Approving.
- A dead deny behind a deny. Option (c): the over-rating is stated in
risk.py:80andseverity-rules.md:63-66, with #447 open for the repro. That is what I asked for as the minimum. - The ordering paragraph. Still reads "R-2 would match the quoted blocker" before saying it is a citation (
risk.py:93-95,severity-rules.md:84-86). That is the fix in #445, so the two need to land together. #445 is still based on this branch: please retarget it tomainbefore this merges (gh pr edit 445 --base main), or it closes when this lands. - The citation now points at PR #434.
- The group report says what R-1 does now (
6b33709). #434's table already reads "high once #443 lands", so it stays true.
Nits handled too (the test comment, the anchor test). re.DOTALL wasn't taken; fine.
CI-style suite on the merge: 2460 passed, 63 skipped. Ruff clean.
@patelankeet2, your changes-requested from 5 November is still the PR's status. Worth a look before this merges.
shubhamkataria2005
left a comment
There was a problem hiding this comment.
Re-approving at 6b33709, as M-2 needs after a new commit.
The new line matches the code. On this head, _rule_dead_rule_is_low returns low only when the evidence lacks the (action DENY) marker. A dead deny keeps its check's severity whatever shadows it. The new line says exactly that, and gives the permit-shadowing case as the reason. That case is the one that matters. The deny-behind-a-deny over-rating stays where it belongs, in the docstring and #447.
Merged result (main f163125 + 6b33709): 2515 passed, 13 skipped. build_group_report.py still compiles. ruff check . over the whole repo reports 8 issues, but main alone reports the same 8. All of them are in docs/document-builds/ scripts that CI does not lint, and none is on line 408, so this commit adds nothing.
#445 is already based on main, so Arsh's point 2 is covered. The two can land together.
Answers the R-1 question raised in #434 (option A).
What changes
R-1 used to rate every dead ACL line
low. Now it lowers a dead permit only. A dead deny keeps the severity its check gives it, which foraccess_controlishigh.Why: a dead deny means traffic someone meant to block is getting through. R-1's old reason was that the exposure "is reported separately by R-2". That only holds when the line shadowing the deny is a blanket permit. Section 4 of the CVSS comparison in PR #434 measured a telnet deny shadowed by a specific permit: R-1 lowered it to
low, R-2 did not fire, and the only finding about that traffic sat at the bottom of the list.docs/severity-rules.mdsection 6 item 2 also records that R-2 has no observed effect on any fixture we have.How the action is read
A finding has no structured field for the dead line's action, and adding one would be an F-1 change. So R-1 reads the fixed marker our own
access_controlcheck writes intoevidence.detail:That is our wording, not config text, and it is the same shape
ai/explain.pyalready reads. A dead-rule finding without a clear(action DENY)marker keeps the old answer,low.A dead deny behind a blanket permit is not promoted by R-2 either: the blocker follows "Blocked by:", so R-2 treats it as a citation. It stays at the check's
high.Files
analysis/checks/risk.py:_rule_dead_rule_is_low, and its docstring.docs/severity-rules.md: R-1 rewritten to match the code, with the reason and the old reasoning kept as history. Also the one line agreed on Research: our severity rules compared with CVSS (closes #425) #434: business context is our equivalent of CVSS Environmental metrics.tests/test_severity_rules.py: new tests for a dead deny behind a specific permit, a dead deny behind a blanket permit, a dead permit, a missing or odd action marker, and the finding the realaccess_controlcheck builds, run throughrefine().No existing test changed its expectation. The two old R-1 tests both use dead permits, which are still
low.Not tested