Repository navigation
Research: our severity rules compared with CVSS (closes #425) - #434
Conversation
Each finding type scored with CVSS 3.1 and 4.0 at its lowest and highest plausible impact. Every scorable row spans two or three bands, because the impact depends on what sits behind the hole and a config does not say, so CVSS numbers on findings would be false precision. One measured disagreement where CVSS is right: R-1 rates a dead deny low, and when its blocker is not a blanket permit, nothing else reports the traffic it fails to block. Raised for Samika, not changed.
The interface and access-list header lines made the file match two of the no-configs-in-git signatures, which is the guard working: it cannot tell an example from a real config. The ACL lines alone carry the example.
patelankeet2
left a comment
There was a problem hiding this comment.
Reviewed by recomputing the numbers and reproducing the measured bug, not by reading the table.
Installed the exact library cited (cvss==3.2) and recomputed all 14 vectors in the appendix myself. Every one matches the doc to the decimal, both versions:
CVSS 3.1: 4.3, 8.8, 5.3, 10.0, 3.7, 6.5, 5.3, 7.5 -- all 8 match
CVSS 4.0: 5.3, 8.7, 6.9, 10.0, 6.3, 8.3 -- all 6 match
Nothing in the appendix is asserted; it's all independently reproducible, and it reproduces cleanly.
Reproduced the AC-002/R-1 measurement end to end, on a fixture I built from the doc's own example, not the one it shipped with. Three-line ACL (permit the LAN to the server, a dead deny of one host's telnet, deny ip any any), run through the real pipeline:
AC-002 found low ACL rule never takes effect in lan_in
Unreachable line: deny tcp host 10.10.10.66 host 10.20.0.5 eq telnet
(action DENY). Blocked by: permit tcp 10.10.10.0 0.0.0.255 host 10.20.0.5
Then read access_control.py directly to confirm the "high" the doc says the check assigns before risk touches it:
# A dead DENY is worse than a dead PERMIT. ... a deny that never fires lets
# through traffic someone meant to block -- a security hole.
severity = "high" if dead_action == "DENY" else "medium"That comment is quoted in the doc almost verbatim, and the live result (low) confirms R-1 is exactly what overrides it. So the measurement behind the question to Samika is real and precisely as described: the check says high, the only thing saying so is now low, and R-2 doesn't pick up the slack because the blocker here is a specific host permit, not a blanket one.
Checked the two real CVE citations. CVE-2017-3881: confirmed 9.8 (CVSS 3.1, AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H) directly from NVD, matches exactly. CVE-2016-6415: couldn't independently pin 7.5 from NVD in the time I spent on it -- it's a pure information-disclosure flaw (BENIGNCERTAIN), and 7.5 is the standard CVSS 3.x score for that shape (AV:N/AC:L/PR:N/UI:N/C:H/I:N/A:N), consistent with everything else I found about it, but I'm not calling it independently confirmed the way the other one is. Flagging that honestly rather than rounding it up to "verified."
On the actual ask to Samika: the question is well-posed and doesn't presume an answer -- either resolution (narrow R-1 to dead permits only, or make R-1's dead-deny exception conditional on the blocker actually being a blanket permit) is representable in the current rule shape, and the doc is explicit that nothing in severity-rules.md or risk.py has been touched pending his view. That's the right way to raise a measured gap in someone else's ruleset.
The other recommendations (no CVSS numbers on findings, no critical band, the cve_mapping cap staying as-is) are each argued from what a config can and can't tell you, consistent with this project's general refusal to claim more precision than the evidence supports.
No em-dashes, no AI-tool mentions.
Not approving formally, since the one substantive recommendation here is addressed to Samika and I'd rather not pre-empt his call on his own ruleset -- but everything in this document checks out, including the one number I couldn't fully pin down being clearly marked as such.
|
@patelankeet2, thanks for recomputing all 14 vectors. On the one you couldn't pin down: NVD's API confirms CVE-2016-6415 at 7.5, primary score from That's the vector you predicted from its shape. Both CVE figures in the doc are now confirmed against NVD. Agreed on leaving the R-1 question to @SamikaPerera. |
SamikaPerera
left a comment
There was a problem hiding this comment.
Thanks Shubham, and thanks Ankeet for recomputing the vectors.
My answer on R-1: A. R-1 should apply only to dead permit lines. A dead deny keeps the severity its check gives it, which is high.
Why: a dead deny means traffic someone meant to block is getting through. The reason R-1 gives, that the exposure is reported separately by R-2, does not hold in the example in section 4. docs/severity-rules.md section 6 (open question 2) also says R-2 has no observed effect on any fixture we have, so I don't want R-1 to depend on it. Option B would need the code to read the blocker out of the evidence text, which is easy to break.
The other recommendations: no CVSS numbers on findings, three levels, no critical band. Agreed. The one line saying business context is our version of CVSS Environmental metrics is fine, and I'll add it to severity-rules.md.
I'll make the change in a small PR: severity-rules.md, _rule_dead_rule_is_low in risk.py, and a test that uses the three-line ACL from the doc.
I read section 4, the recommendations, and the R-2 note in severity-rules.md section 6. I did not recompute the 14 vectors myself. I'm relying on Ankeet's recompute and on the NVD check for the two CVE figures.
risk.py, severity-rules.md and one test docstring pointed at docs/research/cvss-comparison.md, which only exists once #434 merges.
|
Pushed |
SamikaPerera
left a comment
There was a problem hiding this comment.
Re-approving 116c97f. It is docs only (one file, +14 -5), and I checked each new claim against its source.
My R-1 answer matches my review of 3 Nov (option A). The sentences about #445 and about #447 match those PRs. "AC-002 low on main, high with #443" matches Arsh's review of #443 on 4 Nov and yours of 7 Nov.
Two sentences are written as true now but only become true when #443 lands: "R-1 now lowers a dead permit only...", and recommendations 3 and 4 marked Done. Main still has if "unreachable" in _evidence(finding): at risk.py:67, and the CVSS Environmental line is on #443's branch only. The table row already says "once #443 lands", which is right.
Not blocking, but please merge this after #443, or reword those two sentences until it does. One more small thing: the doc does not say who re-measured the numbers. One clause would fix that.
Not checked: the suite (docs only, and CI passed).
Closes #425. One new file,
docs/research/cvss-comparison.md. No code, and nothing indocs/severity-rules.mdchanges: it's @SamikaPerera's, so this needs his view before any recommendation lands. That's why he's the reviewer.Method: I scored each of our finding types with CVSS 3.1 and 4.0 (Red Hat's
cvsslibrary), at the lowest and highest plausible impact. Vectors are in the appendix.Findings:
mediumis a judgement CVSS can't make.highand R-1 made itlow. R-1 says the exposure is reported separately by R-2, but R-2 needs a blanket permit, so here nothing reports the telnet that gets through.@SamikaPerera, the question: should R-1 apply only to dead
permitlines, or to a deaddenyonly when its blocker is a blanket permit? Happy either way. I'd just like the ruleset to say which.