Skip to content

Commit 58318b0

Browse files
NucleonGodXNucleonGodX
authored
Integration of Clarity compliance mechanism (#1705)
* integrating the clarity-compliance in UI Signed-off-by: NucleonGodX <racerpro41@gmail.com> * minor change Signed-off-by: NucleonGodX <racerpro41@gmail.com> * have compliance alerts in the single panel Signed-off-by: NucleonGodX <racerpro41@gmail.com> * add updated check-compliance Signed-off-by: NucleonGodX <racerpro41@gmail.com> * add documentation and support api endpoint for clarity compliance Signed-off-by: NucleonGodX <racerpro41@gmail.com> * fix tests Signed-off-by: NucleonGodX <racerpro41@gmail.com> * create separate method to retrieve compliance alert Signed-off-by: NucleonGodX <racerpro41@gmail.com> * used the new refactored policy code Signed-off-by: NucleonGodX <racerpro41@gmail.com> * add test for api and replace clarity_compliance with license_clarity_compliance everywhere Signed-off-by: NucleonGodX <racerpro41@gmail.com> * suggestions applied Signed-off-by: NucleonGodX <racerpro41@gmail.com> * fix tests Signed-off-by: NucleonGodX <racerpro41@gmail.com> --------- Signed-off-by: NucleonGodX <racerpro41@gmail.com>
1 parent cb2d65e commit 58318b0

File tree

13 files changed

+262
-9
lines changed

13 files changed

+262
-9
lines changed

‎docs/policies.rst‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,43 @@ structure similar to the following:
5454
- ``warning`` — Use with caution; the license may have some restrictions.
5555
- ``error`` — The license is prohibited or incompatible with your policy.
5656

57+
Creating Clarity Thresholds Files
58+
---------------------------------
59+
60+
A valid clarity thresholds file is required to **enable license clarity compliance features**.
61+
62+
The clarity thresholds file, by default named ``policies.yml``, is a **YAML file** with a
63+
structure similar to the following:
64+
65+
.. code-block:: yaml
66+
67+
license_clarity_thresholds:
68+
91: ok
69+
80: warning
70+
0: error
71+
72+
- In the example above, the keys ``91``, ``80``, and ``0`` are integer threshold values
73+
representing **minimum clarity scores**.
74+
- The values ``error``, ``warning``, and ``ok`` are the **compliance alert levels** that
75+
will be triggered if the project's license clarity score meets or exceeds the
76+
corresponding threshold.
77+
- The thresholds must be listed in **strictly descending order**.
78+
79+
How it works:
80+
81+
- If the clarity score is **91 or above**, the alert is **``ok``**.
82+
- If the clarity score is **80 to 90**, the alert is **``warning``**.
83+
- If the clarity score is **below 80**, the alert is **``error``**.
84+
85+
You can adjust the threshold values and alert levels to match your organization's
86+
compliance requirements.
87+
88+
Accepted values for the alert level:
89+
90+
- ``ok``
91+
- ``warning``
92+
- ``error``
93+
5794
App Policies
5895
------------
5996

@@ -115,7 +152,7 @@ REST API
115152
--------
116153

117154
For more details on retrieving compliance data through the REST API, see the
118-
:ref:`rest_api_compliance` section.
155+
:ref:`rest_api_compliance` section and :ref:`rest_api_clarity_compliance` section.
119156

120157
Command Line Interface
121158
----------------------

‎docs/rest-api.rst‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -493,6 +493,31 @@ Data:
493493
}
494494
}
495495
496+
.. _rest_api_license_clarity_compliance:
497+
498+
License Clarity Compliance
499+
^^^^^^^^^^^^^^^^^^^^^^^^^^
500+
501+
This action returns the **license clarity compliance alert** for a project.
502+
503+
The license clarity compliance alert is a single value (``ok``, ``warning``, or ``error``)
504+
that summarizes the project's **license clarity status**, based on the thresholds defined in
505+
the ``policies.yml`` file.
506+
507+
``GET /api/projects/6461408c-726c-4b70-aa7a-c9cc9d1c9685/license_clarity_compliance/``
508+
509+
Data:
510+
- ``license_clarity_compliance_alert``: The overall license clarity compliance alert
511+
for the project.
512+
513+
Possible values: ``ok``, ``warning``, ``error``.
514+
515+
.. code-block:: json
516+
517+
{
518+
"license_clarity_compliance_alert": "warning"
519+
}
520+
496521
Reset
497522
^^^^^
498523

‎docs/tutorial_license_policies.rst‎

Lines changed: 48 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,51 @@ detected license, and computed at the codebase resource level, for example:
8383
"[...]": "[...]"
8484
}
8585
86+
License Clarity Thresholds and Compliance
87+
-----------------------------------------
88+
89+
ScanCode.io also supports **license clarity thresholds**, allowing you to enforce
90+
minimum standards for license detection quality in your codebase. This is managed
91+
through the ``license_clarity_thresholds`` section in your ``policies.yml`` file.
92+
93+
Defining Clarity Thresholds
94+
---------------------------
95+
96+
Add a ``license_clarity_thresholds`` section to your ``policies.yml`` file, for example:
97+
98+
.. code-block:: yaml
99+
100+
license_clarity_thresholds:
101+
91: ok
102+
80: warning
103+
0: error
104+
105+
106+
License Clarity Compliance in Results
107+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
108+
109+
When you run a pipeline with clarity thresholds defined in your ``policies.yml``,
110+
the computed license clarity compliance alert is included in the project's ``extra_data`` field.
111+
112+
For example:
113+
114+
.. code-block:: json
115+
116+
"extra_data": {
117+
"md5": "d23df4a4",
118+
"sha1": "3e9b61cc98c",
119+
"size": 3095,
120+
"sha256": "abacfc8bcee59067",
121+
"sha512": "208f6a83c83a4c770b3c0",
122+
"filename": "cuckoo_filter-1.0.6.tar.gz",
123+
"sha1_git": "3fdb0f82ad59",
124+
"license_clarity_compliance_alert": "error"
125+
}
126+
127+
The ``license_clarity_compliance_alert`` value (e.g., ``"error"``, ``"warning"``, or ``"ok"``)
128+
is computed automatically based on the thresholds you configured and reflects the
129+
overall license clarity status of the scanned codebase.
130+
86131
Run the ``check-compliance`` command
87132
------------------------------------
88133

@@ -95,7 +140,7 @@ in the project:
95140
96141
.. code-block:: bash
97142
98-
4 compliance issues detected on this project.
143+
5 compliance issues detected on this project.
99144
[packages]
100145
> ERROR: 3
101146
pkg:pypi/cuckoo-filter@.
@@ -104,6 +149,8 @@ in the project:
104149
[resources]
105150
> ERROR: 1
106151
cuckoo_filter-1.0.6.tar.gz-extract/cuckoo_filter-1.0.6/README.md
152+
[license clarity]
153+
> ERROR
107154
108155
.. tip::
109156
In case of compliance alerts, the command returns a non-zero exit code which

‎scanpipe/api/views.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -481,6 +481,22 @@ def compliance(self, request, *args, **kwargs):
481481
compliance_alerts = get_project_compliance_alerts(project, fail_level)
482482
return Response({"compliance_alerts": compliance_alerts})
483483

484+
@action(detail=True, methods=["get"])
485+
def license_clarity_compliance(self, request, *args, **kwargs):
486+
"""
487+
Retrieve the license clarity compliance alert for a project.
488+
489+
This endpoint returns the license clarity compliance alert stored in the
490+
project's extra_data.
491+
492+
Example:
493+
GET /api/projects/{project_id}/license_clarity_compliance/
494+
495+
"""
496+
project = self.get_object()
497+
clarity_alert = project.get_license_clarity_compliance_alert()
498+
return Response({"license_clarity_compliance_alert": clarity_alert})
499+
484500

485501
class RunViewSet(mixins.RetrieveModelMixin, viewsets.GenericViewSet):
486502
"""Add actions to the Run viewset."""

‎scanpipe/management/commands/check-compliance.py‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -74,16 +74,25 @@ def check_compliance(self, fail_level):
7474
len(issues) for model in alerts.values() for issues in model.values()
7575
)
7676

77-
if count and self.verbosity > 0:
78-
self.stderr.write(f"{count} compliance issues detected.")
77+
clarity_alert = self.project.get_license_clarity_compliance_alert()
78+
has_clarity_issue = clarity_alert not in (None, "ok")
79+
80+
total_issues = count + (1 if has_clarity_issue else 0)
81+
82+
if total_issues and self.verbosity > 0:
83+
self.stderr.write(f"{total_issues} compliance issues detected.")
7984
for label, model in alerts.items():
8085
self.stderr.write(f"[{label}]")
8186
for severity, entries in model.items():
8287
self.stderr.write(f" > {severity.upper()}: {len(entries)}")
8388
if self.verbosity > 1:
8489
self.stderr.write(" " + "\n ".join(entries))
8590

86-
return count > 0
91+
if has_clarity_issue:
92+
self.stderr.write("[license clarity]")
93+
self.stderr.write(f" > {clarity_alert.upper()}")
94+
95+
return total_issues > 0
8796

8897
def check_vulnerabilities(self):
8998
packages = self.project.discoveredpackages.vulnerable_ordered()

‎scanpipe/models.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1544,6 +1544,13 @@ def get_policies_dict(self):
15441544

15451545
return scanpipe_app.policies
15461546

1547+
def get_license_clarity_compliance_alert(self):
1548+
"""
1549+
Return the license clarity compliance alert value for the project,
1550+
or None if not set.
1551+
"""
1552+
return self.extra_data.get("license_clarity_compliance_alert")
1553+
15471554
def get_license_policy_index(self):
15481555
"""Return the policy license index for this project instance."""
15491556
if policies_dict := self.get_policies_dict():

‎scanpipe/pipes/license_clarity.py‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -156,10 +156,29 @@ def load_clarity_thresholds_from_file(file_path):
156156
file_path = Path(file_path)
157157

158158
if not file_path.exists():
159-
return None
159+
return
160160

161161
try:
162162
yaml_content = file_path.read_text(encoding="utf-8")
163163
return load_clarity_thresholds_from_yaml(yaml_content)
164164
except (OSError, UnicodeDecodeError) as e:
165165
raise ValidationError(f"Error reading file {file_path}: {e}")
166+
167+
168+
def get_project_clarity_thresholds(project):
169+
"""
170+
Get clarity thresholds for a project using the unified policy loading logic.
171+
172+
Returns:
173+
ClarityThresholdsPolicy or None: Policy object if thresholds are configured
174+
175+
"""
176+
policies_dict = project.get_policies_dict()
177+
if not policies_dict:
178+
return
179+
180+
clarity_thresholds = policies_dict.get("license_clarity_thresholds")
181+
if not clarity_thresholds:
182+
return
183+
184+
return ClarityThresholdsPolicy(clarity_thresholds)

‎scanpipe/pipes/scancode.py‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@
6060
from scanpipe.models import DiscoveredDependency
6161
from scanpipe.models import DiscoveredPackage
6262
from scanpipe.pipes import flag
63+
from scanpipe.pipes.license_clarity import get_project_clarity_thresholds
6364

6465
logger = logging.getLogger("scanpipe.pipes")
6566

@@ -1134,7 +1135,10 @@ def make_results_summary(project, scan_results_location):
11341135
Extract selected sections of the Scan results, such as the `summary`
11351136
`license_clarity_score`, and `license_matches` related data.
11361137
The `key_files` are also collected and injected in the `summary` output.
1138+
Additionally, store license_clarity_compliance_alert in project's extra_data.
11371139
"""
1140+
import json
1141+
11381142
from scanpipe.api.serializers import CodebaseResourceSerializer
11391143
from scanpipe.api.serializers import DiscoveredPackageSerializer
11401144

@@ -1167,4 +1171,12 @@ def make_results_summary(project, scan_results_location):
11671171
DiscoveredPackageSerializer(package).data for package in key_files_packages_qs
11681172
]
11691173

1174+
clarity_score = summary.get("license_clarity_score", {}).get("score")
1175+
if clarity_score is not None:
1176+
clarity_policy = get_project_clarity_thresholds(project)
1177+
if clarity_policy:
1178+
alert = clarity_policy.get_alert_for_score(clarity_score)
1179+
summary["license_clarity_compliance_alert"] = alert
1180+
1181+
project.update_extra_data({"license_clarity_compliance_alert": alert})
11701182
return summary

‎scanpipe/templates/scanpipe/panels/project_compliance.html‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
{% load humanize %}
2-
{% if compliance_alerts %}
2+
{% if compliance_alerts or license_clarity_compliance_alert %}
33
<div class="column is-half">
44
<nav id="compliance-panel" class="panel is-dark">
55
<p class="panel-heading">
66
Compliance alerts
77
</p>
8-
{% for model_name, model_alerts in compliance_alerts.items %}
8+
{% for model_name, model_alerts in compliance_alerts.items %}
99
<div class="panel-block">
1010
<span class="pr-1">
1111
{{ model_name|title }}
@@ -19,6 +19,20 @@
1919
{% endfor %}
2020
</div>
2121
{% endfor %}
22+
{% if license_clarity_compliance_alert %}
23+
<div class="panel-block">
24+
<span class="pr-1">
25+
License clarity
26+
</span>
27+
<span class="tag is-rounded ml-1
28+
{% if license_clarity_compliance_alert == 'error' %}is-danger
29+
{% elif license_clarity_compliance_alert == 'warning' %}is-warning
30+
{% elif license_clarity_compliance_alert == 'ok' %}is-success
31+
{% else %}is-light{% endif %}">
32+
{{ license_clarity_compliance_alert|title }}
33+
</span>
34+
</div>
35+
{% endif %}
2236
</nav>
2337
</div>
24-
{% endif %}
38+
{% endif %}

‎scanpipe/tests/test_api.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1252,6 +1252,24 @@ def test_scanpipe_api_project_action_compliance(self):
12521252
}
12531253
self.assertDictEqual(expected, response.data)
12541254

1255+
def test_scanpipe_api_project_action_license_clarity_compliance(self):
1256+
project = make_project()
1257+
url = reverse("project-license-clarity-compliance", args=[project.uuid])
1258+
1259+
response = self.csrf_client.get(url)
1260+
expected = {"license_clarity_compliance_alert": None}
1261+
self.assertEqual(expected, response.data)
1262+
1263+
project.update_extra_data({"license_clarity_compliance_alert": "ok"})
1264+
response = self.csrf_client.get(url)
1265+
expected = {"license_clarity_compliance_alert": "ok"}
1266+
self.assertEqual(expected, response.data)
1267+
1268+
project.update_extra_data({"license_clarity_compliance_alert": "error"})
1269+
response = self.csrf_client.get(url)
1270+
expected = {"license_clarity_compliance_alert": "error"}
1271+
self.assertEqual(expected, response.data)
1272+
12551273
def test_scanpipe_api_serializer_get_model_serializer(self):
12561274
self.assertEqual(
12571275
DiscoveredPackageSerializer, get_model_serializer(DiscoveredPackage)

0 commit comments

Comments
 (0)