Skip to content

Fix trivy failures when the vulnerability database registry rate-limits the run #8813

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nvuillam wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix trivy failures when the vulnerability database registry rate-limits the run #8813

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions .automation/build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,17 @@
"vscode": {"label": "Visual Studio Code", "url": "https://code.visualstudio.com/"},
}

# Vulnerability database settings of the trivy linters, added to the generated
# MegaLinter configuration schema from their descriptor variables
TRIVY_DB_SCHEMA_LINTERS = ["REPOSITORY_TRIVY", "REPOSITORY_TRIVY_SBOM"]
TRIVY_DB_SCHEMA_VARIABLE_TITLES = {
"DB_REPOSITORIES": "Vulnerability database repositories",
"JAVA_DB_REPOSITORIES": "Java vulnerability database repositories",
"DB_RETRY_ATTEMPTS": "Vulnerability database download attempts",
"DB_RETRY_INITIAL_DELAY": "Vulnerability database first retry delay",
"DB_RETRY_MAX_DELAY": "Vulnerability database maximum retry delay",
}

DESCRIPTORS_FOR_BUILD_CACHE = None

MAIN_DOCKERFILE = f"{REPO_HOME}/Dockerfile"
Expand Down Expand Up @@ -1728,6 +1739,14 @@ def process_type(linters_by_type, type1, type_label, linters_tables_md):
],
]
)
# Trivy linters expose vulnerability database mirrors and download
# retry settings consumed by TrivyLinter. Same narrow approach as
# betterleaks above: their descriptor variables are converted into
# configuration schema entries from their known name suffixes.
if linter.name in TRIVY_DB_SCHEMA_LINTERS:
add_in_config_schema_file(
build_trivy_db_config_schema_variables(linter, title_prefix)
)
linter_doc_md += [
f"| {linter.name}_ARGUMENTS | User custom arguments to add in linter CLI call<br/>"
f'Ex: `-s --foo "bar"` | |'
Expand Down Expand Up @@ -3331,6 +3350,32 @@ def validate_config_schema_root_x_metadata() -> None:
raise Exception("Config schema root properties missing x-keys")


def build_trivy_db_config_schema_variables(linter, title_prefix):
schema_variables = []
for variable in linter.variables:
suffix = variable["name"].replace(f"{linter.name}_", "", 1)
if suffix not in TRIVY_DB_SCHEMA_VARIABLE_TITLES:
continue
default_value = variable["default_value"]
variable_schema = {
"$id": f"#/properties/{variable['name']}",
"description": f"{linter.name}: {variable['description']}",
"title": (
f"{title_prefix}{linter.name}: "
f"{TRIVY_DB_SCHEMA_VARIABLE_TITLES[suffix]}"
),
}
if suffix.endswith("_REPOSITORIES"):
variable_schema["type"] = ["array", "string"]
variable_schema["items"] = {"type": "string"}
variable_schema["default"] = default_value.split(",")
else:
variable_schema["type"] = "integer"
variable_schema["default"] = int(default_value)
schema_variables += [[variable["name"], variable_schema]]
return schema_variables


def add_in_config_schema_file(variables):
with open(CONFIG_JSON_SCHEMA, "r", encoding="utf-8") as json_file:
json_schema = json.load(json_file)
Expand Down
5 changes: 5 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,11 @@ Note: Can be used with `oxsecurity/megalinter@beta` in your GitHub Action mega-l
- MegaLinter does not crash anymore with **`This module only works with the 'fork' start method`** right after `Processing linters on [N] parallel cores`, which made every v10.0.0 run fail unless `PARALLEL: false` was set ([#8808](https://github.com/oxsecurity/megalinter/issues/8808))
- The **`PARALLEL: false`** workaround is not needed anymore, on the main image as well as on custom flavors
- Messages logged by linters **running in parallel** are back in the console and in `megalinter.log`, including the extra output of `LOG_LEVEL: DEBUG`
- **REPOSITORY_TRIVY** and **REPOSITORY_TRIVY_SBOM** do not end the run with `--skip-db-update cannot be specified on the first run` anymore when a registry rate-limits the download of the vulnerability database ([#8807](https://github.com/oxsecurity/megalinter/issues/8807))
- trivy is now pointed at **all official database mirrors** (`mirror.gcr.io`, `ghcr.io` and `public.ecr.aws`) and uses the first one that answers
- Download retries are **spaced with increasing waits** (10s, 20s, 40s, 60s), so they no longer all land within the same rate limit minute
- The final attempt against an already downloaded database now runs only when there is one, and an explicit message tells you what to do when there is not
- New variables to tune it: `REPOSITORY_TRIVY_DB_REPOSITORIES`, `REPOSITORY_TRIVY_JAVA_DB_REPOSITORIES`, `REPOSITORY_TRIVY_DB_RETRY_ATTEMPTS`, `REPOSITORY_TRIVY_DB_RETRY_INITIAL_DELAY`, `REPOSITORY_TRIVY_DB_RETRY_MAX_DELAY`, and their `REPOSITORY_TRIVY_SBOM_` counterparts
- Fixed random **`Segmentation fault`** crashes of MegaLinter itself, which stopped the whole run with no error message ([#8733](https://github.com/oxsecurity/megalinter/issues/8733)). MegaLinter threads now get a full-size stack instead of the 128 KiB default of the Alpine images
- Fixed leaked **`git` processes** when **APPLY_FIXES** is active: one was left behind by every fixer linter, which could exhaust the available file descriptors on long runs
- Fixed **random crashes of project-mode linters** (`REPOSITORY_TRIVY`, `REPOSITORY_GRYPE`, `REPOSITORY_SYFT`…) caused by MegaLinter writing temporary ignore files inside the analyzed sources: a file appearing then disappearing while another linter walked the repository aborted its scan (`walk dir error: ... no such file or directory`). **MegaLinter now writes only in REPORT_OUTPUT_FOLDER**, never in your sources
Expand Down
99 changes: 76 additions & 23 deletions megalinter/descriptors/repository.megalinter-descriptor.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -946,6 +946,14 @@ linters:

**Note**: You can ignore specific findings by defining a [.trivyignore file](https://aquasecurity.github.io/trivy/latest/docs/configuration/filtering/#by-finding-ids) at your repository root.

**Vulnerability database download**: the registries hosting the trivy database apply rate limits, and a busy CI can be throttled with a `TOOMANYREQUESTS` error. MegaLinter mitigates it out of the box:

- trivy is pointed at all official database mirrors (`mirror.gcr.io`, `ghcr.io` and `public.ecr.aws`), and uses the first one that answers
- a download failure is retried with increasing waits, so the retries span more than one rate limit window
- if every attempt fails, trivy runs against the database downloaded by a previous run or shipped within the MegaLinter docker image, when there is one

You can tune this behaviour with `REPOSITORY_TRIVY_DB_REPOSITORIES`, `REPOSITORY_TRIVY_JAVA_DB_REPOSITORIES`, `REPOSITORY_TRIVY_DB_RETRY_ATTEMPTS`, `REPOSITORY_TRIVY_DB_RETRY_INITIAL_DELAY` and `REPOSITORY_TRIVY_DB_RETRY_MAX_DELAY`. Setting `TRIVY_DB_REPOSITORY` or `TRIVY_JAVA_DB_REPOSITORY`, or defining `db.repository` in your trivy configuration file, disables the MegaLinter mirrors and uses yours instead. Caching the trivy database folder (`TRIVY_CACHE_DIR`) between CI runs avoids the download altogether.

**Tip**: if you see errors related to files that do not exist, you can bypass them using "--skip-dirs"

Example:
Expand Down Expand Up @@ -991,28 +999,38 @@ linters:
- identifier: REPOSITORY_TRIVY_ERROR_TOOMANYREQUESTS
regex: "TOOMANYREQUESTS"
message: |-
trivy was rate-limited by GitHub Container Registry (ghcr.io) while downloading the vulnerability database.
trivy was rate-limited by a container registry while downloading its vulnerability database.
This is a registry rate limit, not a vulnerability finding.
MegaLinter already tries every official mirror (mirror.gcr.io, ghcr.io, public.ecr.aws) and retries with increasing waits before giving up.
Workarounds:
- Retry the run later (the limit is per-namespace and resets quickly).
- Authenticate pulls so requests count against your own quota. MegaLinter strips token/password env vars by default, so whitelist them for this linter in your .mega-linter.yml:
REPOSITORY_TRIVY_UNSECURED_ENV_VARIABLES:
- GITHUB_TOKEN
- TRIVY_USERNAME
- TRIVY_PASSWORD
- Mirror the trivy-db to your own registry and point trivy at it via `TRIVY_DB_REPOSITORY`.
- Cache `~/.cache/trivy` across CI runs.
- Persist the trivy cache folder between CI runs and point `TRIVY_CACHE_DIR` at it: no download, no rate limit.
- Give the retries more time in your .mega-linter.yml:
REPOSITORY_TRIVY_DB_RETRY_ATTEMPTS: 8
REPOSITORY_TRIVY_DB_RETRY_MAX_DELAY: 120
- Mirror the trivy database to a registry you control and list it first:
REPOSITORY_TRIVY_DB_REPOSITORIES:
- my.registry.example.com/trivy-db:2
- mirror.gcr.io/aquasec/trivy-db:2
- ghcr.io/aquasecurity/trivy-db:2
- Temporarily mark the linter as non-blocking by adding to your .mega-linter.yml:
DISABLE_ERRORS_LINTERS:
- REPOSITORY_TRIVY
- identifier: REPOSITORY_TRIVY_ERROR_DB_DOWNLOAD_FAILED
regex: "(failed to download (vulnerability )?(DB|database)|database download error|could not pull image)"
message: |-
trivy could not download or refresh its vulnerability database from the configured registry.
trivy could not download or refresh its vulnerability database from any of the configured registries.
Workarounds:
- Retry the run; this is often transient.
- Set `TRIVY_DB_REPOSITORY` to an alternative mirror (e.g. an internal registry or AWS ECR Public Gallery).
- Pre-populate `~/.cache/trivy` in CI.
- Persist the trivy cache folder between CI runs and point `TRIVY_CACHE_DIR` at it.
- List a mirror you control first in `REPOSITORY_TRIVY_DB_REPOSITORIES` (the value replaces the default mirrors, so keep the official ones after yours).
- Check that your runner can reach mirror.gcr.io, ghcr.io and public.ecr.aws; a corporate proxy intercepting TLS makes every mirror fail with a certificate error.
- identifier: REPOSITORY_TRIVY_ERROR_FIRST_RUN_NO_DB
regex: "--skip-db-update cannot be specified on the first run"
message: |-
trivy was asked to skip the database update while no database has ever been downloaded, which it refuses to do.
Resolutions:
- Remove `--skip-db-update` from `REPOSITORY_TRIVY_ARGUMENTS` and from `db.skip-update` in your trivy configuration file, or run trivy once with network access to populate its cache.
- When running in an air-gapped environment, download the database elsewhere and mount it into the folder pointed at by `TRIVY_CACHE_DIR`.
- identifier: REPOSITORY_TRIVY_ERROR_REGISTRY_UNAUTHORIZED
regex: "(UNAUTHORIZED: authentication required|unexpected status (code )?401|status 401 Unauthorized)"
message: |-
Expand Down Expand Up @@ -1044,6 +1062,22 @@ linters:
vscode:
- name: VSCode Trivy
url: https://marketplace.visualstudio.com/items?itemName=AquaSecurityOfficial.trivy-vulnerability-scanner
variables:
- name: REPOSITORY_TRIVY_DB_REPOSITORIES
description: OCI repositories where the vulnerability database is downloaded from, tried in order. Set to an empty value to let trivy use its own defaults
default_value: "mirror.gcr.io/aquasec/trivy-db:2,ghcr.io/aquasecurity/trivy-db:2,public.ecr.aws/aquasecurity/trivy-db:2"
- name: REPOSITORY_TRIVY_JAVA_DB_REPOSITORIES
description: OCI repositories where the Java vulnerability database is downloaded from, tried in order. Set to an empty value to let trivy use its own defaults
default_value: "mirror.gcr.io/aquasec/trivy-java-db:1,ghcr.io/aquasecurity/trivy-java-db:1,public.ecr.aws/aquasecurity/trivy-java-db:1"
- name: REPOSITORY_TRIVY_DB_RETRY_ATTEMPTS
description: Number of times trivy is run again when the vulnerability database can not be downloaded
default_value: "5"
- name: REPOSITORY_TRIVY_DB_RETRY_INITIAL_DELAY
description: Number of seconds waited before the first database download retry, doubled at each new attempt
default_value: "10"
- name: REPOSITORY_TRIVY_DB_RETRY_MAX_DELAY
description: Maximum number of seconds waited between two database download retries
default_value: "60"

# TRIVY SBOM
- class: TrivySbomLinter
Expand Down Expand Up @@ -1100,27 +1134,30 @@ linters:
- identifier: REPOSITORY_TRIVY_SBOM_ERROR_TOOMANYREQUESTS
regex: "TOOMANYREQUESTS"
message: |-
trivy-sbom was rate-limited by GitHub Container Registry (ghcr.io) while downloading the trivy DB.
trivy-sbom was rate-limited by a container registry while downloading the trivy database.
This is a registry rate limit, not an SBOM error.
MegaLinter already tries every official mirror (mirror.gcr.io, ghcr.io, public.ecr.aws) and retries with increasing waits before giving up.
Workarounds:
- Retry the run later.
- Authenticate pulls. MegaLinter strips token/password env vars by default, so whitelist them for this linter in your .mega-linter.yml:
REPOSITORY_TRIVY_SBOM_UNSECURED_ENV_VARIABLES:
- GITHUB_TOKEN
- TRIVY_USERNAME
- TRIVY_PASSWORD
- Mirror trivy-db to your own registry via `TRIVY_DB_REPOSITORY`.
- Persist the trivy cache folder between CI runs and point `TRIVY_CACHE_DIR` at it: no download, no rate limit.
- Give the retries more time in your .mega-linter.yml:
REPOSITORY_TRIVY_SBOM_DB_RETRY_ATTEMPTS: 8
REPOSITORY_TRIVY_SBOM_DB_RETRY_MAX_DELAY: 120
- Mirror the trivy database to a registry you control and list it first:
REPOSITORY_TRIVY_SBOM_DB_REPOSITORIES:
- my.registry.example.com/trivy-db:2
- mirror.gcr.io/aquasec/trivy-db:2
- ghcr.io/aquasecurity/trivy-db:2
- Temporarily mark the linter as non-blocking by adding to your .mega-linter.yml:
DISABLE_ERRORS_LINTERS:
- REPOSITORY_TRIVY_SBOM
- identifier: REPOSITORY_TRIVY_SBOM_ERROR_DB_DOWNLOAD_FAILED
regex: "(failed to download (vulnerability )?(DB|database)|database download error)"
message: |-
trivy-sbom could not download or refresh its vulnerability database.
trivy-sbom could not download or refresh its vulnerability database from any of the configured registries.
Workarounds:
- Retry the run; this is often transient.
- Set `TRIVY_DB_REPOSITORY` to an alternative mirror.
- Pre-populate `~/.cache/trivy` in CI.
- Persist the trivy cache folder between CI runs and point `TRIVY_CACHE_DIR` at it.
- List a mirror you control first in `REPOSITORY_TRIVY_SBOM_DB_REPOSITORIES` (the value replaces the default mirrors, so keep the official ones after yours).
test_folder: trivy
examples:
- "trivy fs --format cyclonedx ."
Expand All @@ -1141,6 +1178,22 @@ linters:
vscode:
- name: VSCode Trivy
url: https://marketplace.visualstudio.com/items?itemName=AquaSecurityOfficial.trivy-vulnerability-scanner
variables:
- name: REPOSITORY_TRIVY_SBOM_DB_REPOSITORIES
description: OCI repositories where the vulnerability database is downloaded from, tried in order. Set to an empty value to let trivy use its own defaults
default_value: "mirror.gcr.io/aquasec/trivy-db:2,ghcr.io/aquasecurity/trivy-db:2,public.ecr.aws/aquasecurity/trivy-db:2"
- name: REPOSITORY_TRIVY_SBOM_JAVA_DB_REPOSITORIES
description: OCI repositories where the Java vulnerability database is downloaded from, tried in order. Set to an empty value to let trivy use its own defaults
default_value: "mirror.gcr.io/aquasec/trivy-java-db:1,ghcr.io/aquasecurity/trivy-java-db:1,public.ecr.aws/aquasecurity/trivy-java-db:1"
- name: REPOSITORY_TRIVY_SBOM_DB_RETRY_ATTEMPTS
description: Number of times trivy is run again when the vulnerability database can not be downloaded
default_value: "5"
- name: REPOSITORY_TRIVY_SBOM_DB_RETRY_INITIAL_DELAY
description: Number of seconds waited before the first database download retry, doubled at each new attempt
default_value: "10"
- name: REPOSITORY_TRIVY_SBOM_DB_RETRY_MAX_DELAY
description: Maximum number of seconds waited between two database download retries
default_value: "60"

# TRUFFLEHOG
- class: TruffleHogLinter
Expand Down
Loading
Loading