Official SBSeg 2026 artifact-evaluation seals awarded to this artifact (Main Track): Available, Functional, Sustainable and Reproducible. Official results. Seal artwork by the SBSeg Artifact Evaluation Committee.
Published in the Anais do XXVI Simpósio Brasileiro de Cibersegurança (SBSeg 2026), pp. 1406-1412: SBC OpenLib.
Artifact for the SBSeg 2026 paper A Uniform Random-Sample Security Measurement of Docker Hub Images. It measures the security posture of a typical Docker Hub image, drawing repositories uniformly at random instead of by popularity, and scanning each one with six open-source tools. 2,879 images were analyzed and 94.4% carry a vulnerability the scanners rate critical, so the high rates reported by earlier studies are not an effect of looking only at popular images. Hand-labeling 1,100 secret detections found 99.5% of them to be false positives, which is why the paper reports a validated rate rather than the detector's raw output.
Paper: A Uniform Random-Sample Security Measurement of Docker Hub Images, SBSeg 2026 Main Track. Artifact evaluation follows the official submission and review instructions.
For the artifact evaluation, this README is the only file you need to read. The other Markdown files in the repository are complementary: they document internals and go deeper than the review requires.
Abstract. Registry-scale security measurements of Docker Hub define their samples by repository class, category, popularity, or estimated reach. Their findings therefore describe selected subsets rather than the registry-wide posture of a typical scannable
latestimage. We estimate that posture from 2,879 such images obtained by uniformly sampling 12.7 million repositories. Six open-source scanners match the pipeline used on our exposure-ranked sample, where exposure is the number of pulls reached directly or through reused layers. 94.4% of random images carry a critical vulnerability, compared with 93.4% of exposure-ranked images. Tool choice strongly affects the reported posture: 70.5% of image-CVE pairs appear in only one of three scanners. TruffleHog reports secrets in 82.4% of images, but hand-labeling shows that 99.5% of these detections are false positives. Exposure-based prioritization identifies vulnerabilities likely to affect more users, but the selected images are not more vulnerable than typical scannablelatestimages.
| Section | Purpose |
|---|---|
| Considered seals | The four seals and why each one holds |
| Basic information | Reference environment and resource requirements |
| Dependencies | Languages, packages, host tools and scanner versions, with the released dataset |
| Security concerns | Safe handling of scanner reports and container images |
| Installation | Required setup |
| Minimal test | Fast offline functionality check |
| Experiments | One-command reproduction of the paper results |
| Cleaning up | One command removes what a run created |
| LICENSE | Artifact license |
| How to cite | Paper reference and machine-readable CITATION.cff |
The repository is organized as follows:
| Path | Contents |
|---|---|
reproduce.sh |
The single reproduction entry point: precomputed and analyze |
analysis/ |
The analyses, the committed numeric outputs, the manual labels, and the figure generators. analysis/README.md maps every file there to its single responsibility, its inputs and its outputs |
data/ |
The canonical 4,800-repository draw |
expected/ |
paper_values.json: each of the 66 asserted numbers, with the section that states it |
docs/ |
Methodological caveats and REPRODUCIBILITY_REPORT.md |
cleanup.sh |
Removes everything a run created |
The seals considered are: Available (SeloD), Functional (SeloF), Sustainable (SeloS) and Reproducible (SeloR).
- Available (SeloD): public source code, data, manual labels, and MIT license.
- Functional (SeloF): the offline minimal test runs from a clean checkout in less than one second.
- Sustainable (SeloS): one script per concern in
analysis/, each writing a documented JSON, JSONL, or TSV output, and each mapped to its responsibility inanalysis/README.md;expected/paper_values.jsonmaps each of the 66 numbers the paper asserts to the section or table that states it, so every claim can be located in the artifact. - Reproducible (SeloR): one command downloads the checksum-verified database, recomputes the analyses, regenerates the figures, and checks 66 paper values.
| Item | Requirement / reference environment |
|---|---|
| Operating system | Linux x86-64; tested on Ubuntu |
| Reference machine | AMD Ryzen 5 8600G, 12 logical CPUs, 32 GB RAM |
| Python | 3.10 to 3.12; 3.12.3 used for the study. NumPy 1.26.4 and Matplotlib 3.8.4 publish no wheels for 3.13 or newer, so a newer interpreter cannot install them |
| System package | python3-venv (Debian and Ubuntu package it separately from python3) |
| Host tools | sha256sum and df from GNU coreutils, used by reproduce.sh to verify the dataset checksums and to check free disk before decompressing. The download itself uses Python's standard library, so no curl or wget is needed |
| Fonts | fonts-liberation, to typeset the figures as published. Without it matplotlib falls back to DejaVu Serif and the labels render wider; every number is unaffected |
| GPU | Not required |
| Minimal test | 15 MB peak RAM, negligible disk, 0.02 s measured |
| Offline figure reproduction | 88 MB peak RAM, less than 10 MB output, 1.6 s measured |
| Full database reproduction | 4 GB RAM recommended, 11 GB free disk. Budget over an hour: 30 to 60 min measured on a Ryzen 7 9700X (16 threads, 59 GB RAM), a fast desktop CPU, and longer on slower hardware, since the cost is dominated by the first read of the 10.3 GB database from disk. A re-run with the file already in the page cache takes 4 to 7 min |
Times are stated with the machine that produced them, because they are hardware- and cache-dependent: the minimal test and the offline figure path were measured on the reference machine above, the full reproduction on the Ryzen 7 9700X noted in its row. Download time depends on the evaluator's network connection.
On the host, the evaluator path needs only git, bash, coreutils and a python3; every command in this README is ./reproduce.sh <mode>. The pinned wheels cover CPython 3.10 to 3.12, and bootstrap.sh fetches a supported interpreter itself when the system one is outside that range. make is a developer convenience that mirrors the same modes and is not required.
Every version below is the one actually declared in this repository; the last column says where it is declared, so each row can be checked against the file that pins it.
| Language / runtime | Version | What it is used for | Declared in |
|---|---|---|---|
| Python (CPython) | 3.10 to 3.12; 3.12.3 used for the study | every analysis and script in analysis/ and scripts/ |
scripts/bootstrap.sh (SUPPORTED="3.12 3.11 3.10"), requirements.txt |
| Bash | no minimum pinned; the scripts declare #!/usr/bin/env bash |
reproduce.sh, cleanup.sh, scripts/bootstrap.sh |
the shebang of each script |
| SQL (SQLite) | whatever SQLite the interpreter links; 3.45.1 with CPython 3.12.3 | read-only queries over the 10.3 GB reports database | Python's standard library; bootstrap.sh prints the linked version |
| GNU Make | not pinned; optional | Makefile targets that mirror ./reproduce.sh |
— |
Installed into .venv by scripts/bootstrap.sh; nothing is installed globally.
| Package | Version | What it is used for | Declared in |
|---|---|---|---|
| Matplotlib | 3.8.4 | draws the three paper figures (analysis/make_figs.py, analysis/analyze_extra.py, style in analysis/figstyle.py) |
requirements.txt |
| NumPy | 1.26.4 | the array math behind those figures | requirements.txt |
| zstandard | 0.25.0 | stream-decompresses the released bl_snap.db.zst (226 MB) to bl_snap.db (10.3 GB) |
requirements.txt |
Python standard library (sqlite3, json, csv, random, statistics, hashlib, urllib) |
ships with the interpreter | reads the reports database, computes every statistic, downloads the dataset (so no curl or wget is needed) |
no install; noted in requirements.txt |
| PyMongo | not pinned and not installed | only scripts/sample_repos.py, which draws a new sample from the private crawl's MongoDB. Outside the evaluator path |
intentionally omitted, with the reason, in requirements.txt |
Matplotlib's own dependencies (contourpy, cycler, fonttools, kiwisolver, packaging, pillow, pyparsing, python-dateutil, six) are not pinned: pip resolves them at install time, and no result depends on which version it picks. Only the three packages above affect the figures, so only those are pinned.
| Tool | Version | What it is used for | Required? |
|---|---|---|---|
git |
not pinned | cloning the repository | yes |
sha256sum, df (GNU coreutils) |
not pinned | verifying the dataset checksums and the free disk before decompressing, in reproduce.sh |
yes |
python3-venv (Debian and Ubuntu package it separately) |
matches the interpreter | creating .venv; bootstrap.sh prints the exact install command when it is missing |
yes, on Debian and Ubuntu |
| uv | not pinned; the installer fetches the current release into .uv/ |
fetching a supported interpreter, and only when the system Python is outside 3.10 to 3.12; also required by ./reproduce.sh scan |
only then |
fonts-liberation |
not pinned | typesetting the figures in the paper's serif face; without it Matplotlib falls back to DejaVu Serif and every number is unaffected | no |
| Docker | not pinned | only ./reproduce.sh scan, which re-executes the scanners and is outside the evaluator path |
no |
The measurement itself ran through the separate ChimangoScan pipeline. Their versions are deliberately not stated as numbers: the scanner images were referenced by the floating latest tag and their vulnerability databases were fetched at scan time, so no version was pinned or recorded, and inventing one here would misrepresent the campaign. The released reports are therefore the record of what was measured; a new scan may resolve newer tools and newer vulnerability data. This is also listed as a known limitation in docs/REPRODUCIBILITY_REPORT.md.
| Scanner | Version | What it contributed |
|---|---|---|
| Trivy | floating latest at scan time (not recorded) |
known vulnerabilities in the installed packages |
| Grype | floating latest at scan time (not recorded) |
known vulnerabilities, independently of Trivy |
| OSV-Scanner | floating latest at scan time (not recorded) |
known vulnerabilities from the OSV database |
| Syft | floating latest at scan time (not recorded) |
the package inventory (SBOM), and with it the base-OS distribution |
| Dockle | floating latest at scan time (not recorded) |
image-hardening (CIS) findings |
| TruffleHog | floating latest at scan time (not recorded), run in unverified mode |
strings that look like embedded credentials, the input to the manual labeling |
The full input is the released bl_snap.db.zst dataset:
| File | Download | Decompressed | SHA-256 verification |
|---|---|---|---|
bl_snap.db.zst |
226 MB | 10.3 GB | Automatic in reproduce.sh |
The script downloads the database from its Zenodo record (10.5281/zenodo.22666268; the same file is attached to the repository's dataset-v1 release), verifies the compressed and decompressed SHA-256 hashes, and stores it under data/. Download and decompression use the isolated Python environment; the evaluator does not need system-wide packages or a manual dataset download.
- The committed sample and manual-review files contain only redacted values and truncated hashes. Do not attempt to validate any suspected credential against a live service.
- The released SQLite database contains raw third-party scanner reports and may include sensitive detections. Keep it local and do not redistribute extracted values.
- The reproduction commands only read released reports; they do not execute software from the measured images.
- Re-running the scanners is outside the evaluator path and should be done only on disposable infrastructure because it downloads and unpacks arbitrary public images.
Clone the repository and run the bootstrap script. It selects a Python version the pinned dependencies support, creates the isolated environment, installs into it, and verifies the imports:
git clone https://github.com/ChimangoScan/chimango-baseline.git && cd chimango-baseline && ./scripts/bootstrap.shIf the system Python is newer than 3.12, the script fetches a supported one with uv, downloading uv into the clone first when it is not already installed. That path needs no root and works on any distribution; nothing outside the clone is modified.
If a system package is missing instead, the script stops and prints the single command to run for the distribution it detects (apt, dnf, pacman or zypper). Re-run the script afterwards.
Nothing is installed globally. reproduce.sh automatically uses .venv/bin/python when the environment exists; activation is unnecessary.
Expected time: approximately 30 to 90 s, depending on the package index and network; 8 s measured on the reference machine with a warm package cache.
Expected resources: less than 500 MB RAM and 500 MB disk.
Run the offline functionality check:
./reproduce.sh testExpected time: less than 1 s; 0.02 s measured on the reference machine.
Expected resources: 15 MB peak RAM; no network, database, Docker, or GPU.
Expected output:
OK: random_sample.jsonl has 4800 repositories
OK: committed outputs match the paper (N=2879, 96.8% any-vuln, 0 official, 5/1100 secret TPs)
Pick the path that fits the time and disk you have:
| Command | Cost | What it does |
|---|---|---|
./reproduce.sh test |
under 1 s | confirms the setup works; no network, no dataset |
./reproduce.sh precomputed |
~2 s | regenerates every paper figure and checks all 66 paper values from the committed outputs. Not a reduced path: it asserts the same numbers as the full run |
./reproduce.sh analyze |
over an hour | recomputes those same 66 values from the 10.3 GB database instead of reading them |
./reproduce.sh scan re-executes the six scanners over thousands of public images. It is outside the evaluator path and should only run on disposable infrastructure.
Description. This experiment reproduces the paper's data-backed results from the released reports database, and regenerates the paper's three figures. It checks:
- the uniform draw and the reachability outcomes of the 4,800 repositories;
- vulnerability prevalence and per-image counts;
- scanner disagreement, meaning how often a finding is reported by only one tool;
- base-image distributions and hardening findings;
- the comparisons against prior Docker Hub analyses;
- the manually labeled secret-detection rates.
Execution. Run one command from the repository root:
./reproduce.sh analyzeThe command automatically downloads and verifies the dataset on first use. Later runs reuse the verified local database.
Expected time: over an hour. 30 to 60 min measured on a Ryzen 7 9700X (16 threads, 59 GB RAM), plus the 226 MB download and its decompression to 10.3 GB; expect longer on slower hardware, since the dominant cost is the first read of the database from disk, and the first stage alone took 426 s there. Re-running once the file is in the page cache takes 4 to 7 min. Each stage prints only when it completes, so silence within a stage is normal and does not mean it hung.
Expected resources: 4 GB RAM recommended and 11 GB free disk; no GPU or Docker.
Expected result: the final line reports:
verify: 66 pass, 0 fail (skips listed in expected/_skip_note)
The command writes the regenerated figures to:
figures/fig_panels3.pdf: vulnerabilities per image, severity, and scanner completion;figures/fig_repro.pdf: prior analyses repeated on the random sample;figures/fig_extra.pdf: scanner agreement, base distribution, and secret-label categories.
The principal reproduced values are 2,879/4,800 scanned (60.0%), 94.4% with a critical vulnerability, 96.8% with any vulnerability, 947 median merged findings, 70.5% of distinct per-image CVE findings reported by only one scanner, and 5 genuine credentials among 1,100 sampled secret detections (99.55% false positives). Detailed source-to-value mappings are in expected/paper_values.json; methodological caveats and intentionally unavailable cross-corpus values are documented in docs/REPRODUCIBILITY_REPORT.md.
One command removes everything a run created, the database, the environment and the regenerated figures. It never touches anything tracked by git.
./cleanup.shPass --dry-run to list what would go without removing it (about ~10 GB).
This artifact is released under the MIT License. Scanned third-party images and their contents remain the property of their respective owners.
Cite the paper, not the repository:
Kapelinski, C. and Kreutz, D. (2026). A Uniform Random-Sample Security Measurement of Docker Hub Images. In Anais do XXVII Simpósio Brasileiro de Segurança da Informação e de Sistemas Computacionais (SBSeg 2026). Sociedade Brasileira de Computação.
@inproceedings{kapelinski2026chimangobase,
author = {Kapelinski, Cristhian and Kreutz, Diego},
title = {A Uniform Random-Sample Security Measurement of Docker Hub Images},
booktitle = {Anais do XXVI Simpósio Brasileiro de Cibersegurança (SBSeg 2026)},
pages = {1406--1412},
url = {https://sol.sbc.org.br/index.php/sbseg/article/view/44371},
year = {2026},
publisher = {Sociedade Brasileira de Computação},
}CITATION.cff carries the same metadata in machine-readable form, so GitHub's
"Cite this repository" button and tools such as Zenodo pick it up automatically.