download: no checksum or signature verification on any fetched tarball

[crippledgeek]

Summary

fetch() downloads every dependency tarball and verifies nothing about it. There is no
checksum field, no signature check, and no integrity mechanism anywhere in the tree — the
only thing standing between a tampered tarball and the compiler is TLS transport security
and whatever the far end happens to serve.

Evidence

lib/download.sh:30 is the entire trust decision:

if curl -fL -sS -o "$DISTDIR/$_file" "$_url"; then

A grep for any verification primitive across the whole repo returns nothing:

$ grep -rniE 'sha256|sha512|checksum|gpg|signify|md5' --include='*.sh' --include='*.conf' .
recipes/crypto/libressl.sh:53:  #   aesni_encrypt / sha256_block_data_order.   <- a comment
tests/avs2-reorder-dts.sh:66: ... -f framemd5 -                                <- unrelated test

Two properties make it worse than "no checksum":

1. The cache is trusted unconditionally. lib/download.sh:24 is
   if [ ! -f "$DISTDIR/$_file" ]. Once a file exists in packages/, it is reused
   forever with no check of any kind. A tarball that was corrupted, truncated, or
   swapped after landing is never re-examined. The retry loop rm -fs a failed curl,
   but a curl that returns 200 with wrong bytes is kept and cached.
2. -L follows redirects, including cross-host ones, so the host named in PKG_URL
   is not necessarily the host that serves the bytes.

Why this is worth doing

110 recipes carry a PKG_URL, spread over 24 distinct hosts. Nothing uses plain
http:// — transport is at least TLS everywhere, which is the good news.

The sharp part is that 18 recipes fetch through mirror redirectors
(ftpmirror.gnu.org, sourceforge.net) that deliberately hand off to third-party
volunteer mirrors:

recipes/crypto/gmp.sh          recipes/crypto/nettle.sh
recipes/audio/lame.sh          recipes/audio/twolame.sh
recipes/audio/soxr.sh          recipes/audio/opencore.sh
recipes/audio/vo_amrwbenc.sh   recipes/audio/lv2.sh
recipes/image/libpng.sh        recipes/other/freetype2.sh
recipes/other/freetype2-harfbuzz.sh  recipes/other/gettext.sh
recipes/other/bs2b.sh          recipes/tools/m4.sh
recipes/tools/autoconf.sh      recipes/tools/automake.sh
recipes/tools/libtool.sh       recipes/tools/giflib.sh

gmp and nettle are gnutls's crypto dependencies, and gnutls is the default TLS
backend. So the default build's crypto stack is fetched from a rotating set of volunteer
mirrors with no integrity check — which is precisely the scenario GNU's own download
guidance tells you to verify signatures for.

This surfaced during review of the libressl cluster (#16/#17/#18). That PR spent its
effort making one TLS library current and correctly configured; this is the weakest link
in the same chain, and it applies to all 110 recipes rather than one.

Prior art in our own tree

The sibling mediaforge-meson port already solves this: every wrap carries a
source_hash, and subprojects/libressl.wrap documents the discipline explicitly —
prefer upstream's published digest over one computed from your own download (a hash
derived from the artifact it authenticates is trust-on-first-use and authenticates
nothing), and prefer the signify-signed SHA256.sig over the plain SHA256 served from
the same host as the tarball.

The shell port has no equivalent. Bringing it to parity is the ask.

Suggested shape

• A PKG_SHA256 recipe field, verified in fetch() after download and on every cache
  hit — the cache path is where the current design is weakest, so verifying only on
  fresh download would fix the smaller half.
• die on mismatch, naming expected vs actual, and remove the offending file so a retry
  cannot silently reuse it.
• Populate from upstream-published digests where they exist, not from our own downloads.
• Roll out by risk rather than alphabetically: the crypto/TLS arms and the mirror-redirector
  recipes above first, then the rest.
• A --skip-checksum escape hatch is probably needed for bumping a version locally, but it
  should be loud.

One complication to settle before implementing

35 of the 61 GitHub URLs use the archive/refs/tags/... auto-generated tarball form.
Auto-generated archives are not guaranteed byte-stable — a compression change on GitHub's
side in early 2023 altered checksums ecosystem-wide and broke pinned hashes for a lot of
packagers. This needs confirming against GitHub's current guarantees before we commit to
a design, but if it still holds, the options are to prefer real release assets where a
project publishes them, or to accept that those 35 pins will occasionally churn. Worth
deciding up front rather than discovering it on the first mass-bump.

Not claimed

This is not a report of a compromise, and there is no evidence any fetched tarball has
ever been tampered with. It is a missing defence-in-depth control. Nothing is blocked on
it, and the current builds are fine.

Environment

• branch develop @ 3f76539, SCRIPT_VERSION="3.0"
• Counts above measured 2026-08-23 against recipes/ on that commit.

[crippledgeek]

Research findings

Diagnosis in the issue holds. Three structural findings change the suggested shape, and one class of
recipe can't be verified by digest at all. Everything below was probed live on 2026-08-24.

The surface is 199 digests, not 110

	count
recipes with a PKG_URL	107 (3 more clone from git: vaapi, libplacebo, librtmp)
nested fetch calls with no PKG_* context	10 (7 in lv2.sh, plus ffmpeg.sh, opencl.sh, libcdio.sh)
distinct (package, version) pairs needing a digest	199

PKG_VERSION is profile-overridable (recipes/crypto/gmp.sh:6), and 45 packages carry a different
version in each of the 4 profiles. A digest is a property of (package, version), so a bare
PKG_SHA256= in a recipe is only correct for whichever version happens to be selected — under
--profile=7.1 it would be either unverified or wrongly-verified.

The digests belong in profiles/*.conf beside the versions, with the recipe carrying only the
default-version value as its ${PKG_DIGEST_GMP:-…} fallback — mirroring the existing PKG_VERSION_*
mechanism exactly.

Covering only the current profile plus recipe defaults is ~110 digests; full historical coverage of
all 4 profiles is 199. Suggest the former, older profiles left warn-only.

42 recipes fetch forge-generated archives; 1 can't be pinned at all

• 35 x GitHub archive/refs/tags/…
• 6 x GitLab-style -/archive/… (5 on code.videolan.org, including x264 and dav1d, plus SVT-AV1)
• 1 x aomedia.googlesource.com/aom/+archive/…

These are recompressed on demand rather than stored. Probed by downloading each URL twice, one second apart:

googlesource(+archive)       VARIES  4e50c8f937a8bd7a / 5c90e5f4c1a00465
github(archive/refs/tags)    STABLE  7178c4833639e6b9 / 7178c4833639e6b9
gitlab.com(-/archive)        STABLE  d0d73bfea42fdcc1 / d0d73bfea42fdcc1
code.videolan.org(-/archive) STABLE  fa635e2bdb25147b / fa635e2bdb25147b

libaom cannot be pinned by digest. Google deliberately varies the bytes of gitiles +archive
output to stop people depending on the checksum, and declined to fix the timestamp non-determinism
for that reason (google/gitiles#217, and the 2023-07 rb-general thread). The only fix is changing
where that recipe fetches from — GitHub releases/download, or a clone pinned to a commit SHA.

GitHub's position on the other 35: the byte-for-byte stability promise ran one year from 2023-02-21
and has expired. What stands is six months' notice before any change, plus an explicit
recommendation to use uploaded release assets when a stable checksum is needed
(https://github.blog/open-source/git/update-on-the-future-stability-of-source-code-archives-and-hashes/).
Tolerable — a compression change fails loudly at die, with warning — but it's maintenance risk the
22 releases/download recipes don't carry. Where an upstream offers both, moving the URL to the
release asset is worth more than the digest.

The 3 git-clone recipes are a hole this issue doesn't mention. librtmp.sh:16 and
libplacebo.sh:35 clone --depth 1 --branch v$VERSION, and a git tag is mutable server-side.
No digest applies; the fix is pinning a commit SHA.

Where digests can actually come from

tier	recipes	source	strength
GNU (ftp.gnu.org)	7	.sig, GPG-signed — 200 on every one checked	signature
gnutls, openssl, LibreSSL, libtiff, libssh, libwebp, bzip2, pkg-config	~6	.sig / .asc / signify SHA256.sig	signature
SourceForge	~6	sha256 in the file-listing JSON	digest from SF master
videolan download.	1	.sha512 sidecar (libbluray); x265 has none	digest
xiph	5	no sidecar of any kind (.sha256, .sha256sum, .asc, .sig all 404)	none
GitHub release assets	22	per-project, often none	varies
forge archives	42	none possible	TOFU only

Note all 7 GNU recipes use ftpmirror.gnu.org, which returned 502 for a .sig that ftp.gnu.org
served fine. Those are the recipes this issue rightly calls highest-risk (gmp and nettle feed the
default gnutls stack) — the cheapest real improvement there is fetching from ftp.gnu.org and
verifying the signature, not pinning a hash.

One pushback on the issue text. "Populate from upstream-published digests where they exist" is
right, but a genuinely independent published digest exists for maybe 20 of 107 recipes. For the
other ~85 the digest will be computed from our own download — which subprojects/libressl.wrap
correctly calls trust-on-first-use that authenticates nothing. Still worth doing: it converts the
cache from unconditionally-trusted to verified, and pins the artifact against later mirror drift.
But it should be labelled honestly per-recipe, not presented as upstream verification.

Algorithms

What upstreams in this tree actually publish:

algorithm	recipes
GPG/signify signature	~13
SHA-256	~6 SourceForge + openssl
SHA-512	1 (libbluray)
SHA-1	SourceForge only, always alongside MD5
MD5	SourceForge only, never alone
BLAKE2 / BLAKE3 / SHA-3 / xxhash	0

Two SourceForge recipes publish no SHA-256 — twolame and libpng have only sha1+md5:

lame       sha256 sha1 md5     twolame    sha1 md5    <- no sha256
opencore   sha256 sha1 md5     libpng     sha1 md5    <- no sha256
vo-amr     sha256 sha1 md5
soxr       sha256 sha1 md5
bs2b       sha256 sha1 md5

MD5 is not worth supporting. It never appears without SHA-1 alongside it, so a SHA-1 fallback
dominates it — and MD5 needs a third output-parsing shape (macOS ships BSD md5, not md5sum)
for zero coverage gain. A weak digest is worth having, though, and the reasoning matters: the
threat here is mirror substitution against an already-published artifact, which requires a second
preimage, not a collision. MD5 and SHA-1 are dead for collisions but not for second preimage
(best known MD5 preimage attack is 2^123.4; RFC 6151 calls it impractical and scopes its prohibition
to "where collision resistance is required such as digital signatures"). So a weak upstream digest
defends the volunteer-mirror threat this issue is about, and does not defend against a malicious
upstream — worth stating in the recipe comment rather than leaving implied.

twolame doesn't need it anyway: upstream ships twolame-0.4.0.tar.gz as an uploaded GitHub
release asset on njh/twolame. Repointing the URL gets a normal SHA-256 pin. libpng is the one
genuine holdout (no .asc/.sig/.sha256 on SourceForge — all 404).

Nothing stronger than SHA-256 buys security here. SHA-256's second-preimage resistance is at
full strength; there's no gap for SHA-512 or BLAKE2 to close, and length extension is irrelevant to
comparing a digest against a published constant. Portability settles the rest: shasum -a N
(SHA-1/2 family) is the only digest tool spanning Linux and macOS. b2sum is Linux-only, SHA-3 has
no coreutils tool on either platform, and openssl dgst isn't a fallback because /usr/bin/openssl
on macOS is LibreSSL 3.3.6, which predates its SHA-3 support and never had BLAKE2.

Two details the suggested shape misses

Digest-tool portability. There is no hashing helper anywhere in the tree, and mediaforge targets
macOS. sha256sum is coreutils/Linux-only; macOS ships shasum and openssl. fetch() needs a
_sha256()-style helper with a fallback chain, in lib/utils.sh beside the other primitives.

The 10 nested fetch calls. lv2.sh pulls 7 sub-tarballs by positional argument with no PKG_*
context at all — a recipe-field design leaves them silently unverified. fetch() needs an optional
4th argument, or the sub-fetches need their own digest variables.

Proposed shape

1. _verify_digest in lib/utils.sh: sha256sum -> shasum -a 256 -> openssl dgst, die on
   mismatch naming expected vs actual, and rm -f the offending file so a retry can't reuse it.
2. Verify on both paths in fetch() — fresh download and cache hit. The issue is right that
   lib/download.sh:24 is the weaker half.
3. One tagged field, not a per-algorithm namespace:

   PKG_DIGEST="${PKG_DIGEST_GMP:-sha256:a3c2b80…}"
   PKG_DIGEST="${PKG_DIGEST_LIBBLURAY:-sha512:94dbf3b6…}"
   PKG_DIGEST="${PKG_DIGEST_LIBPNG:-sha1:d4d1e59…}"   # mirror-integrity only

   Accept sha256 (canonical) / sha512 (1 user) / sha1 (1 user, comment mandatory); die on an
   unknown or absent prefix, never silently skip. Empty digest = warn-and-continue during rollout so
   it isn't all-or-nothing. Meson's wrap format offers no algorithm choice at all (source_hash is
   SHA-256, full stop) — the shell port needs the tagged form only because it fetches from 24 hosts
   rather than curated wraps.
4. Fix the source before pinning where the source is the problem: GNU -> ftp.gnu.org; libaom off
   gitiles; librtmp/libplacebo -> commit SHA.
5. Roll out by risk as the issue says: crypto/TLS -> GNU + SourceForge mirror redirectors -> GitHub
   release assets -> forge archives last, since those are TOFU regardless.
6. check-updates (lib/updates.sh) already walks every recipe and queries GitHub; extending it to
   report digest drift is the natural maintenance path for ~200 pins.

Incidental finding: downloads.xvid.com is a soft-404 host

It returns HTTP 200 with tarball-shaped bytes for any path under /downloads/ — a .zzzznope
suffix came back as a bzip2 stream. A first sidecar probe reported xvidcore as publishing every
digest format; all of it was false. Any sidecar-fetching logic must validate the content of what
it retrieves, not the status code, or that host will hand it a tarball and call it a digest file.

[crippledgeek]

The GitHub auto-tarball question, answered

The research comment above flagged one thing to settle before designing: whether GitHub's
auto-generated archive/refs/tags/... tarballs are byte-stable enough to pin. Researched
2026-08-25. Short answer: no, and the situation is worse than in 2023 — it is now
unguaranteed rather than merely unpromised. Design for churn.

The 2023 guarantee was time-boxed and has expired

GitHub's commitment of 2023-02-21 reads "byte-for-byte stable for no less than a year
from today", with a pledge of six months' notice before any future format change. That year
ended February 2024. No renewal or replacement statement has been published.

Community discussion #46034 — the thread where the commitment was made — ends with a
September 2024 report of a SHA256 change on an archive untouched since 2019, a staff member
beginning to investigate, and no resolution or updated policy. GitHub's standing documented
position remains the older one: checksums of automatically generated archives are not
guaranteed, because git archive generates them on the fly.

Two post-expiry recurrences are on record. The instructive one is March 2025: kubespray hit a
changed hash on the Calico CRD archive and proposed switching to the REST archive endpoint.
The PR (kubernetes-sigs/kubespray#12087) was closed — a reviewer noted the API endpoint
carries no stability guarantee either, and recommended real uploaded release assets instead.

Immutable Releases does not rescue this

Immutable Releases reached GA on 2025-10-28. It does not help here:

• Opt-in per repository or organization — across ~50 upstream projects we do not control,
  not a lever we can pull.
• Non-retroactive: "existing releases remain mutable unless you republish them."
• Both the changelog and the docs describe the protection over files attached to the
  release. Neither states that auto-generated source archives are covered or attested.

The mitigation this issue assumed is not available

The research comment proposed preferring real release assets where projects publish them.
Measured against develop today, here is what that would actually buy.

106 PKG_URLs over 23 hosts; 61 on github.com, split 22 release assets / 39
auto-generated. The 39 is 35 archive/refs/tags/ plus 4 legacy archive/<tag>.tar.gz
(srt, vid_stab, vapoursynth, uavs3d) that serve identical bytes — the earlier
count of 35 undercounts by those four.

Querying each of those 39 upstream releases for its uploaded assets:

upstream state	count
no release object for that tag at all	14
release exists, zero assets	14
assets exist, but platform binaries only (Windows installers, .deb, prebuilt libs)	9
a real uploaded source tarball we could switch to	2

The two are openssl (openssl-3.5.4.tar.gz, shipped with .sha256 and .asc) and
libgme (libgme-0.6.3-src.tar.gz). zvbi is a near-miss worth recording: its only asset is
v0.2.44.tar.gz.asc, a detached signature over the auto-generated archive — upstream there
treats the unstable artifact as canonical.

Two out of thirty-nine is not a strategy. 37 recipes stay on byte-unstable archives no
matter what we decide.

Consequence for the acceptance criteria

This weakens one criterion in the issue body: "populate from upstream-published digests where
they exist, not from our own downloads." For most of the GitHub set no upstream digest exists
to populate from, so a self-observed digest is the only option available. That is still worth
having — it closes the unconditionally-trusted cache at lib/download.sh:24, truncation, and
the 18 mirror-redirector recipes where the host named in PKG_URL is not the host serving the
bytes — but it is a different and weaker claim than an upstream-anchored digest, and the two
should not be recorded as if they were the same thing.

Direction this points the design

• Ship digests for all 106 URLs, but record which tier each digest covers, so a churn
  event reads differently from an attack. A simultaneous mismatch across the auto-archive set
  is a GitHub recompression and wants a one-shot re-baseline path; a single mismatch on a
  contractually-stable artifact is the alarm that stops a build cold.
• Move openssl and libgme to their real release assets. openssl's upstream .sha256 and
  .asc make its digest upstream-published rather than self-observed.
• Consider fetch_git for the sharpest of the remaining 39. The machinery landed in librtmp/libplacebo: git clones pin mutable tags, and librtmp fails outright under every profile #28/av1: libaom is fetched from a non-reproducible gitiles archive and cannot be integrity-checked #27
  and its docstring at lib/download.sh:9-17 already makes the argument: a commit name is the
  hash of the commit's content, "the integrity guarantee no tarball digest can give these
  recipes." uavs3d is the obvious first conversion — it already pins a commit SHA, just
  through an archive URL, so there is no tag semantics to preserve.

Sources

• Update on the future stability of source code archives and hashes (2023-02-21) —
  https://github.blog/open-source/git/update-on-the-future-stability-of-source-code-archives-and-hashes/
• Git archive checksums may change (2023-01-30) —
  https://github.blog/changelog/2023-01-30-git-archive-checksums-may-change/
• GitHub archive hash stability, community discussion #46034 —
  https://github.com/orgs/community/discussions/46034
• Update the calico crd archive download url kubernetes-sigs/kubespray#12087 (closed; REST endpoint rejected as equally unstable)
• Immutable releases are now generally available (2025-10-28) —
  https://github.blog/changelog/2025-10-28-immutable-releases-are-now-generally-available/
• Immutable releases docs —
  https://docs.github.com/en/enterprise-cloud@latest/code-security/concepts/supply-chain-security/immutable-releases

Environment

Counts measured 2026-08-25 against develop @ ec95b02. Upstream asset states queried live via
gh api repos/<owner>/<repo>/releases/tags/<tag> on the same date; they can change as projects
publish new assets.