Ship pinned prebuilt external cargo tools in the sc-lint release package

[randlee]

Context

atm-core is planning to migrate its lint provisioning to sc-lint (the tooling was dogfooded in atm-core and split out here). The main blocker to a fast, clean migration is that the external cargo tools sc-lint's lint surface depends on (cargo-deny, cargo-shear, cargo-modules) are not part of the sc-lint package — every consumer (including sc-lint's own CI) provisions them separately, and any registry-install fallback path risks silent multi-minute source compiles in CI (atm-core just spent PR randlee/atm-core#1026 eliminating exactly that failure mode: CI medians went 12→42 min from source-building these tools).

Requested changes

1. Bundle pinned upstream prebuilt binaries in the release archives. The release pipeline downloads the official upstream prebuilt binaries for the pinned tool versions, verifies them, and packs them into sc-lint_<ver>_<target>.tar.gz alongside the sc-lint binaries, covered by the existing checksums.txt. Consumers then get one download + one checksum verification for the whole lint toolchain — no binstall, no registry fallback, no compile path at all.
   • Current pins to bundle: cargo-deny 0.20.2, cargo-shear 1.13.3, cargo-modules 0.26.0 (all publish official GitHub-release prebuilts for the 4 sc-lint targets).
   • Include a NOTICE/attribution file for redistribution (deny/audit MIT OR Apache-2.0, shear MIT, cargo-modules MPL-2.0 — MPL requires a pointer to upstream source).
   • The pinned tool set becomes part of the versioned release contract, which is a feature: consumers stop drifting (atm-core is on shear 1.12.0, sc-lint CI on 1.13.3 today).
2. Action setup operation installs the bundled tools onto PATH (and verifies them), so a consumer workflow needs exactly one step: uses: randlee/sc-lint@<sha> with: {operation: setup, version: X.Y.Z}. Hard-fail on any verification mismatch; no compile fallback of any kind.
3. Decide cargo-audit ownership. atm-core CI also runs cargo audit (pinned 0.22.2) as a security gate. Either include cargo-audit in the bundled set (preferred — one supply channel for all CI cargo tools) or document explicitly that it stays consumer-owned.
4. Dogfood: switch sc-lint's own CI off taiki-e/install-action + binstall onto the bundled artifacts, so the packaging path is exercised on every release.

Acceptance sketch

• gh release view on the next release shows the tool binaries inside the platform archives (or as separately checksummed assets) and a NOTICE file.
• A consumer job on ubuntu/macos/windows runners gets sc-lint + all bundled tools from a single verified download, with zero cargo install/compile lines in the log.
• Version receipts: setup records/asserts exact tool versions; mismatch is a hard failure.

Why now

This lands before atm-core's migration so the migration PR can delete atm-core's four-tool bootstrap section and duplicated .just/lint_cargo_*.py wrappers in one move, collapsing atm-core bootstrap to two verified prebuilt downloads (sc-lint + sc-compose).

Filed on behalf of Rand from the atm-core CI wall-clock investigation (PR randlee/atm-core#1026).

[randlee]

Feature Design: Bundled Verified Lint Toolchain in sc-lint Releases

Status: proposed · Tracking issue: #118 · Author: fenix (atm-dev), on behalf of Rand · Date: 2026-08-26

1. Summary

Make the external cargo tools that sc-lint's lint surface depends on — cargo-deny, cargo-shear, cargo-modules, and (pending ownership decision) cargo-audit — part of the sc-lint release package itself. Each platform release archive ships the pinned, SHA256-verified tool binaries alongside the sc-lint binaries, and the GitHub Action's setup operation puts them on PATH with hard receipt verification. Consumers get the entire lint toolchain from one download verified by one checksums.txt, with zero compile paths and zero third-party installer actions in their workflows.

Goals

• One first-party supply channel for the whole lint toolchain (same trust model consumers already accept for sc-lint itself).
• Zero source compiles and zero registry fetches in consumer CI — eliminates the silent-compile-fallback failure class that regressed atm-core CI medians from 12 to 42 minutes (atm-core#1026).
• The pinned tool-version set becomes part of the versioned sc-lint release contract (no consumer drift; atm-core is on shear 1.12.0 while sc-lint CI is on 1.13.3 today).
• Dogfooding: sc-lint's own CI consumes the bundle, so packaging breakage is caught before consumers see it.

Non-goals

• No change to which lints run or how sc-lint lint ci orchestrates them.
• No new upstream targets beyond the existing 4 release targets (aarch64-linux can be added later; no current consumer needs it).
• No auto-update mechanism; pin bumps are deliberate releases.

2. Background

sc-lint v0.5.0 archives contain exactly the 4 workspace binaries (sc-lint, sc-lint-boundary, sc-lint-portability, sc-lint-runtime) plus the sc-lint-docs bundle, enforced by validate_release_archive (exact-file-set check). External tools are provisioned per-consumer: sc-lint's own CI uses taiki-e/install-action (deny 0.20.2, shear 1.13.3) + cargo binstall (modules 0.26.0); atm-core's bootstrap uses binstall with a registry-compile fallback. atm-core PR #1026 documented the failure mode: when binstall or its receipt verification fails, the fallback silently compiles from source, tripling CI wall-clock; the only alternative (hard-fail) is brittle because one tool has no upstream prebuilts at all (§4.3).

Upstream prebuilt inventory (verified 2026-08-26)

Tool	Upstream	Pinned	Release assets	Notes
cargo-deny	EmbarkStudios/cargo-deny	0.20.2	cargo-deny-0.20.2-<target>.tar.gz + per-asset .sha256	Tag has no v prefix. Linux assets are musl-only (static — runs fine on gnu runners). Windows msvc asset is a .tar.gz.
cargo-audit	rustsec/rustsec	0.22.2	cargo-audit-<target>-v0.22.2.tgz (windows: .zip)	Monorepo tag with a slash: cargo-audit/v0.22.2 (URL-encode as cargo-audit%2Fv0.22.2). Linux: both gnu and musl published.
cargo-shear	Boshen/cargo-shear	1.13.3	cargo-shear-<target>.tar.gz (windows also .zip)	No version in asset filename — the manifest-pinned SHA256 is the only thing preventing a moved tag from changing content. All 4 sc-lint targets covered, incl. universal-apple-darwin.
cargo-modules	regexident/cargo-modules	0.26.0	NONE — repo publishes git tags only, zero GitHub releases	No upstream prebuilt exists anywhere first-party. binstall only ever worked for it via the third-party cargo-quickinstall channel or by compiling. See §4.3.

This inventory is the core justification for bundling: the "just binstall everything" model has three different asset conventions, one tool with no assets at all, and no single checksum manifest — every consumer re-solves (and mis-solves) the same verification problem.

3. Current release pipeline (what we build on)

• release/publish-artifacts.toml (schema_version 1): [[crates]] publish plan, [[release_binaries]] (archive contents), [documentation_bundle].
• scripts/release_artifacts.py: manifest loader + subcommands used by the workflow — emit-inventory, cargo-build-bin-args, stage-release-archive, validate-release-archive (exact-set check: declared binaries + docs bundle, nothing else), stage-homebrew-layout.
• .github/workflows/release.yml: gate-and-tag → inventory → pre-publish audit → publish-crates → build matrix (4 targets) → package + validate → release job downloads all archives, generates one checksums.txt (sha256sum), publishes via softprops/action-gh-release → moves Action v1 tag → updates Homebrew.
• action/index.js: resolves sc-lint_<ver>_<triple>.<ext> + checksums.txt from the release, verifies SHA256, extracts to a temp root, sanity-checks layout (sc-lint binary + sc-lint-docs), runs compatibility preflight, then setup stops (adds dir to PATH) or lint/test executes the consumer contract.

4. Design

4.1 Manifest extension: [[external_tools]]

Extend release/publish-artifacts.toml (bump to schema_version = 2; loader accepts 1 only if no external tools are declared, to keep old branches releasable):

[[external_tools]]
name = "cargo-deny"
version = "0.20.2"
license = "MIT OR Apache-2.0"
provenance = "upstream-release"          # upstream-release | first-party-build
upstream_repo = "EmbarkStudios/cargo-deny"
upstream_tag = "0.20.2"                  # exact tag, no derivation

  # one entry per sc-lint release target
  [external_tools.assets.x86_64-unknown-linux-gnu]
  asset = "cargo-deny-0.20.2-x86_64-unknown-linux-musl.tar.gz"   # musl on gnu is intentional (static)
  sha256 = "<pinned>"
  binary_path = "cargo-deny-0.20.2-x86_64-unknown-linux-musl/cargo-deny"  # path inside upstream archive

  [external_tools.assets.x86_64-apple-darwin]
  asset = "cargo-deny-0.20.2-x86_64-apple-darwin.tar.gz"
  sha256 = "<pinned>"
  binary_path = "cargo-deny-0.20.2-x86_64-apple-darwin/cargo-deny"

  [external_tools.assets.aarch64-apple-darwin]
  # ...

  [external_tools.assets.x86_64-pc-windows-msvc]
  asset = "cargo-deny-0.20.2-x86_64-pc-windows-msvc.tar.gz"      # yes, tar.gz on windows
  sha256 = "<pinned>"
  binary_path = "cargo-deny-0.20.2-x86_64-pc-windows-msvc/cargo-deny.exe"

[[external_tools]]
name = "cargo-audit"
version = "0.22.2"
license = "MIT OR Apache-2.0"
provenance = "upstream-release"
upstream_repo = "rustsec/rustsec"
upstream_tag = "cargo-audit/v0.22.2"     # slash tag — download URLs must URL-encode
# assets: cargo-audit-<target>-v0.22.2.tgz / .zip (windows) ...

[[external_tools]]
name = "cargo-shear"
version = "1.13.3"
license = "MIT"
provenance = "upstream-release"
upstream_repo = "Boshen/cargo-shear"
upstream_tag = "v1.13.3"
# assets: cargo-shear-<target>.tar.gz — unversioned filenames; pinned sha256 is the integrity anchor

[[external_tools]]
name = "cargo-modules"
version = "0.26.0"
license = "MPL-2.0"
provenance = "first-party-build"         # §4.3 — no upstream prebuilts exist
upstream_repo = "regexident/cargo-modules"
upstream_tag = "v0.26.0"
# no [external_tools.assets.*] tables for first-party-build entries

Loader validation rules (mirroring the existing strictness):

• provenance = "upstream-release" requires an assets table covering every release target; a missing target is a manifest error, not a runtime skip.
• provenance = "first-party-build" forbids assets and requires upstream_repo + upstream_tag.
• Every sha256 must be 64 lowercase hex chars; every license non-empty; duplicate name rejected; entries sorted by name (same determinism rule as the inventory items).
• release_binary_names() stays sc-lint-only; a new external_tool_entries() accessor feeds the new steps, so the crates.io publish plan and Homebrew logic are untouched by default (§4.7).

Pin capture tooling: a new subcommand release_artifacts.py pin-external-tool --name cargo-deny --version 0.20.2 fetches the assets for all targets, computes SHA256s, cross-checks against upstream's own .sha256 sidecar files where published (deny has them), and prints the manifest stanza. Pins are captured once at PR time on a trusted machine and reviewed in the diff — release runs never trust upstream checksums alone, only the manifest.

4.2 Release workflow: acquiring upstream tools

New step in the existing per-target build job (before packaging), implemented as release_artifacts.py fetch-external-tools --manifest ... --target <triple> --dest target/<triple>/external-tools:

1. For each upstream-release tool: download https://github.com/<upstream_repo>/releases/download/<url-encoded upstream_tag>/<asset> (bounded retries with backoff; any final failure fails the release job — there is deliberately no alternate source).
2. Verify SHA256 against the manifest pin (upstream sidecars were already cross-checked at pin time; the manifest is authoritative at release time).
3. Extract only the declared binary_path from the archive (ignore docs/completions upstream ships); reject archives where the declared path is missing.
4. Normalize the binary name to <name> / <name>.exe, set the executable bit, and record a receipt line {name} {version} {sha256-of-extracted-binary} into external-tools/receipts.txt.
5. Sanity-run <tool> --version on the build runner (native target only — all 4 targets build on their native OS in the current matrix, and the one cross-compile pair, x86_64-apple-darwin on arm64 macos-latest runners, executes fine under Rosetta) and assert the output contains the pinned version. This is the same receipt philosophy as atm-core's bootstrap, moved to release time where it runs once instead of in every consumer job.

4.3 cargo-modules: first-party build at release time

cargo-modules has no GitHub releases (verified: tags only), so there is no upstream prebuilt to bundle and there never was one for binstall either — every downstream install of it has been either a source compile or a third-party cargo-quickinstall binary. The design makes sc-lint the party that compiles it, once per release, in the release pipeline:

• In the build job (the Rust toolchain and target are already installed): cargo install cargo-modules --version 0.26.0 --locked --target <triple> --root target/<triple>/external-tools/cargo-modules-root, then move the binary into the staging set and emit the same receipt line (name, version, sha256 of the produced binary).
• --locked pins the full dependency graph via the crates.io-published lockfile; the crates.io package for an already-published version is immutable, so this is reproducible-enough for a receipt (bit-for-bit reproducibility is a non-goal; the SHA256 in checksums.txt covers whatever we shipped).
• Release-time compile cost (~2–4 min per target, parallel across the matrix) is paid once per sc-lint release instead of in every consumer CI job forever. This is the decisive argument for bundling over any "consumers install at CI time" scheme — no consumer-side strategy can conjure a prebuilt that doesn't exist.
• If upstream starts publishing releases later, the manifest entry flips provenance to upstream-release with pinned assets; nothing else changes.

emit-inventory gains one item per external tool (publishTarget: "github-release-bundle", verifyCommands: ["<name> --version"], required: true) so the release inventory remains the single audit surface, and the existing schema-validation step is extended for the new publishTarget value.

4.4 Archive layout

sc-lint_<ver>_<target>.tar.gz|zip
├── sc-lint[.exe]
├── sc-lint-boundary[.exe]
├── sc-lint-portability[.exe]
├── sc-lint-runtime[.exe]
├── sc-lint-docs/…                      (unchanged)
└── tools/
    ├── cargo-deny[.exe]
    ├── cargo-shear[.exe]
    ├── cargo-modules[.exe]
    ├── cargo-audit[.exe]               (if §8 Q1 resolves to "include")
    ├── receipts.txt                    (name version sha256 per line)
    └── THIRD-PARTY-NOTICES.md          (§5)

• Tools live in a tools/ subdirectory so the archive root stays exactly the historical contract and PATH ordering is explicit (consumers may have other copies of these tools; the Action prepends tools/ deliberately, see §4.6).
• stage_release_archive copies the fetched/built tools + receipts + notices into tools/; release_archive_expected_files() adds them so validate_release_archive's exact-set check keeps rejecting any undeclared file — the property that makes the archive contract auditable is preserved, now covering the bundled tools too.
• Size impact: roughly +25–40 MB compressed per archive (deny ~8 MB, audit ~5 MB, shear ~2 MB, modules ~8 MB, varying by target). Well under GitHub's 2 GiB asset limit; acceptable for a CI-oriented artifact and offset by consumers dropping 4 separate downloads + installer actions.
• checksums.txt is unchanged in shape: it covers the archives, and the archives now contain the tools, so the single existing verification in the Action transitively covers every tool binary. receipts.txt inside the archive additionally pins each tool binary individually for post-extraction auditing.

4.5 Version/pin policy

• The [[external_tools]] pin set is part of the release contract: bumping a tool version is a manifest PR + a normal sc-lint release. Consumers upgrade tools by bumping one version: input.
• CHANGELOG.md gains a "Bundled tools" section per release listing the pin set, and the release notes template gets a matching table.
• Pin-freshness is advisory, not enforced: a scheduled workflow (or manual release_artifacts.py check-external-tool-updates) reports newer upstream versions; RustSec advisories against a bundled tool version are handled as ordinary pin-bump releases.

4.6 Action changes (action/index.js + action.yml)

• After the existing extract + layout check, when the archive contains tools/: verify each manifest… (the Action doesn't have the manifest — it has receipts.txt) — concretely: parse tools/receipts.txt, assert each listed binary exists, re-hash it, compare to the receipt SHA256, run <tool> --version and assert it contains the receipt version. Any mismatch is a hard ActionError (new ERROR.tool_receipt code). This is defense-in-depth on top of the archive checksum, and it's what makes "the tool on PATH is exactly the pinned one" a checked invariant rather than an assumption.
• addActionPath prepends both the archive root (as today) and tools/, tools first is not necessary — prepend tools/ so the bundled copies shadow any runner-preinstalled versions (GitHub runners do not preinstall these, but consumer self-hosted runners might).
• New outputs: tools-path (absolute path of tools/) and tool-versions (JSON object {"cargo-deny":"0.20.2", ...} built from receipts) so consumer workflows/bootstrap scripts can assert against their own expectations.
• Backwards compatibility: archives from releases predating this feature have no tools/; the Action treats that as "no bundled tools" (no error) so sc-lint@v1 keeps working against old pinned version: inputs. A new optional input require-tools: true (default false for one release, flipped to true once consumers migrate) lets a consumer hard-require the bundle.
• operation: lint/test also get tools/ on PATH for the child process env, so sc-lint lint ci --consumer finds the bundled tools without consumer setup.

4.7 Homebrew and non-CI consumers

• The Homebrew formula does not repackage the third-party binaries: stage_homebrew_layout keeps its current sc-lint-only file set, and the formula instead declares cargo-deny/cargo-audit etc. as formula dependencies where formulae exist, or documents brew install/upstream instructions in docs-bundle/installation.md. Rationale: brew users are interactive developers with a package manager already managing those tools; bundling there invites version shadowing confusion. CI is where the hermetic bundle matters.
• winget/manual installs: unchanged; the docs bundle's just-setup.md/ci.md are updated to describe the bundled-tools contract and the receipts file.

4.8 Dogfooding: sc-lint's own CI

Replace the taiki-e/install-action + binstall steps in .github/workflows/ci.yml with the repo-local action at operation: setup against the previous published release (CI of commit N can't consume the unreleased bundle of commit N). Additionally, action-fixtures.yml gains a fixture that builds a local archive with tools/ via the release scripts and runs the Action's receipt-verification path against it, so the new packaging is exercised pre-release, not just post-release.

5. Supply-chain & licensing

Trust model: consumers already trust randlee/sc-lint release assets + checksums.txt. This design moves the third-party fetch from N consumer CI jobs (each trusting binstall's resolution logic + upstream at run time) to one release-time fetch verified against SHA256 pins that were captured and human-reviewed in a manifest PR. Content-addressing (pinned SHA256, not tags) means a retagged or compromised-later upstream release cannot change what ships without a manifest diff. The cargo-quickinstall third-party build channel is eliminated entirely.

Licensing (tools/THIRD-PARTY-NOTICES.md, generated by release_artifacts.py from manifest fields at stage time):

Tool	License	Redistribution obligations
cargo-deny	MIT OR Apache-2.0	include license text + copyright notice
cargo-audit	MIT OR Apache-2.0	include license text + copyright notice
cargo-shear	MIT	include license text + copyright notice
cargo-modules	MPL-2.0	binary redistribution permitted; must inform recipients where source is available (link to upstream repo + exact tag) and note MPL §3.2 source-availability terms

The notices file carries, per tool: name, version, upstream repo URL, exact tag, license identifier, full license text, and for cargo-modules the MPL source-availability statement. (Action item at implementation time: re-verify each license from the pinned tag's LICENSE files rather than trusting this table.)

No cryptographic signing exists on the sc-lint release surface today; this design doesn't add it, but the receipts + manifest pins are structured so a future sigstore/cosign layer can sign checksums.txt without reshaping anything.

6. Consumer migration (atm-core)

Before (current, post-#1026 stopgap): bootstrap.toml pins 4 tools + sc-compose; bootstrap.py runs binstall per tool with receipt verification, hard-fail in CI; separate binstall action pin; ~5 network fetches from 5 origins per cold job.

After:

- name: Set up lint toolchain
  uses: randlee/sc-lint@<immutable-sha>          # v1
  with:
    operation: setup
    version: 0.6.0
    require-tools: true

• Delete from atm-core: bootstrap.toml [cargo] four-tool section, bootstrap.py binstall/fallback/receipt code for those tools, the cargo-binstall action step, and the duplicated .just/lint_cargo_{deny,shear,modules}.py wrappers (atm-core's just lint delegates those checks to sc-lint lint ci --consumer, which is the extraction plan's intended end state).
• cargo-audit: if bundled (§8 Q1), atm-core's standalone cargo audit CI step keeps working with the bundled binary on PATH and its bootstrap entry is deleted too; if not bundled, it stays in atm-core's bootstrap as the single remaining tool.
• sc-compose remains its own verified prebuilt channel (different repo, already solved in #1026).
• atm-core keeps actions/cache only for the cargo build caches; tool caching becomes unnecessary (one ~35 MB download replaces it — at GitHub-runner bandwidth this is faster than the current 4-tool binstall sequence, and immune to its failure modes).

7. Testing & acceptance criteria

Unit (release scripts): manifest schema-v2 loader (valid/invalid: missing target, bad sha, dup names, provenance/assets contradictions); expected-files set includes tools/; notices generation; receipt emission; pin-capture cross-check against upstream sidecar checksums.

Fixtures (Action): archive with valid tools/ → setup passes, PATH order correct, outputs populated; tampered tool binary → ERROR.tool_receipt; receipts/version mismatch → hard fail; pre-feature archive without tools/ → passes with require-tools: false, fails with true.

Release rehearsal (workflow_dispatch dry-run mode or a -rc tag): all 4 targets produce archives passing validate-release-archive; a consumer-shaped job on ubuntu/macos/windows runs setup then executes each tool --version with zero cargo install/compile lines in the log.

Acceptance (mirrors issue #118):

1. gh release view on the next release shows archives whose contents include tools/ with the 3(+1) binaries, receipts.txt, THIRD-PARTY-NOTICES.md; checksums.txt covers the archives.
2. A consumer job on all 3 runner OSes obtains sc-lint + all bundled tools from a single verified download; hard failure on any hash/version mismatch; no compile fallback exists anywhere in the path.
3. sc-lint's own CI consumes the bundle (previous release) and the fixture suite exercises the packaging of the current commit.
4. CHANGELOG/release notes list the bundled pin set.

8. Open questions

1. cargo-audit ownership — include in the bundle (recommended: one supply channel, atm-core needs it) or declare consumer-owned? Including costs ~5 MB/archive.
2. require-tools default flip timing — one release at false then flip, or keep opt-in until atm-core migration completes?
3. shear pin — bundle at 1.13.3 (sc-lint CI's current) — atm-core accepts the bump as part of migration? (1.12.0 → 1.13.x is bugfix-line churn; assumed yes.)
4. aarch64-unknown-linux-gnu target — all four tools upstream-support it (modules via first-party build); defer until a consumer runs arm64-linux CI runners?

9. Rollout plan (sprint-sized, sequential PRs in sc-lint)

1. S1 — Manifest + scripts: schema v2, loader validation, pin-external-tool, fetch-external-tools, staging/validation/notices/receipts, unit tests. (No workflow changes; releasable as no-op since manifest has no external tools yet… then the pin PR adds them.)
2. S2 — Release workflow: build-job integration (fetch + modules build + receipts), inventory extension, rehearsal run producing -rc assets; capture and land the actual pins.
3. S3 — Action: receipt verification, PATH, outputs, require-tools, fixtures.
4. S4 — Dogfood + docs: sc-lint CI switch, docs-bundle updates, CHANGELOG contract; cut the first bundled release.
5. atm-core follow-up (separate repo): migration PR per §6, after the bundled release exists.

S1–S4 are each small enough for a single dev+QA cycle; S2 is the only one touching release credentials/workflow and deserves the most careful review.

[randlee]

Live evidence for §2/§4.1 an hour after posting: on atm-core PR #1026, cargo binstall --disable-strategies quick-install,compile cargo-audit@0.22.2 hard-failed (exit 94) on windows-latest while succeeding on linux — binstall cannot resolve the rustsec monorepo slash-tag (cargo-audit/v0.22.2) windows asset. Strategy availability is per-tool-per-platform, which is exactly the resolution chaos the bundled tools/ directory removes: the sc-lint release pipeline downloads that asset by explicit URL + pinned SHA256 once, and no consumer ever runs binstall again.