Revert "ci: add nightly job to sync source mirror with pkgs in builtin" - #6245
Conversation
|
I'm happy to proceed with the revert here if that's what folks want to do.
This workflow isn't just adding the source tarballs. It's also picking up the required patches and resources that apply to missing versions. The concretization comes from
Agreed. This workflow is a good canary in the coal mine to finding binary packages and non-redistributable packages that have made it into the repository without being correctly labeled. As such I kinda lean keeping it in and fixing the packages that pop up as problems over time. |
|
Would rather revert than have this job run needlessly. GitHub clearly needs the help :). |
The source mirror analysis turned up packages we can't legally mirror (acfl was already failing this way in #6245). Add license() and redistribute(source=False, binary=False) to: - acfl, armpl-gcc: Arm EULA - aocc: AMD EULA (install already requires +license-agreed) - aqlprofile: proprietary binaries; MIT as of rocm 7.0.0, so scope to @:6 - dislin: proprietary freeware - namd: UIUC non-commercial license (was already non-redistributable) - psipred: UCL non-commercial license - py-x21: closed-source wheels - charmpp: Apache-2.0 as of v7.1; older versions are non-commercial Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Re-adds the nightly source mirror sync from #6069 (reverted in #6245). Nothing about mirroring sources requires concretization: URLs and sha256 checksums for version tarballs, resources, and patches are all known statically from `package.py` files. The `find-missing-mirror-artifacts.py` script here: 1. lists every sha256-addressed artifact in the builtin repo: - version tarballs - resources - URL patches (in packages and from dependencies) while skipping manual-download and non-redistributable packages; 2. compares digests against a listing of the mirror's content-addressed `_source-cache/archive/` prefix; and 3. emits one TSV line per missing artifact with its: - `sha256` - exact mirror path, computed with Spack's `default_mirror_layout()` - its candidate URLs in order of preference. A GitHub Actions workflow then downloads each artifact with curl, verifies the sha256, and uploads artifacts to S3 one at a time. disk usage is bounded and individual fetch failures only skip that artifact until the next nightly run. Co-authored-by: Alec Scott <alec@llnl.gov> Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Todd Gamblin <tgamblin@llnl.gov> * sync-src-mirror: fetch with spack instead of curl Merge the download/verify/upload loop into the finder script and fetch with spack's own fetch strategies via Stage.fetch()/check(). This honors per-package fetch_options, reuses spack's checksum and redirect handling, and drops the TSV/bash handoff. Uploads still go through `aws s3 cp`, one artifact at a time, so disk stays bounded. Without --upload-to, the script just lists missing artifacts, which makes it easy to run locally. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Todd Gamblin <tgamblin@llnl.gov> --------- Signed-off-by: Todd Gamblin <tgamblin@llnl.gov> Co-authored-by: Alec Scott <alec@llnl.gov>
Reverts #6069
Job is failing with:
which implies several things I missed in review:
acflreally needs aredistribute(source=False, binary=False)directive (probably along with some of the other missing packages, as it's proprietary software that we cannot mirror.