diff --git a/dev-handbook/README.md b/dev-handbook/README.md index d12aafb..9dc35cd 100644 --- a/dev-handbook/README.md +++ b/dev-handbook/README.md @@ -40,6 +40,7 @@ * [Adding support for a new distribution](add-new-distro.md) * [Adding support for a new Ruby version](add-new-ruby-version.md) + * [Archiving EOL packages](archiving-eol-packages.md) ## Organizational (for team members) diff --git a/dev-handbook/apt-yum-repo-infra.drawio.svg b/dev-handbook/apt-yum-repo-infra.drawio.svg index f125c00..3f50073 100644 --- a/dev-handbook/apt-yum-repo-infra.drawio.svg +++ b/dev-handbook/apt-yum-repo-infra.drawio.svg @@ -34,7 +34,7 @@ - fullstaq-ruby-server-edition-apt-repo + fsruby-server-edition-apt-repo
@@ -81,7 +81,7 @@
- fullstaq-ruby-server-edition-yum-repo + fsruby-server-edition-yum-repo
diff --git a/dev-handbook/apt-yum-repo.md b/dev-handbook/apt-yum-repo.md index fb265d3..c513ea2 100644 --- a/dev-handbook/apt-yum-repo.md +++ b/dev-handbook/apt-yum-repo.md @@ -48,11 +48,15 @@ Our design was made with the following requirements in mind, and addresses the f We self-host our repositories: we store them in Google Cloud Storage buckets. [Google Cloud Storage is strongly consistent.](https://cloud.google.com/storage/docs/consistency) -There are three buckets: +There are five buckets: - - `fullstaq-ruby-server-edition-apt-repo` stores the production APT repository. - - `fullstaq-ruby-server-edition-yum-repo` stores the production YUM repository. - - `fullstaq-ruby-server-edition-ci-artifacts` stores the temporary repositories created during CI runs. + - `fsruby-server-edition-apt-repo` stores the production APT repository. + - `fsruby-server-edition-yum-repo` stores the production YUM repository. + - `fsruby-server-edition-apt-archive-repo` stores APT packages whose distribution or Ruby version is EOL. + - `fsruby-server-edition-yum-archive-repo` stores YUM packages whose distribution or Ruby version is EOL. + - `fsruby-server-edition-ci-artifacts` stores the temporary repositories created during CI runs. + +The production and archive buckets never contain the same package. The archive buckets are only updated by [archiving EOL packages](archiving-eol-packages.md) — CI never writes to them. Each archival run moves packages from the production bucket into the archive bucket, creating a new version of both. The archive buckets are served at `apt-archive.fullstaqruby.org` and `yum-archive.fullstaqruby.org`. We don't let users use the production bucket URLs directly. Instead, we let users use `https://apt.fullstaqruby.org` and `https://yum.fullstaqruby.org`. These domains redirect to the appropriate bucket URLs. We do this so that we avoid strongly coupling users with Google Cloud Storage. If in the future we want to move off Google Cloud, we can do so without breaking users' URLs. @@ -144,7 +148,7 @@ A downside of this versioning approach is that each version consumes a lot of sp ### CI bucket -During CI runs, we create temporary repositories in `gs://fullstaq-ruby-server-edition-ci-artifacts/$CI_RUN_NUMBER/{apt,yum}-repo`. These directories look as follows: +During CI runs, we create temporary repositories in `gs://fsruby-server-edition-ci-artifacts/$CI_RUN_NUMBER/{apt,yum}-repo`. These directories look as follows: ~~~ /$CI_RUN_NUMBER/{apt,yum}-repo/ @@ -162,7 +166,7 @@ These directories are very similar to the production buckets' contents. But ther The CI tests run directly against this bucket URL instead of going through `https://{apt,yum}.fullstaqruby.org`. -We create a temporary repository by copying over `gs://fullstaq-ruby-server-edition-{apt,yum}-repo/versions/$LATEST_VERSION`. This way we achieve production data parity during testing. +We create a temporary repository by copying over `gs://fsruby-server-edition-{apt,yum}-repo/versions/$LATEST_VERSION`. This way we achieve production data parity during testing. ## Locking @@ -179,10 +183,10 @@ The lock's critical section is quite large, and covers: The locks are located in the following URLs: - * `gs://fullstaq-ruby-server-edition-apt-repo/locks/apt` - * `gs://fullstaq-ruby-server-edition-yum-repo/locks/yum` + * `gs://fsruby-server-edition-apt-repo/locks/apt` + * `gs://fsruby-server-edition-yum-repo/locks/yum` -When publishing to a testing repository (in `fullstaq-ruby-server-edition-ci-artifacts`) we don't perform any locking, because each CI run is guaranteed to write to its own temporary repository. +When publishing to a testing repository (in `fsruby-server-edition-ci-artifacts`) we don't perform any locking, because each CI run is guaranteed to write to its own temporary repository. ## Backups diff --git a/dev-handbook/archiving-eol-packages.md b/dev-handbook/archiving-eol-packages.md new file mode 100644 index 0000000..38d8f68 --- /dev/null +++ b/dev-handbook/archiving-eol-packages.md @@ -0,0 +1,242 @@ +# Archiving EOL packages + +This document describes how to move packages for end-of-life (EOL) distributions and EOL Ruby versions out of the live repositories and into the archive repositories. This is a routine maintenance task that frees CI disk space and keeps the live repositories lean, while keeping EOL packages installable. + +## Background + +The CI publish step downloads the full Aptly state archive (`state.tar.zst`) from Google Cloud Storage on every run. This archive grows with every distribution and Ruby version ever published. When distributions or Ruby versions reach EOL, their packages remain in the state archive indefinitely, consuming disk space on GitHub Actions runners. + +To address this, we maintain **archive repositories** alongside the live repositories: + +| Repository | Bucket | Domain | Contents | +|------------|--------|--------|----------| +| APT (live) | `fsruby-server-edition-apt-repo` | `apt.fullstaqruby.org` | Packages whose distribution *and* Ruby version are both supported | +| APT (archive) | `fsruby-server-edition-apt-archive-repo` | `apt-archive.fullstaqruby.org` | Packages whose distribution *or* Ruby version is EOL | +| YUM (live) | `fsruby-server-edition-yum-repo` | `yum.fullstaqruby.org` | Packages whose distribution *and* Ruby version are both supported | +| YUM (archive) | `fsruby-server-edition-yum-archive-repo` | `yum-archive.fullstaqruby.org` | Packages whose distribution *or* Ruby version is EOL | + +The live and archive repositories never contain the same package. Packages are moved, never deleted, so EOL packages remain installable from the archive. + +The archive only grows: packages are added to it as distributions and Ruby versions reach EOL. It uses the same versioned bucket structure as the live repositories, so each archival run creates a new archive version. + +CI never writes to the archive buckets. Packages only enter the archive through the archive scripts described below, run by hand. This is enforced in IAM (see the infra repo's `terraform/repo_buckets.tf`). + +This pattern follows the precedent set by [PostgreSQL](https://apt-archive.postgresql.org/) (`apt-archive.postgresql.org`) and [HashiCorp](https://www.hashicorp.com/en/blog/announcing-the-linux-package-archive-site) (`archive.releases.hashicorp.com`). + +## What gets archived + +Each script decides per package, based on the current `config.yml`: + + * **EOL distribution** — the distribution has a repository in the live repo, but is no longer built (see [Step 1](#step-1-remove-from-the-build-system)). *All* its packages are archived, including `fullstaq-ruby-common` and `fullstaq-rbenv`, and the distribution is removed from the live repo. + * **EOL Ruby version** — on a supported distribution, any `fullstaq-ruby-X.Y*` package where `X.Y` is not an active minor version (see below). This includes variants (`-jemalloc`, `-malloctrim`) and tiny-version packages (`fullstaq-ruby-X.Y.Z`). + +Everything else stays in the live repo. In particular: + + * `fullstaq-ruby-common` and `fullstaq-rbenv` stay on supported distributions. + * A tiny-version package stays as long as its minor version is active, even if it has been removed from `tiny_version_packages`. + * A minor version counts as active while it appears in `minor_version_packages` *or* `tiny_version_packages`. So when retiring a Ruby minor version, remove it from both, or nothing for it is archived. + +The scripts are: + + * `internal-scripts/ci-cd/archive/archive-apt-packages.rb` + * `internal-scripts/ci-cd/archive/archive-yum-packages.rb` + +Both handle EOL distributions and EOL Ruby versions in the same run, so there is no ordering to get right. + +## Procedure + +### Step 1: Remove from the build system + +For an EOL distribution: + + 1. Delete the `environments//` directory. With `distributions: all` in `config.yml` (the default), that is what removes a distribution from the build. + 2. If `config.yml` lists distributions explicitly instead, remove it from that list too. + +`distribution_exclusions` is not a way to retire a distribution: it only stops particular Ruby versions being built for it. + +For an EOL Ruby version: + + 1. Edit `config.yml` and remove the version from both `minor_version_packages` and `tiny_version_packages`. + +Then: + + 1. Regenerate CI/CD workflows: + + ~~~bash + ./internal-scripts/generate-ci-cd-yaml.rb + ~~~ + + 2. Commit and merge these changes. + +The scripts read `config.yml` from your working copy, so run them from an up-to-date checkout of `main`. + +### Step 2: Announce the change + +Users who install an EOL Ruby version, or any package on an EOL distribution, must add the archive repository after this step. Add an entry to the release notes of the next release. For example: + +> **Packages for EOL Ruby versions and distributions have moved to the archive repositories.** Packages for Ruby X.Y and for Distro N are no longer available from `apt.fullstaqruby.org` / `yum.fullstaqruby.org`. They remain installable from `apt-archive.fullstaqruby.org` / `yum-archive.fullstaqruby.org`, which use the same signing key. To keep installing them, add the archive repository alongside the regular one: +> +> APT: +> +> deb https://apt-archive.fullstaqruby.org main +> +> YUM: +> +> [fullstaq-ruby-archive] +> name=fullstaq-ruby-archive +> baseurl=https://yum-archive.fullstaqruby.org//$basearch +> gpgcheck=0 +> repo_gpgcheck=1 +> enabled=1 +> gpgkey=https://raw.githubusercontent.com/fullstaq-ruby/server-edition/main/fullstaq-ruby.asc +> sslverify=1 + +Replace `X.Y`, `Distro N` and `` with what's being archived. Use the same `Signed-By` option in the APT line as the user's existing Fullstaq Ruby line, if it has one. + +### Step 3: Run the archive scripts + +**Prerequisites:** + + * `gcloud` CLI authenticated with write access to the live and archive buckets + * `az` CLI authenticated with access to the `fsruby2infraowners` Key Vault (for the GPG signing key) + * `aptly`, `zstd` and `gpg` installed locally (for APT) + * Docker running (for `createrepo_c`, for YUM) + * Enough free disk space for the full live APT state (roughly 30 GB at the time of writing) + +**Dry run first.** A dry run makes every change locally, including the checks described in [How the scripts work](#how-the-scripts-work), and prints the list of packages it would move, but uploads nothing. It still takes the lock and downloads the full live state: + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-archive-repo \ +./internal-scripts/ci-cd/archive/archive-apt-packages.rb --dry-run +~~~ + +Review the list. Then run for real: + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-archive-repo \ +./internal-scripts/ci-cd/archive/archive-apt-packages.rb +~~~ + +Repeat for YUM: + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-archive-repo \ +./internal-scripts/ci-cd/archive/archive-yum-packages.rb --dry-run + +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-archive-repo \ +./internal-scripts/ci-cd/archive/archive-yum-packages.rb +~~~ + +The scripts hold the live repo's lock (`locks/apt` or `locks/yum`) for the whole run, which can take an hour for APT. A CI publish waits at most 5 minutes for the lock, then fails. **Don't merge to `main` or re-run CI publishing during an archival run.** If a publish fails this way, re-run it afterwards. + +### Step 4: Restart the web server + +Caddy only reads repo version numbers at startup, and nothing restarts it after a manual archival run. Until it restarts, the live domains keep serving the pre-archival version and the archive domains keep serving the previous archive version (`versions/0/` on the first run, which returns 404). + +The `/admin/restart_web_server` endpoint only accepts OIDC tokens from GitHub-hosted runners in this repo's `deploy` environment, so it can't be called by hand. Restart Caddy over SSH on the backend server instead: + +~~~bash +sudo systemctl restart caddy + +# Confirm the new versions are loaded +sudo cat /etc/caddy/env-repo-versions +~~~ + +### Step 5: Verify + +~~~bash +# Archive should serve the archived distributions +curl -fsS https://apt-archive.fullstaqruby.org/dists//Release +curl -fsS https://yum-archive.fullstaqruby.org//x86_64/repodata/repomd.xml + +# Live repo should no longer list EOL packages +curl -fsS https://apt.fullstaqruby.org/dists//main/binary-amd64/Packages | grep '^Package: fullstaq-ruby-X.Y' + +# State archive size should have decreased +gsutil ls -l gs://fsruby-server-edition-apt-repo/versions/*/state.tar.zst | tail -5 +~~~ + +## Rollback + +Each run creates a new version of both repositories. Old versions are never modified, so rolling back means pointing `latest_version.txt` back at the previous version and restarting Caddy. + +**Revert the live repo:** + +~~~bash +gsutil cat gs://fsruby-server-edition-apt-repo/versions/latest_version.txt + +echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-repo/versions/latest_version.txt +~~~ + +**Revert the archive:** + +~~~bash +gsutil cat gs://fsruby-server-edition-apt-archive-repo/versions/latest_version.txt + +echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-archive-repo/versions/latest_version.txt +~~~ + +Revert the live repo, not only the archive: reverting only the archive makes the moved packages unavailable from both. + +The same commands apply to the YUM buckets. + +## How the scripts work + +Both scripts follow the same sequence: + + 1. Take the live repo's lock. + 2. Download the latest live repo version. + 3. Decide which packages to move (see [What gets archived](#what-gets-archived)). + 4. Download the latest archive version, if any. + 5. Add the packages to the local archive copy and remove them from the local live copy. + 6. Regenerate the signed metadata of every distribution in both copies, not only the ones that changed. + 7. **Verify** that the archive's metadata lists every package being moved, and that the live repo's metadata lists none of them. The script aborts here, before uploading anything, if either check fails. + 8. Upload the archive as version M+1 and activate it. + 9. Upload the live repo as version N+1 and activate it. + +The archive is activated before the live repo, so the moved packages are always available from at least one of them. Before activating each version, the script checks that it still holds the lock. + +**Reruns are safe.** If a run fails after step 9, rerun the script: + + * packages already in the archive are skipped or re-added without change; + * the live repo is then trimmed as normal. + +A run with nothing to archive exits without uploading anything. + +### APT specifics + + * Packages are added to the archive's Aptly instance individually, with `aptly repo add -force-replace` using the `.deb` files from the live state's pool. Only the moved packages' files end up in the archive's pool. + * EOL Ruby packages are removed from live with `aptly repo remove`. EOL distributions are unpublished and dropped with `aptly publish drop` and `aptly repo drop`. + * Both instances run `aptly db cleanup`, so unreferenced files leave the pool. + * Like `publish-debs.rb`, the state archive (`state.tar.zst`) is created *before* publishing, so it contains no publications and an empty `repo/` directory. `publish-debs.rb` creates publications without dropping existing ones, so it would fail on a state that has them. And an uploaded `repo/` would contain symlinks into a temporary directory that no longer exists. + * Every distribution is republished, because the upload replaces the whole public tree with the local one. A distribution that wasn't republished would disappear from it. + * Each distribution in the archive is its own Aptly repo and publication, named after the distribution, like in the live repo. + +### YUM specifics + + * RPM files are copied into the archive's `//` directory individually. Files already present are skipped. + * `createrepo_c` (in the utility Docker image) regenerates `repodata/` for every directory that changed, in both repositories, and `repomd.xml` is re-signed with the signing key. + * EOL distribution directories are deleted from the live copy. + +## Testing + +`internal-scripts/ci-cd/archive/test/run-tests.sh` runs both scripts against a fake Google Cloud Storage inside a Docker container with the same Aptly, `createrepo_c` and GPG versions CI uses. It seeds live repositories with a supported and an EOL distribution and an active and an EOL Ruby version, then checks that: + + * a dry run uploads nothing; + * the right packages end up in each repository, and the two never overlap; + * distributions with nothing to archive are left alone; + * all metadata is signed; + * the uploaded state has no publications, and `publish-debs.rb` can still publish afterwards; + * a second run changes nothing; + * a rerun after a failure between the archive and live uploads completes correctly; + * a second archival round keeps previously archived distributions intact. + +The seeded state is built the way `publish-debs.rb` builds it. It needs only Docker. It is not run by CI. Run it after changing either script: + +~~~bash +./internal-scripts/ci-cd/archive/test/run-tests.sh +~~~ diff --git a/dev-handbook/ci-cd-resumption.md b/dev-handbook/ci-cd-resumption.md index 7b7c3a0..e4078e1 100644 --- a/dev-handbook/ci-cd-resumption.md +++ b/dev-handbook/ci-cd-resumption.md @@ -8,12 +8,12 @@ In order to aleviate this problem, we implement the ability to re-run only faile Resumption support works by checking, for each CI job, whether the artifact that that job should produce, already exists. If so, then that job can be skipped. -When you re-run a CI run, Github Actions wipes all previous state (including artifacts). Therefore, we store artifacts primarily in a Google Cloud Storage bucket ([fullstaq-ruby-server-edition-ci-artifacts](https://storage.googleapis.com/fullstaq-ruby-server-edition-ci-artifacts), part of the [infrastructure](https://github.com/fullstaq-labs/fullstaq-ruby-infra)), which isn't wiped before a re-run. +When you re-run a CI run, Github Actions wipes all previous state (including artifacts). Therefore, we store artifacts primarily in a Google Cloud Storage bucket ([fsruby-server-edition-ci-artifacts](https://storage.googleapis.com/fsruby-server-edition-ci-artifacts), part of the [infrastructure](https://github.com/fullstaq-labs/fullstaq-ruby-infra)), which isn't wiped before a re-run. Here's an example artifact URL: ~~~ -gs://fullstaq-ruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst +gs://fsruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst ~~~ Artifacts are stored on a per-CI-run basis. Thus, they always contains the CI run's number. Note that the CI run number does not change even for re-runs. @@ -23,7 +23,7 @@ At the beginning of a CI run, a job named `determine_necessary_jobs` checks whic ~~~ ##### Determine whether Rbenv DEB needs to be built ##### --> Run ./.github/actions/check-artifact-exists - Checking gs://fullstaq-ruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst + Checking gs://fsruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst Artifact exists ~~~ diff --git a/internal-scripts/ci-cd/archive/archive-apt-packages.rb b/internal-scripts/ci-cd/archive/archive-apt-packages.rb new file mode 100755 index 0000000..44539fb --- /dev/null +++ b/internal-scripts/ci-cd/archive/archive-apt-packages.rb @@ -0,0 +1,630 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Moves packages that no longer belong in the live APT repository into the +# archive APT repository. A package leaves the live repository when its +# distribution, or its Ruby minor version, is no longer listed in config.yml. +# The live and archive repositories never contain the same package. +# +# Usage: +# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-archive-repo \ +# ./internal-scripts/ci-cd/archive/archive-apt-packages.rb [--dry-run] +# +# Safe to rerun: packages already in the archive are re-added as a no-op, +# and the live repository is only modified after the archive has been +# uploaded and activated. +# +# See dev-handbook/archiving-eol-packages.md. + +require_relative '../../../lib/gcloud_storage_lock' +require_relative '../../../lib/ci_workflow_support' +require_relative '../../../lib/shell_scripting_support' +require_relative '../../../lib/publishing_support' +require 'json' +require 'shellwords' +require 'tmpdir' +require 'fileutils' +require 'optparse' + +class ArchiveAptPackages + REPO_ORIGIN = 'Fullstaq-Ruby' + REPO_LABEL = 'Fullstaq-Ruby' + + # Matches: fullstaq-ruby-3.1, fullstaq-ruby-3.1-jemalloc, fullstaq-ruby-3.1.7, etc. + # Does NOT match: fullstaq-ruby-common, fullstaq-rbenv + RUBY_PACKAGE_PATTERN = /\Afullstaq-ruby-(\d+\.\d+)/ + + include CiWorkflowSupport + include ShellScriptingSupport + include PublishingSupport + + def main + parse_options + require_envvar 'PRODUCTION_REPO_BUCKET_NAME' + require_envvar 'ARCHIVE_REPO_BUCKET_NAME' + + print_header 'Initializing' + load_config + create_temp_dirs + begin + ensure_gpg_state_isolated + activate_wrappers_bin_dir + initialize_locking + initialize_aptly(@live_aptly_config_path, @live_state_path) + initialize_aptly(@archive_aptly_config_path, @archive_state_path) + fetch_and_import_signing_key + + synchronize do + print_header 'Fetching live repository state' + @live_version = get_latest_version(live_bucket) + abort 'ERROR: No production repository exists yet' if @live_version == 0 + fetch_state(live_bucket, @live_version, @live_state_path) + + print_header 'Determining packages to archive' + @moves = determine_moves + if @moves.empty? + log_notice 'Nothing to archive' + return + end + print_moves + + print_header 'Fetching archive repository state' + @archive_version = get_latest_version(archive_bucket) + if @archive_version > 0 + fetch_state(archive_bucket, @archive_version, @archive_state_path) + else + log_notice 'No archive exists yet, creating a new one' + end + + print_header 'Moving packages' + add_to_archive + remove_from_live + compact_aptly_db(@archive_aptly_config_path) + compact_aptly_db(@live_aptly_config_path) + + # Like publish-debs.rb, the state is archived *before* publishing, + # so that it contains no publications and an empty repo/ directory. + # publish-debs.rb creates publications without dropping them first, + # which fails if the state already has one. + if !@dry_run + print_header 'Creating state archives' + create_state_tarball(@archive_state_path, @archive_state_tarball_path) + create_state_tarball(@live_state_path, @live_state_tarball_path) + end + check_lock_health + + # Every distribution is published, not only the ones that changed: + # the upload replaces the entire public tree with repo/. + print_header 'Publishing repositories' + publish_all(@archive_aptly_config_path) + publish_all(@live_aptly_config_path) + verify_archive + verify_live + check_lock_health + + if @dry_run + log_notice 'Dry run: not uploading changes' + return + end + + # The archive must be complete and activated before the live + # repository stops serving these packages. + print_header 'Uploading archive repository' + upload_version(archive_bucket, @archive_version, @archive_state_path, @archive_state_tarball_path) + + print_header 'Uploading live repository' + upload_version(live_bucket, @live_version, @live_state_path, @live_state_tarball_path) + + print_header 'Success!' + print_summary + end + ensure + cleanup + end + end + +private + def parse_options + @dry_run = false + OptionParser.new do |opts| + opts.banner = "Usage: #{$0} [options]" + opts.on('--dry-run', 'Make all changes locally, but do not upload them') { @dry_run = true } + end.parse! + end + + def live_bucket + ENV['PRODUCTION_REPO_BUCKET_NAME'] + end + + def archive_bucket + ENV['ARCHIVE_REPO_BUCKET_NAME'] + end + + def create_temp_dirs + log_notice 'Creating temporary directories' + # Under /tmp, not $TMPDIR: on macOS $TMPDIR is too long for gpg-agent's + # socket path, and the GPG home directory lives here. + @temp_dir = Dir.mktmpdir('archive-apt', '/tmp') + @wrappers_bin_dir = "#{@temp_dir}/wrappers" + @signing_key_path = "#{@temp_dir}/key.gpg" + + @live_aptly_config_path = "#{@temp_dir}/aptly-live.conf" + @live_state_path = "#{@temp_dir}/live-state" + @live_state_tarball_path = "#{@temp_dir}/live-state.tar.zst" + + @archive_aptly_config_path = "#{@temp_dir}/aptly-archive.conf" + @archive_state_path = "#{@temp_dir}/archive-state" + @archive_state_tarball_path = "#{@temp_dir}/archive-state.tar.zst" + + Dir.mkdir(@wrappers_bin_dir) + [@live_state_path, @archive_state_path].each do |state_path| + FileUtils.mkdir_p("#{state_path}/db") + FileUtils.mkdir_p("#{state_path}/repo") + end + end + + def ensure_gpg_state_isolated + # Aptly invokes 'gpg' from PATH. We make that use an isolated + # home directory, so that only the signing key is available. + log_notice 'Creating GPG wrapper' + File.open("#{@wrappers_bin_dir}/gpg", 'w:utf-8') do |f| + f.write("#!/bin/sh\n") + f.write( + sprintf( + "exec %s --homedir %s \"$@\"\n", + Shellwords.escape(find_gpg), + Shellwords.escape(@temp_dir) + ) + ) + end + File.chmod(0755, "#{@wrappers_bin_dir}/gpg") + end + + def find_gpg + ENV['PATH'].split(':').each do |dir| + next if dir == @wrappers_bin_dir + candidate = "#{dir}/gpg" + return candidate if File.exist?(candidate) + end + abort('GPG not found') + end + + def activate_wrappers_bin_dir + ENV['PATH'] = "#{@wrappers_bin_dir}:#{ENV['PATH']}" + end + + def initialize_locking + # The archive bucket has no lock of its own. It is only ever written + # while holding the live repository's lock. + @lock = GCloudStorageLock.new(url: "gs://#{live_bucket}/locks/apt") + end + + def synchronize(&block) + @lock.synchronize(&block) + end + + def check_lock_health + abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? + end + + def initialize_aptly(config_path, state_path) + log_notice "Creating Aptly config file #{config_path}" + File.open(config_path, 'w:utf-8') do |f| + f.write(JSON.generate( + rootDir: state_path, + FileSystemPublishEndpoints: { + main: { + rootDir: "#{state_path}/repo", + linkMethod: 'symlink', + verifyMethod: 'md5' + } + } + )) + end + end + + def fetch_and_import_signing_key + log_notice 'Fetching and importing signing key' + File.open(@signing_key_path, 'wb') do |f| + f.write(fetch_signing_key) + end + @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) + log_info "Signing key ID: #{@gpg_key_id}" + import_gpg_key(@temp_dir, @signing_key_path) + end + + def get_latest_version(bucket) + url = "gs://#{bucket}/versions/latest_version.txt" + stdout_output, stderr_output, status = run_command_capture_output( + 'gsutil', 'cp', url, '-', + log_invocation: false, + check_error: false + ) + if status.success? + version = stdout_output.strip + abort("ERROR: invalid version number stored in #{url}") if version !~ /\A[0-9]+\Z/ + version = version.to_i + elsif stderr_output =~ /No URLs matched/ + version = 0 + else + abort("ERROR: error fetching #{url}: #{stderr_output.chomp}") + end + log_notice "Latest version of #{bucket}: #{version}" + version + end + + def fetch_state(bucket, version, state_path) + log_notice "Fetching state of #{bucket} version #{version}" + run_bash( + sprintf('gsutil -m cp %s - | zstd -dc | tar -xC %s', + Shellwords.escape("gs://#{bucket}/versions/#{version}/state.tar.zst"), + Shellwords.escape(state_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def supported_distros + distributions.find_all { |d| d[:package_format] == :DEB }.map { |d| d[:name] } + end + + # A minor version is active while CI still builds any package for it: + # its minor-version package, or one of its tiny-version packages. + # Archiving a package CI still builds would put it in both repositories. + def active_ruby_minor_versions + minors = (config[:ruby][:minor_version_packages] || []).map { |p| p[:minor_version] } + tinys = (config[:ruby][:tiny_version_packages] || []).map { |p| p[:full_version].split('.')[0..1].join('.') } + (minors + tinys).uniq + end + + def eol_ruby_package?(package_key, active_minors) + package_key =~ RUBY_PACKAGE_PATTERN && !active_minors.include?($1) + end + + # Returns a hash: distro => { whole_distro: Boolean, packages: [package keys] } + def determine_moves + supported = supported_distros + active_minors = active_ruby_minor_versions + log_info "Supported DEB distributions: #{supported.join(', ')}" + log_info "Active Ruby minor versions: #{active_minors.join(', ')}" + + moves = {} + list_aptly_repos(@live_aptly_config_path).each do |distro| + packages = list_aptly_packages(@live_aptly_config_path, distro) + if !supported.include?(distro) + moves[distro] = { whole_distro: true, packages: packages } + else + eol_packages = packages.find_all { |key| eol_ruby_package?(key, active_minors) } + moves[distro] = { whole_distro: false, packages: eol_packages } if !eol_packages.empty? + end + end + moves + end + + def print_moves + @moves.each_pair do |distro, move| + reason = move[:whole_distro] ? 'EOL distribution' : 'EOL Ruby versions' + log_notice "[#{distro}] #{move[:packages].size} packages to archive (#{reason})" + move[:packages].each { |key| log_info " #{YELLOW}ARCHIVE#{RESET} #{key}" } + end + end + + def add_to_archive + existing_repos = list_aptly_repos(@archive_aptly_config_path) + + @moves.each_pair do |distro, move| + next if move[:packages].empty? + + log_notice "[#{distro}] Adding #{move[:packages].size} packages to archive" + if !existing_repos.include?(distro) + run_command( + 'aptly', 'repo', 'create', + "-config=#{@archive_aptly_config_path}", + distro, + log_invocation: true, + check_error: true + ) + end + + move[:packages].each_slice(50) do |batch| + # -force-replace makes re-adding a package that is already in + # the archive a no-op, which is what makes reruns safe. + run_command( + 'aptly', 'repo', 'add', '-force-replace', + "-config=#{@archive_aptly_config_path}", + distro, + *live_pool_paths(batch), + log_invocation: false, + check_error: true + ) + end + end + end + + # Paths of the .deb files in the live state's pool for the given package keys. + def live_pool_paths(package_keys) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'package', 'show', + "-config=#{@live_aptly_config_path}", + '-with-files', + package_keys.join(' | '), + log_invocation: false, + check_error: true + ) + paths = stdout_output.scan(/^Files in the pool:\n((?: .+\n?)+)/).flat_map do |(block)| + block.split("\n").map(&:strip) + end + if paths.size != package_keys.size || !paths.all? { |path| File.exist?(path) } + abort("ERROR: expected #{package_keys.size} pool files, found #{paths.size}: #{package_keys.join(', ')}") + end + paths + end + + def verify_archive + log_notice 'Verifying archive repository' + @moves.each_pair do |distro, move| + next if move[:packages].empty? + + archived = list_aptly_packages(@archive_aptly_config_path, distro) + missing = move[:packages] - archived + if !missing.empty? + abort("ERROR: [#{distro}] packages missing from archive: #{missing.join(', ')}") + end + + published = published_packages("#{@archive_state_path}/repo", distro) + missing = move[:packages] - published + if !missing.empty? + abort("ERROR: [#{distro}] packages missing from published archive: #{missing.join(', ')}") + end + log_info "[#{distro}] All #{move[:packages].size} packages present in archive" + end + end + + def remove_from_live + @moves.each_pair do |distro, move| + if move[:whole_distro] + log_notice "[#{distro}] Dropping distribution from live repository" + drop_publication(@live_aptly_config_path, distro) + run_command( + 'aptly', 'repo', 'drop', + "-config=#{@live_aptly_config_path}", + distro, + log_invocation: true, + check_error: true + ) + else + log_notice "[#{distro}] Removing #{move[:packages].size} packages from live repository" + move[:packages].each_slice(50) do |batch| + run_command( + 'aptly', 'repo', 'remove', + "-config=#{@live_aptly_config_path}", + distro, + batch.join(' | '), + log_invocation: false, + check_error: true + ) + end + end + end + end + + def publish_all(config_path) + list_aptly_repos(config_path).each do |distro| + log_notice "[#{distro}] Publishing #{config_path}" + publish_aptly_repo(config_path, distro) + end + end + + def verify_live + log_notice 'Verifying live repository' + remaining_repos = list_aptly_repos(@live_aptly_config_path) + @moves.each_pair do |distro, move| + if move[:whole_distro] + abort("ERROR: [#{distro}] still present in live repository") if remaining_repos.include?(distro) + if File.exist?("#{@live_state_path}/repo/dists/#{distro}") + abort("ERROR: [#{distro}] still published in live repository") + end + else + leftover = move[:packages] & list_aptly_packages(@live_aptly_config_path, distro) + leftover |= move[:packages] & published_packages("#{@live_state_path}/repo", distro) + if !leftover.empty? + abort("ERROR: [#{distro}] packages still in live repository: #{leftover.join(', ')}") + end + end + end + end + + # Package keys (name_version_arch) listed in a published distribution's + # Packages indexes. This is what APT clients actually see. + def published_packages(repo_path, distro) + Dir["#{repo_path}/dists/#{distro}/*/binary-*/Packages"].flat_map do |path| + File.read(path).split("\n\n").map do |stanza| + fields = stanza.scan(/^(Package|Version|Architecture): (.+)$/).to_h + next if fields.empty? + "#{fields['Package']}_#{fields['Version']}_#{fields['Architecture']}" + end.compact + end.uniq + end + + def list_aptly_repos(config_path) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'repo', 'list', + "-config=#{config_path}", + '-raw', + log_invocation: false, + check_error: true + ) + stdout_output.split("\n").map(&:strip).reject(&:empty?) + end + + def list_aptly_packages(config_path, distro) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'repo', 'show', + "-config=#{config_path}", + '-with-packages', + distro, + log_invocation: false, + check_error: true + ) + stdout_output.sub(/.*^Packages:$/m, '').split("\n").map(&:strip).reject(&:empty?) + end + + def published?(config_path, distro) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'publish', 'list', + "-config=#{config_path}", + '-raw', + log_invocation: false, + check_error: true + ) + stdout_output.split("\n").any? { |line| line.split(' ') == ['filesystem:main:.', distro] } + end + + def drop_publication(config_path, distro) + return if !published?(config_path, distro) + run_command( + 'aptly', 'publish', 'drop', + "-config=#{config_path}", + distro, 'filesystem:main:.', + log_invocation: true, + check_error: true + ) + end + + def publish_aptly_repo(config_path, distro) + # 'aptly publish repo' refuses to overwrite an existing publication, + # and 'publish update' doesn't add architectures that weren't + # published before. So we drop and recreate the publication. + drop_publication(config_path, distro) + + publish_command = [ + 'aptly', 'publish', 'repo', + '-batch', '-force-overwrite', + "-config=#{config_path}", + "-gpg-key=#{@gpg_key_id}", + "-distribution=#{distro}", + "-origin=#{REPO_ORIGIN}", + "-label=#{REPO_LABEL}", + ] + _, stderr_output, status = run_command_capture_output( + *publish_command, distro, 'filesystem:main:.', + log_invocation: true, + check_error: false + ) + return if status.success? + + if stderr_output =~ /unable to figure out list of architectures/ + run_command( + *publish_command, '-architectures=all', distro, 'filesystem:main:.', + log_invocation: true, + check_error: true + ) + else + abort("ERROR publishing #{distro}: #{stderr_output.chomp}") + end + end + + def compact_aptly_db(config_path) + log_notice "Compacting #{config_path}" + run_command( + 'aptly', 'db', 'cleanup', + "-config=#{config_path}", + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def create_state_tarball(state_path, tarball_path) + log_notice "Creating #{tarball_path}" + run_bash( + sprintf("tar -C %s -cf - . | zstd -T0 > %s", + Shellwords.escape(state_path), + Shellwords.escape(tarball_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def upload_version(bucket, old_version, state_path, tarball_path) + new_version = old_version + 1 + version_url = "gs://#{bucket}/versions/#{new_version}" + log_notice "Uploading #{bucket} version #{new_version}" + + run_command( + 'gsutil', '-h', 'Cache-Control:public', + 'cp', tarball_path, "#{version_url}/state.tar.zst", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + if old_version > 0 + # Server-side copy first, so that the upload below only transfers changes. + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "gs://#{bucket}/versions/#{old_version}/public", + "#{version_url}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "#{state_path}/repo", + "#{version_url}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + write_gcs_text("#{version_url}/version.txt", new_version, 'public') + check_lock_health + log_notice "Activating #{bucket} version #{new_version}" + write_gcs_text("gs://#{bucket}/versions/latest_version.txt", new_version, 'no-store') + end + + def write_gcs_text(url, content, cache_control) + run_bash( + sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:%s cp - %s <<<%s', + cache_control, + Shellwords.escape(url), + Shellwords.escape(content.to_s)), + log_invocation: true, + check_error: true, + pipefail: false + ) + end + + def print_summary + log_notice 'Summary' + @moves.each_pair do |distro, move| + suffix = move[:whole_distro] ? ' (distribution removed from live)' : '' + log_info "#{distro}: archived #{move[:packages].size} packages#{suffix}" + end + log_info "Live repository: version #{@live_version} -> #{@live_version + 1}" + log_info "Archive repository: version #{@archive_version} -> #{@archive_version + 1}" + log_info '' + log_info 'Next steps:' + log_info ' 1. Restart Caddy on the backend server: sudo systemctl restart caddy' + log_info ' Until then, apt.fullstaqruby.org and apt-archive.fullstaqruby.org serve the old versions.' + log_info ' 2. Verify: curl -fsS https://apt-archive.fullstaqruby.org/dists//Release' + end + + def cleanup + return if @temp_dir.nil? + log_info "Cleaning up #{@temp_dir}" + FileUtils.remove_entry_secure(@temp_dir) + end +end + +ArchiveAptPackages.new.main diff --git a/internal-scripts/ci-cd/archive/archive-yum-packages.rb b/internal-scripts/ci-cd/archive/archive-yum-packages.rb new file mode 100755 index 0000000..41adf6e --- /dev/null +++ b/internal-scripts/ci-cd/archive/archive-yum-packages.rb @@ -0,0 +1,421 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Moves packages that no longer belong in the live YUM repository into the +# archive YUM repository. A package leaves the live repository when its +# distribution, or its Ruby minor version, is no longer listed in config.yml. +# The live and archive repositories never contain the same package. +# +# Usage: +# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-archive-repo \ +# ./internal-scripts/ci-cd/archive/archive-yum-packages.rb [--dry-run] +# +# Safe to rerun: RPMs already in the archive are skipped, and the live +# repository is only modified after the archive has been uploaded and +# activated. +# +# See dev-handbook/archiving-eol-packages.md. + +require_relative '../../../lib/gcloud_storage_lock' +require_relative '../../../lib/ci_workflow_support' +require_relative '../../../lib/general_support' +require_relative '../../../lib/shell_scripting_support' +require_relative '../../../lib/publishing_support' +require 'shellwords' +require 'tmpdir' +require 'fileutils' +require 'optparse' +require 'zlib' + +class ArchiveYumPackages + # Matches: fullstaq-ruby-3.1-rev5-el9.x86_64.rpm, fullstaq-ruby-3.1-jemalloc-..., fullstaq-ruby-3.1.7-... + # Does NOT match: fullstaq-ruby-common-*, fullstaq-rbenv-* + RUBY_RPM_PATTERN = /\Afullstaq-ruby-(\d+\.\d+)/ + + include CiWorkflowSupport + include GeneralSupport + include ShellScriptingSupport + include PublishingSupport + + def main + parse_options + require_envvar 'PRODUCTION_REPO_BUCKET_NAME' + require_envvar 'ARCHIVE_REPO_BUCKET_NAME' + + print_header 'Initializing' + load_config + create_temp_dirs + begin + pull_utility_image_if_not_exists + initialize_locking + fetch_and_import_signing_key + + synchronize do + print_header 'Fetching live repository' + @live_version = get_latest_version(live_bucket) + abort 'ERROR: No production repository exists yet' if @live_version == 0 + fetch_repo(live_bucket, @live_version, @live_repo_path) + + print_header 'Determining packages to archive' + @moves = determine_moves + if @moves.empty? + log_notice 'Nothing to archive' + return + end + print_moves + + print_header 'Fetching archive repository' + @archive_version = get_latest_version(archive_bucket) + if @archive_version > 0 + fetch_repo(archive_bucket, @archive_version, @archive_repo_path) + else + log_notice 'No archive exists yet, creating a new one' + end + + print_header 'Adding packages to archive repository' + add_to_archive + verify_archive + check_lock_health + + print_header 'Removing packages from live repository' + remove_from_live + verify_live + check_lock_health + + if @dry_run + log_notice 'Dry run: not uploading changes' + return + end + + # The archive must be complete and activated before the live + # repository stops serving these packages. + print_header 'Uploading archive repository' + upload_version(archive_bucket, @archive_version, @archive_repo_path) + + print_header 'Uploading live repository' + upload_version(live_bucket, @live_version, @live_repo_path) + + print_header 'Success!' + print_summary + end + ensure + cleanup + end + end + +private + def parse_options + @dry_run = false + OptionParser.new do |opts| + opts.banner = "Usage: #{$0} [options]" + opts.on('--dry-run', 'Make all changes locally, but do not upload them') { @dry_run = true } + end.parse! + end + + def live_bucket + ENV['PRODUCTION_REPO_BUCKET_NAME'] + end + + def archive_bucket + ENV['ARCHIVE_REPO_BUCKET_NAME'] + end + + def create_temp_dirs + log_notice 'Creating temporary directories' + # Under /tmp, not $TMPDIR: on macOS $TMPDIR is too long for gpg-agent's + # socket path, and the GPG home directory lives here. + @temp_dir = Dir.mktmpdir('archive-yum', '/tmp') + @signing_key_path = "#{@temp_dir}/key.gpg" + @live_repo_path = "#{@temp_dir}/live-repo" + @archive_repo_path = "#{@temp_dir}/archive-repo" + Dir.mkdir(@live_repo_path) + Dir.mkdir(@archive_repo_path) + end + + def initialize_locking + # The archive bucket has no lock of its own. It is only ever written + # while holding the live repository's lock. + @lock = GCloudStorageLock.new(url: "gs://#{live_bucket}/locks/yum") + end + + def synchronize(&block) + @lock.synchronize(&block) + end + + def check_lock_health + abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? + end + + def fetch_and_import_signing_key + log_notice 'Fetching and importing signing key' + File.open(@signing_key_path, 'wb') do |f| + f.write(fetch_signing_key) + end + @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) + log_info "Signing key ID: #{@gpg_key_id}" + import_gpg_key(@temp_dir, @signing_key_path) + end + + def get_latest_version(bucket) + url = "gs://#{bucket}/versions/latest_version.txt" + stdout_output, stderr_output, status = run_command_capture_output( + 'gsutil', 'cp', url, '-', + log_invocation: false, + check_error: false + ) + if status.success? + version = stdout_output.strip + abort("ERROR: invalid version number stored in #{url}") if version !~ /\A[0-9]+\Z/ + version = version.to_i + elsif stderr_output =~ /No URLs matched/ + version = 0 + else + abort("ERROR: error fetching #{url}: #{stderr_output.chomp}") + end + log_notice "Latest version of #{bucket}: #{version}" + version + end + + def fetch_repo(bucket, version, repo_path) + log_notice "Fetching #{bucket} version #{version}" + run_command( + 'gsutil', '-m', 'rsync', '-r', + "gs://#{bucket}/versions/#{version}/public", + repo_path, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def supported_distros + distributions.find_all { |d| d[:package_format] == :RPM }.map { |d| d[:name] } + end + + # A minor version is active while CI still builds any package for it: + # its minor-version package, or one of its tiny-version packages. + # Archiving a package CI still builds would put it in both repositories. + def active_ruby_minor_versions + minors = (config[:ruby][:minor_version_packages] || []).map { |p| p[:minor_version] } + tinys = (config[:ruby][:tiny_version_packages] || []).map { |p| p[:full_version].split('.')[0..1].join('.') } + (minors + tinys).uniq + end + + def eol_ruby_rpm?(filename, active_minors) + filename =~ RUBY_RPM_PATTERN && !active_minors.include?($1) + end + + def distro_dirs(repo_path) + Dir["#{repo_path}/*"].find_all { |path| File.directory?(path) }.map { |path| File.basename(path) }.sort + end + + def arch_dirs(repo_path, distro) + Dir["#{repo_path}/#{distro}/*"].find_all { |path| File.directory?(path) }.map { |path| File.basename(path) }.sort + end + + # Returns a hash: distro => { whole_distro: Boolean, rpms: { arch => [filenames] } } + def determine_moves + supported = supported_distros + active_minors = active_ruby_minor_versions + log_info "Supported RPM distributions: #{supported.join(', ')}" + log_info "Active Ruby minor versions: #{active_minors.join(', ')}" + + moves = {} + distro_dirs(@live_repo_path).each do |distro| + whole_distro = !supported.include?(distro) + rpms = {} + arch_dirs(@live_repo_path, distro).each do |arch| + filenames = Dir["#{@live_repo_path}/#{distro}/#{arch}/*.rpm"].map { |path| File.basename(path) }.sort + filenames = filenames.find_all { |filename| eol_ruby_rpm?(filename, active_minors) } if !whole_distro + rpms[arch] = filenames if !filenames.empty? + end + moves[distro] = { whole_distro: whole_distro, rpms: rpms } if whole_distro || !rpms.empty? + end + moves + end + + def print_moves + @moves.each_pair do |distro, move| + reason = move[:whole_distro] ? 'EOL distribution' : 'EOL Ruby versions' + count = move[:rpms].values.sum(&:size) + log_notice "[#{distro}] #{count} packages to archive (#{reason})" + move[:rpms].each_pair do |arch, filenames| + filenames.each { |filename| log_info " #{YELLOW}ARCHIVE#{RESET} #{arch}/#{filename}" } + end + end + end + + def add_to_archive + @moves.each_pair do |distro, move| + move[:rpms].each_pair do |arch, filenames| + target_dir = "#{@archive_repo_path}/#{distro}/#{arch}" + FileUtils.mkdir_p(target_dir) + copied = 0 + filenames.each do |filename| + target_path = "#{target_dir}/#{filename}" + next if File.exist?(target_path) + hardlink_or_copy_file("#{@live_repo_path}/#{distro}/#{arch}/#{filename}", target_path) + copied += 1 + end + log_notice "[#{distro}/#{arch}] Copied #{copied} packages to archive (#{filenames.size - copied} already present)" + regenerate_repo_metadata(target_dir) + end + end + end + + def verify_archive + log_notice 'Verifying archive repository' + @moves.each_pair do |distro, move| + move[:rpms].each_pair do |arch, filenames| + dir = "#{@archive_repo_path}/#{distro}/#{arch}" + missing = filenames - indexed_rpms(dir) + if !missing.empty? + abort("ERROR: [#{distro}/#{arch}] packages missing from archive metadata: #{missing.join(', ')}") + end + log_info "[#{distro}/#{arch}] All #{filenames.size} packages present in archive" + end + end + end + + def remove_from_live + @moves.each_pair do |distro, move| + if move[:whole_distro] + log_notice "[#{distro}] Removing distribution from live repository" + FileUtils.rm_rf("#{@live_repo_path}/#{distro}") + else + move[:rpms].each_pair do |arch, filenames| + dir = "#{@live_repo_path}/#{distro}/#{arch}" + log_notice "[#{distro}/#{arch}] Removing #{filenames.size} packages from live repository" + filenames.each { |filename| File.delete("#{dir}/#{filename}") } + regenerate_repo_metadata(dir) + end + end + end + end + + def verify_live + log_notice 'Verifying live repository' + @moves.each_pair do |distro, move| + if move[:whole_distro] + abort("ERROR: [#{distro}] still present in live repository") if File.exist?("#{@live_repo_path}/#{distro}") + else + move[:rpms].each_pair do |arch, filenames| + dir = "#{@live_repo_path}/#{distro}/#{arch}" + leftover = filenames & (indexed_rpms(dir) + Dir["#{dir}/*.rpm"].map { |path| File.basename(path) }) + if !leftover.empty? + abort("ERROR: [#{distro}/#{arch}] packages still in live repository: #{leftover.join(', ')}") + end + end + end + end + end + + # RPM filenames listed in a directory's repodata. This is what YUM clients actually see. + def indexed_rpms(dir) + primary = Dir["#{dir}/repodata/*-primary.xml.gz"] + return [] if primary.empty? + abort("ERROR: multiple primary.xml.gz files in #{dir}/repodata") if primary.size > 1 + xml = Zlib::GzipReader.open(primary[0], &:read) + xml.scan(/ 0 + # Server-side copy first, so that the upload below only transfers changes. + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "gs://#{bucket}/versions/#{old_version}/public", + "#{version_url}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + repo_path, + "#{version_url}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + write_gcs_text("#{version_url}/version.txt", new_version, 'public') + check_lock_health + log_notice "Activating #{bucket} version #{new_version}" + write_gcs_text("gs://#{bucket}/versions/latest_version.txt", new_version, 'no-store') + end + + def write_gcs_text(url, content, cache_control) + run_bash( + sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:%s cp - %s <<<%s', + cache_control, + Shellwords.escape(url), + Shellwords.escape(content.to_s)), + log_invocation: true, + check_error: true, + pipefail: false + ) + end + + def print_summary + log_notice 'Summary' + @moves.each_pair do |distro, move| + suffix = move[:whole_distro] ? ' (distribution removed from live)' : '' + log_info "#{distro}: archived #{move[:rpms].values.sum(&:size)} packages#{suffix}" + end + log_info "Live repository: version #{@live_version} -> #{@live_version + 1}" + log_info "Archive repository: version #{@archive_version} -> #{@archive_version + 1}" + log_info '' + log_info 'Next steps:' + log_info ' 1. Restart Caddy on the backend server: sudo systemctl restart caddy' + log_info ' Until then, yum.fullstaqruby.org and yum-archive.fullstaqruby.org serve the old versions.' + log_info ' 2. Verify: curl -fsS https://yum-archive.fullstaqruby.org///repodata/repomd.xml' + end + + def cleanup + return if @temp_dir.nil? + log_info "Cleaning up #{@temp_dir}" + FileUtils.remove_entry_secure(@temp_dir) + end +end + +ArchiveYumPackages.new.main diff --git a/internal-scripts/ci-cd/archive/test/Dockerfile b/internal-scripts/ci-cd/archive/test/Dockerfile new file mode 100644 index 0000000..9408e09 --- /dev/null +++ b/internal-scripts/ci-cd/archive/test/Dockerfile @@ -0,0 +1,16 @@ +# Test environment for the archive scripts. Mirrors the CI publish job: +# Aptly comes from the Aptly repo (like install-aptly.sh), createrepo_c +# and GPG from Ubuntu. gsutil, az and docker are replaced by the fakes +# in fake-bin/. +FROM ubuntu:22.04 + +RUN set -x && \ + apt update && \ + apt install -y ca-certificates curl gpg ruby createrepo-c rpm \ + bzip2 xz-utils zstd && \ + echo 'deb http://repo.aptly.info/ squeeze main' > /etc/apt/sources.list.d/aptly.list && \ + curl -fsSL https://www.aptly.info/pubkey.txt | gpg --dearmor -o /etc/apt/trusted.gpg.d/aptly.gpg && \ + apt update && \ + apt install -y aptly && \ + apt clean && \ + rm -rf /var/lib/apt/lists/* diff --git a/internal-scripts/ci-cd/archive/test/fake-bin/az b/internal-scripts/ci-cd/archive/test/fake-bin/az new file mode 100755 index 0000000..e2e2781 --- /dev/null +++ b/internal-scripts/ci-cd/archive/test/fake-bin/az @@ -0,0 +1,4 @@ +#!/bin/bash +# Fake 'az keyvault secret show' for tests: prints the test signing key. +set -euo pipefail +cat "${FAKE_SIGNING_KEY_PATH}" diff --git a/internal-scripts/ci-cd/archive/test/fake-bin/docker b/internal-scripts/ci-cd/archive/test/fake-bin/docker new file mode 100755 index 0000000..0c40580 --- /dev/null +++ b/internal-scripts/ci-cd/archive/test/fake-bin/docker @@ -0,0 +1,38 @@ +#!/bin/bash +# Fake docker for tests. 'docker inspect' succeeds, so the utility image +# is never pulled. 'docker run ... -v DIR:/input... IMAGE createrepo_c ARGS' +# runs createrepo_c locally against DIR. +set -euo pipefail + +case "$1" in + inspect) + exit 0 + ;; + run) + shift + input_dir= + while [[ $# -gt 0 ]]; do + case "$1" in + --rm) shift ;; + --user) shift 2 ;; + -v) input_dir="${2%%:*}"; shift 2 ;; + createrepo_c) break ;; + *) shift ;; + esac + done + shift + args=() + for arg in "$@"; do + if [[ "${arg}" == /input ]]; then + args+=("${input_dir}") + else + args+=("${arg}") + fi + done + exec createrepo_c "${args[@]}" + ;; + *) + echo "fake docker: unsupported command: $*" >&2 + exit 2 + ;; +esac diff --git a/internal-scripts/ci-cd/archive/test/fake-bin/gsutil b/internal-scripts/ci-cd/archive/test/fake-bin/gsutil new file mode 100755 index 0000000..8b8d590 --- /dev/null +++ b/internal-scripts/ci-cd/archive/test/fake-bin/gsutil @@ -0,0 +1,79 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Minimal gsutil replacement for tests. Maps gs://bucket/path to +# $FAKE_GCS_ROOT/bucket/path. Supports only the invocations used by the +# archive scripts and GCloudStorageLock. + +require 'fileutils' +require 'time' + +ROOT = ENV.fetch('FAKE_GCS_ROOT') + +def local_path(url) + raise "not a gs:// URL: #{url}" if url !~ %r{\Ags://(.+)\z} + File.join(ROOT, $1) +end + +def gs?(arg) + arg.start_with?('gs://') +end + +def no_match(url) + warn "CommandException: No URLs matched: #{url}" + exit 1 +end + +args = ARGV.dup +# Drop global options: -m, -q, and -h
. +headers = [] +while (arg = args.first) && arg.start_with?('-') + args.shift + headers << args.shift if arg == '-h' +end + +command = args.shift +case command +when 'cp' + src, dst = args + if src == '-' + path = local_path(dst) + precondition = headers.find { |h| h.start_with?('x-goog-if-generation-match:') } + if precondition && precondition.split(':', 2)[1] == '0' && File.exist?(path) + warn 'PreconditionException: 412 Precondition Failed' + exit 1 + end + FileUtils.mkdir_p(File.dirname(path)) + File.binwrite(path, $stdin.read) + elsif dst == '-' + path = local_path(src) + no_match(src) if !File.file?(path) + $stdout.write(File.binread(path)) + else + path = local_path(dst) + FileUtils.mkdir_p(File.dirname(path)) + FileUtils.cp(src, path) + end +when 'rsync' + delete = args.delete('-d') + args.delete('-r') + src, dst = args.map { |arg| gs?(arg) ? local_path(arg) : arg } + FileUtils.mkdir_p(dst) + FileUtils.rm_rf(Dir.glob("#{dst}/*", File::FNM_DOTMATCH).reject { |p| p.end_with?('/.', '/..') }) if delete + # cp -a would copy symlinks as symlinks; real gsutil uploads their targets. + system('cp', '-rL', "#{src}/.", dst, exception: true) if File.exist?(src) +when 'stat' + path = local_path(args[0]) + no_match(args[0]) if !File.exist?(path) + stat = File.stat(path) + puts args[0] + puts " Update time: #{stat.mtime.utc.httpdate}" + puts " Generation: #{stat.mtime.to_f.to_s.delete('.')}#{stat.ino}" +when 'rm' + path = local_path(args[0]) + no_match(args[0]) if !File.exist?(path) + File.delete(path) +else + warn "fake gsutil: unsupported command: #{ARGV.join(' ')}" + exit 2 +end diff --git a/internal-scripts/ci-cd/archive/test/run-tests.sh b/internal-scripts/ci-cd/archive/test/run-tests.sh new file mode 100755 index 0000000..ddf39f0 --- /dev/null +++ b/internal-scripts/ci-cd/archive/test/run-tests.sh @@ -0,0 +1,422 @@ +#!/bin/bash +# Tests archive-apt-packages.rb and archive-yum-packages.rb against a +# local fake GCS, inside a container with the real Aptly, createrepo_c +# and GPG. Also runs publish-debs.rb against the result, to check that +# CI can still publish afterwards. Not run by CI. +# +# Usage: ./internal-scripts/ci-cd/archive/test/run-tests.sh +set -euo pipefail + +SELFDIR=$(dirname "$0") +SELFDIR=$(cd "${SELFDIR}" && pwd) +ROOTDIR=$(cd "${SELFDIR}/../../../.." && pwd) +IMAGE=fsruby-archive-test + +if [[ "${INSIDE_ARCHIVE_TEST_CONTAINER:-}" != true ]]; then + docker build --platform linux/amd64 -q -t "${IMAGE}" "${SELFDIR}" >/dev/null + exec docker run --rm --platform linux/amd64 \ + -e INSIDE_ARCHIVE_TEST_CONTAINER=true \ + -v "${ROOTDIR}:/src:ro" \ + "${IMAGE}" \ + bash /src/internal-scripts/ci-cd/archive/test/run-tests.sh +fi + +# ---- Inside the container ---- + +WORK=$(mktemp -d) +export FAKE_GCS_ROOT="${WORK}/gcs" +export FAKE_SIGNING_KEY_PATH="${WORK}/signing-key.asc" +export PATH="/src/internal-scripts/ci-cd/archive/test/fake-bin:${PATH}" +export GNUPGHOME="${WORK}/gnupg" +LIVE_APT=fsruby-server-edition-apt-repo +ARCHIVE_APT=fsruby-server-edition-apt-archive-repo +LIVE_YUM=fsruby-server-edition-yum-repo +ARCHIVE_YUM=fsruby-server-edition-yum-archive-repo + +FAILURES=0 + +function header() { + echo + echo "===== $* =====" +} + +function check() { + local description="$1" + shift + if "$@"; then + echo "PASS: ${description}" + else + echo "FAIL: ${description}" + FAILURES=$((FAILURES + 1)) + fi +} + +function gcs_version() { + cat "${FAKE_GCS_ROOT}/$1/versions/latest_version.txt" +} + +function public_dir() { + echo "${FAKE_GCS_ROOT}/$1/versions/$(gcs_version "$1")/public" +} + +# Package names (Package_Version_Architecture) in a published APT distribution. +function apt_packages() { + local file + file="$(public_dir "$1")/dists/$2/main/binary-amd64/Packages" + [[ -f "${file}" ]] || return 0 + # Field order within a stanza is not fixed, so print at the end of each stanza. + awk '/^Package:/{p=$2} /^Version:/{v=$2} /^Architecture:/{a=$2} /^$/{if(p)print p"_"v"_"a; p=""} END{if(p)print p"_"v"_"a}' "${file}" | sort +} + +# RPM filenames indexed in a published YUM distro/arch directory. +function yum_packages() { + local repodata + repodata="$(public_dir "$1")/$2/repodata" + [[ -d "${repodata}" ]] || return 0 + zcat "${repodata}"/*-primary.xml.gz | { grep -o 'location href="[^"]*"' || true; } | sed 's/.*href="//; s/"$//; s|.*/||' | sort +} + +function contains() { + grep -qx -- "$2" <<<"$1" +} + +function lacks() { + ! grep -qx -- "$2" <<<"$1" +} + +function empty() { + [[ -z "$1" ]] +} + +function count_is() { + [[ "$(grep -c . <<<"$1" || true)" -eq "$2" ]] +} + +function apt_signed() { + local dir + dir="$(public_dir "$1")/dists/$2" + gpg --verify "${dir}/Release.gpg" "${dir}/Release" 2>/dev/null +} + +function yum_signed() { + local dir + dir="$(public_dir "$1")/$2/repodata" + gpg --verify "${dir}/repomd.xml.asc" "${dir}/repomd.xml" 2>/dev/null +} + +# Checks that a state tarball matches what publish-debs.rb produces: +# no publications, and an empty repo/ directory. +function state_is_unpublished() { + local bucket="$1" + local dir="${WORK}/state-check" + rm -rf "${dir}" + mkdir -p "${dir}" + zstd -dc "${FAKE_GCS_ROOT}/${bucket}/versions/$(gcs_version "${bucket}")/state.tar.zst" | tar -xC "${dir}" + printf '{"rootDir":"%s"}' "${dir}" >"${WORK}/state-check.conf" + [[ -z "$(aptly -config="${WORK}/state-check.conf" publish list -raw)" ]] && + [[ -z "$(find "${dir}/repo" -mindepth 1 -print -quit)" ]] +} + +header 'Creating signing key' +mkdir -m 700 "${GNUPGHOME}" +gpg -q --batch --passphrase '' --quick-gen-key 'Archive Test ' default default never 2>/dev/null +gpg -q --batch --export-secret-keys --armor >"${FAKE_SIGNING_KEY_PATH}" +KEY_ID=$(gpg --list-keys --with-colons | awk -F: '/^pub/{print $5; exit}') +echo "Key: ${KEY_ID}" + +header 'Building fixture packages' +PKGS="${WORK}/pkgs" +mkdir -p "${PKGS}/deb" "${PKGS}/rpm" "${PKGS}/new" + +# make_deb NAME VERSION ARCH [DISTRIBUTION] [OUTPUT_DIR] +function make_deb() { + local name="$1" version="$2" arch="$3" distribution="${4:-}" output_dir="${5:-${PKGS}/deb}" + local dir="${WORK}/debbuild/${name}_${version}_${arch}" + mkdir -p "${dir}/DEBIAN" + printf 'Package: %s\nVersion: %s\nArchitecture: %s\nMaintainer: Test \nDescription: test\n' \ + "${name}" "${version}" "${arch}" >"${dir}/DEBIAN/control" + if [[ -n "${distribution}" ]]; then + echo "Distribution: ${distribution}" >>"${dir}/DEBIAN/control" + fi + dpkg-deb -Zgzip -b "${dir}" "${output_dir}/${name}_${version}_${arch}.deb" >/dev/null +} + +function make_rpm() { + local name="$1" version="$2" release="$3" arch="$4" + local topdir="${WORK}/rpmbuild/${name}-${version}-${release}" + mkdir -p "${topdir}/SPECS" + cat >"${topdir}/SPECS/pkg.spec" </dev/null 2>&1 + find "${topdir}/RPMS" -name '*.rpm' -exec cp {} "${PKGS}/rpm/" \; +} + +# DEB distributions: +# debian-12 supported, has EOL Ruby packages +# debian-13 supported, nothing to archive +# debian-10 EOL +# Ruby 3.4 is active, 3.1 is EOL (per config.yml). +for distro in debian-12 debian-10; do + make_deb fullstaq-ruby-3.4 "9-${distro}" amd64 + make_deb fullstaq-ruby-3.4-jemalloc "9-${distro}" amd64 + make_deb fullstaq-ruby-3.4.1 "0-${distro}" amd64 + make_deb fullstaq-ruby-3.1 "5-${distro}" amd64 + make_deb fullstaq-ruby-3.1-jemalloc "5-${distro}" amd64 + make_deb fullstaq-ruby-3.1.7 "0-${distro}" amd64 +done +make_deb fullstaq-ruby-3.4 9-debian-13 amd64 +make_deb fullstaq-ruby-common 1.0-0 all +make_deb fullstaq-rbenv 1.1.2-16 all +# New packages for the publish-debs.rb runs. +make_deb fullstaq-ruby-3.4 10-debian-12 amd64 debian-12 "${PKGS}/new" +make_deb fullstaq-ruby-3.4 10-debian-13 amd64 debian-13 "${PKGS}/new" + +# RPM distributions: +# el-9 supported, has EOL Ruby packages +# centos-8 supported, nothing to archive +# centos-7 EOL +for distro in el9 centos7; do + make_rpm fullstaq-ruby-3.4 rev9 "${distro}" x86_64 + make_rpm fullstaq-ruby-3.4-jemalloc rev9 "${distro}" x86_64 + make_rpm fullstaq-ruby-3.1 rev5 "${distro}" x86_64 + make_rpm fullstaq-ruby-3.1-jemalloc rev5 "${distro}" x86_64 + make_rpm fullstaq-ruby-3.1.7 rev0 "${distro}" x86_64 +done +make_rpm fullstaq-ruby-3.4 rev9 centos8 x86_64 +make_rpm fullstaq-ruby-common 1.0 0 noarch +make_rpm fullstaq-rbenv 1.1.2_16 1 noarch +ls "${PKGS}/deb" "${PKGS}/rpm" + +header 'Seeding live APT repository (version 1)' +# Like publish-debs.rb: the state is archived before publishing, so it +# contains no publications. +function seed_apt() { + local state="${WORK}/seed-apt" + mkdir -p "${state}/db" "${state}/repo" + printf '{"rootDir":"%s","FileSystemPublishEndpoints":{"main":{"rootDir":"%s/repo","linkMethod":"symlink","verifyMethod":"md5"}}}' \ + "${state}" "${state}" >"${WORK}/seed-apt.conf" + local distro + for distro in debian-12 debian-13 debian-10; do + aptly -config="${WORK}/seed-apt.conf" repo create "${distro}" >/dev/null + aptly -config="${WORK}/seed-apt.conf" repo add "${distro}" "${PKGS}"/deb/*"${distro}"*.deb \ + "${PKGS}/deb/fullstaq-ruby-common_1.0-0_all.deb" "${PKGS}/deb/fullstaq-rbenv_1.1.2-16_all.deb" >/dev/null + done + local version_dir="${FAKE_GCS_ROOT}/${LIVE_APT}/versions/1" + mkdir -p "${version_dir}/public" + tar -C "${state}" -cf - . | zstd -q -T0 >"${version_dir}/state.tar.zst" + for distro in debian-12 debian-13 debian-10; do + aptly -config="${WORK}/seed-apt.conf" publish repo -batch -gpg-key="${KEY_ID}" -distribution="${distro}" \ + -origin=Fullstaq-Ruby -label=Fullstaq-Ruby "${distro}" filesystem:main:. >/dev/null + done + cp -rL "${state}/repo/." "${version_dir}/public/" + echo 1 >"${version_dir}/version.txt" + echo 1 >"${FAKE_GCS_ROOT}/${LIVE_APT}/versions/latest_version.txt" +} +seed_apt + +header 'Seeding live YUM repository (version 1)' +function seed_yum() { + local version_dir="${FAKE_GCS_ROOT}/${LIVE_YUM}/versions/1" + local distro sanitized dir + for distro in el-9 centos-8 centos-7; do + sanitized=${distro//-/} + dir="${version_dir}/public/${distro}/x86_64" + mkdir -p "${dir}" + cp "${PKGS}"/rpm/*"${sanitized}"*.rpm "${PKGS}"/rpm/*.noarch.rpm "${dir}/" + createrepo_c -q "${dir}" + gpg -q --batch --yes --detach-sign --armor "${dir}/repodata/repomd.xml" + done + echo 1 >"${version_dir}/version.txt" + echo 1 >"${FAKE_GCS_ROOT}/${LIVE_YUM}/versions/latest_version.txt" +} +seed_yum + +# The scripts read config.yml and environments/ from the working copy. +# Use a writable copy so the second round can retire a distribution. +cp -r /src "${WORK}/src" +cd "${WORK}/src" + +APT_ENV=(env PRODUCTION_REPO_BUCKET_NAME="${LIVE_APT}" ARCHIVE_REPO_BUCKET_NAME="${ARCHIVE_APT}") +YUM_ENV=(env PRODUCTION_REPO_BUCKET_NAME="${LIVE_YUM}" ARCHIVE_REPO_BUCKET_NAME="${ARCHIVE_YUM}") + +function snapshot() { + (cd "${FAKE_GCS_ROOT}" && find . -type f ! -path '*/locks/*' -exec md5sum {} + | sort -k2) +} + +# Runs publish-debs.rb in production mode, as CI does on main, with +# DRY_RUN so it uploads nothing. Fails if it can't publish the state +# the archive script left behind. +function publish_debs_succeeds() { + env PRODUCTION_REPO_BUCKET_NAME="${LIVE_APT}" TESTING=false OVERWRITE_EXISTING=false DRY_RUN=true CLEANUP=true \ + ./internal-scripts/ci-cd/publish/publish-debs.rb "$1" >"${WORK}/publish-debs.log" 2>&1 || + { tail -20 "${WORK}/publish-debs.log"; return 1; } +} + +# ---------------------------------------------------------------- APT + +header 'APT: dry run uploads nothing' +before=$(snapshot) +"${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb --dry-run +check 'dry run leaves fake GCS unchanged' [ "${before}" == "$(snapshot)" ] +check 'dry run leaves no temp dir behind' empty "$(find /tmp -maxdepth 1 -name 'archive-apt*')" + +header 'APT: first run' +LIVE_D13_BEFORE=$(apt_packages "${LIVE_APT}" debian-13) +"${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb + +LIVE_D12=$(apt_packages "${LIVE_APT}" debian-12) +LIVE_D13=$(apt_packages "${LIVE_APT}" debian-13) +ARCH_D12=$(apt_packages "${ARCHIVE_APT}" debian-12) +ARCH_D10=$(apt_packages "${ARCHIVE_APT}" debian-10) +echo "live debian-12:"; sed 's/^/ /' <<<"${LIVE_D12}" +echo "live debian-13:"; sed 's/^/ /' <<<"${LIVE_D13}" +echo "archive debian-12:"; sed 's/^/ /' <<<"${ARCH_D12}" +echo "archive debian-10:"; sed 's/^/ /' <<<"${ARCH_D10}" + +check 'live version bumped to 2' [ "$(gcs_version "${LIVE_APT}")" == 2 ] +check 'archive version is 1' [ "$(gcs_version "${ARCHIVE_APT}")" == 1 ] +check 'live debian-12 keeps active Ruby' contains "${LIVE_D12}" fullstaq-ruby-3.4_9-debian-12_amd64 +check 'live debian-12 keeps active Ruby variant' contains "${LIVE_D12}" fullstaq-ruby-3.4-jemalloc_9-debian-12_amd64 +check 'live debian-12 keeps tiny of active minor' contains "${LIVE_D12}" fullstaq-ruby-3.4.1_0-debian-12_amd64 +check 'live debian-12 keeps common' contains "${LIVE_D12}" fullstaq-ruby-common_1.0-0_all +check 'live debian-12 keeps rbenv' contains "${LIVE_D12}" fullstaq-rbenv_1.1.2-16_all +check 'live debian-12 drops EOL Ruby' lacks "${LIVE_D12}" fullstaq-ruby-3.1_5-debian-12_amd64 +check 'live debian-12 drops EOL Ruby variant' lacks "${LIVE_D12}" fullstaq-ruby-3.1-jemalloc_5-debian-12_amd64 +check 'live debian-12 drops tiny of EOL minor' lacks "${LIVE_D12}" fullstaq-ruby-3.1.7_0-debian-12_amd64 +check 'live debian-13 (nothing to archive) still published, unchanged' [ "${LIVE_D13}" == "${LIVE_D13_BEFORE}" ] +check 'live debian-13 Release is signed' apt_signed "${LIVE_APT}" debian-13 +check 'live debian-10 no longer published' empty "$(apt_packages "${LIVE_APT}" debian-10)" +check 'archive debian-12 has exactly the EOL Ruby packages' [ "${ARCH_D12}" == "$(printf '%s\n' \ + fullstaq-ruby-3.1-jemalloc_5-debian-12_amd64 fullstaq-ruby-3.1.7_0-debian-12_amd64 fullstaq-ruby-3.1_5-debian-12_amd64 | sort)" ] +check 'archive debian-10 has all 8 packages' count_is "${ARCH_D10}" 8 +check 'archive debian-10 has common' contains "${ARCH_D10}" fullstaq-ruby-common_1.0-0_all +check 'archive has no debian-13' empty "$(apt_packages "${ARCHIVE_APT}" debian-13)" +check 'live and archive debian-12 are disjoint' empty "$(comm -12 <(echo "${LIVE_D12}") <(echo "${ARCH_D12}"))" +check 'live debian-12 Release is signed' apt_signed "${LIVE_APT}" debian-12 +check 'archive debian-12 Release is signed' apt_signed "${ARCHIVE_APT}" debian-12 +check 'archive debian-10 Release is signed' apt_signed "${ARCHIVE_APT}" debian-10 +check 'archive pool contains debian-10 debs' [ -n "$(find "$(public_dir "${ARCHIVE_APT}")/pool" -name '*debian-10*.deb')" ] +check 'live pool no longer contains EOL Ruby debs' empty "$(find "$(public_dir "${LIVE_APT}")/pool" -name 'fullstaq-ruby-3.1*')" +check 'live pool no longer contains debian-10 debs' empty "$(find "$(public_dir "${LIVE_APT}")/pool" -name '*debian-10*')" +check 'live state has no publications (like publish-debs)' state_is_unpublished "${LIVE_APT}" +check 'archive state has no publications' state_is_unpublished "${ARCHIVE_APT}" +check 'lock released' [ ! -e "${FAKE_GCS_ROOT}/${LIVE_APT}/locks/apt" ] +check 'no temp dir left behind' empty "$(find /tmp -maxdepth 1 -name 'archive-apt*')" +check 'publish-debs.rb can publish afterwards' publish_debs_succeeds "${PKGS}/new/fullstaq-ruby-3.4_10-debian-12_amd64.deb" + +header 'APT: second run is a no-op' +before=$(snapshot) +"${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb +check 'second run changes nothing' [ "${before}" == "$(snapshot)" ] + +header 'APT: rerun after crash between archive and live upload' +# Simulate: archive version 1 was activated, live upload never happened. +# Roll the live pointer back to version 1 (still intact in the bucket). +echo 1 >"${FAKE_GCS_ROOT}/${LIVE_APT}/versions/latest_version.txt" +rm -rf "${FAKE_GCS_ROOT}/${LIVE_APT}/versions/2" +"${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb +check 'archive debian-12 unchanged after rerun' [ "${ARCH_D12}" == "$(apt_packages "${ARCHIVE_APT}" debian-12)" ] +check 'archive debian-10 unchanged after rerun' [ "${ARCH_D10}" == "$(apt_packages "${ARCHIVE_APT}" debian-10)" ] +check 'live debian-12 trimmed after rerun' [ "${LIVE_D12}" == "$(apt_packages "${LIVE_APT}" debian-12)" ] +check 'live debian-13 unchanged after rerun' [ "${LIVE_D13}" == "$(apt_packages "${LIVE_APT}" debian-13)" ] +check 'archive debian-12 still signed after rerun' apt_signed "${ARCHIVE_APT}" debian-12 + +header 'APT: second archival round (debian-12 reaches EOL)' +# debian-10 is already archived and is not touched this round. Its +# publication must survive being rebuilt from the previous archive state. +mv environments/debian-12 "${WORK}/debian-12-env" +"${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb +ARCH2_D12=$(apt_packages "${ARCHIVE_APT}" debian-12) +echo "archive debian-12:"; sed 's/^/ /' <<<"${ARCH2_D12}" +check 'live debian-12 no longer published' empty "$(apt_packages "${LIVE_APT}" debian-12)" +check 'live debian-13 still unchanged' [ "${LIVE_D13}" == "$(apt_packages "${LIVE_APT}" debian-13)" ] +check 'archive debian-12 now has all 8 packages' count_is "${ARCH2_D12}" 8 +check 'archive debian-10 unchanged by second round' [ "${ARCH_D10}" == "$(apt_packages "${ARCHIVE_APT}" debian-10)" ] +check 'archive debian-10 still signed' apt_signed "${ARCHIVE_APT}" debian-10 +check 'archive debian-12 signed' apt_signed "${ARCHIVE_APT}" debian-12 +check 'publish-debs.rb can publish after second round' publish_debs_succeeds "${PKGS}/new/fullstaq-ruby-3.4_10-debian-13_amd64.deb" +mv "${WORK}/debian-12-env" environments/debian-12 + +# ---------------------------------------------------------------- YUM + +header 'YUM: dry run uploads nothing' +before=$(snapshot) +"${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb --dry-run +check 'dry run leaves fake GCS unchanged' [ "${before}" == "$(snapshot)" ] + +header 'YUM: first run' +LIVE_C8_BEFORE=$(yum_packages "${LIVE_YUM}" centos-8/x86_64) +"${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb + +LIVE_EL9=$(yum_packages "${LIVE_YUM}" el-9/x86_64) +LIVE_C8=$(yum_packages "${LIVE_YUM}" centos-8/x86_64) +ARCH_EL9=$(yum_packages "${ARCHIVE_YUM}" el-9/x86_64) +ARCH_C7=$(yum_packages "${ARCHIVE_YUM}" centos-7/x86_64) +echo "live el-9:"; sed 's/^/ /' <<<"${LIVE_EL9}" +echo "archive el-9:"; sed 's/^/ /' <<<"${ARCH_EL9}" +echo "archive centos-7:"; sed 's/^/ /' <<<"${ARCH_C7}" + +check 'live version bumped to 2' [ "$(gcs_version "${LIVE_YUM}")" == 2 ] +check 'archive version is 1' [ "$(gcs_version "${ARCHIVE_YUM}")" == 1 ] +check 'live el-9 keeps active Ruby' contains "${LIVE_EL9}" fullstaq-ruby-3.4-rev9-el9.x86_64.rpm +check 'live el-9 keeps common' contains "${LIVE_EL9}" fullstaq-ruby-common-1.0-0.noarch.rpm +check 'live el-9 keeps rbenv' contains "${LIVE_EL9}" fullstaq-rbenv-1.1.2_16-1.noarch.rpm +check 'live el-9 drops EOL Ruby' lacks "${LIVE_EL9}" fullstaq-ruby-3.1-rev5-el9.x86_64.rpm +check 'live el-9 drops tiny of EOL minor' lacks "${LIVE_EL9}" fullstaq-ruby-3.1.7-rev0-el9.x86_64.rpm +check 'live el-9 has no leftover EOL RPM files' empty "$(find "$(public_dir "${LIVE_YUM}")/el-9" -name 'fullstaq-ruby-3.1*')" +check 'live centos-8 (nothing to archive) unchanged' [ "${LIVE_C8}" == "${LIVE_C8_BEFORE}" ] +check 'live centos-8 repomd is signed' yum_signed "${LIVE_YUM}" centos-8/x86_64 +check 'live centos-7 removed' [ ! -e "$(public_dir "${LIVE_YUM}")/centos-7" ] +check 'archive el-9 has exactly the EOL Ruby packages' [ "${ARCH_EL9}" == "$(printf '%s\n' \ + fullstaq-ruby-3.1-jemalloc-rev5-el9.x86_64.rpm fullstaq-ruby-3.1-rev5-el9.x86_64.rpm fullstaq-ruby-3.1.7-rev0-el9.x86_64.rpm | sort)" ] +check 'archive centos-7 has all 7 packages' count_is "${ARCH_C7}" 7 +check 'archive has no centos-8' [ ! -e "$(public_dir "${ARCHIVE_YUM}")/centos-8" ] +check 'live and archive el-9 are disjoint' empty "$(comm -12 <(echo "${LIVE_EL9}") <(echo "${ARCH_EL9}"))" +check 'live el-9 repomd is signed' yum_signed "${LIVE_YUM}" el-9/x86_64 +check 'archive el-9 repomd is signed' yum_signed "${ARCHIVE_YUM}" el-9/x86_64 +check 'archive centos-7 repomd is signed' yum_signed "${ARCHIVE_YUM}" centos-7/x86_64 +check 'no nested distro directory in archive' [ ! -e "$(public_dir "${ARCHIVE_YUM}")/centos-7/centos-7" ] +check 'lock released' [ ! -e "${FAKE_GCS_ROOT}/${LIVE_YUM}/locks/yum" ] +check 'no temp dir left behind' empty "$(find /tmp -maxdepth 1 -name 'archive-yum*')" + +header 'YUM: second run is a no-op' +before=$(snapshot) +"${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb +check 'second run changes nothing' [ "${before}" == "$(snapshot)" ] + +header 'YUM: rerun after crash between archive and live upload' +echo 1 >"${FAKE_GCS_ROOT}/${LIVE_YUM}/versions/latest_version.txt" +rm -rf "${FAKE_GCS_ROOT}/${LIVE_YUM}/versions/2" +"${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb +check 'archive el-9 unchanged after rerun' [ "${ARCH_EL9}" == "$(yum_packages "${ARCHIVE_YUM}" el-9/x86_64)" ] +check 'archive centos-7 unchanged after rerun' [ "${ARCH_C7}" == "$(yum_packages "${ARCHIVE_YUM}" centos-7/x86_64)" ] +check 'live el-9 trimmed after rerun' [ "${LIVE_EL9}" == "$(yum_packages "${LIVE_YUM}" el-9/x86_64)" ] +check 'no nested distro directory after rerun' empty "$(find "${FAKE_GCS_ROOT}/${ARCHIVE_YUM}" -path '*/centos-7/centos-7')" + +header 'YUM: second archival round (el-9 reaches EOL)' +mv environments/el-9 "${WORK}/el-9-env" +"${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb +ARCH2_EL9=$(yum_packages "${ARCHIVE_YUM}" el-9/x86_64) +check 'live el-9 removed' [ ! -e "$(public_dir "${LIVE_YUM}")/el-9" ] +check 'live centos-8 still unchanged' [ "${LIVE_C8}" == "$(yum_packages "${LIVE_YUM}" centos-8/x86_64)" ] +check 'archive el-9 now has all 7 packages' count_is "${ARCH2_EL9}" 7 +check 'archive centos-7 unchanged by second round' [ "${ARCH_C7}" == "$(yum_packages "${ARCHIVE_YUM}" centos-7/x86_64)" ] +check 'archive el-9 signed' yum_signed "${ARCHIVE_YUM}" el-9/x86_64 +mv "${WORK}/el-9-env" environments/el-9 + +header 'Result' +if [[ "${FAILURES}" -eq 0 ]]; then + echo 'All checks passed' +else + echo "${FAILURES} checks failed" + exit 1 +fi