From b1b5a5011dca3ce1bd3936b73f7ecae1c60e332a Mon Sep 17 00:00:00 2001 From: abtreece Date: Fri, 24 Apr 2026 22:03:24 -0500 Subject: [PATCH 01/12] feat: add EOL package archival and pruning tooling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add incremental APT/YUM archive migration scripts that move EOL distro packages to archive buckets, and pruning scripts that remove EOL Ruby version packages from still-supported distro repos. Scripts auto-detect EOL content by comparing repo state against config.yml and support --dry-run. Migration scripts are incremental — safe to re-run when future distros reach EOL. Includes dev-handbook runbook and updated architecture docs. --- dev-handbook/README.md | 1 + dev-handbook/apt-yum-repo-infra.drawio.svg | 4 +- dev-handbook/apt-yum-repo.md | 22 +- dev-handbook/archiving-eol-packages.md | 229 +++++++ dev-handbook/ci-cd-resumption.md | 6 +- .../ci-cd/archive/migrate-apt-to-archive.rb | 585 ++++++++++++++++++ .../ci-cd/archive/migrate-yum-to-archive.rb | 324 ++++++++++ .../ci-cd/archive/prune-apt-packages.rb | 362 +++++++++++ .../ci-cd/archive/prune-yum-packages.rb | 219 +++++++ 9 files changed, 1738 insertions(+), 14 deletions(-) create mode 100644 dev-handbook/archiving-eol-packages.md create mode 100755 internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb create mode 100755 internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb create mode 100755 internal-scripts/ci-cd/archive/prune-apt-packages.rb create mode 100755 internal-scripts/ci-cd/archive/prune-yum-packages.rb diff --git a/dev-handbook/README.md b/dev-handbook/README.md index d12aafbc..9dc35cd3 100644 --- a/dev-handbook/README.md +++ b/dev-handbook/README.md @@ -40,6 +40,7 @@ * [Adding support for a new distribution](add-new-distro.md) * [Adding support for a new Ruby version](add-new-ruby-version.md) + * [Archiving EOL packages](archiving-eol-packages.md) ## Organizational (for team members) diff --git a/dev-handbook/apt-yum-repo-infra.drawio.svg b/dev-handbook/apt-yum-repo-infra.drawio.svg index f125c005..3f50073f 100644 --- a/dev-handbook/apt-yum-repo-infra.drawio.svg +++ b/dev-handbook/apt-yum-repo-infra.drawio.svg @@ -34,7 +34,7 @@ - fullstaq-ruby-server-edition-apt-repo + fsruby-server-edition-apt-repo
@@ -81,7 +81,7 @@
- fullstaq-ruby-server-edition-yum-repo + fsruby-server-edition-yum-repo
diff --git a/dev-handbook/apt-yum-repo.md b/dev-handbook/apt-yum-repo.md index fb265d31..ad25346b 100644 --- a/dev-handbook/apt-yum-repo.md +++ b/dev-handbook/apt-yum-repo.md @@ -48,11 +48,15 @@ Our design was made with the following requirements in mind, and addresses the f We self-host our repositories: we store them in Google Cloud Storage buckets. [Google Cloud Storage is strongly consistent.](https://cloud.google.com/storage/docs/consistency) -There are three buckets: +There are five buckets: - - `fullstaq-ruby-server-edition-apt-repo` stores the production APT repository. - - `fullstaq-ruby-server-edition-yum-repo` stores the production YUM repository. - - `fullstaq-ruby-server-edition-ci-artifacts` stores the temporary repositories created during CI runs. + - `fsruby-server-edition-apt-repo` stores the production APT repository. + - `fsruby-server-edition-yum-repo` stores the production YUM repository. + - `fsruby-server-edition-apt-repo-archive` stores packages for EOL distributions (APT). + - `fsruby-server-edition-yum-repo-archive` stores packages for EOL distributions (YUM). + - `fsruby-server-edition-ci-artifacts` stores the temporary repositories created during CI runs. + +The archive buckets are only updated during [EOL migration](archiving-eol-packages.md) — CI never writes to them. Each migration creates a new version that merges newly-archived distros with the existing archive contents. They are served at `apt-archive.fullstaqruby.org` and `yum-archive.fullstaqruby.org`. We don't let users use the production bucket URLs directly. Instead, we let users use `https://apt.fullstaqruby.org` and `https://yum.fullstaqruby.org`. These domains redirect to the appropriate bucket URLs. We do this so that we avoid strongly coupling users with Google Cloud Storage. If in the future we want to move off Google Cloud, we can do so without breaking users' URLs. @@ -144,7 +148,7 @@ A downside of this versioning approach is that each version consumes a lot of sp ### CI bucket -During CI runs, we create temporary repositories in `gs://fullstaq-ruby-server-edition-ci-artifacts/$CI_RUN_NUMBER/{apt,yum}-repo`. These directories look as follows: +During CI runs, we create temporary repositories in `gs://fsruby-server-edition-ci-artifacts/$CI_RUN_NUMBER/{apt,yum}-repo`. These directories look as follows: ~~~ /$CI_RUN_NUMBER/{apt,yum}-repo/ @@ -162,7 +166,7 @@ These directories are very similar to the production buckets' contents. But ther The CI tests run directly against this bucket URL instead of going through `https://{apt,yum}.fullstaqruby.org`. -We create a temporary repository by copying over `gs://fullstaq-ruby-server-edition-{apt,yum}-repo/versions/$LATEST_VERSION`. This way we achieve production data parity during testing. +We create a temporary repository by copying over `gs://fsruby-server-edition-{apt,yum}-repo/versions/$LATEST_VERSION`. This way we achieve production data parity during testing. ## Locking @@ -179,10 +183,10 @@ The lock's critical section is quite large, and covers: The locks are located in the following URLs: - * `gs://fullstaq-ruby-server-edition-apt-repo/locks/apt` - * `gs://fullstaq-ruby-server-edition-yum-repo/locks/yum` + * `gs://fsruby-server-edition-apt-repo/locks/apt` + * `gs://fsruby-server-edition-yum-repo/locks/yum` -When publishing to a testing repository (in `fullstaq-ruby-server-edition-ci-artifacts`) we don't perform any locking, because each CI run is guaranteed to write to its own temporary repository. +When publishing to a testing repository (in `fsruby-server-edition-ci-artifacts`) we don't perform any locking, because each CI run is guaranteed to write to its own temporary repository. ## Backups diff --git a/dev-handbook/archiving-eol-packages.md b/dev-handbook/archiving-eol-packages.md new file mode 100644 index 00000000..3519ee78 --- /dev/null +++ b/dev-handbook/archiving-eol-packages.md @@ -0,0 +1,229 @@ +# Archiving EOL packages + +This document describes how to archive packages for end-of-life (EOL) distributions and prune EOL Ruby versions from the repositories. This is a routine maintenance task that frees CI disk space and keeps the repository lean. + +## Background + +The CI publish step downloads the full Aptly state archive (`state.tar.zst`) from Google Cloud Storage on every run. This archive grows with every distribution and Ruby version ever published. When distributions or Ruby versions reach EOL, their packages remain in the state archive indefinitely, consuming disk space on GitHub Actions runners. + +To address this, we maintain **archive repositories** alongside the main repositories: + +| Repository | Bucket | Domain | Purpose | +|------------|--------|--------|---------| +| APT (main) | `fsruby-server-edition-apt-repo` | `apt.fullstaqruby.org` | Current, supported packages | +| APT (archive) | `fsruby-server-edition-apt-repo-archive` | `apt-archive.fullstaqruby.org` | Frozen packages for EOL distributions | +| YUM (main) | `fsruby-server-edition-yum-repo` | `yum.fullstaqruby.org` | Current, supported packages | +| YUM (archive) | `fsruby-server-edition-yum-repo-archive` | `yum-archive.fullstaqruby.org` | Frozen packages for EOL distributions | + +Archive repositories are static — CI never writes to them. They use the same versioned bucket structure as the main repos. Each migration creates a new version that merges newly-archived distros with the existing archive contents, so the archive grows incrementally over time. + +This pattern follows the precedent set by [PostgreSQL](https://apt-archive.postgresql.org/) (`apt-archive.postgresql.org`) and [HashiCorp](https://www.hashicorp.com/en/blog/announcing-the-linux-package-archive-site) (`archive.releases.hashicorp.com`). + +## Two types of cleanup + +There are two independent axes of cleanup, each with its own script: + +### 1. Distro archival — moving entire EOL distribution repos + +When a Linux distribution reaches EOL, we stop building packages for it and move its existing packages to the archive. Users on EOL distributions can still install packages by pointing at the archive repo. + +**Scripts:** + * `internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb` + * `internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb` + +### 2. Package pruning — removing EOL Ruby version packages + +When a Ruby version reaches EOL, we stop building it (by removing it from `config.yml`), but its packages persist inside every distro's repository. Pruning removes these stale packages from the still-supported distro repos to reduce state size. + +**Scripts:** + * `internal-scripts/ci-cd/archive/prune-apt-packages.rb` + * `internal-scripts/ci-cd/archive/prune-yum-packages.rb` + +## Removing an EOL distribution + +### Step 1: Remove from the build system + + 1. Edit `config.yml` and remove the distribution from the `distributions` list (or add it to an exclusion). + 2. Delete the `environments//` directory. + 3. Regenerate CI/CD workflows: + + ~~~bash + ./internal-scripts/generate-ci-cd-yaml.rb + ~~~ + + 4. Commit and merge these changes. + +### Step 2: Migrate packages to the archive + +**Prerequisites:** + * `gcloud` CLI authenticated with write access to the GCS buckets + * `az` CLI authenticated with access to the `fsruby2infraowners` Key Vault (for the GPG signing key) + * `aptly`, `zstd`, and `gpg` installed locally + * Docker running (for `createrepo_c` in YUM migration) + +**Dry run first** to verify which distros will be archived: + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \ +./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb --dry-run +~~~ + +The script auto-detects EOL distros by comparing `aptly repo list` output against the distributions defined in `config.yml`. You can also specify distros explicitly: + +~~~bash +./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb --dry-run --distros centos-8,debian-9 +~~~ + +**Execute the migration** (removes `--dry-run`): + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \ +./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb +~~~ + +**Repeat for YUM:** + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo-archive \ +./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb --dry-run + +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo-archive \ +./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb +~~~ + +### Step 3: Restart the web server + +After migration, restart the web server so Caddy picks up the new version numbers: + +~~~bash +curl -X POST https://apt.fullstaqruby.org/admin/restart_web_server \ + -H "Authorization: Bearer $ID_TOKEN" +~~~ + +Or restart the Caddy service directly via Ansible/SSH. + +### Step 4: Verify + +~~~bash +# Archive should list the archived distros +curl -s https://apt-archive.fullstaqruby.org/dists/ + +# Main repo should only contain supported distros +curl -s https://apt.fullstaqruby.org/dists/ + +# Verify state archive size decreased +gsutil ls -l gs://fsruby-server-edition-apt-repo/versions/*/state.tar.zst | tail -5 +~~~ + +## Pruning EOL Ruby versions + +After removing a Ruby version from `config.yml`, its packages persist in the Aptly state. Run the pruning scripts to remove them. + +**Dry run:** + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +./internal-scripts/ci-cd/archive/prune-apt-packages.rb --dry-run +~~~ + +The script compares packages in the Aptly state against `minor_version_packages` in `config.yml` and identifies any `fullstaq-ruby-X.Y*` packages where `X.Y` is not an active minor version. + +**Execute:** + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +./internal-scripts/ci-cd/archive/prune-apt-packages.rb +~~~ + +**Repeat for YUM:** + +~~~bash +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +./internal-scripts/ci-cd/archive/prune-yum-packages.rb --dry-run + +PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +./internal-scripts/ci-cd/archive/prune-yum-packages.rb +~~~ + +Restart the web server after pruning (same as above). + +## Execution order + +When performing both distro archival and package pruning in the same session, always run distro archival **first**. This ensures the archive captures the full historical packages for EOL distros before any pruning happens. + + 1. `migrate-apt-to-archive.rb` + 2. `prune-apt-packages.rb` + 3. `migrate-yum-to-archive.rb` + 4. `prune-yum-packages.rb` + 5. Restart web server + +## Rollback + +The versioned bucket structure makes rollback straightforward. Each migration creates a new version — the old version is never modified. + +**Revert the main APT repo to a previous version:** + +~~~bash +# Find the pre-migration version number +gsutil cat gs://fsruby-server-edition-apt-repo/versions/latest_version.txt + +# Point back to the old version +echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-repo/versions/latest_version.txt +~~~ + +**Revert the archive to a previous version:** + +~~~bash +gsutil cat gs://fsruby-server-edition-apt-repo-archive/versions/latest_version.txt + +echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-repo-archive/versions/latest_version.txt +~~~ + +**Delete all archive contents** (if no users depend on it yet): + +~~~bash +gsutil -m rm -r gs://fsruby-server-edition-apt-repo-archive/versions/ +~~~ + +## How the migration scripts work + +### APT migration (`migrate-apt-to-archive.rb`) + + 1. Downloads the current Aptly state archive from the main bucket. + 2. Identifies EOL distros (published in Aptly but not in `config.yml`). + 3. Fetches the existing archive state (if any) so new distros are merged into it. + 4. Creates or extends the archive Aptly instance with EOL distro data and package pool. + 5. Publishes all archive distros (existing + newly archived). + 6. Drops the EOL distro repos from the main Aptly database. + 7. Runs `aptly db cleanup` to compact the database and reclaim pool space. + 8. Re-publishes remaining distros in the main repo. + 9. Uploads the merged archive as a new archive version (N+1). + 10. Uploads the trimmed state as a new version of the main repo. + +### YUM migration (`migrate-yum-to-archive.rb`) + + 1. Downloads the current YUM repo from the main bucket via `gsutil rsync`. + 2. Identifies EOL distro directories. + 3. Fetches the existing archive repo (if any) so new distros are merged into it. + 4. Copies EOL distro directories into the local archive copy. + 5. Uploads the merged archive as a new archive version (N+1). + 6. Removes EOL distro directories from the main local copy. + 7. Uploads the trimmed repo as a new version of the main bucket. + +### APT pruning (`prune-apt-packages.rb`) + + 1. Downloads the Aptly state. + 2. Scans all packages across all distro repos, matching `fullstaq-ruby-X.Y*` against active minor versions. + 3. Removes EOL Ruby packages using `aptly repo remove`. + 4. Compacts, re-publishes, and uploads. + +### YUM pruning (`prune-yum-packages.rb`) + + 1. Downloads the YUM repo. + 2. Deletes RPM files matching EOL Ruby versions from the filesystem. + 3. Regenerates `repodata/` with `createrepo_c` and re-signs. + 4. Uploads as a new version. diff --git a/dev-handbook/ci-cd-resumption.md b/dev-handbook/ci-cd-resumption.md index 7b7c3a0c..e4078e14 100644 --- a/dev-handbook/ci-cd-resumption.md +++ b/dev-handbook/ci-cd-resumption.md @@ -8,12 +8,12 @@ In order to aleviate this problem, we implement the ability to re-run only faile Resumption support works by checking, for each CI job, whether the artifact that that job should produce, already exists. If so, then that job can be skipped. -When you re-run a CI run, Github Actions wipes all previous state (including artifacts). Therefore, we store artifacts primarily in a Google Cloud Storage bucket ([fullstaq-ruby-server-edition-ci-artifacts](https://storage.googleapis.com/fullstaq-ruby-server-edition-ci-artifacts), part of the [infrastructure](https://github.com/fullstaq-labs/fullstaq-ruby-infra)), which isn't wiped before a re-run. +When you re-run a CI run, Github Actions wipes all previous state (including artifacts). Therefore, we store artifacts primarily in a Google Cloud Storage bucket ([fsruby-server-edition-ci-artifacts](https://storage.googleapis.com/fsruby-server-edition-ci-artifacts), part of the [infrastructure](https://github.com/fullstaq-labs/fullstaq-ruby-infra)), which isn't wiped before a re-run. Here's an example artifact URL: ~~~ -gs://fullstaq-ruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst +gs://fsruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst ~~~ Artifacts are stored on a per-CI-run basis. Thus, they always contains the CI run's number. Note that the CI run number does not change even for re-runs. @@ -23,7 +23,7 @@ At the beginning of a CI run, a job named `determine_necessary_jobs` checks whic ~~~ ##### Determine whether Rbenv DEB needs to be built ##### --> Run ./.github/actions/check-artifact-exists - Checking gs://fullstaq-ruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst + Checking gs://fsruby-server-edition-ci-artifacts/249/rbenv-deb.tar.zst Artifact exists ~~~ diff --git a/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb b/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb new file mode 100755 index 00000000..1bb16f40 --- /dev/null +++ b/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb @@ -0,0 +1,585 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Incremental migration script to move EOL distribution packages +# from the main APT repository to the archive APT repository. +# Safe to run repeatedly — merges new EOL distros into the existing archive. +# +# Usage: +# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \ +# ./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb [--dry-run] [--distros centos-8,debian-9] +# +# If --distros is not specified, automatically detects EOL distros by comparing +# the Aptly state against the current environments/ directory. + +require_relative '../../../lib/gcloud_storage_lock' +require_relative '../../../lib/ci_workflow_support' +require_relative '../../../lib/shell_scripting_support' +require_relative '../../../lib/publishing_support' +require 'json' +require 'shellwords' +require 'tmpdir' +require 'set' +require 'fileutils' +require 'optparse' + +class MigrateAptToArchive + REPO_ORIGIN = 'Fullstaq-Ruby' + REPO_LABEL = 'Fullstaq-Ruby' + + include CiWorkflowSupport + include ShellScriptingSupport + include PublishingSupport + + def main + parse_options + require_envvar 'PRODUCTION_REPO_BUCKET_NAME' + require_envvar 'ARCHIVE_REPO_BUCKET_NAME' + + print_header 'Initializing' + load_config + create_temp_dirs + ensure_gpg_state_isolated + activate_wrappers_bin_dir + initialize_aptly(@main_aptly_config_path, @main_state_path, @main_state_repo_path) + fetch_and_import_signing_key + + print_header 'Downloading main repository state' + version = get_latest_production_repo_version + if version == 0 + abort 'ERROR: No production repository exists yet' + end + fetch_main_state(version) + + print_header 'Identifying EOL distributions' + eol_distros = identify_eol_distros + if eol_distros.empty? + log_notice 'No EOL distributions found to archive' + exit + end + log_notice "EOL distributions to archive: #{eol_distros.join(', ')}" + + print_header 'Fetching existing archive state (if any)' + @archive_version = get_latest_archive_version + if @archive_version > 0 + log_notice "Existing archive at version #{@archive_version}, will merge" + fetch_archive_state(@archive_version) + else + log_notice 'No existing archive — creating fresh' + end + + print_header 'Creating archive repository' + create_archive_state(eol_distros) + + print_header 'Trimming main repository state' + trim_main_state(eol_distros) + + if @dry_run + log_notice 'DRY RUN — not uploading changes' + print_summary(eol_distros, version) + exit + end + + print_header 'Uploading archive repository' + upload_archive + + print_header 'Uploading trimmed main repository' + upload_main(version) + + print_header 'Success!' + print_summary(eol_distros, version) + + cleanup + end + +private + def parse_options + @dry_run = false + @explicit_distros = nil + + OptionParser.new do |opts| + opts.banner = "Usage: #{$0} [options]" + opts.on('--dry-run', 'Do not upload changes') { @dry_run = true } + opts.on('--distros DISTROS', 'Comma-separated list of distros to archive') do |v| + @explicit_distros = v.split(',').map(&:strip) + end + end.parse! + end + + def create_temp_dirs + log_notice 'Creating temporary directories' + @temp_dir = Dir.mktmpdir('apt-archive-migration') + @wrappers_bin_dir = "#{@temp_dir}/wrappers" + + @main_aptly_config_path = "#{@temp_dir}/aptly-main.conf" + @main_state_path = "#{@temp_dir}/main-state" + @main_state_db_path = "#{@main_state_path}/db" + @main_state_repo_path = "#{@main_state_path}/repo" + @main_state_archive_path = "#{@temp_dir}/main-state.tar.zst" + + @archive_aptly_config_path = "#{@temp_dir}/aptly-archive.conf" + @archive_state_path = "#{@temp_dir}/archive-state" + @archive_state_db_path = "#{@archive_state_path}/db" + @archive_state_repo_path = "#{@archive_state_path}/repo" + @archive_state_archive_path = "#{@temp_dir}/archive-state.tar.zst" + + @signing_key_path = "#{@temp_dir}/key.gpg" + + Dir.mkdir(@wrappers_bin_dir) + [@main_state_path, @main_state_db_path, @main_state_repo_path, + @archive_state_path, @archive_state_db_path, @archive_state_repo_path].each do |dir| + FileUtils.mkdir_p(dir) + end + end + + def ensure_gpg_state_isolated + log_notice 'Creating GPG wrapper' + File.open("#{@wrappers_bin_dir}/gpg", 'w:utf-8') do |f| + f.write("#!/bin/sh\n") + f.write( + sprintf( + "exec %s --homedir %s \"$@\"\n", + Shellwords.escape(find_gpg), + Shellwords.escape(@temp_dir) + ) + ) + end + File.chmod(0755, "#{@wrappers_bin_dir}/gpg") + end + + def find_gpg + ENV['PATH'].split(':').each do |dir| + next if dir == @wrappers_bin_dir + candidate = "#{dir}/gpg" + return candidate if File.exist?(candidate) + end + abort('GPG not found') + end + + def activate_wrappers_bin_dir + ENV['PATH'] = "#{@wrappers_bin_dir}:#{ENV['PATH']}" + end + + def initialize_aptly(config_path, state_path, repo_path) + log_notice "Creating Aptly config: #{config_path}" + File.open(config_path, 'w:utf-8') do |f| + f.write(JSON.generate( + rootDir: state_path, + FileSystemPublishEndpoints: { + main: { + rootDir: repo_path, + linkMethod: 'symlink', + verifyMethod: 'md5' + } + } + )) + end + end + + def fetch_and_import_signing_key + log_notice 'Fetching and importing signing key' + File.open(@signing_key_path, 'wb') do |f| + f.write(fetch_signing_key) + end + @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) + log_info "Signing key ID: #{@gpg_key_id}" + import_gpg_key(@temp_dir, @signing_key_path) + end + + def fetch_main_state(version) + log_notice "Fetching main state version #{version}" + url = "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/state.tar.zst" + run_bash( + sprintf('gsutil -m cp %s - | zstd -dc | tar -xC %s', + Shellwords.escape(url), + Shellwords.escape(@main_state_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def latest_production_version_note_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt" + end + + def get_latest_archive_version + url = "gs://#{ENV['ARCHIVE_REPO_BUCKET_NAME']}/versions/latest_version.txt" + stdout_output, stderr_output, status = run_command_capture_output( + 'gsutil', 'cp', url, '-', + log_invocation: false, + check_error: false + ) + if status.success? + v = stdout_output.strip + if v =~ /\A[0-9]+\Z/ + v.to_i + else + abort("ERROR: invalid version number stored in #{url}") + end + elsif stderr_output =~ /No URLs matched/ + 0 + else + abort("ERROR: error fetching #{url}: #{stderr_output.chomp}") + end + end + + def fetch_archive_state(version) + log_notice "Fetching archive state version #{version}" + url = "gs://#{ENV['ARCHIVE_REPO_BUCKET_NAME']}/versions/#{version}/state.tar.zst" + run_bash( + sprintf('gsutil -m cp %s - | zstd -dc | tar -xC %s', + Shellwords.escape(url), + Shellwords.escape(@archive_state_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def identify_eol_distros + if @explicit_distros + log_notice "Using explicitly specified distros: #{@explicit_distros.join(', ')}" + return @explicit_distros + end + + published = list_aptly_repos(@main_aptly_config_path) + supported = distributions + .select { |d| d[:package_format] == :DEB } + .map { |d| d[:name] } + + eol = published - supported + log_info "Published distros in Aptly state: #{published.join(', ')}" + log_info "Currently supported DEB distros: #{supported.join(', ')}" + log_info "EOL distros (published but not supported): #{eol.join(', ')}" + eol.sort + end + + def list_aptly_repos(config_path) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'repo', 'list', + "-config=#{config_path}", + '-raw', + log_invocation: true, + check_error: true + ) + stdout_output.split("\n").map(&:strip).reject(&:empty?) + end + + def list_aptly_packages(config_path, repo_name) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'repo', 'show', + "-config=#{config_path}", + '-with-packages', + repo_name, + log_invocation: false, + check_error: true + ) + stdout_output.sub!(/.*^Packages:$/m, '') + stdout_output.split("\n").map(&:strip).reject(&:empty?) + end + + def create_archive_state(eol_distros) + if @archive_version == 0 + initialize_aptly(@archive_aptly_config_path, @archive_state_path, @archive_state_repo_path) + else + # Existing archive was fetched — just initialize the Aptly config pointing at it + initialize_aptly(@archive_aptly_config_path, @archive_state_path, @archive_state_repo_path) + existing = list_aptly_repos(@archive_aptly_config_path) + log_notice "Existing archive contains distros: #{existing.join(', ')}" + end + + # Copy EOL packages from main pool into archive pool + main_pool = "#{@main_state_path}/pool" + archive_pool = "#{@archive_state_path}/pool" + if File.exist?(main_pool) + if File.exist?(archive_pool) + # Merge: copy new pool files that don't already exist + run_bash( + sprintf('cp -rn %s/* %s/ 2>/dev/null || true', + Shellwords.escape(main_pool), + Shellwords.escape(archive_pool)), + log_invocation: false, check_error: false, pipefail: false + ) + else + FileUtils.cp_r(main_pool, @archive_state_path) + end + end + + eol_distros.each do |distro| + packages = list_aptly_packages(@main_aptly_config_path, distro) + log_notice "[#{distro}] Exporting #{packages.size} packages to archive" + + # Create the archive repo for this distro + run_command( + 'aptly', 'repo', 'create', + "-config=#{@archive_aptly_config_path}", + distro, + log_invocation: true, + check_error: true + ) + + # Import packages by copying the Aptly database directory for this repo + main_repo_db = "#{@main_state_db_path}/repo/#{distro}" + if File.exist?(main_repo_db) + FileUtils.cp_r(main_repo_db, "#{@archive_state_db_path}/repo/") + end + end + + # Publish all distros in the archive (existing + newly added) + all_archive_distros = list_aptly_repos(@archive_aptly_config_path) + all_archive_distros.each do |distro| + log_notice "[#{distro}] Publishing in archive" + publish_aptly_repo(@archive_aptly_config_path, distro) + end + + # Archive the state + log_notice 'Creating archive state archive' + run_bash( + sprintf("tar -C %s -cf - . | zstd -T0 > %s", + Shellwords.escape(@archive_state_path), + Shellwords.escape(@archive_state_archive_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def trim_main_state(eol_distros) + eol_distros.each do |distro| + log_notice "[#{distro}] Dropping from main repository" + + # Drop the publication first (if it exists) + run_command( + 'aptly', 'publish', 'drop', + "-config=#{@main_aptly_config_path}", + distro, 'filesystem:main:.', + log_invocation: true, + check_error: false + ) + + # Drop the repo + run_command( + 'aptly', 'repo', 'drop', + "-config=#{@main_aptly_config_path}", + distro, + log_invocation: true, + check_error: true + ) + end + + # Compact the database to reclaim space from dropped packages + log_notice 'Compacting main state' + run_command( + 'aptly', 'db', 'cleanup', + "-config=#{@main_aptly_config_path}", + '-verbose', + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Re-publish remaining distros + remaining = list_aptly_repos(@main_aptly_config_path) + remaining.each do |distro| + log_notice "[#{distro}] Re-publishing in main repository" + publish_aptly_repo(@main_aptly_config_path, distro) + end + + # Re-archive the trimmed state + log_notice 'Creating trimmed main state archive' + run_bash( + sprintf("tar -C %s -cf - . | zstd -T0 > %s", + Shellwords.escape(@main_state_path), + Shellwords.escape(@main_state_archive_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def publish_aptly_repo(config_path, distro) + _, stderr_output, status = run_command_capture_output( + 'aptly', 'publish', 'repo', + '-batch', '-force-overwrite', + "-config=#{config_path}", + "-gpg-key=#{@gpg_key_id}", + "-distribution=#{distro}", + "-origin=#{REPO_ORIGIN}", + "-label=#{REPO_LABEL}", + distro, 'filesystem:main:.', + log_invocation: true, + check_error: false + ) + if !status.success? + if stderr_output =~ /unable to figure out list of architectures/ + run_command( + 'aptly', 'publish', 'repo', + '-batch', '-force-overwrite', '-architectures=all', + "-config=#{config_path}", + "-gpg-key=#{@gpg_key_id}", + "-distribution=#{distro}", + "-origin=#{REPO_ORIGIN}", + "-label=#{REPO_LABEL}", + distro, 'filesystem:main:.', + log_invocation: true, + check_error: true + ) + else + abort("ERROR publishing #{distro}: #{stderr_output.chomp}") + end + end + end + + def upload_archive + archive_bucket = ENV['ARCHIVE_REPO_BUCKET_NAME'] + new_archive_version = @archive_version + 1 + + # Upload state + log_notice "Uploading archive state as version #{new_archive_version}" + run_command( + 'gsutil', + '-h', 'Cache-Control:public', + 'cp', + @archive_state_archive_path, + "gs://#{archive_bucket}/versions/#{new_archive_version}/state.tar.zst", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Upload published repo + log_notice 'Uploading archive repository' + run_command( + 'gsutil', '-m', + '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + @archive_state_repo_path, + "gs://#{archive_bucket}/versions/#{new_archive_version}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Create version note + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{archive_bucket}/versions/#{new_archive_version}/version.txt"), + Shellwords.escape(new_archive_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + + # Declare latest version + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{archive_bucket}/versions/latest_version.txt"), + Shellwords.escape(new_archive_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + end + + def upload_main(old_version) + new_version = old_version + 1 + bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] + + # Upload state + log_notice "Uploading trimmed main state as version #{new_version}" + run_command( + 'gsutil', + '-h', 'Cache-Control:public', + 'cp', + @main_state_archive_path, + "gs://#{bucket}/versions/#{new_version}/state.tar.zst", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Copy previous repo version, then overwrite with trimmed version + log_notice "Copying repo version #{old_version} to #{new_version}" + run_command( + 'gsutil', '-m', + '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "gs://#{bucket}/versions/#{old_version}/public", + "gs://#{bucket}/versions/#{new_version}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + log_notice "Uploading trimmed repo as version #{new_version}" + run_command( + 'gsutil', '-m', + '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + @main_state_repo_path, + "gs://#{bucket}/versions/#{new_version}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Version note + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:public cp - %s <<<%s', + Shellwords.escape("gs://#{bucket}/versions/#{new_version}/version.txt"), + Shellwords.escape(new_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + + # Activate new version + log_notice "Activating main repo version #{new_version}" + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{bucket}/versions/latest_version.txt"), + Shellwords.escape(new_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + end + + def print_summary(eol_distros, old_version) + new_archive_version = @archive_version + 1 + archive_bucket = ENV['ARCHIVE_REPO_BUCKET_NAME'] + log_notice 'Migration summary' + log_info "Archived distributions: #{eol_distros.join(', ')}" + log_info "Main repo: version #{old_version} -> #{old_version + 1}" + log_info "Archive repo: version #{@archive_version} -> #{new_archive_version}" + log_info '' + log_info 'Archive APT repo URL:' + log_info " https://storage.googleapis.com/#{archive_bucket}/versions/#{new_archive_version}/public" + log_info '' + log_info 'Next steps:' + log_info ' 1. Restart the web server to pick up new archive version' + log_info ' 2. Verify archive repo: curl https://apt-archive.fullstaqruby.org/dists/' + log_info ' 3. Verify main repo still works: apt-get update on a supported distro' + end + + def cleanup + log_info "Cleaning up #{@temp_dir}" + FileUtils.remove_entry_secure(@temp_dir) + end +end + +MigrateAptToArchive.new.main diff --git a/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb b/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb new file mode 100755 index 00000000..6ef14685 --- /dev/null +++ b/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb @@ -0,0 +1,324 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Incremental migration script to move EOL distribution packages +# from the main YUM repository to the archive YUM repository. +# Safe to run repeatedly — merges new EOL distros into the existing archive. +# +# Usage: +# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo-archive \ +# ./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb [--dry-run] [--distros centos-8] +# +# If --distros is not specified, automatically detects EOL distros by comparing +# the repo contents against the current environments/ directory. + +require_relative '../../../lib/ci_workflow_support' +require_relative '../../../lib/shell_scripting_support' +require_relative '../../../lib/publishing_support' +require 'shellwords' +require 'tmpdir' +require 'fileutils' +require 'optparse' + +class MigrateYumToArchive + include CiWorkflowSupport + include ShellScriptingSupport + include PublishingSupport + + def main + parse_options + require_envvar 'PRODUCTION_REPO_BUCKET_NAME' + require_envvar 'ARCHIVE_REPO_BUCKET_NAME' + + print_header 'Initializing' + load_config + create_temp_dirs + fetch_and_import_signing_key + + print_header 'Downloading main repository' + version = get_latest_production_repo_version + if version == 0 + abort 'ERROR: No production repository exists yet' + end + fetch_main_repo(version) + + print_header 'Identifying EOL distributions' + eol_distros = identify_eol_distros + if eol_distros.empty? + log_notice 'No EOL distributions found to archive' + exit + end + log_notice "EOL distributions to archive: #{eol_distros.join(', ')}" + + print_header 'Fetching existing archive (if any)' + @archive_version = get_latest_archive_version + if @archive_version > 0 + log_notice "Existing archive at version #{@archive_version}, will merge" + fetch_archive_repo(@archive_version) + else + log_notice 'No existing archive — creating fresh' + @archive_repo_path = "#{@temp_dir}/archive-repo" + Dir.mkdir(@archive_repo_path) + end + + if @dry_run + log_notice 'DRY RUN — not uploading changes' + print_summary(eol_distros, version) + exit + end + + print_header 'Uploading EOL distros to archive' + upload_archive(eol_distros) + + print_header 'Removing EOL distros from main repository' + remove_from_main(eol_distros, version) + + print_header 'Success!' + print_summary(eol_distros, version) + + cleanup + end + +private + def parse_options + @dry_run = false + @explicit_distros = nil + + OptionParser.new do |opts| + opts.banner = "Usage: #{$0} [options]" + opts.on('--dry-run', 'Do not upload changes') { @dry_run = true } + opts.on('--distros DISTROS', 'Comma-separated list of distros to archive') do |v| + @explicit_distros = v.split(',').map(&:strip) + end + end.parse! + end + + def create_temp_dirs + @temp_dir = Dir.mktmpdir('yum-archive-migration') + @local_repo_path = "#{@temp_dir}/repo" + @signing_key_path = "#{@temp_dir}/key.gpg" + Dir.mkdir(@local_repo_path) + end + + def fetch_and_import_signing_key + log_notice 'Fetching and importing signing key' + File.open(@signing_key_path, 'wb') do |f| + f.write(fetch_signing_key) + end + @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) + log_info "Signing key ID: #{@gpg_key_id}" + import_gpg_key(@temp_dir, @signing_key_path) + end + + def latest_production_version_note_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt" + end + + def get_latest_archive_version + url = "gs://#{ENV['ARCHIVE_REPO_BUCKET_NAME']}/versions/latest_version.txt" + stdout_output, stderr_output, status = run_command_capture_output( + 'gsutil', 'cp', url, '-', + log_invocation: false, + check_error: false + ) + if status.success? + v = stdout_output.strip + if v =~ /\A[0-9]+\Z/ + v.to_i + else + abort("ERROR: invalid version number stored in #{url}") + end + elsif stderr_output =~ /No URLs matched/ + 0 + else + abort("ERROR: error fetching #{url}: #{stderr_output.chomp}") + end + end + + def fetch_archive_repo(version) + @archive_repo_path = "#{@temp_dir}/archive-repo" + Dir.mkdir(@archive_repo_path) + log_notice "Fetching archive repo version #{version}" + run_command( + 'gsutil', '-m', 'rsync', '-r', + "gs://#{ENV['ARCHIVE_REPO_BUCKET_NAME']}/versions/#{version}/public", + @archive_repo_path, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def fetch_main_repo(version) + log_notice "Fetching main repo version #{version}" + run_command( + 'gsutil', '-m', 'rsync', '-r', + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/public", + @local_repo_path, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def identify_eol_distros + if @explicit_distros + log_notice "Using explicitly specified distros: #{@explicit_distros.join(', ')}" + return @explicit_distros + end + + # List distro directories in the downloaded repo + published = Dir.entries(@local_repo_path) + .select { |e| File.directory?("#{@local_repo_path}/#{e}") } + .reject { |e| e.start_with?('.') } + + supported = distributions + .select { |d| d[:package_format] == :RPM } + .map { |d| d[:name] } + + eol = published - supported + log_info "Published distros in YUM repo: #{published.join(', ')}" + log_info "Currently supported RPM distros: #{supported.join(', ')}" + log_info "EOL distros (published but not supported): #{eol.join(', ')}" + eol.sort + end + + def upload_archive(eol_distros) + archive_bucket = ENV['ARCHIVE_REPO_BUCKET_NAME'] + new_archive_version = @archive_version + 1 + + # Copy EOL distro directories into the local archive repo + eol_distros.each do |distro| + src = "#{@local_repo_path}/#{distro}" + dst = "#{@archive_repo_path}/#{distro}" + log_notice "[#{distro}] Copying to archive staging area" + FileUtils.cp_r(src, dst) + end + + # Upload merged archive repo + log_notice "Uploading archive repo as version #{new_archive_version}" + run_command( + 'gsutil', '-m', + '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + @archive_repo_path, + "gs://#{archive_bucket}/versions/#{new_archive_version}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Create version note + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{archive_bucket}/versions/#{new_archive_version}/version.txt"), + Shellwords.escape(new_archive_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + + # Declare latest version + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{archive_bucket}/versions/latest_version.txt"), + Shellwords.escape(new_archive_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + end + + def remove_from_main(eol_distros, old_version) + new_version = old_version + 1 + bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] + + # Remove EOL distro directories from local copy + eol_distros.each do |distro| + distro_path = "#{@local_repo_path}/#{distro}" + if File.exist?(distro_path) + log_notice "[#{distro}] Removing from local repo" + FileUtils.rm_rf(distro_path) + end + end + + # Upload trimmed repo as new version + log_notice "Copying repo version #{old_version} to #{new_version}" + run_command( + 'gsutil', '-m', + '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "gs://#{bucket}/versions/#{old_version}/public", + "gs://#{bucket}/versions/#{new_version}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + log_notice "Uploading trimmed repo as version #{new_version}" + run_command( + 'gsutil', '-m', + '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + @local_repo_path, + "gs://#{bucket}/versions/#{new_version}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + # Version note + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:public cp - %s <<<%s', + Shellwords.escape("gs://#{bucket}/versions/#{new_version}/version.txt"), + Shellwords.escape(new_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + + # Activate + log_notice "Activating main repo version #{new_version}" + run_bash( + sprintf( + 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{bucket}/versions/latest_version.txt"), + Shellwords.escape(new_version.to_s) + ), + log_invocation: true, + check_error: true, + pipefail: false + ) + end + + def print_summary(eol_distros, old_version) + new_archive_version = @archive_version + 1 + archive_bucket = ENV['ARCHIVE_REPO_BUCKET_NAME'] + log_notice 'Migration summary' + log_info "Archived distributions: #{eol_distros.join(', ')}" + log_info "Main repo: version #{old_version} -> #{old_version + 1}" + log_info "Archive repo: version #{@archive_version} -> #{new_archive_version}" + log_info '' + log_info 'Archive YUM repo URL:' + log_info " https://storage.googleapis.com/#{archive_bucket}/versions/#{new_archive_version}/public" + log_info '' + log_info 'Next steps:' + log_info ' 1. Restart the web server to pick up new archive version' + log_info ' 2. Verify archive repo: curl https://yum-archive.fullstaqruby.org/' + log_info ' 3. Verify main repo still works on a supported distro' + end + + def cleanup + log_info "Cleaning up #{@temp_dir}" + FileUtils.remove_entry_secure(@temp_dir) + end +end + +MigrateYumToArchive.new.main diff --git a/internal-scripts/ci-cd/archive/prune-apt-packages.rb b/internal-scripts/ci-cd/archive/prune-apt-packages.rb new file mode 100755 index 00000000..b5d90bd3 --- /dev/null +++ b/internal-scripts/ci-cd/archive/prune-apt-packages.rb @@ -0,0 +1,362 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Prune EOL Ruby version packages from the APT repository state. +# +# Removes packages for Ruby versions no longer in config.yml from all +# distro repos in the Aptly state, then compacts and re-uploads. +# +# Usage: +# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +# ./internal-scripts/ci-cd/archive/prune-apt-packages.rb [--dry-run] +# +# Automatically detects which Ruby versions are EOL by comparing +# packages in the Aptly state against minor_version_packages in config.yml. + +require_relative '../../../lib/gcloud_storage_lock' +require_relative '../../../lib/ci_workflow_support' +require_relative '../../../lib/shell_scripting_support' +require_relative '../../../lib/publishing_support' +require 'json' +require 'shellwords' +require 'tmpdir' +require 'set' +require 'fileutils' +require 'optparse' + +class PruneAptPackages + REPO_ORIGIN = 'Fullstaq-Ruby' + REPO_LABEL = 'Fullstaq-Ruby' + + # Matches: fullstaq-ruby-3.1, fullstaq-ruby-3.1-jemalloc, fullstaq-ruby-3.1.7, etc. + # Does NOT match: fullstaq-ruby-common, fullstaq-rbenv + RUBY_PACKAGE_PATTERN = /\Afullstaq-ruby-(\d+\.\d+)/ + + include CiWorkflowSupport + include ShellScriptingSupport + include PublishingSupport + + def main + parse_options + require_envvar 'PRODUCTION_REPO_BUCKET_NAME' + + print_header 'Initializing' + load_config + create_temp_dirs + ensure_gpg_state_isolated + activate_wrappers_bin_dir + initialize_aptly + fetch_and_import_signing_key + + print_header 'Downloading repository state' + version = get_latest_production_repo_version + if version == 0 + abort 'ERROR: No production repository exists yet' + end + fetch_state(version) + + print_header 'Identifying EOL Ruby versions' + active_minors = active_ruby_minor_versions + log_info "Active Ruby minor versions: #{active_minors.join(', ')}" + + print_header 'Scanning packages' + total_pruned = 0 + repos = list_aptly_repos + repos.each do |distro| + pruned = prune_eol_packages_from_repo(distro, active_minors) + total_pruned += pruned + end + + if total_pruned == 0 + log_notice 'No EOL Ruby packages found to prune' + exit + end + + log_notice "Total packages to prune: #{total_pruned}" + + if @dry_run + log_notice 'DRY RUN — not uploading changes' + exit + end + + print_header 'Compacting state' + compact_state + + print_header 'Re-publishing repository' + repos.each do |distro| + publish_repo(distro) + end + + print_header 'Archiving and uploading state' + archive_state + upload_state(version + 1) + upload_repo(version + 1, version) + create_version_note(version + 1) + declare_latest_version(version + 1) + + print_header 'Success!' + log_info "Pruned #{total_pruned} packages across #{repos.size} repos" + log_info "Main repo: version #{version} -> #{version + 1}" + + cleanup + end + +private + def parse_options + @dry_run = false + OptionParser.new do |opts| + opts.banner = "Usage: #{$0} [options]" + opts.on('--dry-run', 'Do not upload changes') { @dry_run = true } + end.parse! + end + + def create_temp_dirs + @temp_dir = Dir.mktmpdir('apt-prune') + @wrappers_bin_dir = "#{@temp_dir}/wrappers" + @aptly_config_path = "#{@temp_dir}/aptly.conf" + @signing_key_path = "#{@temp_dir}/key.gpg" + @local_state_path = "#{@temp_dir}/state" + @local_state_db_path = "#{@local_state_path}/db" + @local_state_repo_path = "#{@local_state_path}/repo" + @local_state_archive_path = "#{@temp_dir}/state.tar.zst" + + Dir.mkdir(@wrappers_bin_dir) + FileUtils.mkdir_p(@local_state_db_path) + FileUtils.mkdir_p(@local_state_repo_path) + end + + def ensure_gpg_state_isolated + File.open("#{@wrappers_bin_dir}/gpg", 'w:utf-8') do |f| + f.write("#!/bin/sh\n") + f.write( + sprintf("exec %s --homedir %s \"$@\"\n", + Shellwords.escape(find_gpg), + Shellwords.escape(@temp_dir)) + ) + end + File.chmod(0755, "#{@wrappers_bin_dir}/gpg") + end + + def find_gpg + ENV['PATH'].split(':').each do |dir| + next if dir == @wrappers_bin_dir + candidate = "#{dir}/gpg" + return candidate if File.exist?(candidate) + end + abort('GPG not found') + end + + def activate_wrappers_bin_dir + ENV['PATH'] = "#{@wrappers_bin_dir}:#{ENV['PATH']}" + end + + def initialize_aptly + File.open(@aptly_config_path, 'w:utf-8') do |f| + f.write(JSON.generate( + rootDir: @local_state_path, + FileSystemPublishEndpoints: { + main: { + rootDir: @local_state_repo_path, + linkMethod: 'symlink', + verifyMethod: 'md5' + } + } + )) + end + end + + def fetch_and_import_signing_key + log_notice 'Fetching and importing signing key' + File.open(@signing_key_path, 'wb') do |f| + f.write(fetch_signing_key) + end + @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) + log_info "Signing key ID: #{@gpg_key_id}" + import_gpg_key(@temp_dir, @signing_key_path) + end + + def latest_production_version_note_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt" + end + + def fetch_state(version) + log_notice "Fetching state version #{version}" + run_bash( + sprintf('gsutil -m cp %s - | zstd -dc | tar -xC %s', + Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/state.tar.zst"), + Shellwords.escape(@local_state_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def active_ruby_minor_versions + config['ruby']['minor_version_packages'].map { |p| p['minor_version'] } + end + + def list_aptly_repos + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'repo', 'list', + "-config=#{@aptly_config_path}", '-raw', + log_invocation: true, check_error: true + ) + stdout_output.split("\n").map(&:strip).reject(&:empty?) + end + + def list_packages(distro) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'repo', 'show', + "-config=#{@aptly_config_path}", + '-with-packages', distro, + log_invocation: false, check_error: true + ) + stdout_output.sub!(/.*^Packages:$/m, '') + stdout_output.split("\n").map(&:strip).reject(&:empty?) + end + + def prune_eol_packages_from_repo(distro, active_minors) + packages = list_packages(distro) + eol_packages = packages.select do |pkg| + if pkg =~ RUBY_PACKAGE_PATTERN + minor = $1 + !active_minors.include?(minor) + else + false + end + end + + if eol_packages.empty? + log_info "[#{distro}] No EOL packages found" + return 0 + end + + log_notice "[#{distro}] Pruning #{eol_packages.size} EOL packages (of #{packages.size} total)" + eol_packages.each { |pkg| log_info " #{YELLOW}PRUNE#{RESET} #{pkg}" } + + if !@dry_run + eol_packages.each_slice(50) do |batch| + query = batch.join(' | ') + run_command( + 'aptly', 'repo', 'remove', + "-config=#{@aptly_config_path}", + distro, query, + log_invocation: false, + check_error: true + ) + end + end + + eol_packages.size + end + + def compact_state + run_command( + 'aptly', 'db', 'cleanup', + "-config=#{@aptly_config_path}", '-verbose', + log_invocation: true, check_error: true, passthru_output: true + ) + end + + def publish_repo(distro) + log_notice "[#{distro}] Publishing" + _, stderr_output, status = run_command_capture_output( + 'aptly', 'publish', 'repo', + '-batch', '-force-overwrite', + "-config=#{@aptly_config_path}", + "-gpg-key=#{@gpg_key_id}", + "-distribution=#{distro}", + "-origin=#{REPO_ORIGIN}", + "-label=#{REPO_LABEL}", + distro, 'filesystem:main:.', + log_invocation: true, check_error: false + ) + if !status.success? + if stderr_output =~ /unable to figure out list of architectures/ + run_command( + 'aptly', 'publish', 'repo', + '-batch', '-force-overwrite', '-architectures=all', + "-config=#{@aptly_config_path}", + "-gpg-key=#{@gpg_key_id}", + "-distribution=#{distro}", + "-origin=#{REPO_ORIGIN}", + "-label=#{REPO_LABEL}", + distro, 'filesystem:main:.', + log_invocation: true, check_error: true + ) + else + abort("ERROR publishing #{distro}: #{stderr_output.chomp}") + end + end + end + + def archive_state + log_notice 'Creating state archive' + run_bash( + sprintf("tar -C %s -cf - . | zstd -T0 > %s", + Shellwords.escape(@local_state_path), + Shellwords.escape(@local_state_archive_path)), + pipefail: true, + log_invocation: true, check_error: true, passthru_output: true + ) + end + + def upload_state(version) + bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] + run_command( + 'gsutil', '-h', 'Cache-Control:public', 'cp', + @local_state_archive_path, + "gs://#{bucket}/versions/#{version}/state.tar.zst", + log_invocation: true, check_error: true, passthru_output: true + ) + end + + def upload_repo(version, old_version) + bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] + + log_notice "Copying repo version #{old_version} to #{version}" + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "gs://#{bucket}/versions/#{old_version}/public", + "gs://#{bucket}/versions/#{version}/public", + log_invocation: true, check_error: true, passthru_output: true + ) + + log_notice "Uploading pruned repo as version #{version}" + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + @local_state_repo_path, + "gs://#{bucket}/versions/#{version}/public", + log_invocation: true, check_error: true, passthru_output: true + ) + end + + def create_version_note(version) + run_bash( + sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:public cp - %s <<<%s', + Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/version.txt"), + Shellwords.escape(version.to_s)), + log_invocation: true, check_error: true, pipefail: false + ) + end + + def declare_latest_version(version) + log_notice "Activating version #{version}" + run_bash( + sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt"), + Shellwords.escape(version.to_s)), + log_invocation: true, check_error: true, pipefail: false + ) + end + + def cleanup + log_info "Cleaning up #{@temp_dir}" + FileUtils.remove_entry_secure(@temp_dir) + end +end + +PruneAptPackages.new.main diff --git a/internal-scripts/ci-cd/archive/prune-yum-packages.rb b/internal-scripts/ci-cd/archive/prune-yum-packages.rb new file mode 100755 index 00000000..6bd9e1f5 --- /dev/null +++ b/internal-scripts/ci-cd/archive/prune-yum-packages.rb @@ -0,0 +1,219 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Prune EOL Ruby version packages from the YUM repository. +# +# Removes RPM files for Ruby versions no longer in config.yml from all +# distro/arch directories, regenerates repodata, and re-uploads. +# +# Usage: +# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +# ./internal-scripts/ci-cd/archive/prune-yum-packages.rb [--dry-run] + +require_relative '../../../lib/ci_workflow_support' +require_relative '../../../lib/shell_scripting_support' +require_relative '../../../lib/publishing_support' +require 'shellwords' +require 'tmpdir' +require 'fileutils' +require 'optparse' + +class PruneYumPackages + # Matches: fullstaq-ruby-3.1.7-jemalloc-rev1-centos8.x86_64.rpm + RUBY_RPM_PATTERN = /\Afullstaq-ruby-(\d+\.\d+)/ + + include CiWorkflowSupport + include ShellScriptingSupport + include PublishingSupport + + def main + parse_options + require_envvar 'PRODUCTION_REPO_BUCKET_NAME' + + print_header 'Initializing' + load_config + create_temp_dirs + pull_utility_image_if_not_exists + fetch_and_import_signing_key + + print_header 'Downloading repository' + version = get_latest_production_repo_version + if version == 0 + abort 'ERROR: No production repository exists yet' + end + fetch_repo(version) + + print_header 'Identifying EOL Ruby versions' + active_minors = active_ruby_minor_versions + log_info "Active Ruby minor versions: #{active_minors.join(', ')}" + + print_header 'Scanning and pruning packages' + total_pruned = 0 + affected_dirs = [] + + Dir.glob("#{@local_repo_path}/*/*").each do |arch_dir| + next unless File.directory?(arch_dir) + distro = File.basename(File.dirname(arch_dir)) + arch = File.basename(arch_dir) + + rpms = Dir.glob("#{arch_dir}/fullstaq-ruby-*.rpm") + eol_rpms = rpms.select do |rpm| + basename = File.basename(rpm) + if basename =~ RUBY_RPM_PATTERN + !active_minors.include?($1) + else + false + end + end + + next if eol_rpms.empty? + + log_notice "[#{distro}/#{arch}] Pruning #{eol_rpms.size} EOL packages (of #{rpms.size} total)" + eol_rpms.each { |rpm| log_info " #{YELLOW}PRUNE#{RESET} #{File.basename(rpm)}" } + total_pruned += eol_rpms.size + + if !@dry_run + eol_rpms.each { |rpm| File.delete(rpm) } + affected_dirs << arch_dir + end + end + + if total_pruned == 0 + log_notice 'No EOL Ruby packages found to prune' + exit + end + + log_notice "Total packages pruned: #{total_pruned}" + + if @dry_run + log_notice 'DRY RUN — not uploading changes' + exit + end + + print_header 'Regenerating repo metadata' + affected_dirs.each do |dir| + invoke_createrepo(dir) + sign_repo(dir) + end + + print_header 'Uploading pruned repository' + upload_repo(version + 1, version) + create_version_note(version + 1) + declare_latest_version(version + 1) + + print_header 'Success!' + log_info "Pruned #{total_pruned} packages" + log_info "Main repo: version #{version} -> #{version + 1}" + + cleanup + end + +private + def parse_options + @dry_run = false + OptionParser.new do |opts| + opts.banner = "Usage: #{$0} [options]" + opts.on('--dry-run', 'Do not upload changes') { @dry_run = true } + end.parse! + end + + def create_temp_dirs + @temp_dir = Dir.mktmpdir('yum-prune') + @local_repo_path = "#{@temp_dir}/repo" + @signing_key_path = "#{@temp_dir}/key.gpg" + Dir.mkdir(@local_repo_path) + end + + def fetch_and_import_signing_key + log_notice 'Fetching and importing signing key' + File.open(@signing_key_path, 'wb') do |f| + f.write(fetch_signing_key) + end + @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) + import_gpg_key(@temp_dir, @signing_key_path) + end + + def latest_production_version_note_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt" + end + + def fetch_repo(version) + run_command( + 'gsutil', '-m', 'rsync', '-r', + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/public", + @local_repo_path, + log_invocation: true, check_error: true, passthru_output: true + ) + end + + def active_ruby_minor_versions + config['ruby']['minor_version_packages'].map { |p| p['minor_version'] } + end + + def invoke_createrepo(dir) + run_command( + 'docker', 'run', '--rm', + '-v', "#{dir}:/input:delegated", + '--user', "#{Process.uid}:#{Process.gid}", + utility_image_name, + 'createrepo_c', '--update', '/input', + log_invocation: true, check_error: true + ) + end + + def sign_repo(path) + run_command( + 'gpg', "--homedir=#{@temp_dir}", "--local-user=#{@gpg_key_id}", + '--batch', '--yes', '--detach-sign', '--armor', + "#{path}/repodata/repomd.xml", + log_invocation: true, check_error: true + ) + end + + def upload_repo(version, old_version) + bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] + + log_notice "Copying repo version #{old_version} to #{version}" + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "gs://#{bucket}/versions/#{old_version}/public", + "gs://#{bucket}/versions/#{version}/public", + log_invocation: true, check_error: true, passthru_output: true + ) + + log_notice "Uploading pruned repo as version #{version}" + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + @local_repo_path, + "gs://#{bucket}/versions/#{version}/public", + log_invocation: true, check_error: true, passthru_output: true + ) + end + + def create_version_note(version) + run_bash( + sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:public cp - %s <<<%s', + Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/version.txt"), + Shellwords.escape(version.to_s)), + log_invocation: true, check_error: true, pipefail: false + ) + end + + def declare_latest_version(version) + run_bash( + sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', + Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt"), + Shellwords.escape(version.to_s)), + log_invocation: true, check_error: true, pipefail: false + ) + end + + def cleanup + log_info "Cleaning up #{@temp_dir}" + FileUtils.remove_entry_secure(@temp_dir) + end +end + +PruneYumPackages.new.main From 29516b9756a534fe5e538d04e00fdc848b73750d Mon Sep 17 00:00:00 2001 From: abtreece Date: Mon, 27 Apr 2026 21:49:58 -0500 Subject: [PATCH 02/12] fix: harden EOL archive tooling - Wire up GCloudStorageLock in all four archive scripts so migrations and prunes mutually exclude with the publish workflow's pointer updates. The lock library was already imported but never used. - Wrap all four scripts in begin/ensure cleanup so a failure mid-run no longer leaves the imported GPG signing key on disk. - migrate-apt-to-archive: stop swallowing pool-merge errors via '|| true'. Real I/O/permission failures must surface; only no-clobber skips were ever benign. - migrate-apt-to-archive / migrate-yum-to-archive: comments claimed EOL detection compared against environments/, but the code compares against config.yml. Align the comments with the code. - archiving-eol-packages.md: APT explicit-distros example listed centos-8, which is RPM-only and would fail in the APT script. Replace with debian-10,ubuntu-20.04. --- dev-handbook/archiving-eol-packages.md | 4 +- .../ci-cd/archive/migrate-apt-to-archive.rb | 124 +++++++++------ .../ci-cd/archive/migrate-yum-to-archive.rb | 108 ++++++++----- .../ci-cd/archive/prune-apt-packages.rb | 124 +++++++++------ .../ci-cd/archive/prune-yum-packages.rb | 149 +++++++++++------- 5 files changed, 309 insertions(+), 200 deletions(-) diff --git a/dev-handbook/archiving-eol-packages.md b/dev-handbook/archiving-eol-packages.md index 3519ee78..619e9ef1 100644 --- a/dev-handbook/archiving-eol-packages.md +++ b/dev-handbook/archiving-eol-packages.md @@ -69,10 +69,10 @@ ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \ ./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb --dry-run ~~~ -The script auto-detects EOL distros by comparing `aptly repo list` output against the distributions defined in `config.yml`. You can also specify distros explicitly: +The script auto-detects EOL distros by comparing `aptly repo list` output against the DEB distributions defined in `config.yml`. You can also specify distros explicitly (DEB-only — RPM distros belong to the YUM script): ~~~bash -./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb --dry-run --distros centos-8,debian-9 +./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb --dry-run --distros debian-10,ubuntu-20.04 ~~~ **Execute the migration** (removes `--dry-run`): diff --git a/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb b/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb index 1bb16f40..815ab1af 100755 --- a/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb +++ b/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb @@ -11,7 +11,7 @@ # ./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb [--dry-run] [--distros centos-8,debian-9] # # If --distros is not specified, automatically detects EOL distros by comparing -# the Aptly state against the current environments/ directory. +# the Aptly state against the DEB distributions defined in config.yml. require_relative '../../../lib/gcloud_storage_lock' require_relative '../../../lib/ci_workflow_support' @@ -42,55 +42,66 @@ def main create_temp_dirs ensure_gpg_state_isolated activate_wrappers_bin_dir + initialize_locking initialize_aptly(@main_aptly_config_path, @main_state_path, @main_state_repo_path) fetch_and_import_signing_key - print_header 'Downloading main repository state' - version = get_latest_production_repo_version - if version == 0 - abort 'ERROR: No production repository exists yet' - end - fetch_main_state(version) - - print_header 'Identifying EOL distributions' - eol_distros = identify_eol_distros - if eol_distros.empty? - log_notice 'No EOL distributions found to archive' - exit - end - log_notice "EOL distributions to archive: #{eol_distros.join(', ')}" - - print_header 'Fetching existing archive state (if any)' - @archive_version = get_latest_archive_version - if @archive_version > 0 - log_notice "Existing archive at version #{@archive_version}, will merge" - fetch_archive_state(@archive_version) - else - log_notice 'No existing archive — creating fresh' - end - - print_header 'Creating archive repository' - create_archive_state(eol_distros) - - print_header 'Trimming main repository state' - trim_main_state(eol_distros) + eol_distros = nil + version = nil + + begin + synchronize do + print_header 'Downloading main repository state' + version = get_latest_production_repo_version + if version == 0 + abort 'ERROR: No production repository exists yet' + end + fetch_main_state(version) + + print_header 'Identifying EOL distributions' + eol_distros = identify_eol_distros + if eol_distros.empty? + log_notice 'No EOL distributions found to archive' + return + end + log_notice "EOL distributions to archive: #{eol_distros.join(', ')}" + + print_header 'Fetching existing archive state (if any)' + @archive_version = get_latest_archive_version + if @archive_version > 0 + log_notice "Existing archive at version #{@archive_version}, will merge" + fetch_archive_state(@archive_version) + else + log_notice 'No existing archive — creating fresh' + end + + print_header 'Creating archive repository' + create_archive_state(eol_distros) + check_lock_health + + print_header 'Trimming main repository state' + trim_main_state(eol_distros) + check_lock_health + + if @dry_run + log_notice 'DRY RUN — not uploading changes' + print_summary(eol_distros, version) + return + end + + print_header 'Uploading archive repository' + upload_archive + check_lock_health + + print_header 'Uploading trimmed main repository' + upload_main(version) + end - if @dry_run - log_notice 'DRY RUN — not uploading changes' - print_summary(eol_distros, version) - exit + print_header 'Success!' + print_summary(eol_distros, version) if eol_distros && !eol_distros.empty? + ensure + cleanup end - - print_header 'Uploading archive repository' - upload_archive - - print_header 'Uploading trimmed main repository' - upload_main(version) - - print_header 'Success!' - print_summary(eol_distros, version) - - cleanup end private @@ -161,6 +172,22 @@ def activate_wrappers_bin_dir ENV['PATH'] = "#{@wrappers_bin_dir}:#{ENV['PATH']}" end + def initialize_locking + @lock = GCloudStorageLock.new(url: lock_url) + end + + def lock_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/locks/apt" + end + + def synchronize(&block) + @lock.synchronize(&block) + end + + def check_lock_health + abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? + end + def initialize_aptly(config_path, state_path, repo_path) log_notice "Creating Aptly config: #{config_path}" File.open(config_path, 'w:utf-8') do |f| @@ -297,12 +324,13 @@ def create_archive_state(eol_distros) archive_pool = "#{@archive_state_path}/pool" if File.exist?(main_pool) if File.exist?(archive_pool) - # Merge: copy new pool files that don't already exist + # Merge: copy new pool files that don't already exist (-n: no-clobber). + # Real I/O errors must surface — only no-clobber skips are benign. run_bash( - sprintf('cp -rn %s/* %s/ 2>/dev/null || true', + sprintf('cp -rn %s/* %s/', Shellwords.escape(main_pool), Shellwords.escape(archive_pool)), - log_invocation: false, check_error: false, pipefail: false + log_invocation: true, check_error: true, pipefail: false ) else FileUtils.cp_r(main_pool, @archive_state_path) diff --git a/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb b/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb index 6ef14685..e5b85cad 100755 --- a/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb +++ b/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb @@ -11,8 +11,9 @@ # ./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb [--dry-run] [--distros centos-8] # # If --distros is not specified, automatically detects EOL distros by comparing -# the repo contents against the current environments/ directory. +# the repo contents against the RPM distributions defined in config.yml. +require_relative '../../../lib/gcloud_storage_lock' require_relative '../../../lib/ci_workflow_support' require_relative '../../../lib/shell_scripting_support' require_relative '../../../lib/publishing_support' @@ -34,50 +35,59 @@ def main print_header 'Initializing' load_config create_temp_dirs + initialize_locking fetch_and_import_signing_key - print_header 'Downloading main repository' - version = get_latest_production_repo_version - if version == 0 - abort 'ERROR: No production repository exists yet' - end - fetch_main_repo(version) - - print_header 'Identifying EOL distributions' - eol_distros = identify_eol_distros - if eol_distros.empty? - log_notice 'No EOL distributions found to archive' - exit - end - log_notice "EOL distributions to archive: #{eol_distros.join(', ')}" - - print_header 'Fetching existing archive (if any)' - @archive_version = get_latest_archive_version - if @archive_version > 0 - log_notice "Existing archive at version #{@archive_version}, will merge" - fetch_archive_repo(@archive_version) - else - log_notice 'No existing archive — creating fresh' - @archive_repo_path = "#{@temp_dir}/archive-repo" - Dir.mkdir(@archive_repo_path) - end + eol_distros = nil + version = nil + + begin + synchronize do + print_header 'Downloading main repository' + version = get_latest_production_repo_version + if version == 0 + abort 'ERROR: No production repository exists yet' + end + fetch_main_repo(version) + + print_header 'Identifying EOL distributions' + eol_distros = identify_eol_distros + if eol_distros.empty? + log_notice 'No EOL distributions found to archive' + return + end + log_notice "EOL distributions to archive: #{eol_distros.join(', ')}" + + print_header 'Fetching existing archive (if any)' + @archive_version = get_latest_archive_version + if @archive_version > 0 + log_notice "Existing archive at version #{@archive_version}, will merge" + fetch_archive_repo(@archive_version) + else + log_notice 'No existing archive — creating fresh' + @archive_repo_path = "#{@temp_dir}/archive-repo" + Dir.mkdir(@archive_repo_path) + end + + if @dry_run + log_notice 'DRY RUN — not uploading changes' + print_summary(eol_distros, version) + return + end + + print_header 'Uploading EOL distros to archive' + upload_archive(eol_distros) + check_lock_health + + print_header 'Removing EOL distros from main repository' + remove_from_main(eol_distros, version) + end - if @dry_run - log_notice 'DRY RUN — not uploading changes' - print_summary(eol_distros, version) - exit + print_header 'Success!' + print_summary(eol_distros, version) if eol_distros && !eol_distros.empty? + ensure + cleanup end - - print_header 'Uploading EOL distros to archive' - upload_archive(eol_distros) - - print_header 'Removing EOL distros from main repository' - remove_from_main(eol_distros, version) - - print_header 'Success!' - print_summary(eol_distros, version) - - cleanup end private @@ -101,6 +111,22 @@ def create_temp_dirs Dir.mkdir(@local_repo_path) end + def initialize_locking + @lock = GCloudStorageLock.new(url: lock_url) + end + + def lock_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/locks/yum" + end + + def synchronize(&block) + @lock.synchronize(&block) + end + + def check_lock_health + abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? + end + def fetch_and_import_signing_key log_notice 'Fetching and importing signing key' File.open(@signing_key_path, 'wb') do |f| diff --git a/internal-scripts/ci-cd/archive/prune-apt-packages.rb b/internal-scripts/ci-cd/archive/prune-apt-packages.rb index b5d90bd3..db709eb6 100755 --- a/internal-scripts/ci-cd/archive/prune-apt-packages.rb +++ b/internal-scripts/ci-cd/archive/prune-apt-packages.rb @@ -45,60 +45,72 @@ def main create_temp_dirs ensure_gpg_state_isolated activate_wrappers_bin_dir + initialize_locking initialize_aptly fetch_and_import_signing_key - print_header 'Downloading repository state' - version = get_latest_production_repo_version - if version == 0 - abort 'ERROR: No production repository exists yet' - end - fetch_state(version) - - print_header 'Identifying EOL Ruby versions' - active_minors = active_ruby_minor_versions - log_info "Active Ruby minor versions: #{active_minors.join(', ')}" - - print_header 'Scanning packages' total_pruned = 0 - repos = list_aptly_repos - repos.each do |distro| - pruned = prune_eol_packages_from_repo(distro, active_minors) - total_pruned += pruned - end - - if total_pruned == 0 - log_notice 'No EOL Ruby packages found to prune' - exit - end - - log_notice "Total packages to prune: #{total_pruned}" - - if @dry_run - log_notice 'DRY RUN — not uploading changes' - exit - end - - print_header 'Compacting state' - compact_state + version = nil + repos = nil + + begin + synchronize do + print_header 'Downloading repository state' + version = get_latest_production_repo_version + if version == 0 + abort 'ERROR: No production repository exists yet' + end + fetch_state(version) + + print_header 'Identifying EOL Ruby versions' + active_minors = active_ruby_minor_versions + log_info "Active Ruby minor versions: #{active_minors.join(', ')}" + + print_header 'Scanning packages' + repos = list_aptly_repos + repos.each do |distro| + pruned = prune_eol_packages_from_repo(distro, active_minors) + total_pruned += pruned + end + + if total_pruned == 0 + log_notice 'No EOL Ruby packages found to prune' + return + end + + log_notice "Total packages to prune: #{total_pruned}" + + if @dry_run + log_notice 'DRY RUN — not uploading changes' + return + end + + print_header 'Compacting state' + compact_state + check_lock_health + + print_header 'Re-publishing repository' + repos.each do |distro| + publish_repo(distro) + end + check_lock_health + + print_header 'Archiving and uploading state' + archive_state + upload_state(version + 1) + upload_repo(version + 1, version) + create_version_note(version + 1) + declare_latest_version(version + 1) + end - print_header 'Re-publishing repository' - repos.each do |distro| - publish_repo(distro) + print_header 'Success!' + if total_pruned > 0 && !@dry_run + log_info "Pruned #{total_pruned} packages across #{repos.size} repos" + log_info "Main repo: version #{version} -> #{version + 1}" + end + ensure + cleanup end - - print_header 'Archiving and uploading state' - archive_state - upload_state(version + 1) - upload_repo(version + 1, version) - create_version_note(version + 1) - declare_latest_version(version + 1) - - print_header 'Success!' - log_info "Pruned #{total_pruned} packages across #{repos.size} repos" - log_info "Main repo: version #{version} -> #{version + 1}" - - cleanup end private @@ -150,6 +162,22 @@ def activate_wrappers_bin_dir ENV['PATH'] = "#{@wrappers_bin_dir}:#{ENV['PATH']}" end + def initialize_locking + @lock = GCloudStorageLock.new(url: lock_url) + end + + def lock_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/locks/apt" + end + + def synchronize(&block) + @lock.synchronize(&block) + end + + def check_lock_health + abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? + end + def initialize_aptly File.open(@aptly_config_path, 'w:utf-8') do |f| f.write(JSON.generate( diff --git a/internal-scripts/ci-cd/archive/prune-yum-packages.rb b/internal-scripts/ci-cd/archive/prune-yum-packages.rb index 6bd9e1f5..ab45fe2f 100755 --- a/internal-scripts/ci-cd/archive/prune-yum-packages.rb +++ b/internal-scripts/ci-cd/archive/prune-yum-packages.rb @@ -10,6 +10,7 @@ # PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ # ./internal-scripts/ci-cd/archive/prune-yum-packages.rb [--dry-run] +require_relative '../../../lib/gcloud_storage_lock' require_relative '../../../lib/ci_workflow_support' require_relative '../../../lib/shell_scripting_support' require_relative '../../../lib/publishing_support' @@ -33,79 +34,89 @@ def main print_header 'Initializing' load_config create_temp_dirs + initialize_locking pull_utility_image_if_not_exists fetch_and_import_signing_key - print_header 'Downloading repository' - version = get_latest_production_repo_version - if version == 0 - abort 'ERROR: No production repository exists yet' - end - fetch_repo(version) - - print_header 'Identifying EOL Ruby versions' - active_minors = active_ruby_minor_versions - log_info "Active Ruby minor versions: #{active_minors.join(', ')}" - - print_header 'Scanning and pruning packages' total_pruned = 0 - affected_dirs = [] - - Dir.glob("#{@local_repo_path}/*/*").each do |arch_dir| - next unless File.directory?(arch_dir) - distro = File.basename(File.dirname(arch_dir)) - arch = File.basename(arch_dir) - - rpms = Dir.glob("#{arch_dir}/fullstaq-ruby-*.rpm") - eol_rpms = rpms.select do |rpm| - basename = File.basename(rpm) - if basename =~ RUBY_RPM_PATTERN - !active_minors.include?($1) - else - false + version = nil + + begin + synchronize do + print_header 'Downloading repository' + version = get_latest_production_repo_version + if version == 0 + abort 'ERROR: No production repository exists yet' + end + fetch_repo(version) + + print_header 'Identifying EOL Ruby versions' + active_minors = active_ruby_minor_versions + log_info "Active Ruby minor versions: #{active_minors.join(', ')}" + + print_header 'Scanning and pruning packages' + affected_dirs = [] + + Dir.glob("#{@local_repo_path}/*/*").each do |arch_dir| + next unless File.directory?(arch_dir) + distro = File.basename(File.dirname(arch_dir)) + arch = File.basename(arch_dir) + + rpms = Dir.glob("#{arch_dir}/fullstaq-ruby-*.rpm") + eol_rpms = rpms.select do |rpm| + basename = File.basename(rpm) + if basename =~ RUBY_RPM_PATTERN + !active_minors.include?($1) + else + false + end + end + + next if eol_rpms.empty? + + log_notice "[#{distro}/#{arch}] Pruning #{eol_rpms.size} EOL packages (of #{rpms.size} total)" + eol_rpms.each { |rpm| log_info " #{YELLOW}PRUNE#{RESET} #{File.basename(rpm)}" } + total_pruned += eol_rpms.size + + if !@dry_run + eol_rpms.each { |rpm| File.delete(rpm) } + affected_dirs << arch_dir + end end - end - next if eol_rpms.empty? + if total_pruned == 0 + log_notice 'No EOL Ruby packages found to prune' + return + end - log_notice "[#{distro}/#{arch}] Pruning #{eol_rpms.size} EOL packages (of #{rpms.size} total)" - eol_rpms.each { |rpm| log_info " #{YELLOW}PRUNE#{RESET} #{File.basename(rpm)}" } - total_pruned += eol_rpms.size + log_notice "Total packages pruned: #{total_pruned}" - if !@dry_run - eol_rpms.each { |rpm| File.delete(rpm) } - affected_dirs << arch_dir - end - end - - if total_pruned == 0 - log_notice 'No EOL Ruby packages found to prune' - exit - end + if @dry_run + log_notice 'DRY RUN — not uploading changes' + return + end - log_notice "Total packages pruned: #{total_pruned}" + print_header 'Regenerating repo metadata' + affected_dirs.each do |dir| + invoke_createrepo(dir) + sign_repo(dir) + end + check_lock_health - if @dry_run - log_notice 'DRY RUN — not uploading changes' - exit - end + print_header 'Uploading pruned repository' + upload_repo(version + 1, version) + create_version_note(version + 1) + declare_latest_version(version + 1) + end - print_header 'Regenerating repo metadata' - affected_dirs.each do |dir| - invoke_createrepo(dir) - sign_repo(dir) + print_header 'Success!' + if total_pruned > 0 && !@dry_run + log_info "Pruned #{total_pruned} packages" + log_info "Main repo: version #{version} -> #{version + 1}" + end + ensure + cleanup end - - print_header 'Uploading pruned repository' - upload_repo(version + 1, version) - create_version_note(version + 1) - declare_latest_version(version + 1) - - print_header 'Success!' - log_info "Pruned #{total_pruned} packages" - log_info "Main repo: version #{version} -> #{version + 1}" - - cleanup end private @@ -124,6 +135,22 @@ def create_temp_dirs Dir.mkdir(@local_repo_path) end + def initialize_locking + @lock = GCloudStorageLock.new(url: lock_url) + end + + def lock_url + "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/locks/yum" + end + + def synchronize(&block) + @lock.synchronize(&block) + end + + def check_lock_health + abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? + end + def fetch_and_import_signing_key log_notice 'Fetching and importing signing key' File.open(@signing_key_path, 'wb') do |f| From 70a7ec0322426d0e680563a5e11aa27c37a4f08b Mon Sep 17 00:00:00 2001 From: abtreece Date: Fri, 22 May 2026 21:09:52 -0500 Subject: [PATCH 03/12] docs: sync archive bucket names with infra rename infra#47 renames the archive buckets from `-repo-archive` to `-archive-repo` for naming consistency with the live buckets (see fullstaq-ruby/infra@c1814eb). Scripts here are bucket-name agnostic (they read `ENV['ARCHIVE_REPO_BUCKET_NAME']`), but the old names were baked into the runbook examples, the architecture doc, and the script header docstrings -- updated here so the runbook stays correct after both PRs merge. No code changes. --- dev-handbook/apt-yum-repo.md | 4 ++-- dev-handbook/archiving-eol-packages.md | 18 +++++++++--------- .../ci-cd/archive/migrate-apt-to-archive.rb | 2 +- .../ci-cd/archive/migrate-yum-to-archive.rb | 2 +- 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/dev-handbook/apt-yum-repo.md b/dev-handbook/apt-yum-repo.md index ad25346b..f19a2013 100644 --- a/dev-handbook/apt-yum-repo.md +++ b/dev-handbook/apt-yum-repo.md @@ -52,8 +52,8 @@ There are five buckets: - `fsruby-server-edition-apt-repo` stores the production APT repository. - `fsruby-server-edition-yum-repo` stores the production YUM repository. - - `fsruby-server-edition-apt-repo-archive` stores packages for EOL distributions (APT). - - `fsruby-server-edition-yum-repo-archive` stores packages for EOL distributions (YUM). + - `fsruby-server-edition-apt-archive-repo` stores packages for EOL distributions (APT). + - `fsruby-server-edition-yum-archive-repo` stores packages for EOL distributions (YUM). - `fsruby-server-edition-ci-artifacts` stores the temporary repositories created during CI runs. The archive buckets are only updated during [EOL migration](archiving-eol-packages.md) — CI never writes to them. Each migration creates a new version that merges newly-archived distros with the existing archive contents. They are served at `apt-archive.fullstaqruby.org` and `yum-archive.fullstaqruby.org`. diff --git a/dev-handbook/archiving-eol-packages.md b/dev-handbook/archiving-eol-packages.md index 619e9ef1..df1cb767 100644 --- a/dev-handbook/archiving-eol-packages.md +++ b/dev-handbook/archiving-eol-packages.md @@ -11,9 +11,9 @@ To address this, we maintain **archive repositories** alongside the main reposit | Repository | Bucket | Domain | Purpose | |------------|--------|--------|---------| | APT (main) | `fsruby-server-edition-apt-repo` | `apt.fullstaqruby.org` | Current, supported packages | -| APT (archive) | `fsruby-server-edition-apt-repo-archive` | `apt-archive.fullstaqruby.org` | Frozen packages for EOL distributions | +| APT (archive) | `fsruby-server-edition-apt-archive-repo` | `apt-archive.fullstaqruby.org` | Frozen packages for EOL distributions | | YUM (main) | `fsruby-server-edition-yum-repo` | `yum.fullstaqruby.org` | Current, supported packages | -| YUM (archive) | `fsruby-server-edition-yum-repo-archive` | `yum-archive.fullstaqruby.org` | Frozen packages for EOL distributions | +| YUM (archive) | `fsruby-server-edition-yum-archive-repo` | `yum-archive.fullstaqruby.org` | Frozen packages for EOL distributions | Archive repositories are static — CI never writes to them. They use the same versioned bucket structure as the main repos. Each migration creates a new version that merges newly-archived distros with the existing archive contents, so the archive grows incrementally over time. @@ -65,7 +65,7 @@ When a Ruby version reaches EOL, we stop building it (by removing it from `confi ~~~bash PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ -ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-archive-repo \ ./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb --dry-run ~~~ @@ -79,7 +79,7 @@ The script auto-detects EOL distros by comparing `aptly repo list` output agains ~~~bash PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ -ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-archive-repo \ ./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb ~~~ @@ -87,11 +87,11 @@ ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \ ~~~bash PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ -ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo-archive \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-archive-repo \ ./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb --dry-run PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ -ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo-archive \ +ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-archive-repo \ ./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb ~~~ @@ -178,15 +178,15 @@ echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-st **Revert the archive to a previous version:** ~~~bash -gsutil cat gs://fsruby-server-edition-apt-repo-archive/versions/latest_version.txt +gsutil cat gs://fsruby-server-edition-apt-archive-repo/versions/latest_version.txt -echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-repo-archive/versions/latest_version.txt +echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-archive-repo/versions/latest_version.txt ~~~ **Delete all archive contents** (if no users depend on it yet): ~~~bash -gsutil -m rm -r gs://fsruby-server-edition-apt-repo-archive/versions/ +gsutil -m rm -r gs://fsruby-server-edition-apt-archive-repo/versions/ ~~~ ## How the migration scripts work diff --git a/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb b/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb index 815ab1af..560dd1fa 100755 --- a/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb +++ b/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb @@ -7,7 +7,7 @@ # # Usage: # PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ -# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo-archive \ +# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-archive-repo \ # ./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb [--dry-run] [--distros centos-8,debian-9] # # If --distros is not specified, automatically detects EOL distros by comparing diff --git a/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb b/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb index e5b85cad..262ffe19 100755 --- a/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb +++ b/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb @@ -7,7 +7,7 @@ # # Usage: # PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ -# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo-archive \ +# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-archive-repo \ # ./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb [--dry-run] [--distros centos-8] # # If --distros is not specified, automatically detects EOL distros by comparing From add88a08b7edb6a786d313e315cde8a20ba66678 Mon Sep 17 00:00:00 2001 From: abtreece Date: Wed, 23 Sep 2026 21:29:40 -0500 Subject: [PATCH 04/12] docs: fix Caddy restart step after archive migration The /admin/restart_web_server endpoint only accepts deploy-environment OIDC tokens, so operators must restart Caddy over SSH. Make the runbook and the migration scripts' next-steps output say so explicitly. --- dev-handbook/archiving-eol-packages.md | 12 +++++++----- .../ci-cd/archive/migrate-apt-to-archive.rb | 2 +- .../ci-cd/archive/migrate-yum-to-archive.rb | 2 +- 3 files changed, 9 insertions(+), 7 deletions(-) diff --git a/dev-handbook/archiving-eol-packages.md b/dev-handbook/archiving-eol-packages.md index df1cb767..abd74333 100644 --- a/dev-handbook/archiving-eol-packages.md +++ b/dev-handbook/archiving-eol-packages.md @@ -97,14 +97,16 @@ ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-archive-repo \ ### Step 3: Restart the web server -After migration, restart the web server so Caddy picks up the new version numbers: +Caddy only reads repo version numbers at startup, and nothing restarts it after a manual migration. Until it restarts, archive requests keep redirecting to the old version (`versions/0/` on first migration) and return 404. + +The `/admin/restart_web_server` endpoint only accepts OIDC tokens from GitHub-hosted runners in this repo's `deploy` environment, so it can't be called by hand. Restart Caddy over SSH on the backend server instead: ~~~bash -curl -X POST https://apt.fullstaqruby.org/admin/restart_web_server \ - -H "Authorization: Bearer $ID_TOKEN" -~~~ +sudo systemctl restart caddy -Or restart the Caddy service directly via Ansible/SSH. +# Confirm the archive versions are no longer 0 +sudo cat /etc/caddy/env-repo-versions +~~~ ### Step 4: Verify diff --git a/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb b/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb index 560dd1fa..40b95b84 100755 --- a/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb +++ b/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb @@ -599,7 +599,7 @@ def print_summary(eol_distros, old_version) log_info " https://storage.googleapis.com/#{archive_bucket}/versions/#{new_archive_version}/public" log_info '' log_info 'Next steps:' - log_info ' 1. Restart the web server to pick up new archive version' + log_info ' 1. Restart Caddy on the backend server (required; archive stays 404 until then): sudo systemctl restart caddy' log_info ' 2. Verify archive repo: curl https://apt-archive.fullstaqruby.org/dists/' log_info ' 3. Verify main repo still works: apt-get update on a supported distro' end diff --git a/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb b/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb index 262ffe19..c01a06e0 100755 --- a/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb +++ b/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb @@ -336,7 +336,7 @@ def print_summary(eol_distros, old_version) log_info " https://storage.googleapis.com/#{archive_bucket}/versions/#{new_archive_version}/public" log_info '' log_info 'Next steps:' - log_info ' 1. Restart the web server to pick up new archive version' + log_info ' 1. Restart Caddy on the backend server (required; archive stays 404 until then): sudo systemctl restart caddy' log_info ' 2. Verify archive repo: curl https://yum-archive.fullstaqruby.org/' log_info ' 3. Verify main repo still works on a supported distro' end From 076182dc9112995e09532d7c875e7285ab180876 Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 21:36:35 -0500 Subject: [PATCH 05/12] fix: use symbolized config keys when reading Ruby minor versions load_config symbolizes every key, so config['ruby'] is nil and both prune scripts raised NoMethodError after acquiring the production lock. --- internal-scripts/ci-cd/archive/prune-apt-packages.rb | 2 +- internal-scripts/ci-cd/archive/prune-yum-packages.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/internal-scripts/ci-cd/archive/prune-apt-packages.rb b/internal-scripts/ci-cd/archive/prune-apt-packages.rb index db709eb6..eed086ac 100755 --- a/internal-scripts/ci-cd/archive/prune-apt-packages.rb +++ b/internal-scripts/ci-cd/archive/prune-apt-packages.rb @@ -221,7 +221,7 @@ def fetch_state(version) end def active_ruby_minor_versions - config['ruby']['minor_version_packages'].map { |p| p['minor_version'] } + config[:ruby][:minor_version_packages].map { |p| p[:minor_version] } end def list_aptly_repos diff --git a/internal-scripts/ci-cd/archive/prune-yum-packages.rb b/internal-scripts/ci-cd/archive/prune-yum-packages.rb index ab45fe2f..ed8520fb 100755 --- a/internal-scripts/ci-cd/archive/prune-yum-packages.rb +++ b/internal-scripts/ci-cd/archive/prune-yum-packages.rb @@ -174,7 +174,7 @@ def fetch_repo(version) end def active_ruby_minor_versions - config['ruby']['minor_version_packages'].map { |p| p['minor_version'] } + config[:ruby][:minor_version_packages].map { |p| p[:minor_version] } end def invoke_createrepo(dir) From c78afc48ce9935c8f62773087e813834104d64a8 Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 21:56:44 -0500 Subject: [PATCH 06/12] feat: add archive-apt-packages.rb Moves every package whose distribution or Ruby minor version is EOL from the live APT repo into the archive repo, in one pass under locks/apt. Imports packages individually via aptly repo add (the old migrate script copied a per-distro db directory that LevelDB never creates), and verifies the archive contains every moved package before touching live. --- .../ci-cd/archive/archive-apt-packages.rb | 614 ++++++++++++++++++ 1 file changed, 614 insertions(+) create mode 100755 internal-scripts/ci-cd/archive/archive-apt-packages.rb diff --git a/internal-scripts/ci-cd/archive/archive-apt-packages.rb b/internal-scripts/ci-cd/archive/archive-apt-packages.rb new file mode 100755 index 00000000..c6519520 --- /dev/null +++ b/internal-scripts/ci-cd/archive/archive-apt-packages.rb @@ -0,0 +1,614 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Moves packages that no longer belong in the live APT repository into the +# archive APT repository. A package leaves the live repository when its +# distribution, or its Ruby minor version, is no longer listed in config.yml. +# The live and archive repositories never contain the same package. +# +# Usage: +# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ +# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-archive-repo \ +# ./internal-scripts/ci-cd/archive/archive-apt-packages.rb [--dry-run] +# +# Safe to rerun: packages already in the archive are re-added as a no-op, +# and the live repository is only modified after the archive has been +# uploaded and activated. +# +# See dev-handbook/archiving-eol-packages.md. + +require_relative '../../../lib/gcloud_storage_lock' +require_relative '../../../lib/ci_workflow_support' +require_relative '../../../lib/shell_scripting_support' +require_relative '../../../lib/publishing_support' +require 'json' +require 'shellwords' +require 'tmpdir' +require 'fileutils' +require 'optparse' + +class ArchiveAptPackages + REPO_ORIGIN = 'Fullstaq-Ruby' + REPO_LABEL = 'Fullstaq-Ruby' + + # Matches: fullstaq-ruby-3.1, fullstaq-ruby-3.1-jemalloc, fullstaq-ruby-3.1.7, etc. + # Does NOT match: fullstaq-ruby-common, fullstaq-rbenv + RUBY_PACKAGE_PATTERN = /\Afullstaq-ruby-(\d+\.\d+)/ + + include CiWorkflowSupport + include ShellScriptingSupport + include PublishingSupport + + def main + parse_options + require_envvar 'PRODUCTION_REPO_BUCKET_NAME' + require_envvar 'ARCHIVE_REPO_BUCKET_NAME' + + print_header 'Initializing' + load_config + create_temp_dirs + ensure_gpg_state_isolated + activate_wrappers_bin_dir + initialize_locking + initialize_aptly(@live_aptly_config_path, @live_state_path) + initialize_aptly(@archive_aptly_config_path, @archive_state_path) + fetch_and_import_signing_key + + begin + synchronize do + print_header 'Fetching live repository state' + @live_version = get_latest_version(live_bucket) + abort 'ERROR: No production repository exists yet' if @live_version == 0 + fetch_state(live_bucket, @live_version, @live_state_path) + + print_header 'Determining packages to archive' + @moves = determine_moves + if @moves.empty? + log_notice 'Nothing to archive' + return + end + print_moves + + print_header 'Fetching archive repository state' + @archive_version = get_latest_version(archive_bucket) + if @archive_version > 0 + fetch_state(archive_bucket, @archive_version, @archive_state_path) + else + log_notice 'No archive exists yet, creating a new one' + end + + print_header 'Adding packages to archive repository' + add_to_archive + verify_archive + check_lock_health + + print_header 'Removing packages from live repository' + remove_from_live + verify_live + check_lock_health + + print_header 'Creating state archives' + create_state_tarball(@archive_state_path, @archive_state_tarball_path) + create_state_tarball(@live_state_path, @live_state_tarball_path) + + if @dry_run + log_notice 'Dry run: not uploading changes' + return + end + + # The archive must be complete and activated before the live + # repository stops serving these packages. + print_header 'Uploading archive repository' + upload_version(archive_bucket, @archive_version, @archive_state_path, @archive_state_tarball_path) + check_lock_health + + print_header 'Uploading live repository' + upload_version(live_bucket, @live_version, @live_state_path, @live_state_tarball_path) + end + + if @moves && !@moves.empty? && !@dry_run + print_header 'Success!' + print_summary + end + ensure + cleanup + end + end + +private + def parse_options + @dry_run = false + OptionParser.new do |opts| + opts.banner = "Usage: #{$0} [options]" + opts.on('--dry-run', 'Make all changes locally, but do not upload them') { @dry_run = true } + end.parse! + end + + def live_bucket + ENV['PRODUCTION_REPO_BUCKET_NAME'] + end + + def archive_bucket + ENV['ARCHIVE_REPO_BUCKET_NAME'] + end + + def create_temp_dirs + log_notice 'Creating temporary directories' + @temp_dir = Dir.mktmpdir('archive-apt-packages') + @wrappers_bin_dir = "#{@temp_dir}/wrappers" + @signing_key_path = "#{@temp_dir}/key.gpg" + + @live_aptly_config_path = "#{@temp_dir}/aptly-live.conf" + @live_state_path = "#{@temp_dir}/live-state" + @live_state_tarball_path = "#{@temp_dir}/live-state.tar.zst" + + @archive_aptly_config_path = "#{@temp_dir}/aptly-archive.conf" + @archive_state_path = "#{@temp_dir}/archive-state" + @archive_state_tarball_path = "#{@temp_dir}/archive-state.tar.zst" + + Dir.mkdir(@wrappers_bin_dir) + [@live_state_path, @archive_state_path].each do |state_path| + FileUtils.mkdir_p("#{state_path}/db") + FileUtils.mkdir_p("#{state_path}/repo") + end + end + + def ensure_gpg_state_isolated + # Aptly invokes 'gpg' from PATH. We make that use an isolated + # home directory, so that only the signing key is available. + log_notice 'Creating GPG wrapper' + File.open("#{@wrappers_bin_dir}/gpg", 'w:utf-8') do |f| + f.write("#!/bin/sh\n") + f.write( + sprintf( + "exec %s --homedir %s \"$@\"\n", + Shellwords.escape(find_gpg), + Shellwords.escape(@temp_dir) + ) + ) + end + File.chmod(0755, "#{@wrappers_bin_dir}/gpg") + end + + def find_gpg + ENV['PATH'].split(':').each do |dir| + next if dir == @wrappers_bin_dir + candidate = "#{dir}/gpg" + return candidate if File.exist?(candidate) + end + abort('GPG not found') + end + + def activate_wrappers_bin_dir + ENV['PATH'] = "#{@wrappers_bin_dir}:#{ENV['PATH']}" + end + + def initialize_locking + # The archive bucket has no lock of its own. It is only ever written + # while holding the live repository's lock. + @lock = GCloudStorageLock.new(url: "gs://#{live_bucket}/locks/apt") + end + + def synchronize(&block) + @lock.synchronize(&block) + end + + def check_lock_health + abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? + end + + def initialize_aptly(config_path, state_path) + log_notice "Creating Aptly config file #{config_path}" + File.open(config_path, 'w:utf-8') do |f| + f.write(JSON.generate( + rootDir: state_path, + FileSystemPublishEndpoints: { + main: { + rootDir: "#{state_path}/repo", + linkMethod: 'symlink', + verifyMethod: 'md5' + } + } + )) + end + end + + def fetch_and_import_signing_key + log_notice 'Fetching and importing signing key' + File.open(@signing_key_path, 'wb') do |f| + f.write(fetch_signing_key) + end + @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) + log_info "Signing key ID: #{@gpg_key_id}" + import_gpg_key(@temp_dir, @signing_key_path) + end + + def get_latest_version(bucket) + url = "gs://#{bucket}/versions/latest_version.txt" + stdout_output, stderr_output, status = run_command_capture_output( + 'gsutil', 'cp', url, '-', + log_invocation: false, + check_error: false + ) + if status.success? + version = stdout_output.strip + abort("ERROR: invalid version number stored in #{url}") if version !~ /\A[0-9]+\Z/ + version = version.to_i + elsif stderr_output =~ /No URLs matched/ + version = 0 + else + abort("ERROR: error fetching #{url}: #{stderr_output.chomp}") + end + log_notice "Latest version of #{bucket}: #{version}" + version + end + + def fetch_state(bucket, version, state_path) + log_notice "Fetching state of #{bucket} version #{version}" + run_bash( + sprintf('gsutil -m cp %s - | zstd -dc | tar -xC %s', + Shellwords.escape("gs://#{bucket}/versions/#{version}/state.tar.zst"), + Shellwords.escape(state_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def supported_distros + distributions.find_all { |d| d[:package_format] == :DEB }.map { |d| d[:name] } + end + + def active_ruby_minor_versions + config[:ruby][:minor_version_packages].map { |p| p[:minor_version] } + end + + def eol_ruby_package?(package_key, active_minors) + package_key =~ RUBY_PACKAGE_PATTERN && !active_minors.include?($1) + end + + # Returns a hash: distro => { whole_distro: Boolean, packages: [package keys] } + def determine_moves + supported = supported_distros + active_minors = active_ruby_minor_versions + log_info "Supported DEB distributions: #{supported.join(', ')}" + log_info "Active Ruby minor versions: #{active_minors.join(', ')}" + + moves = {} + list_aptly_repos(@live_aptly_config_path).each do |distro| + packages = list_aptly_packages(@live_aptly_config_path, distro) + if !supported.include?(distro) + moves[distro] = { whole_distro: true, packages: packages } + else + eol_packages = packages.find_all { |key| eol_ruby_package?(key, active_minors) } + moves[distro] = { whole_distro: false, packages: eol_packages } if !eol_packages.empty? + end + end + moves + end + + def print_moves + @moves.each_pair do |distro, move| + reason = move[:whole_distro] ? 'EOL distribution' : 'EOL Ruby versions' + log_notice "[#{distro}] #{move[:packages].size} packages to archive (#{reason})" + move[:packages].each { |key| log_info " #{YELLOW}ARCHIVE#{RESET} #{key}" } + end + end + + def add_to_archive + existing_repos = list_aptly_repos(@archive_aptly_config_path) + + @moves.each_pair do |distro, move| + next if move[:packages].empty? + + log_notice "[#{distro}] Adding #{move[:packages].size} packages to archive" + if !existing_repos.include?(distro) + run_command( + 'aptly', 'repo', 'create', + "-config=#{@archive_aptly_config_path}", + distro, + log_invocation: true, + check_error: true + ) + end + + move[:packages].each_slice(50) do |batch| + # -force-replace makes re-adding a package that is already in + # the archive a no-op, which is what makes reruns safe. + run_command( + 'aptly', 'repo', 'add', '-force-replace', + "-config=#{@archive_aptly_config_path}", + distro, + *live_pool_paths(batch), + log_invocation: false, + check_error: true + ) + end + + publish_aptly_repo(@archive_aptly_config_path, distro) + end + + compact_aptly_db(@archive_aptly_config_path) + end + + # Paths of the .deb files in the live state's pool for the given package keys. + def live_pool_paths(package_keys) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'package', 'show', + "-config=#{@live_aptly_config_path}", + '-with-files', + package_keys.join(' | '), + log_invocation: false, + check_error: true + ) + paths = stdout_output.scan(/^Files in the pool:\n((?: .+\n?)+)/).flat_map do |(block)| + block.split("\n").map(&:strip) + end + if paths.size != package_keys.size || !paths.all? { |path| File.exist?(path) } + abort("ERROR: expected #{package_keys.size} pool files, found #{paths.size}: #{package_keys.join(', ')}") + end + paths + end + + def verify_archive + log_notice 'Verifying archive repository' + @moves.each_pair do |distro, move| + next if move[:packages].empty? + + archived = list_aptly_packages(@archive_aptly_config_path, distro) + missing = move[:packages] - archived + if !missing.empty? + abort("ERROR: [#{distro}] packages missing from archive: #{missing.join(', ')}") + end + + published = published_packages("#{@archive_state_path}/repo", distro) + missing = move[:packages] - published + if !missing.empty? + abort("ERROR: [#{distro}] packages missing from published archive: #{missing.join(', ')}") + end + log_info "[#{distro}] All #{move[:packages].size} packages present in archive" + end + end + + def remove_from_live + @moves.each_pair do |distro, move| + if move[:whole_distro] + log_notice "[#{distro}] Dropping distribution from live repository" + drop_publication(@live_aptly_config_path, distro) + run_command( + 'aptly', 'repo', 'drop', + "-config=#{@live_aptly_config_path}", + distro, + log_invocation: true, + check_error: true + ) + else + log_notice "[#{distro}] Removing #{move[:packages].size} packages from live repository" + move[:packages].each_slice(50) do |batch| + run_command( + 'aptly', 'repo', 'remove', + "-config=#{@live_aptly_config_path}", + distro, + batch.join(' | '), + log_invocation: false, + check_error: true + ) + end + publish_aptly_repo(@live_aptly_config_path, distro) + end + end + + # Must run after republishing: published files keep pool files referenced. + compact_aptly_db(@live_aptly_config_path) + end + + def verify_live + log_notice 'Verifying live repository' + remaining_repos = list_aptly_repos(@live_aptly_config_path) + @moves.each_pair do |distro, move| + if move[:whole_distro] + abort("ERROR: [#{distro}] still present in live repository") if remaining_repos.include?(distro) + if File.exist?("#{@live_state_path}/repo/dists/#{distro}") + abort("ERROR: [#{distro}] still published in live repository") + end + else + leftover = move[:packages] & list_aptly_packages(@live_aptly_config_path, distro) + leftover |= move[:packages] & published_packages("#{@live_state_path}/repo", distro) + if !leftover.empty? + abort("ERROR: [#{distro}] packages still in live repository: #{leftover.join(', ')}") + end + end + end + end + + # Package keys (name_version_arch) listed in a published distribution's + # Packages indexes. This is what APT clients actually see. + def published_packages(repo_path, distro) + Dir["#{repo_path}/dists/#{distro}/*/binary-*/Packages"].flat_map do |path| + File.read(path).split("\n\n").map do |stanza| + fields = stanza.scan(/^(Package|Version|Architecture): (.+)$/).to_h + next if fields.empty? + "#{fields['Package']}_#{fields['Version']}_#{fields['Architecture']}" + end.compact + end.uniq + end + + def list_aptly_repos(config_path) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'repo', 'list', + "-config=#{config_path}", + '-raw', + log_invocation: false, + check_error: true + ) + stdout_output.split("\n").map(&:strip).reject(&:empty?) + end + + def list_aptly_packages(config_path, distro) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'repo', 'show', + "-config=#{config_path}", + '-with-packages', + distro, + log_invocation: false, + check_error: true + ) + stdout_output.sub(/.*^Packages:$/m, '').split("\n").map(&:strip).reject(&:empty?) + end + + def published?(config_path, distro) + stdout_output, _, _ = run_command_capture_output( + 'aptly', 'publish', 'list', + "-config=#{config_path}", + '-raw', + log_invocation: false, + check_error: true + ) + stdout_output.split("\n").any? { |line| line.split(' ') == ['filesystem:main:.', distro] } + end + + def drop_publication(config_path, distro) + return if !published?(config_path, distro) + run_command( + 'aptly', 'publish', 'drop', + "-config=#{config_path}", + distro, 'filesystem:main:.', + log_invocation: true, + check_error: true + ) + end + + def publish_aptly_repo(config_path, distro) + # 'aptly publish repo' refuses to overwrite an existing publication, + # and 'publish update' doesn't add architectures that weren't + # published before. So we drop and recreate the publication. + drop_publication(config_path, distro) + + publish_command = [ + 'aptly', 'publish', 'repo', + '-batch', '-force-overwrite', + "-config=#{config_path}", + "-gpg-key=#{@gpg_key_id}", + "-distribution=#{distro}", + "-origin=#{REPO_ORIGIN}", + "-label=#{REPO_LABEL}", + ] + _, stderr_output, status = run_command_capture_output( + *publish_command, distro, 'filesystem:main:.', + log_invocation: true, + check_error: false + ) + return if status.success? + + if stderr_output =~ /unable to figure out list of architectures/ + run_command( + *publish_command, '-architectures=all', distro, 'filesystem:main:.', + log_invocation: true, + check_error: true + ) + else + abort("ERROR publishing #{distro}: #{stderr_output.chomp}") + end + end + + def compact_aptly_db(config_path) + log_notice "Compacting #{config_path}" + run_command( + 'aptly', 'db', 'cleanup', + "-config=#{config_path}", + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def create_state_tarball(state_path, tarball_path) + log_notice "Creating #{tarball_path}" + run_bash( + sprintf("tar -C %s -cf - . | zstd -T0 > %s", + Shellwords.escape(state_path), + Shellwords.escape(tarball_path)), + pipefail: true, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def upload_version(bucket, old_version, state_path, tarball_path) + new_version = old_version + 1 + version_url = "gs://#{bucket}/versions/#{new_version}" + log_notice "Uploading #{bucket} version #{new_version}" + + run_command( + 'gsutil', '-h', 'Cache-Control:public', + 'cp', tarball_path, "#{version_url}/state.tar.zst", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + if old_version > 0 + # Server-side copy first, so that the upload below only transfers changes. + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "gs://#{bucket}/versions/#{old_version}/public", + "#{version_url}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "#{state_path}/repo", + "#{version_url}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + write_gcs_text("#{version_url}/version.txt", new_version, 'public') + log_notice "Activating #{bucket} version #{new_version}" + write_gcs_text("gs://#{bucket}/versions/latest_version.txt", new_version, 'no-store') + end + + def write_gcs_text(url, content, cache_control) + run_bash( + sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:%s cp - %s <<<%s', + cache_control, + Shellwords.escape(url), + Shellwords.escape(content.to_s)), + log_invocation: true, + check_error: true, + pipefail: false + ) + end + + def print_summary + log_notice 'Summary' + @moves.each_pair do |distro, move| + suffix = move[:whole_distro] ? ' (distribution removed from live)' : '' + log_info "#{distro}: archived #{move[:packages].size} packages#{suffix}" + end + log_info "Live repository: version #{@live_version} -> #{@live_version + 1}" + log_info "Archive repository: version #{@archive_version} -> #{@archive_version + 1}" + log_info '' + log_info 'Next steps:' + log_info ' 1. Restart Caddy on the backend server: sudo systemctl restart caddy' + log_info ' Until then, apt.fullstaqruby.org and apt-archive.fullstaqruby.org serve the old versions.' + log_info ' 2. Verify: curl -fsS https://apt-archive.fullstaqruby.org/dists//Release' + end + + def cleanup + return if @temp_dir.nil? + log_info "Cleaning up #{@temp_dir}" + FileUtils.remove_entry_secure(@temp_dir) + end +end + +ArchiveAptPackages.new.main From 18b584ac2e3b0ac1f427698120073b877d0a51af Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 21:56:46 -0500 Subject: [PATCH 07/12] feat: add archive-yum-packages.rb and archive script test harness Copies RPMs individually into the archive, regenerates and signs repodata on both sides, and verifies the archive metadata before touching live. The harness in internal-scripts/ci-cd/archive/test runs both scripts against a fake GCS inside a container with real Aptly, createrepo_c and GPG. Not wired into CI. --- .../ci-cd/archive/archive-yum-packages.rb | 416 ++++++++++++++++++ .../ci-cd/archive/test/Dockerfile | 16 + .../ci-cd/archive/test/fake-bin/az | 4 + .../ci-cd/archive/test/fake-bin/docker | 39 ++ .../ci-cd/archive/test/fake-bin/gsutil | 79 ++++ .../ci-cd/archive/test/run-tests.sh | 339 ++++++++++++++ 6 files changed, 893 insertions(+) create mode 100755 internal-scripts/ci-cd/archive/archive-yum-packages.rb create mode 100644 internal-scripts/ci-cd/archive/test/Dockerfile create mode 100755 internal-scripts/ci-cd/archive/test/fake-bin/az create mode 100755 internal-scripts/ci-cd/archive/test/fake-bin/docker create mode 100755 internal-scripts/ci-cd/archive/test/fake-bin/gsutil create mode 100755 internal-scripts/ci-cd/archive/test/run-tests.sh diff --git a/internal-scripts/ci-cd/archive/archive-yum-packages.rb b/internal-scripts/ci-cd/archive/archive-yum-packages.rb new file mode 100755 index 00000000..a1a8913b --- /dev/null +++ b/internal-scripts/ci-cd/archive/archive-yum-packages.rb @@ -0,0 +1,416 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Moves packages that no longer belong in the live YUM repository into the +# archive YUM repository. A package leaves the live repository when its +# distribution, or its Ruby minor version, is no longer listed in config.yml. +# The live and archive repositories never contain the same package. +# +# Usage: +# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ +# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-archive-repo \ +# ./internal-scripts/ci-cd/archive/archive-yum-packages.rb [--dry-run] +# +# Safe to rerun: RPMs already in the archive are skipped, and the live +# repository is only modified after the archive has been uploaded and +# activated. +# +# See dev-handbook/archiving-eol-packages.md. + +require_relative '../../../lib/gcloud_storage_lock' +require_relative '../../../lib/ci_workflow_support' +require_relative '../../../lib/general_support' +require_relative '../../../lib/shell_scripting_support' +require_relative '../../../lib/publishing_support' +require 'shellwords' +require 'tmpdir' +require 'fileutils' +require 'optparse' +require 'zlib' + +class ArchiveYumPackages + # Matches: fullstaq-ruby-3.1-rev5-el9.x86_64.rpm, fullstaq-ruby-3.1-jemalloc-..., fullstaq-ruby-3.1.7-... + # Does NOT match: fullstaq-ruby-common-*, fullstaq-rbenv-* + RUBY_RPM_PATTERN = /\Afullstaq-ruby-(\d+\.\d+)/ + + include CiWorkflowSupport + include GeneralSupport + include ShellScriptingSupport + include PublishingSupport + + def main + parse_options + require_envvar 'PRODUCTION_REPO_BUCKET_NAME' + require_envvar 'ARCHIVE_REPO_BUCKET_NAME' + + print_header 'Initializing' + load_config + create_temp_dirs + pull_utility_image_if_not_exists + initialize_locking + fetch_and_import_signing_key + + begin + synchronize do + print_header 'Fetching live repository' + @live_version = get_latest_version(live_bucket) + abort 'ERROR: No production repository exists yet' if @live_version == 0 + fetch_repo(live_bucket, @live_version, @live_repo_path) + + print_header 'Determining packages to archive' + @moves = determine_moves + if @moves.empty? + log_notice 'Nothing to archive' + return + end + print_moves + + print_header 'Fetching archive repository' + @archive_version = get_latest_version(archive_bucket) + if @archive_version > 0 + fetch_repo(archive_bucket, @archive_version, @archive_repo_path) + else + log_notice 'No archive exists yet, creating a new one' + end + + print_header 'Adding packages to archive repository' + add_to_archive + verify_archive + check_lock_health + + print_header 'Removing packages from live repository' + remove_from_live + verify_live + check_lock_health + + if @dry_run + log_notice 'Dry run: not uploading changes' + return + end + + # The archive must be complete and activated before the live + # repository stops serving these packages. + print_header 'Uploading archive repository' + upload_version(archive_bucket, @archive_version, @archive_repo_path) + check_lock_health + + print_header 'Uploading live repository' + upload_version(live_bucket, @live_version, @live_repo_path) + end + + if @moves && !@moves.empty? && !@dry_run + print_header 'Success!' + print_summary + end + ensure + cleanup + end + end + +private + def parse_options + @dry_run = false + OptionParser.new do |opts| + opts.banner = "Usage: #{$0} [options]" + opts.on('--dry-run', 'Make all changes locally, but do not upload them') { @dry_run = true } + end.parse! + end + + def live_bucket + ENV['PRODUCTION_REPO_BUCKET_NAME'] + end + + def archive_bucket + ENV['ARCHIVE_REPO_BUCKET_NAME'] + end + + def create_temp_dirs + log_notice 'Creating temporary directories' + @temp_dir = Dir.mktmpdir('archive-yum-packages') + @signing_key_path = "#{@temp_dir}/key.gpg" + @live_repo_path = "#{@temp_dir}/live-repo" + @archive_repo_path = "#{@temp_dir}/archive-repo" + Dir.mkdir(@live_repo_path) + Dir.mkdir(@archive_repo_path) + end + + def initialize_locking + # The archive bucket has no lock of its own. It is only ever written + # while holding the live repository's lock. + @lock = GCloudStorageLock.new(url: "gs://#{live_bucket}/locks/yum") + end + + def synchronize(&block) + @lock.synchronize(&block) + end + + def check_lock_health + abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? + end + + def fetch_and_import_signing_key + log_notice 'Fetching and importing signing key' + File.open(@signing_key_path, 'wb') do |f| + f.write(fetch_signing_key) + end + @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) + log_info "Signing key ID: #{@gpg_key_id}" + import_gpg_key(@temp_dir, @signing_key_path) + end + + def get_latest_version(bucket) + url = "gs://#{bucket}/versions/latest_version.txt" + stdout_output, stderr_output, status = run_command_capture_output( + 'gsutil', 'cp', url, '-', + log_invocation: false, + check_error: false + ) + if status.success? + version = stdout_output.strip + abort("ERROR: invalid version number stored in #{url}") if version !~ /\A[0-9]+\Z/ + version = version.to_i + elsif stderr_output =~ /No URLs matched/ + version = 0 + else + abort("ERROR: error fetching #{url}: #{stderr_output.chomp}") + end + log_notice "Latest version of #{bucket}: #{version}" + version + end + + def fetch_repo(bucket, version, repo_path) + log_notice "Fetching #{bucket} version #{version}" + run_command( + 'gsutil', '-m', 'rsync', '-r', + "gs://#{bucket}/versions/#{version}/public", + repo_path, + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + def supported_distros + distributions.find_all { |d| d[:package_format] == :RPM }.map { |d| d[:name] } + end + + def active_ruby_minor_versions + config[:ruby][:minor_version_packages].map { |p| p[:minor_version] } + end + + def eol_ruby_rpm?(filename, active_minors) + filename =~ RUBY_RPM_PATTERN && !active_minors.include?($1) + end + + def distro_dirs(repo_path) + Dir["#{repo_path}/*"].find_all { |path| File.directory?(path) }.map { |path| File.basename(path) }.sort + end + + def arch_dirs(repo_path, distro) + Dir["#{repo_path}/#{distro}/*"].find_all { |path| File.directory?(path) }.map { |path| File.basename(path) }.sort + end + + # Returns a hash: distro => { whole_distro: Boolean, rpms: { arch => [filenames] } } + def determine_moves + supported = supported_distros + active_minors = active_ruby_minor_versions + log_info "Supported RPM distributions: #{supported.join(', ')}" + log_info "Active Ruby minor versions: #{active_minors.join(', ')}" + + moves = {} + distro_dirs(@live_repo_path).each do |distro| + whole_distro = !supported.include?(distro) + rpms = {} + arch_dirs(@live_repo_path, distro).each do |arch| + filenames = Dir["#{@live_repo_path}/#{distro}/#{arch}/*.rpm"].map { |path| File.basename(path) }.sort + filenames = filenames.find_all { |filename| eol_ruby_rpm?(filename, active_minors) } if !whole_distro + rpms[arch] = filenames if !filenames.empty? + end + moves[distro] = { whole_distro: whole_distro, rpms: rpms } if whole_distro || !rpms.empty? + end + moves + end + + def print_moves + @moves.each_pair do |distro, move| + reason = move[:whole_distro] ? 'EOL distribution' : 'EOL Ruby versions' + count = move[:rpms].values.sum(&:size) + log_notice "[#{distro}] #{count} packages to archive (#{reason})" + move[:rpms].each_pair do |arch, filenames| + filenames.each { |filename| log_info " #{YELLOW}ARCHIVE#{RESET} #{arch}/#{filename}" } + end + end + end + + def add_to_archive + @moves.each_pair do |distro, move| + move[:rpms].each_pair do |arch, filenames| + target_dir = "#{@archive_repo_path}/#{distro}/#{arch}" + FileUtils.mkdir_p(target_dir) + copied = 0 + filenames.each do |filename| + target_path = "#{target_dir}/#{filename}" + next if File.exist?(target_path) + hardlink_or_copy_file("#{@live_repo_path}/#{distro}/#{arch}/#{filename}", target_path) + copied += 1 + end + log_notice "[#{distro}/#{arch}] Copied #{copied} packages to archive (#{filenames.size - copied} already present)" + regenerate_repo_metadata(target_dir) + end + end + end + + def verify_archive + log_notice 'Verifying archive repository' + @moves.each_pair do |distro, move| + move[:rpms].each_pair do |arch, filenames| + dir = "#{@archive_repo_path}/#{distro}/#{arch}" + missing = filenames - indexed_rpms(dir) + if !missing.empty? + abort("ERROR: [#{distro}/#{arch}] packages missing from archive metadata: #{missing.join(', ')}") + end + log_info "[#{distro}/#{arch}] All #{filenames.size} packages present in archive" + end + end + end + + def remove_from_live + @moves.each_pair do |distro, move| + if move[:whole_distro] + log_notice "[#{distro}] Removing distribution from live repository" + FileUtils.rm_rf("#{@live_repo_path}/#{distro}") + else + move[:rpms].each_pair do |arch, filenames| + dir = "#{@live_repo_path}/#{distro}/#{arch}" + log_notice "[#{distro}/#{arch}] Removing #{filenames.size} packages from live repository" + filenames.each { |filename| File.delete("#{dir}/#{filename}") } + regenerate_repo_metadata(dir) + end + end + end + end + + def verify_live + log_notice 'Verifying live repository' + @moves.each_pair do |distro, move| + if move[:whole_distro] + abort("ERROR: [#{distro}] still present in live repository") if File.exist?("#{@live_repo_path}/#{distro}") + else + move[:rpms].each_pair do |arch, filenames| + dir = "#{@live_repo_path}/#{distro}/#{arch}" + leftover = filenames & (indexed_rpms(dir) + Dir["#{dir}/*.rpm"].map { |path| File.basename(path) }) + if !leftover.empty? + abort("ERROR: [#{distro}/#{arch}] packages still in live repository: #{leftover.join(', ')}") + end + end + end + end + end + + # RPM filenames listed in a directory's repodata. This is what YUM clients actually see. + def indexed_rpms(dir) + primary = Dir["#{dir}/repodata/*-primary.xml.gz"] + return [] if primary.empty? + abort("ERROR: multiple primary.xml.gz files in #{dir}/repodata") if primary.size > 1 + xml = Zlib::GzipReader.open(primary[0], &:read) + xml.scan(/ 0 + # Server-side copy first, so that the upload below only transfers changes. + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + "gs://#{bucket}/versions/#{old_version}/public", + "#{version_url}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + end + + run_command( + 'gsutil', '-m', '-h', 'Cache-Control:public', + 'rsync', '-r', '-d', + repo_path, + "#{version_url}/public", + log_invocation: true, + check_error: true, + passthru_output: true + ) + + write_gcs_text("#{version_url}/version.txt", new_version, 'public') + log_notice "Activating #{bucket} version #{new_version}" + write_gcs_text("gs://#{bucket}/versions/latest_version.txt", new_version, 'no-store') + end + + def write_gcs_text(url, content, cache_control) + run_bash( + sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:%s cp - %s <<<%s', + cache_control, + Shellwords.escape(url), + Shellwords.escape(content.to_s)), + log_invocation: true, + check_error: true, + pipefail: false + ) + end + + def print_summary + log_notice 'Summary' + @moves.each_pair do |distro, move| + suffix = move[:whole_distro] ? ' (distribution removed from live)' : '' + log_info "#{distro}: archived #{move[:rpms].values.sum(&:size)} packages#{suffix}" + end + log_info "Live repository: version #{@live_version} -> #{@live_version + 1}" + log_info "Archive repository: version #{@archive_version} -> #{@archive_version + 1}" + log_info '' + log_info 'Next steps:' + log_info ' 1. Restart Caddy on the backend server: sudo systemctl restart caddy' + log_info ' Until then, yum.fullstaqruby.org and yum-archive.fullstaqruby.org serve the old versions.' + log_info ' 2. Verify: curl -fsS https://yum-archive.fullstaqruby.org///repodata/repomd.xml' + end + + def cleanup + return if @temp_dir.nil? + log_info "Cleaning up #{@temp_dir}" + FileUtils.remove_entry_secure(@temp_dir) + end +end + +ArchiveYumPackages.new.main diff --git a/internal-scripts/ci-cd/archive/test/Dockerfile b/internal-scripts/ci-cd/archive/test/Dockerfile new file mode 100644 index 00000000..9408e092 --- /dev/null +++ b/internal-scripts/ci-cd/archive/test/Dockerfile @@ -0,0 +1,16 @@ +# Test environment for the archive scripts. Mirrors the CI publish job: +# Aptly comes from the Aptly repo (like install-aptly.sh), createrepo_c +# and GPG from Ubuntu. gsutil, az and docker are replaced by the fakes +# in fake-bin/. +FROM ubuntu:22.04 + +RUN set -x && \ + apt update && \ + apt install -y ca-certificates curl gpg ruby createrepo-c rpm \ + bzip2 xz-utils zstd && \ + echo 'deb http://repo.aptly.info/ squeeze main' > /etc/apt/sources.list.d/aptly.list && \ + curl -fsSL https://www.aptly.info/pubkey.txt | gpg --dearmor -o /etc/apt/trusted.gpg.d/aptly.gpg && \ + apt update && \ + apt install -y aptly && \ + apt clean && \ + rm -rf /var/lib/apt/lists/* diff --git a/internal-scripts/ci-cd/archive/test/fake-bin/az b/internal-scripts/ci-cd/archive/test/fake-bin/az new file mode 100755 index 00000000..70280db7 --- /dev/null +++ b/internal-scripts/ci-cd/archive/test/fake-bin/az @@ -0,0 +1,4 @@ +#!/bin/sh +# Fake 'az keyvault secret show' for tests: prints the test signing key. +set -eu +cat "$FAKE_SIGNING_KEY_PATH" diff --git a/internal-scripts/ci-cd/archive/test/fake-bin/docker b/internal-scripts/ci-cd/archive/test/fake-bin/docker new file mode 100755 index 00000000..af587b11 --- /dev/null +++ b/internal-scripts/ci-cd/archive/test/fake-bin/docker @@ -0,0 +1,39 @@ +#!/bin/sh +# Fake docker for tests. 'docker inspect' succeeds, so the utility image +# is never pulled. 'docker run ... -v DIR:/input... IMAGE createrepo_c ARGS' +# runs createrepo_c locally against DIR. +set -eu + +case "$1" in + inspect) + exit 0 + ;; + run) + shift + input_dir= + while [ $# -gt 0 ]; do + case "$1" in + --rm) shift ;; + --user) shift 2 ;; + -v) input_dir=${2%%:*}; shift 2 ;; + createrepo_c) break ;; + *) shift ;; + esac + done + shift + args= + for arg in "$@"; do + if [ "$arg" = /input ]; then + args="$args $input_dir" + else + args="$args $arg" + fi + done + # shellcheck disable=SC2086 + exec createrepo_c $args + ;; + *) + echo "fake docker: unsupported command: $*" >&2 + exit 2 + ;; +esac diff --git a/internal-scripts/ci-cd/archive/test/fake-bin/gsutil b/internal-scripts/ci-cd/archive/test/fake-bin/gsutil new file mode 100755 index 00000000..8b8d590e --- /dev/null +++ b/internal-scripts/ci-cd/archive/test/fake-bin/gsutil @@ -0,0 +1,79 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Minimal gsutil replacement for tests. Maps gs://bucket/path to +# $FAKE_GCS_ROOT/bucket/path. Supports only the invocations used by the +# archive scripts and GCloudStorageLock. + +require 'fileutils' +require 'time' + +ROOT = ENV.fetch('FAKE_GCS_ROOT') + +def local_path(url) + raise "not a gs:// URL: #{url}" if url !~ %r{\Ags://(.+)\z} + File.join(ROOT, $1) +end + +def gs?(arg) + arg.start_with?('gs://') +end + +def no_match(url) + warn "CommandException: No URLs matched: #{url}" + exit 1 +end + +args = ARGV.dup +# Drop global options: -m, -q, and -h
. +headers = [] +while (arg = args.first) && arg.start_with?('-') + args.shift + headers << args.shift if arg == '-h' +end + +command = args.shift +case command +when 'cp' + src, dst = args + if src == '-' + path = local_path(dst) + precondition = headers.find { |h| h.start_with?('x-goog-if-generation-match:') } + if precondition && precondition.split(':', 2)[1] == '0' && File.exist?(path) + warn 'PreconditionException: 412 Precondition Failed' + exit 1 + end + FileUtils.mkdir_p(File.dirname(path)) + File.binwrite(path, $stdin.read) + elsif dst == '-' + path = local_path(src) + no_match(src) if !File.file?(path) + $stdout.write(File.binread(path)) + else + path = local_path(dst) + FileUtils.mkdir_p(File.dirname(path)) + FileUtils.cp(src, path) + end +when 'rsync' + delete = args.delete('-d') + args.delete('-r') + src, dst = args.map { |arg| gs?(arg) ? local_path(arg) : arg } + FileUtils.mkdir_p(dst) + FileUtils.rm_rf(Dir.glob("#{dst}/*", File::FNM_DOTMATCH).reject { |p| p.end_with?('/.', '/..') }) if delete + # cp -a would copy symlinks as symlinks; real gsutil uploads their targets. + system('cp', '-rL', "#{src}/.", dst, exception: true) if File.exist?(src) +when 'stat' + path = local_path(args[0]) + no_match(args[0]) if !File.exist?(path) + stat = File.stat(path) + puts args[0] + puts " Update time: #{stat.mtime.utc.httpdate}" + puts " Generation: #{stat.mtime.to_f.to_s.delete('.')}#{stat.ino}" +when 'rm' + path = local_path(args[0]) + no_match(args[0]) if !File.exist?(path) + File.delete(path) +else + warn "fake gsutil: unsupported command: #{ARGV.join(' ')}" + exit 2 +end diff --git a/internal-scripts/ci-cd/archive/test/run-tests.sh b/internal-scripts/ci-cd/archive/test/run-tests.sh new file mode 100755 index 00000000..d0863095 --- /dev/null +++ b/internal-scripts/ci-cd/archive/test/run-tests.sh @@ -0,0 +1,339 @@ +#!/bin/bash +# Tests archive-apt-packages.rb and archive-yum-packages.rb against a +# local fake GCS, inside a container with the real Aptly, createrepo_c +# and GPG. Not run by CI. +# +# Usage: ./internal-scripts/ci-cd/archive/test/run-tests.sh +set -euo pipefail + +SELFDIR=$(dirname "$0") +SELFDIR=$(cd "$SELFDIR" && pwd) +ROOTDIR=$(cd "$SELFDIR/../../../.." && pwd) +IMAGE=fsruby-archive-test + +if [[ "${INSIDE_ARCHIVE_TEST_CONTAINER:-}" != true ]]; then + docker build --platform linux/amd64 -q -t "$IMAGE" "$SELFDIR" >/dev/null + exec docker run --rm --platform linux/amd64 \ + -e INSIDE_ARCHIVE_TEST_CONTAINER=true \ + -v "$ROOTDIR:/src:ro" \ + "$IMAGE" \ + bash /src/internal-scripts/ci-cd/archive/test/run-tests.sh +fi + +# ---- Inside the container ---- + +WORK=$(mktemp -d) +export FAKE_GCS_ROOT="$WORK/gcs" +export FAKE_SIGNING_KEY_PATH="$WORK/signing-key.asc" +export PATH="/src/internal-scripts/ci-cd/archive/test/fake-bin:$PATH" +export GNUPGHOME="$WORK/gnupg" +LIVE_APT=fsruby-server-edition-apt-repo +ARCHIVE_APT=fsruby-server-edition-apt-archive-repo +LIVE_YUM=fsruby-server-edition-yum-repo +ARCHIVE_YUM=fsruby-server-edition-yum-archive-repo + +FAILURES=0 + +function header() { + echo + echo "===== $* =====" +} + +function check() { + local description="$1" + shift + if "$@"; then + echo "PASS: $description" + else + echo "FAIL: $description" + FAILURES=$((FAILURES + 1)) + fi +} + +function gcs_version() { + cat "$FAKE_GCS_ROOT/$1/versions/latest_version.txt" +} + +# Package names (Package_Version_Architecture) in a published APT distribution. +function apt_packages() { + local bucket="$1" distro="$2" + local version + version=$(gcs_version "$bucket") + local file="$FAKE_GCS_ROOT/$bucket/versions/$version/public/dists/$distro/main/binary-amd64/Packages" + [[ -f "$file" ]] || return 0 + # Field order within a stanza is not fixed, so print at the end of each stanza. + awk '/^Package:/{p=$2} /^Version:/{v=$2} /^Architecture:/{a=$2} /^$/{if(p)print p"_"v"_"a; p=""} END{if(p)print p"_"v"_"a}' "$file" | sort +} + +# RPM filenames indexed in a published YUM distro/arch directory. +function yum_packages() { + local bucket="$1" dir="$2" + local version + version=$(gcs_version "$bucket") + local repodata="$FAKE_GCS_ROOT/$bucket/versions/$version/public/$dir/repodata" + [[ -d "$repodata" ]] || return 0 + zcat "$repodata"/*-primary.xml.gz | grep -o 'location href="[^"]*"' | sed 's/.*href="//; s/"$//; s|.*/||' | sort +} + +function contains() { + grep -qx -- "$2" <<<"$1" +} + +function lacks() { + ! grep -qx -- "$2" <<<"$1" +} + +function empty() { + [[ -z "$1" ]] +} + +function apt_signed() { + local version + version=$(gcs_version "$1") + gpg --verify "$FAKE_GCS_ROOT/$1/versions/$version/public/dists/$2/Release.gpg" \ + "$FAKE_GCS_ROOT/$1/versions/$version/public/dists/$2/Release" 2>/dev/null +} + +function yum_signed() { + local version + version=$(gcs_version "$1") + gpg --verify "$FAKE_GCS_ROOT/$1/versions/$version/public/$2/repodata/repomd.xml.asc" \ + "$FAKE_GCS_ROOT/$1/versions/$version/public/$2/repodata/repomd.xml" 2>/dev/null +} + +header 'Creating signing key' +mkdir -m 700 "$GNUPGHOME" +gpg -q --batch --passphrase '' --quick-gen-key 'Archive Test ' default default never 2>/dev/null +gpg -q --batch --export-secret-keys --armor >"$FAKE_SIGNING_KEY_PATH" +KEY_ID=$(gpg --list-keys --with-colons | awk -F: '/^pub/{print $5; exit}') +echo "Key: $KEY_ID" + +header 'Building fixture packages' +PKGS="$WORK/pkgs" +mkdir -p "$PKGS/deb" "$PKGS/rpm" + +function make_deb() { + local name="$1" version="$2" arch="$3" + local dir="$WORK/debbuild/${name}_${version}_${arch}" + mkdir -p "$dir/DEBIAN" + printf 'Package: %s\nVersion: %s\nArchitecture: %s\nMaintainer: Test \nDescription: test\n' \ + "$name" "$version" "$arch" >"$dir/DEBIAN/control" + dpkg-deb -Zgzip -b "$dir" "$PKGS/deb/${name}_${version}_${arch}.deb" >/dev/null +} + +function make_rpm() { + local name="$1" version="$2" release="$3" arch="$4" + local topdir="$WORK/rpmbuild/$name-$version-$release" + mkdir -p "$topdir/SPECS" + cat >"$topdir/SPECS/pkg.spec" </dev/null 2>&1 + find "$topdir/RPMS" -name '*.rpm' -exec cp {} "$PKGS/rpm/" \; +} + +# DEB: debian-12 (supported), debian-10 (EOL). Ruby 3.4 active, 3.1 EOL. +for distro in debian-12 debian-10; do + make_deb fullstaq-ruby-3.4 "9-$distro" amd64 + make_deb fullstaq-ruby-3.4-jemalloc "9-$distro" amd64 + make_deb fullstaq-ruby-3.4.1 "0-$distro" amd64 + make_deb fullstaq-ruby-3.1 "5-$distro" amd64 + make_deb fullstaq-ruby-3.1-jemalloc "5-$distro" amd64 + make_deb fullstaq-ruby-3.1.7 "0-$distro" amd64 +done +make_deb fullstaq-ruby-common 1.0-0 all +make_deb fullstaq-rbenv 1.1.2-16 all + +# RPM: el-9 (supported), centos-7 (EOL). +for distro in el9 centos7; do + make_rpm fullstaq-ruby-3.4 rev9 "$distro" x86_64 + make_rpm fullstaq-ruby-3.4-jemalloc rev9 "$distro" x86_64 + make_rpm fullstaq-ruby-3.1 rev5 "$distro" x86_64 + make_rpm fullstaq-ruby-3.1-jemalloc rev5 "$distro" x86_64 + make_rpm fullstaq-ruby-3.1.7 rev0 "$distro" x86_64 +done +make_rpm fullstaq-ruby-common 1.0 0 noarch +make_rpm fullstaq-rbenv 1.1.2_16 1 noarch +ls "$PKGS/deb" "$PKGS/rpm" + +header 'Seeding live APT repository (version 1)' +function seed_apt() { + local state="$WORK/seed-apt" + mkdir -p "$state/db" "$state/repo" + printf '{"rootDir":"%s","FileSystemPublishEndpoints":{"main":{"rootDir":"%s/repo","linkMethod":"symlink","verifyMethod":"md5"}}}' \ + "$state" "$state" >"$WORK/seed-apt.conf" + local distro + for distro in debian-12 debian-10; do + aptly -config="$WORK/seed-apt.conf" repo create "$distro" >/dev/null + aptly -config="$WORK/seed-apt.conf" repo add "$distro" "$PKGS"/deb/*"$distro"*.deb \ + "$PKGS/deb/fullstaq-ruby-common_1.0-0_all.deb" "$PKGS/deb/fullstaq-rbenv_1.1.2-16_all.deb" >/dev/null + aptly -config="$WORK/seed-apt.conf" publish repo -batch -gpg-key="$KEY_ID" -distribution="$distro" \ + -origin=Fullstaq-Ruby -label=Fullstaq-Ruby "$distro" filesystem:main:. >/dev/null + done + local version_dir="$FAKE_GCS_ROOT/$LIVE_APT/versions/1" + mkdir -p "$version_dir/public" + tar -C "$state" -cf - . | zstd -q -T0 >"$version_dir/state.tar.zst" + cp -rL "$state/repo/." "$version_dir/public/" + echo 1 >"$version_dir/version.txt" + echo 1 >"$FAKE_GCS_ROOT/$LIVE_APT/versions/latest_version.txt" +} +seed_apt + +header 'Seeding live YUM repository (version 1)' +function seed_yum() { + local version_dir="$FAKE_GCS_ROOT/$LIVE_YUM/versions/1" + local distro sanitized dir + for distro in el-9 centos-7; do + sanitized=${distro//-/} + dir="$version_dir/public/$distro/x86_64" + mkdir -p "$dir" + cp "$PKGS"/rpm/*"$sanitized"*.rpm "$PKGS"/rpm/*.noarch.rpm "$dir/" + createrepo_c -q "$dir" + gpg -q --batch --yes --detach-sign --armor "$dir/repodata/repomd.xml" + done + echo 1 >"$version_dir/version.txt" + echo 1 >"$FAKE_GCS_ROOT/$LIVE_YUM/versions/latest_version.txt" +} +seed_yum + +# The scripts must run from a writable copy: they don't write to the repo, +# but GCloudStorageLock logs relative to cwd and we keep /src read-only. +cp -r /src "$WORK/src" +cd "$WORK/src" + +APT_ENV=(env PRODUCTION_REPO_BUCKET_NAME="$LIVE_APT" ARCHIVE_REPO_BUCKET_NAME="$ARCHIVE_APT") +YUM_ENV=(env PRODUCTION_REPO_BUCKET_NAME="$LIVE_YUM" ARCHIVE_REPO_BUCKET_NAME="$ARCHIVE_YUM") + +function snapshot() { + (cd "$FAKE_GCS_ROOT" && find . -type f ! -path '*/locks/*' -exec md5sum {} + | sort -k2) +} + +# ---------------------------------------------------------------- APT + +header 'APT: dry run uploads nothing' +before=$(snapshot) +"${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb --dry-run +check 'dry run leaves fake GCS unchanged' [ "$before" == "$(snapshot)" ] + +header 'APT: first run' +"${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb + +LIVE_D12=$(apt_packages $LIVE_APT debian-12) +ARCH_D12=$(apt_packages $ARCHIVE_APT debian-12) +ARCH_D10=$(apt_packages $ARCHIVE_APT debian-10) +echo "live debian-12:"; sed 's/^/ /' <<<"$LIVE_D12" +echo "archive debian-12:"; sed 's/^/ /' <<<"$ARCH_D12" +echo "archive debian-10:"; sed 's/^/ /' <<<"$ARCH_D10" + +check 'live version bumped to 2' [ "$(gcs_version $LIVE_APT)" == 2 ] +check 'archive version is 1' [ "$(gcs_version $ARCHIVE_APT)" == 1 ] +check 'live debian-12 keeps active Ruby' contains "$LIVE_D12" fullstaq-ruby-3.4_9-debian-12_amd64 +check 'live debian-12 keeps active Ruby variant' contains "$LIVE_D12" fullstaq-ruby-3.4-jemalloc_9-debian-12_amd64 +check 'live debian-12 keeps tiny of active minor' contains "$LIVE_D12" fullstaq-ruby-3.4.1_0-debian-12_amd64 +check 'live debian-12 keeps common' contains "$LIVE_D12" fullstaq-ruby-common_1.0-0_all +check 'live debian-12 keeps rbenv' contains "$LIVE_D12" fullstaq-rbenv_1.1.2-16_all +check 'live debian-12 drops EOL Ruby' lacks "$LIVE_D12" fullstaq-ruby-3.1_5-debian-12_amd64 +check 'live debian-12 drops EOL Ruby variant' lacks "$LIVE_D12" fullstaq-ruby-3.1-jemalloc_5-debian-12_amd64 +check 'live debian-12 drops tiny of EOL minor' lacks "$LIVE_D12" fullstaq-ruby-3.1.7_0-debian-12_amd64 +check 'live debian-10 no longer published' empty "$(apt_packages $LIVE_APT debian-10)" +check 'archive debian-12 has exactly the EOL Ruby packages' [ "$ARCH_D12" == "$(printf '%s\n' \ + fullstaq-ruby-3.1-jemalloc_5-debian-12_amd64 fullstaq-ruby-3.1.7_0-debian-12_amd64 fullstaq-ruby-3.1_5-debian-12_amd64 | sort)" ] +check 'archive debian-10 has all 8 packages' [ "$(wc -l <<<"$ARCH_D10")" -eq 8 ] +check 'archive debian-10 has common' contains "$ARCH_D10" fullstaq-ruby-common_1.0-0_all +check 'live and archive debian-12 are disjoint' empty "$(comm -12 <(echo "$LIVE_D12") <(echo "$ARCH_D12"))" +check 'live debian-12 Release is signed' apt_signed $LIVE_APT debian-12 +check 'archive debian-12 Release is signed' apt_signed $ARCHIVE_APT debian-12 +check 'archive debian-10 Release is signed' apt_signed $ARCHIVE_APT debian-10 +v=$(gcs_version $ARCHIVE_APT) +check 'archive pool contains debian-10 debs' \ + [ -n "$(find "$FAKE_GCS_ROOT/$ARCHIVE_APT/versions/$v/public/pool" -name '*debian-10*.deb')" ] +v=$(gcs_version $LIVE_APT) +check 'live pool no longer contains EOL Ruby debs' \ + [ -z "$(find "$FAKE_GCS_ROOT/$LIVE_APT/versions/$v/public/pool" -name 'fullstaq-ruby-3.1*')" ] +check 'live pool no longer contains debian-10 debs' \ + [ -z "$(find "$FAKE_GCS_ROOT/$LIVE_APT/versions/$v/public/pool" -name '*debian-10*')" ] +check 'lock released' [ ! -e "$FAKE_GCS_ROOT/$LIVE_APT/locks/apt" ] + +header 'APT: second run is a no-op' +before=$(snapshot) +"${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb +check 'second run changes nothing' [ "$before" == "$(snapshot)" ] + +header 'APT: rerun after crash between archive and live upload' +# Simulate: archive version 2 was activated, live upload never happened. +# Roll the live pointer back to version 1 (still intact in the bucket). +echo 1 >"$FAKE_GCS_ROOT/$LIVE_APT/versions/latest_version.txt" +rm -rf "$FAKE_GCS_ROOT/$LIVE_APT/versions/2" +"${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb +check 'archive debian-12 unchanged after rerun' [ "$ARCH_D12" == "$(apt_packages $ARCHIVE_APT debian-12)" ] +check 'archive debian-10 unchanged after rerun' [ "$ARCH_D10" == "$(apt_packages $ARCHIVE_APT debian-10)" ] +check 'live debian-12 trimmed after rerun' [ "$LIVE_D12" == "$(apt_packages $LIVE_APT debian-12)" ] +check 'archive debian-12 still signed after rerun' apt_signed $ARCHIVE_APT debian-12 + +# ---------------------------------------------------------------- YUM + +header 'YUM: dry run uploads nothing' +before=$(snapshot) +"${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb --dry-run +check 'dry run leaves fake GCS unchanged' [ "$before" == "$(snapshot)" ] + +header 'YUM: first run' +"${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb + +LIVE_EL9=$(yum_packages $LIVE_YUM el-9/x86_64) +ARCH_EL9=$(yum_packages $ARCHIVE_YUM el-9/x86_64) +ARCH_C7=$(yum_packages $ARCHIVE_YUM centos-7/x86_64) +echo "live el-9:"; sed 's/^/ /' <<<"$LIVE_EL9" +echo "archive el-9:"; sed 's/^/ /' <<<"$ARCH_EL9" +echo "archive centos-7:"; sed 's/^/ /' <<<"$ARCH_C7" + +check 'live version bumped to 2' [ "$(gcs_version $LIVE_YUM)" == 2 ] +check 'archive version is 1' [ "$(gcs_version $ARCHIVE_YUM)" == 1 ] +check 'live el-9 keeps active Ruby' contains "$LIVE_EL9" fullstaq-ruby-3.4-rev9-el9.x86_64.rpm +check 'live el-9 keeps common' contains "$LIVE_EL9" fullstaq-ruby-common-1.0-0.noarch.rpm +check 'live el-9 keeps rbenv' contains "$LIVE_EL9" fullstaq-rbenv-1.1.2_16-1.noarch.rpm +check 'live el-9 drops EOL Ruby' lacks "$LIVE_EL9" fullstaq-ruby-3.1-rev5-el9.x86_64.rpm +check 'live el-9 drops tiny of EOL minor' lacks "$LIVE_EL9" fullstaq-ruby-3.1.7-rev0-el9.x86_64.rpm +check 'live el-9 has no leftover EOL RPM files' \ + [ -z "$(find "$FAKE_GCS_ROOT/$LIVE_YUM/versions/2/public/el-9" -name 'fullstaq-ruby-3.1*')" ] +check 'live centos-7 removed' [ ! -e "$FAKE_GCS_ROOT/$LIVE_YUM/versions/2/public/centos-7" ] +check 'archive el-9 has exactly the EOL Ruby packages' [ "$ARCH_EL9" == "$(printf '%s\n' \ + fullstaq-ruby-3.1-jemalloc-rev5-el9.x86_64.rpm fullstaq-ruby-3.1-rev5-el9.x86_64.rpm fullstaq-ruby-3.1.7-rev0-el9.x86_64.rpm | sort)" ] +check 'archive centos-7 has all 7 packages' [ "$(wc -l <<<"$ARCH_C7")" -eq 7 ] +check 'live and archive el-9 are disjoint' empty "$(comm -12 <(echo "$LIVE_EL9") <(echo "$ARCH_EL9"))" +check 'live el-9 repomd is signed' yum_signed $LIVE_YUM el-9/x86_64 +check 'archive el-9 repomd is signed' yum_signed $ARCHIVE_YUM el-9/x86_64 +check 'archive centos-7 repomd is signed' yum_signed $ARCHIVE_YUM centos-7/x86_64 +check 'no nested distro directory in archive' [ ! -e "$FAKE_GCS_ROOT/$ARCHIVE_YUM/versions/1/public/centos-7/centos-7" ] +check 'lock released' [ ! -e "$FAKE_GCS_ROOT/$LIVE_YUM/locks/yum" ] + +header 'YUM: second run is a no-op' +before=$(snapshot) +"${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb +check 'second run changes nothing' [ "$before" == "$(snapshot)" ] + +header 'YUM: rerun after crash between archive and live upload' +echo 1 >"$FAKE_GCS_ROOT/$LIVE_YUM/versions/latest_version.txt" +rm -rf "$FAKE_GCS_ROOT/$LIVE_YUM/versions/2" +"${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb +check 'archive el-9 unchanged after rerun' [ "$ARCH_EL9" == "$(yum_packages $ARCHIVE_YUM el-9/x86_64)" ] +check 'archive centos-7 unchanged after rerun' [ "$ARCH_C7" == "$(yum_packages $ARCHIVE_YUM centos-7/x86_64)" ] +check 'live el-9 trimmed after rerun' [ "$LIVE_EL9" == "$(yum_packages $LIVE_YUM el-9/x86_64)" ] +check 'no nested distro directory after rerun' \ + [ -z "$(find "$FAKE_GCS_ROOT/$ARCHIVE_YUM" -path '*/centos-7/centos-7')" ] + +header 'Result' +if [[ $FAILURES -eq 0 ]]; then + echo 'All checks passed' +else + echo "$FAILURES checks failed" + exit 1 +fi From dbc7e1c92da737c6dec1abc76cd081c6cbe6a6fd Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 21:56:53 -0500 Subject: [PATCH 08/12] refactor: remove migrate and prune archive scripts Replaced by archive-apt-packages.rb and archive-yum-packages.rb. --- .../ci-cd/archive/migrate-apt-to-archive.rb | 613 ------------------ .../ci-cd/archive/migrate-yum-to-archive.rb | 350 ---------- .../ci-cd/archive/prune-apt-packages.rb | 390 ----------- .../ci-cd/archive/prune-yum-packages.rb | 246 ------- 4 files changed, 1599 deletions(-) delete mode 100755 internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb delete mode 100755 internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb delete mode 100755 internal-scripts/ci-cd/archive/prune-apt-packages.rb delete mode 100755 internal-scripts/ci-cd/archive/prune-yum-packages.rb diff --git a/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb b/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb deleted file mode 100755 index 40b95b84..00000000 --- a/internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb +++ /dev/null @@ -1,613 +0,0 @@ -#!/usr/bin/env ruby -# frozen_string_literal: true - -# Incremental migration script to move EOL distribution packages -# from the main APT repository to the archive APT repository. -# Safe to run repeatedly — merges new EOL distros into the existing archive. -# -# Usage: -# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ -# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-archive-repo \ -# ./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb [--dry-run] [--distros centos-8,debian-9] -# -# If --distros is not specified, automatically detects EOL distros by comparing -# the Aptly state against the DEB distributions defined in config.yml. - -require_relative '../../../lib/gcloud_storage_lock' -require_relative '../../../lib/ci_workflow_support' -require_relative '../../../lib/shell_scripting_support' -require_relative '../../../lib/publishing_support' -require 'json' -require 'shellwords' -require 'tmpdir' -require 'set' -require 'fileutils' -require 'optparse' - -class MigrateAptToArchive - REPO_ORIGIN = 'Fullstaq-Ruby' - REPO_LABEL = 'Fullstaq-Ruby' - - include CiWorkflowSupport - include ShellScriptingSupport - include PublishingSupport - - def main - parse_options - require_envvar 'PRODUCTION_REPO_BUCKET_NAME' - require_envvar 'ARCHIVE_REPO_BUCKET_NAME' - - print_header 'Initializing' - load_config - create_temp_dirs - ensure_gpg_state_isolated - activate_wrappers_bin_dir - initialize_locking - initialize_aptly(@main_aptly_config_path, @main_state_path, @main_state_repo_path) - fetch_and_import_signing_key - - eol_distros = nil - version = nil - - begin - synchronize do - print_header 'Downloading main repository state' - version = get_latest_production_repo_version - if version == 0 - abort 'ERROR: No production repository exists yet' - end - fetch_main_state(version) - - print_header 'Identifying EOL distributions' - eol_distros = identify_eol_distros - if eol_distros.empty? - log_notice 'No EOL distributions found to archive' - return - end - log_notice "EOL distributions to archive: #{eol_distros.join(', ')}" - - print_header 'Fetching existing archive state (if any)' - @archive_version = get_latest_archive_version - if @archive_version > 0 - log_notice "Existing archive at version #{@archive_version}, will merge" - fetch_archive_state(@archive_version) - else - log_notice 'No existing archive — creating fresh' - end - - print_header 'Creating archive repository' - create_archive_state(eol_distros) - check_lock_health - - print_header 'Trimming main repository state' - trim_main_state(eol_distros) - check_lock_health - - if @dry_run - log_notice 'DRY RUN — not uploading changes' - print_summary(eol_distros, version) - return - end - - print_header 'Uploading archive repository' - upload_archive - check_lock_health - - print_header 'Uploading trimmed main repository' - upload_main(version) - end - - print_header 'Success!' - print_summary(eol_distros, version) if eol_distros && !eol_distros.empty? - ensure - cleanup - end - end - -private - def parse_options - @dry_run = false - @explicit_distros = nil - - OptionParser.new do |opts| - opts.banner = "Usage: #{$0} [options]" - opts.on('--dry-run', 'Do not upload changes') { @dry_run = true } - opts.on('--distros DISTROS', 'Comma-separated list of distros to archive') do |v| - @explicit_distros = v.split(',').map(&:strip) - end - end.parse! - end - - def create_temp_dirs - log_notice 'Creating temporary directories' - @temp_dir = Dir.mktmpdir('apt-archive-migration') - @wrappers_bin_dir = "#{@temp_dir}/wrappers" - - @main_aptly_config_path = "#{@temp_dir}/aptly-main.conf" - @main_state_path = "#{@temp_dir}/main-state" - @main_state_db_path = "#{@main_state_path}/db" - @main_state_repo_path = "#{@main_state_path}/repo" - @main_state_archive_path = "#{@temp_dir}/main-state.tar.zst" - - @archive_aptly_config_path = "#{@temp_dir}/aptly-archive.conf" - @archive_state_path = "#{@temp_dir}/archive-state" - @archive_state_db_path = "#{@archive_state_path}/db" - @archive_state_repo_path = "#{@archive_state_path}/repo" - @archive_state_archive_path = "#{@temp_dir}/archive-state.tar.zst" - - @signing_key_path = "#{@temp_dir}/key.gpg" - - Dir.mkdir(@wrappers_bin_dir) - [@main_state_path, @main_state_db_path, @main_state_repo_path, - @archive_state_path, @archive_state_db_path, @archive_state_repo_path].each do |dir| - FileUtils.mkdir_p(dir) - end - end - - def ensure_gpg_state_isolated - log_notice 'Creating GPG wrapper' - File.open("#{@wrappers_bin_dir}/gpg", 'w:utf-8') do |f| - f.write("#!/bin/sh\n") - f.write( - sprintf( - "exec %s --homedir %s \"$@\"\n", - Shellwords.escape(find_gpg), - Shellwords.escape(@temp_dir) - ) - ) - end - File.chmod(0755, "#{@wrappers_bin_dir}/gpg") - end - - def find_gpg - ENV['PATH'].split(':').each do |dir| - next if dir == @wrappers_bin_dir - candidate = "#{dir}/gpg" - return candidate if File.exist?(candidate) - end - abort('GPG not found') - end - - def activate_wrappers_bin_dir - ENV['PATH'] = "#{@wrappers_bin_dir}:#{ENV['PATH']}" - end - - def initialize_locking - @lock = GCloudStorageLock.new(url: lock_url) - end - - def lock_url - "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/locks/apt" - end - - def synchronize(&block) - @lock.synchronize(&block) - end - - def check_lock_health - abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? - end - - def initialize_aptly(config_path, state_path, repo_path) - log_notice "Creating Aptly config: #{config_path}" - File.open(config_path, 'w:utf-8') do |f| - f.write(JSON.generate( - rootDir: state_path, - FileSystemPublishEndpoints: { - main: { - rootDir: repo_path, - linkMethod: 'symlink', - verifyMethod: 'md5' - } - } - )) - end - end - - def fetch_and_import_signing_key - log_notice 'Fetching and importing signing key' - File.open(@signing_key_path, 'wb') do |f| - f.write(fetch_signing_key) - end - @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) - log_info "Signing key ID: #{@gpg_key_id}" - import_gpg_key(@temp_dir, @signing_key_path) - end - - def fetch_main_state(version) - log_notice "Fetching main state version #{version}" - url = "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/state.tar.zst" - run_bash( - sprintf('gsutil -m cp %s - | zstd -dc | tar -xC %s', - Shellwords.escape(url), - Shellwords.escape(@main_state_path)), - pipefail: true, - log_invocation: true, - check_error: true, - passthru_output: true - ) - end - - def latest_production_version_note_url - "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt" - end - - def get_latest_archive_version - url = "gs://#{ENV['ARCHIVE_REPO_BUCKET_NAME']}/versions/latest_version.txt" - stdout_output, stderr_output, status = run_command_capture_output( - 'gsutil', 'cp', url, '-', - log_invocation: false, - check_error: false - ) - if status.success? - v = stdout_output.strip - if v =~ /\A[0-9]+\Z/ - v.to_i - else - abort("ERROR: invalid version number stored in #{url}") - end - elsif stderr_output =~ /No URLs matched/ - 0 - else - abort("ERROR: error fetching #{url}: #{stderr_output.chomp}") - end - end - - def fetch_archive_state(version) - log_notice "Fetching archive state version #{version}" - url = "gs://#{ENV['ARCHIVE_REPO_BUCKET_NAME']}/versions/#{version}/state.tar.zst" - run_bash( - sprintf('gsutil -m cp %s - | zstd -dc | tar -xC %s', - Shellwords.escape(url), - Shellwords.escape(@archive_state_path)), - pipefail: true, - log_invocation: true, - check_error: true, - passthru_output: true - ) - end - - def identify_eol_distros - if @explicit_distros - log_notice "Using explicitly specified distros: #{@explicit_distros.join(', ')}" - return @explicit_distros - end - - published = list_aptly_repos(@main_aptly_config_path) - supported = distributions - .select { |d| d[:package_format] == :DEB } - .map { |d| d[:name] } - - eol = published - supported - log_info "Published distros in Aptly state: #{published.join(', ')}" - log_info "Currently supported DEB distros: #{supported.join(', ')}" - log_info "EOL distros (published but not supported): #{eol.join(', ')}" - eol.sort - end - - def list_aptly_repos(config_path) - stdout_output, _, _ = run_command_capture_output( - 'aptly', 'repo', 'list', - "-config=#{config_path}", - '-raw', - log_invocation: true, - check_error: true - ) - stdout_output.split("\n").map(&:strip).reject(&:empty?) - end - - def list_aptly_packages(config_path, repo_name) - stdout_output, _, _ = run_command_capture_output( - 'aptly', 'repo', 'show', - "-config=#{config_path}", - '-with-packages', - repo_name, - log_invocation: false, - check_error: true - ) - stdout_output.sub!(/.*^Packages:$/m, '') - stdout_output.split("\n").map(&:strip).reject(&:empty?) - end - - def create_archive_state(eol_distros) - if @archive_version == 0 - initialize_aptly(@archive_aptly_config_path, @archive_state_path, @archive_state_repo_path) - else - # Existing archive was fetched — just initialize the Aptly config pointing at it - initialize_aptly(@archive_aptly_config_path, @archive_state_path, @archive_state_repo_path) - existing = list_aptly_repos(@archive_aptly_config_path) - log_notice "Existing archive contains distros: #{existing.join(', ')}" - end - - # Copy EOL packages from main pool into archive pool - main_pool = "#{@main_state_path}/pool" - archive_pool = "#{@archive_state_path}/pool" - if File.exist?(main_pool) - if File.exist?(archive_pool) - # Merge: copy new pool files that don't already exist (-n: no-clobber). - # Real I/O errors must surface — only no-clobber skips are benign. - run_bash( - sprintf('cp -rn %s/* %s/', - Shellwords.escape(main_pool), - Shellwords.escape(archive_pool)), - log_invocation: true, check_error: true, pipefail: false - ) - else - FileUtils.cp_r(main_pool, @archive_state_path) - end - end - - eol_distros.each do |distro| - packages = list_aptly_packages(@main_aptly_config_path, distro) - log_notice "[#{distro}] Exporting #{packages.size} packages to archive" - - # Create the archive repo for this distro - run_command( - 'aptly', 'repo', 'create', - "-config=#{@archive_aptly_config_path}", - distro, - log_invocation: true, - check_error: true - ) - - # Import packages by copying the Aptly database directory for this repo - main_repo_db = "#{@main_state_db_path}/repo/#{distro}" - if File.exist?(main_repo_db) - FileUtils.cp_r(main_repo_db, "#{@archive_state_db_path}/repo/") - end - end - - # Publish all distros in the archive (existing + newly added) - all_archive_distros = list_aptly_repos(@archive_aptly_config_path) - all_archive_distros.each do |distro| - log_notice "[#{distro}] Publishing in archive" - publish_aptly_repo(@archive_aptly_config_path, distro) - end - - # Archive the state - log_notice 'Creating archive state archive' - run_bash( - sprintf("tar -C %s -cf - . | zstd -T0 > %s", - Shellwords.escape(@archive_state_path), - Shellwords.escape(@archive_state_archive_path)), - pipefail: true, - log_invocation: true, - check_error: true, - passthru_output: true - ) - end - - def trim_main_state(eol_distros) - eol_distros.each do |distro| - log_notice "[#{distro}] Dropping from main repository" - - # Drop the publication first (if it exists) - run_command( - 'aptly', 'publish', 'drop', - "-config=#{@main_aptly_config_path}", - distro, 'filesystem:main:.', - log_invocation: true, - check_error: false - ) - - # Drop the repo - run_command( - 'aptly', 'repo', 'drop', - "-config=#{@main_aptly_config_path}", - distro, - log_invocation: true, - check_error: true - ) - end - - # Compact the database to reclaim space from dropped packages - log_notice 'Compacting main state' - run_command( - 'aptly', 'db', 'cleanup', - "-config=#{@main_aptly_config_path}", - '-verbose', - log_invocation: true, - check_error: true, - passthru_output: true - ) - - # Re-publish remaining distros - remaining = list_aptly_repos(@main_aptly_config_path) - remaining.each do |distro| - log_notice "[#{distro}] Re-publishing in main repository" - publish_aptly_repo(@main_aptly_config_path, distro) - end - - # Re-archive the trimmed state - log_notice 'Creating trimmed main state archive' - run_bash( - sprintf("tar -C %s -cf - . | zstd -T0 > %s", - Shellwords.escape(@main_state_path), - Shellwords.escape(@main_state_archive_path)), - pipefail: true, - log_invocation: true, - check_error: true, - passthru_output: true - ) - end - - def publish_aptly_repo(config_path, distro) - _, stderr_output, status = run_command_capture_output( - 'aptly', 'publish', 'repo', - '-batch', '-force-overwrite', - "-config=#{config_path}", - "-gpg-key=#{@gpg_key_id}", - "-distribution=#{distro}", - "-origin=#{REPO_ORIGIN}", - "-label=#{REPO_LABEL}", - distro, 'filesystem:main:.', - log_invocation: true, - check_error: false - ) - if !status.success? - if stderr_output =~ /unable to figure out list of architectures/ - run_command( - 'aptly', 'publish', 'repo', - '-batch', '-force-overwrite', '-architectures=all', - "-config=#{config_path}", - "-gpg-key=#{@gpg_key_id}", - "-distribution=#{distro}", - "-origin=#{REPO_ORIGIN}", - "-label=#{REPO_LABEL}", - distro, 'filesystem:main:.', - log_invocation: true, - check_error: true - ) - else - abort("ERROR publishing #{distro}: #{stderr_output.chomp}") - end - end - end - - def upload_archive - archive_bucket = ENV['ARCHIVE_REPO_BUCKET_NAME'] - new_archive_version = @archive_version + 1 - - # Upload state - log_notice "Uploading archive state as version #{new_archive_version}" - run_command( - 'gsutil', - '-h', 'Cache-Control:public', - 'cp', - @archive_state_archive_path, - "gs://#{archive_bucket}/versions/#{new_archive_version}/state.tar.zst", - log_invocation: true, - check_error: true, - passthru_output: true - ) - - # Upload published repo - log_notice 'Uploading archive repository' - run_command( - 'gsutil', '-m', - '-h', 'Cache-Control:public', - 'rsync', '-r', '-d', - @archive_state_repo_path, - "gs://#{archive_bucket}/versions/#{new_archive_version}/public", - log_invocation: true, - check_error: true, - passthru_output: true - ) - - # Create version note - run_bash( - sprintf( - 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', - Shellwords.escape("gs://#{archive_bucket}/versions/#{new_archive_version}/version.txt"), - Shellwords.escape(new_archive_version.to_s) - ), - log_invocation: true, - check_error: true, - pipefail: false - ) - - # Declare latest version - run_bash( - sprintf( - 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', - Shellwords.escape("gs://#{archive_bucket}/versions/latest_version.txt"), - Shellwords.escape(new_archive_version.to_s) - ), - log_invocation: true, - check_error: true, - pipefail: false - ) - end - - def upload_main(old_version) - new_version = old_version + 1 - bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] - - # Upload state - log_notice "Uploading trimmed main state as version #{new_version}" - run_command( - 'gsutil', - '-h', 'Cache-Control:public', - 'cp', - @main_state_archive_path, - "gs://#{bucket}/versions/#{new_version}/state.tar.zst", - log_invocation: true, - check_error: true, - passthru_output: true - ) - - # Copy previous repo version, then overwrite with trimmed version - log_notice "Copying repo version #{old_version} to #{new_version}" - run_command( - 'gsutil', '-m', - '-h', 'Cache-Control:public', - 'rsync', '-r', '-d', - "gs://#{bucket}/versions/#{old_version}/public", - "gs://#{bucket}/versions/#{new_version}/public", - log_invocation: true, - check_error: true, - passthru_output: true - ) - - log_notice "Uploading trimmed repo as version #{new_version}" - run_command( - 'gsutil', '-m', - '-h', 'Cache-Control:public', - 'rsync', '-r', '-d', - @main_state_repo_path, - "gs://#{bucket}/versions/#{new_version}/public", - log_invocation: true, - check_error: true, - passthru_output: true - ) - - # Version note - run_bash( - sprintf( - 'gsutil -q -h Content-Type:text/plain -h Cache-Control:public cp - %s <<<%s', - Shellwords.escape("gs://#{bucket}/versions/#{new_version}/version.txt"), - Shellwords.escape(new_version.to_s) - ), - log_invocation: true, - check_error: true, - pipefail: false - ) - - # Activate new version - log_notice "Activating main repo version #{new_version}" - run_bash( - sprintf( - 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', - Shellwords.escape("gs://#{bucket}/versions/latest_version.txt"), - Shellwords.escape(new_version.to_s) - ), - log_invocation: true, - check_error: true, - pipefail: false - ) - end - - def print_summary(eol_distros, old_version) - new_archive_version = @archive_version + 1 - archive_bucket = ENV['ARCHIVE_REPO_BUCKET_NAME'] - log_notice 'Migration summary' - log_info "Archived distributions: #{eol_distros.join(', ')}" - log_info "Main repo: version #{old_version} -> #{old_version + 1}" - log_info "Archive repo: version #{@archive_version} -> #{new_archive_version}" - log_info '' - log_info 'Archive APT repo URL:' - log_info " https://storage.googleapis.com/#{archive_bucket}/versions/#{new_archive_version}/public" - log_info '' - log_info 'Next steps:' - log_info ' 1. Restart Caddy on the backend server (required; archive stays 404 until then): sudo systemctl restart caddy' - log_info ' 2. Verify archive repo: curl https://apt-archive.fullstaqruby.org/dists/' - log_info ' 3. Verify main repo still works: apt-get update on a supported distro' - end - - def cleanup - log_info "Cleaning up #{@temp_dir}" - FileUtils.remove_entry_secure(@temp_dir) - end -end - -MigrateAptToArchive.new.main diff --git a/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb b/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb deleted file mode 100755 index c01a06e0..00000000 --- a/internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb +++ /dev/null @@ -1,350 +0,0 @@ -#!/usr/bin/env ruby -# frozen_string_literal: true - -# Incremental migration script to move EOL distribution packages -# from the main YUM repository to the archive YUM repository. -# Safe to run repeatedly — merges new EOL distros into the existing archive. -# -# Usage: -# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ -# ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-archive-repo \ -# ./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb [--dry-run] [--distros centos-8] -# -# If --distros is not specified, automatically detects EOL distros by comparing -# the repo contents against the RPM distributions defined in config.yml. - -require_relative '../../../lib/gcloud_storage_lock' -require_relative '../../../lib/ci_workflow_support' -require_relative '../../../lib/shell_scripting_support' -require_relative '../../../lib/publishing_support' -require 'shellwords' -require 'tmpdir' -require 'fileutils' -require 'optparse' - -class MigrateYumToArchive - include CiWorkflowSupport - include ShellScriptingSupport - include PublishingSupport - - def main - parse_options - require_envvar 'PRODUCTION_REPO_BUCKET_NAME' - require_envvar 'ARCHIVE_REPO_BUCKET_NAME' - - print_header 'Initializing' - load_config - create_temp_dirs - initialize_locking - fetch_and_import_signing_key - - eol_distros = nil - version = nil - - begin - synchronize do - print_header 'Downloading main repository' - version = get_latest_production_repo_version - if version == 0 - abort 'ERROR: No production repository exists yet' - end - fetch_main_repo(version) - - print_header 'Identifying EOL distributions' - eol_distros = identify_eol_distros - if eol_distros.empty? - log_notice 'No EOL distributions found to archive' - return - end - log_notice "EOL distributions to archive: #{eol_distros.join(', ')}" - - print_header 'Fetching existing archive (if any)' - @archive_version = get_latest_archive_version - if @archive_version > 0 - log_notice "Existing archive at version #{@archive_version}, will merge" - fetch_archive_repo(@archive_version) - else - log_notice 'No existing archive — creating fresh' - @archive_repo_path = "#{@temp_dir}/archive-repo" - Dir.mkdir(@archive_repo_path) - end - - if @dry_run - log_notice 'DRY RUN — not uploading changes' - print_summary(eol_distros, version) - return - end - - print_header 'Uploading EOL distros to archive' - upload_archive(eol_distros) - check_lock_health - - print_header 'Removing EOL distros from main repository' - remove_from_main(eol_distros, version) - end - - print_header 'Success!' - print_summary(eol_distros, version) if eol_distros && !eol_distros.empty? - ensure - cleanup - end - end - -private - def parse_options - @dry_run = false - @explicit_distros = nil - - OptionParser.new do |opts| - opts.banner = "Usage: #{$0} [options]" - opts.on('--dry-run', 'Do not upload changes') { @dry_run = true } - opts.on('--distros DISTROS', 'Comma-separated list of distros to archive') do |v| - @explicit_distros = v.split(',').map(&:strip) - end - end.parse! - end - - def create_temp_dirs - @temp_dir = Dir.mktmpdir('yum-archive-migration') - @local_repo_path = "#{@temp_dir}/repo" - @signing_key_path = "#{@temp_dir}/key.gpg" - Dir.mkdir(@local_repo_path) - end - - def initialize_locking - @lock = GCloudStorageLock.new(url: lock_url) - end - - def lock_url - "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/locks/yum" - end - - def synchronize(&block) - @lock.synchronize(&block) - end - - def check_lock_health - abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? - end - - def fetch_and_import_signing_key - log_notice 'Fetching and importing signing key' - File.open(@signing_key_path, 'wb') do |f| - f.write(fetch_signing_key) - end - @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) - log_info "Signing key ID: #{@gpg_key_id}" - import_gpg_key(@temp_dir, @signing_key_path) - end - - def latest_production_version_note_url - "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt" - end - - def get_latest_archive_version - url = "gs://#{ENV['ARCHIVE_REPO_BUCKET_NAME']}/versions/latest_version.txt" - stdout_output, stderr_output, status = run_command_capture_output( - 'gsutil', 'cp', url, '-', - log_invocation: false, - check_error: false - ) - if status.success? - v = stdout_output.strip - if v =~ /\A[0-9]+\Z/ - v.to_i - else - abort("ERROR: invalid version number stored in #{url}") - end - elsif stderr_output =~ /No URLs matched/ - 0 - else - abort("ERROR: error fetching #{url}: #{stderr_output.chomp}") - end - end - - def fetch_archive_repo(version) - @archive_repo_path = "#{@temp_dir}/archive-repo" - Dir.mkdir(@archive_repo_path) - log_notice "Fetching archive repo version #{version}" - run_command( - 'gsutil', '-m', 'rsync', '-r', - "gs://#{ENV['ARCHIVE_REPO_BUCKET_NAME']}/versions/#{version}/public", - @archive_repo_path, - log_invocation: true, - check_error: true, - passthru_output: true - ) - end - - def fetch_main_repo(version) - log_notice "Fetching main repo version #{version}" - run_command( - 'gsutil', '-m', 'rsync', '-r', - "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/public", - @local_repo_path, - log_invocation: true, - check_error: true, - passthru_output: true - ) - end - - def identify_eol_distros - if @explicit_distros - log_notice "Using explicitly specified distros: #{@explicit_distros.join(', ')}" - return @explicit_distros - end - - # List distro directories in the downloaded repo - published = Dir.entries(@local_repo_path) - .select { |e| File.directory?("#{@local_repo_path}/#{e}") } - .reject { |e| e.start_with?('.') } - - supported = distributions - .select { |d| d[:package_format] == :RPM } - .map { |d| d[:name] } - - eol = published - supported - log_info "Published distros in YUM repo: #{published.join(', ')}" - log_info "Currently supported RPM distros: #{supported.join(', ')}" - log_info "EOL distros (published but not supported): #{eol.join(', ')}" - eol.sort - end - - def upload_archive(eol_distros) - archive_bucket = ENV['ARCHIVE_REPO_BUCKET_NAME'] - new_archive_version = @archive_version + 1 - - # Copy EOL distro directories into the local archive repo - eol_distros.each do |distro| - src = "#{@local_repo_path}/#{distro}" - dst = "#{@archive_repo_path}/#{distro}" - log_notice "[#{distro}] Copying to archive staging area" - FileUtils.cp_r(src, dst) - end - - # Upload merged archive repo - log_notice "Uploading archive repo as version #{new_archive_version}" - run_command( - 'gsutil', '-m', - '-h', 'Cache-Control:public', - 'rsync', '-r', '-d', - @archive_repo_path, - "gs://#{archive_bucket}/versions/#{new_archive_version}/public", - log_invocation: true, - check_error: true, - passthru_output: true - ) - - # Create version note - run_bash( - sprintf( - 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', - Shellwords.escape("gs://#{archive_bucket}/versions/#{new_archive_version}/version.txt"), - Shellwords.escape(new_archive_version.to_s) - ), - log_invocation: true, - check_error: true, - pipefail: false - ) - - # Declare latest version - run_bash( - sprintf( - 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', - Shellwords.escape("gs://#{archive_bucket}/versions/latest_version.txt"), - Shellwords.escape(new_archive_version.to_s) - ), - log_invocation: true, - check_error: true, - pipefail: false - ) - end - - def remove_from_main(eol_distros, old_version) - new_version = old_version + 1 - bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] - - # Remove EOL distro directories from local copy - eol_distros.each do |distro| - distro_path = "#{@local_repo_path}/#{distro}" - if File.exist?(distro_path) - log_notice "[#{distro}] Removing from local repo" - FileUtils.rm_rf(distro_path) - end - end - - # Upload trimmed repo as new version - log_notice "Copying repo version #{old_version} to #{new_version}" - run_command( - 'gsutil', '-m', - '-h', 'Cache-Control:public', - 'rsync', '-r', '-d', - "gs://#{bucket}/versions/#{old_version}/public", - "gs://#{bucket}/versions/#{new_version}/public", - log_invocation: true, - check_error: true, - passthru_output: true - ) - - log_notice "Uploading trimmed repo as version #{new_version}" - run_command( - 'gsutil', '-m', - '-h', 'Cache-Control:public', - 'rsync', '-r', '-d', - @local_repo_path, - "gs://#{bucket}/versions/#{new_version}/public", - log_invocation: true, - check_error: true, - passthru_output: true - ) - - # Version note - run_bash( - sprintf( - 'gsutil -q -h Content-Type:text/plain -h Cache-Control:public cp - %s <<<%s', - Shellwords.escape("gs://#{bucket}/versions/#{new_version}/version.txt"), - Shellwords.escape(new_version.to_s) - ), - log_invocation: true, - check_error: true, - pipefail: false - ) - - # Activate - log_notice "Activating main repo version #{new_version}" - run_bash( - sprintf( - 'gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', - Shellwords.escape("gs://#{bucket}/versions/latest_version.txt"), - Shellwords.escape(new_version.to_s) - ), - log_invocation: true, - check_error: true, - pipefail: false - ) - end - - def print_summary(eol_distros, old_version) - new_archive_version = @archive_version + 1 - archive_bucket = ENV['ARCHIVE_REPO_BUCKET_NAME'] - log_notice 'Migration summary' - log_info "Archived distributions: #{eol_distros.join(', ')}" - log_info "Main repo: version #{old_version} -> #{old_version + 1}" - log_info "Archive repo: version #{@archive_version} -> #{new_archive_version}" - log_info '' - log_info 'Archive YUM repo URL:' - log_info " https://storage.googleapis.com/#{archive_bucket}/versions/#{new_archive_version}/public" - log_info '' - log_info 'Next steps:' - log_info ' 1. Restart Caddy on the backend server (required; archive stays 404 until then): sudo systemctl restart caddy' - log_info ' 2. Verify archive repo: curl https://yum-archive.fullstaqruby.org/' - log_info ' 3. Verify main repo still works on a supported distro' - end - - def cleanup - log_info "Cleaning up #{@temp_dir}" - FileUtils.remove_entry_secure(@temp_dir) - end -end - -MigrateYumToArchive.new.main diff --git a/internal-scripts/ci-cd/archive/prune-apt-packages.rb b/internal-scripts/ci-cd/archive/prune-apt-packages.rb deleted file mode 100755 index eed086ac..00000000 --- a/internal-scripts/ci-cd/archive/prune-apt-packages.rb +++ /dev/null @@ -1,390 +0,0 @@ -#!/usr/bin/env ruby -# frozen_string_literal: true - -# Prune EOL Ruby version packages from the APT repository state. -# -# Removes packages for Ruby versions no longer in config.yml from all -# distro repos in the Aptly state, then compacts and re-uploads. -# -# Usage: -# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ -# ./internal-scripts/ci-cd/archive/prune-apt-packages.rb [--dry-run] -# -# Automatically detects which Ruby versions are EOL by comparing -# packages in the Aptly state against minor_version_packages in config.yml. - -require_relative '../../../lib/gcloud_storage_lock' -require_relative '../../../lib/ci_workflow_support' -require_relative '../../../lib/shell_scripting_support' -require_relative '../../../lib/publishing_support' -require 'json' -require 'shellwords' -require 'tmpdir' -require 'set' -require 'fileutils' -require 'optparse' - -class PruneAptPackages - REPO_ORIGIN = 'Fullstaq-Ruby' - REPO_LABEL = 'Fullstaq-Ruby' - - # Matches: fullstaq-ruby-3.1, fullstaq-ruby-3.1-jemalloc, fullstaq-ruby-3.1.7, etc. - # Does NOT match: fullstaq-ruby-common, fullstaq-rbenv - RUBY_PACKAGE_PATTERN = /\Afullstaq-ruby-(\d+\.\d+)/ - - include CiWorkflowSupport - include ShellScriptingSupport - include PublishingSupport - - def main - parse_options - require_envvar 'PRODUCTION_REPO_BUCKET_NAME' - - print_header 'Initializing' - load_config - create_temp_dirs - ensure_gpg_state_isolated - activate_wrappers_bin_dir - initialize_locking - initialize_aptly - fetch_and_import_signing_key - - total_pruned = 0 - version = nil - repos = nil - - begin - synchronize do - print_header 'Downloading repository state' - version = get_latest_production_repo_version - if version == 0 - abort 'ERROR: No production repository exists yet' - end - fetch_state(version) - - print_header 'Identifying EOL Ruby versions' - active_minors = active_ruby_minor_versions - log_info "Active Ruby minor versions: #{active_minors.join(', ')}" - - print_header 'Scanning packages' - repos = list_aptly_repos - repos.each do |distro| - pruned = prune_eol_packages_from_repo(distro, active_minors) - total_pruned += pruned - end - - if total_pruned == 0 - log_notice 'No EOL Ruby packages found to prune' - return - end - - log_notice "Total packages to prune: #{total_pruned}" - - if @dry_run - log_notice 'DRY RUN — not uploading changes' - return - end - - print_header 'Compacting state' - compact_state - check_lock_health - - print_header 'Re-publishing repository' - repos.each do |distro| - publish_repo(distro) - end - check_lock_health - - print_header 'Archiving and uploading state' - archive_state - upload_state(version + 1) - upload_repo(version + 1, version) - create_version_note(version + 1) - declare_latest_version(version + 1) - end - - print_header 'Success!' - if total_pruned > 0 && !@dry_run - log_info "Pruned #{total_pruned} packages across #{repos.size} repos" - log_info "Main repo: version #{version} -> #{version + 1}" - end - ensure - cleanup - end - end - -private - def parse_options - @dry_run = false - OptionParser.new do |opts| - opts.banner = "Usage: #{$0} [options]" - opts.on('--dry-run', 'Do not upload changes') { @dry_run = true } - end.parse! - end - - def create_temp_dirs - @temp_dir = Dir.mktmpdir('apt-prune') - @wrappers_bin_dir = "#{@temp_dir}/wrappers" - @aptly_config_path = "#{@temp_dir}/aptly.conf" - @signing_key_path = "#{@temp_dir}/key.gpg" - @local_state_path = "#{@temp_dir}/state" - @local_state_db_path = "#{@local_state_path}/db" - @local_state_repo_path = "#{@local_state_path}/repo" - @local_state_archive_path = "#{@temp_dir}/state.tar.zst" - - Dir.mkdir(@wrappers_bin_dir) - FileUtils.mkdir_p(@local_state_db_path) - FileUtils.mkdir_p(@local_state_repo_path) - end - - def ensure_gpg_state_isolated - File.open("#{@wrappers_bin_dir}/gpg", 'w:utf-8') do |f| - f.write("#!/bin/sh\n") - f.write( - sprintf("exec %s --homedir %s \"$@\"\n", - Shellwords.escape(find_gpg), - Shellwords.escape(@temp_dir)) - ) - end - File.chmod(0755, "#{@wrappers_bin_dir}/gpg") - end - - def find_gpg - ENV['PATH'].split(':').each do |dir| - next if dir == @wrappers_bin_dir - candidate = "#{dir}/gpg" - return candidate if File.exist?(candidate) - end - abort('GPG not found') - end - - def activate_wrappers_bin_dir - ENV['PATH'] = "#{@wrappers_bin_dir}:#{ENV['PATH']}" - end - - def initialize_locking - @lock = GCloudStorageLock.new(url: lock_url) - end - - def lock_url - "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/locks/apt" - end - - def synchronize(&block) - @lock.synchronize(&block) - end - - def check_lock_health - abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? - end - - def initialize_aptly - File.open(@aptly_config_path, 'w:utf-8') do |f| - f.write(JSON.generate( - rootDir: @local_state_path, - FileSystemPublishEndpoints: { - main: { - rootDir: @local_state_repo_path, - linkMethod: 'symlink', - verifyMethod: 'md5' - } - } - )) - end - end - - def fetch_and_import_signing_key - log_notice 'Fetching and importing signing key' - File.open(@signing_key_path, 'wb') do |f| - f.write(fetch_signing_key) - end - @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) - log_info "Signing key ID: #{@gpg_key_id}" - import_gpg_key(@temp_dir, @signing_key_path) - end - - def latest_production_version_note_url - "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt" - end - - def fetch_state(version) - log_notice "Fetching state version #{version}" - run_bash( - sprintf('gsutil -m cp %s - | zstd -dc | tar -xC %s', - Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/state.tar.zst"), - Shellwords.escape(@local_state_path)), - pipefail: true, - log_invocation: true, - check_error: true, - passthru_output: true - ) - end - - def active_ruby_minor_versions - config[:ruby][:minor_version_packages].map { |p| p[:minor_version] } - end - - def list_aptly_repos - stdout_output, _, _ = run_command_capture_output( - 'aptly', 'repo', 'list', - "-config=#{@aptly_config_path}", '-raw', - log_invocation: true, check_error: true - ) - stdout_output.split("\n").map(&:strip).reject(&:empty?) - end - - def list_packages(distro) - stdout_output, _, _ = run_command_capture_output( - 'aptly', 'repo', 'show', - "-config=#{@aptly_config_path}", - '-with-packages', distro, - log_invocation: false, check_error: true - ) - stdout_output.sub!(/.*^Packages:$/m, '') - stdout_output.split("\n").map(&:strip).reject(&:empty?) - end - - def prune_eol_packages_from_repo(distro, active_minors) - packages = list_packages(distro) - eol_packages = packages.select do |pkg| - if pkg =~ RUBY_PACKAGE_PATTERN - minor = $1 - !active_minors.include?(minor) - else - false - end - end - - if eol_packages.empty? - log_info "[#{distro}] No EOL packages found" - return 0 - end - - log_notice "[#{distro}] Pruning #{eol_packages.size} EOL packages (of #{packages.size} total)" - eol_packages.each { |pkg| log_info " #{YELLOW}PRUNE#{RESET} #{pkg}" } - - if !@dry_run - eol_packages.each_slice(50) do |batch| - query = batch.join(' | ') - run_command( - 'aptly', 'repo', 'remove', - "-config=#{@aptly_config_path}", - distro, query, - log_invocation: false, - check_error: true - ) - end - end - - eol_packages.size - end - - def compact_state - run_command( - 'aptly', 'db', 'cleanup', - "-config=#{@aptly_config_path}", '-verbose', - log_invocation: true, check_error: true, passthru_output: true - ) - end - - def publish_repo(distro) - log_notice "[#{distro}] Publishing" - _, stderr_output, status = run_command_capture_output( - 'aptly', 'publish', 'repo', - '-batch', '-force-overwrite', - "-config=#{@aptly_config_path}", - "-gpg-key=#{@gpg_key_id}", - "-distribution=#{distro}", - "-origin=#{REPO_ORIGIN}", - "-label=#{REPO_LABEL}", - distro, 'filesystem:main:.', - log_invocation: true, check_error: false - ) - if !status.success? - if stderr_output =~ /unable to figure out list of architectures/ - run_command( - 'aptly', 'publish', 'repo', - '-batch', '-force-overwrite', '-architectures=all', - "-config=#{@aptly_config_path}", - "-gpg-key=#{@gpg_key_id}", - "-distribution=#{distro}", - "-origin=#{REPO_ORIGIN}", - "-label=#{REPO_LABEL}", - distro, 'filesystem:main:.', - log_invocation: true, check_error: true - ) - else - abort("ERROR publishing #{distro}: #{stderr_output.chomp}") - end - end - end - - def archive_state - log_notice 'Creating state archive' - run_bash( - sprintf("tar -C %s -cf - . | zstd -T0 > %s", - Shellwords.escape(@local_state_path), - Shellwords.escape(@local_state_archive_path)), - pipefail: true, - log_invocation: true, check_error: true, passthru_output: true - ) - end - - def upload_state(version) - bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] - run_command( - 'gsutil', '-h', 'Cache-Control:public', 'cp', - @local_state_archive_path, - "gs://#{bucket}/versions/#{version}/state.tar.zst", - log_invocation: true, check_error: true, passthru_output: true - ) - end - - def upload_repo(version, old_version) - bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] - - log_notice "Copying repo version #{old_version} to #{version}" - run_command( - 'gsutil', '-m', '-h', 'Cache-Control:public', - 'rsync', '-r', '-d', - "gs://#{bucket}/versions/#{old_version}/public", - "gs://#{bucket}/versions/#{version}/public", - log_invocation: true, check_error: true, passthru_output: true - ) - - log_notice "Uploading pruned repo as version #{version}" - run_command( - 'gsutil', '-m', '-h', 'Cache-Control:public', - 'rsync', '-r', '-d', - @local_state_repo_path, - "gs://#{bucket}/versions/#{version}/public", - log_invocation: true, check_error: true, passthru_output: true - ) - end - - def create_version_note(version) - run_bash( - sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:public cp - %s <<<%s', - Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/version.txt"), - Shellwords.escape(version.to_s)), - log_invocation: true, check_error: true, pipefail: false - ) - end - - def declare_latest_version(version) - log_notice "Activating version #{version}" - run_bash( - sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', - Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt"), - Shellwords.escape(version.to_s)), - log_invocation: true, check_error: true, pipefail: false - ) - end - - def cleanup - log_info "Cleaning up #{@temp_dir}" - FileUtils.remove_entry_secure(@temp_dir) - end -end - -PruneAptPackages.new.main diff --git a/internal-scripts/ci-cd/archive/prune-yum-packages.rb b/internal-scripts/ci-cd/archive/prune-yum-packages.rb deleted file mode 100755 index ed8520fb..00000000 --- a/internal-scripts/ci-cd/archive/prune-yum-packages.rb +++ /dev/null @@ -1,246 +0,0 @@ -#!/usr/bin/env ruby -# frozen_string_literal: true - -# Prune EOL Ruby version packages from the YUM repository. -# -# Removes RPM files for Ruby versions no longer in config.yml from all -# distro/arch directories, regenerates repodata, and re-uploads. -# -# Usage: -# PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ -# ./internal-scripts/ci-cd/archive/prune-yum-packages.rb [--dry-run] - -require_relative '../../../lib/gcloud_storage_lock' -require_relative '../../../lib/ci_workflow_support' -require_relative '../../../lib/shell_scripting_support' -require_relative '../../../lib/publishing_support' -require 'shellwords' -require 'tmpdir' -require 'fileutils' -require 'optparse' - -class PruneYumPackages - # Matches: fullstaq-ruby-3.1.7-jemalloc-rev1-centos8.x86_64.rpm - RUBY_RPM_PATTERN = /\Afullstaq-ruby-(\d+\.\d+)/ - - include CiWorkflowSupport - include ShellScriptingSupport - include PublishingSupport - - def main - parse_options - require_envvar 'PRODUCTION_REPO_BUCKET_NAME' - - print_header 'Initializing' - load_config - create_temp_dirs - initialize_locking - pull_utility_image_if_not_exists - fetch_and_import_signing_key - - total_pruned = 0 - version = nil - - begin - synchronize do - print_header 'Downloading repository' - version = get_latest_production_repo_version - if version == 0 - abort 'ERROR: No production repository exists yet' - end - fetch_repo(version) - - print_header 'Identifying EOL Ruby versions' - active_minors = active_ruby_minor_versions - log_info "Active Ruby minor versions: #{active_minors.join(', ')}" - - print_header 'Scanning and pruning packages' - affected_dirs = [] - - Dir.glob("#{@local_repo_path}/*/*").each do |arch_dir| - next unless File.directory?(arch_dir) - distro = File.basename(File.dirname(arch_dir)) - arch = File.basename(arch_dir) - - rpms = Dir.glob("#{arch_dir}/fullstaq-ruby-*.rpm") - eol_rpms = rpms.select do |rpm| - basename = File.basename(rpm) - if basename =~ RUBY_RPM_PATTERN - !active_minors.include?($1) - else - false - end - end - - next if eol_rpms.empty? - - log_notice "[#{distro}/#{arch}] Pruning #{eol_rpms.size} EOL packages (of #{rpms.size} total)" - eol_rpms.each { |rpm| log_info " #{YELLOW}PRUNE#{RESET} #{File.basename(rpm)}" } - total_pruned += eol_rpms.size - - if !@dry_run - eol_rpms.each { |rpm| File.delete(rpm) } - affected_dirs << arch_dir - end - end - - if total_pruned == 0 - log_notice 'No EOL Ruby packages found to prune' - return - end - - log_notice "Total packages pruned: #{total_pruned}" - - if @dry_run - log_notice 'DRY RUN — not uploading changes' - return - end - - print_header 'Regenerating repo metadata' - affected_dirs.each do |dir| - invoke_createrepo(dir) - sign_repo(dir) - end - check_lock_health - - print_header 'Uploading pruned repository' - upload_repo(version + 1, version) - create_version_note(version + 1) - declare_latest_version(version + 1) - end - - print_header 'Success!' - if total_pruned > 0 && !@dry_run - log_info "Pruned #{total_pruned} packages" - log_info "Main repo: version #{version} -> #{version + 1}" - end - ensure - cleanup - end - end - -private - def parse_options - @dry_run = false - OptionParser.new do |opts| - opts.banner = "Usage: #{$0} [options]" - opts.on('--dry-run', 'Do not upload changes') { @dry_run = true } - end.parse! - end - - def create_temp_dirs - @temp_dir = Dir.mktmpdir('yum-prune') - @local_repo_path = "#{@temp_dir}/repo" - @signing_key_path = "#{@temp_dir}/key.gpg" - Dir.mkdir(@local_repo_path) - end - - def initialize_locking - @lock = GCloudStorageLock.new(url: lock_url) - end - - def lock_url - "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/locks/yum" - end - - def synchronize(&block) - @lock.synchronize(&block) - end - - def check_lock_health - abort 'ERROR: lock is unhealthy. Aborting operation' if !@lock.healthy? - end - - def fetch_and_import_signing_key - log_notice 'Fetching and importing signing key' - File.open(@signing_key_path, 'wb') do |f| - f.write(fetch_signing_key) - end - @gpg_key_id = infer_gpg_key_id(@temp_dir, @signing_key_path) - import_gpg_key(@temp_dir, @signing_key_path) - end - - def latest_production_version_note_url - "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt" - end - - def fetch_repo(version) - run_command( - 'gsutil', '-m', 'rsync', '-r', - "gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/public", - @local_repo_path, - log_invocation: true, check_error: true, passthru_output: true - ) - end - - def active_ruby_minor_versions - config[:ruby][:minor_version_packages].map { |p| p[:minor_version] } - end - - def invoke_createrepo(dir) - run_command( - 'docker', 'run', '--rm', - '-v', "#{dir}:/input:delegated", - '--user', "#{Process.uid}:#{Process.gid}", - utility_image_name, - 'createrepo_c', '--update', '/input', - log_invocation: true, check_error: true - ) - end - - def sign_repo(path) - run_command( - 'gpg', "--homedir=#{@temp_dir}", "--local-user=#{@gpg_key_id}", - '--batch', '--yes', '--detach-sign', '--armor', - "#{path}/repodata/repomd.xml", - log_invocation: true, check_error: true - ) - end - - def upload_repo(version, old_version) - bucket = ENV['PRODUCTION_REPO_BUCKET_NAME'] - - log_notice "Copying repo version #{old_version} to #{version}" - run_command( - 'gsutil', '-m', '-h', 'Cache-Control:public', - 'rsync', '-r', '-d', - "gs://#{bucket}/versions/#{old_version}/public", - "gs://#{bucket}/versions/#{version}/public", - log_invocation: true, check_error: true, passthru_output: true - ) - - log_notice "Uploading pruned repo as version #{version}" - run_command( - 'gsutil', '-m', '-h', 'Cache-Control:public', - 'rsync', '-r', '-d', - @local_repo_path, - "gs://#{bucket}/versions/#{version}/public", - log_invocation: true, check_error: true, passthru_output: true - ) - end - - def create_version_note(version) - run_bash( - sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:public cp - %s <<<%s', - Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/#{version}/version.txt"), - Shellwords.escape(version.to_s)), - log_invocation: true, check_error: true, pipefail: false - ) - end - - def declare_latest_version(version) - run_bash( - sprintf('gsutil -q -h Content-Type:text/plain -h Cache-Control:no-store cp - %s <<<%s', - Shellwords.escape("gs://#{ENV['PRODUCTION_REPO_BUCKET_NAME']}/versions/latest_version.txt"), - Shellwords.escape(version.to_s)), - log_invocation: true, check_error: true, pipefail: false - ) - end - - def cleanup - log_info "Cleaning up #{@temp_dir}" - FileUtils.remove_entry_secure(@temp_dir) - end -end - -PruneYumPackages.new.main From f777caf671e13c40e0942fa28a3d9efe1dea1145 Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 21:57:54 -0500 Subject: [PATCH 09/12] docs: rewrite EOL archiving handbook for archive-instead-of-prune Documents the archive-*.rb scripts, the disjoint live/archive rule, the user-facing release-notes template, and the test harness. --- dev-handbook/apt-yum-repo.md | 6 +- dev-handbook/archiving-eol-packages.md | 262 +++++++++++++------------ 2 files changed, 136 insertions(+), 132 deletions(-) diff --git a/dev-handbook/apt-yum-repo.md b/dev-handbook/apt-yum-repo.md index f19a2013..c513ea23 100644 --- a/dev-handbook/apt-yum-repo.md +++ b/dev-handbook/apt-yum-repo.md @@ -52,11 +52,11 @@ There are five buckets: - `fsruby-server-edition-apt-repo` stores the production APT repository. - `fsruby-server-edition-yum-repo` stores the production YUM repository. - - `fsruby-server-edition-apt-archive-repo` stores packages for EOL distributions (APT). - - `fsruby-server-edition-yum-archive-repo` stores packages for EOL distributions (YUM). + - `fsruby-server-edition-apt-archive-repo` stores APT packages whose distribution or Ruby version is EOL. + - `fsruby-server-edition-yum-archive-repo` stores YUM packages whose distribution or Ruby version is EOL. - `fsruby-server-edition-ci-artifacts` stores the temporary repositories created during CI runs. -The archive buckets are only updated during [EOL migration](archiving-eol-packages.md) — CI never writes to them. Each migration creates a new version that merges newly-archived distros with the existing archive contents. They are served at `apt-archive.fullstaqruby.org` and `yum-archive.fullstaqruby.org`. +The production and archive buckets never contain the same package. The archive buckets are only updated by [archiving EOL packages](archiving-eol-packages.md) — CI never writes to them. Each archival run moves packages from the production bucket into the archive bucket, creating a new version of both. The archive buckets are served at `apt-archive.fullstaqruby.org` and `yum-archive.fullstaqruby.org`. We don't let users use the production bucket URLs directly. Instead, we let users use `https://apt.fullstaqruby.org` and `https://yum.fullstaqruby.org`. These domains redirect to the appropriate bucket URLs. We do this so that we avoid strongly coupling users with Google Cloud Storage. If in the future we want to move off Google Cloud, we can do so without breaking users' URLs. diff --git a/dev-handbook/archiving-eol-packages.md b/dev-handbook/archiving-eol-packages.md index abd74333..62aacd23 100644 --- a/dev-handbook/archiving-eol-packages.md +++ b/dev-handbook/archiving-eol-packages.md @@ -1,231 +1,235 @@ # Archiving EOL packages -This document describes how to archive packages for end-of-life (EOL) distributions and prune EOL Ruby versions from the repositories. This is a routine maintenance task that frees CI disk space and keeps the repository lean. +This document describes how to move packages for end-of-life (EOL) distributions and EOL Ruby versions out of the live repositories and into the archive repositories. This is a routine maintenance task that frees CI disk space and keeps the live repositories lean, while keeping EOL packages installable. ## Background The CI publish step downloads the full Aptly state archive (`state.tar.zst`) from Google Cloud Storage on every run. This archive grows with every distribution and Ruby version ever published. When distributions or Ruby versions reach EOL, their packages remain in the state archive indefinitely, consuming disk space on GitHub Actions runners. -To address this, we maintain **archive repositories** alongside the main repositories: +To address this, we maintain **archive repositories** alongside the live repositories: -| Repository | Bucket | Domain | Purpose | -|------------|--------|--------|---------| -| APT (main) | `fsruby-server-edition-apt-repo` | `apt.fullstaqruby.org` | Current, supported packages | -| APT (archive) | `fsruby-server-edition-apt-archive-repo` | `apt-archive.fullstaqruby.org` | Frozen packages for EOL distributions | -| YUM (main) | `fsruby-server-edition-yum-repo` | `yum.fullstaqruby.org` | Current, supported packages | -| YUM (archive) | `fsruby-server-edition-yum-archive-repo` | `yum-archive.fullstaqruby.org` | Frozen packages for EOL distributions | +| Repository | Bucket | Domain | Contents | +|------------|--------|--------|----------| +| APT (live) | `fsruby-server-edition-apt-repo` | `apt.fullstaqruby.org` | Packages whose distribution *and* Ruby version are both supported | +| APT (archive) | `fsruby-server-edition-apt-archive-repo` | `apt-archive.fullstaqruby.org` | Packages whose distribution *or* Ruby version is EOL | +| YUM (live) | `fsruby-server-edition-yum-repo` | `yum.fullstaqruby.org` | Packages whose distribution *and* Ruby version are both supported | +| YUM (archive) | `fsruby-server-edition-yum-archive-repo` | `yum-archive.fullstaqruby.org` | Packages whose distribution *or* Ruby version is EOL | -Archive repositories are static — CI never writes to them. They use the same versioned bucket structure as the main repos. Each migration creates a new version that merges newly-archived distros with the existing archive contents, so the archive grows incrementally over time. +The live and archive repositories never contain the same package. Packages are moved, never deleted, so EOL packages remain installable from the archive. + +The archive only grows: packages are added to it as distributions and Ruby versions reach EOL. It uses the same versioned bucket structure as the live repositories, so each archival run creates a new archive version. + +CI never writes to the archive buckets. Packages only enter the archive through the archive scripts described below, run by hand. This is enforced in IAM (see the infra repo's `terraform/repo_buckets.tf`). This pattern follows the precedent set by [PostgreSQL](https://apt-archive.postgresql.org/) (`apt-archive.postgresql.org`) and [HashiCorp](https://www.hashicorp.com/en/blog/announcing-the-linux-package-archive-site) (`archive.releases.hashicorp.com`). -## Two types of cleanup +## What gets archived -There are two independent axes of cleanup, each with its own script: +Each script decides per package, based on the current `config.yml`: -### 1. Distro archival — moving entire EOL distribution repos + * **EOL distribution** — the distribution has a repository in the live repo, but is no longer in `config.yml`'s distributions. *All* its packages are archived, including `fullstaq-ruby-common` and `fullstaq-rbenv`, and the distribution is removed from the live repo. + * **EOL Ruby version** — on a supported distribution, any `fullstaq-ruby-X.Y*` package where `X.Y` is not in `minor_version_packages`. This includes variants (`-jemalloc`, `-malloctrim`) and tiny-version packages (`fullstaq-ruby-X.Y.Z`). -When a Linux distribution reaches EOL, we stop building packages for it and move its existing packages to the archive. Users on EOL distributions can still install packages by pointing at the archive repo. +Everything else stays in the live repo. In particular: -**Scripts:** - * `internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb` - * `internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb` + * `fullstaq-ruby-common` and `fullstaq-rbenv` stay on supported distributions. + * A tiny-version package stays as long as its minor version is active, even if it has been removed from `tiny_version_packages`. -### 2. Package pruning — removing EOL Ruby version packages +The scripts are: -When a Ruby version reaches EOL, we stop building it (by removing it from `config.yml`), but its packages persist inside every distro's repository. Pruning removes these stale packages from the still-supported distro repos to reduce state size. + * `internal-scripts/ci-cd/archive/archive-apt-packages.rb` + * `internal-scripts/ci-cd/archive/archive-yum-packages.rb` -**Scripts:** - * `internal-scripts/ci-cd/archive/prune-apt-packages.rb` - * `internal-scripts/ci-cd/archive/prune-yum-packages.rb` +Both handle EOL distributions and EOL Ruby versions in the same run, so there is no ordering to get right. -## Removing an EOL distribution +## Procedure ### Step 1: Remove from the build system +For an EOL distribution: + 1. Edit `config.yml` and remove the distribution from the `distributions` list (or add it to an exclusion). 2. Delete the `environments//` directory. - 3. Regenerate CI/CD workflows: + +For an EOL Ruby version: + + 1. Edit `config.yml` and remove the version from `minor_version_packages` (and `tiny_version_packages`). + +Then: + + 1. Regenerate CI/CD workflows: ~~~bash ./internal-scripts/generate-ci-cd-yaml.rb ~~~ - 4. Commit and merge these changes. + 2. Commit and merge these changes. -### Step 2: Migrate packages to the archive +The scripts read `config.yml` from your working copy, so run them from an up-to-date checkout of `main`. + +### Step 2: Announce the change + +Users who install an EOL Ruby version, or any package on an EOL distribution, must add the archive repository after this step. Add an entry to the release notes of the next release. For example: + +> **Packages for EOL Ruby versions and distributions have moved to the archive repositories.** Packages for Ruby X.Y and for Distro N are no longer available from `apt.fullstaqruby.org` / `yum.fullstaqruby.org`. They remain installable from `apt-archive.fullstaqruby.org` / `yum-archive.fullstaqruby.org`, which use the same signing key. To keep installing them, add the archive repository alongside the regular one: +> +> APT: +> +> deb https://apt-archive.fullstaqruby.org main +> +> YUM: +> +> [fullstaq-ruby-archive] +> name=fullstaq-ruby-archive +> baseurl=https://yum-archive.fullstaqruby.org//$basearch +> gpgcheck=0 +> repo_gpgcheck=1 +> enabled=1 +> gpgkey=https://raw.githubusercontent.com/fullstaq-ruby/server-edition/main/fullstaq-ruby.asc +> sslverify=1 + +Replace `X.Y`, `Distro N` and `` with what's being archived. Use the same `Signed-By` option in the APT line as the user's existing Fullstaq Ruby line, if it has one. + +### Step 3: Run the archive scripts **Prerequisites:** - * `gcloud` CLI authenticated with write access to the GCS buckets + + * `gcloud` CLI authenticated with write access to the live and archive buckets * `az` CLI authenticated with access to the `fsruby2infraowners` Key Vault (for the GPG signing key) - * `aptly`, `zstd`, and `gpg` installed locally - * Docker running (for `createrepo_c` in YUM migration) + * `aptly`, `zstd` and `gpg` installed locally (for APT) + * Docker running (for `createrepo_c`, for YUM) + * Enough free disk space for the full live APT state (roughly 30 GB at the time of writing) -**Dry run first** to verify which distros will be archived: +**Dry run first.** A dry run makes every change locally, including the checks described in [How the scripts work](#how-the-scripts-work), and prints the list of packages it would move, but uploads nothing: ~~~bash PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-archive-repo \ -./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb --dry-run +./internal-scripts/ci-cd/archive/archive-apt-packages.rb --dry-run ~~~ -The script auto-detects EOL distros by comparing `aptly repo list` output against the DEB distributions defined in `config.yml`. You can also specify distros explicitly (DEB-only — RPM distros belong to the YUM script): - -~~~bash -./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb --dry-run --distros debian-10,ubuntu-20.04 -~~~ - -**Execute the migration** (removes `--dry-run`): +Review the list. Then run for real: ~~~bash PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-apt-archive-repo \ -./internal-scripts/ci-cd/archive/migrate-apt-to-archive.rb +./internal-scripts/ci-cd/archive/archive-apt-packages.rb ~~~ -**Repeat for YUM:** +Repeat for YUM: ~~~bash PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-archive-repo \ -./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb --dry-run +./internal-scripts/ci-cd/archive/archive-yum-packages.rb --dry-run PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-archive-repo \ -./internal-scripts/ci-cd/archive/migrate-yum-to-archive.rb +./internal-scripts/ci-cd/archive/archive-yum-packages.rb ~~~ -### Step 3: Restart the web server +The scripts hold the live repo's lock (`locks/apt` or `locks/yum`) for the whole run, so a CI publish can't run at the same time. + +### Step 4: Restart the web server -Caddy only reads repo version numbers at startup, and nothing restarts it after a manual migration. Until it restarts, archive requests keep redirecting to the old version (`versions/0/` on first migration) and return 404. +Caddy only reads repo version numbers at startup, and nothing restarts it after a manual archival run. Until it restarts, the live domains keep serving the pre-archival version and the archive domains keep serving the previous archive version (`versions/0/` on the first run, which returns 404). The `/admin/restart_web_server` endpoint only accepts OIDC tokens from GitHub-hosted runners in this repo's `deploy` environment, so it can't be called by hand. Restart Caddy over SSH on the backend server instead: ~~~bash sudo systemctl restart caddy -# Confirm the archive versions are no longer 0 +# Confirm the new versions are loaded sudo cat /etc/caddy/env-repo-versions ~~~ -### Step 4: Verify +### Step 5: Verify ~~~bash -# Archive should list the archived distros -curl -s https://apt-archive.fullstaqruby.org/dists/ +# Archive should serve the archived distributions +curl -fsS https://apt-archive.fullstaqruby.org/dists//Release +curl -fsS https://yum-archive.fullstaqruby.org//x86_64/repodata/repomd.xml -# Main repo should only contain supported distros -curl -s https://apt.fullstaqruby.org/dists/ +# Live repo should no longer list EOL packages +curl -fsS https://apt.fullstaqruby.org/dists//main/binary-amd64/Packages | grep '^Package: fullstaq-ruby-X.Y' -# Verify state archive size decreased +# State archive size should have decreased gsutil ls -l gs://fsruby-server-edition-apt-repo/versions/*/state.tar.zst | tail -5 ~~~ -## Pruning EOL Ruby versions +## Rollback -After removing a Ruby version from `config.yml`, its packages persist in the Aptly state. Run the pruning scripts to remove them. +Each run creates a new version of both repositories. Old versions are never modified, so rolling back means pointing `latest_version.txt` back at the previous version and restarting Caddy. -**Dry run:** +**Revert the live repo:** ~~~bash -PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ -./internal-scripts/ci-cd/archive/prune-apt-packages.rb --dry-run -~~~ - -The script compares packages in the Aptly state against `minor_version_packages` in `config.yml` and identifies any `fullstaq-ruby-X.Y*` packages where `X.Y` is not an active minor version. - -**Execute:** +gsutil cat gs://fsruby-server-edition-apt-repo/versions/latest_version.txt -~~~bash -PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ -./internal-scripts/ci-cd/archive/prune-apt-packages.rb +echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-repo/versions/latest_version.txt ~~~ -**Repeat for YUM:** +**Revert the archive:** ~~~bash -PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ -./internal-scripts/ci-cd/archive/prune-yum-packages.rb --dry-run +gsutil cat gs://fsruby-server-edition-apt-archive-repo/versions/latest_version.txt -PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-yum-repo \ -./internal-scripts/ci-cd/archive/prune-yum-packages.rb +echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-archive-repo/versions/latest_version.txt ~~~ -Restart the web server after pruning (same as above). - -## Execution order - -When performing both distro archival and package pruning in the same session, always run distro archival **first**. This ensures the archive captures the full historical packages for EOL distros before any pruning happens. - - 1. `migrate-apt-to-archive.rb` - 2. `prune-apt-packages.rb` - 3. `migrate-yum-to-archive.rb` - 4. `prune-yum-packages.rb` - 5. Restart web server - -## Rollback - -The versioned bucket structure makes rollback straightforward. Each migration creates a new version — the old version is never modified. +Revert the live repo, not only the archive: reverting only the archive makes the moved packages unavailable from both. -**Revert the main APT repo to a previous version:** +The same commands apply to the YUM buckets. -~~~bash -# Find the pre-migration version number -gsutil cat gs://fsruby-server-edition-apt-repo/versions/latest_version.txt +## How the scripts work -# Point back to the old version -echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-repo/versions/latest_version.txt -~~~ +Both scripts follow the same sequence: -**Revert the archive to a previous version:** + 1. Take the live repo's lock. + 2. Download the latest live repo version. + 3. Decide which packages to move (see [What gets archived](#what-gets-archived)). + 4. Download the latest archive version, if any. + 5. Add the packages to the local archive copy and regenerate its signed metadata. + 6. **Verify** that the archive's published metadata lists every package being moved. The script aborts here, before touching the live repo, if anything is missing. + 7. Remove the packages from the local live copy and regenerate its signed metadata. + 8. Verify that the live repo's published metadata no longer lists any of them. + 9. Upload the archive as version M+1 and activate it. + 10. Upload the live repo as version N+1 and activate it. -~~~bash -gsutil cat gs://fsruby-server-edition-apt-archive-repo/versions/latest_version.txt +The archive is activated before the live repo, so the moved packages are always available from at least one of them. -echo -n "OLD_VERSION" | gsutil -h Content-Type:text/plain -h Cache-Control:no-store cp - gs://fsruby-server-edition-apt-archive-repo/versions/latest_version.txt -~~~ +**Reruns are safe.** If a run fails after step 9, rerun the script: -**Delete all archive contents** (if no users depend on it yet): + * packages already in the archive are skipped or re-added without change; + * the live repo is then trimmed as normal. -~~~bash -gsutil -m rm -r gs://fsruby-server-edition-apt-archive-repo/versions/ -~~~ +A run with nothing to archive exits without uploading anything. -## How the migration scripts work +### APT specifics -### APT migration (`migrate-apt-to-archive.rb`) + * Packages are added to the archive's Aptly instance individually, with `aptly repo add -force-replace` using the `.deb` files from the live state's pool. Only the moved packages' files end up in the archive's pool. + * EOL Ruby packages are removed from live with `aptly repo remove`. EOL distributions are unpublished and dropped with `aptly publish drop` and `aptly repo drop`. + * Both instances run `aptly db cleanup` after republishing, so unreferenced files leave the pool. + * Each distribution in the archive is its own Aptly repo and publication, named after the distribution, like in the live repo. - 1. Downloads the current Aptly state archive from the main bucket. - 2. Identifies EOL distros (published in Aptly but not in `config.yml`). - 3. Fetches the existing archive state (if any) so new distros are merged into it. - 4. Creates or extends the archive Aptly instance with EOL distro data and package pool. - 5. Publishes all archive distros (existing + newly archived). - 6. Drops the EOL distro repos from the main Aptly database. - 7. Runs `aptly db cleanup` to compact the database and reclaim pool space. - 8. Re-publishes remaining distros in the main repo. - 9. Uploads the merged archive as a new archive version (N+1). - 10. Uploads the trimmed state as a new version of the main repo. +### YUM specifics -### YUM migration (`migrate-yum-to-archive.rb`) + * RPM files are copied into the archive's `//` directory individually. Files already present are skipped. + * `createrepo_c` (in the utility Docker image) regenerates `repodata/` for every directory that changed, in both repositories, and `repomd.xml` is re-signed with the signing key. + * EOL distribution directories are deleted from the live copy. - 1. Downloads the current YUM repo from the main bucket via `gsutil rsync`. - 2. Identifies EOL distro directories. - 3. Fetches the existing archive repo (if any) so new distros are merged into it. - 4. Copies EOL distro directories into the local archive copy. - 5. Uploads the merged archive as a new archive version (N+1). - 6. Removes EOL distro directories from the main local copy. - 7. Uploads the trimmed repo as a new version of the main bucket. +## Testing -### APT pruning (`prune-apt-packages.rb`) +`internal-scripts/ci-cd/archive/test/run-tests.sh` runs both scripts against a fake Google Cloud Storage inside a Docker container with the same Aptly, `createrepo_c` and GPG versions CI uses. It seeds live repositories with a supported and an EOL distribution and an active and an EOL Ruby version, then checks that: - 1. Downloads the Aptly state. - 2. Scans all packages across all distro repos, matching `fullstaq-ruby-X.Y*` against active minor versions. - 3. Removes EOL Ruby packages using `aptly repo remove`. - 4. Compacts, re-publishes, and uploads. + * a dry run uploads nothing; + * the right packages end up in each repository, and the two never overlap; + * all metadata is signed; + * a second run changes nothing; + * a rerun after a failure between the archive and live uploads completes correctly. -### YUM pruning (`prune-yum-packages.rb`) +It needs only Docker. It is not run by CI. Run it after changing either script: - 1. Downloads the YUM repo. - 2. Deletes RPM files matching EOL Ruby versions from the filesystem. - 3. Regenerates `repodata/` with `createrepo_c` and re-signs. - 4. Uploads as a new version. +~~~bash +./internal-scripts/ci-cd/archive/test/run-tests.sh +~~~ From b7f49ea5fecfaaa548d27381a8100ec7b83ece41 Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 22:43:04 -0500 Subject: [PATCH 10/12] fix: treat Ruby minors with tiny-version packages as active when archiving A minor removed from minor_version_packages but still in tiny_version_packages is still built by CI. Archiving its packages would let CI re-publish them to live, so live and archive would overlap. --- internal-scripts/ci-cd/archive/archive-apt-packages.rb | 7 ++++++- internal-scripts/ci-cd/archive/archive-yum-packages.rb | 7 ++++++- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/internal-scripts/ci-cd/archive/archive-apt-packages.rb b/internal-scripts/ci-cd/archive/archive-apt-packages.rb index c6519520..3fb94709 100755 --- a/internal-scripts/ci-cd/archive/archive-apt-packages.rb +++ b/internal-scripts/ci-cd/archive/archive-apt-packages.rb @@ -260,8 +260,13 @@ def supported_distros distributions.find_all { |d| d[:package_format] == :DEB }.map { |d| d[:name] } end + # A minor version is active while CI still builds any package for it: + # its minor-version package, or one of its tiny-version packages. + # Archiving a package CI still builds would put it in both repositories. def active_ruby_minor_versions - config[:ruby][:minor_version_packages].map { |p| p[:minor_version] } + minors = (config[:ruby][:minor_version_packages] || []).map { |p| p[:minor_version] } + tinys = (config[:ruby][:tiny_version_packages] || []).map { |p| p[:full_version].split('.')[0..1].join('.') } + (minors + tinys).uniq end def eol_ruby_package?(package_key, active_minors) diff --git a/internal-scripts/ci-cd/archive/archive-yum-packages.rb b/internal-scripts/ci-cd/archive/archive-yum-packages.rb index a1a8913b..bb43bdd5 100755 --- a/internal-scripts/ci-cd/archive/archive-yum-packages.rb +++ b/internal-scripts/ci-cd/archive/archive-yum-packages.rb @@ -194,8 +194,13 @@ def supported_distros distributions.find_all { |d| d[:package_format] == :RPM }.map { |d| d[:name] } end + # A minor version is active while CI still builds any package for it: + # its minor-version package, or one of its tiny-version packages. + # Archiving a package CI still builds would put it in both repositories. def active_ruby_minor_versions - config[:ruby][:minor_version_packages].map { |p| p[:minor_version] } + minors = (config[:ruby][:minor_version_packages] || []).map { |p| p[:minor_version] } + tinys = (config[:ruby][:tiny_version_packages] || []).map { |p| p[:full_version].split('.')[0..1].join('.') } + (minors + tinys).uniq end def eol_ruby_rpm?(filename, active_minors) From 89283bee9e65938a0bd78b400f6dbd116c368b64 Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 22:43:18 -0500 Subject: [PATCH 11/12] fix: archive state before publishing and republish every APT distro Tarring after 'aptly publish' left publications in the uploaded state, which makes publish-debs.rb fail with 'prefix/distribution already used', and left repo/ symlinks into a deleted temp dir that broke the next archival round. Republishing only changed distros made 'gsutil rsync -d' delete untouched distros from live. Now matches publish-debs.rb: tar first, then publish every distro. Also: import the signing key inside begin/ensure so it is always cleaned up, put the temp dir under /tmp (macOS $TMPDIR is too long for gpg-agent's socket), check lock health before activating each version, and skip tarballs on dry runs. The harness now seeds state the way publish-debs.rb does, includes a distro with nothing to archive, runs a second archival round, and runs publish-debs.rb afterwards. The previous scripts fail 5 of its checks. --- .../ci-cd/archive/archive-apt-packages.rb | 63 +-- .../ci-cd/archive/archive-yum-packages.rb | 16 +- .../ci-cd/archive/test/fake-bin/az | 6 +- .../ci-cd/archive/test/fake-bin/docker | 19 +- .../ci-cd/archive/test/run-tests.sh | 437 +++++++++++------- 5 files changed, 317 insertions(+), 224 deletions(-) diff --git a/internal-scripts/ci-cd/archive/archive-apt-packages.rb b/internal-scripts/ci-cd/archive/archive-apt-packages.rb index 3fb94709..44539fb1 100755 --- a/internal-scripts/ci-cd/archive/archive-apt-packages.rb +++ b/internal-scripts/ci-cd/archive/archive-apt-packages.rb @@ -47,14 +47,14 @@ def main print_header 'Initializing' load_config create_temp_dirs - ensure_gpg_state_isolated - activate_wrappers_bin_dir - initialize_locking - initialize_aptly(@live_aptly_config_path, @live_state_path) - initialize_aptly(@archive_aptly_config_path, @archive_state_path) - fetch_and_import_signing_key - begin + ensure_gpg_state_isolated + activate_wrappers_bin_dir + initialize_locking + initialize_aptly(@live_aptly_config_path, @live_state_path) + initialize_aptly(@archive_aptly_config_path, @archive_state_path) + fetch_and_import_signing_key + synchronize do print_header 'Fetching live repository state' @live_version = get_latest_version(live_bucket) @@ -77,20 +77,32 @@ def main log_notice 'No archive exists yet, creating a new one' end - print_header 'Adding packages to archive repository' + print_header 'Moving packages' add_to_archive - verify_archive + remove_from_live + compact_aptly_db(@archive_aptly_config_path) + compact_aptly_db(@live_aptly_config_path) + + # Like publish-debs.rb, the state is archived *before* publishing, + # so that it contains no publications and an empty repo/ directory. + # publish-debs.rb creates publications without dropping them first, + # which fails if the state already has one. + if !@dry_run + print_header 'Creating state archives' + create_state_tarball(@archive_state_path, @archive_state_tarball_path) + create_state_tarball(@live_state_path, @live_state_tarball_path) + end check_lock_health - print_header 'Removing packages from live repository' - remove_from_live + # Every distribution is published, not only the ones that changed: + # the upload replaces the entire public tree with repo/. + print_header 'Publishing repositories' + publish_all(@archive_aptly_config_path) + publish_all(@live_aptly_config_path) + verify_archive verify_live check_lock_health - print_header 'Creating state archives' - create_state_tarball(@archive_state_path, @archive_state_tarball_path) - create_state_tarball(@live_state_path, @live_state_tarball_path) - if @dry_run log_notice 'Dry run: not uploading changes' return @@ -100,13 +112,10 @@ def main # repository stops serving these packages. print_header 'Uploading archive repository' upload_version(archive_bucket, @archive_version, @archive_state_path, @archive_state_tarball_path) - check_lock_health print_header 'Uploading live repository' upload_version(live_bucket, @live_version, @live_state_path, @live_state_tarball_path) - end - if @moves && !@moves.empty? && !@dry_run print_header 'Success!' print_summary end @@ -134,7 +143,9 @@ def archive_bucket def create_temp_dirs log_notice 'Creating temporary directories' - @temp_dir = Dir.mktmpdir('archive-apt-packages') + # Under /tmp, not $TMPDIR: on macOS $TMPDIR is too long for gpg-agent's + # socket path, and the GPG home directory lives here. + @temp_dir = Dir.mktmpdir('archive-apt', '/tmp') @wrappers_bin_dir = "#{@temp_dir}/wrappers" @signing_key_path = "#{@temp_dir}/key.gpg" @@ -330,11 +341,7 @@ def add_to_archive check_error: true ) end - - publish_aptly_repo(@archive_aptly_config_path, distro) end - - compact_aptly_db(@archive_aptly_config_path) end # Paths of the .deb files in the live state's pool for the given package keys. @@ -400,12 +407,15 @@ def remove_from_live check_error: true ) end - publish_aptly_repo(@live_aptly_config_path, distro) end end + end - # Must run after republishing: published files keep pool files referenced. - compact_aptly_db(@live_aptly_config_path) + def publish_all(config_path) + list_aptly_repos(config_path).each do |distro| + log_notice "[#{distro}] Publishing #{config_path}" + publish_aptly_repo(config_path, distro) + end end def verify_live @@ -578,6 +588,7 @@ def upload_version(bucket, old_version, state_path, tarball_path) ) write_gcs_text("#{version_url}/version.txt", new_version, 'public') + check_lock_health log_notice "Activating #{bucket} version #{new_version}" write_gcs_text("gs://#{bucket}/versions/latest_version.txt", new_version, 'no-store') end diff --git a/internal-scripts/ci-cd/archive/archive-yum-packages.rb b/internal-scripts/ci-cd/archive/archive-yum-packages.rb index bb43bdd5..41adf6e8 100755 --- a/internal-scripts/ci-cd/archive/archive-yum-packages.rb +++ b/internal-scripts/ci-cd/archive/archive-yum-packages.rb @@ -46,11 +46,11 @@ def main print_header 'Initializing' load_config create_temp_dirs - pull_utility_image_if_not_exists - initialize_locking - fetch_and_import_signing_key - begin + pull_utility_image_if_not_exists + initialize_locking + fetch_and_import_signing_key + synchronize do print_header 'Fetching live repository' @live_version = get_latest_version(live_bucket) @@ -92,13 +92,10 @@ def main # repository stops serving these packages. print_header 'Uploading archive repository' upload_version(archive_bucket, @archive_version, @archive_repo_path) - check_lock_health print_header 'Uploading live repository' upload_version(live_bucket, @live_version, @live_repo_path) - end - if @moves && !@moves.empty? && !@dry_run print_header 'Success!' print_summary end @@ -126,7 +123,9 @@ def archive_bucket def create_temp_dirs log_notice 'Creating temporary directories' - @temp_dir = Dir.mktmpdir('archive-yum-packages') + # Under /tmp, not $TMPDIR: on macOS $TMPDIR is too long for gpg-agent's + # socket path, and the GPG home directory lives here. + @temp_dir = Dir.mktmpdir('archive-yum', '/tmp') @signing_key_path = "#{@temp_dir}/key.gpg" @live_repo_path = "#{@temp_dir}/live-repo" @archive_repo_path = "#{@temp_dir}/archive-repo" @@ -380,6 +379,7 @@ def upload_version(bucket, old_version, repo_path) ) write_gcs_text("#{version_url}/version.txt", new_version, 'public') + check_lock_health log_notice "Activating #{bucket} version #{new_version}" write_gcs_text("gs://#{bucket}/versions/latest_version.txt", new_version, 'no-store') end diff --git a/internal-scripts/ci-cd/archive/test/fake-bin/az b/internal-scripts/ci-cd/archive/test/fake-bin/az index 70280db7..e2e27816 100755 --- a/internal-scripts/ci-cd/archive/test/fake-bin/az +++ b/internal-scripts/ci-cd/archive/test/fake-bin/az @@ -1,4 +1,4 @@ -#!/bin/sh +#!/bin/bash # Fake 'az keyvault secret show' for tests: prints the test signing key. -set -eu -cat "$FAKE_SIGNING_KEY_PATH" +set -euo pipefail +cat "${FAKE_SIGNING_KEY_PATH}" diff --git a/internal-scripts/ci-cd/archive/test/fake-bin/docker b/internal-scripts/ci-cd/archive/test/fake-bin/docker index af587b11..0c40580e 100755 --- a/internal-scripts/ci-cd/archive/test/fake-bin/docker +++ b/internal-scripts/ci-cd/archive/test/fake-bin/docker @@ -1,8 +1,8 @@ -#!/bin/sh +#!/bin/bash # Fake docker for tests. 'docker inspect' succeeds, so the utility image # is never pulled. 'docker run ... -v DIR:/input... IMAGE createrepo_c ARGS' # runs createrepo_c locally against DIR. -set -eu +set -euo pipefail case "$1" in inspect) @@ -11,26 +11,25 @@ case "$1" in run) shift input_dir= - while [ $# -gt 0 ]; do + while [[ $# -gt 0 ]]; do case "$1" in --rm) shift ;; --user) shift 2 ;; - -v) input_dir=${2%%:*}; shift 2 ;; + -v) input_dir="${2%%:*}"; shift 2 ;; createrepo_c) break ;; *) shift ;; esac done shift - args= + args=() for arg in "$@"; do - if [ "$arg" = /input ]; then - args="$args $input_dir" + if [[ "${arg}" == /input ]]; then + args+=("${input_dir}") else - args="$args $arg" + args+=("${arg}") fi done - # shellcheck disable=SC2086 - exec createrepo_c $args + exec createrepo_c "${args[@]}" ;; *) echo "fake docker: unsupported command: $*" >&2 diff --git a/internal-scripts/ci-cd/archive/test/run-tests.sh b/internal-scripts/ci-cd/archive/test/run-tests.sh index d0863095..ddf39f07 100755 --- a/internal-scripts/ci-cd/archive/test/run-tests.sh +++ b/internal-scripts/ci-cd/archive/test/run-tests.sh @@ -1,32 +1,33 @@ #!/bin/bash # Tests archive-apt-packages.rb and archive-yum-packages.rb against a # local fake GCS, inside a container with the real Aptly, createrepo_c -# and GPG. Not run by CI. +# and GPG. Also runs publish-debs.rb against the result, to check that +# CI can still publish afterwards. Not run by CI. # # Usage: ./internal-scripts/ci-cd/archive/test/run-tests.sh set -euo pipefail SELFDIR=$(dirname "$0") -SELFDIR=$(cd "$SELFDIR" && pwd) -ROOTDIR=$(cd "$SELFDIR/../../../.." && pwd) +SELFDIR=$(cd "${SELFDIR}" && pwd) +ROOTDIR=$(cd "${SELFDIR}/../../../.." && pwd) IMAGE=fsruby-archive-test if [[ "${INSIDE_ARCHIVE_TEST_CONTAINER:-}" != true ]]; then - docker build --platform linux/amd64 -q -t "$IMAGE" "$SELFDIR" >/dev/null + docker build --platform linux/amd64 -q -t "${IMAGE}" "${SELFDIR}" >/dev/null exec docker run --rm --platform linux/amd64 \ -e INSIDE_ARCHIVE_TEST_CONTAINER=true \ - -v "$ROOTDIR:/src:ro" \ - "$IMAGE" \ + -v "${ROOTDIR}:/src:ro" \ + "${IMAGE}" \ bash /src/internal-scripts/ci-cd/archive/test/run-tests.sh fi # ---- Inside the container ---- WORK=$(mktemp -d) -export FAKE_GCS_ROOT="$WORK/gcs" -export FAKE_SIGNING_KEY_PATH="$WORK/signing-key.asc" -export PATH="/src/internal-scripts/ci-cd/archive/test/fake-bin:$PATH" -export GNUPGHOME="$WORK/gnupg" +export FAKE_GCS_ROOT="${WORK}/gcs" +export FAKE_SIGNING_KEY_PATH="${WORK}/signing-key.asc" +export PATH="/src/internal-scripts/ci-cd/archive/test/fake-bin:${PATH}" +export GNUPGHOME="${WORK}/gnupg" LIVE_APT=fsruby-server-edition-apt-repo ARCHIVE_APT=fsruby-server-edition-apt-archive-repo LIVE_YUM=fsruby-server-edition-yum-repo @@ -43,36 +44,36 @@ function check() { local description="$1" shift if "$@"; then - echo "PASS: $description" + echo "PASS: ${description}" else - echo "FAIL: $description" + echo "FAIL: ${description}" FAILURES=$((FAILURES + 1)) fi } function gcs_version() { - cat "$FAKE_GCS_ROOT/$1/versions/latest_version.txt" + cat "${FAKE_GCS_ROOT}/$1/versions/latest_version.txt" +} + +function public_dir() { + echo "${FAKE_GCS_ROOT}/$1/versions/$(gcs_version "$1")/public" } # Package names (Package_Version_Architecture) in a published APT distribution. function apt_packages() { - local bucket="$1" distro="$2" - local version - version=$(gcs_version "$bucket") - local file="$FAKE_GCS_ROOT/$bucket/versions/$version/public/dists/$distro/main/binary-amd64/Packages" - [[ -f "$file" ]] || return 0 + local file + file="$(public_dir "$1")/dists/$2/main/binary-amd64/Packages" + [[ -f "${file}" ]] || return 0 # Field order within a stanza is not fixed, so print at the end of each stanza. - awk '/^Package:/{p=$2} /^Version:/{v=$2} /^Architecture:/{a=$2} /^$/{if(p)print p"_"v"_"a; p=""} END{if(p)print p"_"v"_"a}' "$file" | sort + awk '/^Package:/{p=$2} /^Version:/{v=$2} /^Architecture:/{a=$2} /^$/{if(p)print p"_"v"_"a; p=""} END{if(p)print p"_"v"_"a}' "${file}" | sort } # RPM filenames indexed in a published YUM distro/arch directory. function yum_packages() { - local bucket="$1" dir="$2" - local version - version=$(gcs_version "$bucket") - local repodata="$FAKE_GCS_ROOT/$bucket/versions/$version/public/$dir/repodata" - [[ -d "$repodata" ]] || return 0 - zcat "$repodata"/*-primary.xml.gz | grep -o 'location href="[^"]*"' | sed 's/.*href="//; s/"$//; s|.*/||' | sort + local repodata + repodata="$(public_dir "$1")/$2/repodata" + [[ -d "${repodata}" ]] || return 0 + zcat "${repodata}"/*-primary.xml.gz | { grep -o 'location href="[^"]*"' || true; } | sed 's/.*href="//; s/"$//; s|.*/||' | sort } function contains() { @@ -87,133 +88,177 @@ function empty() { [[ -z "$1" ]] } +function count_is() { + [[ "$(grep -c . <<<"$1" || true)" -eq "$2" ]] +} + function apt_signed() { - local version - version=$(gcs_version "$1") - gpg --verify "$FAKE_GCS_ROOT/$1/versions/$version/public/dists/$2/Release.gpg" \ - "$FAKE_GCS_ROOT/$1/versions/$version/public/dists/$2/Release" 2>/dev/null + local dir + dir="$(public_dir "$1")/dists/$2" + gpg --verify "${dir}/Release.gpg" "${dir}/Release" 2>/dev/null } function yum_signed() { - local version - version=$(gcs_version "$1") - gpg --verify "$FAKE_GCS_ROOT/$1/versions/$version/public/$2/repodata/repomd.xml.asc" \ - "$FAKE_GCS_ROOT/$1/versions/$version/public/$2/repodata/repomd.xml" 2>/dev/null + local dir + dir="$(public_dir "$1")/$2/repodata" + gpg --verify "${dir}/repomd.xml.asc" "${dir}/repomd.xml" 2>/dev/null +} + +# Checks that a state tarball matches what publish-debs.rb produces: +# no publications, and an empty repo/ directory. +function state_is_unpublished() { + local bucket="$1" + local dir="${WORK}/state-check" + rm -rf "${dir}" + mkdir -p "${dir}" + zstd -dc "${FAKE_GCS_ROOT}/${bucket}/versions/$(gcs_version "${bucket}")/state.tar.zst" | tar -xC "${dir}" + printf '{"rootDir":"%s"}' "${dir}" >"${WORK}/state-check.conf" + [[ -z "$(aptly -config="${WORK}/state-check.conf" publish list -raw)" ]] && + [[ -z "$(find "${dir}/repo" -mindepth 1 -print -quit)" ]] } header 'Creating signing key' -mkdir -m 700 "$GNUPGHOME" +mkdir -m 700 "${GNUPGHOME}" gpg -q --batch --passphrase '' --quick-gen-key 'Archive Test ' default default never 2>/dev/null -gpg -q --batch --export-secret-keys --armor >"$FAKE_SIGNING_KEY_PATH" +gpg -q --batch --export-secret-keys --armor >"${FAKE_SIGNING_KEY_PATH}" KEY_ID=$(gpg --list-keys --with-colons | awk -F: '/^pub/{print $5; exit}') -echo "Key: $KEY_ID" +echo "Key: ${KEY_ID}" header 'Building fixture packages' -PKGS="$WORK/pkgs" -mkdir -p "$PKGS/deb" "$PKGS/rpm" +PKGS="${WORK}/pkgs" +mkdir -p "${PKGS}/deb" "${PKGS}/rpm" "${PKGS}/new" +# make_deb NAME VERSION ARCH [DISTRIBUTION] [OUTPUT_DIR] function make_deb() { - local name="$1" version="$2" arch="$3" - local dir="$WORK/debbuild/${name}_${version}_${arch}" - mkdir -p "$dir/DEBIAN" + local name="$1" version="$2" arch="$3" distribution="${4:-}" output_dir="${5:-${PKGS}/deb}" + local dir="${WORK}/debbuild/${name}_${version}_${arch}" + mkdir -p "${dir}/DEBIAN" printf 'Package: %s\nVersion: %s\nArchitecture: %s\nMaintainer: Test \nDescription: test\n' \ - "$name" "$version" "$arch" >"$dir/DEBIAN/control" - dpkg-deb -Zgzip -b "$dir" "$PKGS/deb/${name}_${version}_${arch}.deb" >/dev/null + "${name}" "${version}" "${arch}" >"${dir}/DEBIAN/control" + if [[ -n "${distribution}" ]]; then + echo "Distribution: ${distribution}" >>"${dir}/DEBIAN/control" + fi + dpkg-deb -Zgzip -b "${dir}" "${output_dir}/${name}_${version}_${arch}.deb" >/dev/null } function make_rpm() { local name="$1" version="$2" release="$3" arch="$4" - local topdir="$WORK/rpmbuild/$name-$version-$release" - mkdir -p "$topdir/SPECS" - cat >"$topdir/SPECS/pkg.spec" <"${topdir}/SPECS/pkg.spec" </dev/null 2>&1 - find "$topdir/RPMS" -name '*.rpm' -exec cp {} "$PKGS/rpm/" \; + rpmbuild -bb --define "_topdir ${topdir}" --define '_build_id_links none' "${topdir}/SPECS/pkg.spec" >/dev/null 2>&1 + find "${topdir}/RPMS" -name '*.rpm' -exec cp {} "${PKGS}/rpm/" \; } -# DEB: debian-12 (supported), debian-10 (EOL). Ruby 3.4 active, 3.1 EOL. +# DEB distributions: +# debian-12 supported, has EOL Ruby packages +# debian-13 supported, nothing to archive +# debian-10 EOL +# Ruby 3.4 is active, 3.1 is EOL (per config.yml). for distro in debian-12 debian-10; do - make_deb fullstaq-ruby-3.4 "9-$distro" amd64 - make_deb fullstaq-ruby-3.4-jemalloc "9-$distro" amd64 - make_deb fullstaq-ruby-3.4.1 "0-$distro" amd64 - make_deb fullstaq-ruby-3.1 "5-$distro" amd64 - make_deb fullstaq-ruby-3.1-jemalloc "5-$distro" amd64 - make_deb fullstaq-ruby-3.1.7 "0-$distro" amd64 + make_deb fullstaq-ruby-3.4 "9-${distro}" amd64 + make_deb fullstaq-ruby-3.4-jemalloc "9-${distro}" amd64 + make_deb fullstaq-ruby-3.4.1 "0-${distro}" amd64 + make_deb fullstaq-ruby-3.1 "5-${distro}" amd64 + make_deb fullstaq-ruby-3.1-jemalloc "5-${distro}" amd64 + make_deb fullstaq-ruby-3.1.7 "0-${distro}" amd64 done +make_deb fullstaq-ruby-3.4 9-debian-13 amd64 make_deb fullstaq-ruby-common 1.0-0 all make_deb fullstaq-rbenv 1.1.2-16 all - -# RPM: el-9 (supported), centos-7 (EOL). +# New packages for the publish-debs.rb runs. +make_deb fullstaq-ruby-3.4 10-debian-12 amd64 debian-12 "${PKGS}/new" +make_deb fullstaq-ruby-3.4 10-debian-13 amd64 debian-13 "${PKGS}/new" + +# RPM distributions: +# el-9 supported, has EOL Ruby packages +# centos-8 supported, nothing to archive +# centos-7 EOL for distro in el9 centos7; do - make_rpm fullstaq-ruby-3.4 rev9 "$distro" x86_64 - make_rpm fullstaq-ruby-3.4-jemalloc rev9 "$distro" x86_64 - make_rpm fullstaq-ruby-3.1 rev5 "$distro" x86_64 - make_rpm fullstaq-ruby-3.1-jemalloc rev5 "$distro" x86_64 - make_rpm fullstaq-ruby-3.1.7 rev0 "$distro" x86_64 + make_rpm fullstaq-ruby-3.4 rev9 "${distro}" x86_64 + make_rpm fullstaq-ruby-3.4-jemalloc rev9 "${distro}" x86_64 + make_rpm fullstaq-ruby-3.1 rev5 "${distro}" x86_64 + make_rpm fullstaq-ruby-3.1-jemalloc rev5 "${distro}" x86_64 + make_rpm fullstaq-ruby-3.1.7 rev0 "${distro}" x86_64 done +make_rpm fullstaq-ruby-3.4 rev9 centos8 x86_64 make_rpm fullstaq-ruby-common 1.0 0 noarch make_rpm fullstaq-rbenv 1.1.2_16 1 noarch -ls "$PKGS/deb" "$PKGS/rpm" +ls "${PKGS}/deb" "${PKGS}/rpm" header 'Seeding live APT repository (version 1)' +# Like publish-debs.rb: the state is archived before publishing, so it +# contains no publications. function seed_apt() { - local state="$WORK/seed-apt" - mkdir -p "$state/db" "$state/repo" + local state="${WORK}/seed-apt" + mkdir -p "${state}/db" "${state}/repo" printf '{"rootDir":"%s","FileSystemPublishEndpoints":{"main":{"rootDir":"%s/repo","linkMethod":"symlink","verifyMethod":"md5"}}}' \ - "$state" "$state" >"$WORK/seed-apt.conf" + "${state}" "${state}" >"${WORK}/seed-apt.conf" local distro - for distro in debian-12 debian-10; do - aptly -config="$WORK/seed-apt.conf" repo create "$distro" >/dev/null - aptly -config="$WORK/seed-apt.conf" repo add "$distro" "$PKGS"/deb/*"$distro"*.deb \ - "$PKGS/deb/fullstaq-ruby-common_1.0-0_all.deb" "$PKGS/deb/fullstaq-rbenv_1.1.2-16_all.deb" >/dev/null - aptly -config="$WORK/seed-apt.conf" publish repo -batch -gpg-key="$KEY_ID" -distribution="$distro" \ - -origin=Fullstaq-Ruby -label=Fullstaq-Ruby "$distro" filesystem:main:. >/dev/null + for distro in debian-12 debian-13 debian-10; do + aptly -config="${WORK}/seed-apt.conf" repo create "${distro}" >/dev/null + aptly -config="${WORK}/seed-apt.conf" repo add "${distro}" "${PKGS}"/deb/*"${distro}"*.deb \ + "${PKGS}/deb/fullstaq-ruby-common_1.0-0_all.deb" "${PKGS}/deb/fullstaq-rbenv_1.1.2-16_all.deb" >/dev/null + done + local version_dir="${FAKE_GCS_ROOT}/${LIVE_APT}/versions/1" + mkdir -p "${version_dir}/public" + tar -C "${state}" -cf - . | zstd -q -T0 >"${version_dir}/state.tar.zst" + for distro in debian-12 debian-13 debian-10; do + aptly -config="${WORK}/seed-apt.conf" publish repo -batch -gpg-key="${KEY_ID}" -distribution="${distro}" \ + -origin=Fullstaq-Ruby -label=Fullstaq-Ruby "${distro}" filesystem:main:. >/dev/null done - local version_dir="$FAKE_GCS_ROOT/$LIVE_APT/versions/1" - mkdir -p "$version_dir/public" - tar -C "$state" -cf - . | zstd -q -T0 >"$version_dir/state.tar.zst" - cp -rL "$state/repo/." "$version_dir/public/" - echo 1 >"$version_dir/version.txt" - echo 1 >"$FAKE_GCS_ROOT/$LIVE_APT/versions/latest_version.txt" + cp -rL "${state}/repo/." "${version_dir}/public/" + echo 1 >"${version_dir}/version.txt" + echo 1 >"${FAKE_GCS_ROOT}/${LIVE_APT}/versions/latest_version.txt" } seed_apt header 'Seeding live YUM repository (version 1)' function seed_yum() { - local version_dir="$FAKE_GCS_ROOT/$LIVE_YUM/versions/1" + local version_dir="${FAKE_GCS_ROOT}/${LIVE_YUM}/versions/1" local distro sanitized dir - for distro in el-9 centos-7; do + for distro in el-9 centos-8 centos-7; do sanitized=${distro//-/} - dir="$version_dir/public/$distro/x86_64" - mkdir -p "$dir" - cp "$PKGS"/rpm/*"$sanitized"*.rpm "$PKGS"/rpm/*.noarch.rpm "$dir/" - createrepo_c -q "$dir" - gpg -q --batch --yes --detach-sign --armor "$dir/repodata/repomd.xml" + dir="${version_dir}/public/${distro}/x86_64" + mkdir -p "${dir}" + cp "${PKGS}"/rpm/*"${sanitized}"*.rpm "${PKGS}"/rpm/*.noarch.rpm "${dir}/" + createrepo_c -q "${dir}" + gpg -q --batch --yes --detach-sign --armor "${dir}/repodata/repomd.xml" done - echo 1 >"$version_dir/version.txt" - echo 1 >"$FAKE_GCS_ROOT/$LIVE_YUM/versions/latest_version.txt" + echo 1 >"${version_dir}/version.txt" + echo 1 >"${FAKE_GCS_ROOT}/${LIVE_YUM}/versions/latest_version.txt" } seed_yum -# The scripts must run from a writable copy: they don't write to the repo, -# but GCloudStorageLock logs relative to cwd and we keep /src read-only. -cp -r /src "$WORK/src" -cd "$WORK/src" +# The scripts read config.yml and environments/ from the working copy. +# Use a writable copy so the second round can retire a distribution. +cp -r /src "${WORK}/src" +cd "${WORK}/src" -APT_ENV=(env PRODUCTION_REPO_BUCKET_NAME="$LIVE_APT" ARCHIVE_REPO_BUCKET_NAME="$ARCHIVE_APT") -YUM_ENV=(env PRODUCTION_REPO_BUCKET_NAME="$LIVE_YUM" ARCHIVE_REPO_BUCKET_NAME="$ARCHIVE_YUM") +APT_ENV=(env PRODUCTION_REPO_BUCKET_NAME="${LIVE_APT}" ARCHIVE_REPO_BUCKET_NAME="${ARCHIVE_APT}") +YUM_ENV=(env PRODUCTION_REPO_BUCKET_NAME="${LIVE_YUM}" ARCHIVE_REPO_BUCKET_NAME="${ARCHIVE_YUM}") function snapshot() { - (cd "$FAKE_GCS_ROOT" && find . -type f ! -path '*/locks/*' -exec md5sum {} + | sort -k2) + (cd "${FAKE_GCS_ROOT}" && find . -type f ! -path '*/locks/*' -exec md5sum {} + | sort -k2) +} + +# Runs publish-debs.rb in production mode, as CI does on main, with +# DRY_RUN so it uploads nothing. Fails if it can't publish the state +# the archive script left behind. +function publish_debs_succeeds() { + env PRODUCTION_REPO_BUCKET_NAME="${LIVE_APT}" TESTING=false OVERWRITE_EXISTING=false DRY_RUN=true CLEANUP=true \ + ./internal-scripts/ci-cd/publish/publish-debs.rb "$1" >"${WORK}/publish-debs.log" 2>&1 || + { tail -20 "${WORK}/publish-debs.log"; return 1; } } # ---------------------------------------------------------------- APT @@ -221,119 +266,157 @@ function snapshot() { header 'APT: dry run uploads nothing' before=$(snapshot) "${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb --dry-run -check 'dry run leaves fake GCS unchanged' [ "$before" == "$(snapshot)" ] +check 'dry run leaves fake GCS unchanged' [ "${before}" == "$(snapshot)" ] +check 'dry run leaves no temp dir behind' empty "$(find /tmp -maxdepth 1 -name 'archive-apt*')" header 'APT: first run' +LIVE_D13_BEFORE=$(apt_packages "${LIVE_APT}" debian-13) "${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb -LIVE_D12=$(apt_packages $LIVE_APT debian-12) -ARCH_D12=$(apt_packages $ARCHIVE_APT debian-12) -ARCH_D10=$(apt_packages $ARCHIVE_APT debian-10) -echo "live debian-12:"; sed 's/^/ /' <<<"$LIVE_D12" -echo "archive debian-12:"; sed 's/^/ /' <<<"$ARCH_D12" -echo "archive debian-10:"; sed 's/^/ /' <<<"$ARCH_D10" - -check 'live version bumped to 2' [ "$(gcs_version $LIVE_APT)" == 2 ] -check 'archive version is 1' [ "$(gcs_version $ARCHIVE_APT)" == 1 ] -check 'live debian-12 keeps active Ruby' contains "$LIVE_D12" fullstaq-ruby-3.4_9-debian-12_amd64 -check 'live debian-12 keeps active Ruby variant' contains "$LIVE_D12" fullstaq-ruby-3.4-jemalloc_9-debian-12_amd64 -check 'live debian-12 keeps tiny of active minor' contains "$LIVE_D12" fullstaq-ruby-3.4.1_0-debian-12_amd64 -check 'live debian-12 keeps common' contains "$LIVE_D12" fullstaq-ruby-common_1.0-0_all -check 'live debian-12 keeps rbenv' contains "$LIVE_D12" fullstaq-rbenv_1.1.2-16_all -check 'live debian-12 drops EOL Ruby' lacks "$LIVE_D12" fullstaq-ruby-3.1_5-debian-12_amd64 -check 'live debian-12 drops EOL Ruby variant' lacks "$LIVE_D12" fullstaq-ruby-3.1-jemalloc_5-debian-12_amd64 -check 'live debian-12 drops tiny of EOL minor' lacks "$LIVE_D12" fullstaq-ruby-3.1.7_0-debian-12_amd64 -check 'live debian-10 no longer published' empty "$(apt_packages $LIVE_APT debian-10)" -check 'archive debian-12 has exactly the EOL Ruby packages' [ "$ARCH_D12" == "$(printf '%s\n' \ +LIVE_D12=$(apt_packages "${LIVE_APT}" debian-12) +LIVE_D13=$(apt_packages "${LIVE_APT}" debian-13) +ARCH_D12=$(apt_packages "${ARCHIVE_APT}" debian-12) +ARCH_D10=$(apt_packages "${ARCHIVE_APT}" debian-10) +echo "live debian-12:"; sed 's/^/ /' <<<"${LIVE_D12}" +echo "live debian-13:"; sed 's/^/ /' <<<"${LIVE_D13}" +echo "archive debian-12:"; sed 's/^/ /' <<<"${ARCH_D12}" +echo "archive debian-10:"; sed 's/^/ /' <<<"${ARCH_D10}" + +check 'live version bumped to 2' [ "$(gcs_version "${LIVE_APT}")" == 2 ] +check 'archive version is 1' [ "$(gcs_version "${ARCHIVE_APT}")" == 1 ] +check 'live debian-12 keeps active Ruby' contains "${LIVE_D12}" fullstaq-ruby-3.4_9-debian-12_amd64 +check 'live debian-12 keeps active Ruby variant' contains "${LIVE_D12}" fullstaq-ruby-3.4-jemalloc_9-debian-12_amd64 +check 'live debian-12 keeps tiny of active minor' contains "${LIVE_D12}" fullstaq-ruby-3.4.1_0-debian-12_amd64 +check 'live debian-12 keeps common' contains "${LIVE_D12}" fullstaq-ruby-common_1.0-0_all +check 'live debian-12 keeps rbenv' contains "${LIVE_D12}" fullstaq-rbenv_1.1.2-16_all +check 'live debian-12 drops EOL Ruby' lacks "${LIVE_D12}" fullstaq-ruby-3.1_5-debian-12_amd64 +check 'live debian-12 drops EOL Ruby variant' lacks "${LIVE_D12}" fullstaq-ruby-3.1-jemalloc_5-debian-12_amd64 +check 'live debian-12 drops tiny of EOL minor' lacks "${LIVE_D12}" fullstaq-ruby-3.1.7_0-debian-12_amd64 +check 'live debian-13 (nothing to archive) still published, unchanged' [ "${LIVE_D13}" == "${LIVE_D13_BEFORE}" ] +check 'live debian-13 Release is signed' apt_signed "${LIVE_APT}" debian-13 +check 'live debian-10 no longer published' empty "$(apt_packages "${LIVE_APT}" debian-10)" +check 'archive debian-12 has exactly the EOL Ruby packages' [ "${ARCH_D12}" == "$(printf '%s\n' \ fullstaq-ruby-3.1-jemalloc_5-debian-12_amd64 fullstaq-ruby-3.1.7_0-debian-12_amd64 fullstaq-ruby-3.1_5-debian-12_amd64 | sort)" ] -check 'archive debian-10 has all 8 packages' [ "$(wc -l <<<"$ARCH_D10")" -eq 8 ] -check 'archive debian-10 has common' contains "$ARCH_D10" fullstaq-ruby-common_1.0-0_all -check 'live and archive debian-12 are disjoint' empty "$(comm -12 <(echo "$LIVE_D12") <(echo "$ARCH_D12"))" -check 'live debian-12 Release is signed' apt_signed $LIVE_APT debian-12 -check 'archive debian-12 Release is signed' apt_signed $ARCHIVE_APT debian-12 -check 'archive debian-10 Release is signed' apt_signed $ARCHIVE_APT debian-10 -v=$(gcs_version $ARCHIVE_APT) -check 'archive pool contains debian-10 debs' \ - [ -n "$(find "$FAKE_GCS_ROOT/$ARCHIVE_APT/versions/$v/public/pool" -name '*debian-10*.deb')" ] -v=$(gcs_version $LIVE_APT) -check 'live pool no longer contains EOL Ruby debs' \ - [ -z "$(find "$FAKE_GCS_ROOT/$LIVE_APT/versions/$v/public/pool" -name 'fullstaq-ruby-3.1*')" ] -check 'live pool no longer contains debian-10 debs' \ - [ -z "$(find "$FAKE_GCS_ROOT/$LIVE_APT/versions/$v/public/pool" -name '*debian-10*')" ] -check 'lock released' [ ! -e "$FAKE_GCS_ROOT/$LIVE_APT/locks/apt" ] +check 'archive debian-10 has all 8 packages' count_is "${ARCH_D10}" 8 +check 'archive debian-10 has common' contains "${ARCH_D10}" fullstaq-ruby-common_1.0-0_all +check 'archive has no debian-13' empty "$(apt_packages "${ARCHIVE_APT}" debian-13)" +check 'live and archive debian-12 are disjoint' empty "$(comm -12 <(echo "${LIVE_D12}") <(echo "${ARCH_D12}"))" +check 'live debian-12 Release is signed' apt_signed "${LIVE_APT}" debian-12 +check 'archive debian-12 Release is signed' apt_signed "${ARCHIVE_APT}" debian-12 +check 'archive debian-10 Release is signed' apt_signed "${ARCHIVE_APT}" debian-10 +check 'archive pool contains debian-10 debs' [ -n "$(find "$(public_dir "${ARCHIVE_APT}")/pool" -name '*debian-10*.deb')" ] +check 'live pool no longer contains EOL Ruby debs' empty "$(find "$(public_dir "${LIVE_APT}")/pool" -name 'fullstaq-ruby-3.1*')" +check 'live pool no longer contains debian-10 debs' empty "$(find "$(public_dir "${LIVE_APT}")/pool" -name '*debian-10*')" +check 'live state has no publications (like publish-debs)' state_is_unpublished "${LIVE_APT}" +check 'archive state has no publications' state_is_unpublished "${ARCHIVE_APT}" +check 'lock released' [ ! -e "${FAKE_GCS_ROOT}/${LIVE_APT}/locks/apt" ] +check 'no temp dir left behind' empty "$(find /tmp -maxdepth 1 -name 'archive-apt*')" +check 'publish-debs.rb can publish afterwards' publish_debs_succeeds "${PKGS}/new/fullstaq-ruby-3.4_10-debian-12_amd64.deb" header 'APT: second run is a no-op' before=$(snapshot) "${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb -check 'second run changes nothing' [ "$before" == "$(snapshot)" ] +check 'second run changes nothing' [ "${before}" == "$(snapshot)" ] header 'APT: rerun after crash between archive and live upload' -# Simulate: archive version 2 was activated, live upload never happened. +# Simulate: archive version 1 was activated, live upload never happened. # Roll the live pointer back to version 1 (still intact in the bucket). -echo 1 >"$FAKE_GCS_ROOT/$LIVE_APT/versions/latest_version.txt" -rm -rf "$FAKE_GCS_ROOT/$LIVE_APT/versions/2" +echo 1 >"${FAKE_GCS_ROOT}/${LIVE_APT}/versions/latest_version.txt" +rm -rf "${FAKE_GCS_ROOT}/${LIVE_APT}/versions/2" "${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb -check 'archive debian-12 unchanged after rerun' [ "$ARCH_D12" == "$(apt_packages $ARCHIVE_APT debian-12)" ] -check 'archive debian-10 unchanged after rerun' [ "$ARCH_D10" == "$(apt_packages $ARCHIVE_APT debian-10)" ] -check 'live debian-12 trimmed after rerun' [ "$LIVE_D12" == "$(apt_packages $LIVE_APT debian-12)" ] -check 'archive debian-12 still signed after rerun' apt_signed $ARCHIVE_APT debian-12 +check 'archive debian-12 unchanged after rerun' [ "${ARCH_D12}" == "$(apt_packages "${ARCHIVE_APT}" debian-12)" ] +check 'archive debian-10 unchanged after rerun' [ "${ARCH_D10}" == "$(apt_packages "${ARCHIVE_APT}" debian-10)" ] +check 'live debian-12 trimmed after rerun' [ "${LIVE_D12}" == "$(apt_packages "${LIVE_APT}" debian-12)" ] +check 'live debian-13 unchanged after rerun' [ "${LIVE_D13}" == "$(apt_packages "${LIVE_APT}" debian-13)" ] +check 'archive debian-12 still signed after rerun' apt_signed "${ARCHIVE_APT}" debian-12 + +header 'APT: second archival round (debian-12 reaches EOL)' +# debian-10 is already archived and is not touched this round. Its +# publication must survive being rebuilt from the previous archive state. +mv environments/debian-12 "${WORK}/debian-12-env" +"${APT_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-apt-packages.rb +ARCH2_D12=$(apt_packages "${ARCHIVE_APT}" debian-12) +echo "archive debian-12:"; sed 's/^/ /' <<<"${ARCH2_D12}" +check 'live debian-12 no longer published' empty "$(apt_packages "${LIVE_APT}" debian-12)" +check 'live debian-13 still unchanged' [ "${LIVE_D13}" == "$(apt_packages "${LIVE_APT}" debian-13)" ] +check 'archive debian-12 now has all 8 packages' count_is "${ARCH2_D12}" 8 +check 'archive debian-10 unchanged by second round' [ "${ARCH_D10}" == "$(apt_packages "${ARCHIVE_APT}" debian-10)" ] +check 'archive debian-10 still signed' apt_signed "${ARCHIVE_APT}" debian-10 +check 'archive debian-12 signed' apt_signed "${ARCHIVE_APT}" debian-12 +check 'publish-debs.rb can publish after second round' publish_debs_succeeds "${PKGS}/new/fullstaq-ruby-3.4_10-debian-13_amd64.deb" +mv "${WORK}/debian-12-env" environments/debian-12 # ---------------------------------------------------------------- YUM header 'YUM: dry run uploads nothing' before=$(snapshot) "${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb --dry-run -check 'dry run leaves fake GCS unchanged' [ "$before" == "$(snapshot)" ] +check 'dry run leaves fake GCS unchanged' [ "${before}" == "$(snapshot)" ] header 'YUM: first run' +LIVE_C8_BEFORE=$(yum_packages "${LIVE_YUM}" centos-8/x86_64) "${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb -LIVE_EL9=$(yum_packages $LIVE_YUM el-9/x86_64) -ARCH_EL9=$(yum_packages $ARCHIVE_YUM el-9/x86_64) -ARCH_C7=$(yum_packages $ARCHIVE_YUM centos-7/x86_64) -echo "live el-9:"; sed 's/^/ /' <<<"$LIVE_EL9" -echo "archive el-9:"; sed 's/^/ /' <<<"$ARCH_EL9" -echo "archive centos-7:"; sed 's/^/ /' <<<"$ARCH_C7" - -check 'live version bumped to 2' [ "$(gcs_version $LIVE_YUM)" == 2 ] -check 'archive version is 1' [ "$(gcs_version $ARCHIVE_YUM)" == 1 ] -check 'live el-9 keeps active Ruby' contains "$LIVE_EL9" fullstaq-ruby-3.4-rev9-el9.x86_64.rpm -check 'live el-9 keeps common' contains "$LIVE_EL9" fullstaq-ruby-common-1.0-0.noarch.rpm -check 'live el-9 keeps rbenv' contains "$LIVE_EL9" fullstaq-rbenv-1.1.2_16-1.noarch.rpm -check 'live el-9 drops EOL Ruby' lacks "$LIVE_EL9" fullstaq-ruby-3.1-rev5-el9.x86_64.rpm -check 'live el-9 drops tiny of EOL minor' lacks "$LIVE_EL9" fullstaq-ruby-3.1.7-rev0-el9.x86_64.rpm -check 'live el-9 has no leftover EOL RPM files' \ - [ -z "$(find "$FAKE_GCS_ROOT/$LIVE_YUM/versions/2/public/el-9" -name 'fullstaq-ruby-3.1*')" ] -check 'live centos-7 removed' [ ! -e "$FAKE_GCS_ROOT/$LIVE_YUM/versions/2/public/centos-7" ] -check 'archive el-9 has exactly the EOL Ruby packages' [ "$ARCH_EL9" == "$(printf '%s\n' \ +LIVE_EL9=$(yum_packages "${LIVE_YUM}" el-9/x86_64) +LIVE_C8=$(yum_packages "${LIVE_YUM}" centos-8/x86_64) +ARCH_EL9=$(yum_packages "${ARCHIVE_YUM}" el-9/x86_64) +ARCH_C7=$(yum_packages "${ARCHIVE_YUM}" centos-7/x86_64) +echo "live el-9:"; sed 's/^/ /' <<<"${LIVE_EL9}" +echo "archive el-9:"; sed 's/^/ /' <<<"${ARCH_EL9}" +echo "archive centos-7:"; sed 's/^/ /' <<<"${ARCH_C7}" + +check 'live version bumped to 2' [ "$(gcs_version "${LIVE_YUM}")" == 2 ] +check 'archive version is 1' [ "$(gcs_version "${ARCHIVE_YUM}")" == 1 ] +check 'live el-9 keeps active Ruby' contains "${LIVE_EL9}" fullstaq-ruby-3.4-rev9-el9.x86_64.rpm +check 'live el-9 keeps common' contains "${LIVE_EL9}" fullstaq-ruby-common-1.0-0.noarch.rpm +check 'live el-9 keeps rbenv' contains "${LIVE_EL9}" fullstaq-rbenv-1.1.2_16-1.noarch.rpm +check 'live el-9 drops EOL Ruby' lacks "${LIVE_EL9}" fullstaq-ruby-3.1-rev5-el9.x86_64.rpm +check 'live el-9 drops tiny of EOL minor' lacks "${LIVE_EL9}" fullstaq-ruby-3.1.7-rev0-el9.x86_64.rpm +check 'live el-9 has no leftover EOL RPM files' empty "$(find "$(public_dir "${LIVE_YUM}")/el-9" -name 'fullstaq-ruby-3.1*')" +check 'live centos-8 (nothing to archive) unchanged' [ "${LIVE_C8}" == "${LIVE_C8_BEFORE}" ] +check 'live centos-8 repomd is signed' yum_signed "${LIVE_YUM}" centos-8/x86_64 +check 'live centos-7 removed' [ ! -e "$(public_dir "${LIVE_YUM}")/centos-7" ] +check 'archive el-9 has exactly the EOL Ruby packages' [ "${ARCH_EL9}" == "$(printf '%s\n' \ fullstaq-ruby-3.1-jemalloc-rev5-el9.x86_64.rpm fullstaq-ruby-3.1-rev5-el9.x86_64.rpm fullstaq-ruby-3.1.7-rev0-el9.x86_64.rpm | sort)" ] -check 'archive centos-7 has all 7 packages' [ "$(wc -l <<<"$ARCH_C7")" -eq 7 ] -check 'live and archive el-9 are disjoint' empty "$(comm -12 <(echo "$LIVE_EL9") <(echo "$ARCH_EL9"))" -check 'live el-9 repomd is signed' yum_signed $LIVE_YUM el-9/x86_64 -check 'archive el-9 repomd is signed' yum_signed $ARCHIVE_YUM el-9/x86_64 -check 'archive centos-7 repomd is signed' yum_signed $ARCHIVE_YUM centos-7/x86_64 -check 'no nested distro directory in archive' [ ! -e "$FAKE_GCS_ROOT/$ARCHIVE_YUM/versions/1/public/centos-7/centos-7" ] -check 'lock released' [ ! -e "$FAKE_GCS_ROOT/$LIVE_YUM/locks/yum" ] +check 'archive centos-7 has all 7 packages' count_is "${ARCH_C7}" 7 +check 'archive has no centos-8' [ ! -e "$(public_dir "${ARCHIVE_YUM}")/centos-8" ] +check 'live and archive el-9 are disjoint' empty "$(comm -12 <(echo "${LIVE_EL9}") <(echo "${ARCH_EL9}"))" +check 'live el-9 repomd is signed' yum_signed "${LIVE_YUM}" el-9/x86_64 +check 'archive el-9 repomd is signed' yum_signed "${ARCHIVE_YUM}" el-9/x86_64 +check 'archive centos-7 repomd is signed' yum_signed "${ARCHIVE_YUM}" centos-7/x86_64 +check 'no nested distro directory in archive' [ ! -e "$(public_dir "${ARCHIVE_YUM}")/centos-7/centos-7" ] +check 'lock released' [ ! -e "${FAKE_GCS_ROOT}/${LIVE_YUM}/locks/yum" ] +check 'no temp dir left behind' empty "$(find /tmp -maxdepth 1 -name 'archive-yum*')" header 'YUM: second run is a no-op' before=$(snapshot) "${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb -check 'second run changes nothing' [ "$before" == "$(snapshot)" ] +check 'second run changes nothing' [ "${before}" == "$(snapshot)" ] header 'YUM: rerun after crash between archive and live upload' -echo 1 >"$FAKE_GCS_ROOT/$LIVE_YUM/versions/latest_version.txt" -rm -rf "$FAKE_GCS_ROOT/$LIVE_YUM/versions/2" +echo 1 >"${FAKE_GCS_ROOT}/${LIVE_YUM}/versions/latest_version.txt" +rm -rf "${FAKE_GCS_ROOT}/${LIVE_YUM}/versions/2" +"${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb +check 'archive el-9 unchanged after rerun' [ "${ARCH_EL9}" == "$(yum_packages "${ARCHIVE_YUM}" el-9/x86_64)" ] +check 'archive centos-7 unchanged after rerun' [ "${ARCH_C7}" == "$(yum_packages "${ARCHIVE_YUM}" centos-7/x86_64)" ] +check 'live el-9 trimmed after rerun' [ "${LIVE_EL9}" == "$(yum_packages "${LIVE_YUM}" el-9/x86_64)" ] +check 'no nested distro directory after rerun' empty "$(find "${FAKE_GCS_ROOT}/${ARCHIVE_YUM}" -path '*/centos-7/centos-7')" + +header 'YUM: second archival round (el-9 reaches EOL)' +mv environments/el-9 "${WORK}/el-9-env" "${YUM_ENV[@]}" ./internal-scripts/ci-cd/archive/archive-yum-packages.rb -check 'archive el-9 unchanged after rerun' [ "$ARCH_EL9" == "$(yum_packages $ARCHIVE_YUM el-9/x86_64)" ] -check 'archive centos-7 unchanged after rerun' [ "$ARCH_C7" == "$(yum_packages $ARCHIVE_YUM centos-7/x86_64)" ] -check 'live el-9 trimmed after rerun' [ "$LIVE_EL9" == "$(yum_packages $LIVE_YUM el-9/x86_64)" ] -check 'no nested distro directory after rerun' \ - [ -z "$(find "$FAKE_GCS_ROOT/$ARCHIVE_YUM" -path '*/centos-7/centos-7')" ] +ARCH2_EL9=$(yum_packages "${ARCHIVE_YUM}" el-9/x86_64) +check 'live el-9 removed' [ ! -e "$(public_dir "${LIVE_YUM}")/el-9" ] +check 'live centos-8 still unchanged' [ "${LIVE_C8}" == "$(yum_packages "${LIVE_YUM}" centos-8/x86_64)" ] +check 'archive el-9 now has all 7 packages' count_is "${ARCH2_EL9}" 7 +check 'archive centos-7 unchanged by second round' [ "${ARCH_C7}" == "$(yum_packages "${ARCHIVE_YUM}" centos-7/x86_64)" ] +check 'archive el-9 signed' yum_signed "${ARCHIVE_YUM}" el-9/x86_64 +mv "${WORK}/el-9-env" environments/el-9 header 'Result' -if [[ $FAILURES -eq 0 ]]; then +if [[ "${FAILURES}" -eq 0 ]]; then echo 'All checks passed' else - echo "$FAILURES checks failed" + echo "${FAILURES} checks failed" exit 1 fi From 70344a261d9c8b9d4781618285148f8e1370bd40 Mon Sep 17 00:00:00 2001 From: abtreece Date: Tue, 29 Sep 2026 22:43:18 -0500 Subject: [PATCH 12/12] docs: correct archiving handbook on distro retirement, lock timeouts, and active minors --- dev-handbook/archiving-eol-packages.md | 41 +++++++++++++++----------- 1 file changed, 24 insertions(+), 17 deletions(-) diff --git a/dev-handbook/archiving-eol-packages.md b/dev-handbook/archiving-eol-packages.md index 62aacd23..38d8f688 100644 --- a/dev-handbook/archiving-eol-packages.md +++ b/dev-handbook/archiving-eol-packages.md @@ -27,13 +27,14 @@ This pattern follows the precedent set by [PostgreSQL](https://apt-archive.postg Each script decides per package, based on the current `config.yml`: - * **EOL distribution** — the distribution has a repository in the live repo, but is no longer in `config.yml`'s distributions. *All* its packages are archived, including `fullstaq-ruby-common` and `fullstaq-rbenv`, and the distribution is removed from the live repo. - * **EOL Ruby version** — on a supported distribution, any `fullstaq-ruby-X.Y*` package where `X.Y` is not in `minor_version_packages`. This includes variants (`-jemalloc`, `-malloctrim`) and tiny-version packages (`fullstaq-ruby-X.Y.Z`). + * **EOL distribution** — the distribution has a repository in the live repo, but is no longer built (see [Step 1](#step-1-remove-from-the-build-system)). *All* its packages are archived, including `fullstaq-ruby-common` and `fullstaq-rbenv`, and the distribution is removed from the live repo. + * **EOL Ruby version** — on a supported distribution, any `fullstaq-ruby-X.Y*` package where `X.Y` is not an active minor version (see below). This includes variants (`-jemalloc`, `-malloctrim`) and tiny-version packages (`fullstaq-ruby-X.Y.Z`). Everything else stays in the live repo. In particular: * `fullstaq-ruby-common` and `fullstaq-rbenv` stay on supported distributions. * A tiny-version package stays as long as its minor version is active, even if it has been removed from `tiny_version_packages`. + * A minor version counts as active while it appears in `minor_version_packages` *or* `tiny_version_packages`. So when retiring a Ruby minor version, remove it from both, or nothing for it is archived. The scripts are: @@ -48,12 +49,14 @@ Both handle EOL distributions and EOL Ruby versions in the same run, so there is For an EOL distribution: - 1. Edit `config.yml` and remove the distribution from the `distributions` list (or add it to an exclusion). - 2. Delete the `environments//` directory. + 1. Delete the `environments//` directory. With `distributions: all` in `config.yml` (the default), that is what removes a distribution from the build. + 2. If `config.yml` lists distributions explicitly instead, remove it from that list too. + +`distribution_exclusions` is not a way to retire a distribution: it only stops particular Ruby versions being built for it. For an EOL Ruby version: - 1. Edit `config.yml` and remove the version from `minor_version_packages` (and `tiny_version_packages`). + 1. Edit `config.yml` and remove the version from both `minor_version_packages` and `tiny_version_packages`. Then: @@ -100,7 +103,7 @@ Replace `X.Y`, `Distro N` and `` with what's being archived. Use the sam * Docker running (for `createrepo_c`, for YUM) * Enough free disk space for the full live APT state (roughly 30 GB at the time of writing) -**Dry run first.** A dry run makes every change locally, including the checks described in [How the scripts work](#how-the-scripts-work), and prints the list of packages it would move, but uploads nothing: +**Dry run first.** A dry run makes every change locally, including the checks described in [How the scripts work](#how-the-scripts-work), and prints the list of packages it would move, but uploads nothing. It still takes the lock and downloads the full live state: ~~~bash PRODUCTION_REPO_BUCKET_NAME=fsruby-server-edition-apt-repo \ @@ -128,7 +131,7 @@ ARCHIVE_REPO_BUCKET_NAME=fsruby-server-edition-yum-archive-repo \ ./internal-scripts/ci-cd/archive/archive-yum-packages.rb ~~~ -The scripts hold the live repo's lock (`locks/apt` or `locks/yum`) for the whole run, so a CI publish can't run at the same time. +The scripts hold the live repo's lock (`locks/apt` or `locks/yum`) for the whole run, which can take an hour for APT. A CI publish waits at most 5 minutes for the lock, then fails. **Don't merge to `main` or re-run CI publishing during an archival run.** If a publish fails this way, re-run it afterwards. ### Step 4: Restart the web server @@ -189,14 +192,13 @@ Both scripts follow the same sequence: 2. Download the latest live repo version. 3. Decide which packages to move (see [What gets archived](#what-gets-archived)). 4. Download the latest archive version, if any. - 5. Add the packages to the local archive copy and regenerate its signed metadata. - 6. **Verify** that the archive's published metadata lists every package being moved. The script aborts here, before touching the live repo, if anything is missing. - 7. Remove the packages from the local live copy and regenerate its signed metadata. - 8. Verify that the live repo's published metadata no longer lists any of them. - 9. Upload the archive as version M+1 and activate it. - 10. Upload the live repo as version N+1 and activate it. + 5. Add the packages to the local archive copy and remove them from the local live copy. + 6. Regenerate the signed metadata of every distribution in both copies, not only the ones that changed. + 7. **Verify** that the archive's metadata lists every package being moved, and that the live repo's metadata lists none of them. The script aborts here, before uploading anything, if either check fails. + 8. Upload the archive as version M+1 and activate it. + 9. Upload the live repo as version N+1 and activate it. -The archive is activated before the live repo, so the moved packages are always available from at least one of them. +The archive is activated before the live repo, so the moved packages are always available from at least one of them. Before activating each version, the script checks that it still holds the lock. **Reruns are safe.** If a run fails after step 9, rerun the script: @@ -209,7 +211,9 @@ A run with nothing to archive exits without uploading anything. * Packages are added to the archive's Aptly instance individually, with `aptly repo add -force-replace` using the `.deb` files from the live state's pool. Only the moved packages' files end up in the archive's pool. * EOL Ruby packages are removed from live with `aptly repo remove`. EOL distributions are unpublished and dropped with `aptly publish drop` and `aptly repo drop`. - * Both instances run `aptly db cleanup` after republishing, so unreferenced files leave the pool. + * Both instances run `aptly db cleanup`, so unreferenced files leave the pool. + * Like `publish-debs.rb`, the state archive (`state.tar.zst`) is created *before* publishing, so it contains no publications and an empty `repo/` directory. `publish-debs.rb` creates publications without dropping existing ones, so it would fail on a state that has them. And an uploaded `repo/` would contain symlinks into a temporary directory that no longer exists. + * Every distribution is republished, because the upload replaces the whole public tree with the local one. A distribution that wasn't republished would disappear from it. * Each distribution in the archive is its own Aptly repo and publication, named after the distribution, like in the live repo. ### YUM specifics @@ -224,11 +228,14 @@ A run with nothing to archive exits without uploading anything. * a dry run uploads nothing; * the right packages end up in each repository, and the two never overlap; + * distributions with nothing to archive are left alone; * all metadata is signed; + * the uploaded state has no publications, and `publish-debs.rb` can still publish afterwards; * a second run changes nothing; - * a rerun after a failure between the archive and live uploads completes correctly. + * a rerun after a failure between the archive and live uploads completes correctly; + * a second archival round keeps previously archived distributions intact. -It needs only Docker. It is not run by CI. Run it after changing either script: +The seeded state is built the way `publish-debs.rb` builds it. It needs only Docker. It is not run by CI. Run it after changing either script: ~~~bash ./internal-scripts/ci-cd/archive/test/run-tests.sh