diff --git a/AGENTS.md b/AGENTS.md index 311f5f1..9e4dc1e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -120,6 +120,7 @@ monitoring/ # Grafana dashboards + Prometheus alert rules (maintaine sponsord/ # sponsord's (+ promtool unit tests), Ansible/Compose only iroh-relay/ # the relay's (+ promtool unit tests, exported-metrics.txt), Ansible and Compose docs/ # cross-path operator docs: node-operators.md + publishers.md (the front doors), requirements.md, lifecycle.md +examples/collection-fleet/ # a consumer project for the Galaxy collections: requirements.yml, ansible.cfg, .gitignore (the inventory comes from ansible/inventory/'s examples) scripts/ # upstream-mirror generators, the compose Alloy config generator (render-compose-alloy.sh), the decdn bump (open-decdn-bump.sh, bump_decdn.py), the release script (release.sh, git-cliff: ../cliff.toml) and its gate (+ its Artifact Hub changes generator), the molecule driver (molecule.sh), ansible/Makefile's LIMIT preflight (limit-guard.sh) ``` @@ -718,9 +719,21 @@ which depends on `decdn.node`). Each overlay lives in `ansible/galaxy/` — there is **no** `galaxy.yml` at the `ansible/` root (that would make ansible-lint treat the deploy project as a collection). -A new role goes in one `roles.txt`. Build/validate from `ansible/` with `make build` / +A new role goes in one `roles.txt`. **The playbooks ship too**, so a consumer needs only +an inventory (`ansible-playbook decdn.publisher.site`): `playbooks.txt` lists what each +collection takes from `ansible/playbooks/` (a new playbook goes in at least one; every +`import_playbook` target in the same one, so a run never mixes two releases' +`group_vars/`), `playbooks/group_vars/` ships beside them (a collection playbook loads +it as this project does, same precedence), and `galaxy/stage-playbooks.py` rewrites +role references to FQCNs at staging only: never qualify them, or add `collections:`, in +`ansible/playbooks/`, where an installed collection would then shadow `roles/`. +`decdn.node`'s node-only `backup.yml`/`decommission.yml` are a second source in +`galaxy/node/playbooks/`, kept equal to the fleet-wide node plays by `make +test-scripts`: change both. Build/validate from `ansible/` with `make build` / `make galaxy-check` (both; `-node`/`-publisher` suffixes for one; the root's -`galaxy-build`/`galaxy-check` run both). **Publishing** is per artifact, each on +`galaxy-build`/`galaxy-check` run both; each check also installs the tarball and runs +`galaxy/check-playbooks.sh`: every playbook's syntax by FQCN and, through a probe +collection playbook, the firewall holes its `group_vars/` give). **Publishing** is per artifact, each on its own version: `release-collection.yml` on a `node-collection-vX.Y.Z` or `publisher-collection-vX.Y.Z` tag (the repo's Latest follows `decdn.node`; release.sh refuses a publisher release until a node tag satisfies its `decdn.node` constraint), `release-chart.yml` on a diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ffa8d69..edea5cc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -42,9 +42,9 @@ targets, so a local pass means a CI pass. Deploy targets live in | `make lint-alloy` | renders `roles/grafana_alloy`'s templates and validates them with the **real** digest-pinned Alloy binary. The molecule stub exits 0 for everything, so this is the only gate that proves the config loads. It also checks that `compose/alloy/config.alloy` (the Compose `alloy` profile's render of the same template) is current, and runs it in Alloy: every component healthy, each daemon scraped only under the profiles that start it. `ALLOY_BIN=` skips the download. | | `make lint-compose` | reads `compose/compose.yaml` as written (`yq`) against `compose/tests/bind-sources.jq` (every bind mount's `create_host_path`), then renders it with every profile on (with its example env, without env files, with an empty `.env`) and asserts `compose/tests/invariants.jq`, `inline-env.jq` and `fail-closed.jq` | | `make lint-cloud-init` | `cloud-init schema` on `cloud-init/user-data-node.yaml` and `cloud-init/user-data-publisher.yaml`, then `cloud-init/tests/lint.py` on each: only the templates' top-level modules and no YAML anchors, no secrets (only the bootstrap's own files, once each, plain `content` only, no secret-looking keys or assignments), no hardening skip, `release` installs verified against the vendored keys with a host-generated node wallet (trust knobs only in their own group's `vars`, no moved keyring or secret path), only localhost, in `decdn_nodes` and/or `sponsord_hosts` (an origin only beside `decdn_nodes` and with a backend, every backend key only in `decdn_origin_nodes.vars`, the onramp only beside sponsord), a keyed admin account, `runcmd` exactly stage 1, shellcheck-clean scripts, and a collection lock that covers `ansible/requirements.yml` (needs `cloud-init`, `shellcheck`, `yq`). `CLOUD_INIT_FILE=` checks your own filled-in copy. | -| `make test-scripts` | `tests/scripts-test.sh`: the `ansible/Makefile` scoping guards (dry runs, and `limit-guard.sh` on a fixture inventory), the molecule driver's selection guards and locks (no containers), the split scenarios' shared inventories, the release gate, `scripts/release.sh` on a fixture repo and `pr-title.yml`'s check (the former skipped without `git-cliff` or `ssh-keygen` on PATH, failed in CI), the negative cases of `lint-compose` and `lint-cloud-init` (the latter skipped without `cloud-init` on PATH), `compose/decdn-compose`'s unit tests (`compose/tests/test_decdn_compose.py`), `scripts/bump_decdn.py`'s (`scripts/tests/test_bump_decdn.py`), that `compose/alloy/config.alloy` is a fresh render of the `grafana_alloy` template (`scripts/render-compose-alloy.sh --check`; needs ansible-core, failed in CI without it) and that Compose's `grafana/alloy` image is `grafana_alloy_version`, and the cloud-init bootstrap's contracts: `baseline-plays.sh` on fixtures and on the real playbooks (every node.yml, origin.yml and sponsord.yml play runs `baseline` tagged `baseline`), the two templates' shared stage 1, login hint and `final_message`, `bootstrap.sh`'s groups against lint.py's, and lint.py's `FORBIDDEN_VARS` against the roles' keyrings and secret paths. `UPSTREAM=` adds the sync generators' exit codes. | +| `make test-scripts` | `tests/scripts-test.sh`: the `ansible/Makefile` scoping guards (dry runs, and `limit-guard.sh` on a fixture inventory), the molecule driver's selection guards and locks (no containers), the split scenarios' shared inventories, the release gate, `scripts/release.sh` on a fixture repo and `pr-title.yml`'s check (the former skipped without `git-cliff` or `ssh-keygen` on PATH, failed in CI), the negative cases of `lint-compose` and `lint-cloud-init` (the latter skipped without `cloud-init` on PATH), `compose/decdn-compose`'s unit tests (`compose/tests/test_decdn_compose.py`), `scripts/bump_decdn.py`'s (`scripts/tests/test_bump_decdn.py`), that `compose/alloy/config.alloy` is a fresh render of the `grafana_alloy` template (`scripts/render-compose-alloy.sh --check`; needs ansible-core, failed in CI without it) and that Compose's `grafana/alloy` image is `grafana_alloy_version`, and the cloud-init bootstrap's contracts: `baseline-plays.sh` on fixtures and on the real playbooks (every node.yml, origin.yml and sponsord.yml play runs `baseline` tagged `baseline`), the two templates' shared stage 1, login hint and `final_message`, `bootstrap.sh`'s groups against lint.py's, and lint.py's `FORBIDDEN_VARS` against the roles' keyrings and secret paths; and the Galaxy collections' shipped playbooks: every playbook in a `playbooks.txt`, `galaxy/stage-playbooks.py` on both collections and its refusals, that it changes only role names, `decdn.node`'s `backup.yml`/`decommission.yml` against the fleet-wide node plays in both directions (needs yq, failed in CI without it), and the two roles' shared `*_backup_local_dir` default. `UPSTREAM=` adds the sync generators' exit codes. | | `make security` | KICS IaC scan of `ansible/`, the rendered chart and `compose/` (digest-pinned engine, fail on HIGH) | -| `make galaxy-check` | build the `decdn.node` and `decdn.publisher` collections and run galaxy-importer's checks on each (`make -C ansible galaxy-check-node` / `galaxy-check-publisher` for one) | +| `make galaxy-check` | build the `decdn.node` and `decdn.publisher` collections, run galaxy-importer's checks on each, then install each and check its playbooks by FQCN, firewall holes included (`ansible/galaxy/check-playbooks.sh`; `make -C ansible galaxy-check-node` / `galaxy-check-publisher` for one) | `ansible-lint` is **not** a per-commit hook (it needs the collections installed). Run it with `make lint-ansible`, or `pre-commit run ansible-lint --hook-stage manual`. diff --git a/Makefile b/Makefile index f7f28af..ef4c4a3 100644 --- a/Makefile +++ b/Makefile @@ -199,5 +199,5 @@ molecule-serial: ## same selection, one scenario at a time (readable output galaxy-build: ## stage + build the decdn.node and decdn.publisher Galaxy collection artifacts $(MAKE) -C ansible build -galaxy-check: ## build + validate both collections (galaxy-importer) +galaxy-check: ## build + validate both collections (galaxy-importer, shipped playbooks by FQCN) $(MAKE) -C ansible galaxy-check diff --git a/README.md b/README.md index 1b1ae05..4a833f6 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ Four ways to deploy the node, cache or origin: | Path | For | Start here | |------|-----|------------| | **Docker Compose** (`compose/`), recommended | One host that already runs Docker. The quickest start, [below](#quickstart-docker-compose). Profiles for a cache node or an origin, the onboarding sponsor (`sponsord` + its onramp), an iroh relay, an iroh DNS server and Grafana Alloy, driven by the `decdn-compose` wrapper (set-up, preflight, guarded restarts). | [`compose/README.md`](compose/README.md) | -| **Ansible** (`ansible/`) | VMs and bare metal, one node or a fleet. Hardens the host too (firewall, SSH, patching). Also published on Ansible Galaxy as the [`decdn.node`](https://galaxy.ansible.com/ui/repo/published/decdn/node/) and [`decdn.publisher`](https://galaxy.ansible.com/ui/repo/published/decdn/publisher/) collections. | [`ansible/README.md`](ansible/README.md) | +| **Ansible** (`ansible/`) | VMs and bare metal, one node or a fleet. Hardens the host too (firewall, SSH, patching). Also published on Ansible Galaxy as the [`decdn.node`](https://galaxy.ansible.com/ui/repo/published/decdn/node/) and [`decdn.publisher`](https://galaxy.ansible.com/ui/repo/published/decdn/publisher/) collections, roles and playbooks, so a fleet needs only an inventory of its own ([`examples/collection-fleet/`](examples/collection-fleet/README.md)). | [`ansible/README.md`](ansible/README.md) | | **cloud-init** (`cloud-init/`) | One VM, no control machine: paste the user-data into your provider's "create server" form. It runs the Ansible playbook on the host itself, hardening included. One template per kind of operator: a cache node, or a publisher's origin with the onboarding sponsor (`sponsord` + its onramp). | [`cloud-init/README.md`](cloud-init/README.md) | | **Helm** (`charts/decdn-node/`) | Kubernetes, one release per node, cache or origin. Published to `oci://ghcr.io/decdn/charts/decdn-node`, signed, and listed on [Artifact Hub](https://artifacthub.io/packages/helm/decdn-node/decdn-node). | [`charts/decdn-node/README.md`](charts/decdn-node/README.md) | @@ -131,6 +131,7 @@ This is the canonical statement; the per-path READMEs add only what is specific | [`charts/decdn-node/`](charts/decdn-node/README.md) | The Helm chart; it renders the node's dashboards and alert rules from `monitoring/`. | | [`monitoring/`](monitoring/README.md) | The deCDN Grafana dashboards and Prometheus alert rules, for the node (`decdn-node/`, every deploy path), sponsord (`sponsord/`, Ansible and Compose) and the iroh relay (`iroh-relay/`, Ansible and Compose). | | [`docs/`](docs/requirements.md) | Cross-path operator docs: the two front doors ([node operators](docs/node-operators.md), [publishers](docs/publishers.md)), requirements, lifecycle. | +| [`examples/collection-fleet/`](examples/collection-fleet/README.md) | A starting point for a private repository that deploys from the Galaxy collections alone, with only its own inventory. | | `scripts/` | Generators for the upstream mirrors (network profiles) and Compose's Alloy config (`render-compose-alloy.sh`), and the release gate. | | `Makefile` | Lint, test and security targets; CI runs the same ones. `make help` lists them. | | `.github/workflows/` | CI (`ci.yml`, `molecule.yml`), releases (`release-collection.yml`, `release-chart.yml`), the weekly upstream drift check. | diff --git a/RELEASING.md b/RELEASING.md index 81578b0..642062f 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -55,9 +55,11 @@ would commit, and changes no file. The script: 1. **Refuses unconventional commits.** It looks at the commits since the artifact's last tag that touched what it ships: for a collection, the roles its - `ansible/galaxy//roles.txt` lists (the list `galaxy/build.sh` reads), that - overlay, `ansible/galaxy/build.sh` and `LICENSE`; for the chart, `charts/decdn-node/` - and `monitoring/decdn-node/`. + `ansible/galaxy//roles.txt` lists and the playbooks its `playbooks.txt` + lists (the lists `galaxy/build.sh` and `galaxy/stage-playbooks.py` read) with + `ansible/playbooks/group_vars/`, that overlay, `ansible/galaxy/build.sh`, + `ansible/galaxy/stage-playbooks.py` and `LICENSE`; for the chart, + `charts/decdn-node/` and `monitoring/decdn-node/`. git-cliff drops a commit it cannot parse as a Conventional Commit, from the changelog and from the bump, and only logs a count. That covers an unconventional subject, but also an empty scope (`feat(): …`) and a body right under the subject diff --git a/ansible/.ansible-lint b/ansible/.ansible-lint index 60423e7..8fe26e7 100644 --- a/ansible/.ansible-lint +++ b/ansible/.ansible-lint @@ -18,6 +18,13 @@ skip_list: exclude_paths: - build/ - galaxy/ + # Imports the decdn.node collection's playbooks by FQCN, which exist only once the + # scenario's prepare has built and installed the collection, so ansible-lint's + # syntax check cannot load it. Of the playbooks it imports, node.yml is linted + # here; the node-only backup.yml and decommission.yml live under the excluded + # galaxy/, linted by galaxy-importer in the build and kept equal to playbooks/'s + # node plays by make test-scripts. + - molecule/collection/converge.yml # Task files a molecule scenario includes (ansible-lint would otherwise read them as # playbooks, and a `tasks/` directory would make the scenario look like a role). diff --git a/ansible/Makefile b/ansible/Makefile index a94dd72..e3cd477 100644 --- a/ansible/Makefile +++ b/ansible/Makefile @@ -263,8 +263,9 @@ molecule-list: @$(molecule_env) ../scripts/molecule.sh list # --- Galaxy collections (decdn.node, decdn.publisher) ------------------------- -# Stage each collection's roles (galaxy//roles.txt) into a clean collection -# tree and build the artifact under build/: decdn.node is what a node operator runs +# Stage each collection's roles (galaxy//roles.txt) and playbooks +# (galaxy//playbooks.txt) into a clean collection tree and build the +# artifact under build/: decdn.node is what a node operator runs # (baseline + decdn_node + grafana_alloy), decdn.publisher what a publisher runs beside # its origins (sponsord + sponsord_onramp + iroh_relay + iroh_dns_server). See # galaxy/*/README.md. Publishing runs from a node-collection-vX.Y.Z or @@ -281,8 +282,14 @@ build-node build-publisher: build-%: # pipeline exit (set -o pipefail catches a crash / "No module named …" that prints # no error keyword) AND the importer's explicit success line (it can log content # errors while still exiting 0). Warnings are kept in build/importer-.log. +# Then galaxy/check-playbooks.sh installs the artifact and checks its playbooks as a +# consumer runs them (syntax by FQCN, the firewall holes their group_vars/ give), +# which the importer's ansible-lint, seeing one collection outside any collections +# path, cannot. decdn.publisher installs with the decdn.node it depends on. galaxy-check: galaxy-check-node galaxy-check-publisher +galaxy-check-publisher: build-node + galaxy-check-node galaxy-check-publisher: galaxy-check-%: build-% @set -o pipefail; \ tarball=$$(ls -1 build/decdn-$*-*.tar.gz); \ @@ -291,3 +298,4 @@ galaxy-check-node galaxy-check-publisher: galaxy-check-%: build-% || ! grep -q "Importer processing completed successfully" build/importer-$*.log; then \ echo "FAIL: galaxy-importer reported errors or did not complete (see build/importer-$*.log)"; exit 1; \ fi + ./galaxy/check-playbooks.sh $* diff --git a/ansible/README.md b/ansible/README.md index 68b897a..2ff8060 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -1,10 +1,11 @@ # ansible — deCDN deployment Declarative Ansible project for deploying deCDN nodes, one or a fleet, over a -hardened host baseline. The roles also ship as two collections on Ansible Galaxy: -[`decdn.node`](https://galaxy.ansible.com/ui/repo/published/decdn/node/) (what a node operator runs) and +hardened host baseline. The roles and these playbooks also ship as two collections on +Ansible Galaxy: [`decdn.node`](https://galaxy.ansible.com/ui/repo/published/decdn/node/) (what a node operator runs) and [`decdn.publisher`](https://galaxy.ansible.com/ui/repo/published/decdn/publisher/) (what a publisher runs beside its -origin nodes). +origin nodes), so a fleet can run from an inventory and the collections alone (see +[Packaging as Galaxy collections](#packaging-as-galaxy-collections-decdnnode-decdnpublisher)). | Playbook | Purpose | Make target | |----------|---------|-------------| @@ -514,26 +515,42 @@ the RPC URL when it is not provisioned on the host instead). Highlights: ## Packaging as Galaxy collections (`decdn.node`, `decdn.publisher`) -The seven roles are also packaged as two collections, published on Ansible Galaxy, for -operators who bring their own playbooks (`ansible-galaxy collection install -decdn.node` or `decdn.publisher`): +The seven roles and the playbooks are also packaged as two collections, published on +Ansible Galaxy (`ansible-galaxy collection install decdn.node` or `decdn.publisher`), +for operators who run a fleet from an inventory and the collections alone +(`ansible-playbook -i inventory/hosts.yml decdn.publisher.site`) or bring their own playbooks: -| Collection | Roles | For | -|---|---|---| -| **`decdn.node`** | `baseline`, `decdn_node`, `grafana_alloy` | node operators, and a publisher's origin nodes | -| **`decdn.publisher`** | `sponsord`, `sponsord_onramp`, `iroh_relay`, `iroh_dns_server` | publishers; depends on `decdn.node` | +| Collection | Roles | Playbooks | For | +|---|---|---|---| +| **`decdn.node`** | `baseline`, `decdn_node`, `grafana_alloy` | `node`, and node-only `backup` and `decommission` | node operators | +| **`decdn.publisher`** | `sponsord`, `sponsord_onramp`, `iroh_relay`, `iroh_dns_server` | every playbook here (`site`, `node`, `publisher`, `origin`, `sponsord`, `iroh_relay`, `iroh_dns_server`, `backup`, `decommission`) | publishers; depends on `decdn.node` | Each collection's overlay lives in [`galaxy//`](galaxy/) (`galaxy.yml`, the -collection `README.md`/`CHANGELOG.md`, `meta/runtime.yml`, and `roles.txt`, the one -list of the roles it ships). It is deliberately **not** a `galaxy.yml` at the project -root: `galaxy/build.sh ` stages only that collection's roles into a clean +collection `README.md`/`CHANGELOG.md`, `meta/runtime.yml`, `roles.txt` and +`playbooks.txt`, the lists of the roles and playbooks it ships, and for `decdn.node` +a `playbooks/` holding its node-only `backup.yml` and `decommission.yml`). It is +deliberately **not** a `galaxy.yml` at the project root: `galaxy/build.sh ` +stages only that collection's roles and playbooks into a clean `ansible_collections/decdn//` tree and builds the artifact, so this project stays a plain Ansible project (the `make deploy`/`lint` flow is unchanged). Every role -under `roles/` ships in exactly one collection (`make test-scripts` checks). +under `roles/` ships in exactly one collection, and every playbook under `playbooks/` in +at least one (`make test-scripts` checks). + +The playbooks ship as they are here, with `playbooks/group_vars/` beside them: a +collection playbook loads the `group_vars/` next to it as this project does, with the +same precedence, so the firewall holes and Alloy toggles come along unchanged. +`galaxy/stage-playbooks.py` qualifies their role references (`role: baseline` → +`decdn.node.baseline`) at staging; the playbooks here stay unqualified, so an +installed collection can never stand in for `roles/` on a deploy. Every playbook a +collection imports must ship in it too, so a run never leaves the collection and never +mixes two releases' `group_vars/`: that is why `decdn.publisher` carries its own +`node.yml`. `decdn.node`'s `backup.yml` and `decommission.yml` are the node plays of +the fleet-wide ones (`make test-scripts` keeps them equal). ```bash make build # stage + build -> build/decdn-{node,publisher}-.tar.gz -make galaxy-check # build + validate both with galaxy-importer (the checks Galaxy runs) +make galaxy-check # build + validate both: galaxy-importer (the checks Galaxy runs), then + # galaxy/check-playbooks.sh (the playbooks by FQCN, their firewall holes) make build-node # or galaxy-check-node, build-publisher, galaxy-check-publisher: one ``` @@ -543,5 +560,6 @@ change. Publishing happens from a `node-collection-vX.Y.Z` or each collection on its own version (the Helm chart is released separately), cut on `main` by `scripts/release.sh`: see [RELEASING.md](../RELEASING.md). Each `galaxy//CHANGELOG.md` section is generated at release from the -conventional commit subjects that touched that collection's roles, its overlay or -`galaxy/build.sh`, so give the PR a title that reads as a changelog entry. +conventional commit subjects that touched that collection's roles, its playbooks +(with `playbooks/group_vars/`), its overlay, `galaxy/build.sh` or +`galaxy/stage-playbooks.py`, so give the PR a title that reads as a changelog entry. diff --git a/ansible/galaxy/build.sh b/ansible/galaxy/build.sh index feb9006..4a1b244 100755 --- a/ansible/galaxy/build.sh +++ b/ansible/galaxy/build.sh @@ -5,13 +5,15 @@ # galaxy/build.sh publisher decdn.publisher: what a publisher runs beside its origins # # Only the roles listed in galaxy//roles.txt ship, with that directory's -# overlay (galaxy.yml, README.md, CHANGELOG.md, meta/). All deploy machinery -# (inventory, Makefile, ansible.cfg) is excluded BY CONSTRUCTION — it is simply +# overlay (galaxy.yml, README.md, CHANGELOG.md, meta/), and the playbooks its +# playbooks.txt lists with their group_vars/ (galaxy/stage-playbooks.py). All deploy +# machinery (inventory, Makefile, ansible.cfg) is excluded BY CONSTRUCTION — it is simply # never copied into the staging tree. This keeps the artifact clean and leaves the # internal project untouched (no galaxy.yml at the project root, so ansible-lint # and ansible still see a plain project). # -# Output: ansible/build/decdn--.tar.gz +# Output: ansible/build/decdn--.tar.gz, or under +# $DECDN_GALAXY_BUILD_DIR (molecule/collection builds into its ephemeral dir). set -euo pipefail here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # ansible/galaxy @@ -27,7 +29,7 @@ overlay="$here/$collection" mapfile -t roles < <(grep -vE '^[[:space:]]*(#|$)' "$overlay/roles.txt") ((${#roles[@]} > 0)) || { echo "build.sh: $overlay/roles.txt lists no role" >&2; exit 1; } -build_dir="$ansible_dir/build" +build_dir="${DECDN_GALAXY_BUILD_DIR:-$ansible_dir/build}" stage="$build_dir/ansible_collections/decdn/$collection" echo "staging decdn.$collection -> $stage" @@ -43,6 +45,10 @@ for role in "${roles[@]}"; do cp -R "$ansible_dir/roles/$role" "$stage/roles/$role" done +# The playbooks in playbooks.txt (+ their group_vars/ and the overlay's playbooks/), +# role references rewritten to FQCNs and checked. +python3 "$here/stage-playbooks.py" "$collection" "$stage" + # Collection overlay + license (the artifact must be self-contained). cp "$overlay/galaxy.yml" "$stage/galaxy.yml" cp "$overlay/README.md" "$stage/README.md" diff --git a/ansible/galaxy/check-playbooks.sh b/ansible/galaxy/check-playbooks.sh new file mode 100755 index 0000000..641fb3c --- /dev/null +++ b/ansible/galaxy/check-playbooks.sh @@ -0,0 +1,68 @@ +#!/usr/bin/env bash +# Check the playbooks the built collections ship, as a Galaxy consumer runs them: +# +# galaxy/check-playbooks.sh node|publisher (`make galaxy-check-` runs it) +# +# Installs the collection's tarball under build/ (decdn.publisher's with the decdn.node +# one it depends on) into a scratch collections path, then +# - syntax-checks every playbook it ships by its FQCN (decdn.node.node, …), which +# resolves each static role reference against the installed collections (not an +# include_role's: stage-playbooks.py checks those names and their tasks_from, and +# molecule/collection runs decdn.node's); +# - runs tests/firewall-holes/check.sh on its playbooks/, which runs a probe AS a +# collection playbook: the firewall holes must come out as they do from +# ansible/playbooks/, so the shipped group_vars/ load and agree. +# galaxy-importer cannot do the first: its ansible-lint sees one collection, outside +# any collections path, so it only warns that it cannot find the roles. +# The roles' own dependencies (devsec.hardening, ansible.posix) come from `make deps` +# (ansible/collections) or ANSIBLE_COLLECTIONS_PATH. +set -euo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # ansible/galaxy +ansible_dir="$(cd "$here/.." && pwd)" # ansible/ +build_dir="$ansible_dir/build" + +collection="${1:-}" +case "$collection" in + node) needs=(node) ;; + publisher) needs=(node publisher) ;; + *) echo "usage: $0 node|publisher" >&2; exit 2 ;; +esac +shopt -s nullglob +tarballs=() +for c in "${needs[@]}"; do + found=("$build_dir"/decdn-"$c"-*.tar.gz) + ((${#found[@]} == 1)) || { echo "check-playbooks.sh: want one decdn-$c tarball under build/ (make build-$c): ${found[*]:-none}" >&2; exit 1; } + tarballs+=("${found[0]}") +done + +work="$(mktemp -d)" +trap 'rm -rf "$work"' EXIT + +# --no-deps: decdn.publisher's dependency is the decdn.node tarball installed with +# it, and the rest is already vendored. Ansible refuses a non-blocking stdio, so the +# output goes to a file, never a pipe. +ansible-galaxy collection install --no-deps -p "$work" "${tarballs[@]}" "$work/install.log" 2>&1 \ + || { cat "$work/install.log" >&2; echo "check-playbooks.sh: could not install the built collections" >&2; exit 1; } +export ANSIBLE_COLLECTIONS_PATH="$work:$ansible_dir/collections${ANSIBLE_COLLECTIONS_PATH:+:$ANSIBLE_COLLECTIONS_PATH}" +export ANSIBLE_DEPRECATION_WARNINGS=0 +# Hermetic: no roles path (an exported ANSIBLE_ROLES_PATH, ~/.ansible/roles or a +# config's roles_path could stand in for a role the collections lack) and no config +# file from the caller. +mkdir "$work/no-roles" +export ANSIBLE_ROLES_PATH="$work/no-roles" +unset ANSIBLE_CONFIG + +inventory="$ansible_dir/tests/firewall-holes/inventory.yml" +playbooks="$work/ansible_collections/decdn/$collection/playbooks" +n=0 +for pb in "$playbooks"/*.yml; do + fqcn="decdn.$collection.$(basename "$pb" .yml)" + # From the scratch dir, so no ansible.cfg is read. + (cd "$work" && ansible-playbook -i "$inventory" "$fqcn" --syntax-check "$work/syntax.log" 2>&1) \ + || { cat "$work/syntax.log" >&2; echo "check-playbooks.sh: $fqcn fails its syntax check" >&2; exit 1; } + n=$((n + 1)) +done +((n > 0)) || { echo "check-playbooks.sh: decdn.$collection ships no playbook" >&2; exit 1; } +echo "ok decdn.$collection: $n playbooks pass a syntax check by FQCN" +"$ansible_dir/tests/firewall-holes/check.sh" "$playbooks" diff --git a/ansible/galaxy/node/README.md b/ansible/galaxy/node/README.md index 208c6ad..b7bed00 100644 --- a/ansible/galaxy/node/README.md +++ b/ansible/galaxy/node/README.md @@ -3,7 +3,8 @@ Deploy and harden a **public [deCDN](https://decdn.org) node**: what a **node operator** runs, and the origin nodes a **publisher** runs. This collection is part of the public, reusable slice of the [`decdn/devops`](https://github.com/decdn/devops) -repository — three roles and nothing else. A publisher's other services (the +repository: three roles, and the playbooks that deploy, back up and decommission +nodes with them (see [Usage](#usage)). A publisher's other services (the `sponsord` sponsor and its onramp, an iroh relay, an iroh DNS server) are the [`decdn.publisher`](https://github.com/decdn/devops/tree/main/ansible/galaxy/publisher) collection, which depends on this one. @@ -36,7 +37,7 @@ Or pin it in a `requirements.yml`: ```yaml collections: - name: decdn.node - version: ">=0.1.0" + version: ">=0.2.0" # the first release with the playbooks ``` Each release is listed in the @@ -55,8 +56,53 @@ ansible-galaxy collection install --force build/decdn-node-*.tar.gz ## Usage -A minimal node playbook — baseline first (so the admin key lands before SSH -hardening), then the node: +### The shipped playbooks + +The collection ships the node playbooks `decdn/devops` deploys its own fleets with +([`ansible/playbooks/`](https://github.com/decdn/devops/tree/main/ansible/playbooks)), +roles qualified: + +| Playbook | Runs | +|----------|------| +| `decdn.node.node` | `decdn_nodes` outside `decdn_origin_nodes`: baseline, then Grafana Alloy (opt-in, off by default), then the node | +| `decdn.node.backup` | an encrypted backup of each node's identity (or, with `-e decdn_backup_scope=full`, its state), one node at a time | +| `decdn.node.decommission` | stops the node and removes its unit, then the Alloy agent, after one typed confirmation of the run's hosts; keeps the keys, never touches the chain | + +```bash +ansible-galaxy collection install -r requirements.yml +ansible-playbook -i inventory/hosts.yml decdn.node.node --check --diff +ansible-playbook -i inventory/hosts.yml decdn.node.node +ansible-playbook -i inventory/hosts.yml decdn.node.decommission -l +``` + +Start the inventory from +[`hosts-node.yml.example`](https://github.com/decdn/devops/blob/main/ansible/inventory/hosts-node.yml.example) +and the `group_vars/` and `host_vars/` beside it. The playbooks open the node's QUIC +port in the baseline firewall themselves (`baseline_extra_inbound`, from their own +`group_vars/`, which ship beside them), and `decommission` refuses more hosts than +`decdn_decommission_max_hosts` (default 1). + +- **Overrides.** The playbooks' `group_vars/decdn_nodes.yml` sets + `baseline_extra_inbound`, and playbook group vars beat every inventory group var: + change it per host, in `host_vars` (or with `-e`). The Alloy toggles come from their + `group_vars/all.yml`, which beats the inventory's `group_vars/all.yml` and group vars + written inline in `hosts.yml` but not its `group_vars/.yml` or `host_vars`. + Importing a shipped playbook from one of your own changes none of this: its plays + read the collection's `group_vars/`, not those beside your playbook. +- **Backups.** `decdn.node.backup` fetches each identity archive to `backups/` beside + your project's `ansible.cfg` (never into the installed collection). Without a + project `ansible.cfg`, set `decdn_backup_local_dir`. +- **Scope.** The repository's Makefile guards (a `LIMIT` preflight, a required `LIMIT` + for decommission) do not ship: scope a run with `-l` yourself. +- **Publisher hosts.** A publisher's origin nodes, sponsord, relays and DNS servers + are `decdn.publisher`'s playbooks. Decommission a host that also runs one of them + with `decdn.publisher.decommission`: `decdn.node.decommission` would tear down the + Alloy agent they share and leave them running. + +### Your own playbook + +To compose the roles yourself, a minimal node playbook — baseline first (so the admin +key lands before SSH hardening), then the node: ```yaml - name: Provision a hardened deCDN node diff --git a/ansible/galaxy/node/galaxy.yml b/ansible/galaxy/node/galaxy.yml index 6b94b37..bca6e06 100644 --- a/ansible/galaxy/node/galaxy.yml +++ b/ansible/galaxy/node/galaxy.yml @@ -47,8 +47,8 @@ documentation: https://github.com/decdn/devops/tree/main/ansible homepage: https://decdn.org issues: https://github.com/decdn/devops/issues # No build_ignore. It used to carry "*.example" as a guard against shipping an -# inventory secret template, but build.sh stages ONLY roles/ + this overlay — the -# inventory never enters the tree, so the pattern excluded nothing real while +# inventory secret template, but build.sh stages ONLY roles/, playbooks/ (with their +# group_vars/) and this overlay — the inventory never enters the tree, so the pattern excluded nothing real while # silently dropping roles/decdn_node/files/decdn.env.example, which external # operators need (it is the template for the host-provisioned 0600 env file). # Add a NARROW, path-anchored pattern here if something must ever be excluded. diff --git a/ansible/galaxy/node/playbooks.txt b/ansible/galaxy/node/playbooks.txt new file mode 100644 index 0000000..f356383 --- /dev/null +++ b/ansible/galaxy/node/playbooks.txt @@ -0,0 +1,5 @@ +# Playbooks galaxy/build.sh ships in decdn.node, from ansible/playbooks/ (with its +# group_vars/), beside the overlay playbooks/ in this directory (the node-scoped +# backup.yml and decommission.yml). Role references are rewritten to FQCNs at +# staging (galaxy/stage-playbooks.py). +node.yml diff --git a/ansible/galaxy/node/playbooks/backup.yml b/ansible/galaxy/node/playbooks/backup.yml new file mode 100644 index 0000000..b9c5ab3 --- /dev/null +++ b/ansible/galaxy/node/playbooks/backup.yml @@ -0,0 +1,20 @@ +--- +# decdn.node's backup: each node's identity (default) or full state, encrypted on +# the host to decdn_backup_age_recipients (public keys, required). +# ansible-playbook -i inventory/hosts.yml decdn.node.backup -l [-e decdn_backup_scope=full] +# The node play of ansible/playbooks/backup.yml, which decdn.publisher ships whole +# (with sponsord's credentials); tests/scripts-test.sh keeps the two plays equal. See +# roles/decdn_node/tasks/backup.yml for what is archived and docs/lifecycle.md for +# restore and host migration. +- name: Back up deCDN nodes + hosts: decdn_nodes + become: true + # One host at a time: a "full" backup stops the node for the copy, and a fleet + # should never lose every node at once. + serial: 1 + gather_facts: false + tasks: + - name: Back up this node + ansible.builtin.include_role: + name: decdn_node + tasks_from: backup diff --git a/ansible/galaxy/node/playbooks/decommission.yml b/ansible/galaxy/node/playbooks/decommission.yml new file mode 100644 index 0000000..4fe0ef3 --- /dev/null +++ b/ansible/galaxy/node/playbooks/decommission.yml @@ -0,0 +1,52 @@ +--- +# decdn.node's decommission: stop the node and remove its unit, then tear down the +# opt-in Grafana Alloy agent. Keeps the identity and keys (the node's keystore +# withdraws the bond) and does NOT touch the chain; the role's last task prints what +# is left to do. One typed confirmation per run (the sorted host list). +# ansible-playbook -i inventory/hosts.yml decdn.node.decommission -l +# On a host that also runs a publisher service (sponsord, a relay, a DNS server), use +# decdn.publisher.decommission: this one would tear down the Alloy agent they share. +# The node plays of ansible/playbooks/decommission.yml, which decdn.publisher ships +# whole (with sponsord, its onramp, iroh relays and DNS servers); +# tests/scripts-test.sh keeps the plays equal. Take a backup first +# (decdn.node.backup). See docs/lifecycle.md. + +# Before anything stops: the host cap, against every host this run decommissions. +- name: Check the hosts before anything stops + hosts: decdn_nodes + become: true + gather_facts: false + tasks: + - name: Refuse to decommission more hosts than allowed at once + ansible.builtin.assert: + that: + - decdn_decommission_max_hosts | default(1) | int > 0 + - ansible_play_hosts_all | length <= decdn_decommission_max_hosts | default(1) | int + fail_msg: >- + This run decommissions {{ ansible_play_hosts_all | length }} hosts + ({{ ansible_play_hosts_all | sort | join(', ') }}), more than + decdn_decommission_max_hosts ({{ decdn_decommission_max_hosts | default(1) }}). + Scope it with LIMIT (-l), or raise decdn_decommission_max_hosts deliberately. + quiet: true + +- name: Decommission deCDN nodes + hosts: decdn_nodes + become: true + tasks: + - name: Stop and remove the node service + ansible.builtin.include_role: + name: decdn_node + tasks_from: decommission + +# grafana_alloy's disabled path removes only an installation this role wrote (it +# checks the managed-by marker in the unit), so this is a no-op on a host that never +# enabled Grafana Cloud or runs a foreign Alloy. +- name: Tear down the Grafana Alloy agent + hosts: decdn_nodes + become: true + tasks: + - name: Tear down the Grafana Alloy agent this repo installed + ansible.builtin.include_role: + name: grafana_alloy + vars: + decdn_grafana_cloud_enabled: false diff --git a/ansible/galaxy/publisher/README.md b/ansible/galaxy/publisher/README.md index 3bbbd76..73f8a6b 100644 --- a/ansible/galaxy/publisher/README.md +++ b/ansible/galaxy/publisher/README.md @@ -7,7 +7,8 @@ An origin is a `decdn-node` with an origin backend, deployed with [`decdn.node`](https://github.com/decdn/devops/tree/main/ansible/galaxy/node)'s `decdn_node` role; this collection holds the publisher's other services. It is part of the public, reusable slice of the [`decdn/devops`](https://github.com/decdn/devops) -repository — four roles and nothing else: +repository: four roles, and the playbooks that deploy a whole publisher fleet with +them and `decdn.node`'s (see [Usage](#usage)): | Role | Purpose | |------|---------| @@ -23,8 +24,8 @@ repository — four roles and nothing else: sudo user. Facts must be gathered (each role derives its release target from the host architecture). - Collection dependencies (installed automatically with this collection): - `decdn.node (>=0.1.0)`, for the host baseline, observability and the origin nodes. - None of this collection's roles calls into it: the playbooks that run them do. + `decdn.node (>=0.2.0)`, for the host baseline, observability and the origin and + cache nodes. None of this collection's roles calls into it: its playbooks do. ## Install @@ -40,7 +41,7 @@ Or pin it in a `requirements.yml`: ```yaml collections: - name: decdn.publisher - version: ">=0.1.0" + version: ">=0.2.0" # the first release with the playbooks ``` Each release is listed in the @@ -60,8 +61,69 @@ ansible-galaxy collection install --force build/decdn-publisher-*.tar.gz ## Usage -A minimal sponsor playbook — `decdn.node`'s baseline first (so the admin key lands -before SSH hardening), then the daemon, then its public onramp: +### The shipped playbooks + +The collection ships the playbooks `decdn/devops` deploys its own fleets with +([`ansible/playbooks/`](https://github.com/decdn/devops/tree/main/ansible/playbooks)), +roles qualified. Each play targets an inventory group and matches nothing unless the +inventory defines it, so `site` deploys whatever the inventory holds: + +| Playbook | Runs | +|----------|------| +| `decdn.publisher.site` | `node`, then `publisher`: everything in the inventory | +| `decdn.publisher.publisher` | `origin`, `sponsord`, `iroh_relay`, `iroh_dns_server`, in that order | +| `decdn.publisher.node` | cache nodes: `decdn_nodes` outside `decdn_origin_nodes` (the same playbook as `decdn.node.node`) | +| `decdn.publisher.origin` | origin nodes: `decdn_origin_nodes`, a child group of `decdn_nodes`; refuses one without an origin backend | +| `decdn.publisher.sponsord` | `sponsord_hosts`, then the onramp on `sponsord_onramp_hosts` (which must be `sponsord_hosts` too) | +| `decdn.publisher.iroh_relay` | `iroh_relay_hosts`, one at a time; in allowlist mode (the default) it first reads every node's endpoint ID for the allowlist | +| `decdn.publisher.iroh_dns_server` | `iroh_dns_server_hosts`, one at a time | +| `decdn.publisher.backup` | encrypted backups of the nodes' identities (or full state) and sponsord's credentials | +| `decdn.publisher.decommission` | stops and removes every service on the run's hosts (the onramp's Caddy is stopped and disabled, not removed), then the Alloy agent, after one typed confirmation of the hosts; keeps the keys, never touches the chain | + +```bash +ansible-galaxy collection install -r requirements.yml +ansible-playbook -i inventory/hosts.yml decdn.publisher.site --check --diff +ansible-playbook -i inventory/hosts.yml decdn.publisher.site +ansible-playbook -i inventory/hosts.yml decdn.publisher.decommission -l +``` + +Start the inventory from +[`hosts-publisher.yml.example`](https://github.com/decdn/devops/blob/main/ansible/inventory/hosts-publisher.yml.example) +and the `group_vars/` and `host_vars/` beside it. Beyond the roles, the playbooks +bring: + +- **the firewall holes**, from the groups a host is in: the node's udp/4433, Caddy's + tcp/80 + tcp/443 on an onramp host, the relay's tcp/80 + tcp/443 + udp/7842 and the + DNS server's tcp/443 + udp/53 + tcp/53, one list per host whatever the play, so a + co-located host renders one firewall. They and the Grafana Alloy per-daemon toggles + live in the playbooks' own `group_vars/`, which ship beside them; +- **the placement checks** the roles leave out: an origin must be a `decdn_nodes` host + too and have an origin backend, an onramp a `sponsord_hosts` one, a relay may not + share a host with an onramp (both want tcp/80 and tcp/443), and a DNS server with + neither (all want tcp/443); +- **the lifecycle guards**: `decommission` refuses more hosts than + `decdn_decommission_max_hosts` (default 1) and refuses while sponsord holds an + unconfirmed pool top-up. + +- **Overrides.** The playbooks' `group_vars/.yml` set `baseline_extra_inbound`, + and playbook group vars beat every inventory group var: change it per host, in + `host_vars` (or with `-e`). The Alloy toggles come from their `group_vars/all.yml`, + which beats the inventory's `group_vars/all.yml` and group vars written inline in + `hosts.yml`, but not its `group_vars/.yml` or `host_vars`, where you can + override them. Importing a shipped playbook from one of your own changes none of + this: its plays read the collection's `group_vars/`, not those beside your playbook. +- **Backups.** `decdn.publisher.backup` fetches the nodes' identity archives and + sponsord's credentials to `backups/` beside your project's `ansible.cfg` (never into + the installed collection). Without a project `ansible.cfg`, set + `decdn_backup_local_dir` and `sponsord_backup_local_dir`. +- **Scope.** The repository's Makefile guards (a `LIMIT` preflight, a required `LIMIT` + for decommission) do not ship: scope a run with `-l` yourself. + +### Your own playbook + +To compose the roles yourself, a minimal sponsor playbook — `decdn.node`'s baseline +first (so the admin key lands before SSH hardening), then the daemon, then its public +onramp: ```yaml - name: Provision sponsord and its onramp @@ -93,8 +155,8 @@ before SSH hardening), then the daemon, then its public onramp: # git-ignored secret.yml). ``` -This repository's own playbooks enforce three placement rules that the roles do not -check, so a playbook of your own must keep them: `sponsord_onramp` runs on a +Of the shipped playbooks' placement checks, three concern this collection's roles, +which do not check them, so a playbook of your own must keep them: `sponsord_onramp` runs on a `sponsord` host (it reads the daemon's API token and calls it on loopback); an iroh relay does not share a host with the onramp (both want tcp/80 and tcp/443); an iroh DNS server shares a host with neither (all want tcp/443). The relay and DNS server @@ -110,7 +172,7 @@ day-2 ops: Backends bind `127.0.0.1`. `sponsord` has no public listener. `sponsord_onramp` listens on loopback too: Caddy is its public side, so a host that runs it opens -tcp/80 and tcp/443 for Caddy in `baseline_extra_inbound` (this repo's playbooks +tcp/80 and tcp/443 for Caddy in `baseline_extra_inbound` (the shipped playbooks derive that from `sponsord_onramp_hosts`; from your own playbook, add them yourself). `iroh_relay` is public by design and terminates its own TLS: it binds tcp/80, tcp/443 and udp/7842 on every address (its metrics stay on loopback), so a relay host opens diff --git a/ansible/galaxy/publisher/galaxy.yml b/ansible/galaxy/publisher/galaxy.yml index 208ff46..abbf7c6 100644 --- a/ansible/galaxy/publisher/galaxy.yml +++ b/ansible/galaxy/publisher/galaxy.yml @@ -36,15 +36,19 @@ tags: - web3 # Collection-level dependencies. Every SHIPPED role uses ansible.builtin only # (sponsord_onramp's Caddy comes from the distro's apt), and none calls into -# decdn.node. The dependency is for the playbooks that run these roles: a publisher -# host takes decdn.node.baseline first and decdn.node.grafana_alloy for -# observability, and its origin nodes are decdn.node.decdn_node. scripts/release.sh -# refuses a release of this collection while no node-collection tag satisfies it. +# decdn.node. The dependency is for the playbooks this collection ships +# (playbooks.txt): a publisher host takes decdn.node.baseline first and +# decdn.node.grafana_alloy for observability, and its origin and cache nodes are +# decdn.node.decdn_node. The floor is the decdn.node released with the first +# shipped playbooks; raise it whenever they need a newer decdn.node role. +# scripts/release.sh refuses a release of this collection while no node-collection +# tag satisfies it. dependencies: - decdn.node: ">=0.1.0" + decdn.node: ">=0.2.0" repository: https://github.com/decdn/devops documentation: https://github.com/decdn/devops/tree/main/ansible homepage: https://decdn.org issues: https://github.com/decdn/devops/issues -# No build_ignore: build.sh stages ONLY roles/ + this overlay. Add a NARROW, +# No build_ignore: build.sh stages ONLY roles/, playbooks/ (with their group_vars/) +# and this overlay. Add a NARROW, # path-anchored pattern here if something must ever be excluded. diff --git a/ansible/galaxy/publisher/playbooks.txt b/ansible/galaxy/publisher/playbooks.txt new file mode 100644 index 0000000..9710ef3 --- /dev/null +++ b/ansible/galaxy/publisher/playbooks.txt @@ -0,0 +1,12 @@ +# Playbooks galaxy/build.sh ships in decdn.publisher, from ansible/playbooks/ (with +# its group_vars/). Every import_playbook target must be listed here too: a run +# never leaves the collection, so it never mixes two releases' group_vars/. +site.yml +node.yml +publisher.yml +origin.yml +sponsord.yml +iroh_relay.yml +iroh_dns_server.yml +backup.yml +decommission.yml diff --git a/ansible/galaxy/stage-playbooks.py b/ansible/galaxy/stage-playbooks.py new file mode 100755 index 0000000..0131b9c --- /dev/null +++ b/ansible/galaxy/stage-playbooks.py @@ -0,0 +1,262 @@ +#!/usr/bin/env python3 +"""Stage a collection's playbooks for galaxy/build.sh. + + galaxy/stage-playbooks.py + +Copies into /playbooks/: + - the playbooks galaxy//playbooks.txt lists, from ansible/playbooks/; + - ansible/playbooks/group_vars/, which a collection playbook loads as it does in + this repo (the loader's basedir is the playbook's directory), so the firewall + holes and the Alloy toggles ship with the same precedence; + - the overlay galaxy//playbooks/, if any. + +Role references become FQCNs (`role: baseline` -> `role: decdn.node.baseline`, and +an include_role/import_role `name:` likewise), line by line so the comments +survive. The repo's own playbooks stay unqualified: a `collections:` keyword there +would let an installed decdn.node shadow ansible/roles/ on a deploy. + +A role reference is one of exactly two forms, its name bare or quoted: + - a `roles:` item written `- role: `; + - the `name:` key of a block-mapping include_role/import_role (any prefix). +Anything else that names a role is refused, not guessed at: a flow-style or inline +`roles:` list or include_role, a `- ` or `- name: ` roles item, a +templated or block-scalar name, an include_role without a name. That matters +because `ansible-playbook --syntax-check` does not resolve an include_role: a name +left short there would fail only at a consumer's runtime. + +After the rewrite it fails on a reference that is not an FQCN of this collection +or of one it depends on, an include_role `tasks_from:` the role does not have, and +an import_playbook whose target is not staged (a run never leaves the collection, +so it never mixes two releases' group_vars/). +Stdlib only: build.sh runs it with whatever python3 is on PATH. +""" + +from __future__ import annotations + +import re +import shutil +import sys +from dataclasses import dataclass +from pathlib import Path + +ANSIBLE_DIR = Path(__file__).resolve().parent.parent +GALAXY_DIR = ANSIBLE_DIR / "galaxy" +PLAYBOOKS_DIR = ANSIBLE_DIR / "playbooks" +ROLES_DIR = ANSIBLE_DIR / "roles" +COLLECTIONS = ("node", "publisher") +# The collections each one's playbooks may call into: its own and its dependencies +# (galaxy/publisher/galaxy.yml depends on decdn.node). +CALLS_INTO = {"node": ("node",), "publisher": ("publisher", "node")} + +# A role name, bare or in matching quotes: `baseline`, "baseline", 'decdn.node.baseline'. +NAME_VALUE = re.compile(r"""^(?P["']?)(?P[A-Za-z0-9_.]+)(?P=q)$""") +# A tasks_from file name (node-ids, topup-hold), bare or quoted. +TASKS_FROM_VALUE = re.compile(r"""^(?P["']?)(?P[A-Za-z0-9_.-]+)(?P=q)$""") +COMMENT_TAIL = re.compile(r"\s+#.*$") +ROLE_TASK = re.compile(r"^(?:-\s+)?(?:ansible\.(?:builtin|legacy)\.)?(?:include|import)_role:(?P.*)$") +ANY_ROLE_TASK = re.compile(r"(?:include|import)_role\s*:") +ROLES_KEY = re.compile(r"^(?:-\s+)?roles:(?P.*)$") +KEY_VALUE = re.compile(r"^(?:-\s+)?(?P[A-Za-z_][A-Za-z0-9_]*):(?:\s+(?P.*))?$") +IMPORT_PLAYBOOK = re.compile(r"^(?:-\s+)?(?:ansible\.builtin\.)?import_playbook:(?P.*)$") + + +class StageError(Exception): + pass + + +@dataclass +class Site: + """A role reference: the line it is on, its name, and (for an include_role) the + tasks_from it names.""" + + line: int + name: str + quote: str + tasks_from: str | None = None + + +def listed(path: Path) -> list[str]: + if not path.is_file(): + raise StageError(f"{path} is missing") + names = [s for line in path.read_text().splitlines() if (s := line.strip()) and not s.startswith("#")] + if not names: + raise StageError(f"{path} lists nothing") + return names + + +def role_owners() -> dict[str, str]: + owners: dict[str, str] = {} + for collection in COLLECTIONS: + for role in listed(GALAXY_DIR / collection / "roles.txt"): + if role in owners: + raise StageError(f"{role} is in both {owners[role]}/roles.txt and {collection}/roles.txt") + owners[role] = collection + return owners + + +def indent_of(line: str) -> int: + return len(line) - len(line.lstrip(" ")) + + +def value_of(raw: str | None) -> str: + return COMMENT_TAIL.sub("", raw or "").strip() + + +def scan(lines: list[str], where: str) -> tuple[list[Site], list[str]]: + """Every role reference in a playbook, and every reference form it refuses.""" + sites: list[Site] = [] + errors: list[str] = [] + task = None # an open include_role/import_role: [indent, child indent, Site | None, line] + roles = None # an open `roles:` key: [indent, item indent] + + def close_task() -> None: + if task is not None and task[2] is None: + errors.append(f"{where}:{task[3]}: an include_role/import_role without a name: key") + + for n, line in enumerate(lines, 1): + stripped = line.strip() + if not stripped or stripped.startswith("#"): + continue + ind = indent_of(line) + if task is not None and ind <= task[0]: + close_task() + task = None + # A `roles:` sequence may sit at the key's own indent; anything else there, or + # anything shallower, ends it. + if roles is not None and (ind < roles[0] or (ind == roles[0] and not stripped.startswith("- "))): + roles = None + + if m := ROLE_TASK.match(stripped): + if value_of(m["rest"]): + errors.append(f"{where}:{n}: write `{stripped}` as a block mapping, so its role name can be qualified") + task = [ind, None, None, n] + continue + if ANY_ROLE_TASK.search(stripped): + errors.append(f"{where}:{n}: `{stripped}` names a role in a form this script does not parse") + continue + if m := ROLES_KEY.match(stripped): + if value_of(m["rest"]): + errors.append(f"{where}:{n}: write roles as a block sequence of `- role: `, not `{stripped}`") + roles = [ind, None] + continue + + if task is not None: + if task[1] is None: + task[1] = ind + if ind == task[1] and (kv := KEY_VALUE.match(stripped)) and kv["key"] in ("name", "tasks_from"): + value = value_of(kv["value"]) + if not (v := (NAME_VALUE if kv["key"] == "name" else TASKS_FROM_VALUE).match(value)): + errors.append(f"{where}:{n}: write the role's {kv['key']} as a bare name, not `{value}`") + elif kv["key"] == "name": + task[2] = Site(n, v["name"], v["q"]) + sites.append(task[2]) + elif task[2] is not None: + task[2].tasks_from = v["name"] + else: + errors.append(f"{where}:{n}: put an include_role's name: before its tasks_from:") + continue + + if roles is not None and stripped.startswith("- "): + if roles[1] is None: + roles[1] = ind + if ind == roles[1]: + kv = KEY_VALUE.match(stripped) + if kv is None or kv["key"] != "role": + errors.append(f"{where}:{n}: write a roles item as `- role: `, not `{stripped}`") + elif not (v := NAME_VALUE.match(value_of(kv["value"]))): + errors.append(f"{where}:{n}: write the role as a bare name, not `{stripped}`") + else: + sites.append(Site(n, v["name"], v["q"])) + close_task() + return sites, errors + + +def rewrite(path: Path, owners: dict[str, str]) -> None: + """Qualify every role reference in place, keeping the line's quotes and comment.""" + lines = path.read_text().splitlines(keepends=True) + sites, errors = scan([raw.rstrip("\n") for raw in lines], path.name) + if errors: + raise StageError("\n".join(errors)) + for site in sites: + if site.name not in owners: + continue # check() names it + raw = lines[site.line - 1] + old = f"{site.quote}{site.name}{site.quote}" + new = f"{site.quote}decdn.{owners[site.name]}.{site.name}{site.quote}" + head, sep, tail = raw.partition(": ") + lines[site.line - 1] = head + sep + tail.replace(old, new, 1) + path.write_text("".join(lines)) + + +def check(path: Path, collection: str, owners: dict[str, str], staged: set[str]) -> list[str]: + """Every role reference an FQCN this collection can resolve, with the tasks_from it + names; every import local.""" + allowed = {f"decdn.{owners[r]}.{r}": r for r in owners if owners[r] in CALLS_INTO[collection]} + lines = path.read_text().splitlines() + sites, problems = scan(lines, path.name) + for site in sites: + if site.name not in allowed: + problems.append(f"{path.name}:{site.line}: role {site.name} is not one decdn.{collection} can resolve") + elif site.tasks_from is not None and not any( + (ROLES_DIR / allowed[site.name] / "tasks" / f"{site.tasks_from}{ext}").is_file() for ext in (".yml", ".yaml") + ): + problems.append(f"{path.name}:{site.line}: {site.name} has no tasks_from {site.tasks_from}") + for n, line in enumerate(lines, 1): + if m := IMPORT_PLAYBOOK.match(line.strip()): + target = value_of(m["rest"]) + if not target: + problems.append(f"{path.name}:{n}: write import_playbook's target on its line") + elif target not in staged: + problems.append(f"{path.name}:{n}: imports {target}, which decdn.{collection} does not ship") + return problems + + +def stage(collection: str, stage_dir: Path) -> None: + owners = role_owners() + overlay = GALAXY_DIR / collection / "playbooks" + names = listed(GALAXY_DIR / collection / "playbooks.txt") + overlay_names = sorted(p.name for p in overlay.glob("*.yml")) if overlay.is_dir() else [] + if clash := sorted(set(names) & set(overlay_names)): + raise StageError(f"{collection}: {', '.join(clash)} both listed in playbooks.txt and in the overlay playbooks/") + if len(set(names)) != len(names): + raise StageError(f"{collection}/playbooks.txt lists a playbook twice") + if not (PLAYBOOKS_DIR / "group_vars").is_dir(): + raise StageError(f"{PLAYBOOKS_DIR / 'group_vars'} is missing") + + dest = stage_dir / "playbooks" + if dest.exists(): + shutil.rmtree(dest) + dest.mkdir(parents=True) + shutil.copytree(PLAYBOOKS_DIR / "group_vars", dest / "group_vars") + for name in names: + src = PLAYBOOKS_DIR / name + if not src.is_file(): + raise StageError(f"{collection}/playbooks.txt names {name}, which is not under ansible/playbooks/") + shutil.copy2(src, dest / name) + for name in overlay_names: + shutil.copy2(overlay / name, dest / name) + + staged = set(names) | set(overlay_names) + problems = [] + for name in sorted(staged): + rewrite(dest / name, owners) + problems += check(dest / name, collection, owners, staged) + if problems: + raise StageError("\n".join(problems)) + print(f"staged {len(staged)} playbooks (+ group_vars/) -> {dest}") + + +def main(argv: list[str]) -> int: + if len(argv) != 3 or argv[1] not in COLLECTIONS: + print(f"usage: {argv[0]} node|publisher ", file=sys.stderr) + return 2 + try: + stage(argv[1], Path(argv[2])) + except StageError as e: + print(f"stage-playbooks.py: {e}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/ansible/molecule/collection/converge.yml b/ansible/molecule/collection/converge.yml new file mode 100644 index 0000000..8507c48 --- /dev/null +++ b/ansible/molecule/collection/converge.yml @@ -0,0 +1,26 @@ +--- +# The shipped playbooks by FQCN, in a consumer's order. Their plays read the +# collection's playbooks/group_vars/, not anything here. +- name: Provision the node + ansible.builtin.import_playbook: decdn.node.node + +- name: Encrypt the backups to the throwaway age key + hosts: all + become: true + gather_facts: false + tasks: + - name: Read the backup recipient's public key + ansible.builtin.command: + cmd: age-keygen -y /root/molecule-age.key + register: _molecule_age_pub + changed_when: false + + - name: Set the recipient + ansible.builtin.set_fact: + decdn_backup_age_recipients: ["{{ _molecule_age_pub.stdout | trim }}"] + +- name: Identity backup + ansible.builtin.import_playbook: decdn.node.backup + +- name: Decommission + ansible.builtin.import_playbook: decdn.node.decommission diff --git a/ansible/molecule/collection/molecule.yml b/ansible/molecule/collection/molecule.yml new file mode 100644 index 0000000..eec2ca5 --- /dev/null +++ b/ansible/molecule/collection/molecule.yml @@ -0,0 +1,62 @@ +--- +# decdn.node as a Galaxy consumer runs it. prepare builds the collection from this +# checkout (galaxy/build.sh, into the scenario's ephemeral dir) and installs it; +# converge then runs its shipped playbooks by FQCN against the stub daemon: +# decdn.node.node -> decdn.node.backup (identity, fetched) -> decdn.node.decommission +# ANSIBLE_ROLES_PATH is an empty directory, so every role, the lifecycle playbooks' +# runtime include_role calls included, must resolve from the installed collection: +# the part galaxy/check-playbooks.sh's syntax check cannot reach. baseline is skipped +# by its tag (real-host-only, see default/molecule.yml). +# verify.yml checks the fetched backup landed in backups/ beside the ansible.cfg in use +# (never inside the installed collection) and that decommission removed the unit but +# kept the identity. +# +# No idempotence step: backups and decommission are one-shot, as in lifecycle. +driver: + name: docker +platforms: + - name: decdn-node-collection + # Same digest-pinned image as default (repo convention). Re-resolve together. + image: geerlingguy/docker-debian12-ansible@sha256:a131e3b7a9a38dda40e98b49b39d6200ffab0ad853c714a513f5eea41bab9a25 + pre_build_image: true + command: /usr/lib/systemd/systemd + privileged: true + cgroupns_mode: host + volumes: + - /sys/fs/cgroup:/sys/fs/cgroup:rw + groups: + - decdn_nodes +provisioner: + name: ansible + env: + # Empty (prepare creates it): no role may come from ansible/roles. + ANSIBLE_ROLES_PATH: "${MOLECULE_EPHEMERAL_DIRECTORY}/no-roles" + # The collection prepare installs, then the vendored dependencies (make deps). + ANSIBLE_COLLECTIONS_PATH: "${MOLECULE_EPHEMERAL_DIRECTORY}/collections:${MOLECULE_PROJECT_DIRECTORY}/collections" + # See default/molecule.yml for why pipelining is on. + ANSIBLE_PIPELINING: "true" + options: + skip-tags: baseline + inventory: + host_vars: + # Inventory, not play vars: the shipped playbooks are the plays. + decdn-node-collection: + decdn_node_install_method: manual + decdn_node_manual_bin_src: "${MOLECULE_PROJECT_DIRECTORY}/molecule/default/files/decdn-node-stub" + decdn_cli_manual_bin_src: "${MOLECULE_PROJECT_DIRECTORY}/molecule/default/files/decdn-node-stub" + decdn_rpc_url: "https://rpc.example.invalid/" + decdn_region: "US" + decdn_network: arbitrum-sepolia + # decommission's typed confirmation, given non-interactively. + decdn_decommission_confirm: decdn-node-collection +verifier: + name: ansible +scenario: + test_sequence: + # Leading destroy: see default/molecule.yml. + - destroy + - create + - prepare + - converge + - verify + - destroy diff --git a/ansible/molecule/collection/prepare.yml b/ansible/molecule/collection/prepare.yml new file mode 100644 index 0000000..376ea69 --- /dev/null +++ b/ansible/molecule/collection/prepare.yml @@ -0,0 +1,50 @@ +--- +# Build decdn.node from this checkout and install it where molecule.yml's +# ANSIBLE_COLLECTIONS_PATH looks first, replacing an earlier run's. The build goes to +# the ephemeral dir, never ansible/build/, which `make galaxy-check` owns. +- name: Build and install the decdn.node collection + hosts: localhost + connection: local + gather_facts: false + become: false + vars: + _ephemeral: "{{ lookup('ansible.builtin.env', 'MOLECULE_EPHEMERAL_DIRECTORY') }}" + _project: "{{ lookup('ansible.builtin.env', 'MOLECULE_PROJECT_DIRECTORY') }}" + tasks: + - name: Build the collection + ansible.builtin.command: + cmd: "{{ _project }}/galaxy/build.sh node" + environment: + DECDN_GALAXY_BUILD_DIR: "{{ _ephemeral }}/build" + changed_when: true + + - name: Find the built tarball + ansible.builtin.find: + paths: "{{ _ephemeral }}/build" + patterns: ["decdn-node-*.tar.gz"] + register: _tarball + failed_when: _tarball.matched != 1 + + # --no-deps: its dependencies are vendored (make deps), second on the path. + - name: Install it + ansible.builtin.command: + argv: + - ansible-galaxy + - collection + - install + - --no-deps + - --force + - -p + - "{{ _ephemeral }}/collections" + - "{{ _tarball.files[0].path }}" + changed_when: true + + - name: Create the empty roles path + ansible.builtin.file: + path: "{{ _ephemeral }}/no-roles" + state: directory + mode: "0755" + +# The placeholder identity, age and its throwaway key pair, as lifecycle stages them. +- name: Stage the identity and the backup key + ansible.builtin.import_playbook: ../lifecycle/prepare.yml diff --git a/ansible/molecule/collection/verify.yml b/ansible/molecule/collection/verify.yml new file mode 100644 index 0000000..8201467 --- /dev/null +++ b/ansible/molecule/collection/verify.yml @@ -0,0 +1,63 @@ +--- +- name: Verify the node ran, was backed up and was decommissioned from the collection + hosts: all + become: true + vars: + _ephemeral: "{{ lookup('ansible.builtin.env', 'MOLECULE_EPHEMERAL_DIRECTORY') }}" + tasks: + - name: Find the archives on the host + ansible.builtin.find: + paths: /var/backups/decdn + patterns: ["*.tar.age"] + register: _archives + + - name: Require the identity archive on the host + ansible.builtin.assert: + that: + - _archives.matched == 1 + fail_msg: "decdn.node.backup left {{ _archives.matched }} archives on the host, not 1." + + # decdn_backup_local_dir's default for a collection playbook: backups/ beside the + # ansible.cfg in use, which is molecule's, in the ephemeral dir (ANSIBLE_CONFIG). + - name: Find the fetched copy beside ansible.cfg + ansible.builtin.find: + paths: "{{ _ephemeral }}/backups/{{ inventory_hostname }}" + patterns: ["*.tar.age"] + register: _fetched + delegate_to: localhost + become: false + + - name: Find any copy inside the installed collection + ansible.builtin.find: + paths: "{{ _ephemeral }}/collections/ansible_collections/decdn/node" + patterns: ["*.tar.age"] + recurse: true + register: _in_collection + delegate_to: localhost + become: false + + - name: Require the fetched copy beside ansible.cfg and none in the collection + ansible.builtin.assert: + that: + - _fetched.matched == 1 + - _in_collection.matched == 0 + fail_msg: >- + Fetched {{ _fetched.matched }} archive(s) to {{ _ephemeral }}/backups and + {{ _in_collection.matched }} into the installed collection. + + - name: Read the unit and the identity + ansible.builtin.stat: + path: "{{ item }}" + loop: + - /etc/systemd/system/decdn-node.service + - /var/lib/decdn/node.secret + - /var/lib/decdn/keystore.json + register: _paths + + - name: Require the unit removed and the identity kept + ansible.builtin.assert: + that: + - not _paths.results[0].stat.exists + - _paths.results[1].stat.exists + - _paths.results[2].stat.exists + fail_msg: decdn.node.decommission did not remove the unit, or did not keep the identity. diff --git a/ansible/playbooks/backup.yml b/ansible/playbooks/backup.yml index 69ff20a..8b24647 100644 --- a/ansible/playbooks/backup.yml +++ b/ansible/playbooks/backup.yml @@ -3,6 +3,7 @@ # sponsord host's credentials (the treasury keystore and password, the API token, # the onramp's Turnstile secret). # make backup LIMIT= ANSIBLE_ARGS='-e decdn_backup_scope=full' +# ansible-playbook -i inventory/hosts.yml decdn.publisher.backup -l # from the collection # decdn_backup_age_recipients (public keys) is required; set it in group_vars # (sponsord_backup_age_recipients defaults to it). See roles/decdn_node/tasks/backup.yml # and roles/sponsord/tasks/backup.yml for what is archived and docs/lifecycle.md for diff --git a/ansible/playbooks/decommission.yml b/ansible/playbooks/decommission.yml index e79d74c..ff19c1e 100644 --- a/ansible/playbooks/decommission.yml +++ b/ansible/playbooks/decommission.yml @@ -7,6 +7,8 @@ # task prints what is left to do. One typed confirmation, in the first play that # reaches a host, names every service it covers there. # make decommission LIMIT= # LIMIT is required; asks for typed confirmation +# ansible-playbook -i inventory/hosts.yml decdn.publisher.decommission -l # from the +# collection, where nothing requires -l: the host cap below still applies # Take a backup first (make backup). See docs/lifecycle.md. # Before anything stops, over every host this run decommissions: @@ -30,7 +32,7 @@ This run decommissions {{ ansible_play_hosts_all | length }} hosts ({{ ansible_play_hosts_all | sort | join(', ') }}), more than decdn_decommission_max_hosts ({{ decdn_decommission_max_hosts | default(1) }}). - Scope it with LIMIT, or raise decdn_decommission_max_hosts deliberately. + Scope it with LIMIT (-l), or raise decdn_decommission_max_hosts deliberately. quiet: true - name: Refuse while sponsord holds a pool top-up diff --git a/ansible/playbooks/iroh_dns_server.yml b/ansible/playbooks/iroh_dns_server.yml index 9f5b810..d6bc270 100644 --- a/ansible/playbooks/iroh_dns_server.yml +++ b/ansible/playbooks/iroh_dns_server.yml @@ -14,8 +14,8 @@ # and only one of them can have it. (Under site.yml, sponsord.yml and # iroh_relay.yml run first, so on a host in both groups the other service is # already up when this refuses; either way the deploy fails loudly.) The role -# itself refuses ports another process holds, so the decdn.publisher collection gets -# the same protection without this repo's groups. tests/scripts-test.sh runs this +# itself refuses ports another process holds, so a playbook of your own that runs it +# gets the same protection without this repo's groups. tests/scripts-test.sh runs this # play by its name. - name: Refuse an iroh DNS server on an iroh relay or sponsord-onramp host hosts: iroh_dns_server_hosts diff --git a/ansible/playbooks/iroh_relay.yml b/ansible/playbooks/iroh_relay.yml index 8a164d6..2232f5d 100644 --- a/ansible/playbooks/iroh_relay.yml +++ b/ansible/playbooks/iroh_relay.yml @@ -13,8 +13,8 @@ # Before the relay play: the onramp's Caddy needs tcp/80 and tcp/443 too, and only # one of them can have them. (Under site.yml, sponsord.yml runs first, so on a host # in both groups the onramp is already up when this refuses; either way the deploy -# fails loudly.) The role itself refuses ports another process holds, so the -# decdn.publisher collection gets the same protection without this repo's groups. +# fails loudly.) The role itself refuses ports another process holds, so a playbook +# of your own that runs it gets the same protection without this repo's groups. # tests/scripts-test.sh runs this play by its name. - name: Refuse an iroh relay on a sponsord-onramp host hosts: iroh_relay_hosts diff --git a/ansible/playbooks/node.yml b/ansible/playbooks/node.yml index 59da030..1d34671 100644 --- a/ansible/playbooks/node.yml +++ b/ansible/playbooks/node.yml @@ -2,7 +2,8 @@ # NODE OPERATORS: hardened, public deCDN cache nodes. # decdn_nodes (minus decdn_origin_nodes): baseline -> grafana_alloy (opt-in, off by # default) -> decdn-node -# Standalone entry point (`make deploy-node`), also imported by site.yml. The node +# Standalone entry point (`make deploy-node`; from the Galaxy collections, +# `ansible-playbook decdn.node.node`), also imported by site.yml. The node # listens on public QUIC udp/4433; metrics + admin RPC stay loopback. On-chain stake + # registration (ADR 019 Phase 2) is an operator action, not automated here — see # roles/decdn_node/README.md. diff --git a/ansible/playbooks/origin.yml b/ansible/playbooks/origin.yml index 4f92a4c..e71c285 100644 --- a/ansible/playbooks/origin.yml +++ b/ansible/playbooks/origin.yml @@ -15,8 +15,8 @@ pre_tasks: # The firewall hole, backup, decommission, the relay allowlist and Alloy's node # scrape all follow decdn_nodes, so an origin outside it would lose them. The role - # checks no group name, so the decdn.node collection does not impose this repo's - # groups on others. tests/scripts-test.sh runs these tasks by the play's name. + # checks no group name, so a playbook of your own that runs it is not held to this + # repo's groups. tests/scripts-test.sh runs these tasks by the play's name. - name: Require the origin in decdn_nodes too ansible.builtin.assert: that: diff --git a/ansible/playbooks/sponsord.yml b/ansible/playbooks/sponsord.yml index 6b8e9e7..f3eb706 100644 --- a/ansible/playbooks/sponsord.yml +++ b/ansible/playbooks/sponsord.yml @@ -16,7 +16,7 @@ # Before anything runs: an onramp host outside sponsord_hosts would skip the play # below (baseline hardening included) and get no firewall holes for Caddy, which # playbooks/group_vars/ derive from sponsord_hosts. The role itself checks no group -# name, so the decdn.publisher collection does not impose this repo's groups on others. +# name, so a playbook of your own that runs it is not held to this repo's groups. # tests/scripts-test.sh runs this play by its name. - name: Require every sponsord-onramp host to be a sponsord host hosts: sponsord_onramp_hosts diff --git a/ansible/roles/decdn_node/defaults/main.yml b/ansible/roles/decdn_node/defaults/main.yml index 780ffa5..a17a53e 100644 --- a/ansible/roles/decdn_node/defaults/main.yml +++ b/ansible/roles/decdn_node/defaults/main.yml @@ -479,7 +479,21 @@ decdn_backup_dir: /var/backups/decdn # on the host, root 0700 # (kilobytes); off for full ones, because fetch under become reads the whole file # into memory on both ends. The report prints a streaming `ssh … sudo cat` instead. decdn_backup_fetch: "{{ decdn_backup_scope == 'identity' }}" -decdn_backup_local_dir: "{{ playbook_dir }}/../backups" # control machine; git-ignored in this repo +# On the control machine: backups/ beside this repo's playbooks/ (git-ignored here). +# From a Galaxy collection's playbook, playbook_dir is inside the installed +# collection, which an upgrade replaces, so it is backups/ beside the project's +# ansible.cfg instead; with no project ansible.cfg (none, ~/.ansible.cfg or +# /etc/ansible's), a run that fetches fails and asks for this variable. +# decdn_backup_local_dir and sponsord_backup_local_dir share this expression +# (make test-scripts checks). +decdn_backup_local_dir: >- + {{ ((playbook_dir ~ '/..') if playbook_dir is not search('/ansible_collections/') + else (ansible_config_file | dirname) + if (ansible_config_file | default('', true) | basename) == 'ansible.cfg' + and (ansible_config_file | dirname) != '/etc/ansible' + else undef(hint='decdn_backup_local_dir: a collection playbook run without a project' + ~ ' ansible.cfg has no project directory to fetch backups to; set it')) + ~ '/backups' }} # # Decommission refuses to touch more hosts than this in one run. Its typed # confirmation is interactive; a non-interactive run sets decdn_decommission_confirm diff --git a/ansible/roles/sponsord/defaults/main.yml b/ansible/roles/sponsord/defaults/main.yml index 7c38ca8..8864b34 100644 --- a/ansible/roles/sponsord/defaults/main.yml +++ b/ansible/roles/sponsord/defaults/main.yml @@ -203,7 +203,21 @@ sponsord_backup_age_recipients: "{{ decdn_backup_age_recipients | default([]) }} sponsord_backup_include_env: false # also archive secret.env (the RPC URL, maybe an API key) sponsord_backup_dir: /var/backups/sponsord # on the host, root 0700 sponsord_backup_fetch: true # copy the ENCRYPTED archive (kilobytes) to the control machine -sponsord_backup_local_dir: "{{ playbook_dir }}/../backups" # control machine; git-ignored in this repo +# On the control machine: backups/ beside this repo's playbooks/ (git-ignored here). +# From a Galaxy collection's playbook, playbook_dir is inside the installed +# collection, which an upgrade replaces, so it is backups/ beside the project's +# ansible.cfg instead; with no project ansible.cfg (none, ~/.ansible.cfg or +# /etc/ansible's), a run that fetches fails and asks for this variable. +# decdn_backup_local_dir and sponsord_backup_local_dir share this expression +# (make test-scripts checks). +sponsord_backup_local_dir: >- + {{ ((playbook_dir ~ '/..') if playbook_dir is not search('/ansible_collections/') + else (ansible_config_file | dirname) + if (ansible_config_file | default('', true) | basename) == 'ansible.cfg' + and (ansible_config_file | dirname) != '/etc/ansible' + else undef(hint='sponsord_backup_local_dir: a collection playbook run without a project' + ~ ' ansible.cfg has no project directory to fetch backups to; set it')) + ~ '/backups' }} # # Shared with decdn_node (and identical there, `make test-scripts` checks): one typed # confirmation covers every service on the host. A non-interactive run sets diff --git a/ansible/tests/firewall-holes/check.sh b/ansible/tests/firewall-holes/check.sh index 406348d..4e0b687 100755 --- a/ansible/tests/firewall-holes/check.sh +++ b/ansible/tests/firewall-holes/check.sh @@ -6,24 +6,60 @@ # every host shape in inventory.yml, outside any role, which is how the sponsord # play sees a co-located node, and compare with expected.json. # Needs ansible-core and jq. tests/scripts-test.sh runs it. +# check.sh [] default ansible/playbooks. For an installed +# collection's .../ansible_collections///playbooks (galaxy/check-playbooks.sh), +# it runs a probe as the collection playbook ..firewall_holes_probe +# instead, which proves the group_vars/ beside a collection playbook load. set -euo pipefail here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -playbooks="$here/../../playbooks" +playbooks="${1:-$here/../../playbooks}" +[[ -d "$playbooks/group_vars" ]] || { echo "$playbooks has no group_vars/" >&2; exit 1; } work="$(mktemp -d)" -trap 'rm -rf "$work"' EXIT +probe="" # the probe playbook, while it sits in a collection's playbooks/ +trap 'rm -rf "$work"; [[ -z "$probe" ]] || rm -f "$probe"' EXIT +collection_re='^(.*)/ansible_collections/([a-z0-9_]+)/([a-z0-9_]+)/playbooks$' +playbooks="$(cd "$playbooks" && pwd)" +mode="playbook-dir" +if [[ "$playbooks" =~ $collection_re ]]; then + mode="collection" + root="${BASH_REMATCH[1]}" fqcn="${BASH_REMATCH[2]}.${BASH_REMATCH[3]}.firewall_holes_probe" + probe="$playbooks/firewall_holes_probe.yml" +elif [[ "$playbooks" == */ansible_collections/* ]]; then + # Never fall back to the weaker mode for something that looks like a collection. + echo "$playbooks is under ansible_collections/ but not a collection's playbooks/" >&2 + exit 1 +fi # Each host writes its resolved list to .json with the core copy module (the # json stdout callback lives in ansible.posix, which a core-only install lacks). # From this directory, so ansible/ansible.cfg (inventory, become) is not read. ( cd "$here" - ANSIBLE_DEPRECATION_WARNINGS=0 ansible all -i "$here/inventory.yml" --playbook-dir "$playbooks" \ - -m ansible.builtin.copy \ - -a "dest=$work/{{ inventory_hostname }}.json content={{ baseline_extra_inbound | to_json }} mode=0600" \ - "$work/ansible.log" 2>&1 -) || { cat "$work/ansible.log" >&2; echo "ansible could not resolve the inventory" >&2; exit 1; } + export ANSIBLE_DEPRECATION_WARNINGS=0 + if [[ "$mode" == collection ]]; then + cat >"$probe" <<'PROBE' +--- +- name: Write each host's baseline_extra_inbound + hosts: all + gather_facts: false + tasks: + - name: Write it to .json + ansible.builtin.copy: + dest: "{{ probe_out }}/{{ inventory_hostname }}.json" + content: "{{ baseline_extra_inbound | to_json }}" + mode: "0600" +PROBE + ANSIBLE_COLLECTIONS_PATH="$root" ansible-playbook -i "$here/inventory.yml" "$fqcn" -e "probe_out=$work" \ + "$work/ansible.log" 2>&1 + else + ansible all -i "$here/inventory.yml" --playbook-dir "$playbooks" \ + -m ansible.builtin.copy \ + -a "dest=$work/{{ inventory_hostname }}.json content={{ baseline_extra_inbound | to_json }} mode=0600" \ + "$work/ansible.log" 2>&1 + fi +) || { cat "$work/ansible.log" >&2 2>/dev/null || true; echo "ansible could not resolve the inventory" >&2; exit 1; } got="$(cd "$work" && jq -S -n '[inputs | {key: (input_filename | rtrimstr(".json")), value: .}] | from_entries' ./*.json \ | jq -S 'with_entries(.key |= ltrimstr("./"))')" want="$(jq -S . "$here/expected.json")" @@ -42,4 +78,4 @@ grep -qE '^sponsord_onramp_proxy: caddy([[:space:]]|$)' "$roles/sponsord_onramp/ grep -qE '^iroh_relay_enable_quic_addr_discovery: true([[:space:]]|$)' "$roles/iroh_relay/defaults/main.yml" \ || { echo "iroh_relay_enable_quic_addr_discovery's default is not true: update the fallback in playbooks/group_vars/all.yml" >&2; exit 1; } -echo "baseline_extra_inbound matches for $(jq -r 'keys | join(", ")' <<<"$want")" +echo "ok baseline_extra_inbound matches for $(jq -r 'keys | join(", ")' <<<"$want") ($mode: $playbooks)" diff --git a/docs/node-operators.md b/docs/node-operators.md index cf2c1c0..8625856 100644 --- a/docs/node-operators.md +++ b/docs/node-operators.md @@ -24,9 +24,12 @@ loopback, and the chain's contract addresses taken from upstream's deployment ma | **Helm** (Kubernetes, one release per node) | [`charts/decdn-node/README.md`](../charts/decdn-node/README.md) | `oci://ghcr.io/decdn/charts/decdn-node`, with no `config.cache.origin` | [`requirements.md`](requirements.md) has the hardware, network and platform requirements, -and compares the paths. For your own playbooks, the roles are the +and compares the paths. The roles and the node playbooks are also the [`decdn.node`](../ansible/galaxy/node/README.md) collection on -[Ansible Galaxy](https://galaxy.ansible.com/ui/repo/published/decdn/node/) (`ansible-galaxy collection install decdn.node`). +[Ansible Galaxy](https://galaxy.ansible.com/ui/repo/published/decdn/node/) (`ansible-galaxy collection install decdn.node`): +with only an inventory of your own, `ansible-playbook -i inventory/hosts.yml decdn.node.node` +deploys what `make deploy-node` does, firewall hole included; or call the roles from +your own playbooks. A node without an origin backend fills a miss by pulling from other nodes (`node_to_node_pull_through_enabled`). The Ansible role and the chart turn that on for diff --git a/docs/publishers.md b/docs/publishers.md index 1460589..8bc8fa2 100644 --- a/docs/publishers.md +++ b/docs/publishers.md @@ -29,11 +29,14 @@ should read [`node-operators.md`](node-operators.md) instead. Only the origin is required. With Ansible, start from `ansible/inventory/hosts-publisher.yml.example`; `make deploy-publisher` (`playbooks/publisher.yml`) deploys all four in that order, and each has its own target -(`deploy-origin`, `deploy-sponsord`, `deploy-relay`, `deploy-dns`). The roles also ship as -the [`decdn.publisher`](../ansible/galaxy/publisher/README.md) collection on +(`deploy-origin`, `deploy-sponsord`, `deploy-relay`, `deploy-dns`). The roles and these +playbooks also ship as the [`decdn.publisher`](../ansible/galaxy/publisher/README.md) collection on [Ansible Galaxy](https://galaxy.ansible.com/ui/repo/published/decdn/publisher/) (`ansible-galaxy collection install decdn.publisher`), which depends on [`decdn.node`](../ansible/galaxy/node/README.md) for the baseline, -observability and the origin node itself. +observability and the origin node itself: with only an inventory of your own, +`ansible-playbook -i inventory/hosts.yml decdn.publisher.publisher` (or `decdn.publisher.origin`, +`.sponsord`, `.iroh_relay`, `.iroh_dns_server`) deploys what the make targets do, +firewall holes and placement checks included. ## Origin nodes diff --git a/examples/collection-fleet/.gitignore b/examples/collection-fleet/.gitignore new file mode 100644 index 0000000..335b37d --- /dev/null +++ b/examples/collection-fleet/.gitignore @@ -0,0 +1,12 @@ +# Installed by `ansible-galaxy collection install -r requirements.yml -p collections`. +/collections/ +# Secrets (decdn_rpc_url, grafana_alloy_api_token, …): never commit them. Any +# secret.* (Ansible loads .yml, .yaml and .json alike), but the copied +# secret.*.example templates. +inventory/host_vars/*/secret.* +!inventory/host_vars/*/secret.*.example +inventory/group_vars/*/secret.* +!inventory/group_vars/*/secret.*.example +# Identity and sponsord backups fetched by decdn.node.backup / decdn.publisher.backup +# (encrypted, but they belong with the operator, not in git). +/backups/ diff --git a/examples/collection-fleet/README.md b/examples/collection-fleet/README.md new file mode 100644 index 0000000..177cb53 --- /dev/null +++ b/examples/collection-fleet/README.md @@ -0,0 +1,55 @@ +# A deCDN fleet from the Galaxy collections + +A starting point for a **private** repository that deploys deCDN from the +[`decdn.node`](../../ansible/galaxy/node/README.md) and +[`decdn.publisher`](../../ansible/galaxy/publisher/README.md) collections alone: the +collections bring the roles and the playbooks (firewall holes, placement checks and +lifecycle guards included), and this repository holds only your inventory. + +| File | What it is | +|------|------------| +| `requirements.yml` | the collections, from Ansible Galaxy (delete `decdn.publisher` for cache nodes only) | +| `ansible.cfg` | this repo's `ansible/ansible.cfg`, minus `roles_path` | +| `.gitignore` | keeps the installed collections, per-host secrets and backups out of git | + +## Set up + +Copy this directory into a new private repository, then add an inventory, starting +from this repository's examples: + +```bash +ansible-galaxy collection install -r requirements.yml -p collections +mkdir -p inventory +cp /ansible/inventory/hosts-node.yml.example inventory/hosts.yml # or hosts-publisher.yml.example +cp -R /ansible/inventory/group_vars /ansible/inventory/host_vars inventory/ +``` + +Fill in `inventory/hosts.yml` and its `group_vars/` and `host_vars/` (each file says +what it needs; [`ansible/README.md`](../../ansible/README.md) has the variables and the +on-host secrets). Rename `host_vars/decdn-node-1/` (and `origin-1/`) to your hosts, and +keep secrets in the git-ignored `host_vars//secret.yml` (copied from its +`secret.yml.example`). The copied files name this repository's make targets; here, +`make deploy-node LIMIT=` is `ansible-playbook decdn.node.node -l `, and +`ANSIBLE_ARGS='…'` are plain `ansible-playbook` options. + +## Run + +```bash +ansible-playbook decdn.node.node -l -u root # first converge only; swap root for your image's bootstrap user +ansible-playbook decdn.node.node --check --diff -l # dry run +ansible-playbook decdn.node.node -l # cache nodes +ansible-playbook decdn.publisher.site # everything in the inventory +ansible-playbook decdn.node.backup -l # fetched to backups/, beside ansible.cfg +ansible-playbook decdn.node.decommission -l # decdn.publisher.decommission on a host with publisher services +``` + +The collection READMEs list every playbook and what it runs. Unlike this repository's +Makefile, nothing here refuses a run without `-l`: scope deploys and every +decommission yourself (decommission refuses more than `decdn_decommission_max_hosts` +hosts, default 1). + +## Upgrade + +Raise the versions in `requirements.yml` (pin `==X.Y.Z` once the fleet runs), read +each collection's CHANGELOG, then `ansible-galaxy collection install -r +requirements.yml -p collections --upgrade` and dry-run before deploying. diff --git a/examples/collection-fleet/ansible.cfg b/examples/collection-fleet/ansible.cfg new file mode 100644 index 0000000..a9e3be3 --- /dev/null +++ b/examples/collection-fleet/ansible.cfg @@ -0,0 +1,21 @@ +# Settings for running the decdn.node / decdn.publisher playbooks from this +# directory: decdn/devops's ansible/ansible.cfg, except roles_path (every role comes +# from the collections). Fetched backups land in backups/ beside this file. +[defaults] +inventory = inventory/hosts.yml +collections_path = collections +host_key_checking = True +retry_files_enabled = False +stdout_callback = default +result_format = yaml +nocows = True +interpreter_python = auto_silent +# Keep it on: modules run over the SSH session instead of being staged in the +# target's ~/.ansible/tmp, where the roles' `copy: content=` of a secret (the node's +# decdn.env, Alloy's token file) would otherwise sit in plaintext. Needs `requiretty` +# off in sudoers, the default on the Debian/Ubuntu targets. +pipelining = True + +[privilege_escalation] +become = True +become_method = sudo diff --git a/examples/collection-fleet/requirements.yml b/examples/collection-fleet/requirements.yml new file mode 100644 index 0000000..f027b46 --- /dev/null +++ b/examples/collection-fleet/requirements.yml @@ -0,0 +1,15 @@ +--- +# The deCDN collections from Ansible Galaxy. Install into ./collections (ansible.cfg +# reads it from there): +# ansible-galaxy collection install -r requirements.yml -p collections +# 0.2.0 is the first release of each that ships the playbooks. Pin exact versions +# (==X.Y.Z) once the fleet runs, and bump them deliberately, reading each +# CHANGELOG: https://github.com/decdn/devops/tree/main/ansible/galaxy +collections: + # Node operators: cache nodes only. Delete decdn.publisher below. + - name: decdn.node + version: ">=0.2.0" + # Publishers: origin nodes, sponsord and its onramp, iroh relays and DNS servers + # (and cache nodes). Depends on decdn.node. + - name: decdn.publisher + version: ">=0.2.0" diff --git a/scripts/release.sh b/scripts/release.sh index 0c39b44..d819e80 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -14,9 +14,10 @@ # Without --execute it is a dry run: it prints the new tag and the diff it would # commit, and changes no file. The level defaults to `auto`, the bump git-cliff # derives (cliff.toml) from the conventional commits since the artifact's last tag -# that touched what it ships: the roles in its galaxy//roles.txt, that -# overlay, galaxy/build.sh and LICENSE for a collection; the chart and -# monitoring/decdn-node/ for the chart. +# that touched what it ships: the roles in its galaxy//roles.txt, the +# playbooks in its playbooks.txt with ansible/playbooks/group_vars/, that overlay, +# galaxy/build.sh, galaxy/stage-playbooks.py and LICENSE for a collection; the chart +# and monitoring/decdn-node/ for the chart. # Before 1.0, a feature (or, from 0.1 on, a breaking change) bumps the minor and any # other commit kept in the changelog bumps the patch. The section is git-cliff's # rendering of those commits, under a `## [X.Y.Z] — YYYY-MM-DD` heading. A commit there @@ -74,15 +75,23 @@ case "$artifact" in changelog="$overlay/CHANGELOG.md" workflow="release-collection.yml" check=(make -C ansible "galaxy-check-$collection") - # What `galaxy/build.sh ` ships: the roles in its roles.txt, its - # overlay, and LICENSE; build.sh itself too. The role list is the one build.sh - # reads, so a role added there is counted here too. + # What `galaxy/build.sh ` ships: the roles in its roles.txt, the + # playbooks in its playbooks.txt with their group_vars/, its overlay (the overlay + # playbooks/ included), and LICENSE; build.sh and stage-playbooks.py themselves + # too. The lists are the ones build.sh reads, so an entry added there is counted + # here too. [[ -f "$overlay/roles.txt" ]] || die "$overlay/roles.txt is missing" + [[ -f "$overlay/playbooks.txt" ]] || die "$overlay/playbooks.txt is missing" roles=() while IFS= read -r role; do roles+=("$role"); done < <(grep -vE '^[[:space:]]*(#|$)' "$overlay/roles.txt") ((${#roles[@]} > 0)) || die "$overlay/roles.txt lists no role" - paths=("$overlay/**" 'ansible/galaxy/build.sh' 'LICENSE') + paths=("$overlay/**" 'ansible/galaxy/build.sh' 'ansible/galaxy/stage-playbooks.py' + 'ansible/playbooks/group_vars/**' 'LICENSE') for role in "${roles[@]}"; do paths+=("ansible/roles/$role/**"); done + playbooks=() + while IFS= read -r pb; do playbooks+=("$pb"); done < <(grep -vE '^[[:space:]]*(#|$)' "$overlay/playbooks.txt") + ((${#playbooks[@]} > 0)) || die "$overlay/playbooks.txt lists no playbook" + for pb in "${playbooks[@]}"; do paths+=("ansible/playbooks/$pb"); done ;; chart) prefix="decdn-node-" diff --git a/tests/scripts-test.sh b/tests/scripts-test.sh index a3ddf6a..5f0c5a6 100755 --- a/tests/scripts-test.sh +++ b/tests/scripts-test.sh @@ -221,6 +221,126 @@ roles_dir="$(cd "$repo/ansible/roles" && printf '%s\n' */ | tr -d / | LC_ALL=C s pass "every role under ansible/roles ships in exactly one Galaxy collection" expect 2 "galaxy/build.sh rejects an unknown collection" "$repo/ansible/galaxy/build.sh" all expect 2 "galaxy/build.sh rejects no collection" "$repo/ansible/galaxy/build.sh" +expect 2 "galaxy/check-playbooks.sh rejects an unknown collection" "$repo/ansible/galaxy/check-playbooks.sh" all + +# --- Galaxy collections: the shipped playbooks ----------------------------------------- +# galaxy//playbooks.txt lists what build.sh ships from ansible/playbooks/; a +# playbook in neither list would never reach a Galaxy consumer. +listed_pbs="$(cat "$repo"/ansible/galaxy/{node,publisher}/playbooks.txt | grep -vE '^[[:space:]]*(#|$)' | LC_ALL=C sort -u)" +pbs_dir="$(cd "$repo/ansible/playbooks" && printf '%s\n' *.yml | LC_ALL=C sort)" +[[ "$(comm -13 <(echo "$listed_pbs") <(echo "$pbs_dir") | xargs)" == "" ]] \ + || fail "playbooks no collection ships (add them to a galaxy/*/playbooks.txt): $(comm -13 <(echo "$listed_pbs") <(echo "$pbs_dir") | xargs)" +pass "every playbook under ansible/playbooks ships in a Galaxy collection" +# stage-playbooks.py on a copy of ansible/{galaxy,playbooks} (it finds them, and the +# roles it checks tasks_from against, from its own path). +sp="$work/stage-playbooks" +stage_copy() { + rm -rf "$sp" && mkdir -p "$sp/ansible" + cp -R "$repo/ansible/galaxy" "$repo/ansible/playbooks" "$sp/ansible/" + ln -s "$repo/ansible/roles" "$sp/ansible/roles" +} +stage_run() { python3 "$sp/ansible/galaxy/stage-playbooks.py" "$1" "$sp/out-$1"; } +# : a fresh copy with that one change, which must +# change something. +stage_break() { + stage_copy + cp "$sp/ansible/$1" "$work/stage-orig" + sed -i "$2" "$sp/ansible/$1" + ! cmp -s "$work/stage-orig" "$sp/ansible/$1" || fail "stage_break: '$2' changed nothing in $1" +} +# : refused (exit 1) for that reason; +# an uncaught exception exits 1 too, so the message is what proves it. +stage_refuses() { + expect 1 "stage-playbooks.py refuses $1" stage_run "$2" + grep -qF -- "$3" "$work/out" || { cat "$work/out" >&2; fail "stage-playbooks.py refused $1 for another reason"; } +} +stage_copy +for c in node publisher; do + expect 0 "stage-playbooks.py stages decdn.$c" stage_run "$c" + [[ -f "$sp/out-$c/playbooks/group_vars/all.yml" ]] || fail "stage-playbooks.py did not stage decdn.$c's group_vars/" + # Strip the qualifier from role:/name: values and each staged playbook is its source. + for pb in "$sp/out-$c/playbooks/"*.yml; do + src="$repo/ansible/galaxy/$c/playbooks/${pb##*/}" + [[ -f "$src" ]] || src="$repo/ansible/playbooks/${pb##*/}" + diff <(sed -E "s/^([[:space:]]*(- )?(role|name): [\"']?)decdn\.(node|publisher)\./\1/" "$pb") "$src" >/dev/null \ + || fail "stage-playbooks.py changed more than role names in decdn.$c's ${pb##*/}" + done +done +if ! grep -q '^ - role: decdn.node.baseline$' "$sp/out-publisher/playbooks/sponsord.yml" \ + || ! grep -q '^ name: decdn.publisher.sponsord$' "$sp/out-publisher/playbooks/backup.yml"; then + fail "stage-playbooks.py did not qualify roles: items and include_role names" +fi +pass "stage-playbooks.py qualifies role: and include_role names, stages group_vars/ and changes nothing else" +expect 2 "stage-playbooks.py rejects an unknown collection" python3 "$sp/ansible/galaxy/stage-playbooks.py" relay "$sp/out" +# Quoted names and the ansible.legacy prefix are qualified, not passed through. +stage_break playbooks/decommission.yml 's/^ name: grafana_alloy$/ name: "grafana_alloy"/' +expect 0 "stage-playbooks.py takes a quoted include_role name" stage_run publisher +grep -q '^ name: "decdn.node.grafana_alloy"$' "$sp/out-publisher/playbooks/decommission.yml" \ + || fail "stage-playbooks.py left a quoted include_role name unqualified" +stage_break playbooks/decommission.yml 's/^ ansible.builtin.include_role:$/ ansible.legacy.include_role:/' +expect 0 "stage-playbooks.py takes ansible.legacy.include_role" stage_run publisher +! grep -qE '^ name: [a-z_]+$' "$sp/out-publisher/playbooks/decommission.yml" \ + || fail "stage-playbooks.py left an ansible.legacy.include_role name unqualified" +stage_break playbooks/node.yml "s/^ - role: baseline$/ - role: 'baseline'/" +expect 0 "stage-playbooks.py takes a quoted roles item" stage_run node +grep -q "^ - role: 'decdn.node.baseline'$" "$sp/out-node/playbooks/node.yml" \ + || fail "stage-playbooks.py left a quoted roles item unqualified" +pass "stage-playbooks.py qualifies quoted names and ansible.legacy.include_role" +stage_break playbooks/sponsord.yml 's/^ - role: sponsord$/ - role: sponsordd/' +stage_refuses "a role no collection ships" publisher "role sponsordd is not one decdn.publisher can resolve" +stage_break playbooks/node.yml 's/^ - role: decdn_node$/ - role: decdn.publisher.decdn_node/' +stage_refuses "an FQCN in the wrong collection" node "role decdn.publisher.decdn_node is not one decdn.node can resolve" +stage_break galaxy/publisher/playbooks.txt '/^iroh_relay.yml$/d' +stage_refuses "an import_playbook it does not ship" publisher "imports iroh_relay.yml" +stage_break playbooks/backup.yml 's/^ ansible.builtin.include_role:$/ ansible.builtin.include_role: {name: decdn_node, tasks_from: backup}/' +stage_refuses "a flow-style include_role" publisher "as a block mapping" +stage_break playbooks/decommission.yml 's/^ name: grafana_alloy$/ name: "{{ r }}"/' +stage_refuses "a templated include_role name" publisher "write the role's name as a bare name" +stage_break playbooks/backup.yml 's/^ name: decdn_node$/ role_name: decdn_node/' +stage_refuses "an include_role without a name" publisher "without a name: key" +stage_break playbooks/backup.yml 's/^ tasks_from: backup$/ tasks_from: bakup/' +stage_refuses "a tasks_from the role lacks" publisher "has no tasks_from bakup" +stage_break playbooks/node.yml 's/^ - role: baseline$/ - baseline/' +stage_refuses "a bare roles item" node "write a roles item as" +stage_break playbooks/node.yml 's/^ - role: baseline$/ - { role: baseline }/' +stage_refuses "a flow-style roles item" node "write a roles item as" +stage_break playbooks/node.yml 's/^ - role: baseline$/ - name: baseline/' +stage_refuses "a name: roles item" node "write a roles item as" +stage_break playbooks/node.yml 's/^ roles:$/ roles: [baseline]/' +stage_refuses "an inline roles list" node "block sequence" +stage_break galaxy/node/playbooks.txt '/^node.yml$/d' +stage_refuses "an empty playbooks.txt" node "lists nothing" +stage_copy; rm -r "$sp/ansible/playbooks/group_vars" +stage_refuses "a missing group_vars/" node "group_vars is missing" +# decdn.node ships node-scoped backup.yml and decommission.yml (its overlay +# playbooks/): each play must be the fleet-wide play of the same name, but for its +# hosts and the sponsord top-up check, and every fleet play on decdn_nodes must be +# there. Play names reach yq through the environment, never the expression. +if command -v yq >/dev/null; then + norm='del(.hosts) | (.tasks // []) |= map(select(.name != "Refuse while sponsord holds a pool top-up"))' + for pb in backup.yml decommission.yml; do + node_pb="$repo/ansible/galaxy/node/playbooks/$pb" fleet_pb="$repo/ansible/playbooks/$pb" + node_plays="$(yq '.[].name' "$node_pb")" + [[ -n "$node_plays" ]] || fail "galaxy/node/playbooks/$pb has no named play" + while IFS= read -r play; do + mine="$(PLAY="$play" yq -o=json -I0 ".[] | select(.name == strenv(PLAY)) | $norm" "$node_pb")" + theirs="$(PLAY="$play" yq -o=json -I0 ".[] | select(.name == strenv(PLAY)) | $norm" "$fleet_pb")" + [[ -n "$mine" && "$mine" == "$theirs" ]] \ + || fail "galaxy/node/playbooks/$pb: play \"$play\" differs from playbooks/$pb's (beyond hosts and the sponsord check)" + done <<<"$node_plays" + while IFS= read -r play; do + grep -qxF -- "$play" <<<"$node_plays" \ + || fail "galaxy/node/playbooks/$pb lacks playbooks/$pb's decdn_nodes play \"$play\"" + done < <(yq '.[] | select([.hosts] | flatten | map(split(":")) | flatten | any_c(. == "decdn_nodes")) | .name' "$fleet_pb") + [[ "$(yq '[.[] | select([.hosts] | flatten | any_c(. != "decdn_nodes"))] | length' "$node_pb")" == 0 ]] \ + || fail "galaxy/node/playbooks/$pb: a play targets more than decdn_nodes" + done + pass "decdn.node's backup.yml and decommission.yml are exactly the fleet-wide node plays" +elif [[ -n ${CI:-} ]]; then + fail "yq is not on PATH in CI; the node lifecycle playbook check would be skipped" +else + skipped+=("node lifecycle playbook check (needs yq)") +fi # --- release gate ---------------------------------------------------------------------- gate="$repo/scripts/check-release-version.sh" @@ -642,7 +762,7 @@ fi # --- scripts/release.sh: version, changelog section, signed commit and tag ------------- # On a fixture repo with a bare origin: small manifests and changelogs at the 0.0.0 # placeholder, the real scripts, cliff.toml, galaxy/build.sh and each collection's -# roles.txt, and an ephemeral ssh signing key. The host's git config is kept out (GIT_CONFIG_GLOBAL). The +# roles.txt and playbooks.txt, and an ephemeral ssh signing key. The host's git config is kept out (GIT_CONFIG_GLOBAL). The # make checks are stubs that log their target to $RS_MAKE_LOG and fail while # $RS_MAKE_FAIL exists; most cases skip them with --no-verify. if command -v git-cliff >/dev/null && command -v ssh-keygen >/dev/null; then @@ -703,7 +823,7 @@ if command -v git-cliff >/dev/null && command -v ssh-keygen >/dev/null; then printf 'galaxy-check-%%:\n\t@echo galaxy-check-$* >> "$$RS_MAKE_LOG"; test ! -e "$$RS_MAKE_FAIL"\n' > "$rs/ansible/Makefile" printf 'apiVersion: v2\nname: decdn-node\nversion: 0.0.0\nappVersion: "0.0.1"\n' > "$chart/Chart.yaml" for c in node publisher; do - cp "$repo/ansible/galaxy/$c/roles.txt" "$rs/ansible/galaxy/$c/" + cp "$repo/ansible/galaxy/$c/roles.txt" "$repo/ansible/galaxy/$c/playbooks.txt" "$rs/ansible/galaxy/$c/" printf 'namespace: decdn\nname: %s\nversion: 0.0.0 # a comment\n' "$c" > "$rs/ansible/galaxy/$c/galaxy.yml" done printf 'dependencies:\n decdn.node: ">=0.0.2"\n' >> "$rs/ansible/galaxy/publisher/galaxy.yml" @@ -961,13 +1081,27 @@ if command -v git-cliff >/dev/null && command -v ssh-keygen >/dev/null; then rscommit ansible/galaxy/build.sh "fix(galaxy): a build fix" expect 0 "release.sh counts galaxy/build.sh for the node collection" rsrel node-collection rssaid "after a build.sh fix" "+- **galaxy**: A build fix" + # Playbooks count for each collection that lists them in its playbooks.txt, and + # their group_vars/ for both: node.yml ships in both, sponsord.yml in publisher only. + rscommit ansible/playbooks/node.yml "fix(playbooks): a node play fix" + rscommit ansible/playbooks/sponsord.yml "fix(playbooks): a sponsord play fix" + rscommit ansible/playbooks/group_vars/all.yml "fix(playbooks): a firewall hole fix" + rscommit ansible/galaxy/stage-playbooks.py "fix(galaxy): a staging fix" + expect 0 "release.sh counts the node collection's playbooks" rsrel node-collection + rssaid "after a stage-playbooks.py fix" "+- **galaxy**: A staging fix" + rssaid "after a node.yml fix" "+- **playbooks**: A node play fix" + rssaid "after a group_vars fix" "+- **playbooks**: A firewall hole fix" + ! grep -qF "A sponsord play fix" "$work/out" || fail "release.sh counted sponsord.yml, which decdn.node does not ship" echo unshipped >> "$rs/ansible/galaxy/publisher/roles.txt" rsgit commit -qam "build: ship the unshipped role"; rspush expect 0 "release.sh reads the collection's roles from its roles.txt" rsrel publisher-collection rssaid "with a role added to roles.txt" "+- **unshipped**: Not in a roles.txt" + rssaid "after a sponsord.yml fix" "+- **playbooks**: A sponsord play fix" + rssaid "after a node.yml fix, for publisher" "+- **playbooks**: A node play fix" + rssaid "after a group_vars fix, for publisher" "+- **playbooks**: A firewall hole fix" rssaid "with a feature at 0.0.x" "publisher-collection-v0.0.1 -> publisher-collection-v0.1.0" ! grep -qF "Ship the unshipped role" "$work/out" || fail "release.sh's collection section lists a build commit" - pass "release.sh releases each collection on its own tags, roles.txt and overlay, publisher after node" + pass "release.sh releases each collection on its own tags, roles.txt, playbooks.txt and overlay, publisher after node" unset GIT_CONFIG_GLOBAL GIT_CONFIG_NOSYSTEM RS_MAKE_LOG RS_MAKE_FAIL elif [[ -n ${CI:-} ]]; then fail "git-cliff or ssh-keygen is not on PATH in CI; the release.sh tests would be skipped" @@ -1492,7 +1626,16 @@ done cmp -s "$repo/ansible/roles/decdn_node/files/install-build-output.py" \ "$repo/ansible/roles/sponsord/files/install-build-output.py" \ || fail "decdn_node and sponsord files/install-build-output.py differ" -pass "source-build and decommission defaults, and the installer, identical across roles" +# decdn_backup_local_dir and sponsord_backup_local_dir: one expression (backups/ beside +# the repo's playbooks/, or beside the project's ansible.cfg from a collection), the +# key's own name aside. Only decdn_node's runs in a molecule scenario. +backup_dir_default() { # + awk -v k="$2" '$0 ~ "^" k ":" {on = 1; next} on && /^ / {print; next} on {exit}' "$1" | sed "s/$2/KEY/g" +} +want="$(backup_dir_default "$repo/ansible/roles/decdn_node/defaults/main.yml" decdn_backup_local_dir)" +[[ -n "$want" && "$(backup_dir_default "$repo/ansible/roles/sponsord/defaults/main.yml" sponsord_backup_local_dir)" == "$want" ]] \ + || fail "sponsord_backup_local_dir's default differs from decdn_backup_local_dir's" +pass "source-build and decommission defaults, the backup directory and the installer, identical across roles" # iroh_relay and iroh_dns_server have no source build (and their own per-triple # sha256 pins), so only their decommission keys are compared. decom_keys() { grep -E '^decdn_decommission_(max_hosts|prompt_seconds):' "$1" || true; }