diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d2c37af..40acc32 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -127,8 +127,8 @@ jobs: - name: Lint (yamllint + ansible-lint) run: make lint - name: Syntax-check playbooks - # Dummy inventory: no real hosts.yml is committed upstream (only hosts.yml.example), - # and --syntax-check only parses, it never connects. Every playbook, so a new + # Dummy inventory: no real hosts.yml is committed upstream (only the + # hosts-*.yml.example files), and --syntax-check only parses, it never connects. Every playbook, so a new # one (backup, decommission, ...) is covered without editing this step. run: | for pb in playbooks/*.yml; do diff --git a/AGENTS.md b/AGENTS.md index d1e3060..5204aa6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,7 +72,7 @@ mirror (`vars/main/networks.yml`), never hand-copied into inventory or docs. ``` ansible/ # the deployment project (DevSec-hardened, lean roles) - playbooks/ # site.yml (node + sponsord + relay + DNS server), sponsord.yml (+ onramp), iroh_relay.yml, iroh_dns_server.yml, backup.yml, decommission.yml + playbooks/ # site.yml = node.yml (node operators: cache nodes) + publisher.yml (publishers: origin.yml, sponsord.yml (+ onramp), iroh_relay.yml, iroh_dns_server.yml); backup.yml, decommission.yml roles/ # baseline, decdn_node, grafana_alloy, sponsord, sponsord_onramp, iroh_relay, iroh_dns_server inventory/ galaxy/ molecule/ # see ansible/README.md cloud-init/ # user-data{,-sponsord}.yaml + on-host bootstrap.sh; pinned ansible-core/collections (see its README.md) @@ -141,8 +141,21 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). ## Current services -- **`ansible/`** — the declarative deployment project. **The public deCDN node** - (`playbooks/site.yml` → baseline + `decdn-node`), installed by `decdn_node_install_method`: +- **`ansible/`** — the declarative deployment project, with an entry point per persona: + `playbooks/node.yml` (`make deploy-node`) for **node operators**, `playbooks/publisher.yml` + (`make deploy-publisher`) for **publishers**, and `site.yml` (`make deploy`) for both. + **Origin nodes** are `decdn_origin_nodes`, a child of `decdn_nodes` (so they inherit the + node's firewall hole, backup, decommission, relay allowlist and Alloy scrape), deployed by + `playbooks/origin.yml` with the same roles; node.yml's play is + `decdn_nodes:!decdn_origin_nodes`, so each node runs once under site.yml. origin.yml's + `pre_tasks` refuse an origin outside `decdn_nodes` or without an origin backend + (`decdn_cache_origin_kind`/`decdn_cache_origins`); node.yml's warn about a backend outside + `decdn_origin_nodes`. Those group checks live in the playbooks, never the role, as with + sponsord's. A cache node has no origin backend: it fills misses from other nodes + (node-to-node pull-through), and an origin-configured node serves only what its own + backend holds (`relay_foreign_namespaces`, ADR 002 § Retrieval by namespace, ADR 037). Recognition as an origin is on-chain (`OriginAssignment.addOrigin`, + the publisher's step). **The public deCDN node** + (baseline + `decdn-node`), installed by `decdn_node_install_method`: `release` (the default) — a pinned GitHub release tarball verified against the release's GPG-signed `SHA256SUMS` (`decdn_node_version`, default `0.0.1`); `source` — a git ref (any tag/branch/SHA) cloned and `cargo build`-ed **on the node** as the unprivileged `decdn-build` user, sha256-pinned rustup, a `@` @@ -219,7 +232,7 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). - **`ansible/roles/sponsord`** — the deCDN onboarding sponsor (`decdn/sponsord`): the treasury wallet's signer for capped capabilities plus the PaymentPool keeper. It is independent of the node, so `playbooks/sponsord.yml` (`make deploy-sponsord`, also - imported by `site.yml`) targets its own `sponsord_hosts` group, standalone or + imported by `publisher.yml`) targets its own `sponsord_hosts` group, standalone or co-located with a node. - **Install:** a GPG-verified `v*` decdn/sponsord release by default (`sponsord_version`, `0.0.2`), a host-side `source` build, or a manual binary. @@ -290,7 +303,7 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). QUIC address discovery. The ADRs make relays operator-run infrastructure, not an incentivized role (`decdn/adr/architecture.md`, Trust Assumptions); nodes use them through `decdn_relay_urls`, which REPLACES n0's relays (so deploy at least two). - `playbooks/iroh_relay.yml` (`make deploy-relay`, also imported by `site.yml`) runs + `playbooks/iroh_relay.yml` (`make deploy-relay`, also imported by `publisher.yml`) runs baseline → grafana_alloy → iroh_relay on `iroh_relay_hosts`, after a play refusing a host also in `sponsord_onramp_hosts` (both want 80/443; `make test-scripts` runs it). - **Access:** `iroh_relay_access` defaults to `allowlist`. A play before the @@ -351,7 +364,7 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). `dns.iroh.link` (ADR 001, Node Discovery). Nodes use it through `decdn_discovery_pkarr_url` + `decdn_discovery_dns_origin`, which DROP the n0 leg, so clients need the same `dns_origin`. `playbooks/iroh_dns_server.yml` (`make - deploy-dns`, also imported by `site.yml`) runs baseline → grafana_alloy → + deploy-dns`, also imported by `publisher.yml`) runs baseline → grafana_alloy → iroh_dns_server on `iroh_dns_server_hosts`, after a play refusing a host also in `iroh_relay_hosts` or `sponsord_onramp_hosts` (all want tcp/443; `make test-scripts` runs it). One server per origin: upstream has no replication, and @@ -393,7 +406,7 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). - installs ansible-core from the hash-locked `requirements.txt` into a venv (two pins split by Python marker: 2.19 for Debian 12's 3.11, 2.21 for 3.12 and later); - installs the exact collections from `collections.lock.yml`; - - runs `site.yml` (which imports `sponsord.yml`) against localhost. The inventory + - runs `site.yml` (which imports `node.yml` and `publisher.yml`) against localhost. The inventory must put localhost in `decdn_nodes` and/or `sponsord_hosts` (and in `sponsord_hosts` whenever it is in `sponsord_onramp_hosts`), or the plays match nothing and the firewall holes never load. @@ -509,6 +522,9 @@ make security # KICS over ansible/, the rendered chart and compose/ # Ansible — run from ansible/ make deps # vendor pinned Galaxy collections make check / deploy # site.yml; fleet-wide unless LIMIT=; INVENTORY= +make check-node / deploy-node # node.yml only (node operators: cache nodes) +make check-publisher / deploy-publisher # publisher.yml only (origins, sponsord, relays, DNS) +make check-origin / deploy-origin # origin.yml only (decdn_origin_nodes) make check-sponsord / deploy-sponsord # sponsord.yml only (sponsord_hosts) make check-relay / deploy-relay # iroh_relay.yml only (iroh_relay_hosts) make check-dns / deploy-dns # iroh_dns_server.yml only (iroh_dns_server_hosts) @@ -520,13 +536,14 @@ make build / galaxy-check # both collections (build-node, galaxy-check- `ansible/inventory/hosts.yml` is git-ignored and a real fleet lives in a private overlay. Inventory-adjacent group_vars do not load for an overlay, so anything every host needs regardless of inventory lives in `ansible/playbooks/group_vars/`. The public firewall holes -(`baseline_extra_inbound`, today the node's udp/4433, plus tcp/80 + tcp/443 on +(`baseline_extra_inbound`, today the node's udp/4433 (an origin's too), plus tcp/80 + tcp/443 on `sponsord_onramp_hosts` with `sponsord_onramp_proxy: caddy`, tcp/80 + tcp/443 + udp/7842 on `iroh_relay_hosts`, the last unless `iroh_relay_enable_quic_addr_discovery` is false, and tcp/443 + udp/53 + tcp/53 on `iroh_dns_server_hosts`) are built from a host's groups in -`all.yml` (`_baseline_service_inbound`). `decdn_nodes.yml`, `sponsord_hosts.yml`, -`iroh_relay_hosts.yml` and `iroh_dns_server_hosts.yml` all set `baseline_extra_inbound` -to that list, so a co-located host renders one firewall in every play. +`all.yml` (`_baseline_service_inbound`). `decdn_nodes.yml`, `decdn_origin_nodes.yml`, +`sponsord_hosts.yml`, `iroh_relay_hosts.yml` and `iroh_dns_server_hosts.yml` all set +`baseline_extra_inbound` to that list, so a co-located host renders one firewall in +every play. `ansible/tests/firewall-holes/` (run by `make test-scripts`) pins the result per host shape. The Alloy per-daemon toggles also live in `all.yml`. Don't move any of it back under `inventory/`. diff --git a/README.md b/README.md index c4b7e1b..59e30bd 100644 --- a/README.md +++ b/README.md @@ -55,13 +55,15 @@ end to end (with `--dry-run`); this repo stops at host prep and startup. ```bash cd ansible make deps # vendor pinned Galaxy collections -cp inventory/hosts.yml.example inventory/hosts.yml -$EDITOR inventory/hosts.yml # your hosts in decdn_nodes -$EDITOR inventory/host_vars/decdn-node-1/main.yml # binaries, region, origin; chain via decdn_network +cp inventory/hosts-node.yml.example inventory/hosts.yml # node operators; its host is decdn-node-1 +# publishers instead: cp inventory/hosts-publisher.yml.example inventory/hosts.yml (host origin-1) +HOST=decdn-node-1 # or origin-1 +$EDITOR inventory/hosts.yml # your hosts +$EDITOR inventory/host_vars/$HOST/main.yml # binaries, region (an origin: its backend); chain via decdn_network # RPC URL: provision 0600 /etc/decdn/decdn.env on the host (preferred), or secret.yml -make check LIMIT=decdn-node-1 ANSIBLE_ARGS='-u root' # dry run; -u root only until the first converge -make deploy LIMIT=decdn-node-1 ANSIBLE_ARGS='-u root' # first converge creates your admin account -make deploy LIMIT=decdn-node-1 # every run after that +make check LIMIT=$HOST ANSIBLE_ARGS='-u root' # dry run; -u root only until the first converge +make deploy LIMIT=$HOST ANSIBLE_ARGS='-u root' # first converge creates your admin account +make deploy LIMIT=$HOST # every run after that ``` The full flow (bootstrap user, keystore, secrets, fleets in a private inventory) is in @@ -103,7 +105,7 @@ This is the canonical statement; the per-path READMEs add only what is specific | Path | What it is | |------|------------| -| [`ansible/`](ansible/README.md) | The Ansible project: `inventory/`, `playbooks/` (`site.yml`, `sponsord.yml`, `iroh_relay.yml`, `iroh_dns_server.yml`, `backup.yml`, `decommission.yml`), `roles/` (`baseline`, `decdn_node`, `grafana_alloy`, `sponsord`, `sponsord_onramp`, `iroh_relay`, `iroh_dns_server`), `galaxy/` (the `decdn.node` and `decdn.publisher` collections), `molecule/`. | +| [`ansible/`](ansible/README.md) | The Ansible project: `inventory/`, `playbooks/` (`site.yml`, `node.yml` for node operators, `publisher.yml` for publishers with `origin.yml`, `sponsord.yml`, `iroh_relay.yml` and `iroh_dns_server.yml`, `backup.yml`, `decommission.yml`), `roles/` (`baseline`, `decdn_node`, `grafana_alloy`, `sponsord`, `sponsord_onramp`, `iroh_relay`, `iroh_dns_server`), `galaxy/` (the `decdn.node` and `decdn.publisher` collections), `molecule/`. | | [`cloud-init/`](cloud-init/README.md) | The cloud-init deploy path: `user-data.yaml` (a node), `user-data-sponsord.yaml` (a sponsor host), the on-host `bootstrap.sh`, and the pinned ansible-core and collections it installs. | | [`compose/`](compose/README.md) | The Docker Compose deploy path: the node, `sponsord`, `sponsord-onramp` and Caddy, one profile each. | | [`charts/decdn-node/`](charts/decdn-node/README.md) | The Helm chart; it renders the node's dashboards and alert rules from `monitoring/`. | diff --git a/ansible/.gitignore b/ansible/.gitignore index cd6fd99..64708a0 100644 --- a/ansible/.gitignore +++ b/ansible/.gitignore @@ -13,9 +13,10 @@ inventory/*/host_vars/*/secret.* !inventory/*/host_vars/*/secret.*.example # The real inventory. This repo is PUBLIC, so real host IPs must never land here. -# Copy hosts.yml.example to hosts.yml for a local run, or keep a fleet's inventory -# in a private overlay and pass INVENTORY= (see README.md "Private fleet -# inventory"). The .example templates stay tracked. +# Copy hosts-node.yml.example (node operators) or hosts-publisher.yml.example +# (publishers) to hosts.yml for a local run, or keep a fleet's inventory in a private +# overlay and pass INVENTORY= (see README.md "Private fleet inventory"). The +# .example templates stay tracked. inventory/hosts.yml inventory/hosts.yaml diff --git a/ansible/Makefile b/ansible/Makefile index a4e239d..23b4c49 100644 --- a/ansible/Makefile +++ b/ansible/Makefile @@ -1,6 +1,6 @@ # Convenience targets for the deCDN Ansible project. # Always run from the ansible/ directory. -.PHONY: deps lint check deploy check-sponsord deploy-sponsord check-relay deploy-relay check-dns deploy-dns backup decommission molecule molecule-serial molecule-list build build-node build-publisher galaxy-check galaxy-check-node galaxy-check-publisher +.PHONY: deps lint check deploy check-node deploy-node check-origin deploy-origin check-publisher deploy-publisher check-sponsord deploy-sponsord check-relay deploy-relay check-dns deploy-dns backup decommission molecule molecule-serial molecule-list build build-node build-publisher galaxy-check galaxy-check-node galaxy-check-publisher SHELL := /bin/bash # Install the required Galaxy collections (>= constraints in requirements.yml) @@ -14,9 +14,9 @@ lint: yamllint . ansible-lint -# --- Primary: the public deCDN node ------------------------------------------ +# --- Deploys: node operators and publishers ----------------------------------- # Optional scoping for the deploy targets. Both are empty by default, so check/deploy -# stay fleet-wide (every host in decdn_nodes) unless you say otherwise. +# stay fleet-wide (every host in the inventory) unless you say otherwise. # make deploy LIMIT=decdn-node-1 # one host # make deploy LIMIT='decdn-node-1,decdn-node-2' # a list # make deploy ANSIBLE_ARGS='--tags decdn_node -vv' @@ -35,7 +35,7 @@ ANSIBLE_ARGS ?= limit_arg = $(if $(LIMIT),--limit '$(LIMIT)') # Which inventory check/deploy run against. The default is the git-ignored -# inventory/hosts.yml (copy hosts.yml.example). A real fleet keeps its inventory in a +# inventory/hosts.yml (copy hosts-node.yml.example or hosts-publisher.yml.example). A real fleet keeps its inventory in a # private overlay outside this public repo and points at it here: # make deploy INVENTORY=../../decdn-fleet/hosts.yml LIMIT= # The overlay's own group_vars/ and host_vars/ must sit beside that hosts.yml, where @@ -44,21 +44,21 @@ INVENTORY ?= inventory/hosts.yml # A missing or unparseable inventory fails the run. Ansible's default only warns, # skips the `hosts: decdn_nodes` play and exits 0, so a mistyped INVENTORY= would # report a successful deploy that did nothing. This applies to the playbook targets -# (check/deploy/check-sponsord/deploy-sponsord/check-relay/deploy-relay/check-dns/deploy-dns/ +# (check/deploy and their -node/-origin/-publisher/-sponsord/-relay/-dns variants, # backup/decommission) only: # ansible.cfg can't set it globally, because ansible-lint's own syntax checks run # with no inventory present. strict_inventory = ANSIBLE_INVENTORY_UNPARSED_FAILED=True # Fail loud on the two ways a run can look scoped but silently be fleet-wide. Scoped to -# the playbook targets (check/deploy[-sponsord|-relay|-dns]/backup/decommission) via MAKECMDGOALS so an +# the playbook targets (check/deploy[-node|-origin|-publisher|-sponsord|-relay|-dns]/backup/decommission) via MAKECMDGOALS so an # unrelated `make lint` is never blocked. NOTE: this # does not catch an argument that is well-formed but selects nothing — `--tags` with a # typo (decdn-node vs decdn_node) runs zero tasks and still exits 0. See ansible/README.md. -ifneq ($(filter check deploy check-sponsord deploy-sponsord check-relay deploy-relay check-dns deploy-dns backup decommission,$(MAKECMDGOALS)),) +ifneq ($(filter check deploy check-node deploy-node check-origin deploy-origin check-publisher deploy-publisher check-sponsord deploy-sponsord check-relay deploy-relay check-dns deploy-dns backup decommission,$(MAKECMDGOALS)),) # Echo back the goal the operator actually typed, so the hint never nudges a # `make check` dry-run toward the mutating `make deploy`. - scoped_goal := $(firstword $(filter check deploy check-sponsord deploy-sponsord check-relay deploy-relay check-dns deploy-dns backup decommission,$(MAKECMDGOALS))) + scoped_goal := $(firstword $(filter check deploy check-node deploy-node check-origin deploy-origin check-publisher deploy-publisher check-sponsord deploy-sponsord check-relay deploy-relay check-dns deploy-dns backup decommission,$(MAKECMDGOALS))) ifeq ($(origin LIMIT),environment) $(error LIMIT is set in your environment ($(LIMIT)) and would silently scope this run. Unset it or pass it on the command line: make $(scoped_goal) LIMIT=) endif @@ -86,16 +86,40 @@ ifneq ($(filter check deploy check-sponsord deploy-sponsord check-relay deploy-r endif endif -# Dry run against $(INVENTORY) — shows diffs, mutates nothing. +# Dry run of everything in $(INVENTORY) (site.yml: node.yml, then publisher.yml) — +# shows diffs, mutates nothing. check: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/site.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) -# Provision the deCDN node host(s) in $(INVENTORY) (default inventory/hosts.yml). +# Provision every host in $(INVENTORY) (default inventory/hosts.yml). deploy: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/site.yml $(limit_arg) $(ANSIBLE_ARGS) -# sponsord alone (playbooks/sponsord.yml, hosts in sponsord_hosts). check/deploy -# above already include it via site.yml; these leave every decdn-node untouched. +# Node operators: the cache nodes alone (playbooks/node.yml, decdn_nodes that are not +# in decdn_origin_nodes). +check-node: + $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/node.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) + +deploy-node: + $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/node.yml $(limit_arg) $(ANSIBLE_ARGS) + +# Publishers: everything a publisher runs (playbooks/publisher.yml: origins, sponsord +# and its onramp, iroh relays, iroh DNS servers), leaving the cache nodes untouched. +check-publisher: + $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/publisher.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) + +deploy-publisher: + $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/publisher.yml $(limit_arg) $(ANSIBLE_ARGS) + +# A publisher's origin nodes alone (playbooks/origin.yml, hosts in decdn_origin_nodes). +check-origin: + $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/origin.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) + +deploy-origin: + $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/origin.yml $(limit_arg) $(ANSIBLE_ARGS) + +# sponsord alone (playbooks/sponsord.yml, hosts in sponsord_hosts). check/deploy and +# check/deploy-publisher above already include it; these leave every decdn-node untouched. check-sponsord: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/sponsord.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) @@ -103,7 +127,8 @@ deploy-sponsord: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/sponsord.yml $(limit_arg) $(ANSIBLE_ARGS) # iroh relays alone (playbooks/iroh_relay.yml, hosts in iroh_relay_hosts). check/deploy -# above already include them via site.yml; these leave every other host untouched. +# and check/deploy-publisher above already include them; these leave every other host +# untouched. check-relay: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/iroh_relay.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) @@ -111,8 +136,8 @@ deploy-relay: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/iroh_relay.yml $(limit_arg) $(ANSIBLE_ARGS) # iroh DNS servers alone (playbooks/iroh_dns_server.yml, hosts in iroh_dns_server_hosts). -# check/deploy above already include them via site.yml; these leave every other host -# untouched. +# check/deploy and check/deploy-publisher above already include them; these leave every +# other host untouched. check-dns: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/iroh_dns_server.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) diff --git a/ansible/README.md b/ansible/README.md index 6bf8f8a..ae2db70 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -7,12 +7,15 @@ origin nodes). | Playbook | Purpose | Make target | |----------|---------|-------------| -| **`site.yml`** | Harden the host and deploy the public **deCDN node** (`decdn-node`). | `make check` / `make deploy` | +| **`site.yml`** | Everything in the inventory: `node.yml`, then `publisher.yml`. | `make check` / `make deploy` | +| **`node.yml`** | **Node operators.** Harden the host and deploy a public deCDN **cache node** (`decdn-node`) on hosts in `decdn_nodes` that are not in `decdn_origin_nodes`. | `make check-node` / `make deploy-node` | +| **`publisher.yml`** | **Publishers.** `origin.yml`, `sponsord.yml`, `iroh_relay.yml` and `iroh_dns_server.yml`, in that order. | `make check-publisher` / `make deploy-publisher` | +| `origin.yml` | Harden the host and deploy a publisher's **origin node**: `decdn-node` with an origin backend, on hosts in `decdn_origin_nodes` (a child of `decdn_nodes`). Refuses an origin outside `decdn_nodes` or without a backend. | `make check-origin` / `make deploy-origin` | | `backup.yml` | Encrypted backup of a node's identity or full state, and of sponsord's credentials. | `make backup` | | `decommission.yml` | Stop a node, an iroh relay, an iroh DNS server, the onramp and sponsord and remove their services (keeps the keys; no on-chain steps). | `make decommission` | -| `sponsord.yml` | Harden the host and deploy **sponsord**, the onboarding sponsor, on hosts in `sponsord_hosts`, with or without a node; then its public **sponsord-onramp** (behind Caddy) on hosts in `sponsord_onramp_hosts`. Also run by `site.yml`. | `make check-sponsord` / `make deploy-sponsord` | -| `iroh_relay.yml` | Harden the host and deploy a self-hosted **iroh relay** (`iroh-relay`), the fallback path for peers that cannot hole-punch, on hosts in `iroh_relay_hosts`. Also run by `site.yml`. | `make check-relay` / `make deploy-relay` | -| `iroh_dns_server.yml` | Harden the host and deploy a self-hosted **iroh DNS server** (`iroh-dns-server`), the pkarr relay and DNS server nodes are discovered through, on hosts in `iroh_dns_server_hosts`. Also run by `site.yml`. | `make check-dns` / `make deploy-dns` | +| `sponsord.yml` | Harden the host and deploy **sponsord**, the onboarding sponsor, on hosts in `sponsord_hosts`, with or without a node; then its public **sponsord-onramp** (behind Caddy) on hosts in `sponsord_onramp_hosts`. Also run by `publisher.yml`. | `make check-sponsord` / `make deploy-sponsord` | +| `iroh_relay.yml` | Harden the host and deploy a self-hosted **iroh relay** (`iroh-relay`), the fallback path for peers that cannot hole-punch, on hosts in `iroh_relay_hosts`. Also run by `publisher.yml`. | `make check-relay` / `make deploy-relay` | +| `iroh_dns_server.yml` | Harden the host and deploy a self-hosted **iroh DNS server** (`iroh-dns-server`), the pkarr relay and DNS server nodes are discovered through, on hosts in `iroh_dns_server_hosts`. Also run by `publisher.yml`. | `make check-dns` / `make deploy-dns` | ``` baseline host hardening: DevSec os/ssh, nftables default-deny inbound, @@ -66,7 +69,7 @@ specific to this path: which doesn't honor the custom `-p` become prompt Ansible relies on — so `--ask-become-pass` hangs with "Timeout waiting for privilege escalation prompt". On an affected host uncomment `ansible_become_exe: /usr/bin/sudo.ws` for that host in - `hosts.yml` (see the note in `inventory/hosts.yml.example`) to route become through + `hosts.yml` (see the note in `inventory/hosts-node.yml.example`) to route become through classic sudo. ## Setup @@ -74,8 +77,9 @@ specific to this path: ```bash cd ansible make deps # vendor pinned collections into ./collections -cp inventory/hosts.yml.example inventory/hosts.yml -$EDITOR inventory/hosts.yml # set hosts for decdn_nodes +cp inventory/hosts-node.yml.example inventory/hosts.yml # node operators +cp inventory/hosts-publisher.yml.example inventory/hosts.yml # or publishers +$EDITOR inventory/hosts.yml # set your hosts $EDITOR inventory/group_vars/all.yml # optional: override admin user/keys, allowlists ``` @@ -91,8 +95,9 @@ canonical copy; don't duplicate them here. **Or** carry it in inventory, and the role authors `decdn.env` from those values on every run: ```bash -cp inventory/host_vars/decdn-node-1/secret.yml.example inventory/host_vars/decdn-node-1/secret.yml -$EDITOR inventory/host_vars/decdn-node-1/secret.yml # set decdn_rpc_url +HOST=decdn-node-1 # or origin-1, the publisher example's host +cp inventory/host_vars/$HOST/secret.yml.example inventory/host_vars/$HOST/secret.yml +$EDITOR inventory/host_vars/$HOST/secret.yml # set decdn_rpc_url ``` The role fails loud when neither exists, and — once it has recorded a checksum for the file — @@ -110,7 +115,7 @@ point the deploy targets at it: ```bash mkdir -p ../../decdn-fleet # a PRIVATE repo/dir -cp inventory/hosts.yml.example ../../decdn-fleet/hosts.yml # then fill it in +cp inventory/hosts-node.yml.example ../../decdn-fleet/hosts.yml # then fill it in make check INVENTORY=../../decdn-fleet/hosts.yml LIMIT= make deploy INVENTORY=../../decdn-fleet/hosts.yml LIMIT= ``` @@ -120,12 +125,14 @@ the overlay carries its own: create them next to its `hosts.yml`, starting from `inventory/group_vars/` and `inventory/host_vars/` templates. Git-ignore `host_vars/*/secret.*` in the private repo, so it never tracks a per-node secret. The public `inventory/group_vars/` and `inventory/host_vars/` are **not** loaded for it. Add child groups -of your own when hosts differ by group (an origin, an egress cap on metered bandwidth), each -with a `group_vars/.yml`. Settings every host needs regardless of +of your own when hosts differ by group (disk size, an egress cap on metered bandwidth), each +with a `group_vars/.yml`. Origins go in `decdn_origin_nodes`, as a child of +`decdn_nodes` (see `inventory/hosts-publisher.yml.example`). Settings every host needs regardless of inventory, currently the public firewall holes (the node's udp/4433, and tcp/80 + tcp/443 for Caddy on `sponsord_onramp_hosts`), live in `playbooks/group_vars/`, so an overlay can't drop them: `all.yml` builds the list from the -host's groups and `decdn_nodes.yml` / `sponsord_hosts.yml` apply it. Override +host's groups and each group's file (`decdn_nodes.yml`, `decdn_origin_nodes.yml`, +`sponsord_hosts.yml`, …) applies it. Override `baseline_extra_inbound` per host in `host_vars` if you have to. Playbook group_vars beat inventory group_vars. @@ -194,10 +201,12 @@ ignored on the first converge. ```bash make check # dry run (--check --diff); asserts fire if required knobs are missing -make deploy # provision + start the node +make deploy # provision + start every service in the inventory +make deploy-node # node operators: the cache nodes only +make deploy-publisher # publishers: origins, sponsord, relays, DNS servers only ``` -Both targets are **fleet-wide by default** — every host in `decdn_nodes`. Scope a run with +All of them are **fleet-wide by default** — every host their playbooks target. Scope a run with `LIMIT`: ```bash diff --git a/ansible/galaxy/node/CHANGELOG.md b/ansible/galaxy/node/CHANGELOG.md index e347188..2a8c619 100644 --- a/ansible/galaxy/node/CHANGELOG.md +++ b/ansible/galaxy/node/CHANGELOG.md @@ -195,6 +195,10 @@ Initial packaging of the deCDN node roles as a distributable collection ### Changed +- `decdn_node`: the docs and the origin-kind assert no longer say a serving node + needs an origin backend. Without one a node is a cache node, which fills a miss + from other nodes (`decdn_node_to_node_pull_through_enabled`, on by default without + an origin); a node with one is a publisher's origin. - The seven roles ship as two collections. `decdn.node` keeps what a node operator runs (`baseline`, `decdn_node`, `grafana_alloy`); `sponsord`, `sponsord_onramp`, `iroh_relay` and `iroh_dns_server` move to `decdn.publisher`, diff --git a/ansible/inventory/group_vars/decdn_nodes.yml b/ansible/inventory/group_vars/decdn_nodes.yml index 352e450..2cc0cb5 100644 --- a/ansible/inventory/group_vars/decdn_nodes.yml +++ b/ansible/inventory/group_vars/decdn_nodes.yml @@ -1,5 +1,7 @@ --- -# Shared config for public deCDN node hosts (playbooks/site.yml). +# Shared config for public deCDN node hosts, cache nodes (playbooks/node.yml) and a +# publisher's origins alike (playbooks/origin.yml; group_vars/decdn_origin_nodes.yml +# wins over this file for them). # Per-node values live in host_vars//: addresses/region/version in main.yml. # The sensitive rpc_url is provisioned as /etc/decdn/decdn.env on the host, or (if # you would rather the role authored that file) in a git-ignored secret.yml. diff --git a/ansible/inventory/group_vars/decdn_origin_nodes.yml b/ansible/inventory/group_vars/decdn_origin_nodes.yml new file mode 100644 index 0000000..3d04ada --- /dev/null +++ b/ansible/inventory/group_vars/decdn_origin_nodes.yml @@ -0,0 +1,25 @@ +--- +# Shared config for a publisher's origin nodes (playbooks/origin.yml, also run by +# publisher.yml and site.yml). decdn_origin_nodes is a child of decdn_nodes, so +# group_vars/decdn_nodes.yml applies to origins too, and this file wins over it. +# Per-origin values live in host_vars//main.yml (see host_vars/origin-1/). +# +# Every origin needs an origin backend: decdn_cache_origin_kind (http, fs or s3) and +# that kind's fields, or decdn_cache_origins (an ordered fallback list); origin.yml +# refuses a host without one. Set it here when every origin reads the same store, in +# host_vars when they differ. Full list: roles/decdn_node/defaults/main.yml. +# +# decdn_cache_origin_kind: s3 +# decdn_cache_origin_s3_bucket: my-namespace-content +# decdn_cache_origin_s3_region: eu-central-1 +# decdn_cache_origin_s3_endpoint_url: "" # R2/B2/MinIO; omit for AWS +# decdn_cache_origin_s3_use_default_chain: true # keys in /etc/decdn/decdn.env + +# --- Origin behaviour ---------------------------------------------------------------- +# With a backend set, two defaults follow: decdn_relay_foreign_namespaces false (the +# daemon's: the node serves only what its backend holds and declines the rest, even a +# blob already in its cache; ADR 002 § Retrieval by namespace, ADR 037) and +# decdn_node_to_node_pull_through_enabled false (the role's: misses fill from the +# backend). Set either explicitly only to change that. +# decdn_fs_rescan_interval_sec: 60 # fs backend: how often the root is rescanned +# decdn_pinned_hashes: [] # blobs (lowercase BLAKE3) never evicted diff --git a/ansible/inventory/group_vars/iroh_dns_server_hosts.yml b/ansible/inventory/group_vars/iroh_dns_server_hosts.yml index 77d8f4e..39cb403 100644 --- a/ansible/inventory/group_vars/iroh_dns_server_hosts.yml +++ b/ansible/inventory/group_vars/iroh_dns_server_hosts.yml @@ -1,7 +1,7 @@ --- # Shared config for iroh DNS server hosts (playbooks/iroh_dns_server.yml, also run # by site.yml). Uncomment and fill in; per-host values (the hostname, at least) go -# under the host in hosts.yml (as hosts.yml.example shows) or in +# under the host in hosts.yml (as hosts-publisher.yml.example shows) or in # host_vars//main.yml. # Full variable list: roles/iroh_dns_server/defaults/main.yml and its README.md. # Nothing to provision on the host: the server generates its ACME account key and diff --git a/ansible/inventory/group_vars/iroh_relay_hosts.yml b/ansible/inventory/group_vars/iroh_relay_hosts.yml index 05ae5b5..7214679 100644 --- a/ansible/inventory/group_vars/iroh_relay_hosts.yml +++ b/ansible/inventory/group_vars/iroh_relay_hosts.yml @@ -1,7 +1,7 @@ --- # Shared config for iroh relay hosts (playbooks/iroh_relay.yml, also run by # site.yml). Uncomment and fill in; per-host values (the hostname, at least) go -# under the host in hosts.yml (as hosts.yml.example shows) or in +# under the host in hosts.yml (as hosts-publisher.yml.example shows) or in # host_vars//main.yml. # Full variable list: roles/iroh_relay/defaults/main.yml and its README.md. # Nothing to provision on the host: the relay generates its ACME account key and diff --git a/ansible/inventory/host_vars/decdn-node-1/main.yml b/ansible/inventory/host_vars/decdn-node-1/main.yml index 3833177..00c4465 100644 --- a/ansible/inventory/host_vars/decdn-node-1/main.yml +++ b/ansible/inventory/host_vars/decdn-node-1/main.yml @@ -44,12 +44,11 @@ decdn_node_install_method: release # decdn_*_address here only to override it deliberately; the role reports overrides. decdn_network: arbitrum-sepolia -# --- Cache pull-through origin (what the node fetches on a cache miss) --------- -# MUST-EDIT — operator-specific backing origin (NOT a chain fact). A serving node -# needs one, else cache misses fail NoOrigin. http shown; see defaults/main.yml for -# fs / s3 fields, and decdn_cache_origins for an ordered fallback list. -decdn_cache_origin_kind: "http" -decdn_cache_origin_url: "https://your-origin.example/" +# --- Cache ------------------------------------------------------------------- +# A cache node has no origin backend (decdn_cache_origin_kind stays unset): it fills +# a cache miss by pulling from other nodes (node-to-node pull-through, on by default +# when no origin is set) and relays every namespace. An origin backend makes a node a +# publisher's origin instead: see host_vars/origin-1/ and hosts-publisher.yml.example. # --- Node identity / locale --------------------------------------------------- decdn_region: "US" # MUST-EDIT — ISO 3166-1 alpha-2 of the node's physical location diff --git a/ansible/inventory/host_vars/origin-1/main.yml b/ansible/inventory/host_vars/origin-1/main.yml new file mode 100644 index 0000000..38f1a76 --- /dev/null +++ b/ansible/inventory/host_vars/origin-1/main.yml @@ -0,0 +1,73 @@ +--- +# Per-node deployment values for origin-1, a publisher's origin node (COMMITTED — +# non-secret config only). Ansible merges every *.yml in this directory for host +# origin-1. An origin is a node like decdn-node-1 (see that host's main.yml) plus +# the origin backend below. The sensitive +# values (decdn_rpc_url, and any environment-borne secrets) live OFF this file: +# either in a 0600 /etc/decdn/decdn.env provisioned on the target host (preferred — +# nothing sensitive on the control machine), or in the sibling git-ignored +# secret.yml (copy secret.yml.example). The role refuses to start the node until +# this file is set and ONE of those two secret sources exists. +# +# CONTRACT ADDRESSES ARE NOT INVENTED HERE, and not copied here either: +# decdn_network below selects the role's mirror of upstream's deployment manifest +# (roles/decdn_node/vars/main/networks.yml, regenerated by +# scripts/sync-network-profiles.py from contracts/deployments/.json in +# decdn/decdn). +# It is the same manifest `decdn config init --chain arbitrum-sepolia` bakes in, and +# upstream redeploys change every address at once, so one mirror beats a hand copy +# per inventory. To target a different deployment, clear decdn_network and set +# decdn_chain_id plus the decdn_*_address variables from that deployment's manifest. +# MUST-EDIT lines are yours to fill. + +# --- Which binaries to install ------------------------------------------------ +# "release" (the default): the v GitHub Release tarballs, verified against +# the GPG-signed SHA256SUMS manifest. The version defaults to the decdn/decdn release +# the role pins; set it only to hold a node on another one. +decdn_node_install_method: release +# decdn_node_version: "0.0.1" +# +# Alternative: build on the node from a git ref (any tag, branch or SHA), as an +# unprivileged build user with a pinned toolchain (decdn_node_version is not checked +# in this mode): +# decdn_node_install_method: source +# decdn_node_source_ref: v0.0.1 +# Alternative: copy binaries built on this control machine +# (`cargo build --release -p decdn-node -p decdn-cli` in a decdn/decdn checkout): +# decdn_node_install_method: manual +# decdn_node_manual_bin_src: /path/to/decdn/target/release/decdn-node +# decdn_cli_manual_bin_src: /path/to/decdn/target/release/decdn + +# --- Chain (Arbitrum Sepolia, chain 421614) ----------------------------------- +# The RPC endpoint is the one sensitive value — provision it as DECDN_RPC_URL in +# /etc/decdn/decdn.env on the host, or set decdn_rpc_url in secret.yml. +# The profile supplies decdn_chain_id and all eight contract addresses (the four +# required ones, SlashAppeal, OriginAssignment, PublisherRegistry and USDC). Set any +# decdn_*_address here only to override it deliberately; the role reports overrides. +decdn_network: arbitrum-sepolia + +# --- Origin backend (the canonical source of the namespace's content) --------- +# MUST-EDIT — where this origin reads the content it serves (NOT a chain fact). +# http shown; fs (a local disk or an NFS mount: decdn_cache_origin_path) and s3 +# (S3/R2/B2/MinIO: decdn_cache_origin_s3_*) are in roles/decdn_node/defaults/main.yml, +# and decdn_cache_origins takes an ordered fallback list. S3 access keys never go +# here: put them in /etc/decdn/decdn.env on the host (AWS_ACCESS_KEY_ID, …) with +# decdn_cache_origin_s3_use_default_chain: true, or in secret.yml's decdn_extra_env. +# playbooks/origin.yml refuses a decdn_origin_nodes host without a backend. +decdn_cache_origin_kind: "http" +decdn_cache_origin_url: "https://your-origin.example/" +# Being configured is not being recognised: the namespace's publisher seats this +# node's operator with OriginAssignment.addOrigin (decdn/adr/011-content-takedown.md, +# Origin Assignment Authority). With a backend set, the node serves only what the +# backend holds (decdn_relay_foreign_namespaces defaults to false) and does not pull +# from other nodes (decdn_node_to_node_pull_through_enabled). + +# --- Node identity / locale --------------------------------------------------- +decdn_region: "US" # MUST-EDIT — ISO 3166-1 alpha-2 of the node's physical location +# decdn_relay_urls: [] # optional iroh relays for NAT traversal ([] => n0 defaults) + +# --- Economics ---------------------------------------------------------------- +# Seller-side quote: this node's asking price, advertised verbatim and never clamped +# to the delivery floor by the daemon (ADR 003 §Rate-floor enforcement, ADR 019 +# §Step 3.3 — the ADRs, not this file, govern that behaviour). +decdn_rate_per_mb: 10 # USDC base units (6 decimals); 10 = $0.00001/MB diff --git a/ansible/inventory/host_vars/origin-1/secret.yml.example b/ansible/inventory/host_vars/origin-1/secret.yml.example new file mode 100644 index 0000000..c511b8c --- /dev/null +++ b/ansible/inventory/host_vars/origin-1/secret.yml.example @@ -0,0 +1,56 @@ +--- +# Per-node SECRETS for origin-1 — the INVENTORY path, which is optional. +# +# You only need this file if you want the role to author /etc/decdn/decdn.env for +# you. The preferred posture is the other way round: provision that 0600 file on +# the target host (see roles/decdn_node/files/decdn.env.example) and skip this +# file entirely — the secret then never sits in plaintext on your control machine +# and never transits it on a converge. The role fails loud if NEITHER exists. +# +# If you do use it: copy to secret.yml (same directory, git-ignored) and fill in. +# Ansible merges it with main.yml for this host. These are the ONLY per-node +# values kept out of git — everything else lives in the committed main.yml. +# +# SENSITIVE (may embed an API key). Setting this makes the role REWRITE +# /etc/decdn/decdn.env from these values on every run, discarding any lines added +# on the host. The public endpoint below works for light use; run your own or use +# a provider for production reliability. +decdn_rpc_url: "https://sepolia-rollup.arbitrum.io/rpc" + +# Optional, and only meaningful alongside decdn_rpc_url above (the role writes the +# env file all-or-nothing, and rejects extra_env against a host-provisioned file +# rather than silently dropping it). Extra KEY: value pairs appended to the 0600 +# /etc/decdn/decdn.env that systemd hands the daemon — most notably the AWS +# credentials backing an S3 cache origin, which the role deliberately does NOT +# write into node.toml (that file is 0640 and diffable): +# +# decdn_extra_env: +# AWS_ACCESS_KEY_ID: "AKIA..." +# AWS_SECRET_ACCESS_KEY: "..." +# AWS_SESSION_TOKEN: "..." # only for assume-role / SSO flows +# +# Pair that with decdn_cache_origin_s3_use_default_chain: true in main.yml. + +# --- Grafana Cloud (only when decdn_grafana_cloud_enabled is on) --------------- +# +# The opt-in observability agent accepts its API token through the same channel: +# set grafana_alloy_api_token here and the role authors /etc/grafana-alloy.env on +# the host for you (token-only, root 0600), skipping the hand-provisioning shown +# in roles/grafana_alloy/files/grafana-alloy.env.example. SENSITIVE — leave it +# commented out to keep the operator-provisioned host path instead. +# +# grafana_alloy_api_token: "glc_example_token_replace_me" + +# --- sponsord (only on a host in sponsord_hosts) -------------------------------- +# +# Same choice for sponsord's RPC endpoint: set it here and the role authors +# /etc/sponsord/secret.env (root 0600), or leave it commented out and provision that +# file on the host (roles/sponsord/files/secret.env.example). SENSITIVE. +# +# sponsord_rpc_url: "https://sepolia-rollup.arbitrum.io/rpc" +# +# And for the onramp (a host in sponsord_onramp_hosts): the Turnstile widget's +# secret key. Set it here and the role writes /etc/sponsord/turnstile-secret (root +# 0600), or leave it commented out and write that file on the host. SENSITIVE. +# +# sponsord_onramp_turnstile_secret: "0x4AAAAAAA…" diff --git a/ansible/inventory/hosts.yml.example b/ansible/inventory/hosts-node.yml.example similarity index 53% rename from ansible/inventory/hosts.yml.example rename to ansible/inventory/hosts-node.yml.example index a516691..52947f8 100644 --- a/ansible/inventory/hosts.yml.example +++ b/ansible/inventory/hosts-node.yml.example @@ -1,8 +1,9 @@ --- -# Copy to inventory/hosts.yml and fill in your hosts. inventory/hosts.yml is +# NODE OPERATORS: copy to inventory/hosts.yml and fill in your hosts (a publisher +# starts from hosts-publisher.yml.example instead). inventory/hosts.yml is # git-ignored: this repo is public, so real host IPs never belong in it. To track a -# real fleet, keep its inventory in a private overlay and pass INVENTORY= -# (see README.md "Private fleet inventory"). +# real fleet, keep its inventory in a private overlay and pass INVENTORY= (see +# README.md "Private fleet inventory"). # # No ansible_user by default: Ansible connects as your local $USER, which is the admin # account the baseline role creates (the auto-detected runner). A fresh box doesn't have @@ -29,7 +30,10 @@ # affected host: `sudo --version` reports "sudo-rs" and/or /usr/bin/sudo.ws exists. Leave # it commented on Debian bookworm / Ubuntu noble — that path does not exist there. -# PRIMARY: public deCDN node(s) — provisioned by playbooks/site.yml. +# Public deCDN cache nodes — provisioned by playbooks/node.yml (`make deploy-node`) +# and by site.yml (`make deploy`). A cache node has no origin backend: it fills a +# miss from other nodes (node-to-node pull-through). Origins are a publisher's +# (hosts-publisher.yml.example). # Per-node config (contract addresses, region, version, …) goes in the committed # inventory/host_vars//main.yml; the one secret, rpc_url, goes in a git-ignored # secret.yml beside it (copy the shipped secret.yml.example there). @@ -44,54 +48,9 @@ decdn_nodes: # decdn-node-2: # ansible_host: REPLACE_WITH_NODE_VPS_IP_OR_DNS # Child groups, for hosts that differ in a way that is easier to express per group - # than per host (disk size, an origin, an egress cap on pay-per-TB bandwidth). Give + # than per host (disk size, an egress cap on pay-per-TB bandwidth). Give # each one a group_vars/.yml; Ansible merges it over decdn_nodes'. # children: # big_disk: # hosts: # decdn-node-2: - -# OPTIONAL: sponsord, the onboarding sponsor (treasury signer + PaymentPool keeper) — -# provisioned by playbooks/sponsord.yml (`make deploy-sponsord`) and by site.yml. It -# needs no node: give it its own host, or list a node host here as well to co-locate -# them. Its knobs are in group_vars/sponsord_hosts.yml; the treasury keystore and -# password are provisioned on the host (roles/sponsord/README.md). -# sponsord_hosts: -# hosts: -# sponsord-1: -# ansible_host: REPLACE_WITH_SPONSORD_VPS_IP_OR_DNS -# -# OPTIONAL: sponsord-onramp, sponsord's public side (Turnstile gate, installers, the -# decdn-sponsored CLI API), behind Caddy on tcp/80 + tcp/443. List a sponsord host -# again here; it must be in sponsord_hosts too. Its knobs are in -# group_vars/sponsord_onramp_hosts.yml; the Turnstile secret is provisioned on the -# host (roles/sponsord_onramp/README.md). -# sponsord_onramp_hosts: -# hosts: -# sponsord-1: -# -# OPTIONAL: self-hosted iroh relays, the fallback path for peers that cannot -# hole-punch — provisioned by playbooks/iroh_relay.yml (`make deploy-relay`) and by -# site.yml. Public by design: each terminates its own Let's Encrypt TLS on tcp/80 + -# tcp/443 and serves QUIC address discovery on udp/7842, so it cannot share a host -# with sponsord-onramp. Deploy at least two before pointing nodes' decdn_relay_urls -# at them. Its knobs are in group_vars/iroh_relay_hosts.yml (roles/iroh_relay/README.md). -# iroh_relay_hosts: -# hosts: -# relay-1: -# ansible_host: REPLACE_WITH_RELAY_VPS_IP_OR_DNS -# iroh_relay_hostname: relay1.example.org -# -# OPTIONAL: a self-hosted iroh DNS server, the pkarr relay nodes publish their -# address records to and the DNS server peers resolve them through (in place of -# n0's dns.iroh.link) — provisioned by playbooks/iroh_dns_server.yml (`make -# deploy-dns`) and by site.yml. Public by design: it terminates its own Let's -# Encrypt TLS on tcp/443 and answers DNS on udp+tcp/53, so it cannot share a host -# with an iroh relay or sponsord-onramp. The parent zone must delegate its hostname -# to it (NS + glue). Its knobs are in group_vars/iroh_dns_server_hosts.yml -# (roles/iroh_dns_server/README.md). -# iroh_dns_server_hosts: -# hosts: -# dns-1: -# ansible_host: REPLACE_WITH_DNS_VPS_IP_OR_DNS -# iroh_dns_server_hostname: dns.example.org diff --git a/ansible/inventory/hosts-publisher.yml.example b/ansible/inventory/hosts-publisher.yml.example new file mode 100644 index 0000000..6b9a20a --- /dev/null +++ b/ansible/inventory/hosts-publisher.yml.example @@ -0,0 +1,84 @@ +--- +# PUBLISHERS: copy to inventory/hosts.yml and fill in your hosts (a node operator +# starts from hosts-node.yml.example instead). inventory/hosts.yml is git-ignored: +# this repo is public, so real host IPs never belong in it. To track a real fleet, +# keep its inventory in a private overlay and pass INVENTORY= (see README.md +# "Private fleet inventory"). +# +# A publisher runs its origin nodes and, optionally, the services around them. Each +# group below has its own playbook; `make deploy-publisher` (playbooks/publisher.yml) +# runs them all, and `make deploy` (site.yml) also runs any cache nodes listed here. +# +# Bootstrapping a fresh host, ansible_user and the Ubuntu sudo-rs workaround work as +# in hosts-node.yml.example: read its header before the first run. In short, the first +# run per host is `make deploy-publisher LIMIT= ANSIBLE_ARGS='-u root'` (your +# image's bootstrap user in place of root); after it, plain `make deploy-publisher`. + +# Origin nodes — provisioned by playbooks/origin.yml (`make deploy-origin`) and by +# publisher.yml. An origin is a deCDN node with an origin backend: the canonical +# source of its namespace's content. decdn_origin_nodes is a child of decdn_nodes, so +# an origin gets a node's firewall hole, backup, decommission and observability; +# origin.yml refuses one outside decdn_nodes or without a backend +# (group_vars/decdn_origin_nodes.yml). The chain recognises it as the origin only once +# you, the namespace's publisher, seat its operator with OriginAssignment.addOrigin; +# until then it serves its bytes as a cache does. Per-origin config (the backend, +# region, …) goes in host_vars//main.yml (see host_vars/origin-1/). +decdn_nodes: + children: + decdn_origin_nodes: + hosts: + origin-1: + ansible_host: REPLACE_WITH_ORIGIN_VPS_IP_OR_DNS + # More origins: one entry each, with its own host_vars//main.yml + # (decdn_region and the origin backend are REQUIRED per host). + # origin-2: + # ansible_host: REPLACE_WITH_ORIGIN_VPS_IP_OR_DNS + +# OPTIONAL: sponsord, the onboarding sponsor (treasury signer + PaymentPool keeper), +# which pays for your users' downloads — provisioned by playbooks/sponsord.yml (`make +# deploy-sponsord`) and by publisher.yml. It needs no node: give it its own host, or +# list an origin host here as well to co-locate them. Its knobs are in +# group_vars/sponsord_hosts.yml; the treasury keystore and password are provisioned on +# the host (roles/sponsord/README.md). +# sponsord_hosts: +# hosts: +# sponsord-1: +# ansible_host: REPLACE_WITH_SPONSORD_VPS_IP_OR_DNS +# +# OPTIONAL: sponsord-onramp, sponsord's public side (Turnstile gate, installers, the +# decdn-sponsored CLI API), behind Caddy on tcp/80 + tcp/443. List a sponsord host +# again here; it must be in sponsord_hosts too. Its knobs are in +# group_vars/sponsord_onramp_hosts.yml; the Turnstile secret is provisioned on the +# host (roles/sponsord_onramp/README.md). +# sponsord_onramp_hosts: +# hosts: +# sponsord-1: +# +# OPTIONAL: self-hosted iroh relays, the fallback path for peers that cannot +# hole-punch — provisioned by playbooks/iroh_relay.yml (`make deploy-relay`) and by +# publisher.yml. Public by design: each terminates its own Let's Encrypt TLS on +# tcp/80 + tcp/443 and serves QUIC address discovery on udp/7842, so it cannot share a +# host with sponsord-onramp. Deploy at least two before pointing nodes' +# decdn_relay_urls at them; by default a relay admits only this inventory's nodes +# (iroh_relay_access: allowlist), so list other operators' node IDs in +# iroh_relay_allowlist or use everyone. Its knobs are in +# group_vars/iroh_relay_hosts.yml (roles/iroh_relay/README.md). +# iroh_relay_hosts: +# hosts: +# relay-1: +# ansible_host: REPLACE_WITH_RELAY_VPS_IP_OR_DNS +# iroh_relay_hostname: relay1.example.org +# +# OPTIONAL: a self-hosted iroh DNS server, the pkarr relay nodes publish their +# address records to and the DNS server peers resolve them through (in place of +# n0's dns.iroh.link) — provisioned by playbooks/iroh_dns_server.yml (`make +# deploy-dns`) and by publisher.yml. Public by design: it terminates its own Let's +# Encrypt TLS on tcp/443 and answers DNS on udp+tcp/53, so it cannot share a host +# with an iroh relay or sponsord-onramp. The parent zone must delegate its hostname +# to it (NS + glue). Its knobs are in group_vars/iroh_dns_server_hosts.yml +# (roles/iroh_dns_server/README.md). +# iroh_dns_server_hosts: +# hosts: +# dns-1: +# ansible_host: REPLACE_WITH_DNS_VPS_IP_OR_DNS +# iroh_dns_server_hostname: dns.example.org diff --git a/ansible/playbooks/group_vars/all.yml b/ansible/playbooks/group_vars/all.yml index d04418e..e8eac82 100644 --- a/ansible/playbooks/group_vars/all.yml +++ b/ansible/playbooks/group_vars/all.yml @@ -21,12 +21,13 @@ grafana_alloy_iroh_relay_enabled: "{{ 'iroh_relay_hosts' in group_names }}" grafana_alloy_iroh_dns_server_enabled: "{{ 'iroh_dns_server_hosts' in group_names }}" # The public holes in the baseline firewall, from the groups a host is in. Each -# group that runs baseline (decdn_nodes.yml, sponsord_hosts.yml, iroh_relay_hosts.yml, -# iroh_dns_server_hosts.yml) sets baseline_extra_inbound to this, so every play on a -# host renders the same rules. +# group that runs baseline (decdn_nodes.yml, decdn_origin_nodes.yml, sponsord_hosts.yml, +# iroh_relay_hosts.yml, iroh_dns_server_hosts.yml) sets baseline_extra_inbound to this, +# so every play on a host renders the same rules. # Only genuinely public listeners belong here: metrics, admin RPC and the sponsord # API stay on loopback and get no hole. -# decdn_nodes: the node's QUIC listener. decdn_bind_port is a decdn_node role +# decdn_nodes: the node's QUIC listener, an origin's included (decdn_origin_nodes +# are decdn_nodes too; origin.yml refuses one that is not). decdn_bind_port is a decdn_node role # default, which a play without that role (sponsord.yml) does not load; the # fallback must match roles/decdn_node/defaults/main.yml. # sponsord_onramp_hosts: Caddy, the onramp's TLS front (80 for the ACME challenge diff --git a/ansible/playbooks/group_vars/decdn_origin_nodes.yml b/ansible/playbooks/group_vars/decdn_origin_nodes.yml new file mode 100644 index 0000000..6154d20 --- /dev/null +++ b/ansible/playbooks/group_vars/decdn_origin_nodes.yml @@ -0,0 +1,7 @@ +--- +# Playbook-adjacent group_vars for decdn_origin_nodes (see decdn_nodes.yml beside +# this for why playbook-adjacent). An origin is also in decdn_nodes, as a child group +# or by name, and inherits that file's value; it is set here as well because every +# group whose play runs baseline sets it, so no group ordering can leave an origin's +# play without the node's QUIC hole (all.yml builds the list from the host's groups). +baseline_extra_inbound: "{{ _baseline_service_inbound }}" diff --git a/ansible/playbooks/node.yml b/ansible/playbooks/node.yml new file mode 100644 index 0000000..59da030 --- /dev/null +++ b/ansible/playbooks/node.yml @@ -0,0 +1,39 @@ +--- +# NODE OPERATORS: hardened, public deCDN cache nodes. +# decdn_nodes (minus decdn_origin_nodes): baseline -> grafana_alloy (opt-in, off by +# default) -> decdn-node +# Standalone entry point (`make deploy-node`), also imported by site.yml. The node +# listens on public QUIC udp/4433; metrics + admin RPC stay loopback. On-chain stake + +# registration (ADR 019 Phase 2) is an operator action, not automated here — see +# roles/decdn_node/README.md. +# A publisher's origin nodes are decdn_nodes too, but they are listed in +# decdn_origin_nodes and deployed by origin.yml (publisher.yml), so this play leaves +# them alone and site.yml runs each node once. +- name: Provision deCDN cache nodes + hosts: decdn_nodes:!decdn_origin_nodes + become: true + pre_tasks: + # A node with an origin backend serves as its namespace's canonical source and + # declines whatever its backend does not hold (decdn_relay_foreign_namespaces). It + # still works from here, so this warns rather than fails. + - name: Warn about an origin backend on a host outside decdn_origin_nodes + ansible.builtin.debug: + msg: >- + WARNING: {{ inventory_hostname }} has an origin backend + (decdn_cache_origin_kind / decdn_cache_origins) but is not in + decdn_origin_nodes. It serves as an origin all the same; list it under + decdn_origin_nodes so publisher.yml deploys it and its placement is checked. + when: >- + (decdn_cache_origin_kind | default('') | length > 0) + or (decdn_cache_origins | default([]) | length > 0) + tags: [always] + roles: + - role: baseline + tags: [baseline] + # Opt-in Grafana Cloud observability (off by default; one mirrored inventory + # flag drives this + the otlp_endpoint injection inside decdn_node). Runs + # BEFORE decdn_node so Alloy's receivers exist before the daemon exports. + - role: grafana_alloy + tags: [observability, decdn_node, node] + - role: decdn_node + tags: [decdn_node, node] diff --git a/ansible/playbooks/origin.yml b/ansible/playbooks/origin.yml new file mode 100644 index 0000000..4f92a4c --- /dev/null +++ b/ansible/playbooks/origin.yml @@ -0,0 +1,48 @@ +--- +# PUBLISHERS: origin nodes, the decdn-node daemon with an origin backend (S3/R2/B2/ +# MinIO, NFS or local disk, or an HTTP upstream) as its namespace's canonical source. +# decdn_origin_nodes: baseline -> grafana_alloy (opt-in, off by default) -> decdn-node +# Standalone entry point (`make deploy-origin`), also imported by publisher.yml. The +# same roles as node.yml: an origin is a node, configured with +# decdn_cache_origin_kind (+ its fields) or decdn_cache_origins. Being recognised as +# the origin is on-chain: the namespace's publisher seats the operator with +# OriginAssignment.addOrigin (decdn/adr/011-content-takedown.md, Origin Assignment +# Authority); without that, its bytes are served as cache. Stake + registration are +# the same operator steps as any node's — see roles/decdn_node/README.md. +- name: Provision deCDN origin nodes + hosts: decdn_origin_nodes + become: true + pre_tasks: + # The firewall hole, backup, decommission, the relay allowlist and Alloy's node + # scrape all follow decdn_nodes, so an origin outside it would lose them. The role + # checks no group name, so the decdn.node collection does not impose this repo's + # groups on others. tests/scripts-test.sh runs these tasks by the play's name. + - name: Require the origin in decdn_nodes too + ansible.builtin.assert: + that: + - "'decdn_nodes' in group_names" + fail_msg: >- + {{ inventory_hostname }} is in decdn_origin_nodes but not in decdn_nodes. An + origin is a deCDN node: list it under decdn_nodes too (as a child group, or + by name), which gives it the node's firewall hole, backup, decommission and + observability. + tags: [always] + - name: Require an origin backend + ansible.builtin.assert: + that: + - >- + (decdn_cache_origin_kind | default('') | length > 0) + or (decdn_cache_origins | default([]) | length > 0) + fail_msg: >- + {{ inventory_hostname }} is in decdn_origin_nodes but has no origin backend: + set decdn_cache_origin_kind (http, fs or s3) and its fields, or + decdn_cache_origins (see roles/decdn_node/README.md), or move the host out + of decdn_origin_nodes to run it as a cache node. + tags: [always] + roles: + - role: baseline + tags: [baseline] + - role: grafana_alloy + tags: [observability, decdn_node, node] + - role: decdn_node + tags: [decdn_node, node] diff --git a/ansible/playbooks/publisher.yml b/ansible/playbooks/publisher.yml new file mode 100644 index 0000000..5842aac --- /dev/null +++ b/ansible/playbooks/publisher.yml @@ -0,0 +1,21 @@ +--- +# PUBLISHERS: everything a publisher runs, in order. +# origin.yml decdn_origin_nodes +# sponsord.yml sponsord_hosts (+ sponsord_onramp_hosts) +# iroh_relay.yml iroh_relay_hosts (reads the nodes' IDs, origins included) +# iroh_dns_server.yml iroh_dns_server_hosts +# Entry point `make deploy-publisher`, also imported by site.yml after node.yml. Each +# component has its own entry point too (deploy-origin, deploy-sponsord, +# deploy-relay, deploy-dns), and each play matches nothing unless the inventory +# defines its group. +- name: Provision origin nodes + ansible.builtin.import_playbook: origin.yml + +- name: Provision sponsord + ansible.builtin.import_playbook: sponsord.yml + +- name: Provision iroh relays + ansible.builtin.import_playbook: iroh_relay.yml + +- name: Provision iroh DNS servers + ansible.builtin.import_playbook: iroh_dns_server.yml diff --git a/ansible/playbooks/site.yml b/ansible/playbooks/site.yml index 6c0a96d..0e111df 100644 --- a/ansible/playbooks/site.yml +++ b/ansible/playbooks/site.yml @@ -1,32 +1,11 @@ --- -# PRIMARY deployment: a hardened, public deCDN node. -# baseline -> grafana_alloy (opt-in, off by default) -> decdn-node -# The node listens on public QUIC udp/4433; metrics + admin RPC stay loopback. -# On-chain stake + registration (ADR 019 Phase 2) is an operator action, not -# automated here — see roles/decdn_node/README.md. -- name: Provision a hardened deCDN node - hosts: decdn_nodes - become: true - roles: - - role: baseline - tags: [baseline] - # Opt-in Grafana Cloud observability (off by default; one mirrored inventory - # flag drives this + the otlp_endpoint injection inside decdn_node). Runs - # BEFORE decdn_node so Alloy's receivers exist before the daemon exports. - - role: grafana_alloy - tags: [observability, decdn_node, node] - - role: decdn_node - tags: [decdn_node, node] +# Everything in the inventory: what node operators run, then what publishers run. +# node.yml decdn_nodes that are not origins (node operators) +# publisher.yml origins, sponsord (+ onramp), iroh relays, iroh DNS servers +# A host is in as many of those groups as it runs services; each play matches nothing +# unless the inventory defines its group. +- name: Provision deCDN cache nodes + ansible.builtin.import_playbook: node.yml -# sponsord on hosts in sponsord_hosts (none unless the inventory defines the group). -- name: Provision sponsord - ansible.builtin.import_playbook: sponsord.yml - -# iroh relays on hosts in iroh_relay_hosts (none unless the inventory defines the group). -- name: Provision iroh relays - ansible.builtin.import_playbook: iroh_relay.yml - -# iroh DNS servers on hosts in iroh_dns_server_hosts (none unless the inventory -# defines the group). -- name: Provision iroh DNS servers - ansible.builtin.import_playbook: iroh_dns_server.yml +- name: Provision the publisher's services + ansible.builtin.import_playbook: publisher.yml diff --git a/ansible/roles/baseline/README.md b/ansible/roles/baseline/README.md index efc1f32..428bf23 100644 --- a/ansible/roles/baseline/README.md +++ b/ansible/roles/baseline/README.md @@ -90,4 +90,4 @@ nftables, DevSec `os_hardening`/`ssh_hardening` and fail2ban need a real machine. It is verified on real hosts (the deCDN team's own fleet runs Ubuntu 26.04) and with `make check` as a dry run. On Ubuntu 25.10+ note the sudo-rs become workaround in -`inventory/hosts.yml.example`. +`inventory/hosts-node.yml.example`. diff --git a/ansible/roles/decdn_node/README.md b/ansible/roles/decdn_node/README.md index 63e08d8..13fcfeb 100644 --- a/ansible/roles/decdn_node/README.md +++ b/ansible/roles/decdn_node/README.md @@ -246,9 +246,12 @@ Optional (omitted from `node.toml` unless set): file, which also means nothing downstream catches a malformed value — hence the role's own shape assert. (The former `decdn_swap_*` knobs for the on-chain swap path were removed upstream.) -- `decdn_cache_origin_kind` (`http`|`fs`|`s3`) + that kind's fields — the - pull-through origin the node fetches on a cache miss. **A serving node needs - one:** unset ⇒ no `[cache.origin]` and cache misses fail `NoOrigin`. For an +- `decdn_cache_origin_kind` (`http`|`fs`|`s3`) + that kind's fields — the origin + backend a publisher's **origin node** fills a cache miss from (in this repo's + playbooks, a `decdn_origin_nodes` host; `playbooks/origin.yml` requires one). + Unset ⇒ no `[cache.origin]`: a **cache node**, which fills a miss from other nodes + (`decdn_node_to_node_pull_through_enabled`, on by default without an origin) and + fails it `NoOrigin` when none of them has the blob. For an ordered fallback chain use `decdn_cache_origins` (a list of the same per-kind dicts) instead — the two forms are mutually exclusive upstream and the role fails loud if both are set. diff --git a/ansible/roles/decdn_node/defaults/main.yml b/ansible/roles/decdn_node/defaults/main.yml index 8cd7216..34af662 100644 --- a/ansible/roles/decdn_node/defaults/main.yml +++ b/ansible/roles/decdn_node/defaults/main.yml @@ -299,13 +299,15 @@ decdn_node_pull_timeout_sec: "" # daemon dflt 20 — per-upstre # A leg is abandoned when it delivers under min_throughput_bps across the window. decdn_node_pull_stall_window_sec: "" # daemon dflt 20 decdn_node_pull_min_throughput_bps: "" # daemon dflt 4096 -# Origin-only node policy (#1759): whether to relay namespaces this node is not an -# authorized origin for. Daemon default is ROLE-DERIVED — false when an origin -# backend is configured, true when none — so leave "" unless you mean to pin it. +# Origin-only node policy (#1759): whether to relay a miss this node's own backend +# cannot produce (the gate is the backend's content, not the request's namespace; +# ADR 002 § Retrieval by namespace). Daemon default is ROLE-DERIVED — false when an +# origin backend is configured, true when none — so leave "" unless you mean to pin it. decdn_relay_foreign_namespaces: "" # bool -# Pull-through origin (#437): what the node fetches on a cache miss. Empty kind => -# the [cache.origin] table is omitted and misses fail NoOrigin — a serving node -# needs an origin. Pick ONE kind and set that kind's fields; the others are ignored. +# Origin backend (#437): what an origin node fetches on a cache miss. Empty kind => +# the [cache.origin] table is omitted: a cache node, which fills a miss from other +# nodes (node-to-node pull-through above) and fails it NoOrigin when none has it. +# Pick ONE kind and set that kind's fields; the others are ignored. decdn_cache_origin_kind: "" # "" (omit) | http | fs | s3 decdn_cache_origin_url: "" # http: base URL (blobs served at {url}/{blake3_hex}) decdn_cache_origin_decompress: "" # http, optional: "auto" (default) | "strict" diff --git a/ansible/roles/decdn_node/tasks/main.yml b/ansible/roles/decdn_node/tasks/main.yml index fe3fa0c..84b9242 100644 --- a/ansible/roles/decdn_node/tasks/main.yml +++ b/ansible/roles/decdn_node/tasks/main.yml @@ -628,7 +628,8 @@ s3: decdn_cache_origin_s3_bucket + decdn_cache_origin_s3_region. decdn_cache_origin_decompress, if set, must be auto|strict, and decdn_cache_origin_s3_path_style must be a boolean (true/false). Leave the - kind empty to omit [cache.origin] (cache misses then fail NoOrigin). + kind empty to omit [cache.origin] (a cache node, which fills misses from other + nodes). when: decdn_cache_origin_kind | length > 0 # The inverse gate. Every decdn_cache_origin_* default is "", and the validator diff --git a/ansible/tests/firewall-holes/check.sh b/ansible/tests/firewall-holes/check.sh index c1043d7..406348d 100755 --- a/ansible/tests/firewall-holes/check.sh +++ b/ansible/tests/firewall-holes/check.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # The baseline firewall's public holes are derived from a host's groups in -# playbooks/group_vars/ (all.yml builds the list; decdn_nodes.yml, +# playbooks/group_vars/ (all.yml builds the list; decdn_nodes.yml, decdn_origin_nodes.yml, # sponsord_hosts.yml, iroh_relay_hosts.yml and iroh_dns_server_hosts.yml all point # baseline_extra_inbound at it). Resolve them for # every host shape in inventory.yml, outside any role, which is how the sponsord diff --git a/ansible/tests/firewall-holes/expected.json b/ansible/tests/firewall-holes/expected.json index 8262b78..5bbd55e 100644 --- a/ansible/tests/firewall-holes/expected.json +++ b/ansible/tests/firewall-holes/expected.json @@ -48,5 +48,12 @@ {"proto": "tcp", "port": 443, "comment": "iroh-dns-server (https, pkarr, ACME)"}, {"proto": "udp", "port": 53, "comment": "iroh-dns-server (dns)"}, {"proto": "tcp", "port": 53, "comment": "iroh-dns-server (dns over tcp)"} + ], + "origin": [{"proto": "udp", "port": 4433, "comment": "decdn QUIC"}], + "origin-listed": [{"proto": "udp", "port": 4433, "comment": "decdn QUIC"}], + "origin-onramp": [ + {"proto": "udp", "port": 4433, "comment": "decdn QUIC"}, + {"proto": "tcp", "port": 80, "comment": "sponsord-onramp (Caddy, http)"}, + {"proto": "tcp", "port": 443, "comment": "sponsord-onramp (Caddy, https)"} ] } diff --git a/ansible/tests/firewall-holes/inventory.yml b/ansible/tests/firewall-holes/inventory.yml index 9422a7b..d494e3e 100644 --- a/ansible/tests/firewall-holes/inventory.yml +++ b/ansible/tests/firewall-holes/inventory.yml @@ -14,9 +14,18 @@ all: co-located-onramp: {} relay-node: {} dns-node: {} + origin-listed: {} + # A publisher's origins: a child group, so its hosts are decdn_nodes too. + children: + decdn_origin_nodes: + hosts: + origin: {} + origin-listed: {} + origin-onramp: {} sponsord_hosts: hosts: sponsord: {} + origin-onramp: {} co-located: {} onramp: {} onramp-own-proxy: {} @@ -29,6 +38,7 @@ all: onramp-own-proxy: sponsord_onramp_proxy: none co-located-onramp: {} + origin-onramp: {} iroh_relay_hosts: hosts: relay: {} diff --git a/ansible/tests/playbook-guards/inventory.yml b/ansible/tests/playbook-guards/inventory.yml index 9087062..0c13e09 100644 --- a/ansible/tests/playbook-guards/inventory.yml +++ b/ansible/tests/playbook-guards/inventory.yml @@ -9,6 +9,21 @@ all: hosts: node-only: {} dns-node: {} + node-with-origin: + decdn_cache_origin_kind: http + origin-ok: {} + origin-list: {} + origin-none: {} + # A top-level group here, not decdn_nodes' child, so a host can be in it alone. + decdn_origin_nodes: + hosts: + origin-ok: + decdn_cache_origin_kind: http + origin-list: + decdn_cache_origins: [{kind: fs, path: /srv/content}] + origin-none: {} + origin-orphan: + decdn_cache_origin_kind: http sponsord_hosts: hosts: both: {} diff --git a/cloud-init/baseline-plays.awk b/cloud-init/baseline-plays.awk index ed906ff..5be9736 100644 --- a/cloud-init/baseline-plays.awk +++ b/cloud-init/baseline-plays.awk @@ -19,10 +19,18 @@ # Exit 0 when each group has such a play and every one of them selects a baseline task; # 1 otherwise, with one line per group on stderr; 2 when no groups are given. -function end_play() { +# A play counts for each group its host pattern selects: the positive terms of a +# pattern like `decdn_nodes:!decdn_origin_nodes` (decdn_nodes), not its `!` exclusions +# or `&` intersections. +function end_play( n_terms, terms, i, t) { if (pattern != "" && has_localhost) { - plays[pattern]++ - if (baseline == 0) bare[pattern]++ + n_terms = split(pattern, terms, /[:,]/) + for (i = 1; i <= n_terms; i++) { + t = terms[i] + if (t == "" || t ~ /^[!&]/) continue + plays[t]++ + if (baseline == 0) bare[t]++ + } } pattern = ""; has_localhost = 0; baseline = 0; section = "" } diff --git a/docs/requirements.md b/docs/requirements.md index 5d688bf..a5663ba 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -35,7 +35,7 @@ and they run the same roles. | Docker Compose | v2 with `env_file.required` and `depends_on.restart` support (2.24+) | rendered in CI with every profile on | On Ubuntu 25.10 and later, `sudo` is sudo-rs; see the `ansible_become_exe` note in -`ansible/inventory/hosts.yml.example`. +`ansible/inventory/hosts-node.yml.example`. ## Network diff --git a/tests/scripts-test.sh b/tests/scripts-test.sh index 8a3efb2..db83b9d 100755 --- a/tests/scripts-test.sh +++ b/tests/scripts-test.sh @@ -45,6 +45,16 @@ expect 2 "deploy-dns refuses LIMIT from the environment" env LIMIT=h make -s mk deploy-dns LIMIT=h | grep -q -- "playbooks/iroh_dns_server.yml --limit 'h'" \ || fail "deploy-dns LIMIT=h does not run playbooks/iroh_dns_server.yml scoped to h" pass "deploy-dns LIMIT=h scopes playbooks/iroh_dns_server.yml" +# The persona entry points: each goal is guarded and runs its own playbook. +for t in check-node:node deploy-node:node check-origin:origin deploy-origin:origin \ + check-publisher:publisher deploy-publisher:publisher; do + expect 2 "${t%%:*} refuses LIMIT from the environment" env LIMIT=h make -s -C "$repo/ansible" -n "${t%%:*}" + mk "${t%%:*}" LIMIT=h | grep -q -- "playbooks/${t##*:}.yml.* --limit 'h'" \ + || fail "${t%%:*} LIMIT=h does not run playbooks/${t##*:}.yml scoped to h" +done +mk check-publisher | grep -q -- "playbooks/publisher.yml --check --diff" || fail "check-publisher is not a --check run" +mk deploy | grep -q -- "playbooks/site.yml" || fail "deploy does not run playbooks/site.yml" +pass "check/deploy-node, -origin and -publisher scope their own playbooks; deploy runs site.yml" # --- molecule driver (scripts/molecule.sh): selection, guards, locks ------------------ # Against a throwaway ansible/ layout, so nothing here can start a container or touch @@ -1202,6 +1212,18 @@ expect 1 "baseline guard refuses a group whose play does not list localhost" \ grep -q 'no play targets decdn_nodes with localhost in it' "$work/out" \ || { cat "$work/out" >&2; fail "the baseline guard refused, but not for decdn_nodes"; } expect 2 "baseline guard refuses an empty group list" guard_plays "" "$hardened" +# A play's pattern counts for its positive terms only: node.yml's +# decdn_nodes:!decdn_origin_nodes play is a decdn_nodes play, never an origin one. +excl_out() { # + printf " play #1 (decdn_nodes:!decdn_origin_nodes): Provision deCDN cache nodes\tTAGS: []\n pattern: ['decdn_nodes:!decdn_origin_nodes']\n hosts (1):\n localhost\n tasks:\n%b\n" "$1" +} +excl_plays() { excl_out "$2" | awk -v groups="$1" -f "$repo/cloud-init/baseline-plays.awk"; } +expect 0 "baseline guard counts a decdn_nodes:!decdn_origin_nodes play for decdn_nodes" excl_plays decdn_nodes "$hardened" +expect 1 "baseline guard does not count an excluded group's play" excl_plays decdn_origin_nodes "$hardened" +grep -q 'no play targets decdn_origin_nodes with localhost in it' "$work/out" \ + || { cat "$work/out" >&2; fail "the baseline guard counted the exclusion as a decdn_origin_nodes play"; } +expect 1 "baseline guard refuses a bare decdn_nodes:!decdn_origin_nodes play" \ + excl_plays decdn_nodes ' Apply DevSec OS hardening\tTAGS: [baseline]' # bootstrap.sh's gate and lint.py agree on the groups a user-data may use, and the # bootstrap's hint has its own instructions for each. boot="$repo/cloud-init/bootstrap.sh" @@ -1218,17 +1240,19 @@ pass "bootstrap.sh's groups match lint.py's, each with its secrets and its hint" # run the guard too, but in Docker and only for the groups their inventories use. # `hosts` and `tags` may each be a string or a list; tags match exactly. if command -v yq >/dev/null; then - for g in decdn_nodes sponsord_hosts; do + for g in decdn_nodes decdn_origin_nodes sponsord_hosts; do n=0 - for pb in site.yml sponsord.yml; do - plays="[.[] | select([.hosts] | flatten | any_c(. == \"$g\"))]" + for pb in node.yml origin.yml sponsord.yml; do + # A play's positive pattern terms: decdn_nodes:!decdn_origin_nodes targets + # decdn_nodes only. + plays="[.[] | select([.hosts] | flatten | map(split(\":\")) | flatten | map(split(\",\")) | flatten | any_c(. == \"$g\"))]" n=$((n + $(yq "$plays | length" "$repo/ansible/playbooks/$pb"))) [[ "$(yq "$plays | map(select((.roles // []) | map(select(.role == \"baseline\" and ([.tags // []] | flatten | any_c(. == \"baseline\")))) | length == 0)) | length" "$repo/ansible/playbooks/$pb")" == 0 ]] \ || fail "playbooks/$pb: a $g play lacks the baseline role tagged baseline (cloud-init's phase 1 relies on it)" done - ((n >= 1)) || fail "no play in site.yml or sponsord.yml targets $g" + ((n >= 1)) || fail "no play in node.yml, origin.yml or sponsord.yml targets $g" done - pass "every decdn_nodes and sponsord_hosts play runs baseline under the baseline tag" + pass "every decdn_nodes, decdn_origin_nodes and sponsord_hosts play runs baseline under the baseline tag" # baseline-plays.sh, the command bootstrap.sh runs, against the real playbooks and the # templates' inventories, then against a copy whose sponsord play lost the tag (#90). # Listing tasks needs ansible-core only, not the collections. @@ -1443,6 +1467,34 @@ if command -v ansible >/dev/null; then expect 2 "sponsord.yml refuses an onramp host outside sponsord_hosts" guard onramp-only grep -q "is in sponsord_onramp_hosts but not in sponsord_hosts" "$work/out" \ || { cat "$work/out" >&2; fail "the onramp-only refusal did not come from the placement check"; } + # playbooks/origin.yml refuses an origin outside decdn_nodes or without an origin + # backend; node.yml warns about an origin backend outside decdn_origin_nodes. The + # plays' pre_tasks are lifted out as tasks, without the roles or fact gathering. + lift() { # + yq "[.[] | select(.name == \"$2\") | {\"name\": .name, \"hosts\": .hosts, \"gather_facts\": false, \"tasks\": .pre_tasks}]" \ + "$repo/ansible/playbooks/$1" > "$work/guard.yml" + [[ "$(yq 'length' "$work/guard.yml")" == 1 && "$(yq '.[0].tasks | length' "$work/guard.yml")" -ge 1 ]] \ + || fail "playbooks/$1 has no single '$2' play with pre_tasks" + } + lift origin.yml "Provision deCDN origin nodes" + expect 0 "origin.yml accepts an origin in decdn_nodes with decdn_cache_origin_kind" guard origin-ok + expect 0 "origin.yml accepts an origin in decdn_nodes with decdn_cache_origins" guard origin-list + expect 2 "origin.yml refuses an origin outside decdn_nodes" guard origin-orphan + grep -q "is in decdn_origin_nodes but not in decdn_nodes" "$work/out" \ + || { cat "$work/out" >&2; fail "the origin-orphan refusal did not come from the decdn_nodes check"; } + expect 2 "origin.yml refuses an origin without an origin backend" guard origin-none + grep -q "is in decdn_origin_nodes but has no origin backend" "$work/out" \ + || { cat "$work/out" >&2; fail "the origin-none refusal did not come from the backend check"; } + lift node.yml "Provision deCDN cache nodes" + expect 0 "node.yml runs a node with an origin backend outside decdn_origin_nodes" guard node-with-origin + grep -q "WARNING: node-with-origin has an origin backend" "$work/out" \ + || { cat "$work/out" >&2; fail "node.yml does not warn about an origin backend outside decdn_origin_nodes"; } + expect 0 "node.yml runs a cache node" guard node-only + ! grep -q "WARNING:" "$work/out" || { cat "$work/out" >&2; fail "node.yml warns about a cache node"; } + guard origin-ok >"$work/out" 2>&1 || true + grep -q "skipping: no hosts matched" "$work/out" \ + || { cat "$work/out" >&2; fail "node.yml's play matches an origin (decdn_origin_nodes) host"; } + pass "node.yml's play leaves decdn_origin_nodes to origin.yml" # decommission.yml's host cap covers the whole run: a node-only host and a # sponsord host pass each role's own per-play cap but not this one. Only the # cap task is lifted (the hold check needs the roles).