From 57568cbf3756be65edc3852462f3a2d961af4c35 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Sat, 10 Oct 2026 19:03:22 +0300 Subject: [PATCH 1/3] feat(ansible): decdn_origin_nodes and the node.yml / publisher.yml entry points Node operators run cache nodes; publishers run origin nodes and the services around them. The playbooks now have an entry point per persona: node.yml (`make deploy-node`, decdn_nodes:!decdn_origin_nodes) and publisher.yml (`make deploy-publisher`: origin.yml, sponsord.yml, iroh_relay.yml, iroh_dns_server.yml). site.yml imports both, and `make deploy` still deploys everything. Origins are decdn_origin_nodes, a child of decdn_nodes, so they keep the node's firewall hole, backup, decommission, relay allowlist and Alloy scrape. origin.yml refuses an origin outside decdn_nodes or without an origin backend; node.yml warns about a backend outside the group. The group checks live in the playbooks, not the role. baseline-plays.awk counts a play for the positive terms of its host pattern only. The inventory example splits into hosts-node.yml.example (a cache node, no origin backend) and hosts-publisher.yml.example (origin-1, plus the sponsord, relay and DNS blocks). The decdn_node docs no longer say a serving node needs an origin: without one it fills misses from other nodes (ADR 001's pure caches). Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 4 +- AGENTS.md | 39 ++++++--- README.md | 6 +- ansible/.gitignore | 7 +- ansible/Makefile | 55 ++++++++---- ansible/README.md | 34 +++++--- ansible/galaxy/node/CHANGELOG.md | 4 + ansible/inventory/group_vars/decdn_nodes.yml | 4 +- .../group_vars/decdn_origin_nodes.yml | 24 ++++++ .../group_vars/iroh_dns_server_hosts.yml | 2 +- .../inventory/group_vars/iroh_relay_hosts.yml | 2 +- .../inventory/host_vars/decdn-node-1/main.yml | 11 ++- ansible/inventory/host_vars/origin-1/main.yml | 73 ++++++++++++++++ .../host_vars/origin-1/secret.yml.example | 56 +++++++++++++ ...sts.yml.example => hosts-node.yml.example} | 59 ++----------- ansible/inventory/hosts-publisher.yml.example | 84 +++++++++++++++++++ ansible/playbooks/group_vars/all.yml | 9 +- .../group_vars/decdn_origin_nodes.yml | 7 ++ ansible/playbooks/node.yml | 39 +++++++++ ansible/playbooks/origin.yml | 48 +++++++++++ ansible/playbooks/publisher.yml | 21 +++++ ansible/playbooks/site.yml | 39 ++------- ansible/roles/baseline/README.md | 2 +- ansible/roles/decdn_node/README.md | 9 +- ansible/roles/decdn_node/defaults/main.yml | 7 +- ansible/roles/decdn_node/tasks/main.yml | 3 +- ansible/tests/firewall-holes/check.sh | 2 +- ansible/tests/firewall-holes/expected.json | 7 ++ ansible/tests/firewall-holes/inventory.yml | 10 +++ ansible/tests/playbook-guards/inventory.yml | 15 ++++ cloud-init/baseline-plays.awk | 14 +++- docs/requirements.md | 2 +- tests/scripts-test.sh | 62 ++++++++++++-- 33 files changed, 602 insertions(+), 158 deletions(-) create mode 100644 ansible/inventory/group_vars/decdn_origin_nodes.yml create mode 100644 ansible/inventory/host_vars/origin-1/main.yml create mode 100644 ansible/inventory/host_vars/origin-1/secret.yml.example rename ansible/inventory/{hosts.yml.example => hosts-node.yml.example} (53%) create mode 100644 ansible/inventory/hosts-publisher.yml.example create mode 100644 ansible/playbooks/group_vars/decdn_origin_nodes.yml create mode 100644 ansible/playbooks/node.yml create mode 100644 ansible/playbooks/origin.yml create mode 100644 ansible/playbooks/publisher.yml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d2c37af..40acc32 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -127,8 +127,8 @@ jobs: - name: Lint (yamllint + ansible-lint) run: make lint - name: Syntax-check playbooks - # Dummy inventory: no real hosts.yml is committed upstream (only hosts.yml.example), - # and --syntax-check only parses, it never connects. Every playbook, so a new + # Dummy inventory: no real hosts.yml is committed upstream (only the + # hosts-*.yml.example files), and --syntax-check only parses, it never connects. Every playbook, so a new # one (backup, decommission, ...) is covered without editing this step. run: | for pb in playbooks/*.yml; do diff --git a/AGENTS.md b/AGENTS.md index d1e3060..d861d08 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,7 +72,7 @@ mirror (`vars/main/networks.yml`), never hand-copied into inventory or docs. ``` ansible/ # the deployment project (DevSec-hardened, lean roles) - playbooks/ # site.yml (node + sponsord + relay + DNS server), sponsord.yml (+ onramp), iroh_relay.yml, iroh_dns_server.yml, backup.yml, decommission.yml + playbooks/ # site.yml = node.yml (node operators: cache nodes) + publisher.yml (publishers: origin.yml, sponsord.yml (+ onramp), iroh_relay.yml, iroh_dns_server.yml); backup.yml, decommission.yml roles/ # baseline, decdn_node, grafana_alloy, sponsord, sponsord_onramp, iroh_relay, iroh_dns_server inventory/ galaxy/ molecule/ # see ansible/README.md cloud-init/ # user-data{,-sponsord}.yaml + on-host bootstrap.sh; pinned ansible-core/collections (see its README.md) @@ -141,8 +141,21 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). ## Current services -- **`ansible/`** — the declarative deployment project. **The public deCDN node** - (`playbooks/site.yml` → baseline + `decdn-node`), installed by `decdn_node_install_method`: +- **`ansible/`** — the declarative deployment project, with an entry point per persona: + `playbooks/node.yml` (`make deploy-node`) for **node operators**, `playbooks/publisher.yml` + (`make deploy-publisher`) for **publishers**, and `site.yml` (`make deploy`) for both. + **Origin nodes** are `decdn_origin_nodes`, a child of `decdn_nodes` (so they inherit the + node's firewall hole, backup, decommission, relay allowlist and Alloy scrape), deployed by + `playbooks/origin.yml` with the same roles; node.yml's play is + `decdn_nodes:!decdn_origin_nodes`, so each node runs once under site.yml. origin.yml's + `pre_tasks` refuse an origin outside `decdn_nodes` or without an origin backend + (`decdn_cache_origin_kind`/`decdn_cache_origins`); node.yml's warn about a backend outside + `decdn_origin_nodes`. Those group checks live in the playbooks, never the role, as with + sponsord's. A cache node has no origin backend: it fills misses from other nodes + (node-to-node pull-through), and an origin-configured node serves only its own + namespaces (ADR 037). Recognition as an origin is on-chain (`OriginAssignment.addOrigin`, + the publisher's step). **The public deCDN node** + (baseline + `decdn-node`), installed by `decdn_node_install_method`: `release` (the default) — a pinned GitHub release tarball verified against the release's GPG-signed `SHA256SUMS` (`decdn_node_version`, default `0.0.1`); `source` — a git ref (any tag/branch/SHA) cloned and `cargo build`-ed **on the node** as the unprivileged `decdn-build` user, sha256-pinned rustup, a `@` @@ -219,7 +232,7 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). - **`ansible/roles/sponsord`** — the deCDN onboarding sponsor (`decdn/sponsord`): the treasury wallet's signer for capped capabilities plus the PaymentPool keeper. It is independent of the node, so `playbooks/sponsord.yml` (`make deploy-sponsord`, also - imported by `site.yml`) targets its own `sponsord_hosts` group, standalone or + imported by `publisher.yml`) targets its own `sponsord_hosts` group, standalone or co-located with a node. - **Install:** a GPG-verified `v*` decdn/sponsord release by default (`sponsord_version`, `0.0.2`), a host-side `source` build, or a manual binary. @@ -290,7 +303,7 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). QUIC address discovery. The ADRs make relays operator-run infrastructure, not an incentivized role (`decdn/adr/architecture.md`, Trust Assumptions); nodes use them through `decdn_relay_urls`, which REPLACES n0's relays (so deploy at least two). - `playbooks/iroh_relay.yml` (`make deploy-relay`, also imported by `site.yml`) runs + `playbooks/iroh_relay.yml` (`make deploy-relay`, also imported by `publisher.yml`) runs baseline → grafana_alloy → iroh_relay on `iroh_relay_hosts`, after a play refusing a host also in `sponsord_onramp_hosts` (both want 80/443; `make test-scripts` runs it). - **Access:** `iroh_relay_access` defaults to `allowlist`. A play before the @@ -351,7 +364,7 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). `dns.iroh.link` (ADR 001, Node Discovery). Nodes use it through `decdn_discovery_pkarr_url` + `decdn_discovery_dns_origin`, which DROP the n0 leg, so clients need the same `dns_origin`. `playbooks/iroh_dns_server.yml` (`make - deploy-dns`, also imported by `site.yml`) runs baseline → grafana_alloy → + deploy-dns`, also imported by `publisher.yml`) runs baseline → grafana_alloy → iroh_dns_server on `iroh_dns_server_hosts`, after a play refusing a host also in `iroh_relay_hosts` or `sponsord_onramp_hosts` (all want tcp/443; `make test-scripts` runs it). One server per origin: upstream has no replication, and @@ -393,7 +406,7 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). - installs ansible-core from the hash-locked `requirements.txt` into a venv (two pins split by Python marker: 2.19 for Debian 12's 3.11, 2.21 for 3.12 and later); - installs the exact collections from `collections.lock.yml`; - - runs `site.yml` (which imports `sponsord.yml`) against localhost. The inventory + - runs `site.yml` (which imports `node.yml` and `publisher.yml`) against localhost. The inventory must put localhost in `decdn_nodes` and/or `sponsord_hosts` (and in `sponsord_hosts` whenever it is in `sponsord_onramp_hosts`), or the plays match nothing and the firewall holes never load. @@ -509,6 +522,9 @@ make security # KICS over ansible/, the rendered chart and compose/ # Ansible — run from ansible/ make deps # vendor pinned Galaxy collections make check / deploy # site.yml; fleet-wide unless LIMIT=; INVENTORY= +make check-node / deploy-node # node.yml only (node operators: cache nodes) +make check-publisher / deploy-publisher # publisher.yml only (origins, sponsord, relays, DNS) +make check-origin / deploy-origin # origin.yml only (decdn_origin_nodes) make check-sponsord / deploy-sponsord # sponsord.yml only (sponsord_hosts) make check-relay / deploy-relay # iroh_relay.yml only (iroh_relay_hosts) make check-dns / deploy-dns # iroh_dns_server.yml only (iroh_dns_server_hosts) @@ -520,13 +536,14 @@ make build / galaxy-check # both collections (build-node, galaxy-check- `ansible/inventory/hosts.yml` is git-ignored and a real fleet lives in a private overlay. Inventory-adjacent group_vars do not load for an overlay, so anything every host needs regardless of inventory lives in `ansible/playbooks/group_vars/`. The public firewall holes -(`baseline_extra_inbound`, today the node's udp/4433, plus tcp/80 + tcp/443 on +(`baseline_extra_inbound`, today the node's udp/4433 (an origin's too), plus tcp/80 + tcp/443 on `sponsord_onramp_hosts` with `sponsord_onramp_proxy: caddy`, tcp/80 + tcp/443 + udp/7842 on `iroh_relay_hosts`, the last unless `iroh_relay_enable_quic_addr_discovery` is false, and tcp/443 + udp/53 + tcp/53 on `iroh_dns_server_hosts`) are built from a host's groups in -`all.yml` (`_baseline_service_inbound`). `decdn_nodes.yml`, `sponsord_hosts.yml`, -`iroh_relay_hosts.yml` and `iroh_dns_server_hosts.yml` all set `baseline_extra_inbound` -to that list, so a co-located host renders one firewall in every play. +`all.yml` (`_baseline_service_inbound`). `decdn_nodes.yml`, `decdn_origin_nodes.yml`, +`sponsord_hosts.yml`, `iroh_relay_hosts.yml` and `iroh_dns_server_hosts.yml` all set +`baseline_extra_inbound` to that list, so a co-located host renders one firewall in +every play. `ansible/tests/firewall-holes/` (run by `make test-scripts`) pins the result per host shape. The Alloy per-daemon toggles also live in `all.yml`. Don't move any of it back under `inventory/`. diff --git a/README.md b/README.md index c4b7e1b..828934f 100644 --- a/README.md +++ b/README.md @@ -55,9 +55,9 @@ end to end (with `--dry-run`); this repo stops at host prep and startup. ```bash cd ansible make deps # vendor pinned Galaxy collections -cp inventory/hosts.yml.example inventory/hosts.yml +cp inventory/hosts-node.yml.example inventory/hosts.yml # publishers: hosts-publisher.yml.example $EDITOR inventory/hosts.yml # your hosts in decdn_nodes -$EDITOR inventory/host_vars/decdn-node-1/main.yml # binaries, region, origin; chain via decdn_network +$EDITOR inventory/host_vars/decdn-node-1/main.yml # binaries, region; chain via decdn_network # RPC URL: provision 0600 /etc/decdn/decdn.env on the host (preferred), or secret.yml make check LIMIT=decdn-node-1 ANSIBLE_ARGS='-u root' # dry run; -u root only until the first converge make deploy LIMIT=decdn-node-1 ANSIBLE_ARGS='-u root' # first converge creates your admin account @@ -103,7 +103,7 @@ This is the canonical statement; the per-path READMEs add only what is specific | Path | What it is | |------|------------| -| [`ansible/`](ansible/README.md) | The Ansible project: `inventory/`, `playbooks/` (`site.yml`, `sponsord.yml`, `iroh_relay.yml`, `iroh_dns_server.yml`, `backup.yml`, `decommission.yml`), `roles/` (`baseline`, `decdn_node`, `grafana_alloy`, `sponsord`, `sponsord_onramp`, `iroh_relay`, `iroh_dns_server`), `galaxy/` (the `decdn.node` and `decdn.publisher` collections), `molecule/`. | +| [`ansible/`](ansible/README.md) | The Ansible project: `inventory/`, `playbooks/` (`site.yml`, `node.yml` for node operators, `publisher.yml` for publishers with `origin.yml`, `sponsord.yml`, `iroh_relay.yml` and `iroh_dns_server.yml`, `backup.yml`, `decommission.yml`), `roles/` (`baseline`, `decdn_node`, `grafana_alloy`, `sponsord`, `sponsord_onramp`, `iroh_relay`, `iroh_dns_server`), `galaxy/` (the `decdn.node` and `decdn.publisher` collections), `molecule/`. | | [`cloud-init/`](cloud-init/README.md) | The cloud-init deploy path: `user-data.yaml` (a node), `user-data-sponsord.yaml` (a sponsor host), the on-host `bootstrap.sh`, and the pinned ansible-core and collections it installs. | | [`compose/`](compose/README.md) | The Docker Compose deploy path: the node, `sponsord`, `sponsord-onramp` and Caddy, one profile each. | | [`charts/decdn-node/`](charts/decdn-node/README.md) | The Helm chart; it renders the node's dashboards and alert rules from `monitoring/`. | diff --git a/ansible/.gitignore b/ansible/.gitignore index cd6fd99..64708a0 100644 --- a/ansible/.gitignore +++ b/ansible/.gitignore @@ -13,9 +13,10 @@ inventory/*/host_vars/*/secret.* !inventory/*/host_vars/*/secret.*.example # The real inventory. This repo is PUBLIC, so real host IPs must never land here. -# Copy hosts.yml.example to hosts.yml for a local run, or keep a fleet's inventory -# in a private overlay and pass INVENTORY= (see README.md "Private fleet -# inventory"). The .example templates stay tracked. +# Copy hosts-node.yml.example (node operators) or hosts-publisher.yml.example +# (publishers) to hosts.yml for a local run, or keep a fleet's inventory in a private +# overlay and pass INVENTORY= (see README.md "Private fleet inventory"). The +# .example templates stay tracked. inventory/hosts.yml inventory/hosts.yaml diff --git a/ansible/Makefile b/ansible/Makefile index a4e239d..23b4c49 100644 --- a/ansible/Makefile +++ b/ansible/Makefile @@ -1,6 +1,6 @@ # Convenience targets for the deCDN Ansible project. # Always run from the ansible/ directory. -.PHONY: deps lint check deploy check-sponsord deploy-sponsord check-relay deploy-relay check-dns deploy-dns backup decommission molecule molecule-serial molecule-list build build-node build-publisher galaxy-check galaxy-check-node galaxy-check-publisher +.PHONY: deps lint check deploy check-node deploy-node check-origin deploy-origin check-publisher deploy-publisher check-sponsord deploy-sponsord check-relay deploy-relay check-dns deploy-dns backup decommission molecule molecule-serial molecule-list build build-node build-publisher galaxy-check galaxy-check-node galaxy-check-publisher SHELL := /bin/bash # Install the required Galaxy collections (>= constraints in requirements.yml) @@ -14,9 +14,9 @@ lint: yamllint . ansible-lint -# --- Primary: the public deCDN node ------------------------------------------ +# --- Deploys: node operators and publishers ----------------------------------- # Optional scoping for the deploy targets. Both are empty by default, so check/deploy -# stay fleet-wide (every host in decdn_nodes) unless you say otherwise. +# stay fleet-wide (every host in the inventory) unless you say otherwise. # make deploy LIMIT=decdn-node-1 # one host # make deploy LIMIT='decdn-node-1,decdn-node-2' # a list # make deploy ANSIBLE_ARGS='--tags decdn_node -vv' @@ -35,7 +35,7 @@ ANSIBLE_ARGS ?= limit_arg = $(if $(LIMIT),--limit '$(LIMIT)') # Which inventory check/deploy run against. The default is the git-ignored -# inventory/hosts.yml (copy hosts.yml.example). A real fleet keeps its inventory in a +# inventory/hosts.yml (copy hosts-node.yml.example or hosts-publisher.yml.example). A real fleet keeps its inventory in a # private overlay outside this public repo and points at it here: # make deploy INVENTORY=../../decdn-fleet/hosts.yml LIMIT= # The overlay's own group_vars/ and host_vars/ must sit beside that hosts.yml, where @@ -44,21 +44,21 @@ INVENTORY ?= inventory/hosts.yml # A missing or unparseable inventory fails the run. Ansible's default only warns, # skips the `hosts: decdn_nodes` play and exits 0, so a mistyped INVENTORY= would # report a successful deploy that did nothing. This applies to the playbook targets -# (check/deploy/check-sponsord/deploy-sponsord/check-relay/deploy-relay/check-dns/deploy-dns/ +# (check/deploy and their -node/-origin/-publisher/-sponsord/-relay/-dns variants, # backup/decommission) only: # ansible.cfg can't set it globally, because ansible-lint's own syntax checks run # with no inventory present. strict_inventory = ANSIBLE_INVENTORY_UNPARSED_FAILED=True # Fail loud on the two ways a run can look scoped but silently be fleet-wide. Scoped to -# the playbook targets (check/deploy[-sponsord|-relay|-dns]/backup/decommission) via MAKECMDGOALS so an +# the playbook targets (check/deploy[-node|-origin|-publisher|-sponsord|-relay|-dns]/backup/decommission) via MAKECMDGOALS so an # unrelated `make lint` is never blocked. NOTE: this # does not catch an argument that is well-formed but selects nothing — `--tags` with a # typo (decdn-node vs decdn_node) runs zero tasks and still exits 0. See ansible/README.md. -ifneq ($(filter check deploy check-sponsord deploy-sponsord check-relay deploy-relay check-dns deploy-dns backup decommission,$(MAKECMDGOALS)),) +ifneq ($(filter check deploy check-node deploy-node check-origin deploy-origin check-publisher deploy-publisher check-sponsord deploy-sponsord check-relay deploy-relay check-dns deploy-dns backup decommission,$(MAKECMDGOALS)),) # Echo back the goal the operator actually typed, so the hint never nudges a # `make check` dry-run toward the mutating `make deploy`. - scoped_goal := $(firstword $(filter check deploy check-sponsord deploy-sponsord check-relay deploy-relay check-dns deploy-dns backup decommission,$(MAKECMDGOALS))) + scoped_goal := $(firstword $(filter check deploy check-node deploy-node check-origin deploy-origin check-publisher deploy-publisher check-sponsord deploy-sponsord check-relay deploy-relay check-dns deploy-dns backup decommission,$(MAKECMDGOALS))) ifeq ($(origin LIMIT),environment) $(error LIMIT is set in your environment ($(LIMIT)) and would silently scope this run. Unset it or pass it on the command line: make $(scoped_goal) LIMIT=) endif @@ -86,16 +86,40 @@ ifneq ($(filter check deploy check-sponsord deploy-sponsord check-relay deploy-r endif endif -# Dry run against $(INVENTORY) — shows diffs, mutates nothing. +# Dry run of everything in $(INVENTORY) (site.yml: node.yml, then publisher.yml) — +# shows diffs, mutates nothing. check: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/site.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) -# Provision the deCDN node host(s) in $(INVENTORY) (default inventory/hosts.yml). +# Provision every host in $(INVENTORY) (default inventory/hosts.yml). deploy: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/site.yml $(limit_arg) $(ANSIBLE_ARGS) -# sponsord alone (playbooks/sponsord.yml, hosts in sponsord_hosts). check/deploy -# above already include it via site.yml; these leave every decdn-node untouched. +# Node operators: the cache nodes alone (playbooks/node.yml, decdn_nodes that are not +# in decdn_origin_nodes). +check-node: + $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/node.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) + +deploy-node: + $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/node.yml $(limit_arg) $(ANSIBLE_ARGS) + +# Publishers: everything a publisher runs (playbooks/publisher.yml: origins, sponsord +# and its onramp, iroh relays, iroh DNS servers), leaving the cache nodes untouched. +check-publisher: + $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/publisher.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) + +deploy-publisher: + $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/publisher.yml $(limit_arg) $(ANSIBLE_ARGS) + +# A publisher's origin nodes alone (playbooks/origin.yml, hosts in decdn_origin_nodes). +check-origin: + $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/origin.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) + +deploy-origin: + $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/origin.yml $(limit_arg) $(ANSIBLE_ARGS) + +# sponsord alone (playbooks/sponsord.yml, hosts in sponsord_hosts). check/deploy and +# check/deploy-publisher above already include it; these leave every decdn-node untouched. check-sponsord: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/sponsord.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) @@ -103,7 +127,8 @@ deploy-sponsord: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/sponsord.yml $(limit_arg) $(ANSIBLE_ARGS) # iroh relays alone (playbooks/iroh_relay.yml, hosts in iroh_relay_hosts). check/deploy -# above already include them via site.yml; these leave every other host untouched. +# and check/deploy-publisher above already include them; these leave every other host +# untouched. check-relay: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/iroh_relay.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) @@ -111,8 +136,8 @@ deploy-relay: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/iroh_relay.yml $(limit_arg) $(ANSIBLE_ARGS) # iroh DNS servers alone (playbooks/iroh_dns_server.yml, hosts in iroh_dns_server_hosts). -# check/deploy above already include them via site.yml; these leave every other host -# untouched. +# check/deploy and check/deploy-publisher above already include them; these leave every +# other host untouched. check-dns: $(strict_inventory) ansible-playbook -i '$(INVENTORY)' playbooks/iroh_dns_server.yml --check --diff $(limit_arg) $(ANSIBLE_ARGS) diff --git a/ansible/README.md b/ansible/README.md index 6bf8f8a..2d35227 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -7,12 +7,15 @@ origin nodes). | Playbook | Purpose | Make target | |----------|---------|-------------| -| **`site.yml`** | Harden the host and deploy the public **deCDN node** (`decdn-node`). | `make check` / `make deploy` | +| **`site.yml`** | Everything in the inventory: `node.yml`, then `publisher.yml`. | `make check` / `make deploy` | +| **`node.yml`** | **Node operators.** Harden the host and deploy a public deCDN **cache node** (`decdn-node`) on hosts in `decdn_nodes` that are not in `decdn_origin_nodes`. | `make check-node` / `make deploy-node` | +| **`publisher.yml`** | **Publishers.** `origin.yml`, `sponsord.yml`, `iroh_relay.yml` and `iroh_dns_server.yml`, in that order. | `make check-publisher` / `make deploy-publisher` | +| `origin.yml` | Harden the host and deploy a publisher's **origin node**: `decdn-node` with an origin backend, on hosts in `decdn_origin_nodes` (a child of `decdn_nodes`). Refuses an origin outside `decdn_nodes` or without a backend. | `make check-origin` / `make deploy-origin` | | `backup.yml` | Encrypted backup of a node's identity or full state, and of sponsord's credentials. | `make backup` | | `decommission.yml` | Stop a node, an iroh relay, an iroh DNS server, the onramp and sponsord and remove their services (keeps the keys; no on-chain steps). | `make decommission` | -| `sponsord.yml` | Harden the host and deploy **sponsord**, the onboarding sponsor, on hosts in `sponsord_hosts`, with or without a node; then its public **sponsord-onramp** (behind Caddy) on hosts in `sponsord_onramp_hosts`. Also run by `site.yml`. | `make check-sponsord` / `make deploy-sponsord` | -| `iroh_relay.yml` | Harden the host and deploy a self-hosted **iroh relay** (`iroh-relay`), the fallback path for peers that cannot hole-punch, on hosts in `iroh_relay_hosts`. Also run by `site.yml`. | `make check-relay` / `make deploy-relay` | -| `iroh_dns_server.yml` | Harden the host and deploy a self-hosted **iroh DNS server** (`iroh-dns-server`), the pkarr relay and DNS server nodes are discovered through, on hosts in `iroh_dns_server_hosts`. Also run by `site.yml`. | `make check-dns` / `make deploy-dns` | +| `sponsord.yml` | Harden the host and deploy **sponsord**, the onboarding sponsor, on hosts in `sponsord_hosts`, with or without a node; then its public **sponsord-onramp** (behind Caddy) on hosts in `sponsord_onramp_hosts`. Also run by `publisher.yml`. | `make check-sponsord` / `make deploy-sponsord` | +| `iroh_relay.yml` | Harden the host and deploy a self-hosted **iroh relay** (`iroh-relay`), the fallback path for peers that cannot hole-punch, on hosts in `iroh_relay_hosts`. Also run by `publisher.yml`. | `make check-relay` / `make deploy-relay` | +| `iroh_dns_server.yml` | Harden the host and deploy a self-hosted **iroh DNS server** (`iroh-dns-server`), the pkarr relay and DNS server nodes are discovered through, on hosts in `iroh_dns_server_hosts`. Also run by `publisher.yml`. | `make check-dns` / `make deploy-dns` | ``` baseline host hardening: DevSec os/ssh, nftables default-deny inbound, @@ -66,7 +69,7 @@ specific to this path: which doesn't honor the custom `-p` become prompt Ansible relies on — so `--ask-become-pass` hangs with "Timeout waiting for privilege escalation prompt". On an affected host uncomment `ansible_become_exe: /usr/bin/sudo.ws` for that host in - `hosts.yml` (see the note in `inventory/hosts.yml.example`) to route become through + `hosts.yml` (see the note in `inventory/hosts-node.yml.example`) to route become through classic sudo. ## Setup @@ -74,8 +77,9 @@ specific to this path: ```bash cd ansible make deps # vendor pinned collections into ./collections -cp inventory/hosts.yml.example inventory/hosts.yml -$EDITOR inventory/hosts.yml # set hosts for decdn_nodes +cp inventory/hosts-node.yml.example inventory/hosts.yml # node operators +cp inventory/hosts-publisher.yml.example inventory/hosts.yml # or publishers +$EDITOR inventory/hosts.yml # set your hosts $EDITOR inventory/group_vars/all.yml # optional: override admin user/keys, allowlists ``` @@ -110,7 +114,7 @@ point the deploy targets at it: ```bash mkdir -p ../../decdn-fleet # a PRIVATE repo/dir -cp inventory/hosts.yml.example ../../decdn-fleet/hosts.yml # then fill it in +cp inventory/hosts-node.yml.example ../../decdn-fleet/hosts.yml # then fill it in make check INVENTORY=../../decdn-fleet/hosts.yml LIMIT= make deploy INVENTORY=../../decdn-fleet/hosts.yml LIMIT= ``` @@ -120,12 +124,14 @@ the overlay carries its own: create them next to its `hosts.yml`, starting from `inventory/group_vars/` and `inventory/host_vars/` templates. Git-ignore `host_vars/*/secret.*` in the private repo, so it never tracks a per-node secret. The public `inventory/group_vars/` and `inventory/host_vars/` are **not** loaded for it. Add child groups -of your own when hosts differ by group (an origin, an egress cap on metered bandwidth), each -with a `group_vars/.yml`. Settings every host needs regardless of +of your own when hosts differ by group (disk size, an egress cap on metered bandwidth), each +with a `group_vars/.yml`. Origins go in `decdn_origin_nodes`, as a child of +`decdn_nodes` (see `inventory/hosts-publisher.yml.example`). Settings every host needs regardless of inventory, currently the public firewall holes (the node's udp/4433, and tcp/80 + tcp/443 for Caddy on `sponsord_onramp_hosts`), live in `playbooks/group_vars/`, so an overlay can't drop them: `all.yml` builds the list from the -host's groups and `decdn_nodes.yml` / `sponsord_hosts.yml` apply it. Override +host's groups and each group's file (`decdn_nodes.yml`, `decdn_origin_nodes.yml`, +`sponsord_hosts.yml`, …) applies it. Override `baseline_extra_inbound` per host in `host_vars` if you have to. Playbook group_vars beat inventory group_vars. @@ -194,10 +200,12 @@ ignored on the first converge. ```bash make check # dry run (--check --diff); asserts fire if required knobs are missing -make deploy # provision + start the node +make deploy # provision + start every service in the inventory +make deploy-node # node operators: the cache nodes only +make deploy-publisher # publishers: origins, sponsord, relays, DNS servers only ``` -Both targets are **fleet-wide by default** — every host in `decdn_nodes`. Scope a run with +All of them are **fleet-wide by default** — every host their playbooks target. Scope a run with `LIMIT`: ```bash diff --git a/ansible/galaxy/node/CHANGELOG.md b/ansible/galaxy/node/CHANGELOG.md index e347188..2a8c619 100644 --- a/ansible/galaxy/node/CHANGELOG.md +++ b/ansible/galaxy/node/CHANGELOG.md @@ -195,6 +195,10 @@ Initial packaging of the deCDN node roles as a distributable collection ### Changed +- `decdn_node`: the docs and the origin-kind assert no longer say a serving node + needs an origin backend. Without one a node is a cache node, which fills a miss + from other nodes (`decdn_node_to_node_pull_through_enabled`, on by default without + an origin); a node with one is a publisher's origin. - The seven roles ship as two collections. `decdn.node` keeps what a node operator runs (`baseline`, `decdn_node`, `grafana_alloy`); `sponsord`, `sponsord_onramp`, `iroh_relay` and `iroh_dns_server` move to `decdn.publisher`, diff --git a/ansible/inventory/group_vars/decdn_nodes.yml b/ansible/inventory/group_vars/decdn_nodes.yml index 352e450..2cc0cb5 100644 --- a/ansible/inventory/group_vars/decdn_nodes.yml +++ b/ansible/inventory/group_vars/decdn_nodes.yml @@ -1,5 +1,7 @@ --- -# Shared config for public deCDN node hosts (playbooks/site.yml). +# Shared config for public deCDN node hosts, cache nodes (playbooks/node.yml) and a +# publisher's origins alike (playbooks/origin.yml; group_vars/decdn_origin_nodes.yml +# wins over this file for them). # Per-node values live in host_vars//: addresses/region/version in main.yml. # The sensitive rpc_url is provisioned as /etc/decdn/decdn.env on the host, or (if # you would rather the role authored that file) in a git-ignored secret.yml. diff --git a/ansible/inventory/group_vars/decdn_origin_nodes.yml b/ansible/inventory/group_vars/decdn_origin_nodes.yml new file mode 100644 index 0000000..ff15c18 --- /dev/null +++ b/ansible/inventory/group_vars/decdn_origin_nodes.yml @@ -0,0 +1,24 @@ +--- +# Shared config for a publisher's origin nodes (playbooks/origin.yml, also run by +# publisher.yml and site.yml). decdn_origin_nodes is a child of decdn_nodes, so +# group_vars/decdn_nodes.yml applies to origins too, and this file wins over it. +# Per-origin values live in host_vars//main.yml (see host_vars/origin-1/). +# +# Every origin needs an origin backend: decdn_cache_origin_kind (http, fs or s3) and +# that kind's fields, or decdn_cache_origins (an ordered fallback list); origin.yml +# refuses a host without one. Set it here when every origin reads the same store, in +# host_vars when they differ. Full list: roles/decdn_node/defaults/main.yml. +# +# decdn_cache_origin_kind: s3 +# decdn_cache_origin_s3_bucket: my-namespace-content +# decdn_cache_origin_s3_region: eu-central-1 +# decdn_cache_origin_s3_endpoint_url: "" # R2/B2/MinIO; omit for AWS +# decdn_cache_origin_s3_use_default_chain: true # keys in /etc/decdn/decdn.env + +# --- Origin behaviour ---------------------------------------------------------------- +# With a backend set, the role derives two defaults: decdn_relay_foreign_namespaces +# false (the node serves only namespaces it is an origin of, ADR 037) and +# decdn_node_to_node_pull_through_enabled false (misses fill from the backend). Set +# either explicitly only to change that. +# decdn_fs_rescan_interval_sec: 60 # fs backend: how often the root is rescanned +# decdn_pinned_hashes: [] # blobs (lowercase BLAKE3) never evicted diff --git a/ansible/inventory/group_vars/iroh_dns_server_hosts.yml b/ansible/inventory/group_vars/iroh_dns_server_hosts.yml index 77d8f4e..39cb403 100644 --- a/ansible/inventory/group_vars/iroh_dns_server_hosts.yml +++ b/ansible/inventory/group_vars/iroh_dns_server_hosts.yml @@ -1,7 +1,7 @@ --- # Shared config for iroh DNS server hosts (playbooks/iroh_dns_server.yml, also run # by site.yml). Uncomment and fill in; per-host values (the hostname, at least) go -# under the host in hosts.yml (as hosts.yml.example shows) or in +# under the host in hosts.yml (as hosts-publisher.yml.example shows) or in # host_vars//main.yml. # Full variable list: roles/iroh_dns_server/defaults/main.yml and its README.md. # Nothing to provision on the host: the server generates its ACME account key and diff --git a/ansible/inventory/group_vars/iroh_relay_hosts.yml b/ansible/inventory/group_vars/iroh_relay_hosts.yml index 05ae5b5..7214679 100644 --- a/ansible/inventory/group_vars/iroh_relay_hosts.yml +++ b/ansible/inventory/group_vars/iroh_relay_hosts.yml @@ -1,7 +1,7 @@ --- # Shared config for iroh relay hosts (playbooks/iroh_relay.yml, also run by # site.yml). Uncomment and fill in; per-host values (the hostname, at least) go -# under the host in hosts.yml (as hosts.yml.example shows) or in +# under the host in hosts.yml (as hosts-publisher.yml.example shows) or in # host_vars//main.yml. # Full variable list: roles/iroh_relay/defaults/main.yml and its README.md. # Nothing to provision on the host: the relay generates its ACME account key and diff --git a/ansible/inventory/host_vars/decdn-node-1/main.yml b/ansible/inventory/host_vars/decdn-node-1/main.yml index 3833177..00c4465 100644 --- a/ansible/inventory/host_vars/decdn-node-1/main.yml +++ b/ansible/inventory/host_vars/decdn-node-1/main.yml @@ -44,12 +44,11 @@ decdn_node_install_method: release # decdn_*_address here only to override it deliberately; the role reports overrides. decdn_network: arbitrum-sepolia -# --- Cache pull-through origin (what the node fetches on a cache miss) --------- -# MUST-EDIT — operator-specific backing origin (NOT a chain fact). A serving node -# needs one, else cache misses fail NoOrigin. http shown; see defaults/main.yml for -# fs / s3 fields, and decdn_cache_origins for an ordered fallback list. -decdn_cache_origin_kind: "http" -decdn_cache_origin_url: "https://your-origin.example/" +# --- Cache ------------------------------------------------------------------- +# A cache node has no origin backend (decdn_cache_origin_kind stays unset): it fills +# a cache miss by pulling from other nodes (node-to-node pull-through, on by default +# when no origin is set) and relays every namespace. An origin backend makes a node a +# publisher's origin instead: see host_vars/origin-1/ and hosts-publisher.yml.example. # --- Node identity / locale --------------------------------------------------- decdn_region: "US" # MUST-EDIT — ISO 3166-1 alpha-2 of the node's physical location diff --git a/ansible/inventory/host_vars/origin-1/main.yml b/ansible/inventory/host_vars/origin-1/main.yml new file mode 100644 index 0000000..771ba6d --- /dev/null +++ b/ansible/inventory/host_vars/origin-1/main.yml @@ -0,0 +1,73 @@ +--- +# Per-node deployment values for origin-1, a publisher's origin node (COMMITTED — +# non-secret config only). Ansible merges every *.yml in this directory for host +# origin-1. An origin is a node like decdn-node-1 (see that host's main.yml) plus +# the origin backend below. The sensitive +# values (decdn_rpc_url, and any environment-borne secrets) live OFF this file: +# either in a 0600 /etc/decdn/decdn.env provisioned on the target host (preferred — +# nothing sensitive on the control machine), or in the sibling git-ignored +# secret.yml (copy secret.yml.example). The role refuses to start the node until +# this file is set and ONE of those two secret sources exists. +# +# CONTRACT ADDRESSES ARE NOT INVENTED HERE, and not copied here either: +# decdn_network below selects the role's mirror of upstream's deployment manifest +# (roles/decdn_node/vars/main/networks.yml, regenerated by +# scripts/sync-network-profiles.py from contracts/deployments/.json in +# decdn/decdn). +# It is the same manifest `decdn config init --chain arbitrum-sepolia` bakes in, and +# upstream redeploys change every address at once, so one mirror beats a hand copy +# per inventory. To target a different deployment, clear decdn_network and set +# decdn_chain_id plus the decdn_*_address variables from that deployment's manifest. +# MUST-EDIT lines are yours to fill. + +# --- Which binaries to install ------------------------------------------------ +# "release" (the default): the v GitHub Release tarballs, verified against +# the GPG-signed SHA256SUMS manifest. The version defaults to the decdn/decdn release +# the role pins; set it only to hold a node on another one. +decdn_node_install_method: release +# decdn_node_version: "0.0.1" +# +# Alternative: build on the node from a git ref (any tag, branch or SHA), as an +# unprivileged build user with a pinned toolchain (decdn_node_version is not checked +# in this mode): +# decdn_node_install_method: source +# decdn_node_source_ref: v0.0.1 +# Alternative: copy binaries built on this control machine +# (`cargo build --release -p decdn-node -p decdn-cli` in a decdn/decdn checkout): +# decdn_node_install_method: manual +# decdn_node_manual_bin_src: /path/to/decdn/target/release/decdn-node +# decdn_cli_manual_bin_src: /path/to/decdn/target/release/decdn + +# --- Chain (Arbitrum Sepolia, chain 421614) ----------------------------------- +# The RPC endpoint is the one sensitive value — provision it as DECDN_RPC_URL in +# /etc/decdn/decdn.env on the host, or set decdn_rpc_url in secret.yml. +# The profile supplies decdn_chain_id and all eight contract addresses (the four +# required ones, SlashAppeal, OriginAssignment, PublisherRegistry and USDC). Set any +# decdn_*_address here only to override it deliberately; the role reports overrides. +decdn_network: arbitrum-sepolia + +# --- Origin backend (the canonical source of the namespace's content) --------- +# MUST-EDIT — where this origin reads the content it serves (NOT a chain fact). +# http shown; fs (a local disk or an NFS mount: decdn_cache_origin_path) and s3 +# (S3/R2/B2/MinIO: decdn_cache_origin_s3_*) are in roles/decdn_node/defaults/main.yml, +# and decdn_cache_origins takes an ordered fallback list. S3 access keys never go +# here: put them in /etc/decdn/decdn.env on the host (AWS_ACCESS_KEY_ID, …) with +# decdn_cache_origin_s3_use_default_chain: true, or in secret.yml's decdn_extra_env. +# playbooks/origin.yml refuses a decdn_origin_nodes host without a backend. +decdn_cache_origin_kind: "http" +decdn_cache_origin_url: "https://your-origin.example/" +# Being configured is not being recognised: the namespace's publisher seats this +# node's operator with OriginAssignment.addOrigin (decdn/adr/011-content-takedown.md, +# Origin Assignment Authority). With a backend set, the node declines misses for +# namespaces it is not an origin of (decdn_relay_foreign_namespaces defaults to false) +# and does not pull from other nodes (decdn_node_to_node_pull_through_enabled). + +# --- Node identity / locale --------------------------------------------------- +decdn_region: "US" # MUST-EDIT — ISO 3166-1 alpha-2 of the node's physical location +# decdn_relay_urls: [] # optional iroh relays for NAT traversal ([] => n0 defaults) + +# --- Economics ---------------------------------------------------------------- +# Seller-side quote: this node's asking price, advertised verbatim and never clamped +# to the delivery floor by the daemon (ADR 003 §Rate-floor enforcement, ADR 019 +# §Step 3.3 — the ADRs, not this file, govern that behaviour). +decdn_rate_per_mb: 10 # USDC base units (6 decimals); 10 = $0.00001/MB diff --git a/ansible/inventory/host_vars/origin-1/secret.yml.example b/ansible/inventory/host_vars/origin-1/secret.yml.example new file mode 100644 index 0000000..c511b8c --- /dev/null +++ b/ansible/inventory/host_vars/origin-1/secret.yml.example @@ -0,0 +1,56 @@ +--- +# Per-node SECRETS for origin-1 — the INVENTORY path, which is optional. +# +# You only need this file if you want the role to author /etc/decdn/decdn.env for +# you. The preferred posture is the other way round: provision that 0600 file on +# the target host (see roles/decdn_node/files/decdn.env.example) and skip this +# file entirely — the secret then never sits in plaintext on your control machine +# and never transits it on a converge. The role fails loud if NEITHER exists. +# +# If you do use it: copy to secret.yml (same directory, git-ignored) and fill in. +# Ansible merges it with main.yml for this host. These are the ONLY per-node +# values kept out of git — everything else lives in the committed main.yml. +# +# SENSITIVE (may embed an API key). Setting this makes the role REWRITE +# /etc/decdn/decdn.env from these values on every run, discarding any lines added +# on the host. The public endpoint below works for light use; run your own or use +# a provider for production reliability. +decdn_rpc_url: "https://sepolia-rollup.arbitrum.io/rpc" + +# Optional, and only meaningful alongside decdn_rpc_url above (the role writes the +# env file all-or-nothing, and rejects extra_env against a host-provisioned file +# rather than silently dropping it). Extra KEY: value pairs appended to the 0600 +# /etc/decdn/decdn.env that systemd hands the daemon — most notably the AWS +# credentials backing an S3 cache origin, which the role deliberately does NOT +# write into node.toml (that file is 0640 and diffable): +# +# decdn_extra_env: +# AWS_ACCESS_KEY_ID: "AKIA..." +# AWS_SECRET_ACCESS_KEY: "..." +# AWS_SESSION_TOKEN: "..." # only for assume-role / SSO flows +# +# Pair that with decdn_cache_origin_s3_use_default_chain: true in main.yml. + +# --- Grafana Cloud (only when decdn_grafana_cloud_enabled is on) --------------- +# +# The opt-in observability agent accepts its API token through the same channel: +# set grafana_alloy_api_token here and the role authors /etc/grafana-alloy.env on +# the host for you (token-only, root 0600), skipping the hand-provisioning shown +# in roles/grafana_alloy/files/grafana-alloy.env.example. SENSITIVE — leave it +# commented out to keep the operator-provisioned host path instead. +# +# grafana_alloy_api_token: "glc_example_token_replace_me" + +# --- sponsord (only on a host in sponsord_hosts) -------------------------------- +# +# Same choice for sponsord's RPC endpoint: set it here and the role authors +# /etc/sponsord/secret.env (root 0600), or leave it commented out and provision that +# file on the host (roles/sponsord/files/secret.env.example). SENSITIVE. +# +# sponsord_rpc_url: "https://sepolia-rollup.arbitrum.io/rpc" +# +# And for the onramp (a host in sponsord_onramp_hosts): the Turnstile widget's +# secret key. Set it here and the role writes /etc/sponsord/turnstile-secret (root +# 0600), or leave it commented out and write that file on the host. SENSITIVE. +# +# sponsord_onramp_turnstile_secret: "0x4AAAAAAA…" diff --git a/ansible/inventory/hosts.yml.example b/ansible/inventory/hosts-node.yml.example similarity index 53% rename from ansible/inventory/hosts.yml.example rename to ansible/inventory/hosts-node.yml.example index a516691..52947f8 100644 --- a/ansible/inventory/hosts.yml.example +++ b/ansible/inventory/hosts-node.yml.example @@ -1,8 +1,9 @@ --- -# Copy to inventory/hosts.yml and fill in your hosts. inventory/hosts.yml is +# NODE OPERATORS: copy to inventory/hosts.yml and fill in your hosts (a publisher +# starts from hosts-publisher.yml.example instead). inventory/hosts.yml is # git-ignored: this repo is public, so real host IPs never belong in it. To track a -# real fleet, keep its inventory in a private overlay and pass INVENTORY= -# (see README.md "Private fleet inventory"). +# real fleet, keep its inventory in a private overlay and pass INVENTORY= (see +# README.md "Private fleet inventory"). # # No ansible_user by default: Ansible connects as your local $USER, which is the admin # account the baseline role creates (the auto-detected runner). A fresh box doesn't have @@ -29,7 +30,10 @@ # affected host: `sudo --version` reports "sudo-rs" and/or /usr/bin/sudo.ws exists. Leave # it commented on Debian bookworm / Ubuntu noble — that path does not exist there. -# PRIMARY: public deCDN node(s) — provisioned by playbooks/site.yml. +# Public deCDN cache nodes — provisioned by playbooks/node.yml (`make deploy-node`) +# and by site.yml (`make deploy`). A cache node has no origin backend: it fills a +# miss from other nodes (node-to-node pull-through). Origins are a publisher's +# (hosts-publisher.yml.example). # Per-node config (contract addresses, region, version, …) goes in the committed # inventory/host_vars//main.yml; the one secret, rpc_url, goes in a git-ignored # secret.yml beside it (copy the shipped secret.yml.example there). @@ -44,54 +48,9 @@ decdn_nodes: # decdn-node-2: # ansible_host: REPLACE_WITH_NODE_VPS_IP_OR_DNS # Child groups, for hosts that differ in a way that is easier to express per group - # than per host (disk size, an origin, an egress cap on pay-per-TB bandwidth). Give + # than per host (disk size, an egress cap on pay-per-TB bandwidth). Give # each one a group_vars/.yml; Ansible merges it over decdn_nodes'. # children: # big_disk: # hosts: # decdn-node-2: - -# OPTIONAL: sponsord, the onboarding sponsor (treasury signer + PaymentPool keeper) — -# provisioned by playbooks/sponsord.yml (`make deploy-sponsord`) and by site.yml. It -# needs no node: give it its own host, or list a node host here as well to co-locate -# them. Its knobs are in group_vars/sponsord_hosts.yml; the treasury keystore and -# password are provisioned on the host (roles/sponsord/README.md). -# sponsord_hosts: -# hosts: -# sponsord-1: -# ansible_host: REPLACE_WITH_SPONSORD_VPS_IP_OR_DNS -# -# OPTIONAL: sponsord-onramp, sponsord's public side (Turnstile gate, installers, the -# decdn-sponsored CLI API), behind Caddy on tcp/80 + tcp/443. List a sponsord host -# again here; it must be in sponsord_hosts too. Its knobs are in -# group_vars/sponsord_onramp_hosts.yml; the Turnstile secret is provisioned on the -# host (roles/sponsord_onramp/README.md). -# sponsord_onramp_hosts: -# hosts: -# sponsord-1: -# -# OPTIONAL: self-hosted iroh relays, the fallback path for peers that cannot -# hole-punch — provisioned by playbooks/iroh_relay.yml (`make deploy-relay`) and by -# site.yml. Public by design: each terminates its own Let's Encrypt TLS on tcp/80 + -# tcp/443 and serves QUIC address discovery on udp/7842, so it cannot share a host -# with sponsord-onramp. Deploy at least two before pointing nodes' decdn_relay_urls -# at them. Its knobs are in group_vars/iroh_relay_hosts.yml (roles/iroh_relay/README.md). -# iroh_relay_hosts: -# hosts: -# relay-1: -# ansible_host: REPLACE_WITH_RELAY_VPS_IP_OR_DNS -# iroh_relay_hostname: relay1.example.org -# -# OPTIONAL: a self-hosted iroh DNS server, the pkarr relay nodes publish their -# address records to and the DNS server peers resolve them through (in place of -# n0's dns.iroh.link) — provisioned by playbooks/iroh_dns_server.yml (`make -# deploy-dns`) and by site.yml. Public by design: it terminates its own Let's -# Encrypt TLS on tcp/443 and answers DNS on udp+tcp/53, so it cannot share a host -# with an iroh relay or sponsord-onramp. The parent zone must delegate its hostname -# to it (NS + glue). Its knobs are in group_vars/iroh_dns_server_hosts.yml -# (roles/iroh_dns_server/README.md). -# iroh_dns_server_hosts: -# hosts: -# dns-1: -# ansible_host: REPLACE_WITH_DNS_VPS_IP_OR_DNS -# iroh_dns_server_hostname: dns.example.org diff --git a/ansible/inventory/hosts-publisher.yml.example b/ansible/inventory/hosts-publisher.yml.example new file mode 100644 index 0000000..6b9a20a --- /dev/null +++ b/ansible/inventory/hosts-publisher.yml.example @@ -0,0 +1,84 @@ +--- +# PUBLISHERS: copy to inventory/hosts.yml and fill in your hosts (a node operator +# starts from hosts-node.yml.example instead). inventory/hosts.yml is git-ignored: +# this repo is public, so real host IPs never belong in it. To track a real fleet, +# keep its inventory in a private overlay and pass INVENTORY= (see README.md +# "Private fleet inventory"). +# +# A publisher runs its origin nodes and, optionally, the services around them. Each +# group below has its own playbook; `make deploy-publisher` (playbooks/publisher.yml) +# runs them all, and `make deploy` (site.yml) also runs any cache nodes listed here. +# +# Bootstrapping a fresh host, ansible_user and the Ubuntu sudo-rs workaround work as +# in hosts-node.yml.example: read its header before the first run. In short, the first +# run per host is `make deploy-publisher LIMIT= ANSIBLE_ARGS='-u root'` (your +# image's bootstrap user in place of root); after it, plain `make deploy-publisher`. + +# Origin nodes — provisioned by playbooks/origin.yml (`make deploy-origin`) and by +# publisher.yml. An origin is a deCDN node with an origin backend: the canonical +# source of its namespace's content. decdn_origin_nodes is a child of decdn_nodes, so +# an origin gets a node's firewall hole, backup, decommission and observability; +# origin.yml refuses one outside decdn_nodes or without a backend +# (group_vars/decdn_origin_nodes.yml). The chain recognises it as the origin only once +# you, the namespace's publisher, seat its operator with OriginAssignment.addOrigin; +# until then it serves its bytes as a cache does. Per-origin config (the backend, +# region, …) goes in host_vars//main.yml (see host_vars/origin-1/). +decdn_nodes: + children: + decdn_origin_nodes: + hosts: + origin-1: + ansible_host: REPLACE_WITH_ORIGIN_VPS_IP_OR_DNS + # More origins: one entry each, with its own host_vars//main.yml + # (decdn_region and the origin backend are REQUIRED per host). + # origin-2: + # ansible_host: REPLACE_WITH_ORIGIN_VPS_IP_OR_DNS + +# OPTIONAL: sponsord, the onboarding sponsor (treasury signer + PaymentPool keeper), +# which pays for your users' downloads — provisioned by playbooks/sponsord.yml (`make +# deploy-sponsord`) and by publisher.yml. It needs no node: give it its own host, or +# list an origin host here as well to co-locate them. Its knobs are in +# group_vars/sponsord_hosts.yml; the treasury keystore and password are provisioned on +# the host (roles/sponsord/README.md). +# sponsord_hosts: +# hosts: +# sponsord-1: +# ansible_host: REPLACE_WITH_SPONSORD_VPS_IP_OR_DNS +# +# OPTIONAL: sponsord-onramp, sponsord's public side (Turnstile gate, installers, the +# decdn-sponsored CLI API), behind Caddy on tcp/80 + tcp/443. List a sponsord host +# again here; it must be in sponsord_hosts too. Its knobs are in +# group_vars/sponsord_onramp_hosts.yml; the Turnstile secret is provisioned on the +# host (roles/sponsord_onramp/README.md). +# sponsord_onramp_hosts: +# hosts: +# sponsord-1: +# +# OPTIONAL: self-hosted iroh relays, the fallback path for peers that cannot +# hole-punch — provisioned by playbooks/iroh_relay.yml (`make deploy-relay`) and by +# publisher.yml. Public by design: each terminates its own Let's Encrypt TLS on +# tcp/80 + tcp/443 and serves QUIC address discovery on udp/7842, so it cannot share a +# host with sponsord-onramp. Deploy at least two before pointing nodes' +# decdn_relay_urls at them; by default a relay admits only this inventory's nodes +# (iroh_relay_access: allowlist), so list other operators' node IDs in +# iroh_relay_allowlist or use everyone. Its knobs are in +# group_vars/iroh_relay_hosts.yml (roles/iroh_relay/README.md). +# iroh_relay_hosts: +# hosts: +# relay-1: +# ansible_host: REPLACE_WITH_RELAY_VPS_IP_OR_DNS +# iroh_relay_hostname: relay1.example.org +# +# OPTIONAL: a self-hosted iroh DNS server, the pkarr relay nodes publish their +# address records to and the DNS server peers resolve them through (in place of +# n0's dns.iroh.link) — provisioned by playbooks/iroh_dns_server.yml (`make +# deploy-dns`) and by publisher.yml. Public by design: it terminates its own Let's +# Encrypt TLS on tcp/443 and answers DNS on udp+tcp/53, so it cannot share a host +# with an iroh relay or sponsord-onramp. The parent zone must delegate its hostname +# to it (NS + glue). Its knobs are in group_vars/iroh_dns_server_hosts.yml +# (roles/iroh_dns_server/README.md). +# iroh_dns_server_hosts: +# hosts: +# dns-1: +# ansible_host: REPLACE_WITH_DNS_VPS_IP_OR_DNS +# iroh_dns_server_hostname: dns.example.org diff --git a/ansible/playbooks/group_vars/all.yml b/ansible/playbooks/group_vars/all.yml index d04418e..e8eac82 100644 --- a/ansible/playbooks/group_vars/all.yml +++ b/ansible/playbooks/group_vars/all.yml @@ -21,12 +21,13 @@ grafana_alloy_iroh_relay_enabled: "{{ 'iroh_relay_hosts' in group_names }}" grafana_alloy_iroh_dns_server_enabled: "{{ 'iroh_dns_server_hosts' in group_names }}" # The public holes in the baseline firewall, from the groups a host is in. Each -# group that runs baseline (decdn_nodes.yml, sponsord_hosts.yml, iroh_relay_hosts.yml, -# iroh_dns_server_hosts.yml) sets baseline_extra_inbound to this, so every play on a -# host renders the same rules. +# group that runs baseline (decdn_nodes.yml, decdn_origin_nodes.yml, sponsord_hosts.yml, +# iroh_relay_hosts.yml, iroh_dns_server_hosts.yml) sets baseline_extra_inbound to this, +# so every play on a host renders the same rules. # Only genuinely public listeners belong here: metrics, admin RPC and the sponsord # API stay on loopback and get no hole. -# decdn_nodes: the node's QUIC listener. decdn_bind_port is a decdn_node role +# decdn_nodes: the node's QUIC listener, an origin's included (decdn_origin_nodes +# are decdn_nodes too; origin.yml refuses one that is not). decdn_bind_port is a decdn_node role # default, which a play without that role (sponsord.yml) does not load; the # fallback must match roles/decdn_node/defaults/main.yml. # sponsord_onramp_hosts: Caddy, the onramp's TLS front (80 for the ACME challenge diff --git a/ansible/playbooks/group_vars/decdn_origin_nodes.yml b/ansible/playbooks/group_vars/decdn_origin_nodes.yml new file mode 100644 index 0000000..6154d20 --- /dev/null +++ b/ansible/playbooks/group_vars/decdn_origin_nodes.yml @@ -0,0 +1,7 @@ +--- +# Playbook-adjacent group_vars for decdn_origin_nodes (see decdn_nodes.yml beside +# this for why playbook-adjacent). An origin is also in decdn_nodes, as a child group +# or by name, and inherits that file's value; it is set here as well because every +# group whose play runs baseline sets it, so no group ordering can leave an origin's +# play without the node's QUIC hole (all.yml builds the list from the host's groups). +baseline_extra_inbound: "{{ _baseline_service_inbound }}" diff --git a/ansible/playbooks/node.yml b/ansible/playbooks/node.yml new file mode 100644 index 0000000..382ef16 --- /dev/null +++ b/ansible/playbooks/node.yml @@ -0,0 +1,39 @@ +--- +# NODE OPERATORS: hardened, public deCDN cache nodes. +# decdn_nodes (minus decdn_origin_nodes): baseline -> grafana_alloy (opt-in, off by +# default) -> decdn-node +# Standalone entry point (`make deploy-node`), also imported by site.yml. The node +# listens on public QUIC udp/4433; metrics + admin RPC stay loopback. On-chain stake + +# registration (ADR 019 Phase 2) is an operator action, not automated here — see +# roles/decdn_node/README.md. +# A publisher's origin nodes are decdn_nodes too, but they are listed in +# decdn_origin_nodes and deployed by origin.yml (publisher.yml), so this play leaves +# them alone and site.yml runs each node once. +- name: Provision deCDN cache nodes + hosts: decdn_nodes:!decdn_origin_nodes + become: true + pre_tasks: + # A node with an origin backend serves as its namespace's canonical source and + # declines foreign-namespace misses (decdn_relay_foreign_namespaces). It still + # works from here, so this warns rather than fails. + - name: Warn about an origin backend on a host outside decdn_origin_nodes + ansible.builtin.debug: + msg: >- + WARNING: {{ inventory_hostname }} has an origin backend + (decdn_cache_origin_kind / decdn_cache_origins) but is not in + decdn_origin_nodes. It serves as an origin all the same; list it under + decdn_origin_nodes so publisher.yml deploys it and its placement is checked. + when: >- + (decdn_cache_origin_kind | default('') | length > 0) + or (decdn_cache_origins | default([]) | length > 0) + tags: [always] + roles: + - role: baseline + tags: [baseline] + # Opt-in Grafana Cloud observability (off by default; one mirrored inventory + # flag drives this + the otlp_endpoint injection inside decdn_node). Runs + # BEFORE decdn_node so Alloy's receivers exist before the daemon exports. + - role: grafana_alloy + tags: [observability, decdn_node, node] + - role: decdn_node + tags: [decdn_node, node] diff --git a/ansible/playbooks/origin.yml b/ansible/playbooks/origin.yml new file mode 100644 index 0000000..4f92a4c --- /dev/null +++ b/ansible/playbooks/origin.yml @@ -0,0 +1,48 @@ +--- +# PUBLISHERS: origin nodes, the decdn-node daemon with an origin backend (S3/R2/B2/ +# MinIO, NFS or local disk, or an HTTP upstream) as its namespace's canonical source. +# decdn_origin_nodes: baseline -> grafana_alloy (opt-in, off by default) -> decdn-node +# Standalone entry point (`make deploy-origin`), also imported by publisher.yml. The +# same roles as node.yml: an origin is a node, configured with +# decdn_cache_origin_kind (+ its fields) or decdn_cache_origins. Being recognised as +# the origin is on-chain: the namespace's publisher seats the operator with +# OriginAssignment.addOrigin (decdn/adr/011-content-takedown.md, Origin Assignment +# Authority); without that, its bytes are served as cache. Stake + registration are +# the same operator steps as any node's — see roles/decdn_node/README.md. +- name: Provision deCDN origin nodes + hosts: decdn_origin_nodes + become: true + pre_tasks: + # The firewall hole, backup, decommission, the relay allowlist and Alloy's node + # scrape all follow decdn_nodes, so an origin outside it would lose them. The role + # checks no group name, so the decdn.node collection does not impose this repo's + # groups on others. tests/scripts-test.sh runs these tasks by the play's name. + - name: Require the origin in decdn_nodes too + ansible.builtin.assert: + that: + - "'decdn_nodes' in group_names" + fail_msg: >- + {{ inventory_hostname }} is in decdn_origin_nodes but not in decdn_nodes. An + origin is a deCDN node: list it under decdn_nodes too (as a child group, or + by name), which gives it the node's firewall hole, backup, decommission and + observability. + tags: [always] + - name: Require an origin backend + ansible.builtin.assert: + that: + - >- + (decdn_cache_origin_kind | default('') | length > 0) + or (decdn_cache_origins | default([]) | length > 0) + fail_msg: >- + {{ inventory_hostname }} is in decdn_origin_nodes but has no origin backend: + set decdn_cache_origin_kind (http, fs or s3) and its fields, or + decdn_cache_origins (see roles/decdn_node/README.md), or move the host out + of decdn_origin_nodes to run it as a cache node. + tags: [always] + roles: + - role: baseline + tags: [baseline] + - role: grafana_alloy + tags: [observability, decdn_node, node] + - role: decdn_node + tags: [decdn_node, node] diff --git a/ansible/playbooks/publisher.yml b/ansible/playbooks/publisher.yml new file mode 100644 index 0000000..5842aac --- /dev/null +++ b/ansible/playbooks/publisher.yml @@ -0,0 +1,21 @@ +--- +# PUBLISHERS: everything a publisher runs, in order. +# origin.yml decdn_origin_nodes +# sponsord.yml sponsord_hosts (+ sponsord_onramp_hosts) +# iroh_relay.yml iroh_relay_hosts (reads the nodes' IDs, origins included) +# iroh_dns_server.yml iroh_dns_server_hosts +# Entry point `make deploy-publisher`, also imported by site.yml after node.yml. Each +# component has its own entry point too (deploy-origin, deploy-sponsord, +# deploy-relay, deploy-dns), and each play matches nothing unless the inventory +# defines its group. +- name: Provision origin nodes + ansible.builtin.import_playbook: origin.yml + +- name: Provision sponsord + ansible.builtin.import_playbook: sponsord.yml + +- name: Provision iroh relays + ansible.builtin.import_playbook: iroh_relay.yml + +- name: Provision iroh DNS servers + ansible.builtin.import_playbook: iroh_dns_server.yml diff --git a/ansible/playbooks/site.yml b/ansible/playbooks/site.yml index 6c0a96d..0e111df 100644 --- a/ansible/playbooks/site.yml +++ b/ansible/playbooks/site.yml @@ -1,32 +1,11 @@ --- -# PRIMARY deployment: a hardened, public deCDN node. -# baseline -> grafana_alloy (opt-in, off by default) -> decdn-node -# The node listens on public QUIC udp/4433; metrics + admin RPC stay loopback. -# On-chain stake + registration (ADR 019 Phase 2) is an operator action, not -# automated here — see roles/decdn_node/README.md. -- name: Provision a hardened deCDN node - hosts: decdn_nodes - become: true - roles: - - role: baseline - tags: [baseline] - # Opt-in Grafana Cloud observability (off by default; one mirrored inventory - # flag drives this + the otlp_endpoint injection inside decdn_node). Runs - # BEFORE decdn_node so Alloy's receivers exist before the daemon exports. - - role: grafana_alloy - tags: [observability, decdn_node, node] - - role: decdn_node - tags: [decdn_node, node] +# Everything in the inventory: what node operators run, then what publishers run. +# node.yml decdn_nodes that are not origins (node operators) +# publisher.yml origins, sponsord (+ onramp), iroh relays, iroh DNS servers +# A host is in as many of those groups as it runs services; each play matches nothing +# unless the inventory defines its group. +- name: Provision deCDN cache nodes + ansible.builtin.import_playbook: node.yml -# sponsord on hosts in sponsord_hosts (none unless the inventory defines the group). -- name: Provision sponsord - ansible.builtin.import_playbook: sponsord.yml - -# iroh relays on hosts in iroh_relay_hosts (none unless the inventory defines the group). -- name: Provision iroh relays - ansible.builtin.import_playbook: iroh_relay.yml - -# iroh DNS servers on hosts in iroh_dns_server_hosts (none unless the inventory -# defines the group). -- name: Provision iroh DNS servers - ansible.builtin.import_playbook: iroh_dns_server.yml +- name: Provision the publisher's services + ansible.builtin.import_playbook: publisher.yml diff --git a/ansible/roles/baseline/README.md b/ansible/roles/baseline/README.md index efc1f32..428bf23 100644 --- a/ansible/roles/baseline/README.md +++ b/ansible/roles/baseline/README.md @@ -90,4 +90,4 @@ nftables, DevSec `os_hardening`/`ssh_hardening` and fail2ban need a real machine. It is verified on real hosts (the deCDN team's own fleet runs Ubuntu 26.04) and with `make check` as a dry run. On Ubuntu 25.10+ note the sudo-rs become workaround in -`inventory/hosts.yml.example`. +`inventory/hosts-node.yml.example`. diff --git a/ansible/roles/decdn_node/README.md b/ansible/roles/decdn_node/README.md index 63e08d8..13fcfeb 100644 --- a/ansible/roles/decdn_node/README.md +++ b/ansible/roles/decdn_node/README.md @@ -246,9 +246,12 @@ Optional (omitted from `node.toml` unless set): file, which also means nothing downstream catches a malformed value — hence the role's own shape assert. (The former `decdn_swap_*` knobs for the on-chain swap path were removed upstream.) -- `decdn_cache_origin_kind` (`http`|`fs`|`s3`) + that kind's fields — the - pull-through origin the node fetches on a cache miss. **A serving node needs - one:** unset ⇒ no `[cache.origin]` and cache misses fail `NoOrigin`. For an +- `decdn_cache_origin_kind` (`http`|`fs`|`s3`) + that kind's fields — the origin + backend a publisher's **origin node** fills a cache miss from (in this repo's + playbooks, a `decdn_origin_nodes` host; `playbooks/origin.yml` requires one). + Unset ⇒ no `[cache.origin]`: a **cache node**, which fills a miss from other nodes + (`decdn_node_to_node_pull_through_enabled`, on by default without an origin) and + fails it `NoOrigin` when none of them has the blob. For an ordered fallback chain use `decdn_cache_origins` (a list of the same per-kind dicts) instead — the two forms are mutually exclusive upstream and the role fails loud if both are set. diff --git a/ansible/roles/decdn_node/defaults/main.yml b/ansible/roles/decdn_node/defaults/main.yml index 8cd7216..08fdecc 100644 --- a/ansible/roles/decdn_node/defaults/main.yml +++ b/ansible/roles/decdn_node/defaults/main.yml @@ -303,9 +303,10 @@ decdn_node_pull_min_throughput_bps: "" # daemon dflt 4096 # authorized origin for. Daemon default is ROLE-DERIVED — false when an origin # backend is configured, true when none — so leave "" unless you mean to pin it. decdn_relay_foreign_namespaces: "" # bool -# Pull-through origin (#437): what the node fetches on a cache miss. Empty kind => -# the [cache.origin] table is omitted and misses fail NoOrigin — a serving node -# needs an origin. Pick ONE kind and set that kind's fields; the others are ignored. +# Origin backend (#437): what an origin node fetches on a cache miss. Empty kind => +# the [cache.origin] table is omitted: a cache node, which fills a miss from other +# nodes (node-to-node pull-through above) and fails it NoOrigin when none has it. +# Pick ONE kind and set that kind's fields; the others are ignored. decdn_cache_origin_kind: "" # "" (omit) | http | fs | s3 decdn_cache_origin_url: "" # http: base URL (blobs served at {url}/{blake3_hex}) decdn_cache_origin_decompress: "" # http, optional: "auto" (default) | "strict" diff --git a/ansible/roles/decdn_node/tasks/main.yml b/ansible/roles/decdn_node/tasks/main.yml index fe3fa0c..84b9242 100644 --- a/ansible/roles/decdn_node/tasks/main.yml +++ b/ansible/roles/decdn_node/tasks/main.yml @@ -628,7 +628,8 @@ s3: decdn_cache_origin_s3_bucket + decdn_cache_origin_s3_region. decdn_cache_origin_decompress, if set, must be auto|strict, and decdn_cache_origin_s3_path_style must be a boolean (true/false). Leave the - kind empty to omit [cache.origin] (cache misses then fail NoOrigin). + kind empty to omit [cache.origin] (a cache node, which fills misses from other + nodes). when: decdn_cache_origin_kind | length > 0 # The inverse gate. Every decdn_cache_origin_* default is "", and the validator diff --git a/ansible/tests/firewall-holes/check.sh b/ansible/tests/firewall-holes/check.sh index c1043d7..406348d 100755 --- a/ansible/tests/firewall-holes/check.sh +++ b/ansible/tests/firewall-holes/check.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # The baseline firewall's public holes are derived from a host's groups in -# playbooks/group_vars/ (all.yml builds the list; decdn_nodes.yml, +# playbooks/group_vars/ (all.yml builds the list; decdn_nodes.yml, decdn_origin_nodes.yml, # sponsord_hosts.yml, iroh_relay_hosts.yml and iroh_dns_server_hosts.yml all point # baseline_extra_inbound at it). Resolve them for # every host shape in inventory.yml, outside any role, which is how the sponsord diff --git a/ansible/tests/firewall-holes/expected.json b/ansible/tests/firewall-holes/expected.json index 8262b78..5bbd55e 100644 --- a/ansible/tests/firewall-holes/expected.json +++ b/ansible/tests/firewall-holes/expected.json @@ -48,5 +48,12 @@ {"proto": "tcp", "port": 443, "comment": "iroh-dns-server (https, pkarr, ACME)"}, {"proto": "udp", "port": 53, "comment": "iroh-dns-server (dns)"}, {"proto": "tcp", "port": 53, "comment": "iroh-dns-server (dns over tcp)"} + ], + "origin": [{"proto": "udp", "port": 4433, "comment": "decdn QUIC"}], + "origin-listed": [{"proto": "udp", "port": 4433, "comment": "decdn QUIC"}], + "origin-onramp": [ + {"proto": "udp", "port": 4433, "comment": "decdn QUIC"}, + {"proto": "tcp", "port": 80, "comment": "sponsord-onramp (Caddy, http)"}, + {"proto": "tcp", "port": 443, "comment": "sponsord-onramp (Caddy, https)"} ] } diff --git a/ansible/tests/firewall-holes/inventory.yml b/ansible/tests/firewall-holes/inventory.yml index 9422a7b..d494e3e 100644 --- a/ansible/tests/firewall-holes/inventory.yml +++ b/ansible/tests/firewall-holes/inventory.yml @@ -14,9 +14,18 @@ all: co-located-onramp: {} relay-node: {} dns-node: {} + origin-listed: {} + # A publisher's origins: a child group, so its hosts are decdn_nodes too. + children: + decdn_origin_nodes: + hosts: + origin: {} + origin-listed: {} + origin-onramp: {} sponsord_hosts: hosts: sponsord: {} + origin-onramp: {} co-located: {} onramp: {} onramp-own-proxy: {} @@ -29,6 +38,7 @@ all: onramp-own-proxy: sponsord_onramp_proxy: none co-located-onramp: {} + origin-onramp: {} iroh_relay_hosts: hosts: relay: {} diff --git a/ansible/tests/playbook-guards/inventory.yml b/ansible/tests/playbook-guards/inventory.yml index 9087062..0c13e09 100644 --- a/ansible/tests/playbook-guards/inventory.yml +++ b/ansible/tests/playbook-guards/inventory.yml @@ -9,6 +9,21 @@ all: hosts: node-only: {} dns-node: {} + node-with-origin: + decdn_cache_origin_kind: http + origin-ok: {} + origin-list: {} + origin-none: {} + # A top-level group here, not decdn_nodes' child, so a host can be in it alone. + decdn_origin_nodes: + hosts: + origin-ok: + decdn_cache_origin_kind: http + origin-list: + decdn_cache_origins: [{kind: fs, path: /srv/content}] + origin-none: {} + origin-orphan: + decdn_cache_origin_kind: http sponsord_hosts: hosts: both: {} diff --git a/cloud-init/baseline-plays.awk b/cloud-init/baseline-plays.awk index ed906ff..5be9736 100644 --- a/cloud-init/baseline-plays.awk +++ b/cloud-init/baseline-plays.awk @@ -19,10 +19,18 @@ # Exit 0 when each group has such a play and every one of them selects a baseline task; # 1 otherwise, with one line per group on stderr; 2 when no groups are given. -function end_play() { +# A play counts for each group its host pattern selects: the positive terms of a +# pattern like `decdn_nodes:!decdn_origin_nodes` (decdn_nodes), not its `!` exclusions +# or `&` intersections. +function end_play( n_terms, terms, i, t) { if (pattern != "" && has_localhost) { - plays[pattern]++ - if (baseline == 0) bare[pattern]++ + n_terms = split(pattern, terms, /[:,]/) + for (i = 1; i <= n_terms; i++) { + t = terms[i] + if (t == "" || t ~ /^[!&]/) continue + plays[t]++ + if (baseline == 0) bare[t]++ + } } pattern = ""; has_localhost = 0; baseline = 0; section = "" } diff --git a/docs/requirements.md b/docs/requirements.md index 5d688bf..a5663ba 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -35,7 +35,7 @@ and they run the same roles. | Docker Compose | v2 with `env_file.required` and `depends_on.restart` support (2.24+) | rendered in CI with every profile on | On Ubuntu 25.10 and later, `sudo` is sudo-rs; see the `ansible_become_exe` note in -`ansible/inventory/hosts.yml.example`. +`ansible/inventory/hosts-node.yml.example`. ## Network diff --git a/tests/scripts-test.sh b/tests/scripts-test.sh index 8a3efb2..db83b9d 100755 --- a/tests/scripts-test.sh +++ b/tests/scripts-test.sh @@ -45,6 +45,16 @@ expect 2 "deploy-dns refuses LIMIT from the environment" env LIMIT=h make -s mk deploy-dns LIMIT=h | grep -q -- "playbooks/iroh_dns_server.yml --limit 'h'" \ || fail "deploy-dns LIMIT=h does not run playbooks/iroh_dns_server.yml scoped to h" pass "deploy-dns LIMIT=h scopes playbooks/iroh_dns_server.yml" +# The persona entry points: each goal is guarded and runs its own playbook. +for t in check-node:node deploy-node:node check-origin:origin deploy-origin:origin \ + check-publisher:publisher deploy-publisher:publisher; do + expect 2 "${t%%:*} refuses LIMIT from the environment" env LIMIT=h make -s -C "$repo/ansible" -n "${t%%:*}" + mk "${t%%:*}" LIMIT=h | grep -q -- "playbooks/${t##*:}.yml.* --limit 'h'" \ + || fail "${t%%:*} LIMIT=h does not run playbooks/${t##*:}.yml scoped to h" +done +mk check-publisher | grep -q -- "playbooks/publisher.yml --check --diff" || fail "check-publisher is not a --check run" +mk deploy | grep -q -- "playbooks/site.yml" || fail "deploy does not run playbooks/site.yml" +pass "check/deploy-node, -origin and -publisher scope their own playbooks; deploy runs site.yml" # --- molecule driver (scripts/molecule.sh): selection, guards, locks ------------------ # Against a throwaway ansible/ layout, so nothing here can start a container or touch @@ -1202,6 +1212,18 @@ expect 1 "baseline guard refuses a group whose play does not list localhost" \ grep -q 'no play targets decdn_nodes with localhost in it' "$work/out" \ || { cat "$work/out" >&2; fail "the baseline guard refused, but not for decdn_nodes"; } expect 2 "baseline guard refuses an empty group list" guard_plays "" "$hardened" +# A play's pattern counts for its positive terms only: node.yml's +# decdn_nodes:!decdn_origin_nodes play is a decdn_nodes play, never an origin one. +excl_out() { # + printf " play #1 (decdn_nodes:!decdn_origin_nodes): Provision deCDN cache nodes\tTAGS: []\n pattern: ['decdn_nodes:!decdn_origin_nodes']\n hosts (1):\n localhost\n tasks:\n%b\n" "$1" +} +excl_plays() { excl_out "$2" | awk -v groups="$1" -f "$repo/cloud-init/baseline-plays.awk"; } +expect 0 "baseline guard counts a decdn_nodes:!decdn_origin_nodes play for decdn_nodes" excl_plays decdn_nodes "$hardened" +expect 1 "baseline guard does not count an excluded group's play" excl_plays decdn_origin_nodes "$hardened" +grep -q 'no play targets decdn_origin_nodes with localhost in it' "$work/out" \ + || { cat "$work/out" >&2; fail "the baseline guard counted the exclusion as a decdn_origin_nodes play"; } +expect 1 "baseline guard refuses a bare decdn_nodes:!decdn_origin_nodes play" \ + excl_plays decdn_nodes ' Apply DevSec OS hardening\tTAGS: [baseline]' # bootstrap.sh's gate and lint.py agree on the groups a user-data may use, and the # bootstrap's hint has its own instructions for each. boot="$repo/cloud-init/bootstrap.sh" @@ -1218,17 +1240,19 @@ pass "bootstrap.sh's groups match lint.py's, each with its secrets and its hint" # run the guard too, but in Docker and only for the groups their inventories use. # `hosts` and `tags` may each be a string or a list; tags match exactly. if command -v yq >/dev/null; then - for g in decdn_nodes sponsord_hosts; do + for g in decdn_nodes decdn_origin_nodes sponsord_hosts; do n=0 - for pb in site.yml sponsord.yml; do - plays="[.[] | select([.hosts] | flatten | any_c(. == \"$g\"))]" + for pb in node.yml origin.yml sponsord.yml; do + # A play's positive pattern terms: decdn_nodes:!decdn_origin_nodes targets + # decdn_nodes only. + plays="[.[] | select([.hosts] | flatten | map(split(\":\")) | flatten | map(split(\",\")) | flatten | any_c(. == \"$g\"))]" n=$((n + $(yq "$plays | length" "$repo/ansible/playbooks/$pb"))) [[ "$(yq "$plays | map(select((.roles // []) | map(select(.role == \"baseline\" and ([.tags // []] | flatten | any_c(. == \"baseline\")))) | length == 0)) | length" "$repo/ansible/playbooks/$pb")" == 0 ]] \ || fail "playbooks/$pb: a $g play lacks the baseline role tagged baseline (cloud-init's phase 1 relies on it)" done - ((n >= 1)) || fail "no play in site.yml or sponsord.yml targets $g" + ((n >= 1)) || fail "no play in node.yml, origin.yml or sponsord.yml targets $g" done - pass "every decdn_nodes and sponsord_hosts play runs baseline under the baseline tag" + pass "every decdn_nodes, decdn_origin_nodes and sponsord_hosts play runs baseline under the baseline tag" # baseline-plays.sh, the command bootstrap.sh runs, against the real playbooks and the # templates' inventories, then against a copy whose sponsord play lost the tag (#90). # Listing tasks needs ansible-core only, not the collections. @@ -1443,6 +1467,34 @@ if command -v ansible >/dev/null; then expect 2 "sponsord.yml refuses an onramp host outside sponsord_hosts" guard onramp-only grep -q "is in sponsord_onramp_hosts but not in sponsord_hosts" "$work/out" \ || { cat "$work/out" >&2; fail "the onramp-only refusal did not come from the placement check"; } + # playbooks/origin.yml refuses an origin outside decdn_nodes or without an origin + # backend; node.yml warns about an origin backend outside decdn_origin_nodes. The + # plays' pre_tasks are lifted out as tasks, without the roles or fact gathering. + lift() { # + yq "[.[] | select(.name == \"$2\") | {\"name\": .name, \"hosts\": .hosts, \"gather_facts\": false, \"tasks\": .pre_tasks}]" \ + "$repo/ansible/playbooks/$1" > "$work/guard.yml" + [[ "$(yq 'length' "$work/guard.yml")" == 1 && "$(yq '.[0].tasks | length' "$work/guard.yml")" -ge 1 ]] \ + || fail "playbooks/$1 has no single '$2' play with pre_tasks" + } + lift origin.yml "Provision deCDN origin nodes" + expect 0 "origin.yml accepts an origin in decdn_nodes with decdn_cache_origin_kind" guard origin-ok + expect 0 "origin.yml accepts an origin in decdn_nodes with decdn_cache_origins" guard origin-list + expect 2 "origin.yml refuses an origin outside decdn_nodes" guard origin-orphan + grep -q "is in decdn_origin_nodes but not in decdn_nodes" "$work/out" \ + || { cat "$work/out" >&2; fail "the origin-orphan refusal did not come from the decdn_nodes check"; } + expect 2 "origin.yml refuses an origin without an origin backend" guard origin-none + grep -q "is in decdn_origin_nodes but has no origin backend" "$work/out" \ + || { cat "$work/out" >&2; fail "the origin-none refusal did not come from the backend check"; } + lift node.yml "Provision deCDN cache nodes" + expect 0 "node.yml runs a node with an origin backend outside decdn_origin_nodes" guard node-with-origin + grep -q "WARNING: node-with-origin has an origin backend" "$work/out" \ + || { cat "$work/out" >&2; fail "node.yml does not warn about an origin backend outside decdn_origin_nodes"; } + expect 0 "node.yml runs a cache node" guard node-only + ! grep -q "WARNING:" "$work/out" || { cat "$work/out" >&2; fail "node.yml warns about a cache node"; } + guard origin-ok >"$work/out" 2>&1 || true + grep -q "skipping: no hosts matched" "$work/out" \ + || { cat "$work/out" >&2; fail "node.yml's play matches an origin (decdn_origin_nodes) host"; } + pass "node.yml's play leaves decdn_origin_nodes to origin.yml" # decommission.yml's host cap covers the whole run: a node-only host and a # sponsord host pass each role's own per-play cap but not this one. Only the # cap task is lifted (the hold check needs the roles). From 412160fae78536333ff6c5daf5bd4b0d38491438 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Sat, 10 Oct 2026 19:28:31 +0300 Subject: [PATCH 2/3] fix(ansible): say an origin serves what its backend holds, not its namespaces The relay_foreign_namespaces gate is backend-authoritative (ADR 002, Retrieval by namespace): an origin-configured node serves only what its own backend holds and declines the rest, whatever the request's namespace. The comments and AGENTS.md said it served only the namespaces it is an origin of. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 4 ++-- ansible/inventory/group_vars/decdn_origin_nodes.yml | 9 +++++---- ansible/inventory/host_vars/origin-1/main.yml | 6 +++--- ansible/playbooks/node.yml | 4 ++-- ansible/roles/decdn_node/defaults/main.yml | 7 ++++--- 5 files changed, 16 insertions(+), 14 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d861d08..5204aa6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -152,8 +152,8 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`). (`decdn_cache_origin_kind`/`decdn_cache_origins`); node.yml's warn about a backend outside `decdn_origin_nodes`. Those group checks live in the playbooks, never the role, as with sponsord's. A cache node has no origin backend: it fills misses from other nodes - (node-to-node pull-through), and an origin-configured node serves only its own - namespaces (ADR 037). Recognition as an origin is on-chain (`OriginAssignment.addOrigin`, + (node-to-node pull-through), and an origin-configured node serves only what its own + backend holds (`relay_foreign_namespaces`, ADR 002 § Retrieval by namespace, ADR 037). Recognition as an origin is on-chain (`OriginAssignment.addOrigin`, the publisher's step). **The public deCDN node** (baseline + `decdn-node`), installed by `decdn_node_install_method`: `release` (the default) — a pinned GitHub release tarball verified against the diff --git a/ansible/inventory/group_vars/decdn_origin_nodes.yml b/ansible/inventory/group_vars/decdn_origin_nodes.yml index ff15c18..3d04ada 100644 --- a/ansible/inventory/group_vars/decdn_origin_nodes.yml +++ b/ansible/inventory/group_vars/decdn_origin_nodes.yml @@ -16,9 +16,10 @@ # decdn_cache_origin_s3_use_default_chain: true # keys in /etc/decdn/decdn.env # --- Origin behaviour ---------------------------------------------------------------- -# With a backend set, the role derives two defaults: decdn_relay_foreign_namespaces -# false (the node serves only namespaces it is an origin of, ADR 037) and -# decdn_node_to_node_pull_through_enabled false (misses fill from the backend). Set -# either explicitly only to change that. +# With a backend set, two defaults follow: decdn_relay_foreign_namespaces false (the +# daemon's: the node serves only what its backend holds and declines the rest, even a +# blob already in its cache; ADR 002 § Retrieval by namespace, ADR 037) and +# decdn_node_to_node_pull_through_enabled false (the role's: misses fill from the +# backend). Set either explicitly only to change that. # decdn_fs_rescan_interval_sec: 60 # fs backend: how often the root is rescanned # decdn_pinned_hashes: [] # blobs (lowercase BLAKE3) never evicted diff --git a/ansible/inventory/host_vars/origin-1/main.yml b/ansible/inventory/host_vars/origin-1/main.yml index 771ba6d..38f1a76 100644 --- a/ansible/inventory/host_vars/origin-1/main.yml +++ b/ansible/inventory/host_vars/origin-1/main.yml @@ -58,9 +58,9 @@ decdn_cache_origin_kind: "http" decdn_cache_origin_url: "https://your-origin.example/" # Being configured is not being recognised: the namespace's publisher seats this # node's operator with OriginAssignment.addOrigin (decdn/adr/011-content-takedown.md, -# Origin Assignment Authority). With a backend set, the node declines misses for -# namespaces it is not an origin of (decdn_relay_foreign_namespaces defaults to false) -# and does not pull from other nodes (decdn_node_to_node_pull_through_enabled). +# Origin Assignment Authority). With a backend set, the node serves only what the +# backend holds (decdn_relay_foreign_namespaces defaults to false) and does not pull +# from other nodes (decdn_node_to_node_pull_through_enabled). # --- Node identity / locale --------------------------------------------------- decdn_region: "US" # MUST-EDIT — ISO 3166-1 alpha-2 of the node's physical location diff --git a/ansible/playbooks/node.yml b/ansible/playbooks/node.yml index 382ef16..59da030 100644 --- a/ansible/playbooks/node.yml +++ b/ansible/playbooks/node.yml @@ -14,8 +14,8 @@ become: true pre_tasks: # A node with an origin backend serves as its namespace's canonical source and - # declines foreign-namespace misses (decdn_relay_foreign_namespaces). It still - # works from here, so this warns rather than fails. + # declines whatever its backend does not hold (decdn_relay_foreign_namespaces). It + # still works from here, so this warns rather than fails. - name: Warn about an origin backend on a host outside decdn_origin_nodes ansible.builtin.debug: msg: >- diff --git a/ansible/roles/decdn_node/defaults/main.yml b/ansible/roles/decdn_node/defaults/main.yml index 08fdecc..34af662 100644 --- a/ansible/roles/decdn_node/defaults/main.yml +++ b/ansible/roles/decdn_node/defaults/main.yml @@ -299,9 +299,10 @@ decdn_node_pull_timeout_sec: "" # daemon dflt 20 — per-upstre # A leg is abandoned when it delivers under min_throughput_bps across the window. decdn_node_pull_stall_window_sec: "" # daemon dflt 20 decdn_node_pull_min_throughput_bps: "" # daemon dflt 4096 -# Origin-only node policy (#1759): whether to relay namespaces this node is not an -# authorized origin for. Daemon default is ROLE-DERIVED — false when an origin -# backend is configured, true when none — so leave "" unless you mean to pin it. +# Origin-only node policy (#1759): whether to relay a miss this node's own backend +# cannot produce (the gate is the backend's content, not the request's namespace; +# ADR 002 § Retrieval by namespace). Daemon default is ROLE-DERIVED — false when an +# origin backend is configured, true when none — so leave "" unless you mean to pin it. decdn_relay_foreign_namespaces: "" # bool # Origin backend (#437): what an origin node fetches on a cache miss. Empty kind => # the [cache.origin] table is omitted: a cache node, which fills a miss from other From c4009a943694be4f20e3f06a3242bb7c1d02737a Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Sat, 10 Oct 2026 19:43:01 +0300 Subject: [PATCH 3/3] docs(ansible): the quickstarts work with either inventory example Both quickstarts offered the publisher inventory, whose host is origin-1, then hard-coded decdn-node-1 in every following command. A HOST variable now carries the example's host through them. Co-Authored-By: Claude Opus 5.5 --- README.md | 14 ++++++++------ ansible/README.md | 5 +++-- 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 828934f..59e30bd 100644 --- a/README.md +++ b/README.md @@ -55,13 +55,15 @@ end to end (with `--dry-run`); this repo stops at host prep and startup. ```bash cd ansible make deps # vendor pinned Galaxy collections -cp inventory/hosts-node.yml.example inventory/hosts.yml # publishers: hosts-publisher.yml.example -$EDITOR inventory/hosts.yml # your hosts in decdn_nodes -$EDITOR inventory/host_vars/decdn-node-1/main.yml # binaries, region; chain via decdn_network +cp inventory/hosts-node.yml.example inventory/hosts.yml # node operators; its host is decdn-node-1 +# publishers instead: cp inventory/hosts-publisher.yml.example inventory/hosts.yml (host origin-1) +HOST=decdn-node-1 # or origin-1 +$EDITOR inventory/hosts.yml # your hosts +$EDITOR inventory/host_vars/$HOST/main.yml # binaries, region (an origin: its backend); chain via decdn_network # RPC URL: provision 0600 /etc/decdn/decdn.env on the host (preferred), or secret.yml -make check LIMIT=decdn-node-1 ANSIBLE_ARGS='-u root' # dry run; -u root only until the first converge -make deploy LIMIT=decdn-node-1 ANSIBLE_ARGS='-u root' # first converge creates your admin account -make deploy LIMIT=decdn-node-1 # every run after that +make check LIMIT=$HOST ANSIBLE_ARGS='-u root' # dry run; -u root only until the first converge +make deploy LIMIT=$HOST ANSIBLE_ARGS='-u root' # first converge creates your admin account +make deploy LIMIT=$HOST # every run after that ``` The full flow (bootstrap user, keystore, secrets, fleets in a private inventory) is in diff --git a/ansible/README.md b/ansible/README.md index 2d35227..ae2db70 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -95,8 +95,9 @@ canonical copy; don't duplicate them here. **Or** carry it in inventory, and the role authors `decdn.env` from those values on every run: ```bash -cp inventory/host_vars/decdn-node-1/secret.yml.example inventory/host_vars/decdn-node-1/secret.yml -$EDITOR inventory/host_vars/decdn-node-1/secret.yml # set decdn_rpc_url +HOST=decdn-node-1 # or origin-1, the publisher example's host +cp inventory/host_vars/$HOST/secret.yml.example inventory/host_vars/$HOST/secret.yml +$EDITOR inventory/host_vars/$HOST/secret.yml # set decdn_rpc_url ``` The role fails loud when neither exists, and — once it has recorded a checksum for the file —