diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 76599fb..ce3464f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -227,10 +227,13 @@ jobs: - name: cloud-init invariants run: make lint-cloud-init - # Docker Compose path: render compose/compose.yaml with its example env files and - # assert the invariants its README promises (host network, no published ports, - # digest-pinned image, read-only rootfs, no capabilities, graceful stop). Runs - # only when compose/ or the root Makefile changed. + # Docker Compose path: render compose/compose.yaml with every profile on (with its + # example env files, without env files, with an empty .env) and assert the + # invariants its README promises (compose/tests/*.jq: host network, nothing + # published, images by digest, read-only rootfs, no capabilities, exact mounts, + # loopback listeners, no secret inline, graceful stop, unset variables that fail + # closed). Runs only when compose/ or the root + # Makefile changed. compose: needs: changes if: needs.changes.outputs.compose == 'true' diff --git a/AGENTS.md b/AGENTS.md index 66490fa..7776824 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -205,13 +205,35 @@ addresses) the repo carries, and they carry their upstream commit. user-data through cloud-init (skipping `baseline`) against a locally signed release mirror, and is the suite's only coverage of the release download and verify path. -- **`compose/`** — the same node under Docker Compose on one host: the upstream image, - always by digest (`compose.yaml` builds `DECDN_IMAGE_REPO@DECDN_IMAGE_DIGEST`), the - role's host layout (`/etc/decdn` read-only, `/var/lib/decdn`), host - networking (so loopback metrics/admin stay loopback and Docker publishes no ports), - read-only rootfs, no capabilities, 300 s SIGTERM grace. `make lint-compose` asserts - those invariants (and `make test-scripts` that it rejects broken variants); - `make security` scans it. +- **`compose/`** — the node and the sponsor under Docker Compose on one host. The + node: the upstream image, always by digest (`compose.yaml` builds + `DECDN_IMAGE_REPO@DECDN_IMAGE_DIGEST`), the role's host layout (`/etc/decdn` + read-only, `/var/lib/decdn`), host networking (so loopback metrics/admin stay + loopback and Docker publishes no ports), read-only rootfs, no capabilities, 300 s + SIGTERM grace. Every service sits behind a + profile (`COMPOSE_PROFILES` in `.env`): `node`, `sponsord`, `onramp` (also starts + `sponsord`) and `caddy`. + - **sponsord / onramp:** the roles' `/etc/sponsord/` layout, except that the + credential files belong to a host `sponsord` account (bind mounts keep owner and + mode, and upstream rejects a group-readable keystore). They are mounted + read-only one by one into `/run/secrets/`. Listeners and secret paths are set in + `environment:`, which beats env files, because the release images default to + `0.0.0.0`. + - **Caddy:** the official image by digest, non-root, keeping only + `NET_BIND_SERVICE`; `compose/Caddyfile` mirrors the role's. + - **No `${VAR:?}`:** Compose interpolates disabled services too, so a required + variable would break other profiles. An unset variable renders a value its + service refuses instead (invalid image reference, unknown user, a domain with a + non-numeric port). + - **Gates:** `make lint-compose` renders with `--profile '*'` under `env -i`, + once with the example `.env` against `compose/tests/invariants.jq` (allowed + keys and exact mounts per service, so `privileged`, `pid: host` or an extra + mount fail), once without the env files (`--no-env-resolution`) against + `compose/tests/inline-env.jq` (an allow-list of inline env keys, so no RPC URL + lands in the tracked file) and once with an empty `.env` against + `compose/tests/fail-closed.jq`. All print `: ` per violation; `make test-scripts` pins + which one fires for each broken variant (`LINT_COMPOSE_FILE`, not Compose's own + `COMPOSE_FILE`). `make security` scans it. - **`charts/decdn-node/`** — the same node on Kubernetes: a one-replica StatefulSet (one release = one identity) on the upstream daemon-only image (`ghcr.io/decdn/decdn-node`; diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9a19682..2d42d0d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -32,7 +32,7 @@ targets, so a local pass means a CI pass. Deploy targets live in | `make molecule` | every `ansible/molecule/*/` scenario in parallel, in privileged systemd containers (needs Docker and util-linux `flock`; cap with `JOBS=`). `make molecule-serial` runs them one at a time for readable failures. Both take a host-wide lock, so a second suite refuses to start (overlapping runs share container names). | | `make lint-helm` | chart: `helm lint --strict`, positive/negative render tests, kubeconform (digest-pinned image), the shared schema-key check (needs `helm`, `yq`, `python3` ≥ 3.11, Docker). Set `DECDN_CLI=` to also run the real `decdn config validate` (CI can't). | | `make lint-alloy` | renders `roles/grafana_alloy`'s templates and validates them with the **real** digest-pinned Alloy binary. The molecule stub exits 0 for everything, so this is the only gate that proves the config loads. `ALLOY_BIN=` skips the download. | -| `make lint-compose` | renders `compose/compose.yaml` with its example env and asserts its security invariants | +| `make lint-compose` | renders `compose/compose.yaml` with every profile on (with its example env, without env files, with an empty `.env`) and asserts `compose/tests/invariants.jq`, `inline-env.jq` and `fail-closed.jq` | | `make lint-cloud-init` | `cloud-init schema` on `cloud-init/user-data.yaml`, then `cloud-init/tests/lint.py`: no secrets (only the bootstrap's own files, no secret-looking keys or assignments), no hardening skip, a `release` install verified against the vendored key with a host-generated wallet (trust knobs only in `decdn_nodes.vars`), localhost in `decdn_nodes`, a keyed admin account, `runcmd` exactly stage 1, shellcheck-clean scripts, and a collection lock that covers `ansible/requirements.yml` (needs `cloud-init`, `shellcheck`, `yq`). `CLOUD_INIT_FILE=` checks your own filled-in copy. | | `make test-scripts` | `tests/scripts-test.sh`: the `ansible/Makefile` scoping guards (dry runs), the release gate, and the negative cases of `lint-compose` and `lint-cloud-init` (the latter skipped without `cloud-init` on PATH). `UPSTREAM=` adds the sync generators' exit codes. | | `make security` | KICS IaC scan of `ansible/`, the rendered chart and `compose/` (digest-pinned engine, fail on HIGH) | diff --git a/Makefile b/Makefile index aff96ae..3b430f4 100644 --- a/Makefile +++ b/Makefile @@ -57,10 +57,13 @@ security-helm: ## KICS scan of the decdn-node chart's rendered manifests ( --no-progress --fail-on high # One query is excluded, deliberately: "Volume Has Sensitive Host Directory" -# (1c1325ff-…) fires on the read-only /etc/decdn mount. That directory is the node's -# own config dir, mounted :ro, and keeping the Ansible host layout is what lets the -# host CLI, backups and restores (docs/lifecycle.md) work unchanged. The two MEDIUMs -# (host network, no healthcheck) are the documented design; see compose/README.md. +# (1c1325ff-…) flags every host-path mount: /etc/decdn (ro) and /var/lib/decdn, the +# /etc/sponsord credential files (ro, one file each) and /var/lib/caddy. They are +# the Ansible roles' host layout, which is what lets the host CLI, backups and +# restores (docs/lifecycle.md) work unchanged; lint-compose pins each container's +# exact mounts instead. The MEDIUMs (host +# network, no healthcheck, Caddy's NET_BIND_SERVICE) and the INFO (both sponsord +# containers mount the API token) are the documented design; see compose/README.md. security-compose: ## KICS scan of compose/ (pinned engine image) mkdir -p kics-results docker run --rm --user $(shell id -u):$(shell id -g) -w /repo -v "$(CURDIR):/repo" $(KICS_IMAGE) \ @@ -69,27 +72,47 @@ security-compose: ## KICS scan of compose/ (pinned engine image) --report-formats json,sarif --output-path /repo/kics-results/compose \ --no-progress --fail-on high -# Renders compose/compose.yaml with the example env files and asserts the invariants -# the README promises: host networking and no published ports (so metrics and the -# admin RPC stay loopback), a digest-pinned image, read-only rootfs, every capability -# dropped, no-new-privileges, and a stop grace long enough for the daemon's drain. -COMPOSE_INVARIANTS := .services["decdn-node"] as $$s | \ - ($$s.network_mode == "host") and ($$s.ports == null) and \ - ($$s.image | test("@sha256:[0-9a-f]{64}$$")) and ($$s.read_only == true) and \ - ($$s.cap_drop == ["ALL"]) and ($$s.security_opt | index("no-new-privileges:true") != null) and \ - ($$s.stop_signal == "SIGTERM") and ($$s.stop_grace_period == "5m0s") and \ - ($$s.user | test("^[0-9]+:[0-9]+$$")) - -# COMPOSE_FILE is overridable so tests/scripts-test.sh can feed it broken variants. -COMPOSE_FILE ?= compose/compose.yaml +# Renders compose/compose.yaml with every profile on, three times, and checks each: +# - with the example .env, against compose/tests/invariants.jq: the properties the +# README promises (host network, nothing published, images by digest, read-only +# rootfs, no capabilities beyond Caddy's NET_BIND_SERVICE, exact security_opt, +# non-root users, each container's exact mounts, loopback sponsord listeners, +# secret files only, stop graces long enough for each daemon's drain); +# - with every service env file empty (/dev/null), against +# compose/tests/inline-env.jq: compose.yaml itself sets only the allowed +# environment keys, so no secret (DECDN_RPC_URL, SPONSORD_RPC_URL) moves into the +# tracked file. (Not `config --no-env-resolution`: older Compose releases, such +# as the CI runner's, still merge the env files with it.) +# - with an empty .env, against compose/tests/fail-closed.jq: an unset variable +# renders a value its service refuses, since compose.yaml cannot use `:?`. +# check <.env file> does one of them: the service env +# files are compose/'s examples, or /dev/null for each. All run +# under `env -i`, because the caller's shell variables would override the +# .env and the lint would check something other than the committed defaults. +# LINT_COMPOSE_FILE (not Compose's own COMPOSE_FILE, which operators export) is +# overridable so tests/scripts-test.sh can feed it broken variants. +LINT_COMPOSE_FILE ?= compose/compose.yaml lint-compose: ## render compose/ with its examples and check its security invariants (needs docker, jq) @set -o pipefail; \ - rendered="$$(DECDN_ENV_FILE=$(CURDIR)/compose/decdn.env.example docker compose -f '$(COMPOSE_FILE)' \ - --env-file compose/.env.example config --format json)" \ - || { echo "lint-compose: docker compose could not render $(COMPOSE_FILE) (see above)" >&2; exit 2; }; \ - jq -e '$(COMPOSE_INVARIANTS)' <<<"$$rendered" >/dev/null \ - || { echo "$(COMPOSE_FILE) violates an invariant (see the lint-compose comment in Makefile)" >&2; exit 1; } - @echo "compose invariants hold" + check() { \ + mode="$$3"; ef() { if [ "$$mode" = none ]; then echo /dev/null; else echo "$(CURDIR)/compose/$$1"; fi; }; \ + rendered="$$(env -i PATH="$$PATH" HOME="$$HOME" \ + DECDN_ENV_FILE="$$(ef decdn.env.example)" \ + SPONSORD_SECRET_ENV_FILE="$$(ef sponsord-secret.env.example)" \ + SPONSORD_ENV_FILE="$$(ef sponsord.env.example)" \ + SPONSORD_ONRAMP_ENV_FILE="$$(ef sponsord-onramp.env.example)" \ + docker compose -f '$(LINT_COMPOSE_FILE)' --env-file "$$2" --profile '*' config --format json)" \ + || { echo "lint-compose: docker compose could not render $(LINT_COMPOSE_FILE) (see above)" >&2; exit 2; }; \ + [ -n "$$rendered" ] || { echo "lint-compose: docker compose rendered nothing" >&2; exit 2; }; \ + violations="$$(jq -r -f "$$1" <<<"$$rendered")" \ + || { printf '%s\n' "$$violations" >&2; echo "lint-compose: $$1 failed (see above)" >&2; exit 2; }; \ + [ -z "$$violations" ] \ + || { sed 's/^/ /' <<<"$$violations" >&2; echo "$(LINT_COMPOSE_FILE) violates an invariant (see $$1)" >&2; exit 1; }; \ + }; \ + check compose/tests/invariants.jq compose/.env.example examples; \ + check compose/tests/inline-env.jq compose/.env.example none; \ + check compose/tests/fail-closed.jq /dev/null examples + @echo "compose invariants hold: $(LINT_COMPOSE_FILE)" # The cloud-init user-data (cloud-init/README.md): `cloud-init schema` for its shape, then # cloud-init/tests/lint.py for what a schema cannot see. That covers no secrets, no diff --git a/README.md b/README.md index 82a09b8..df4d6f5 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ day-2 tooling for operators anywhere. Four ways to deploy the same node: |------|-----|------------| | **Ansible** (`ansible/`) | VMs and bare metal, one node or a fleet. Hardens the host too (firewall, SSH, patching). Also published as the `decdn.node` Galaxy collection. | [`ansible/README.md`](ansible/README.md) | | **cloud-init** (`cloud-init/`) | One VM, no control machine: paste the user-data into your provider's "create server" form. It runs the Ansible playbook on the host itself, hardening included. | [`cloud-init/README.md`](cloud-init/README.md) | -| **Docker Compose** (`compose/`) | One host that already runs Docker. | [`compose/README.md`](compose/README.md) | +| **Docker Compose** (`compose/`) | One host that already runs Docker. Also runs the onboarding sponsor (`sponsord` + its onramp) behind profiles. | [`compose/README.md`](compose/README.md) | | **Helm** (`charts/decdn-node/`) | Kubernetes, one release per node. | [`charts/decdn-node/README.md`](charts/decdn-node/README.md) | Supported hosts: Debian 12/13 and Ubuntu 24.04/26.04 on x86_64 or aarch64. Network, @@ -82,9 +82,9 @@ This is the canonical statement; the per-path READMEs add only what is specific you choose. - **Localhost-only by default.** Backends bind `127.0.0.1`. A service that must accept public traffic declares its port explicitly, and the node declares exactly one: - udp/4433. The sponsor's public onramp (Ansible only) stays on loopback too, behind - Caddy on tcp/80 + tcp/443 (the default; with `sponsord_onramp_proxy: none` you bring - the proxy and open its ports). On Kubernetes, metrics bind `0.0.0.0` in the pod only behind a ClusterIP + udp/4433. The sponsor's public onramp (Ansible and Compose) stays on loopback too, + behind Caddy on tcp/80 + tcp/443 (the default on Ansible, the `caddy` profile on + Compose; otherwise you bring the proxy and open its ports). On Kubernetes, metrics bind `0.0.0.0` in the pod only behind a ClusterIP Service and a NetworkPolicy. - **Default-deny inbound** (Ansible's `baseline`, nftables). SSH is the only universally open port; extra public ports are declared via `baseline_extra_inbound`. @@ -101,7 +101,7 @@ This is the canonical statement; the per-path READMEs add only what is specific |------|------------| | [`ansible/`](ansible/README.md) | The Ansible project: `inventory/`, `playbooks/` (`site.yml`, `sponsord.yml`, `backup.yml`, `decommission.yml`), `roles/` (`baseline`, `decdn_node`, `grafana_alloy`, `sponsord`, `sponsord_onramp`), `galaxy/` (the `decdn.node` collection), `molecule/`. | | [`cloud-init/`](cloud-init/README.md) | The cloud-init deploy path: `user-data.yaml`, the on-host `bootstrap.sh`, and the pinned ansible-core and collections it installs. | -| [`compose/`](compose/README.md) | The Docker Compose deploy path. | +| [`compose/`](compose/README.md) | The Docker Compose deploy path: the node, `sponsord`, `sponsord-onramp` and Caddy, one profile each. | | [`charts/decdn-node/`](charts/decdn-node/README.md) | The Helm chart, with vendored dashboards and alert rules in `files/monitoring/`. | | [`docs/`](docs/requirements.md) | Cross-path operator docs: requirements, lifecycle. | | `scripts/` | Generators for the upstream mirrors (network profiles, monitoring) and the release gate. | diff --git a/compose/.env.example b/compose/.env.example index 88849b0..4768eb0 100644 --- a/compose/.env.example +++ b/compose/.env.example @@ -1,10 +1,22 @@ # Copy to .env beside compose.yaml. Nothing here is secret. -# The node image, always pinned by digest (compose.yaml builds REPO@DIGEST). Take the -# digest from the release's signed image-digest.txt (verify its .asc against the -# maintainer key in decdn/decdn's KEYS), not from whatever a tag points at today. -# DECDN_IMAGE_REPO defaults to ghcr.io/decdn/decdn-node; override it for a mirror or a -# local registry (compose/README.md, "before a release"). +# Which services this host runs (compose.yaml's header): any of node, sponsord, +# onramp (which also starts sponsord) and caddy, comma-separated. For example +# node a deCDN node only +# sponsord the sponsor daemon only +# onramp,caddy the sponsor daemon, its public onramp and TLS in front +# node,onramp,caddy all of it on one host +COMPOSE_PROFILES=node + +# Every image is pinned by digest (compose.yaml builds REPO@DIGEST). Take a digest +# from the release's signed digest file (image-digest.txt for decdn, +# -image-digest.txt for sponsord; verify its .asc against the maintainer +# key in that repository's KEYS), not from whatever a tag points at today. The +# file holds REPO@DIGEST: copy the part after `@`. Override a REPO for a mirror or +# a local registry (README.md, "Before a release"). Set only the ones your +# profiles use; an unset one fails only the service that needs it. + +# --- node ----------------------------------------------------------------------- DECDN_IMAGE_DIGEST=sha256:0000000000000000000000000000000000000000000000000000000000000000 # DECDN_IMAGE_REPO=ghcr.io/decdn/decdn-node @@ -15,3 +27,32 @@ DECDN_GID=999 # Host path of the 0600 environment file holding DECDN_RPC_URL. DECDN_ENV_FILE=/etc/decdn/decdn.env + +# --- sponsord and onramp ---------------------------------------------------------- +SPONSORD_IMAGE_DIGEST=sha256:0000000000000000000000000000000000000000000000000000000000000000 +# SPONSORD_IMAGE_REPO=ghcr.io/decdn/sponsord +SPONSORD_ONRAMP_IMAGE_DIGEST=sha256:0000000000000000000000000000000000000000000000000000000000000000 +# SPONSORD_ONRAMP_IMAGE_REPO=ghcr.io/decdn/sponsord-onramp + +# uid/gid of the host's dedicated `sponsord` system account, which owns the +# credential files in /etc/sponsord: `id -u sponsord`, `id -g sponsord`. Both +# daemons run as it. +SPONSORD_UID=998 +SPONSORD_GID=998 + +# Host paths of the environment files (README.md lists what goes in each). +SPONSORD_SECRET_ENV_FILE=/etc/sponsord/secret.env +SPONSORD_ENV_FILE=/etc/sponsord/sponsord.env +SPONSORD_ONRAMP_ENV_FILE=/etc/sponsord/sponsord-onramp.env + +# The onramp's public host name: its public URL is https://, and the caddy +# profile requests the certificate for it. Point its DNS at this host. +SPONSORD_ONRAMP_DOMAIN=onramp.example.org + +# --- caddy ------------------------------------------------------------------------ +# uid/gid of the host's dedicated `caddy` system account, which owns /var/lib/caddy. +CADDY_UID=997 +CADDY_GID=997 +# compose.yaml pins the official image by digest; override both to change it. +# CADDY_IMAGE_REPO=docker.io/library/caddy +# CADDY_IMAGE_DIGEST=sha256:… diff --git a/compose/Caddyfile b/compose/Caddyfile new file mode 100644 index 0000000..aaccb6e --- /dev/null +++ b/compose/Caddyfile @@ -0,0 +1,26 @@ +# The public side of sponsord-onramp under Compose (the `caddy` profile), the same +# as the sponsord_onramp role's Caddyfile: TLS for SPONSORD_ONRAMP_DOMAIN (from +# .env), then a reverse proxy to the onramp on loopback. Caddy redirects http to +# https and, with no trusted_proxies, replaces any client-sent X-Forwarded-For with +# the real client address, which the onramp rate limits by. So +# ONRAMP_CLIENT_IP_HEADER=X-Forwarded-For is safe behind this file; with another +# proxy, see sponsord-onramp.env.example. +{ + # No admin API: nothing on this host can reconfigure the running proxy over + # the admin endpoint (:2019). A Caddyfile change needs a restart: + # `docker compose restart caddy`. + admin off + # HTTP/1.1 and HTTP/2 only: HTTP/3 would need udp/443, which this setup does + # not ask you to open. + servers { + protocols h1 h2 + } + # An optional ACME account contact: + # email ops@example.org +} + +{$SPONSORD_ONRAMP_DOMAIN} { + # Caddy's local CA, for staging and tests only (browsers do not trust it): + # tls internal + reverse_proxy 127.0.0.1:8080 +} diff --git a/compose/README.md b/compose/README.md index ba0dec0..a0ef053 100644 --- a/compose/README.md +++ b/compose/README.md @@ -1,8 +1,19 @@ -# deCDN node with Docker Compose +# deCDN services with Docker Compose -The same node the Ansible role deploys, on one host, without Ansible: the upstream -image (`ghcr.io/decdn/decdn-node`, amd64 and arm64) under Compose, with the same host -paths, the same start command and the same hardening as the role's systemd unit. +The same services the Ansible roles deploy, on one host, without Ansible: the upstream +images (amd64 and arm64) under Compose, with the same host paths, the same start +commands and the same hardening as the roles' systemd units. + +| Profile | Service | Image | Ansible equivalent | +|---------|---------|-------|--------------------| +| `node` | `decdn-node`, the deCDN node | `ghcr.io/decdn/decdn-node` | `decdn_node` | +| `sponsord` | `sponsord`, the onboarding sponsor: treasury signer and PaymentPool keeper | `ghcr.io/decdn/sponsord` | `sponsord` | +| `onramp` | `sponsord-onramp`, its public side (Turnstile gate, installers); also starts `sponsord` | `ghcr.io/decdn/sponsord-onramp` | `sponsord_onramp` | +| `caddy` | Caddy, TLS in front of the onramp; leave it out to bring your own proxy | `caddy` (official) | `sponsord_onramp_proxy: caddy` | + +Pick the profiles with `COMPOSE_PROFILES` in `.env`: `node` for a node, +`onramp,caddy` for a sponsor, `node,onramp,caddy` for both on one host. The sponsor +is independent of the node. Pick this path for a single machine you already run Docker on. For a fleet, or a host you want hardened from scratch (firewall, SSH, auto-patching), use the @@ -10,12 +21,18 @@ you want hardened from scratch (firewall, SSH, auto-patching), use the [Helm chart](../charts/decdn-node/README.md). [`docs/requirements.md`](../docs/requirements.md) compares the paths. -> **Upstream has not published a release yet**, so there is no signed image to pin. -> `compose.yaml` only takes an image by digest, and a locally built image has no -> digest until it is pushed somewhere. Until a release exists, see -> [Before a release](#before-a-release-a-local-image). +> **Upstream has not published a release yet** of the node or of sponsord, so there +> is no signed image to pin. `compose.yaml` only takes an image by digest, and a +> locally built image has no digest until it is pushed somewhere. Until a release +> exists, see [Before a release](#before-a-release-a-local-image). + +**Upgrading from a node-only `compose.yaml`?** Every service now sits behind a +profile, so add `COMPOSE_PROFILES=node` to your `.env`. Without it, +`sudo docker compose -f compose/compose.yaml up -d` selects no service. -## How it is laid out +## The node + +### How it is laid out | Host path | In the container | Holds | |-----------|------------------|-------| @@ -27,12 +44,12 @@ compares the paths. The layout matches the Ansible role, so the host `decdn` CLI, backups and restores ([`docs/lifecycle.md`](../docs/lifecycle.md)) work the same way on both. -**Networking.** The container uses the host network. The daemon's metrics +**Networking.** Every container uses the host network. The node's metrics (`127.0.0.1:9090`) and admin RPC (`127.0.0.1:9191`, hard-wired upstream) stay on the host's loopback, and Docker publishes no ports, so its iptables rules never open -anything past your firewall. The only public port is QUIC **udp/4433**. +anything past your firewall. The node's only public port is QUIC **udp/4433**. -## Set up +### Set up 1. **A system account and directories:** @@ -93,7 +110,7 @@ anything past your firewall. The only public port is QUIC **udp/4433**. ```bash cp compose/.env.example compose/.env - $EDITOR compose/.env # DECDN_IMAGE_DIGEST; DECDN_UID/GID = `id -u decdn` / `id -g decdn` + $EDITOR compose/.env # COMPOSE_PROFILES=node; DECDN_IMAGE_DIGEST; DECDN_UID/GID = `id -u decdn` / `id -g decdn` sudo docker compose -f compose/compose.yaml up -d curl -s 127.0.0.1:9090/metrics | head # once "node runtime ready" is in the logs decdn node health # admin RPC, from the host @@ -108,23 +125,181 @@ anything past your firewall. The only public port is QUIC **udp/4433**. which passes your endpoint as `--rpc-url`. The node serves paid traffic only after that. -## Operate +### Operate + +Run these from the repository root with `sudo`, like the set-up steps: Compose reads +the root-only env files whenever it creates a container. -- **Logs:** `sudo docker compose -f compose/compose.yaml logs -f`. To send them to the - journal like the systemd unit, switch the `logging` driver (commented in - `compose.yaml`). -- **Stop:** `docker compose stop`. It sends SIGTERM, the daemon's graceful drain, and - waits up to 300 s. Do not use `decdn node drain` here: `restart: unless-stopped` - starts the drained container again. +- **Logs:** `sudo docker compose -f compose/compose.yaml logs -f decdn-node`. To send + them to the journal like the systemd unit, switch the `x-logging` driver + (commented in `compose.yaml`). +- **Stop:** `sudo docker compose -f compose/compose.yaml stop decdn-node`. It sends + SIGTERM, the daemon's graceful drain, and waits up to 300 s. Do not use + `decdn node drain` here: `restart: unless-stopped` starts the drained container + again. - **Upgrade:** set the new release's digest as `DECDN_IMAGE_DIGEST` in `.env`, then - `docker compose up -d`. -- **Config change:** edit `/etc/decdn/node.toml`, then `docker compose restart` - (or `decdn node reload` for the hot-reloadable sections). + `sudo docker compose -f compose/compose.yaml up -d`. +- **Config change:** edit `/etc/decdn/node.toml`, then + `sudo docker compose -f compose/compose.yaml restart decdn-node` (or + `decdn node reload` for the hot-reloadable sections). - **Health:** there is no container healthcheck. The image has no HTTP client, and the metrics listener serves only `/metrics`. Probe `http://127.0.0.1:9090/metrics` from the host, or ship metrics with Grafana Alloy / Prometheus. Dashboards and alert rules are in [`charts/decdn-node/files/monitoring/`](../charts/decdn-node/files/monitoring/README.md). +## sponsord and its onramp + +`sponsord` holds the treasury wallet: it signs capped spending capabilities for new +users and keeps the treasury's PaymentPool pool topped up. `sponsord-onramp` is its +public side: the Turnstile gate, the `decdn.sh` / `decdn.ps1` installers and the API +the `decdn-sponsored` CLI polls. The onramp calls the daemon on loopback with the +daemon's own API token, so the two run on the same host (the `onramp` profile starts +both). The [`sponsord`](../ansible/roles/sponsord/README.md) and +[`sponsord_onramp`](../ansible/roles/sponsord_onramp/README.md) role READMEs explain +what each setting does; this section covers the Compose side. + +### How it is laid out + +| Host path | Owner, mode | Reaches | As | +|-----------|-------------|---------|----| +| `/etc/sponsord/secret.env` | `root`, `0600` | sponsord | env, read by `sudo docker compose`: `SPONSORD_RPC_URL` only | +| `/etc/sponsord/sponsord.env` | `root`, `0644` | sponsord | env: chain, PaymentPool, pool id, limits | +| `/etc/sponsord/sponsord-onramp.env` | `root`, `0644` | onramp | env: public settings | +| `/etc/sponsord/api-token` | `sponsord`, `0600` | both | read-only file in `/run/secrets/` | +| `/etc/sponsord/treasury-keystore.json` | `sponsord`, `0600` | sponsord | read-only file in `/run/secrets/` | +| `/etc/sponsord/treasury-password` | `sponsord`, `0600` | sponsord | read-only file in `/run/secrets/` | +| `/etc/sponsord/turnstile-secret` | `sponsord`, `0600` | onramp | read-only file in `/run/secrets/` | +| `/var/lib/caddy/` | `caddy`, `0700` | Caddy | `/data`: certificates, ACME account | + +The paths match the Ansible roles, with one difference: the credential files belong +to a `sponsord` account instead of root. Under systemd, root hands them to the +service; here they are bind-mounted, which keeps the host file's owner and mode, and +sponsord refuses a treasury keystore that anyone but its owner can read. Both daemons +run as that account, and each container sees only the files it needs. A host moved +from the Ansible path needs a `chown` of those four files, and running the Ansible +`sponsord` role on the host again chowns them back to root, which stops this path. + +`compose.yaml` sets both listeners (`127.0.0.1:8090` for sponsord, `127.0.0.1:8080` +for the onramp), the onramp's daemon URL and every secret file path in +`environment:`, which wins over the env files, so no env file can change them; the +release images default both listeners to `0.0.0.0`. No container publishes a port: +sponsord is reachable only from the host, and from outside, the onramp only through +the proxy. + +### Set up + +1. **System accounts and directories:** + + ```bash + sudo useradd --system --no-create-home --shell /usr/sbin/nologin sponsord + sudo install -d -m 0755 -o root -g root /etc/sponsord + # with the caddy profile: + sudo useradd --system --no-create-home --shell /usr/sbin/nologin caddy + sudo install -d -m 0700 -o caddy -g caddy /var/lib/caddy + ``` + +2. **The treasury wallet.** Create it with the decdn CLI (`decdn key-gen`, which + writes `keystore.json`), fund it with USDC plus gas, then open its pool from it + (`decdn pool open`, which prints the pool id). It is a hot key: hold only a few + top-ups' worth. Then copy the keystore and its password onto the host: + + ```bash + sudo install -m 0600 -o sponsord -g sponsord keystore.json /etc/sponsord/treasury-keystore.json + sudo install -m 0600 -o sponsord -g sponsord treasury-password /etc/sponsord/treasury-password + ``` + +3. **The API token**, generated on the host and never replaced (the onramp presents + the same token): + + ```bash + openssl rand -hex 32 | sudo install -m 0600 -o sponsord -g sponsord /dev/stdin /etc/sponsord/api-token + ``` + +4. **sponsord's settings.** The RPC URL may embed an API key, so it gets its own + root-only file; the rest is not secret. Take the chain id and PaymentPool address + from the deployment this repo mirrors, + [`roles/sponsord/vars/main/networks.yml`](../ansible/roles/sponsord/vars/main/networks.yml). + + ```bash + sudo install -m 0600 -o root -g root compose/sponsord-secret.env.example /etc/sponsord/secret.env + sudoedit /etc/sponsord/secret.env # SPONSORD_RPC_URL='https://…' + sudo install -m 0644 -o root -g root compose/sponsord.env.example /etc/sponsord/sponsord.env + sudoedit /etc/sponsord/sponsord.env # SPONSORD_CHAIN_ID, SPONSORD_PAYMENT_POOL_ADDR, SPONSORD_POOL_ID + ``` + +5. **The onramp** (`onramp` profile). Create a Cloudflare Turnstile widget for the + onramp's domain, then store its secret and fill in the public settings. The + CapacityBond address is in + [`roles/sponsord_onramp/vars/main/networks.yml`](../ansible/roles/sponsord_onramp/vars/main/networks.yml). + + ```bash + sudo install -m 0600 -o sponsord -g sponsord /dev/null /etc/sponsord/turnstile-secret + sudoedit /etc/sponsord/turnstile-secret # the widget's secret key + sudo install -m 0644 -o root -g root compose/sponsord-onramp.env.example /etc/sponsord/sponsord-onramp.env + sudoedit /etc/sponsord/sponsord-onramp.env + ``` + + `sudoedit` writes the file back with its existing owner and mode. + +6. **TLS in front of the onramp.** Either: + - **Caddy** (`caddy` profile): point the domain's DNS at this host and open + **tcp/80** and **tcp/443** in the host firewall and the cloud security group. + Caddy gets the certificate from Let's Encrypt, redirects http to https and + proxies to `127.0.0.1:8080`. Its config is [`Caddyfile`](Caddyfile): no admin + API, no HTTP/3 (so no udp/443), and an optional ACME contact email. Uncomment + `ONRAMP_CLIENT_IP_HEADER=X-Forwarded-For` in `sponsord-onramp.env`, so the onramp + rate limits each client instead of Caddy as a whole. + - **Your own proxy**: leave `caddy` out of `COMPOSE_PROFILES` and proxy the domain + to `127.0.0.1:8080`. Then decide on `ONRAMP_CLIENT_IP_HEADER` in + `sponsord-onramp.env`. The onramp rate limits by that header's right-most + address, so set it only if your proxy is the **only** way in and overwrites + that header (e.g. `CF-Connecting-IP` behind Cloudflare). Unset, the onramp + limits by the TCP peer, which behind a proxy is the proxy itself. + +7. **Start it:** + + ```bash + cp compose/.env.example compose/.env # unless the node already uses one + $EDITOR compose/.env # COMPOSE_PROFILES=onramp,caddy; SPONSORD_*_IMAGE_DIGEST; + # SPONSORD_UID/GID, CADDY_UID/GID (`id -u sponsord` …); + # SPONSORD_ONRAMP_DOMAIN + sudo docker compose -f compose/compose.yaml up -d + curl -s 127.0.0.1:8090/healthz # {"ok":true} + curl -s 127.0.0.1:8080/healthz # {"ok":true} + curl -s https:///healthz # through Caddy + ``` + + sponsord binds only after it has decrypted the keystore and confirmed on-chain + that the treasury owns `SPONSORD_POOL_ID`. Until then it exits and restarts, and + its log says why. The onramp exits while the daemon is unreachable, so it restarts + until sponsord is up. + +### Operate + +As for the node, run these from the repository root with `sudo`. + +- **Logs:** `sudo docker compose -f compose/compose.yaml logs -f sponsord sponsord-onramp caddy`. +- **Stop:** `sudo docker compose -f compose/compose.yaml stop caddy sponsord-onramp sponsord` + (leave out `caddy` without that profile). On SIGTERM sponsord waits for a pool + top-up it already sent, up to 120 s. Stopping sponsord alone leaves the onramp + running, and stopping both leaves Caddy on tcp/80 and tcp/443 answering with + errors. +- **Restart:** `sudo docker compose -f compose/compose.yaml restart sponsord` restarts + the onramp too, because the onramp reads the daemon's limits only at start. +- **Config change:** edit the env file, then + `sudo docker compose -f compose/compose.yaml up -d`, which recreates a container + whose env changed. A changed credential file needs + `sudo docker compose -f compose/compose.yaml restart sponsord` (the onramp restarts + with it), or `restart sponsord-onramp` for the Turnstile secret alone. +- **Caddyfile change:** `sudo docker compose -f compose/compose.yaml restart caddy`. + The admin API is off, so there is no live reload. +- **Rotate the treasury:** replace the keystore and password files (same owner and + mode), then `sudo docker compose -f compose/compose.yaml restart sponsord`. The new + wallet must own `SPONSORD_POOL_ID`, or sponsord exits at start. +- **Health:** no container healthchecks, as for the node. Probe the two `/healthz` + endpoints above from the host. sponsord also serves `/metrics` on the same port, and + logs errors only unless `RUST_LOG=info` is set in `sponsord.env`. + ## Before a release: a local image Build the daemon image from a `decdn/decdn` checkout (its `Dockerfile` header shows @@ -147,16 +322,46 @@ The upstream image is based on Debian bookworm (glibc 2.36). A `decdn-node` buil natively on a newer distribution will not start in it; build with `cross`, as the upstream release does, or on bookworm. +The sponsord images work the same way. `decdn/sponsord`'s `deploy/Dockerfile` builds +either one from source: + +```bash +sudo docker build -f /deploy/Dockerfile --build-arg BIN=sponsord \ + -t 127.0.0.1:5000/sponsord:dev +sudo docker build -f /deploy/Dockerfile --build-arg BIN=sponsord-onramp \ + -t 127.0.0.1:5000/sponsord-onramp:dev +sudo docker push 127.0.0.1:5000/sponsord:dev && sudo docker push 127.0.0.1:5000/sponsord-onramp:dev +``` + +Then set `SPONSORD_IMAGE_REPO` / `SPONSORD_IMAGE_DIGEST` and +`SPONSORD_ONRAMP_IMAGE_REPO` / `SPONSORD_ONRAMP_IMAGE_DIGEST` in `.env`. + ## Security notes -- The container runs as the host's `decdn` account with a read-only root filesystem, - every capability dropped and `no-new-privileges`. -- The two remaining KICS findings are the design, not an oversight: host networking - (loopback-only metrics and admin, no Docker-published ports) and no healthcheck (see - above). The read-only `/etc/decdn` mount is excluded from the scan for the reason - given in the root `Makefile`. -- The image is always referenced by digest: `compose.yaml` builds - `DECDN_IMAGE_REPO@DECDN_IMAGE_DIGEST` itself, so no `.env` value can turn it into a - mutable tag. -- `make lint-compose` (CI job `compose`) renders this file and fails if any of those - properties regress; `make test-scripts` checks it rejects broken variants. +- Every container runs as a dedicated host account (`decdn`, `sponsord`, `caddy`) with + a read-only root filesystem, every capability dropped and `no-new-privileges`. + Caddy keeps `NET_BIND_SERVICE` and nothing else, for tcp/80 and tcp/443. +- Secret files are mounted read-only and one by one, so a container sees only its + own. The secrets that are environment variables (`DECDN_RPC_URL`, + `SPONSORD_RPC_URL`) live in root-only files on disk; Compose copies them into the + container's config when it creates it. +- The remaining KICS findings are the design, not an oversight: host networking + (loopback-only backends, no Docker-published ports), no healthchecks (see above), + Caddy's one added capability, and the API token both sponsord containers mount. + The "Volume Has Sensitive Host Directory" query, which flags every host-path mount + (the roles' host layout), is excluded for the reason given in the root `Makefile`. +- Every image is referenced by digest: `compose.yaml` builds `REPO@DIGEST` itself, + so no `.env` value can turn it into a mutable tag. +- Compose cannot require a variable without breaking the profiles that do not use + it, so an unset one renders a value its service refuses: an invalid image + reference, an unknown user, an onramp domain with a non-numeric port. That service + fails to start; the others are unaffected. +- No secret is written in this file: each service may set only its listener, + secret-file paths and public URL inline, and everything else (`DECDN_RPC_URL`, + `SPONSORD_RPC_URL`, …) comes from its env file on the host. +- `make lint-compose` (CI job `compose`) renders this file with every profile on: + with the example `.env`, without the env files, and with an empty `.env`. It fails + if any of those properties regress ([`tests/invariants.jq`](tests/invariants.jq), + [`tests/inline-env.jq`](tests/inline-env.jq), + [`tests/fail-closed.jq`](tests/fail-closed.jq)); `make test-scripts` checks it + rejects broken variants. diff --git a/compose/compose.yaml b/compose/compose.yaml index ed11e4a..d2b82bd 100644 --- a/compose/compose.yaml +++ b/compose/compose.yaml @@ -1,27 +1,49 @@ -# deCDN node under Docker Compose: the same daemon, paths and hardening as the -# Ansible role's systemd unit, for a single host without Ansible. See README.md. +# deCDN services under Docker Compose: the same daemons, paths and hardening as the +# Ansible roles' systemd units, for a single host without Ansible. See README.md. # -# cp .env.example .env && $EDITOR .env # image digest + the decdn user's uid/gid +# cp .env.example .env && $EDITOR .env # COMPOSE_PROFILES, image digests, uids/gids # sudo docker compose up -d # -# Host networking, deliberately. The daemon binds metrics to 127.0.0.1 by default -# and its admin RPC is hard-wired to 127.0.0.1, so on the host network both stay -# loopback-only, exactly as on a systemd host, and the host's `decdn` CLI reaches -# the admin RPC directly. There are no `ports:` for Docker to publish, which also -# means Docker's own iptables rules never punch a hole past the host firewall. The -# one public port is the node's QUIC udp/4433: open it in the host firewall. +# Each service sits behind a profile, chosen with COMPOSE_PROFILES in .env: +# node the deCDN node +# sponsord the onboarding sponsor daemon (treasury signer + PaymentPool keeper) +# onramp sponsord-onramp, its public side; also starts sponsord, which it needs +# caddy TLS in front of the onramp (leave it out to bring your own proxy) +# +# Compose interpolates every service, enabled or not, so no variable here may be +# required with `:?`: a host without the node would then need the node's digest. +# An unset value instead renders a value that names the missing variable and that +# the service refuses: an invalid image reference, an unknown user, a host name +# with a non-numeric port. `make lint-compose` renders an empty .env to check it. +# +# Host networking, deliberately. Every backend binds 127.0.0.1, so on the host +# network it stays loopback-only, exactly as on a systemd host, and host tools (the +# `decdn` CLI, curl to /healthz) reach it directly. There are no `ports:` for Docker +# to publish, which also means Docker's own iptables rules never punch a hole past +# the host firewall. The public ports are the node's QUIC udp/4433 and, with the +# caddy profile, tcp/80 + tcp/443: open them in the host firewall. name: decdn +x-logging: &logging + driver: json-file + options: + max-size: 50m + max-file: "5" + # Or send the daemons' logs to the host journal, like the systemd units: + # driver: journald + # options: {tag: "{{.Name}}"} + services: decdn-node: + profiles: [node] # Always by digest, structurally: the reference is built here as repo@digest, so # no .env can swap in a mutable tag. The release's signed image-digest.txt names # the digest; upstream only tags an image after a maintainer signs the release. - image: ${DECDN_IMAGE_REPO:-ghcr.io/decdn/decdn-node}@${DECDN_IMAGE_DIGEST:?set DECDN_IMAGE_DIGEST in .env to sha256:} + image: ${DECDN_IMAGE_REPO:-ghcr.io/decdn/decdn-node}@${DECDN_IMAGE_DIGEST:-sha256:unset-set-DECDN_IMAGE_DIGEST-in-.env} network_mode: host # A dedicated host account (see README.md), not the image's uid 1000, which on # most hosts is the first human login. - user: "${DECDN_UID:?set DECDN_UID in .env}:${DECDN_GID:?set DECDN_GID in .env}" + user: "${DECDN_UID:-unset-DECDN_UID}:${DECDN_GID:-unset-DECDN_GID}" # Mirrors the systemd unit's ExecStart. command: - run @@ -49,13 +71,145 @@ services: stop_signal: SIGTERM stop_grace_period: 300s restart: unless-stopped - logging: - driver: json-file - options: - max-size: 50m - max-file: "5" - # Or send the daemon's JSON logs to the host journal, like the systemd unit: - # driver: journald - # options: {tag: decdn-node} + logging: *logging # No healthcheck: the image has no HTTP client, and the metrics listener only # serves /metrics. Probe http://127.0.0.1:9090/metrics from the host instead. + + sponsord: + profiles: [sponsord, onramp] + image: ${SPONSORD_IMAGE_REPO:-ghcr.io/decdn/sponsord}@${SPONSORD_IMAGE_DIGEST:-sha256:unset-set-SPONSORD_IMAGE_DIGEST-in-.env} + network_mode: host + # The host's `sponsord` account, which owns the credential files below: a bind + # mount keeps the host file's owner and mode, and the daemon refuses a treasury + # keystore that anyone but its owner can read. + user: "${SPONSORD_UID:-unset-SPONSORD_UID}:${SPONSORD_GID:-unset-SPONSORD_GID}" + # Read by the `docker compose` client when it creates the container, so run it + # as root (sudo): secret.env (0600) holds SPONSORD_RPC_URL, which may embed an + # API key; sponsord.env (0644) is the pool and limit settings. The later file + # wins on a repeated key, so a stray SPONSORD_RPC_URL in the world-readable + # file cannot replace the secret one. + env_file: + - path: ${SPONSORD_ENV_FILE:-/etc/sponsord/sponsord.env} + required: true + - path: ${SPONSORD_SECRET_ENV_FILE:-/etc/sponsord/secret.env} + required: true + # In `environment:`, which wins over env_file, so no env file can move this + # API, which signs spending capabilities, off loopback (the release image + # defaults to 0.0.0.0) or swap a secret file for an inline value. + environment: + SPONSORD_BIND: 127.0.0.1:8090 + SPONSORD_API_TOKEN_FILE: /run/secrets/api-token + SPONSORD_TREASURY_PASSWORD_FILE: /run/secrets/treasury-password + SPONSORD_TREASURY_KEYSTORE: /run/secrets/treasury-keystore.json + NO_COLOR: "1" + # One file each, read-only, and only the ones this daemon needs. A missing host + # file fails the start instead of Docker creating a directory in its place. + volumes: + - type: bind + source: /etc/sponsord/api-token + target: /run/secrets/api-token + read_only: true + bind: {create_host_path: false} + - type: bind + source: /etc/sponsord/treasury-password + target: /run/secrets/treasury-password + read_only: true + bind: {create_host_path: false} + - type: bind + source: /etc/sponsord/treasury-keystore.json + target: /run/secrets/treasury-keystore.json + read_only: true + bind: {create_host_path: false} + read_only: true + cap_drop: [ALL] + security_opt: + - no-new-privileges:true + # On SIGTERM the daemon waits for a pool top-up it already sent; this matches + # the unit's TimeoutStopSec. + stop_signal: SIGTERM + stop_grace_period: 120s + restart: unless-stopped + logging: *logging + # No healthcheck: the image has no HTTP client. Probe + # http://127.0.0.1:8090/healthz from the host. + + sponsord-onramp: + profiles: [onramp] + image: ${SPONSORD_ONRAMP_IMAGE_REPO:-ghcr.io/decdn/sponsord-onramp}@${SPONSORD_ONRAMP_IMAGE_DIGEST:-sha256:unset-set-SPONSORD_ONRAMP_IMAGE_DIGEST-in-.env} + network_mode: host + # The same account as sponsord: it reads the daemon's own API token. + user: "${SPONSORD_UID:-unset-SPONSORD_UID}:${SPONSORD_GID:-unset-SPONSORD_GID}" + # Public settings only: everything in it (the RPC URL, contract addresses) is + # served to every user. + env_file: + - path: ${SPONSORD_ONRAMP_ENV_FILE:-/etc/sponsord/sponsord-onramp.env} + required: true + environment: + ONRAMP_BIND: 127.0.0.1:8080 + ONRAMP_DAEMON_URL: http://127.0.0.1:8090 + ONRAMP_PUBLIC_URL: https://${SPONSORD_ONRAMP_DOMAIN:-unset-SPONSORD_ONRAMP_DOMAIN:set-in-.env} + ONRAMP_DAEMON_TOKEN_FILE: /run/secrets/api-token + ONRAMP_TURNSTILE_SECRET_FILE: /run/secrets/turnstile-secret + NO_COLOR: "1" + volumes: + - type: bind + source: /etc/sponsord/api-token + target: /run/secrets/api-token + read_only: true + bind: {create_host_path: false} + - type: bind + source: /etc/sponsord/turnstile-secret + target: /run/secrets/turnstile-secret + read_only: true + bind: {create_host_path: false} + # It reads the daemon's limits only at start, so `docker compose restart + # sponsord` restarts it too, like the unit's PartOf= (a stop does not + # propagate). It exits while the daemon is unreachable; the restart policy + # retries until the daemon is up. + depends_on: + sponsord: + condition: service_started + restart: true + read_only: true + cap_drop: [ALL] + security_opt: + - no-new-privileges:true + stop_signal: SIGTERM + stop_grace_period: 30s + restart: unless-stopped + logging: *logging + # No healthcheck, as above. Probe http://127.0.0.1:8080/healthz from the host. + + caddy: + profiles: [caddy] + # The official image, pinned by its multi-arch index digest (2.11.6). + image: ${CADDY_IMAGE_REPO:-docker.io/library/caddy}@${CADDY_IMAGE_DIGEST:-sha256:3422ce6de165df66534f9b9ba50efaf457114ec961763cc52f5dbdaac2972d73} + network_mode: host + # A dedicated host account that owns /var/lib/caddy. NET_BIND_SERVICE is the + # one capability it keeps, for tcp/80 and tcp/443. + user: "${CADDY_UID:-unset-CADDY_UID}:${CADDY_GID:-unset-CADDY_GID}" + environment: + SPONSORD_ONRAMP_DOMAIN: ${SPONSORD_ONRAMP_DOMAIN:-unset-SPONSORD_ONRAMP_DOMAIN:set-in-.env} + volumes: + - type: bind + source: ./Caddyfile + target: /etc/caddy/Caddyfile + read_only: true + bind: {create_host_path: false} + # Certificates and the ACME account, kept across restarts so a restart does + # not spend Let's Encrypt's rate limit. + - type: bind + source: /var/lib/caddy + target: /data + bind: {create_host_path: false} + # Caddy autosaves the config it loaded here; nothing reads it back. + tmpfs: + - /config:uid=${CADDY_UID:-unset-CADDY_UID},gid=${CADDY_GID:-unset-CADDY_GID},mode=0700 + read_only: true + cap_drop: [ALL] + cap_add: [NET_BIND_SERVICE] + security_opt: + - no-new-privileges:true + stop_signal: SIGTERM + restart: unless-stopped + logging: *logging diff --git a/compose/decdn.env.example b/compose/decdn.env.example index a5cec2a..f1781b9 100644 --- a/compose/decdn.env.example +++ b/compose/decdn.env.example @@ -1,6 +1,7 @@ # Template for /etc/decdn/decdn.env (root:root 0600). Docker reads it as root before # starting the container, so it never needs to be readable by the decdn account. -# One KEY=value per line, no quotes, no `export`. +# One KEY=value per line, no `export`. Compose expands `$VAR` in unquoted and +# double-quoted values, so single-quote a value containing `$`. # # The RPC endpoint. It may embed a provider API key, which is why it lives here and # not in node.toml. Overrides blockchain.rpc_url. diff --git a/compose/sponsord-onramp.env.example b/compose/sponsord-onramp.env.example new file mode 100644 index 0000000..c36b899 --- /dev/null +++ b/compose/sponsord-onramp.env.example @@ -0,0 +1,49 @@ +# Template for /etc/sponsord/sponsord-onramp.env (root:root 0644). No secrets, and +# nothing here may become one: everything in it is served to every user. +# compose.yaml sets the listener (loopback), the daemon URL, the public URL (from +# SPONSORD_ONRAMP_DOMAIN in .env) and the paths of the API token and Turnstile +# secret, and those win over this file. + +# --- Required ------------------------------------------------------------------- +# The public RPC endpoint end users' decdn reads the chain through. No API key, no +# credentials, no query string. +ONRAMP_RPC_URL= +# CapacityBond, for node discovery. Take it from the deployment this repo mirrors, +# ansible/roles/sponsord_onramp/vars/main/networks.yml (capacity_bond_address), on +# the same chain as sponsord. The chain id and PaymentPool come from the daemon. +ONRAMP_CAPACITY_BOND_ADDR= +# Optional, from the same file (slash_judge_address). +# ONRAMP_SLASH_JUDGE_ADDR= +# Cloudflare Turnstile: the widget's public sitekey. Its secret goes in +# /etc/sponsord/turnstile-secret. +ONRAMP_GATE=turnstile +ONRAMP_TURNSTILE_SITEKEY= +# The releases the installers install: a tag and the SHA-256 of that release's +# SHA256SUMS file, for decdn (vX.Y.Z) and for decdn-sponsored +# (decdn-sponsored-vX.Y.Z; both values are in its release notes). +ONRAMP_DECDN_RELEASE= +ONRAMP_DECDN_SUMS_SHA256= +ONRAMP_CLI_RELEASE= +ONRAMP_CLI_SUMS_SHA256= + +# --- Client address --------------------------------------------------------------- +# The header the proxy puts the client address in; the onramp rate limits by it. +# Unset = the TCP peer, which behind any proxy is the proxy itself, so every user +# shares one limit. With the caddy profile, uncomment this line: Caddy replaces +# any X-Forwarded-For the client sent. With your own proxy, set it only if that +# proxy is the ONLY way in and overwrites the header (e.g. CF-Connecting-IP +# behind Cloudflare); otherwise clients pick their own address. +# ONRAMP_CLIENT_IP_HEADER=X-Forwarded-For + +# --- Optional (defaults shown) ---------------------------------------------------- +# Oldest decdn-sponsored accepted; older CLIs are told to re-run the installer. +# ONRAMP_MIN_CLI_VERSION= +# Requested per capability; unset takes the daemon's maximum +# ONRAMP_SPENDING_CAP_MICRO_USDC= +# ONRAMP_TTL_SECS= +# ONRAMP_BRAND_NAME=deCDN +# POST /v1/fund and GET /v1/capability per address per minute; 0 turns a limit off +# ONRAMP_FUND_RATE_PER_MIN=10 +# ONRAMP_POLL_RATE_PER_MIN=120 +# ONRAMP_RELEASES_BASE=https://github.com/decdn +# RUST_LOG=info diff --git a/compose/sponsord-secret.env.example b/compose/sponsord-secret.env.example new file mode 100644 index 0000000..234a1d2 --- /dev/null +++ b/compose/sponsord-secret.env.example @@ -0,0 +1,13 @@ +# Template for /etc/sponsord/secret.env (root:root 0600). `sudo docker compose up` +# reads it when it creates the container, so the sponsord account never needs to. +# It holds SPONSORD_RPC_URL and nothing else: compose.yaml loads it last, so any +# other key here would override sponsord.env. +# +# FORMAT: Compose's env-file parser, not a shell. One KEY=value per line. Compose +# expands `$VAR` in unquoted and double-quoted values, so single-quote a value +# containing `$`. systemd strips single quotes too and rejects `export`, so leave +# that out and the same file works under the Ansible role. +# +# The RPC endpoint the daemon signs and sends pool top-ups through. It may embed a +# provider API key, which is why it lives here and not in sponsord.env. +SPONSORD_RPC_URL= diff --git a/compose/sponsord.env.example b/compose/sponsord.env.example new file mode 100644 index 0000000..341b95e --- /dev/null +++ b/compose/sponsord.env.example @@ -0,0 +1,31 @@ +# Template for /etc/sponsord/sponsord.env (root:root 0644). No secrets: the RPC +# URL is in secret.env (which compose.yaml loads after this file, so it wins), and +# the API token, treasury keystore and password are files bind-mounted by +# compose.yaml, which also sets SPONSORD_BIND (loopback). Same format as the +# Ansible role's sponsord.env, so the file moves between paths. + +# --- Required ------------------------------------------------------------------- +# The chain and its PaymentPool contract. Take both from the deployment this repo +# mirrors, ansible/roles/sponsord/vars/main/networks.yml (generated from upstream's +# manifest; for arbitrum-sepolia: chain_id and payment_pool_address). Never guess. +SPONSORD_CHAIN_ID= +SPONSORD_PAYMENT_POOL_ADDR= +# The pool this treasury owns: 0x + 64 hex, printed by `decdn pool open` run once +# from the treasury wallet. The daemon checks at start that the treasury keystore's +# address owns it on-chain and exits if not. +SPONSORD_POOL_ID= + +# --- Optional (the daemon's defaults shown) --------------------------------------- +# Economic decisions for the operator, not facts this repo carries. +# Largest cap per capability, in micro-USDC ($5) +# SPONSORD_MAX_SPENDING_CAP_MICRO_USDC=5000000 +# Longest TTL per capability, in seconds (48 h) +# SPONSORD_MAX_TTL_SECS=172800 +# Top the pool up below this, in micro-USDC ($20) +# SPONSORD_POOL_LOW_WATER_MICRO_USDC=20000000 +# Each top-up adds this, in micro-USDC ($100) +# SPONSORD_POOL_REFILL_MICRO_USDC=100000000 +# How often the keeper checks the pool, in seconds +# SPONSORD_POOL_WATCH_INTERVAL_SECS=3600 +# Unset logs errors only; info adds the listener and each top-up. +# RUST_LOG=info diff --git a/compose/tests/fail-closed.jq b/compose/tests/fail-closed.jq new file mode 100644 index 0000000..aaac015 --- /dev/null +++ b/compose/tests/fail-closed.jq @@ -0,0 +1,24 @@ +# What an unset variable renders to, checked by `make lint-compose` against +# `docker compose config --format json` with every profile on and an EMPTY .env. +# compose.yaml cannot require variables with `:?` (Compose interpolates disabled +# services too), so each default must instead be something the service refuses. +# A default that works (a real image, uid 0, a resolvable-looking host) would let +# a half-configured host start. Prints one line per violation. + +def check($svc; $what; $ok): if $ok then empty else "\($svc): unset \($what)" end; + +.services as $all +| ($all | to_entries[] | .key as $n | .value as $s + # Caddy's image is pinned in compose.yaml itself; everything else names its + # variable and is not a valid reference. + | (if $n == "caddy" then empty + else check($n; "image digest renders an invalid reference"; ($s.image // "") | test("@sha256:unset-set-[A-Z_]+-in-\\.env$")) + end), + check($n; "uid/gid render unknown users"; ($s.user // "") | test("^unset-[A-Z_]+:unset-[A-Z_]+$"))), + + # A host name with a non-numeric port: the onramp refuses ONRAMP_PUBLIC_URL + # ("invalid port number") and Caddy refuses the site address. + check("sponsord-onramp"; "domain renders an unparsable ONRAMP_PUBLIC_URL"; + ($all["sponsord-onramp"].environment.ONRAMP_PUBLIC_URL // "") | test("^https://unset-[A-Z_]+:[^0-9]")), + check("caddy"; "domain renders an unparsable site address"; + ($all.caddy.environment.SPONSORD_ONRAMP_DOMAIN // "") | test("^unset-[A-Z_]+:[^0-9]")) diff --git a/compose/tests/inline-env.jq b/compose/tests/inline-env.jq new file mode 100644 index 0000000..d7685d7 --- /dev/null +++ b/compose/tests/inline-env.jq @@ -0,0 +1,19 @@ +# The environment written in compose.yaml itself, checked by `make lint-compose` +# against `docker compose config --no-env-resolution --format json`, which leaves the +# env files out. The other renders cannot tell an inline value from one in an env +# file, so this is what keeps a secret (DECDN_RPC_URL, SPONSORD_RPC_URL, …) out of +# the tracked file: each service may set only these keys inline; everything else +# belongs in its env file on the host. Prints one line per violation. + +def allowed: { + "decdn-node": [], + "sponsord": ["NO_COLOR", "SPONSORD_API_TOKEN_FILE", "SPONSORD_BIND", + "SPONSORD_TREASURY_KEYSTORE", "SPONSORD_TREASURY_PASSWORD_FILE"], + "sponsord-onramp": ["NO_COLOR", "ONRAMP_BIND", "ONRAMP_DAEMON_TOKEN_FILE", "ONRAMP_DAEMON_URL", + "ONRAMP_PUBLIC_URL", "ONRAMP_TURNSTILE_SECRET_FILE"], + "caddy": ["SPONSORD_ONRAMP_DOMAIN"] +}; + +.services | to_entries[] | .key as $n +| ((.value.environment // {}) | keys) - (allowed[$n] // []) | select(length > 0) +| "\($n): sets only allowed environment keys inline (extra: \(join(", ")))" diff --git a/compose/tests/invariants.jq b/compose/tests/invariants.jq new file mode 100644 index 0000000..dd8959c --- /dev/null +++ b/compose/tests/invariants.jq @@ -0,0 +1,103 @@ +# The invariants compose/README.md promises, checked by `make lint-compose` against +# `docker compose config --format json` with every profile on and the example +# .env. Prints one ": " line per violation; no output means +# every invariant holds. + +def check($svc; $what; $ok): if $ok then empty else "\($svc): \($what)" end; +def loopback: test("^(http://)?127(\\.[0-9]{1,3}){3}:[0-9]+$"); + +# The keys each service may set, as Compose renders them. Anything else +# (privileged, pid/ipc/userns_mode, devices, ports, cap_add on a daemon, …) is a +# violation until it is added here deliberately. +def base_keys: ["cap_drop", "command", "entrypoint", "environment", "image", "logging", + "network_mode", "profiles", "read_only", "restart", "security_opt", "stop_signal", + "user", "volumes"]; +def allowed_keys: { + "decdn-node": (base_keys + ["stop_grace_period", "tmpfs"]), + "sponsord": (base_keys + ["stop_grace_period"]), + "sponsord-onramp": (base_keys + ["stop_grace_period", "depends_on"]), + "caddy": (base_keys + ["cap_add", "tmpfs"]) +}; + +# Every mount, exactly: [source, target, read-only?]. Each container sees only its +# own files; the only writable ones are the node's data dir and Caddy's ACME state. +# The Caddyfile's source is absolute (compose.yaml's directory), so it is matched +# by name. +def allowed_mounts: { + "decdn-node": [["/etc/decdn", "/etc/decdn", true], ["/var/lib/decdn", "/var/lib/decdn", false]], + "sponsord": [["/etc/sponsord/api-token", "/run/secrets/api-token", true], + ["/etc/sponsord/treasury-password", "/run/secrets/treasury-password", true], + ["/etc/sponsord/treasury-keystore.json", "/run/secrets/treasury-keystore.json", true]], + "sponsord-onramp": [["/etc/sponsord/api-token", "/run/secrets/api-token", true], + ["/etc/sponsord/turnstile-secret", "/run/secrets/turnstile-secret", true]], + "caddy": [["Caddyfile", "/etc/caddy/Caddyfile", true], ["/var/lib/caddy", "/data", false]] +}; +def mounts: [(.volumes // [])[] + | [(if .type == "bind" then .source else "\(.type):\(.source)" end + | if endswith("/Caddyfile") then "Caddyfile" else . end), + .target, (.read_only == true)]] | sort; + +# Secrets reach the sponsord daemons only as the files mounted above, never inline. +def secret_files: { + "sponsord": {"SPONSORD_API_TOKEN_FILE": "/run/secrets/api-token", + "SPONSORD_TREASURY_PASSWORD_FILE": "/run/secrets/treasury-password", + "SPONSORD_TREASURY_KEYSTORE": "/run/secrets/treasury-keystore.json"}, + "sponsord-onramp": {"ONRAMP_DAEMON_TOKEN_FILE": "/run/secrets/api-token", + "ONRAMP_TURNSTILE_SECRET_FILE": "/run/secrets/turnstile-secret"} +}; +def inline_secrets: { + "sponsord": ["SPONSORD_API_TOKEN", "SPONSORD_TREASURY_PASSWORD"], + "sponsord-onramp": ["ONRAMP_DAEMON_TOKEN", "ONRAMP_TURNSTILE_SECRET"] +}; + +.services as $all +| (["caddy", "decdn-node", "sponsord", "sponsord-onramp"] as $want + | check("compose.yaml"; "services are exactly \($want | join(", "))"; ($all | keys) == $want)), + + # Every service: host networking and nothing published, so loopback listeners + # stay loopback and Docker's iptables rules open nothing; an image by digest; a + # read-only rootfs; every capability dropped; no-new-privileges and nothing + # else in security_opt (no seccomp/apparmor opt-out); a non-root account. + ($all | to_entries[] | .key as $n | .value as $s + | check($n; "sets only allowed keys (extra: \(($s | keys) - (allowed_keys[$n] // []) | join(", ")))"; + (($s | keys) - (allowed_keys[$n] // [])) == []), + check($n; "network_mode is host"; $s.network_mode == "host"), + check($n; "publishes no ports"; $s.ports == null), + check($n; "image is pinned by @sha256 digest"; ($s.image // "") | test("@sha256:[0-9a-f]{64}$")), + check($n; "read_only rootfs"; $s.read_only == true), + check($n; "cap_drop is [ALL]"; $s.cap_drop == ["ALL"]), + check($n; "security_opt is exactly [no-new-privileges:true]"; $s.security_opt == ["no-new-privileges:true"]), + check($n; "runs as a non-root uid:gid"; ($s.user // "") | test("^[1-9][0-9]*:[1-9][0-9]*$")), + check($n; "stop_signal is SIGTERM"; $s.stop_signal == "SIGTERM"), + check($n; "mounts are exactly its allow-list"; ($s | mounts) == (allowed_mounts[$n] // [] | sort))), + + # Caddy keeps only the capability for :80/:443 (the daemons may not set cap_add + # at all: allowed_keys). + check("caddy"; "cap_add is exactly [NET_BIND_SERVICE]"; $all.caddy.cap_add == ["NET_BIND_SERVICE"]), + + # Stop grace long enough for each daemon's drain (the units' TimeoutStopSec). + check("decdn-node"; "stop_grace_period is 300s"; $all["decdn-node"].stop_grace_period == "5m0s"), + check("sponsord"; "stop_grace_period is 120s"; $all.sponsord.stop_grace_period == "2m0s"), + check("sponsord-onramp"; "stop_grace_period is 30s"; $all["sponsord-onramp"].stop_grace_period == "30s"), + + # The sponsord daemons run the image's entrypoint with no flags: a flag beats the + # environment, so `--bind 0.0.0.0:…` would get past the loopback checks below. + (["sponsord", "sponsord-onramp"][] as $n + | check($n; "no command or entrypoint override"; $all[$n].command == null and $all[$n].entrypoint == null)), + + # The sponsor's listeners stay on loopback whatever an env file says (the release + # images default to 0.0.0.0); the rendered environment has env_file merged in. + check("sponsord"; "SPONSORD_BIND is 127.x"; ($all.sponsord.environment.SPONSORD_BIND // "") | loopback), + check("sponsord-onramp"; "ONRAMP_BIND is 127.x"; ($all["sponsord-onramp"].environment.ONRAMP_BIND // "") | loopback), + check("sponsord-onramp"; "ONRAMP_DAEMON_URL is http://127.x"; ($all["sponsord-onramp"].environment.ONRAMP_DAEMON_URL // "") | loopback), + + (secret_files | to_entries[] | .key as $n | .value | to_entries[] + | check($n; "\(.key) is \(.value)"; ($all[$n].environment // {})[.key] == .value)), + (inline_secrets | to_entries[] | .key as $n | .value[] as $k + | check($n; "no inline \($k)"; ($all[$n].environment // {}) | has($k) | not)), + + # Profiles: the onramp needs its daemon on the same host, so `onramp` starts both. + check("decdn-node"; "profiles are [node]"; $all["decdn-node"].profiles == ["node"]), + check("sponsord"; "profiles are [sponsord, onramp]"; $all.sponsord.profiles == ["sponsord", "onramp"]), + check("sponsord-onramp"; "profiles are [onramp]"; $all["sponsord-onramp"].profiles == ["onramp"]), + check("caddy"; "profiles are [caddy]"; $all.caddy.profiles == ["caddy"]) diff --git a/docs/lifecycle.md b/docs/lifecycle.md index eadf088..da8f99d 100644 --- a/docs/lifecycle.md +++ b/docs/lifecycle.md @@ -215,7 +215,8 @@ sudo systemd-run --pty --wait --collect -p User=decdn \ - **Compose** ([`compose/`](../compose/README.md)) uses the same host paths (`/var/lib/decdn`, `/etc/decdn`), so the manual backup command and the restore steps - above apply unchanged; stop the node with `docker compose stop`. + above apply unchanged; stop the node with + `sudo docker compose -f compose/compose.yaml stop decdn-node`. - **Helm**: the identity lives in the operator-provisioned `existingSecret`, which you created off-cluster and should already hold elsewhere. The daemon's state is on the PVC; snapshot it with your storage's `VolumeSnapshot` support after scaling the diff --git a/docs/requirements.md b/docs/requirements.md index d9ec9e2..e68b84a 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -32,7 +32,7 @@ and they run the same roles. | Ansible (control machine) | ansible-core ≥ 2.15 | CI runs the current release | | cloud-init | the provider image's own; the bootstrap installs its pinned ansible-core on the host | the user-data is booted with the distro's cloud-init in Debian 12 and Ubuntu 26.04 containers, to a running node | | Kubernetes | ≥ 1.25 | rendered and validated with kubeconform against 1.30 | -| Docker Compose | v2 with `env_file.required` support (2.24+) | rendered in CI | +| Docker Compose | v2 with `env_file.required` and `depends_on.restart` support (2.24+) | rendered in CI with every profile on | On Ubuntu 25.10 and later, `sudo` is sudo-rs; see the `ansible_become_exe` note in `ansible/inventory/hosts.yml.example`. @@ -42,12 +42,14 @@ On Ubuntu 25.10 and later, `sudo` is sudo-rs; see the `ansible_become_exe` note - **Inbound: udp/4433 (QUIC)** from anywhere. This is the node's only public port. Open it in the host firewall (the Ansible `baseline` does this) **and** in your cloud provider's security group or firewall, which the host cannot see. -- **A sponsord-onramp host (Ansible only) also needs inbound tcp/80 and tcp/443** for - Caddy, from anywhere: 443 serves the onramp, 80 answers the ACME challenge and - redirects to https. `baseline` opens both for hosts in `sponsord_onramp_hosts` while - `sponsord_onramp_proxy` is `caddy` (the default; `none` opens neither); open them in - the provider's firewall too, and point the domain's A/AAAA records at the host. - Caddy comes from the distribution: on Ubuntu, enable the **universe** component. +- **A sponsord-onramp host also needs inbound tcp/80 and tcp/443** for Caddy, from + anywhere: 443 serves the onramp, 80 answers the ACME challenge and redirects to + https. On Ansible, `baseline` opens both for hosts in `sponsord_onramp_hosts` while + `sponsord_onramp_proxy` is `caddy` (the default; `none` opens neither), and Caddy + comes from the distribution: on Ubuntu, enable the **universe** component. On + Compose, the `caddy` profile runs the official image and you open both ports in the + host firewall. Either way, open them in the provider's firewall too, and point the + domain's A/AAAA records at the host. - **Outbound:** HTTPS to your RPC provider and to any cache origin; QUIC/UDP to peers; HTTPS to the iroh relays (upstream's defaults unless you set your own). - **NAT:** a node without direct inbound reachability still works through iroh relays, diff --git a/tests/scripts-test.sh b/tests/scripts-test.sh index f5ec4c0..7e97147 100755 --- a/tests/scripts-test.sh +++ b/tests/scripts-test.sh @@ -79,25 +79,86 @@ for section in '^## Ansible collection' '^## Helm chart'; do done pass "release notes carry both changelog sections" -# --- lint-compose negatives: each broken variant must be rejected ---------------------- +# --- lint-compose: the real file passes, each broken variant is rejected ------------- compose="$repo/compose/compose.yaml" -variant() { # - sed -E "$2" "$compose" > "$work/$1.yaml" +# The baseline first: a variant below only proves something if the unmodified file +# does not already trip the invariant it targets. +expect 0 "lint-compose accepts compose/compose.yaml" make -s -C "$repo" lint-compose +# : the fragment pins WHICH +# invariant fired (compose/tests/*.jq print ": "), since one +# edit can trip several. Scope an edit to one service by prefixing a sed range: +# "${node}", "${sd}", "${onr}" or "${cdy}". +variant() { + sed -E "$3" "$compose" > "$work/$1.yaml" cmp -s "$compose" "$work/$1.yaml" && fail "variant $1 did not change compose.yaml" # make exits 2 for any failing recipe, so tell "invariant violated" from "could # not render" by the message, not the exit code. - if make -s -C "$repo" lint-compose COMPOSE_FILE="$work/$1.yaml" >"$work/out" 2>&1; then + if make -s -C "$repo" lint-compose LINT_COMPOSE_FILE="$work/$1.yaml" >"$work/out" 2>&1; then fail "lint-compose accepted: $1" fi grep -q 'violates an invariant' "$work/out" || { cat "$work/out" >&2; fail "lint-compose failed for another reason: $1"; } + grep -qF -- "$2" "$work/out" || { cat "$work/out" >&2; fail "lint-compose rejected $1, but not with: $2"; } pass "lint-compose rejects: $1" } -variant "bridge network" 's/^(\s*)network_mode: host$/\1network_mode: bridge/' -variant "writable rootfs" 's/^(\s*)read_only: true$/\1read_only: false/' -variant "short stop grace" 's/^(\s*)stop_grace_period: 300s$/\1stop_grace_period: 10s/' -variant "capabilities kept" 's/^(\s*)cap_drop: \[ALL\]$/\1cap_drop: [NET_RAW]/' -variant "published port" 's/^(\s*)network_mode: host$/\1ports: ["127.0.0.1:9090:9090"]/' -variant "tag instead of digest" 's#^(\s*)image: .*#\1image: ghcr.io/decdn/decdn-node:latest#' +# Each range ends at the next service or top-level key. +node='/^ decdn-node:$/,/^ {0,2}[a-z]/' +sd='/^ sponsord:$/,/^ {0,2}[a-z]/' +onr='/^ sponsord-onramp:$/,/^ {0,2}[a-z]/' +cdy='/^ caddy:$/,/^ {0,2}[a-z]/' +# Shape +variant "unexpected service" "compose.yaml: services are exactly" "s/^ caddy:$/ proxy:/" +variant "sponsord privileged" "sponsord: sets only allowed keys (extra: privileged)" "${sd} s/^(\s*)cap_drop: \[ALL\]$/&\n\1privileged: true/" +variant "sponsord host pid" "sponsord: sets only allowed keys (extra: pid)" "${sd} s/^(\s*)network_mode: host$/&\n\1pid: host/" +variant "onramp device" "sponsord-onramp: sets only allowed keys (extra: devices)" "${onr} s/^(\s*)network_mode: host$/&\n\1devices: [\"\/dev\/mem:\/dev\/mem\"]/" +# decdn-node +variant "bridge network" "decdn-node: network_mode is host" "${node} s/^(\s*)network_mode: host$/\1network_mode: bridge/" +variant "writable rootfs" "decdn-node: read_only rootfs" "${node} s/^(\s*)read_only: true$/\1read_only: false/" +variant "short stop grace" "decdn-node: stop_grace_period" "${node} s/^(\s*)stop_grace_period: 300s$/\1stop_grace_period: 10s/" +variant "capabilities kept" "decdn-node: cap_drop is [ALL]" "${node} s/^(\s*)cap_drop: \[ALL\]$/\1cap_drop: [NET_RAW]/" +variant "published port" "decdn-node: publishes no ports" "${node} s/^(\s*)network_mode: host$/&\n\1ports: [\"127.0.0.1:9090:9090\"]/" +variant "tag instead of digest" "decdn-node: image is pinned" "${node} s#^(\s*)image: .*#\1image: ghcr.io/decdn/decdn-node:latest#" +variant "node config writable" "decdn-node: mounts are exactly" "${node} s#- /etc/decdn:/etc/decdn:ro\$#- /etc/decdn:/etc/decdn#" +variant "node always on" "decdn-node: profiles are [node]" "${node} {/^ profiles: \[node\]$/d}" +# sponsord +variant "sponsord on 0.0.0.0" "sponsord: SPONSORD_BIND is 127.x" "${sd} s/SPONSORD_BIND: 127\.0\.0\.1:8090/SPONSORD_BIND: 0.0.0.0:8090/" +variant "sponsord bind unset" "sponsord: SPONSORD_BIND is 127.x" "${sd} {/SPONSORD_BIND: /d}" +variant "sponsord bind flag" "sponsord: no command or entrypoint override" "${sd} s/^(\s*)network_mode: host$/&\n\1command: [--bind, \"0.0.0.0:8090\"]/" +variant "sponsord by tag" "sponsord: image is pinned" "${sd} s#^(\s*)image: .*#\1image: ghcr.io/decdn/sponsord:latest#" +variant "sponsord capability" "sponsord: sets only allowed keys (extra: cap_add)" "${sd} s/^(\s*)cap_drop: \[ALL\]$/&\n\1cap_add: [NET_RAW]/" +variant "sponsord short grace" "sponsord: stop_grace_period" "${sd} s/^(\s*)stop_grace_period: 120s$/\1stop_grace_period: 10s/" +variant "sponsord SIGKILL" "sponsord: stop_signal is SIGTERM" "${sd} s/^(\s*)stop_signal: SIGTERM$/\1stop_signal: SIGKILL/" +variant "sponsord as root" "sponsord: runs as a non-root uid:gid" "${sd} s/^(\s*)user: .*/\1user: \"0:0\"/" +variant "privilege escalation" "sponsord: security_opt is exactly" "${sd} s/- no-new-privileges:true$/- no-new-privileges:false/" +variant "seccomp unconfined" "sponsord: security_opt is exactly" "${sd} s/^(\s*)- no-new-privileges:true$/&\n\1- seccomp:unconfined/" +variant "writable keystore mount" "sponsord: mounts are exactly" "${sd} {/target: \/run\/secrets\/treasury-keystore\.json$/{n;s/read_only: true/read_only: false/}}" +variant "host root as data dir" "sponsord: mounts are exactly" "${sd} s#^(\s*)volumes:\$#\1volumes:\n\1 - /:/data#" +variant "docker socket" "sponsord: mounts are exactly" "${sd} s#^(\s*)volumes:\$#\1volumes:\n\1 - /var/run/docker.sock:/var/run/docker.sock:ro#" +variant "keystore path moved" "sponsord: SPONSORD_TREASURY_KEYSTORE is" "${sd} s#SPONSORD_TREASURY_KEYSTORE: .*#SPONSORD_TREASURY_KEYSTORE: /tmp/k.json#" +variant "inline API token" "sponsord: no inline SPONSORD_API_TOKEN" "${sd} s/^(\s*)SPONSORD_BIND: (.*)$/&\n\1SPONSORD_API_TOKEN: x/" +variant "RPC URL in compose.yaml" "sponsord: sets only allowed environment keys inline (extra: SPONSORD_RPC_URL)" "${sd} s/^(\s*)SPONSORD_BIND: (.*)$/&\n\1SPONSORD_RPC_URL: https:\/\/rpc.invalid\/key/" +variant "node RPC URL inline" "decdn-node: sets only allowed environment keys inline (extra: DECDN_RPC_URL)" "${node} s/^(\s*)read_only: true$/\1environment:\n\1 DECDN_RPC_URL: https:\/\/rpc.invalid\/key\n&/" +variant "onramp without daemon" "sponsord: profiles are" "${sd} s/^(\s*)profiles: \[sponsord, onramp\]$/\1profiles: [sponsord]/" +variant "sponsord digest default" "sponsord: unset image digest" "${sd} s#^(\s*)image: .*#\1image: ghcr.io/decdn/sponsord@\\\${SPONSORD_IMAGE_DIGEST:-sha256:$(printf '0%.0s' {1..64})}#" +variant "sponsord uid default" "sponsord: unset uid/gid" "${sd} s/^(\s*)user: .*/\1user: \"\\\${SPONSORD_UID:-998}:\\\${SPONSORD_GID:-998}\"/" +# sponsord-onramp +variant "onramp on 0.0.0.0" "sponsord-onramp: ONRAMP_BIND is 127.x" "${onr} s/ONRAMP_BIND: 127\.0\.0\.1:8080/ONRAMP_BIND: 0.0.0.0:8080/" +variant "onramp bind unset" "sponsord-onramp: ONRAMP_BIND is 127.x" "${onr} {/ONRAMP_BIND: /d}" +variant "onramp remote daemon" "sponsord-onramp: ONRAMP_DAEMON_URL" "${onr} s#ONRAMP_DAEMON_URL: http://127\.0\.0\.1:8090#ONRAMP_DAEMON_URL: http://10.0.0.1:8090#" +variant "onramp published port" "sponsord-onramp: publishes no ports" "${onr} s/^(\s*)network_mode: host$/&\n\1ports: [\"8080:8080\"]/" +variant "onramp writable rootfs" "sponsord-onramp: read_only rootfs" "${onr} s/^(\s*)read_only: true$/\1read_only: false/" +variant "onramp short grace" "sponsord-onramp: stop_grace_period" "${onr} s/^(\s*)stop_grace_period: 30s$/\1stop_grace_period: 5s/" +variant "onramp by tag" "sponsord-onramp: image is pinned" "${onr} s#^(\s*)image: .*#\1image: ghcr.io/decdn/sponsord-onramp:latest#" +variant "onramp holds keystore" "sponsord-onramp: mounts are exactly" "${onr} s#^(\s*)volumes:\$#\1volumes:\n\1 - /etc/sponsord/treasury-keystore.json:/run/secrets/k:ro#" +variant "writable turnstile mount" "sponsord-onramp: mounts are exactly" "${onr} {/target: \/run\/secrets\/turnstile-secret$/{n;s/read_only: true/read_only: false/}}" +variant "inline Turnstile secret" "sponsord-onramp: no inline ONRAMP_TURNSTILE_SECRET" "${onr} s/^(\s*)ONRAMP_BIND: (.*)$/&\n\1ONRAMP_TURNSTILE_SECRET: x/" +variant "onramp on sponsord hosts" "sponsord-onramp: profiles are [onramp]" "${onr} s/^(\s*)profiles: \[onramp\]$/\1profiles: [onramp, sponsord]/" +variant "resolvable domain default" "sponsord-onramp: unset domain" "${onr} s#ONRAMP_PUBLIC_URL: .*#ONRAMP_PUBLIC_URL: https://\\\${SPONSORD_ONRAMP_DOMAIN:-unset-SPONSORD_ONRAMP_DOMAIN.invalid}#" +# caddy +variant "caddy extra capability" "caddy: cap_add is exactly" "${cdy} s/cap_add: \[NET_BIND_SERVICE\]/cap_add: [NET_BIND_SERVICE, NET_ADMIN]/" +variant "caddy keeps capabilities" "caddy: cap_drop is [ALL]" "${cdy} {/^ cap_drop: /d}" +variant "caddy as root" "caddy: runs as a non-root uid:gid" "${cdy} {/^ user: /d}" +variant "caddy published port" "caddy: publishes no ports" "${cdy} s/^(\s*)network_mode: host$/&\n\1ports: [\"443:443\"]/" +variant "caddy always on" "caddy: profiles are [caddy]" "${cdy} {/^ profiles: \[caddy\]$/d}" # --- lint-cloud-init negatives: each broken variant must be rejected ------------------- # Skipped without cloud-init on PATH, except in CI (which installs it), so a broken