Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -227,10 +227,13 @@ jobs:
- name: cloud-init invariants
run: make lint-cloud-init

# Docker Compose path: render compose/compose.yaml with its example env files and
# assert the invariants its README promises (host network, no published ports,
# digest-pinned image, read-only rootfs, no capabilities, graceful stop). Runs
# only when compose/ or the root Makefile changed.
# Docker Compose path: render compose/compose.yaml with every profile on (with its
# example env files, without env files, with an empty .env) and assert the
# invariants its README promises (compose/tests/*.jq: host network, nothing
# published, images by digest, read-only rootfs, no capabilities, exact mounts,
# loopback listeners, no secret inline, graceful stop, unset variables that fail
# closed). Runs only when compose/ or the root
# Makefile changed.
compose:
needs: changes
if: needs.changes.outputs.compose == 'true'
Expand Down
36 changes: 29 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,13 +205,35 @@ addresses) the repo carries, and they carry their upstream commit.
user-data through cloud-init (skipping `baseline`) against a locally signed release
mirror, and is the suite's only coverage of the release download and verify path.

- **`compose/`** — the same node under Docker Compose on one host: the upstream image,
always by digest (`compose.yaml` builds `DECDN_IMAGE_REPO@DECDN_IMAGE_DIGEST`), the
role's host layout (`/etc/decdn` read-only, `/var/lib/decdn`), host
networking (so loopback metrics/admin stay loopback and Docker publishes no ports),
read-only rootfs, no capabilities, 300 s SIGTERM grace. `make lint-compose` asserts
those invariants (and `make test-scripts` that it rejects broken variants);
`make security` scans it.
- **`compose/`** — the node and the sponsor under Docker Compose on one host. The
node: the upstream image, always by digest (`compose.yaml` builds
`DECDN_IMAGE_REPO@DECDN_IMAGE_DIGEST`), the role's host layout (`/etc/decdn`
read-only, `/var/lib/decdn`), host networking (so loopback metrics/admin stay
loopback and Docker publishes no ports), read-only rootfs, no capabilities, 300 s
SIGTERM grace. Every service sits behind a
profile (`COMPOSE_PROFILES` in `.env`): `node`, `sponsord`, `onramp` (also starts
`sponsord`) and `caddy`.
- **sponsord / onramp:** the roles' `/etc/sponsord/` layout, except that the
credential files belong to a host `sponsord` account (bind mounts keep owner and
mode, and upstream rejects a group-readable keystore). They are mounted
read-only one by one into `/run/secrets/`. Listeners and secret paths are set in
`environment:`, which beats env files, because the release images default to
`0.0.0.0`.
- **Caddy:** the official image by digest, non-root, keeping only
`NET_BIND_SERVICE`; `compose/Caddyfile` mirrors the role's.
- **No `${VAR:?}`:** Compose interpolates disabled services too, so a required
variable would break other profiles. An unset variable renders a value its
service refuses instead (invalid image reference, unknown user, a domain with a
non-numeric port).
- **Gates:** `make lint-compose` renders with `--profile '*'` under `env -i`,
once with the example `.env` against `compose/tests/invariants.jq` (allowed
keys and exact mounts per service, so `privileged`, `pid: host` or an extra
mount fail), once without the env files (`--no-env-resolution`) against
`compose/tests/inline-env.jq` (an allow-list of inline env keys, so no RPC URL
lands in the tracked file) and once with an empty `.env` against
`compose/tests/fail-closed.jq`. All print `<service>: <invariant>` per violation; `make test-scripts` pins
which one fires for each broken variant (`LINT_COMPOSE_FILE`, not Compose's own
`COMPOSE_FILE`). `make security` scans it.

- **`charts/decdn-node/`** — the same node on Kubernetes: a one-replica StatefulSet (one
release = one identity) on the upstream daemon-only image (`ghcr.io/decdn/decdn-node`;
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ targets, so a local pass means a CI pass. Deploy targets live in
| `make molecule` | every `ansible/molecule/*/` scenario in parallel, in privileged systemd containers (needs Docker and util-linux `flock`; cap with `JOBS=<n>`). `make molecule-serial` runs them one at a time for readable failures. Both take a host-wide lock, so a second suite refuses to start (overlapping runs share container names). |
| `make lint-helm` | chart: `helm lint --strict`, positive/negative render tests, kubeconform (digest-pinned image), the shared schema-key check (needs `helm`, `yq`, `python3` ≥ 3.11, Docker). Set `DECDN_CLI=<path to decdn>` to also run the real `decdn config validate` (CI can't). |
| `make lint-alloy` | renders `roles/grafana_alloy`'s templates and validates them with the **real** digest-pinned Alloy binary. The molecule stub exits 0 for everything, so this is the only gate that proves the config loads. `ALLOY_BIN=<path>` skips the download. |
| `make lint-compose` | renders `compose/compose.yaml` with its example env and asserts its security invariants |
| `make lint-compose` | renders `compose/compose.yaml` with every profile on (with its example env, without env files, with an empty `.env`) and asserts `compose/tests/invariants.jq`, `inline-env.jq` and `fail-closed.jq` |
| `make lint-cloud-init` | `cloud-init schema` on `cloud-init/user-data.yaml`, then `cloud-init/tests/lint.py`: no secrets (only the bootstrap's own files, no secret-looking keys or assignments), no hardening skip, a `release` install verified against the vendored key with a host-generated wallet (trust knobs only in `decdn_nodes.vars`), localhost in `decdn_nodes`, a keyed admin account, `runcmd` exactly stage 1, shellcheck-clean scripts, and a collection lock that covers `ansible/requirements.yml` (needs `cloud-init`, `shellcheck`, `yq`). `CLOUD_INIT_FILE=<path>` checks your own filled-in copy. |
| `make test-scripts` | `tests/scripts-test.sh`: the `ansible/Makefile` scoping guards (dry runs), the release gate, and the negative cases of `lint-compose` and `lint-cloud-init` (the latter skipped without `cloud-init` on PATH). `UPSTREAM=<decdn checkout>` adds the sync generators' exit codes. |
| `make security` | KICS IaC scan of `ansible/`, the rendered chart and `compose/` (digest-pinned engine, fail on HIGH) |
Expand Down
69 changes: 46 additions & 23 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,13 @@ security-helm: ## KICS scan of the decdn-node chart's rendered manifests (
--no-progress --fail-on high

# One query is excluded, deliberately: "Volume Has Sensitive Host Directory"
# (1c1325ff-…) fires on the read-only /etc/decdn mount. That directory is the node's
# own config dir, mounted :ro, and keeping the Ansible host layout is what lets the
# host CLI, backups and restores (docs/lifecycle.md) work unchanged. The two MEDIUMs
# (host network, no healthcheck) are the documented design; see compose/README.md.
# (1c1325ff-…) flags every host-path mount: /etc/decdn (ro) and /var/lib/decdn, the
# /etc/sponsord credential files (ro, one file each) and /var/lib/caddy. They are
# the Ansible roles' host layout, which is what lets the host CLI, backups and
# restores (docs/lifecycle.md) work unchanged; lint-compose pins each container's
# exact mounts instead. The MEDIUMs (host
# network, no healthcheck, Caddy's NET_BIND_SERVICE) and the INFO (both sponsord
# containers mount the API token) are the documented design; see compose/README.md.
security-compose: ## KICS scan of compose/ (pinned engine image)
mkdir -p kics-results
docker run --rm --user $(shell id -u):$(shell id -g) -w /repo -v "$(CURDIR):/repo" $(KICS_IMAGE) \
Expand All @@ -69,27 +72,47 @@ security-compose: ## KICS scan of compose/ (pinned engine image)
--report-formats json,sarif --output-path /repo/kics-results/compose \
--no-progress --fail-on high

# Renders compose/compose.yaml with the example env files and asserts the invariants
# the README promises: host networking and no published ports (so metrics and the
# admin RPC stay loopback), a digest-pinned image, read-only rootfs, every capability
# dropped, no-new-privileges, and a stop grace long enough for the daemon's drain.
COMPOSE_INVARIANTS := .services["decdn-node"] as $$s | \
($$s.network_mode == "host") and ($$s.ports == null) and \
($$s.image | test("@sha256:[0-9a-f]{64}$$")) and ($$s.read_only == true) and \
($$s.cap_drop == ["ALL"]) and ($$s.security_opt | index("no-new-privileges:true") != null) and \
($$s.stop_signal == "SIGTERM") and ($$s.stop_grace_period == "5m0s") and \
($$s.user | test("^[0-9]+:[0-9]+$$"))

# COMPOSE_FILE is overridable so tests/scripts-test.sh can feed it broken variants.
COMPOSE_FILE ?= compose/compose.yaml
# Renders compose/compose.yaml with every profile on, three times, and checks each:
# - with the example .env, against compose/tests/invariants.jq: the properties the
# README promises (host network, nothing published, images by digest, read-only
# rootfs, no capabilities beyond Caddy's NET_BIND_SERVICE, exact security_opt,
# non-root users, each container's exact mounts, loopback sponsord listeners,
# secret files only, stop graces long enough for each daemon's drain);
# - with every service env file empty (/dev/null), against
# compose/tests/inline-env.jq: compose.yaml itself sets only the allowed
# environment keys, so no secret (DECDN_RPC_URL, SPONSORD_RPC_URL) moves into the
# tracked file. (Not `config --no-env-resolution`: older Compose releases, such
# as the CI runner's, still merge the env files with it.)
# - with an empty .env, against compose/tests/fail-closed.jq: an unset variable
# renders a value its service refuses, since compose.yaml cannot use `:?`.
# check <jq program> <.env file> <examples|none> does one of them: the service env
# files are compose/'s examples, or /dev/null for each. All run
# under `env -i`, because the caller's shell variables would override the
# .env and the lint would check something other than the committed defaults.
# LINT_COMPOSE_FILE (not Compose's own COMPOSE_FILE, which operators export) is
# overridable so tests/scripts-test.sh can feed it broken variants.
LINT_COMPOSE_FILE ?= compose/compose.yaml
lint-compose: ## render compose/ with its examples and check its security invariants (needs docker, jq)
@set -o pipefail; \
rendered="$$(DECDN_ENV_FILE=$(CURDIR)/compose/decdn.env.example docker compose -f '$(COMPOSE_FILE)' \
--env-file compose/.env.example config --format json)" \
|| { echo "lint-compose: docker compose could not render $(COMPOSE_FILE) (see above)" >&2; exit 2; }; \
jq -e '$(COMPOSE_INVARIANTS)' <<<"$$rendered" >/dev/null \
|| { echo "$(COMPOSE_FILE) violates an invariant (see the lint-compose comment in Makefile)" >&2; exit 1; }
@echo "compose invariants hold"
check() { \
mode="$$3"; ef() { if [ "$$mode" = none ]; then echo /dev/null; else echo "$(CURDIR)/compose/$$1"; fi; }; \
rendered="$$(env -i PATH="$$PATH" HOME="$$HOME" \
DECDN_ENV_FILE="$$(ef decdn.env.example)" \
SPONSORD_SECRET_ENV_FILE="$$(ef sponsord-secret.env.example)" \
SPONSORD_ENV_FILE="$$(ef sponsord.env.example)" \
SPONSORD_ONRAMP_ENV_FILE="$$(ef sponsord-onramp.env.example)" \
docker compose -f '$(LINT_COMPOSE_FILE)' --env-file "$$2" --profile '*' config --format json)" \
|| { echo "lint-compose: docker compose could not render $(LINT_COMPOSE_FILE) (see above)" >&2; exit 2; }; \
[ -n "$$rendered" ] || { echo "lint-compose: docker compose rendered nothing" >&2; exit 2; }; \
violations="$$(jq -r -f "$$1" <<<"$$rendered")" \
|| { printf '%s\n' "$$violations" >&2; echo "lint-compose: $$1 failed (see above)" >&2; exit 2; }; \
[ -z "$$violations" ] \
|| { sed 's/^/ /' <<<"$$violations" >&2; echo "$(LINT_COMPOSE_FILE) violates an invariant (see $$1)" >&2; exit 1; }; \
}; \
check compose/tests/invariants.jq compose/.env.example examples; \
check compose/tests/inline-env.jq compose/.env.example none; \
check compose/tests/fail-closed.jq /dev/null examples
@echo "compose invariants hold: $(LINT_COMPOSE_FILE)"

# The cloud-init user-data (cloud-init/README.md): `cloud-init schema` for its shape, then
# cloud-init/tests/lint.py for what a schema cannot see. That covers no secrets, no
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ day-2 tooling for operators anywhere. Four ways to deploy the same node:
|------|-----|------------|
| **Ansible** (`ansible/`) | VMs and bare metal, one node or a fleet. Hardens the host too (firewall, SSH, patching). Also published as the `decdn.node` Galaxy collection. | [`ansible/README.md`](ansible/README.md) |
| **cloud-init** (`cloud-init/`) | One VM, no control machine: paste the user-data into your provider's "create server" form. It runs the Ansible playbook on the host itself, hardening included. | [`cloud-init/README.md`](cloud-init/README.md) |
| **Docker Compose** (`compose/`) | One host that already runs Docker. | [`compose/README.md`](compose/README.md) |
| **Docker Compose** (`compose/`) | One host that already runs Docker. Also runs the onboarding sponsor (`sponsord` + its onramp) behind profiles. | [`compose/README.md`](compose/README.md) |
| **Helm** (`charts/decdn-node/`) | Kubernetes, one release per node. | [`charts/decdn-node/README.md`](charts/decdn-node/README.md) |

Supported hosts: Debian 12/13 and Ubuntu 24.04/26.04 on x86_64 or aarch64. Network,
Expand Down Expand Up @@ -82,9 +82,9 @@ This is the canonical statement; the per-path READMEs add only what is specific
you choose.
- **Localhost-only by default.** Backends bind `127.0.0.1`. A service that must accept
public traffic declares its port explicitly, and the node declares exactly one:
udp/4433. The sponsor's public onramp (Ansible only) stays on loopback too, behind
Caddy on tcp/80 + tcp/443 (the default; with `sponsord_onramp_proxy: none` you bring
the proxy and open its ports). On Kubernetes, metrics bind `0.0.0.0` in the pod only behind a ClusterIP
udp/4433. The sponsor's public onramp (Ansible and Compose) stays on loopback too,
behind Caddy on tcp/80 + tcp/443 (the default on Ansible, the `caddy` profile on
Compose; otherwise you bring the proxy and open its ports). On Kubernetes, metrics bind `0.0.0.0` in the pod only behind a ClusterIP
Service and a NetworkPolicy.
- **Default-deny inbound** (Ansible's `baseline`, nftables). SSH is the only
universally open port; extra public ports are declared via `baseline_extra_inbound`.
Expand All @@ -101,7 +101,7 @@ This is the canonical statement; the per-path READMEs add only what is specific
|------|------------|
| [`ansible/`](ansible/README.md) | The Ansible project: `inventory/`, `playbooks/` (`site.yml`, `sponsord.yml`, `backup.yml`, `decommission.yml`), `roles/` (`baseline`, `decdn_node`, `grafana_alloy`, `sponsord`, `sponsord_onramp`), `galaxy/` (the `decdn.node` collection), `molecule/`. |
| [`cloud-init/`](cloud-init/README.md) | The cloud-init deploy path: `user-data.yaml`, the on-host `bootstrap.sh`, and the pinned ansible-core and collections it installs. |
| [`compose/`](compose/README.md) | The Docker Compose deploy path. |
| [`compose/`](compose/README.md) | The Docker Compose deploy path: the node, `sponsord`, `sponsord-onramp` and Caddy, one profile each. |
| [`charts/decdn-node/`](charts/decdn-node/README.md) | The Helm chart, with vendored dashboards and alert rules in `files/monitoring/`. |
| [`docs/`](docs/requirements.md) | Cross-path operator docs: requirements, lifecycle. |
| `scripts/` | Generators for the upstream mirrors (network profiles, monitoring) and the release gate. |
Expand Down
51 changes: 46 additions & 5 deletions compose/.env.example
Original file line number Diff line number Diff line change
@@ -1,10 +1,22 @@
# Copy to .env beside compose.yaml. Nothing here is secret.

# The node image, always pinned by digest (compose.yaml builds REPO@DIGEST). Take the
# digest from the release's signed image-digest.txt (verify its .asc against the
# maintainer key in decdn/decdn's KEYS), not from whatever a tag points at today.
# DECDN_IMAGE_REPO defaults to ghcr.io/decdn/decdn-node; override it for a mirror or a
# local registry (compose/README.md, "before a release").
# Which services this host runs (compose.yaml's header): any of node, sponsord,
# onramp (which also starts sponsord) and caddy, comma-separated. For example
# node a deCDN node only
# sponsord the sponsor daemon only
# onramp,caddy the sponsor daemon, its public onramp and TLS in front
# node,onramp,caddy all of it on one host
COMPOSE_PROFILES=node

# Every image is pinned by digest (compose.yaml builds REPO@DIGEST). Take a digest
# from the release's signed digest file (image-digest.txt for decdn,
# <image>-image-digest.txt for sponsord; verify its .asc against the maintainer
# key in that repository's KEYS), not from whatever a tag points at today. The
# file holds REPO@DIGEST: copy the part after `@`. Override a REPO for a mirror or
# a local registry (README.md, "Before a release"). Set only the ones your
# profiles use; an unset one fails only the service that needs it.

# --- node -----------------------------------------------------------------------
DECDN_IMAGE_DIGEST=sha256:0000000000000000000000000000000000000000000000000000000000000000
# DECDN_IMAGE_REPO=ghcr.io/decdn/decdn-node

Expand All @@ -15,3 +27,32 @@ DECDN_GID=999

# Host path of the 0600 environment file holding DECDN_RPC_URL.
DECDN_ENV_FILE=/etc/decdn/decdn.env

# --- sponsord and onramp ----------------------------------------------------------
SPONSORD_IMAGE_DIGEST=sha256:0000000000000000000000000000000000000000000000000000000000000000
# SPONSORD_IMAGE_REPO=ghcr.io/decdn/sponsord
SPONSORD_ONRAMP_IMAGE_DIGEST=sha256:0000000000000000000000000000000000000000000000000000000000000000
# SPONSORD_ONRAMP_IMAGE_REPO=ghcr.io/decdn/sponsord-onramp

# uid/gid of the host's dedicated `sponsord` system account, which owns the
# credential files in /etc/sponsord: `id -u sponsord`, `id -g sponsord`. Both
# daemons run as it.
SPONSORD_UID=998
SPONSORD_GID=998

# Host paths of the environment files (README.md lists what goes in each).
SPONSORD_SECRET_ENV_FILE=/etc/sponsord/secret.env
SPONSORD_ENV_FILE=/etc/sponsord/sponsord.env
SPONSORD_ONRAMP_ENV_FILE=/etc/sponsord/sponsord-onramp.env

# The onramp's public host name: its public URL is https://<this>, and the caddy
# profile requests the certificate for it. Point its DNS at this host.
SPONSORD_ONRAMP_DOMAIN=onramp.example.org

# --- caddy ------------------------------------------------------------------------
# uid/gid of the host's dedicated `caddy` system account, which owns /var/lib/caddy.
CADDY_UID=997
CADDY_GID=997
# compose.yaml pins the official image by digest; override both to change it.
# CADDY_IMAGE_REPO=docker.io/library/caddy
# CADDY_IMAGE_DIGEST=sha256:…
26 changes: 26 additions & 0 deletions compose/Caddyfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# The public side of sponsord-onramp under Compose (the `caddy` profile), the same
# as the sponsord_onramp role's Caddyfile: TLS for SPONSORD_ONRAMP_DOMAIN (from
# .env), then a reverse proxy to the onramp on loopback. Caddy redirects http to
# https and, with no trusted_proxies, replaces any client-sent X-Forwarded-For with
# the real client address, which the onramp rate limits by. So
# ONRAMP_CLIENT_IP_HEADER=X-Forwarded-For is safe behind this file; with another
# proxy, see sponsord-onramp.env.example.
{
# No admin API: nothing on this host can reconfigure the running proxy over
# the admin endpoint (:2019). A Caddyfile change needs a restart:
# `docker compose restart caddy`.
admin off
# HTTP/1.1 and HTTP/2 only: HTTP/3 would need udp/443, which this setup does
# not ask you to open.
servers {
protocols h1 h2
}
# An optional ACME account contact:
# email ops@example.org
}

{$SPONSORD_ONRAMP_DOMAIN} {
# Caddy's local CA, for staging and tests only (browsers do not trust it):
# tls internal
reverse_proxy 127.0.0.1:8080
}
Loading
Loading