Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -381,6 +381,9 @@ jobs:
# Without cloud-init on PATH the harness SKIPS the lint-cloud-init cases.
- name: Install cloud-init
run: command -v cloud-init || { sudo apt-get update -qq && sudo apt-get install -y -qq cloud-init; }
# Without ansible-core on PATH the harness FAILS the firewall-holes case in CI.
- name: Install ansible-core
run: command -v ansible || pipx install ansible-core
- name: Script and guard tests
run: make test-scripts

Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/molecule.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ jobs:
runs-on: ubuntu-latest # Docker is preinstalled
# ~4m41s at JOBS=3 with 8 scenarios. os-matrix (3 containers) and lifecycle took the
# suite to ~10m at JOBS=3 on a local 16-thread box; re-measure from CI's first run.
# sponsord added a 12th scenario (2 containers, Debian 12 + Ubuntu 24.04).
# sponsord added a 12th scenario (2 containers, Debian 12 + Ubuntu 24.04), and
# sponsord-onramp a 13th (the same 2, plus an apt install of Caddy each).
# A bound, not a target: without one a wedged privileged systemd container burns
# the 360-minute default, and with cancel-in-progress above, some branch-protection
# setups read the resulting cancelled check as "not failed" rather than as a failure.
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/upstream-drift.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@ name: Upstream drift

# Three things in this repo mirror decdn/decdn and go stale when upstream moves:
# - roles/decdn_node/vars/main/networks.yml (contract addresses; upstream
# redeploys change every one of them at once) and its subset
# roles/sponsord/vars/main/networks.yml (the same script and check)
# redeploys change every one of them at once) and its subsets
# roles/sponsord/vars/main/networks.yml and
# roles/sponsord_onramp/vars/main/networks.yml (the same script and check)
# - charts/decdn-node/files/monitoring/ (dashboards + alert rules)
# - ansible/molecule/schema/files/schema-keys.txt (the node.toml key inventory)
# This job regenerates each from upstream main and fails when a committed copy
Expand Down
61 changes: 46 additions & 15 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,13 @@ mirror (`vars/main/networks.yml`), never hand-copied into inventory or docs.
overrides `node.toml`), keeps the keystore password off the PVC, and refuses
secret-bearing keys in `config`.
2. **Localhost-only by default.** Service daemons bind `127.0.0.1` (e.g. the node's metrics
and admin RPC). A service that must accept public traffic declares exactly one hole (the
node's QUIC udp/4433) via `baseline_extra_inbound`; if a service ever needs an HTTP-facing
public path, front it with an explicit reverse proxy that terminates auth + TLS. Never
bind a *backend* to `0.0.0.0` or expose its raw port.
and admin RPC). A service that must accept public traffic declares its holes explicitly
via `baseline_extra_inbound` (today: the node's QUIC udp/4433, and tcp/80 + tcp/443 for
the `sponsord_onramp` role's Caddy). An HTTP-facing service is fronted by an explicit
reverse proxy that terminates TLS (and auth, where the service needs it), and the
backend stays on loopback. Never bind a *backend* to `0.0.0.0` or expose its raw port.
`sponsord-onramp` is public by design: its Turnstile gate guards the one action that
spends (`POST /v1/fund`), and per-address rate limits cover the rest.
**Kubernetes exception (chart only):** the node's metrics bind `0.0.0.0` inside the pod
so kubelet probes and Prometheus can reach them. That is allowed only behind a
ClusterIP-only Service and the chart's NetworkPolicy (metrics ingress limited to
Expand All @@ -54,8 +57,8 @@ mirror (`vars/main/networks.yml`), never hand-copied into inventory or docs.

```
ansible/ # the deployment project (DevSec-hardened, lean roles)
playbooks/ # site.yml (decdn node + sponsord), sponsord.yml, backup.yml, decommission.yml
roles/ # baseline, decdn_node, grafana_alloy, sponsord
playbooks/ # site.yml (decdn node + sponsord), sponsord.yml (+ onramp), backup.yml, decommission.yml
roles/ # baseline, decdn_node, grafana_alloy, sponsord, sponsord_onramp
inventory/ galaxy/ molecule/ # see ansible/README.md
cloud-init/ # user-data.yaml + on-host bootstrap.sh; pinned ansible-core/collections (see its README.md)
compose/ # Docker Compose deploy path for a single host (see its README.md)
Expand All @@ -69,8 +72,9 @@ scripts/ # upstream-mirror generators + the release gate
```

**Generated mirrors of upstream — regenerate, never hand-edit:**
`ansible/roles/decdn_node/vars/main/networks.yml` and its subset
`ansible/roles/sponsord/vars/main/networks.yml` (both `scripts/sync-network-profiles.py`),
`ansible/roles/decdn_node/vars/main/networks.yml` and its subsets
`ansible/roles/sponsord/vars/main/networks.yml` and
`ansible/roles/sponsord_onramp/vars/main/networks.yml` (all `scripts/sync-network-profiles.py`),
`charts/decdn-node/files/monitoring/` (`scripts/sync-monitoring.sh`) and
`ansible/molecule/schema/files/schema-keys.txt` (`gen-schema-keys.py`). The weekly
`upstream-drift` workflow flags staleness. These are the only protocol facts (contract
Expand Down Expand Up @@ -157,7 +161,29 @@ addresses) the repo carries, and they carry their upstream commit.
- **Molecule:** under docker, `/` must be `--make-rshared` (see
`molecule/sponsord/prepare.yml`) or every credential directory is empty.
- **Not in cloud-init:** its lint only allows `decdn_nodes`.
- The public `sponsord-onramp` is not deployed yet.

- **`ansible/roles/sponsord_onramp`** — `sponsord-onramp`, sponsord's public side (the
Turnstile gate, the installers, the `decdn-sponsored` CLI API), as a second play of
`playbooks/sponsord.yml` on `sponsord_onramp_hosts` (which must also be in
`sponsord_hosts`: it reads the daemon's `/etc/sponsord/api-token` and calls it on
loopback). The playbook's first play enforces that membership (tested by
`make test-scripts`); the role checks no group name, so collection users keep their
own groups, and its token gate and `/healthz` fail without a daemon.
- **Install / unit / gate:** the sponsord role's patterns, copied: manual or
GPG-verified `sponsord-onramp-v*` release (the KEYS are the sponsord role's file,
via `role_path`), `DynamicUser` with the token and the operator-provisioned
Turnstile secret as `LoadCredential=`, `PartOf=sponsord.service`, a restart-inputs
record, and a fatal `/healthz` gate on a loopback listener.
- **Public inputs:** `sponsord_onramp_rpc_url` is served to every user, so preflight
refuses userinfo, a query and the characters upstream's installers cannot quote.
CapacityBond/SlashJudge come from the generated `vars/main/networks.yml`.
- **Caddy (default) or none:** the distro package, a role-owned `/etc/caddy/Caddyfile`
(marker-gated: a foreign one needs `sponsord_onramp_caddy_overwrite_config`),
`admin off`, so the handler restarts. tcp/80 + tcp/443 come from
`playbooks/group_vars/all.yml`, which reads `sponsord_onramp_proxy` from inventory.
- **Molecule:** `molecule/sponsord-onramp` runs the real Caddy with `tls internal`
and checks the https chain and that a forged `X-Forwarded-For` cannot pick the
client address.

- **`cloud-init/`** — the Ansible path with no control machine. `user-data.yaml` carries
only public material (the lint refuses secret-looking keys, credentials in URLs and
Expand Down Expand Up @@ -231,13 +257,18 @@ make build / galaxy-check # the decdn.node collection

**Inventory is private; the firewall hole is not.** This repo is public, so
`ansible/inventory/hosts.yml` is git-ignored and a real fleet lives in a private overlay.
Inventory-adjacent group_vars do not load for an overlay, so anything every node needs regardless of inventory (today the udp/4433
`baseline_extra_inbound` hole) lives in `ansible/playbooks/group_vars/decdn_nodes.yml`, and
the Alloy per-daemon toggles every host needs in `ansible/playbooks/group_vars/all.yml`.
Don't move it back under `inventory/`.
Inventory-adjacent group_vars do not load for an overlay, so anything every host needs
regardless of inventory lives in `ansible/playbooks/group_vars/`. The public firewall holes
(`baseline_extra_inbound`, today the node's udp/4433, plus tcp/80 + tcp/443 on
`sponsord_onramp_hosts` with `sponsord_onramp_proxy: caddy`) are built from a host's groups in
`all.yml` (`_baseline_service_inbound`). `decdn_nodes.yml` and `sponsord_hosts.yml` both set
`baseline_extra_inbound` to that list, so a co-located host renders one firewall in every play.
`ansible/tests/firewall-holes/` (run by `make test-scripts`) pins the result per host shape.
The Alloy per-daemon toggles also live in `all.yml`. Don't move any of it back under
`inventory/`.

**Galaxy collection (`decdn.node`).** The four roles (`baseline` + `decdn_node` +
`grafana_alloy` + `sponsord`) ship as a
**Galaxy collection (`decdn.node`).** The five roles (`baseline` + `decdn_node` +
`grafana_alloy` + `sponsord` + `sponsord_onramp`) ship as a
distributable collection. The overlay lives in `ansible/galaxy/` and is staged into a clean
collection tree by `galaxy/build.sh` — there is **no** `galaxy.yml` at the `ansible/` root
(that would make ansible-lint treat the deploy project as a collection). Build/validate with
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,9 @@ This is the canonical statement; the per-path READMEs add only what is specific
you choose.
- **Localhost-only by default.** Backends bind `127.0.0.1`. A service that must accept
public traffic declares its port explicitly, and the node declares exactly one:
udp/4433. On Kubernetes, metrics bind `0.0.0.0` in the pod only behind a ClusterIP
udp/4433. The sponsor's public onramp (Ansible only) stays on loopback too, behind
Caddy on tcp/80 + tcp/443 (the default; with `sponsord_onramp_proxy: none` you bring
the proxy and open its ports). On Kubernetes, metrics bind `0.0.0.0` in the pod only behind a ClusterIP
Service and a NetworkPolicy.
- **Default-deny inbound** (Ansible's `baseline`, nftables). SSH is the only
universally open port; extra public ports are declared via `baseline_extra_inbound`.
Expand All @@ -97,7 +99,7 @@ This is the canonical statement; the per-path READMEs add only what is specific

| Path | What it is |
|------|------------|
| [`ansible/`](ansible/README.md) | The Ansible project: `inventory/`, `playbooks/` (`site.yml`, `sponsord.yml`, `backup.yml`, `decommission.yml`), `roles/` (`baseline`, `decdn_node`, `grafana_alloy`, `sponsord`), `galaxy/` (the `decdn.node` collection), `molecule/`. |
| [`ansible/`](ansible/README.md) | The Ansible project: `inventory/`, `playbooks/` (`site.yml`, `sponsord.yml`, `backup.yml`, `decommission.yml`), `roles/` (`baseline`, `decdn_node`, `grafana_alloy`, `sponsord`, `sponsord_onramp`), `galaxy/` (the `decdn.node` collection), `molecule/`. |
| [`cloud-init/`](cloud-init/README.md) | The cloud-init deploy path: `user-data.yaml`, the on-host `bootstrap.sh`, and the pinned ansible-core and collections it installs. |
| [`compose/`](compose/README.md) | The Docker Compose deploy path. |
| [`charts/decdn-node/`](charts/decdn-node/README.md) | The Helm chart, with vendored dashboards and alert rules in `files/monitoring/`. |
Expand Down
10 changes: 6 additions & 4 deletions ansible/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -121,8 +121,10 @@ decommission:
# /metrics probe timeout), grafana-cloud and grafana-cloud-token (opt-in Grafana
# Cloud observability, host-provisioned vs inventory token), os-matrix (default's
# plays on Debian 13, Ubuntu 24.04 and 26.04), lifecycle (network profile,
# backup + decommission) and sponsord (the sponsord role on a host with no node,
# through playbooks/sponsord.yml, on Debian 12 and Ubuntu 24.04). The glob below is the authority; this list is a guide.
# backup + decommission), sponsord (the sponsord role on a host with no node,
# through playbooks/sponsord.yml, on Debian 12 and Ubuntu 24.04) and sponsord-onramp
# (the onramp beside its daemon, behind the real distro Caddy). The glob below is the
# authority; this list is a guide.
#
# The scenarios run concurrently: distinct container names, no published host ports,
# per-scenario ephemeral dirs, and what they share on the control machine (the
Expand Down Expand Up @@ -209,8 +211,8 @@ molecule-all: deps
molecule test --all

# --- Galaxy collection (decdn.node) ------------------------------------------
# Stage baseline + decdn_node + grafana_alloy + sponsord into a clean collection tree and
# build the artifact under build/. Only the four roles ship; see galaxy/README.md. Publishing
# Stage baseline + decdn_node + grafana_alloy + sponsord + sponsord_onramp into a clean collection tree and
# build the artifact under build/. Only the five roles ship; see galaxy/README.md. Publishing
# runs from a vX.Y.Z tag in .github/workflows/release.yml (see RELEASING.md).
build:
./galaxy/build.sh
Expand Down
Loading
Loading