Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,14 @@ jobs:
- '.github/workflows/ci.yml'
# The chart shares the schema-key inventory and checker with molecule,
# and renders monitoring/decdn-node/ through a symlink; lint-helm also
# checks monitoring/sponsord/ and monitoring/iroh-relay/.
# checks monitoring/sponsord/ and monitoring/iroh-relay/. The decdn CLI
# it validates against is verified with the role's vendored KEYS.
helm:
- 'charts/**'
- 'monitoring/**'
- 'ansible/molecule/schema/files/**'
- 'ansible/roles/decdn_node/files/decdn-release-KEYS.asc'
- 'scripts/fetch-decdn-cli.sh'
- 'Makefile'
- '.github/workflows/ci.yml'
# lint-compose and security-compose live in the root Makefile.
Expand Down Expand Up @@ -190,8 +193,9 @@ jobs:
if-no-files-found: ignore

# Helm chart: `helm lint --strict`, positive + negative render tests, kubeconform
# and promtool (digest-pinned images) and the upstream schema-key check shared
# with molecule — all via `make lint-helm`, the same command developers run locally.
# and promtool (digest-pinned images), the upstream schema-key check shared
# with molecule and the real `decdn config validate` — all via `make lint-helm`,
# the same command developers run locally.
helm:
needs: changes
if: needs.changes.outputs.helm == 'true'
Expand All @@ -208,10 +212,13 @@ jobs:
with:
python-version: '3.12'
- name: Chart lint + render tests
# yq (mikefarah) and Docker ship on ubuntu-latest. No decdn binary here, so
# the real `decdn config validate` step reports SKIPPED; run it locally with
# DECDN_CLI=... before bumping the decdn version.
run: make lint-helm
# yq (mikefarah), curl, gpg and Docker ship on ubuntu-latest. The decdn CLI is the
# release the chart pins (appVersion), signature-checked against the vendored
# KEYS, so a decdn bump PR validates the chart against the new release.
run: |
DECDN_CLI="$(scripts/fetch-decdn-cli.sh "$RUNNER_TEMP/decdn")"
export DECDN_CLI
make lint-helm

# cloud-init path: `cloud-init schema` plus the invariants cloud-init/README.md
# promises (no secrets, release installs, host-generated node wallet, localhost in
Expand Down
24 changes: 15 additions & 9 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ monitoring/ # Grafana dashboards + Prometheus alert rules (maintaine
iroh-relay/ # the relay's (+ promtool unit tests, exported-metrics.txt), Ansible and Compose
docs/ # cross-path operator docs: node-operators.md + publishers.md (the front doors), requirements.md, lifecycle.md
examples/collection-fleet/ # a consumer project for the Galaxy collections: requirements.yml, ansible.cfg, .gitignore, inventory/ (ansible/inventory/'s templates: every tracked file identical but the hosts-*.yml.example headers, which end at the first blank line and hold only comments; make test-scripts checks, and that both parse to the same inventory, so change both. The root .gitignore keeps a hosts.yml filled in there out of this public repo)
scripts/ # upstream-mirror generators, the compose Alloy config generator (render-compose-alloy.sh), the decdn bump (open-decdn-bump.sh, bump_decdn.py), the release script (release.sh, git-cliff: ../cliff.toml) and its gate (+ its Artifact Hub changes generator), the molecule driver (molecule.sh), ansible/Makefile's LIMIT preflight (limit-guard.sh)
scripts/ # upstream-mirror generators, the compose Alloy config generator (render-compose-alloy.sh), the decdn bump (open-decdn-bump.sh, bump_decdn.py), the chart check's verified decdn CLI (fetch-decdn-cli.sh), the release script (release.sh, git-cliff: ../cliff.toml) and its gate (+ its Artifact Hub changes generator), the molecule driver (molecule.sh), ansible/Makefile's LIMIT preflight (limit-guard.sh)
```

**Generated mirrors of upstream — regenerate, never hand-edit:**
Expand All @@ -148,9 +148,13 @@ tags) are pinned across every path: the role defaults, the onramp's installer pi
(`sponsord_onramp_decdn_release` + `_decdn_sums_sha256`, copied into
`compose/sponsord-onramp.env.example`), `compose/.env.example`'s image digests, the
chart's `appVersion` and the generated mirrors. cloud-init takes the role defaults at
`DEVOPS_REF`. `make test-scripts` checks the pins agree with each other, the Compose
example, `appVersion` and the chart's `artifacthub.io/images` tag, and that both
vendored KEYS hold the keys SECURITY.md publishes; the weekly `upstream-drift` job checks the digests and the mirrors against
`DEVOPS_REF`. Docs name the pin (`decdn_node_version`, `sponsord_version`), never its
number: besides this paragraph, only the commented examples and the comments beside a
pinned value name a number, and they move with the pin (`bump_decdn.py` moves decdn's;
the sponsord checklist lists sponsord's). `make test-scripts` checks the pins agree
with each other, the Compose example, `appVersion`, the chart's `artifacthub.io/images`
tag and this paragraph, and that both vendored KEYS hold the keys SECURITY.md
publishes; the weekly `upstream-drift` job checks the digests and the mirrors against
the releases and flags a newer upstream tag. The `--version` check compares the pin in
`release` mode only (exactly, on `<binary> <version>`). Bump everything together with
the checklists in `roles/decdn_node/README.md` and `roles/sponsord/README.md`;
Expand Down Expand Up @@ -196,7 +200,7 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`).
the publisher's step). **The public deCDN node**
(baseline + `decdn-node`), installed by `decdn_node_install_method`:
`release` (the default) — a pinned GitHub release tarball verified against the
release's GPG-signed `SHA256SUMS` (`decdn_node_version`, default `0.0.4`); `source` — a git ref (any tag/branch/SHA) cloned and `cargo build`-ed **on the
release's GPG-signed `SHA256SUMS` (`decdn_node_version`); `source` — a git ref (any tag/branch/SHA) cloned and `cargo build`-ed **on the
node** as the unprivileged `decdn-build` user, sha256-pinned rustup, a `<repo>@<commit>`
stamp (`tasks/source.yml`, copied into both sponsord roles, which share the user,
home and toolchain). **No build can poison a later one:** root owns the home, the
Expand Down Expand Up @@ -293,7 +297,7 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`).
imported by `publisher.yml`) targets its own `sponsord_hosts` group, standalone or
co-located with a node.
- **Install:** a GPG-verified `v*` decdn/sponsord release by default
(`sponsord_version`, `0.0.4`), a host-side `source` build, or a manual binary.
(`sponsord_version`), a host-side `source` build, or a manual binary.
The `--version` backstop checks the version in `release` mode only (the version
has a default, so other modes would fail against it), as in the other two roles.
- **Unit:** `DynamicUser`; the API token (generated on the host, never replaced),
Expand Down Expand Up @@ -656,9 +660,11 @@ the pinned `iroh-relay` (re-capture it when bumping `iroh_relay_version`).
behind a ClusterIP Service + NetworkPolicy. `values.config` mirrors `node.toml`; the
chart injects the path/port keys and fails on collisions. **The config-schema coupling
above applies here too:** `make lint-helm` runs the same `check-schema-keys.py` +
`schema-keys.txt` on the rendered ConfigMap, so a re-sync covers both paths. Unlike the
role, CI has no real-binary `decdn config validate` for the chart — run
`DECDN_CLI=… make lint-helm` locally when bumping the decdn version. Optional
`schema-keys.txt` on the rendered ConfigMap, so a re-sync covers both paths. CI's
`helm` job also runs the real `decdn config validate` on each `ci/*.yaml` render, against the
`decdn` of the release `appVersion` pins (`scripts/fetch-decdn-cli.sh`: the role's
GPG check against the vendored KEYS), so a decdn bump PR validates the chart against
the new release. Optional
`PrometheusRule` + dashboard ConfigMaps render `files/monitoring/`, the symlink to
`monitoring/decdn-node/` (never through `tpl`:
the alert annotations carry Prometheus templates).
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ targets, so a local pass means a CI pass. Deploy targets live in
| `make lint` | every pre-commit hook on every file (CI job `pre-commit`) |
| `make lint-ansible` | install Galaxy collections + `ansible-lint` (production profile, which includes the Ansible security rules) |
| `make molecule` | every `ansible/molecule/*/` scenario in parallel, in privileged systemd containers (needs Docker and util-linux `flock`; cap with `JOBS=<n>`, pick some with `SCENARIOS='a b'`). `make molecule-serial` runs them one at a time for readable failures. Each scenario takes its own host-wide lock, so two runs can share a host as long as they share no scenario (a run that would refuses to start: runs of one scenario share its container names). `make molecule-list` prints the selection as JSON (CI's matrix). The driver is `scripts/molecule.sh`. |
| `make lint-helm` | chart: `helm lint --strict`, positive/negative render tests, kubeconform and `promtool check rules` on the alert rules (digest-pinned images), the shared schema-key check (needs `helm`, `yq`, `python3` ≥ 3.11, Docker). Set `DECDN_CLI=<path to decdn>` to also run the real `decdn config validate` (CI can't). |
| `make lint-helm` | chart: `helm lint --strict`, positive/negative render tests, kubeconform and `promtool check rules` on the alert rules (digest-pinned images), the shared schema-key check (needs `helm`, `yq`, `python3` ≥ 3.11, Docker). Set `DECDN_CLI=<path to decdn>` to also run the real `decdn config validate`; CI does, with the pinned release's (`cli="$(scripts/fetch-decdn-cli.sh <dir>)" && DECDN_CLI="$cli" make lint-helm`). |
| `make lint-alloy` | renders `roles/grafana_alloy`'s templates and validates them with the **real** digest-pinned Alloy binary. The molecule stub exits 0 for everything, so this is the only gate that proves the config loads. It also checks that `compose/alloy/config.alloy` (the Compose `alloy` profile's render of the same template) is current, and runs it in Alloy: every component healthy, each daemon scraped only under the profiles that start it. `ALLOY_BIN=<path>` skips the download. |
| `make lint-compose` | reads `compose/compose.yaml` as written (`yq`) against `compose/tests/bind-sources.jq` (every bind mount's `create_host_path`), then renders it with every profile on (with its example env, without env files, with an empty `.env`) and asserts `compose/tests/invariants.jq`, `inline-env.jq` and `fail-closed.jq` |
| `make lint-cloud-init` | `cloud-init schema` on `cloud-init/user-data-node.yaml` and `cloud-init/user-data-publisher.yaml`, then `cloud-init/tests/lint.py` on each: only the templates' top-level modules and no YAML anchors, no secrets (only the bootstrap's own files, once each, plain `content` only, no secret-looking keys or assignments), no hardening skip, `release` installs verified against the vendored keys with a host-generated node wallet (trust knobs only in their own group's `vars`, no moved keyring or secret path), only localhost, in `decdn_nodes` and/or `sponsord_hosts` (an origin only beside `decdn_nodes` and with a backend, every backend key only in `decdn_origin_nodes.vars`, the onramp only beside sponsord), a keyed admin account, `runcmd` exactly stage 1, shellcheck-clean scripts, and a collection lock that covers `ansible/requirements.yml` (needs `cloud-init`, `shellcheck`, `yq`). `CLOUD_INIT_FILE=<path>` checks your own filled-in copy. |
Expand Down
7 changes: 4 additions & 3 deletions ansible/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,8 +183,9 @@ ignored on the first converge.
1. **Binaries.** Pick `decdn_node_install_method`:
- **`release`** (the default) downloads the `v<decdn_node_version>` tarballs from
`decdn_node_release_base` and verifies them against the GPG-signed `SHA256SUMS`.
`decdn_node_version` defaults to the decdn/decdn release this repo pins
(`0.0.4`); the sponsord roles pin decdn/sponsord `0.0.4` the same way.
`decdn_node_version` defaults to the decdn/decdn release this repo pins (the
role's `defaults/main.yml`); the sponsord roles pin a decdn/sponsord release the
same way (`sponsord_version`).
- **`source`** clones `decdn_node_source_repo` (default `decdn/decdn` on GitHub) at
`decdn_node_source_ref` (any tag, branch or SHA) **on the node** and builds it there
as an unprivileged `decdn-build` user, in a fresh environment per commit, with a
Expand Down Expand Up @@ -508,7 +509,7 @@ the RPC URL when it is not provisioned on the host instead). Highlights:
| `baseline_extra_inbound` | `[]` | public inbound ports; `decdn_nodes` opens udp/4433. |
| `baseline_preserve_ipv6_autoconf` | `true` | Keep IPv6 RA/autoconf under hardening; set `false` for static-IPv6 hosts. |
| `baseline_rp_filter_loose` | `false` | `true` loosens reverse-path filtering (`rp_filter=2`) for multi-homed nodes. |
| `decdn_node_install_method` | `release` | `release` (the pinned `decdn_node_version`, `0.0.4`), `source` (build `decdn_node_source_ref` on the node) or `manual` (copy a control-machine build). |
| `decdn_node_install_method` | `release` | `release` (the pinned `decdn_node_version`), `source` (build `decdn_node_source_ref` on the node) or `manual` (copy a control-machine build). |
| `decdn_node_source_repo` / `decdn_node_source_ref` | `decdn/decdn` on GitHub / `""` | `source` only: the repo (no credentials in the URL) and the tag, branch or SHA to build. |
| `decdn_node_target` | from the host | The release target triple, derived from the host architecture (x86_64 or aarch64). |
| `decdn_network` | `""` | `arbitrum-sepolia` sets `chain_id` and every contract address from the role's manifest mirror; inventory values still win. `""` = set them yourself. |
Expand Down
4 changes: 2 additions & 2 deletions ansible/galaxy/publisher/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ them and `decdn.node`'s (see [Usage](#usage)):

| Role | Purpose |
|------|---------|
| `decdn.publisher.sponsord` | The `sponsord` onboarding sponsor (treasury signer + PaymentPool keeper), standalone or beside a node — local binary, a host-side source build or a GPG-verified decdn/sponsord `v*` release (pinned to 0.0.4), `DynamicUser` unit with the API token and treasury wallet as systemd credentials, loopback-only API, `/healthz` deploy gate. `sponsord_network` sets the chain and PaymentPool from upstream's manifest. Restarts refuse while a pool top-up is held; `tasks_from: backup` / `decommission` for day 2. |
| `decdn.publisher.sponsord_onramp` | `sponsord-onramp`, sponsord's public side (Turnstile gate, installers, capability API), on the daemon's host — local binary, a host-side source build or a GPG-verified decdn/sponsord `v*` release (pinned to 0.0.4), `DynamicUser` unit with the daemon token and Turnstile secret as systemd credentials, loopback listener behind Caddy (distro package, ACME TLS, admin API off; or your own proxy), `/healthz` deploy gate; `tasks_from: decommission`. |
| `decdn.publisher.sponsord` | The `sponsord` onboarding sponsor (treasury signer + PaymentPool keeper), standalone or beside a node — local binary, a host-side source build or a GPG-verified decdn/sponsord `v*` release (pinned by `sponsord_version`), `DynamicUser` unit with the API token and treasury wallet as systemd credentials, loopback-only API, `/healthz` deploy gate. `sponsord_network` sets the chain and PaymentPool from upstream's manifest. Restarts refuse while a pool top-up is held; `tasks_from: backup` / `decommission` for day 2. |
| `decdn.publisher.sponsord_onramp` | `sponsord-onramp`, sponsord's public side (Turnstile gate, installers, capability API), on the daemon's host — local binary, a host-side source build or a GPG-verified decdn/sponsord `v*` release (pinned by `sponsord_onramp_version`, kept equal to `sponsord_version`), `DynamicUser` unit with the daemon token and Turnstile secret as systemd credentials, loopback listener behind Caddy (distro package, ACME TLS, admin API off; or your own proxy), `/healthz` deploy gate; `tasks_from: decommission`. |
| `decdn.publisher.iroh_relay` | A self-hosted [iroh relay](https://github.com/n0-computer/iroh) (`iroh-relay`) for deCDN peers that cannot hole-punch — sha256-pinned upstream release (or a local binary), its own Let's Encrypt TLS, QUIC address discovery, `DynamicUser` unit holding only `CAP_NET_BIND_SERVICE`, loopback metrics, a deploy gate on every listener, a refusal of ports another process holds; access limited to an allowlist of endpoint IDs by default (`iroh_relay_access`; `tasks_from: node-ids` reads nodes' IDs); `tasks_from: decommission`. Point nodes at it with `decdn_relay_urls`. |
| `decdn.publisher.iroh_dns_server` | A self-hosted [iroh DNS server](https://github.com/n0-computer/iroh) (`iroh-dns-server`): the pkarr relay deCDN nodes publish their address records to and the DNS server peers resolve them through, in place of n0's `dns.iroh.link` — sha256-pinned upstream release (or a local binary), its own Let's Encrypt TLS, DNS on one bind address (clear of systemd-resolved's stub), `DynamicUser` unit holding only `CAP_NET_BIND_SERVICE`, loopback metrics, a deploy gate on every listener, `/healthz`'s version and an SOA answer over udp and tcp, a refusal of ports another process holds; `tasks_from: decommission`. Point nodes at it with `decdn_discovery_pkarr_url` and `decdn_discovery_dns_origin`. |

Expand Down
3 changes: 2 additions & 1 deletion ansible/inventory/group_vars/sponsord_onramp_hosts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@
# sponsord_onramp_rpc_url: https://sepolia-rollup.arbitrum.io/rpc
# sponsord_onramp_turnstile_sitekey: "0x…"
# The decdn release the installers install defaults to the role's pin (decdn/decdn
# v0.0.4), with the SHA-256 of its SHA256SUMS. To serve another, set both:
# v<decdn_node_version>), with the SHA-256 of its SHA256SUMS. To serve another,
# set both:
# sponsord_onramp_decdn_release: v0.0.4
# sponsord_onramp_decdn_sums_sha256: "…"
# CapacityBond and SlashJudge come from sponsord_network (group_vars/sponsord_hosts.yml).
Loading
Loading