From fa04c145a6777f172b1d1df5602b8cd1008f6b14 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Sat, 10 Oct 2026 19:29:02 +0300 Subject: [PATCH 1/2] docs: front doors for node operators and publishers docs/node-operators.md and docs/publishers.md are the two entry points: what each kind of operator runs, on which deploy path and entry point, and the on-chain steps that are theirs, every protocol claim cited to an ADR. A node operator runs cache nodes, which pay for their misses from their own pool; a publisher runs origin nodes (served from their own backend, seated with `decdn publish assign`) and optionally sponsord, iroh relays and an iroh DNS server. The README opens on that split, AGENTS.md describes the persona entry points, and lifecycle.md says decommissioning an origin leaves it seated until the publisher revokes it. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 19 ++++++++-- README.md | 21 +++++++---- docs/lifecycle.md | 9 +++++ docs/node-operators.md | 57 ++++++++++++++++++++++++++++++ docs/publishers.md | 79 ++++++++++++++++++++++++++++++++++++++++++ docs/requirements.md | 6 ++-- 6 files changed, 180 insertions(+), 11 deletions(-) create mode 100644 docs/node-operators.md create mode 100644 docs/publishers.md diff --git a/AGENTS.md b/AGENTS.md index 0b851e7..8c92b16 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,8 +5,21 @@ DevOps repo. ## What this repo is -The official DevOps project for deploying a **deCDN node**: infrastructure, deployment, -and operational tooling, for node operators anywhere. There are four deploy paths: +The official DevOps project for deploying deCDN: infrastructure, deployment, and +operational tooling. It serves **two kinds of operator**, and every entry point is +split along that line: + +- **Node operators** run cache nodes (`decdn-node`, no origin backend): `node.yml` / + `make deploy-node`, `cloud-init/user-data-node.yaml`, Compose's `node` profile, the + chart, the `decdn.node` collection. Front door: `docs/node-operators.md`. +- **Publishers** own namespaces in `PublisherRegistry` and run **origin nodes** (the + same daemon with an origin backend, seated on-chain via `OriginAssignment.addOrigin`, + ADR 002/011), and optionally `sponsord` + its onramp, iroh relays and an iroh DNS + server: `publisher.yml` / `make deploy-publisher` (`origin.yml` + the component + playbooks), `cloud-init/user-data-publisher.yaml`, Compose's `origin`/`onramp` + profiles, the `decdn.publisher` collection. Front door: `docs/publishers.md`. + +There are four deploy paths: **Ansible** (`ansible/`, VMs/bare metal, the primary path, also the `decdn.node` and `decdn.publisher` Galaxy collections, and the only path that also deploys self-hosted iroh relays and the iroh DNS server), **cloud-init** @@ -90,7 +103,7 @@ monitoring/ # Grafana dashboards + Prometheus alert rules (maintaine decdn-node/ # the node's (+ promtool unit tests, .helmignored), rendered by the chart sponsord/ # sponsord's (+ promtool unit tests), Ansible/Compose only iroh-relay/ # the relay's (+ promtool unit tests, exported-metrics.txt), Ansible only -docs/ # cross-path operator docs: requirements.md, lifecycle.md +docs/ # cross-path operator docs: node-operators.md + publishers.md (the front doors), requirements.md, lifecycle.md scripts/ # upstream-mirror generators, the release script (release.sh, git-cliff: ../cliff.toml) and its gate (+ its Artifact Hub changes generator), the molecule driver (molecule.sh) ``` diff --git a/README.md b/README.md index 8e5e6bb..da6dbfd 100644 --- a/README.md +++ b/README.md @@ -8,15 +8,24 @@ [![shellcheck](https://img.shields.io/badge/shellcheck-passing-brightgreen)](https://www.shellcheck.net/) [![Conventional Commits](https://img.shields.io/badge/Conventional%20Commits-1.0.0-yellow.svg)](https://www.conventionalcommits.org) -The official **DevOps repo** for running a deCDN node: infrastructure, deployment and -day-2 tooling for operators anywhere. Four ways to deploy the same node: +The official **DevOps repo** for running deCDN: infrastructure, deployment and day-2 +tooling for operators anywhere. It serves both kinds of operator: + +| You are | You run | Start here | +|---------|---------|------------| +| a **node operator** | deCDN cache nodes: bond, serve, get paid per megabyte | [`docs/node-operators.md`](docs/node-operators.md) | +| a **publisher** | origin nodes for your namespaces, and optionally the services around them (`sponsord` and its onramp, iroh relays, an iroh DNS server) | [`docs/publishers.md`](docs/publishers.md) | + +Four ways to deploy the node, cache or origin: | Path | For | Start here | |------|-----|------------| | **Ansible** (`ansible/`) | VMs and bare metal, one node or a fleet. Hardens the host too (firewall, SSH, patching). Also published as the `decdn.node` and `decdn.publisher` Galaxy collections. | [`ansible/README.md`](ansible/README.md) | -| **cloud-init** (`cloud-init/`) | One VM, no control machine: paste the user-data into your provider's "create server" form. It runs the Ansible playbook on the host itself, hardening included. A second template boots the onboarding sponsor (`sponsord` + its onramp). | [`cloud-init/README.md`](cloud-init/README.md) | -| **Docker Compose** (`compose/`) | One host that already runs Docker. Also runs the onboarding sponsor (`sponsord` + its onramp) behind profiles. | [`compose/README.md`](compose/README.md) | -| **Helm** (`charts/decdn-node/`) | Kubernetes, one release per node. | [`charts/decdn-node/README.md`](charts/decdn-node/README.md) | +| **cloud-init** (`cloud-init/`) | One VM, no control machine: paste the user-data into your provider's "create server" form. It runs the Ansible playbook on the host itself, hardening included. One template per kind of operator: a cache node, or a publisher's origin with the onboarding sponsor (`sponsord` + its onramp). | [`cloud-init/README.md`](cloud-init/README.md) | +| **Docker Compose** (`compose/`) | One host that already runs Docker. Profiles for a cache node or an origin, and the onboarding sponsor (`sponsord` + its onramp). | [`compose/README.md`](compose/README.md) | +| **Helm** (`charts/decdn-node/`) | Kubernetes, one release per node, cache or origin. | [`charts/decdn-node/README.md`](charts/decdn-node/README.md) | + +The iroh relay and DNS server a publisher may run deploy with Ansible only. Supported hosts: Debian 12/13 and Ubuntu 24.04/26.04 on x86_64 or aarch64. Network, disk and platform requirements, and how to choose a path: @@ -110,7 +119,7 @@ This is the canonical statement; the per-path READMEs add only what is specific | [`compose/`](compose/README.md) | The Docker Compose deploy path: the node, `sponsord`, `sponsord-onramp` and Caddy, one profile each. | | [`charts/decdn-node/`](charts/decdn-node/README.md) | The Helm chart; it renders the node's dashboards and alert rules from `monitoring/`. | | [`monitoring/`](monitoring/README.md) | The deCDN Grafana dashboards and Prometheus alert rules, for the node (`decdn-node/`, every deploy path), sponsord (`sponsord/`, Ansible and Compose) and the iroh relay (`iroh-relay/`, Ansible). | -| [`docs/`](docs/requirements.md) | Cross-path operator docs: requirements, lifecycle. | +| [`docs/`](docs/requirements.md) | Cross-path operator docs: the two front doors ([node operators](docs/node-operators.md), [publishers](docs/publishers.md)), requirements, lifecycle. | | `scripts/` | Generators for the upstream mirrors (network profiles) and the release gate. | | `Makefile` | Lint, test and security targets; CI runs the same ones. `make help` lists them. | | `.github/workflows/` | CI (`ci.yml`, `molecule.yml`), releases (`release-collection.yml`, `release-chart.yml`), the weekly upstream drift check. | diff --git a/docs/lifecycle.md b/docs/lifecycle.md index 68987d0..328d0fa 100644 --- a/docs/lifecycle.md +++ b/docs/lifecycle.md @@ -167,6 +167,15 @@ steps with the `decdn` CLI, both with `--dry-run` first (use `decdn_chain` from Delete the keystore only after the withdrawal has landed. The public `udp/4433` firewall rule stays until baseline is re-run without it. +An **origin node** (`decdn_origin_nodes`) decommissions the same way, and the same +command covers it. Its operator also stays seated in `OriginAssignment` for its +namespaces. The namespace's publisher unseats it with `decdn publish revoke` +(`OriginAssignment.removeOrigin`; governance may too, and once the operator +deregisters anyone may prune it as inactive) +([ADR 011 § Origin Assignment Authority](https://github.com/decdn/decdn/blob/main/adr/011-content-takedown.md#origin-assignment-authority)), +ideally after another origin is seated, since a namespace with no origin left becomes +unassigned. + Silence `DecdnNodeDown` for the host before you run `make decommission`: the [`*Down` alerts](../monitoring/README.md#the-down-alerts) also fire on a series that went silent, so it fires for up to a day afterwards. diff --git a/docs/node-operators.md b/docs/node-operators.md new file mode 100644 index 0000000..e0fe390 --- /dev/null +++ b/docs/node-operators.md @@ -0,0 +1,57 @@ +# Node operators + +You run deCDN **cache nodes**: you bond stake, serve the content clients ask for, and are +paid per megabyte. You hold no content of your own. A miss is filled from other nodes, +and an origin is the canonical source. The cache role is permissionless: any bonded +operator may re-serve any blob it holds +([ADR 001](https://github.com/decdn/decdn/blob/main/adr/001-network.md), +[ADR 002 § Publisher Identity and Namespaces](https://github.com/decdn/decdn/blob/main/adr/002-content-addressing.md#publisher-identity-and-namespaces)). + +If you publish content and run its origins, read [`publishers.md`](publishers.md) +instead. + +## What you deploy + +One service per host: `decdn-node`, with no origin backend. Every path deploys it the same +way: a non-root daemon, public QUIC on **udp/4433**, metrics and the admin RPC on +loopback, and the chain's contract addresses taken from upstream's deployment manifest. + +| Path | Start here | Entry point | +|------|------------|-------------| +| **Ansible** (VMs, bare metal, a fleet) | [`ansible/README.md`](../ansible/README.md) | `inventory/hosts-node.yml.example`, `make deploy-node` (`playbooks/node.yml`) | +| **cloud-init** (one VM, no control machine) | [`cloud-init/README.md`](../cloud-init/README.md) | `cloud-init/user-data-node.yaml` | +| **Docker Compose** (one Docker host) | [`compose/README.md`](../compose/README.md) | `COMPOSE_PROFILES=node` | +| **Helm** (Kubernetes, one release per node) | [`charts/decdn-node/README.md`](../charts/decdn-node/README.md) | the chart, with no `config.cache.origin` | + +[`requirements.md`](requirements.md) has the hardware, network and platform requirements, +and compares the paths. The Galaxy collection for your own playbooks is +[`decdn.node`](../ansible/galaxy/node/README.md). + +A node without an origin backend fills a miss by pulling from other nodes +(`node_to_node_pull_through_enabled`). The Ansible role and the chart turn that on for +you. On Compose, set it in `node.toml`; the daemon's default is off. A pull is paid +from the node's own PaymentPool (the node-to-node tier of +[ADR 003](https://github.com/decdn/decdn/blob/main/adr/003-payments.md)), so the node +fronts USDC for its misses, up to `buyer_working_deposit_micro_usdc`, and pull-through +needs that buyer pool in place. + +## Before it earns + +The node serves paid traffic only after its on-chain stake and registration +([ADR 019, Phase 2](https://github.com/decdn/decdn/blob/main/adr/019-node-onboarding.md#phase-2--on-chain-setup)). +That is your step on every path: upstream's `decdn setup` walks it end to end, and +[`lifecycle.md`](lifecycle.md#running-on-chain-commands) shows how to run it on each +path. This repository stops at host preparation and startup. + +## Day 2 + +- Backups, restores, host migration and decommissioning: [`lifecycle.md`](lifecycle.md). +- Dashboards and alert rules: [`monitoring/`](../monitoring/README.md). Grafana Cloud + shipping is opt-in on Ansible (`decdn_grafana_cloud_enabled`). +- Relays and discovery: a node uses n0's iroh relays and DNS discovery by default. To + move to self-hosted ones, set `decdn_relay_urls` and the `decdn_discovery_*` knobs on + Ansible (`network.relay_urls` and `[network.discovery]` in `node.toml` or the chart's + `config`). Two caveats (`roles/iroh_relay/README.md`, `roles/iroh_dns_server/README.md`): + a publisher's relay admits only listed node IDs by default, so ask for yours to be + added; and your own discovery settings drop the n0 leg, so clients must use the same + `dns_origin` to find your node. diff --git a/docs/publishers.md b/docs/publishers.md new file mode 100644 index 0000000..82ad548 --- /dev/null +++ b/docs/publishers.md @@ -0,0 +1,79 @@ +# Publishers + +You publish content on deCDN. A **publisher** is an Ethereum address that owns at least +one namespace in the `PublisherRegistry` contract. It becomes one with its first +`createNamespace()` call. Once governance has vetted you as a publisher, you seat the +operators that serve each namespace as **origins** +([ADR 002 § Publisher Identity and Namespaces](https://github.com/decdn/decdn/blob/main/adr/002-content-addressing.md#publisher-identity-and-namespaces), +[ADR 011 § Origin Assignment Authority](https://github.com/decdn/decdn/blob/main/adr/011-content-takedown.md#origin-assignment-authority)). +You take your steps with the `decdn` CLI, not with this repository +([Appendix: Binaries](https://github.com/decdn/decdn/blob/main/adr/appendix-binaries.md)): +`decdn publish namespace create`, `decdn publish assign` (`OriginAssignment.addOrigin`), +`decdn publish revoke` (`removeOrigin`), and `decdn origin import` to seed an origin's +store. The vetting itself is governance's. + +This repository deploys what a publisher runs. Node operators who only run cache nodes +should read [`node-operators.md`](node-operators.md) instead. + +## What you deploy + +| Service | What it is | Ansible group and playbook | Other paths | +|---------|------------|----------------------------|-------------| +| **Origin node** | `decdn-node` with an origin backend (S3/R2/B2/MinIO, NFS or local disk, or an HTTP store): the canonical source of your namespace's content | `decdn_origin_nodes`, a child of `decdn_nodes`; `playbooks/origin.yml` | cloud-init `user-data-publisher.yaml`; Compose `origin` profile; the Helm chart with `config.cache.origin` | +| **sponsord** + **sponsord-onramp** | Pays for your users' downloads: the treasury wallet that owns a PaymentPool and signs capped capabilities for clients ([ADR 003 § Capability delegation](https://github.com/decdn/decdn/blob/main/adr/003-payments.md#capability-delegation)), and its public gate (Turnstile, installers) behind Caddy | `sponsord_hosts`, `sponsord_onramp_hosts`; `playbooks/sponsord.yml` | cloud-init `user-data-publisher.yaml`; Compose `sponsord`, `onramp`, `caddy` profiles | +| **iroh relay** | NAT-traversal fallback for peers that cannot hole-punch. Relays are operational infrastructure, not an incentivized role ([Architecture § Trust Assumptions](https://github.com/decdn/decdn/blob/main/adr/architecture.md#trust-assumptions)) | `iroh_relay_hosts`; `playbooks/iroh_relay.yml` | Ansible only | +| **iroh DNS server** | The pkarr relay and DNS server nodes publish and resolve their address records through, in place of n0's ([ADR 001 § Node Discovery](https://github.com/decdn/decdn/blob/main/adr/001-network.md#node-discovery-registry)) | `iroh_dns_server_hosts`; `playbooks/iroh_dns_server.yml` | Ansible only | + +Only the origin is required. With Ansible, start from +`ansible/inventory/hosts-publisher.yml.example`; `make deploy-publisher` +(`playbooks/publisher.yml`) deploys all four in that order, and each has its own target +(`deploy-origin`, `deploy-sponsord`, `deploy-relay`, `deploy-dns`). The roles also ship as +the [`decdn.publisher`](../ansible/galaxy/publisher/README.md) Galaxy collection, which +depends on [`decdn.node`](../ansible/galaxy/node/README.md) for the baseline, observability +and the origin node itself. + +## Origin nodes + +An origin is a deCDN node like any other: it is bonded and registered +([ADR 019](https://github.com/decdn/decdn/blob/main/adr/019-node-onboarding.md)), with the +same hardening, port and day-2 tooling as a cache node. Three things differ: + +- **The backend.** Set `decdn_cache_origin_kind` (`http`, `fs` or `s3`) and its fields, + or `decdn_cache_origins` for an ordered fallback list (`roles/decdn_node/README.md`). + S3 keys go in `/etc/decdn/decdn.env` on the host, never in inventory or user-data. + `playbooks/origin.yml` refuses a `decdn_origin_nodes` host without a backend, or one + outside `decdn_nodes`. +- **What it serves.** With a backend, the node fills misses from it rather than from + other nodes. By default it serves only what its own backend holds and declines + everything else, even a blob already in its cache (`relay_foreign_namespaces`, + [ADR 002 § Retrieval by namespace](https://github.com/decdn/decdn/blob/main/adr/002-content-addressing.md#retrieval-by-namespace), + [ADR 037 § Warming is a relay-edge mechanism](https://github.com/decdn/decdn/blob/main/adr/037-regional-proxy-warming.md#warming-is-a-relay-edge-mechanism)). + The gate is the backend's content, not the namespace's on-chain assignment. +- **Recognition is on-chain.** As the namespace's publisher, you seat the node's operator + with `decdn publish assign` (`OriginAssignment.addOrigin`). Configuring a backend without that only means the + node's bytes are served as cache + ([Architecture § Origin Backends](https://github.com/decdn/decdn/blob/main/adr/architecture.md#origin-backends)). + Multiple operators may serve one namespace, so run more than one origin for + redundancy. + +## Relays and the DNS server + +A node's `decdn_relay_urls` replaces n0's relays, so deploy at least two relays before +pointing nodes at them. By default a relay admits only the nodes in the same inventory +(`iroh_relay_access: allowlist`): it reads their IDs from `decdn_nodes`, your origins +included. Other operators' nodes need their IDs in `iroh_relay_allowlist`, or +`everyone`. Clients can never be listed (they use a fresh key per fetch), so a node +behind NAT that serves clients needs its relays in `everyone` mode. The DNS server has +no replication: each DNS origin (`dns_origin`) is served by exactly one server, a single +point of failure for pkarr lookups. Give every node and client that uses it the same +`dns_origin`. Both roles' READMEs have the +details: `roles/iroh_relay/README.md`, `roles/iroh_dns_server/README.md`. + +## Day 2 + +[`lifecycle.md`](lifecycle.md) covers backup, restore, migration and decommission for +every service above. Decommissioning an origin stops the node but leaves its operator +seated on-chain, where `OriginAssignment.getOrigins` still lists it. Unseat it with +`decdn publish revoke` (`OriginAssignment.removeOrigin`); once the operator deregisters, +anyone may also prune it as inactive +([ADR 011 § Origin Assignment Authority](https://github.com/decdn/decdn/blob/main/adr/011-content-takedown.md#origin-assignment-authority)). diff --git a/docs/requirements.md b/docs/requirements.md index a5663ba..abacc29 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -1,8 +1,10 @@ # Requirements and choosing a deploy path What a deCDN node needs from its host and network, and which of this repo's four -deploy paths fits. Protocol facts (bond sizing, fees) are not here: they come from the -deCDN ADRs. +deploy paths fits. It applies to a node operator's cache node and a publisher's origin +node alike: an origin is the same node with an origin backend +([`node-operators.md`](node-operators.md), [`publishers.md`](publishers.md)). Protocol +facts (bond sizing, fees) are not here: they come from the deCDN ADRs. ## Choosing a path From 5e064f5229cef63d1baf69458ce882df8ada7e1c Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Sat, 10 Oct 2026 19:45:58 +0300 Subject: [PATCH 2/2] docs: the deposit is a refill target, origin import writes a local store, and role links resolve Co-Authored-By: Claude Opus 5.5 --- docs/node-operators.md | 8 +++++--- docs/publishers.md | 12 ++++++++---- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/node-operators.md b/docs/node-operators.md index e0fe390..0490e0a 100644 --- a/docs/node-operators.md +++ b/docs/node-operators.md @@ -32,8 +32,9 @@ A node without an origin backend fills a miss by pulling from other nodes you. On Compose, set it in `node.toml`; the daemon's default is off. A pull is paid from the node's own PaymentPool (the node-to-node tier of [ADR 003](https://github.com/decdn/decdn/blob/main/adr/003-payments.md)), so the node -fronts USDC for its misses, up to `buyer_working_deposit_micro_usdc`, and pull-through -needs that buyer pool in place. +fronts USDC for its misses. `buyer_working_deposit_micro_usdc` is not a cap: it is the +deposit the pool opens with and the balance every top-up refills it to, so keep the +node's wallet funded for as long as it pulls. ## Before it earns @@ -51,7 +52,8 @@ path. This repository stops at host preparation and startup. - Relays and discovery: a node uses n0's iroh relays and DNS discovery by default. To move to self-hosted ones, set `decdn_relay_urls` and the `decdn_discovery_*` knobs on Ansible (`network.relay_urls` and `[network.discovery]` in `node.toml` or the chart's - `config`). Two caveats (`roles/iroh_relay/README.md`, `roles/iroh_dns_server/README.md`): + `config`). Two caveats ([`iroh_relay`](../ansible/roles/iroh_relay/README.md), + [`iroh_dns_server`](../ansible/roles/iroh_dns_server/README.md)): a publisher's relay admits only listed node IDs by default, so ask for yours to be added; and your own discovery settings drop the n0 leg, so clients must use the same `dns_origin` to find your node. diff --git a/docs/publishers.md b/docs/publishers.md index 82ad548..a665105 100644 --- a/docs/publishers.md +++ b/docs/publishers.md @@ -9,8 +9,10 @@ operators that serve each namespace as **origins** You take your steps with the `decdn` CLI, not with this repository ([Appendix: Binaries](https://github.com/decdn/decdn/blob/main/adr/appendix-binaries.md)): `decdn publish namespace create`, `decdn publish assign` (`OriginAssignment.addOrigin`), -`decdn publish revoke` (`removeOrigin`), and `decdn origin import` to seed an origin's -store. The vetting itself is governance's. +`decdn publish revoke` (`removeOrigin`), and `decdn origin import`, which writes +content into a local directory in the layout an origin reads: point an `fs` origin at +it, or sync it to an S3 bucket (the layouts are the same). It does not write to S3 or +an HTTP store itself. The vetting itself is governance's. This repository deploys what a publisher runs. Node operators who only run cache nodes should read [`node-operators.md`](node-operators.md) instead. @@ -39,7 +41,8 @@ An origin is a deCDN node like any other: it is bonded and registered same hardening, port and day-2 tooling as a cache node. Three things differ: - **The backend.** Set `decdn_cache_origin_kind` (`http`, `fs` or `s3`) and its fields, - or `decdn_cache_origins` for an ordered fallback list (`roles/decdn_node/README.md`). + or `decdn_cache_origins` for an ordered fallback list + ([`decdn_node` role](../ansible/roles/decdn_node/README.md)). S3 keys go in `/etc/decdn/decdn.env` on the host, never in inventory or user-data. `playbooks/origin.yml` refuses a `decdn_origin_nodes` host without a backend, or one outside `decdn_nodes`. @@ -67,7 +70,8 @@ behind NAT that serves clients needs its relays in `everyone` mode. The DNS serv no replication: each DNS origin (`dns_origin`) is served by exactly one server, a single point of failure for pkarr lookups. Give every node and client that uses it the same `dns_origin`. Both roles' READMEs have the -details: `roles/iroh_relay/README.md`, `roles/iroh_dns_server/README.md`. +details: [`iroh_relay`](../ansible/roles/iroh_relay/README.md), +[`iroh_dns_server`](../ansible/roles/iroh_dns_server/README.md). ## Day 2