diff --git a/AGENTS.md b/AGENTS.md index 0b851e7..8c92b16 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,8 +5,21 @@ DevOps repo. ## What this repo is -The official DevOps project for deploying a **deCDN node**: infrastructure, deployment, -and operational tooling, for node operators anywhere. There are four deploy paths: +The official DevOps project for deploying deCDN: infrastructure, deployment, and +operational tooling. It serves **two kinds of operator**, and every entry point is +split along that line: + +- **Node operators** run cache nodes (`decdn-node`, no origin backend): `node.yml` / + `make deploy-node`, `cloud-init/user-data-node.yaml`, Compose's `node` profile, the + chart, the `decdn.node` collection. Front door: `docs/node-operators.md`. +- **Publishers** own namespaces in `PublisherRegistry` and run **origin nodes** (the + same daemon with an origin backend, seated on-chain via `OriginAssignment.addOrigin`, + ADR 002/011), and optionally `sponsord` + its onramp, iroh relays and an iroh DNS + server: `publisher.yml` / `make deploy-publisher` (`origin.yml` + the component + playbooks), `cloud-init/user-data-publisher.yaml`, Compose's `origin`/`onramp` + profiles, the `decdn.publisher` collection. Front door: `docs/publishers.md`. + +There are four deploy paths: **Ansible** (`ansible/`, VMs/bare metal, the primary path, also the `decdn.node` and `decdn.publisher` Galaxy collections, and the only path that also deploys self-hosted iroh relays and the iroh DNS server), **cloud-init** @@ -90,7 +103,7 @@ monitoring/ # Grafana dashboards + Prometheus alert rules (maintaine decdn-node/ # the node's (+ promtool unit tests, .helmignored), rendered by the chart sponsord/ # sponsord's (+ promtool unit tests), Ansible/Compose only iroh-relay/ # the relay's (+ promtool unit tests, exported-metrics.txt), Ansible only -docs/ # cross-path operator docs: requirements.md, lifecycle.md +docs/ # cross-path operator docs: node-operators.md + publishers.md (the front doors), requirements.md, lifecycle.md scripts/ # upstream-mirror generators, the release script (release.sh, git-cliff: ../cliff.toml) and its gate (+ its Artifact Hub changes generator), the molecule driver (molecule.sh) ``` diff --git a/README.md b/README.md index 8e5e6bb..da6dbfd 100644 --- a/README.md +++ b/README.md @@ -8,15 +8,24 @@ [![shellcheck](https://img.shields.io/badge/shellcheck-passing-brightgreen)](https://www.shellcheck.net/) [![Conventional Commits](https://img.shields.io/badge/Conventional%20Commits-1.0.0-yellow.svg)](https://www.conventionalcommits.org) -The official **DevOps repo** for running a deCDN node: infrastructure, deployment and -day-2 tooling for operators anywhere. Four ways to deploy the same node: +The official **DevOps repo** for running deCDN: infrastructure, deployment and day-2 +tooling for operators anywhere. It serves both kinds of operator: + +| You are | You run | Start here | +|---------|---------|------------| +| a **node operator** | deCDN cache nodes: bond, serve, get paid per megabyte | [`docs/node-operators.md`](docs/node-operators.md) | +| a **publisher** | origin nodes for your namespaces, and optionally the services around them (`sponsord` and its onramp, iroh relays, an iroh DNS server) | [`docs/publishers.md`](docs/publishers.md) | + +Four ways to deploy the node, cache or origin: | Path | For | Start here | |------|-----|------------| | **Ansible** (`ansible/`) | VMs and bare metal, one node or a fleet. Hardens the host too (firewall, SSH, patching). Also published as the `decdn.node` and `decdn.publisher` Galaxy collections. | [`ansible/README.md`](ansible/README.md) | -| **cloud-init** (`cloud-init/`) | One VM, no control machine: paste the user-data into your provider's "create server" form. It runs the Ansible playbook on the host itself, hardening included. A second template boots the onboarding sponsor (`sponsord` + its onramp). | [`cloud-init/README.md`](cloud-init/README.md) | -| **Docker Compose** (`compose/`) | One host that already runs Docker. Also runs the onboarding sponsor (`sponsord` + its onramp) behind profiles. | [`compose/README.md`](compose/README.md) | -| **Helm** (`charts/decdn-node/`) | Kubernetes, one release per node. | [`charts/decdn-node/README.md`](charts/decdn-node/README.md) | +| **cloud-init** (`cloud-init/`) | One VM, no control machine: paste the user-data into your provider's "create server" form. It runs the Ansible playbook on the host itself, hardening included. One template per kind of operator: a cache node, or a publisher's origin with the onboarding sponsor (`sponsord` + its onramp). | [`cloud-init/README.md`](cloud-init/README.md) | +| **Docker Compose** (`compose/`) | One host that already runs Docker. Profiles for a cache node or an origin, and the onboarding sponsor (`sponsord` + its onramp). | [`compose/README.md`](compose/README.md) | +| **Helm** (`charts/decdn-node/`) | Kubernetes, one release per node, cache or origin. | [`charts/decdn-node/README.md`](charts/decdn-node/README.md) | + +The iroh relay and DNS server a publisher may run deploy with Ansible only. Supported hosts: Debian 12/13 and Ubuntu 24.04/26.04 on x86_64 or aarch64. Network, disk and platform requirements, and how to choose a path: @@ -110,7 +119,7 @@ This is the canonical statement; the per-path READMEs add only what is specific | [`compose/`](compose/README.md) | The Docker Compose deploy path: the node, `sponsord`, `sponsord-onramp` and Caddy, one profile each. | | [`charts/decdn-node/`](charts/decdn-node/README.md) | The Helm chart; it renders the node's dashboards and alert rules from `monitoring/`. | | [`monitoring/`](monitoring/README.md) | The deCDN Grafana dashboards and Prometheus alert rules, for the node (`decdn-node/`, every deploy path), sponsord (`sponsord/`, Ansible and Compose) and the iroh relay (`iroh-relay/`, Ansible). | -| [`docs/`](docs/requirements.md) | Cross-path operator docs: requirements, lifecycle. | +| [`docs/`](docs/requirements.md) | Cross-path operator docs: the two front doors ([node operators](docs/node-operators.md), [publishers](docs/publishers.md)), requirements, lifecycle. | | `scripts/` | Generators for the upstream mirrors (network profiles) and the release gate. | | `Makefile` | Lint, test and security targets; CI runs the same ones. `make help` lists them. | | `.github/workflows/` | CI (`ci.yml`, `molecule.yml`), releases (`release-collection.yml`, `release-chart.yml`), the weekly upstream drift check. | diff --git a/docs/lifecycle.md b/docs/lifecycle.md index 68987d0..328d0fa 100644 --- a/docs/lifecycle.md +++ b/docs/lifecycle.md @@ -167,6 +167,15 @@ steps with the `decdn` CLI, both with `--dry-run` first (use `decdn_chain` from Delete the keystore only after the withdrawal has landed. The public `udp/4433` firewall rule stays until baseline is re-run without it. +An **origin node** (`decdn_origin_nodes`) decommissions the same way, and the same +command covers it. Its operator also stays seated in `OriginAssignment` for its +namespaces. The namespace's publisher unseats it with `decdn publish revoke` +(`OriginAssignment.removeOrigin`; governance may too, and once the operator +deregisters anyone may prune it as inactive) +([ADR 011 § Origin Assignment Authority](https://github.com/decdn/decdn/blob/main/adr/011-content-takedown.md#origin-assignment-authority)), +ideally after another origin is seated, since a namespace with no origin left becomes +unassigned. + Silence `DecdnNodeDown` for the host before you run `make decommission`: the [`*Down` alerts](../monitoring/README.md#the-down-alerts) also fire on a series that went silent, so it fires for up to a day afterwards. diff --git a/docs/node-operators.md b/docs/node-operators.md new file mode 100644 index 0000000..0490e0a --- /dev/null +++ b/docs/node-operators.md @@ -0,0 +1,59 @@ +# Node operators + +You run deCDN **cache nodes**: you bond stake, serve the content clients ask for, and are +paid per megabyte. You hold no content of your own. A miss is filled from other nodes, +and an origin is the canonical source. The cache role is permissionless: any bonded +operator may re-serve any blob it holds +([ADR 001](https://github.com/decdn/decdn/blob/main/adr/001-network.md), +[ADR 002 § Publisher Identity and Namespaces](https://github.com/decdn/decdn/blob/main/adr/002-content-addressing.md#publisher-identity-and-namespaces)). + +If you publish content and run its origins, read [`publishers.md`](publishers.md) +instead. + +## What you deploy + +One service per host: `decdn-node`, with no origin backend. Every path deploys it the same +way: a non-root daemon, public QUIC on **udp/4433**, metrics and the admin RPC on +loopback, and the chain's contract addresses taken from upstream's deployment manifest. + +| Path | Start here | Entry point | +|------|------------|-------------| +| **Ansible** (VMs, bare metal, a fleet) | [`ansible/README.md`](../ansible/README.md) | `inventory/hosts-node.yml.example`, `make deploy-node` (`playbooks/node.yml`) | +| **cloud-init** (one VM, no control machine) | [`cloud-init/README.md`](../cloud-init/README.md) | `cloud-init/user-data-node.yaml` | +| **Docker Compose** (one Docker host) | [`compose/README.md`](../compose/README.md) | `COMPOSE_PROFILES=node` | +| **Helm** (Kubernetes, one release per node) | [`charts/decdn-node/README.md`](../charts/decdn-node/README.md) | the chart, with no `config.cache.origin` | + +[`requirements.md`](requirements.md) has the hardware, network and platform requirements, +and compares the paths. The Galaxy collection for your own playbooks is +[`decdn.node`](../ansible/galaxy/node/README.md). + +A node without an origin backend fills a miss by pulling from other nodes +(`node_to_node_pull_through_enabled`). The Ansible role and the chart turn that on for +you. On Compose, set it in `node.toml`; the daemon's default is off. A pull is paid +from the node's own PaymentPool (the node-to-node tier of +[ADR 003](https://github.com/decdn/decdn/blob/main/adr/003-payments.md)), so the node +fronts USDC for its misses. `buyer_working_deposit_micro_usdc` is not a cap: it is the +deposit the pool opens with and the balance every top-up refills it to, so keep the +node's wallet funded for as long as it pulls. + +## Before it earns + +The node serves paid traffic only after its on-chain stake and registration +([ADR 019, Phase 2](https://github.com/decdn/decdn/blob/main/adr/019-node-onboarding.md#phase-2--on-chain-setup)). +That is your step on every path: upstream's `decdn setup` walks it end to end, and +[`lifecycle.md`](lifecycle.md#running-on-chain-commands) shows how to run it on each +path. This repository stops at host preparation and startup. + +## Day 2 + +- Backups, restores, host migration and decommissioning: [`lifecycle.md`](lifecycle.md). +- Dashboards and alert rules: [`monitoring/`](../monitoring/README.md). Grafana Cloud + shipping is opt-in on Ansible (`decdn_grafana_cloud_enabled`). +- Relays and discovery: a node uses n0's iroh relays and DNS discovery by default. To + move to self-hosted ones, set `decdn_relay_urls` and the `decdn_discovery_*` knobs on + Ansible (`network.relay_urls` and `[network.discovery]` in `node.toml` or the chart's + `config`). Two caveats ([`iroh_relay`](../ansible/roles/iroh_relay/README.md), + [`iroh_dns_server`](../ansible/roles/iroh_dns_server/README.md)): + a publisher's relay admits only listed node IDs by default, so ask for yours to be + added; and your own discovery settings drop the n0 leg, so clients must use the same + `dns_origin` to find your node. diff --git a/docs/publishers.md b/docs/publishers.md new file mode 100644 index 0000000..a665105 --- /dev/null +++ b/docs/publishers.md @@ -0,0 +1,83 @@ +# Publishers + +You publish content on deCDN. A **publisher** is an Ethereum address that owns at least +one namespace in the `PublisherRegistry` contract. It becomes one with its first +`createNamespace()` call. Once governance has vetted you as a publisher, you seat the +operators that serve each namespace as **origins** +([ADR 002 § Publisher Identity and Namespaces](https://github.com/decdn/decdn/blob/main/adr/002-content-addressing.md#publisher-identity-and-namespaces), +[ADR 011 § Origin Assignment Authority](https://github.com/decdn/decdn/blob/main/adr/011-content-takedown.md#origin-assignment-authority)). +You take your steps with the `decdn` CLI, not with this repository +([Appendix: Binaries](https://github.com/decdn/decdn/blob/main/adr/appendix-binaries.md)): +`decdn publish namespace create`, `decdn publish assign` (`OriginAssignment.addOrigin`), +`decdn publish revoke` (`removeOrigin`), and `decdn origin import`, which writes +content into a local directory in the layout an origin reads: point an `fs` origin at +it, or sync it to an S3 bucket (the layouts are the same). It does not write to S3 or +an HTTP store itself. The vetting itself is governance's. + +This repository deploys what a publisher runs. Node operators who only run cache nodes +should read [`node-operators.md`](node-operators.md) instead. + +## What you deploy + +| Service | What it is | Ansible group and playbook | Other paths | +|---------|------------|----------------------------|-------------| +| **Origin node** | `decdn-node` with an origin backend (S3/R2/B2/MinIO, NFS or local disk, or an HTTP store): the canonical source of your namespace's content | `decdn_origin_nodes`, a child of `decdn_nodes`; `playbooks/origin.yml` | cloud-init `user-data-publisher.yaml`; Compose `origin` profile; the Helm chart with `config.cache.origin` | +| **sponsord** + **sponsord-onramp** | Pays for your users' downloads: the treasury wallet that owns a PaymentPool and signs capped capabilities for clients ([ADR 003 § Capability delegation](https://github.com/decdn/decdn/blob/main/adr/003-payments.md#capability-delegation)), and its public gate (Turnstile, installers) behind Caddy | `sponsord_hosts`, `sponsord_onramp_hosts`; `playbooks/sponsord.yml` | cloud-init `user-data-publisher.yaml`; Compose `sponsord`, `onramp`, `caddy` profiles | +| **iroh relay** | NAT-traversal fallback for peers that cannot hole-punch. Relays are operational infrastructure, not an incentivized role ([Architecture § Trust Assumptions](https://github.com/decdn/decdn/blob/main/adr/architecture.md#trust-assumptions)) | `iroh_relay_hosts`; `playbooks/iroh_relay.yml` | Ansible only | +| **iroh DNS server** | The pkarr relay and DNS server nodes publish and resolve their address records through, in place of n0's ([ADR 001 § Node Discovery](https://github.com/decdn/decdn/blob/main/adr/001-network.md#node-discovery-registry)) | `iroh_dns_server_hosts`; `playbooks/iroh_dns_server.yml` | Ansible only | + +Only the origin is required. With Ansible, start from +`ansible/inventory/hosts-publisher.yml.example`; `make deploy-publisher` +(`playbooks/publisher.yml`) deploys all four in that order, and each has its own target +(`deploy-origin`, `deploy-sponsord`, `deploy-relay`, `deploy-dns`). The roles also ship as +the [`decdn.publisher`](../ansible/galaxy/publisher/README.md) Galaxy collection, which +depends on [`decdn.node`](../ansible/galaxy/node/README.md) for the baseline, observability +and the origin node itself. + +## Origin nodes + +An origin is a deCDN node like any other: it is bonded and registered +([ADR 019](https://github.com/decdn/decdn/blob/main/adr/019-node-onboarding.md)), with the +same hardening, port and day-2 tooling as a cache node. Three things differ: + +- **The backend.** Set `decdn_cache_origin_kind` (`http`, `fs` or `s3`) and its fields, + or `decdn_cache_origins` for an ordered fallback list + ([`decdn_node` role](../ansible/roles/decdn_node/README.md)). + S3 keys go in `/etc/decdn/decdn.env` on the host, never in inventory or user-data. + `playbooks/origin.yml` refuses a `decdn_origin_nodes` host without a backend, or one + outside `decdn_nodes`. +- **What it serves.** With a backend, the node fills misses from it rather than from + other nodes. By default it serves only what its own backend holds and declines + everything else, even a blob already in its cache (`relay_foreign_namespaces`, + [ADR 002 § Retrieval by namespace](https://github.com/decdn/decdn/blob/main/adr/002-content-addressing.md#retrieval-by-namespace), + [ADR 037 § Warming is a relay-edge mechanism](https://github.com/decdn/decdn/blob/main/adr/037-regional-proxy-warming.md#warming-is-a-relay-edge-mechanism)). + The gate is the backend's content, not the namespace's on-chain assignment. +- **Recognition is on-chain.** As the namespace's publisher, you seat the node's operator + with `decdn publish assign` (`OriginAssignment.addOrigin`). Configuring a backend without that only means the + node's bytes are served as cache + ([Architecture § Origin Backends](https://github.com/decdn/decdn/blob/main/adr/architecture.md#origin-backends)). + Multiple operators may serve one namespace, so run more than one origin for + redundancy. + +## Relays and the DNS server + +A node's `decdn_relay_urls` replaces n0's relays, so deploy at least two relays before +pointing nodes at them. By default a relay admits only the nodes in the same inventory +(`iroh_relay_access: allowlist`): it reads their IDs from `decdn_nodes`, your origins +included. Other operators' nodes need their IDs in `iroh_relay_allowlist`, or +`everyone`. Clients can never be listed (they use a fresh key per fetch), so a node +behind NAT that serves clients needs its relays in `everyone` mode. The DNS server has +no replication: each DNS origin (`dns_origin`) is served by exactly one server, a single +point of failure for pkarr lookups. Give every node and client that uses it the same +`dns_origin`. Both roles' READMEs have the +details: [`iroh_relay`](../ansible/roles/iroh_relay/README.md), +[`iroh_dns_server`](../ansible/roles/iroh_dns_server/README.md). + +## Day 2 + +[`lifecycle.md`](lifecycle.md) covers backup, restore, migration and decommission for +every service above. Decommissioning an origin stops the node but leaves its operator +seated on-chain, where `OriginAssignment.getOrigins` still lists it. Unseat it with +`decdn publish revoke` (`OriginAssignment.removeOrigin`); once the operator deregisters, +anyone may also prune it as inactive +([ADR 011 § Origin Assignment Authority](https://github.com/decdn/decdn/blob/main/adr/011-content-takedown.md#origin-assignment-authority)). diff --git a/docs/requirements.md b/docs/requirements.md index a5663ba..abacc29 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -1,8 +1,10 @@ # Requirements and choosing a deploy path What a deCDN node needs from its host and network, and which of this repo's four -deploy paths fits. Protocol facts (bond sizing, fees) are not here: they come from the -deCDN ADRs. +deploy paths fits. It applies to a node operator's cache node and a publisher's origin +node alike: an origin is the same node with an origin backend +([`node-operators.md`](node-operators.md), [`publishers.md`](publishers.md)). Protocol +facts (bond sizing, fees) are not here: they come from the deCDN ADRs. ## Choosing a path