Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,21 @@ DevOps repo.

## What this repo is

The official DevOps project for deploying a **deCDN node**: infrastructure, deployment,
and operational tooling, for node operators anywhere. There are four deploy paths:
The official DevOps project for deploying deCDN: infrastructure, deployment, and
operational tooling. It serves **two kinds of operator**, and every entry point is
split along that line:

- **Node operators** run cache nodes (`decdn-node`, no origin backend): `node.yml` /
`make deploy-node`, `cloud-init/user-data-node.yaml`, Compose's `node` profile, the
chart, the `decdn.node` collection. Front door: `docs/node-operators.md`.
- **Publishers** own namespaces in `PublisherRegistry` and run **origin nodes** (the
same daemon with an origin backend, seated on-chain via `OriginAssignment.addOrigin`,
ADR 002/011), and optionally `sponsord` + its onramp, iroh relays and an iroh DNS
server: `publisher.yml` / `make deploy-publisher` (`origin.yml` + the component
playbooks), `cloud-init/user-data-publisher.yaml`, Compose's `origin`/`onramp`
profiles, the `decdn.publisher` collection. Front door: `docs/publishers.md`.

There are four deploy paths:
**Ansible** (`ansible/`, VMs/bare metal, the primary path, also the `decdn.node` and
`decdn.publisher` Galaxy collections, and the only path that also deploys self-hosted iroh relays and the iroh
DNS server), **cloud-init**
Expand Down Expand Up @@ -90,7 +103,7 @@ monitoring/ # Grafana dashboards + Prometheus alert rules (maintaine
decdn-node/ # the node's (+ promtool unit tests, .helmignored), rendered by the chart
sponsord/ # sponsord's (+ promtool unit tests), Ansible/Compose only
iroh-relay/ # the relay's (+ promtool unit tests, exported-metrics.txt), Ansible only
docs/ # cross-path operator docs: requirements.md, lifecycle.md
docs/ # cross-path operator docs: node-operators.md + publishers.md (the front doors), requirements.md, lifecycle.md
scripts/ # upstream-mirror generators, the release script (release.sh, git-cliff: ../cliff.toml) and its gate (+ its Artifact Hub changes generator), the molecule driver (molecule.sh)
```

Expand Down
21 changes: 15 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,24 @@
[![shellcheck](https://img.shields.io/badge/shellcheck-passing-brightgreen)](https://www.shellcheck.net/)
[![Conventional Commits](https://img.shields.io/badge/Conventional%20Commits-1.0.0-yellow.svg)](https://www.conventionalcommits.org)

The official **DevOps repo** for running a deCDN node: infrastructure, deployment and
day-2 tooling for operators anywhere. Four ways to deploy the same node:
The official **DevOps repo** for running deCDN: infrastructure, deployment and day-2
tooling for operators anywhere. It serves both kinds of operator:

| You are | You run | Start here |
|---------|---------|------------|
| a **node operator** | deCDN cache nodes: bond, serve, get paid per megabyte | [`docs/node-operators.md`](docs/node-operators.md) |
| a **publisher** | origin nodes for your namespaces, and optionally the services around them (`sponsord` and its onramp, iroh relays, an iroh DNS server) | [`docs/publishers.md`](docs/publishers.md) |

Four ways to deploy the node, cache or origin:

| Path | For | Start here |
|------|-----|------------|
| **Ansible** (`ansible/`) | VMs and bare metal, one node or a fleet. Hardens the host too (firewall, SSH, patching). Also published as the `decdn.node` and `decdn.publisher` Galaxy collections. | [`ansible/README.md`](ansible/README.md) |
| **cloud-init** (`cloud-init/`) | One VM, no control machine: paste the user-data into your provider's "create server" form. It runs the Ansible playbook on the host itself, hardening included. A second template boots the onboarding sponsor (`sponsord` + its onramp). | [`cloud-init/README.md`](cloud-init/README.md) |
| **Docker Compose** (`compose/`) | One host that already runs Docker. Also runs the onboarding sponsor (`sponsord` + its onramp) behind profiles. | [`compose/README.md`](compose/README.md) |
| **Helm** (`charts/decdn-node/`) | Kubernetes, one release per node. | [`charts/decdn-node/README.md`](charts/decdn-node/README.md) |
| **cloud-init** (`cloud-init/`) | One VM, no control machine: paste the user-data into your provider's "create server" form. It runs the Ansible playbook on the host itself, hardening included. One template per kind of operator: a cache node, or a publisher's origin with the onboarding sponsor (`sponsord` + its onramp). | [`cloud-init/README.md`](cloud-init/README.md) |
| **Docker Compose** (`compose/`) | One host that already runs Docker. Profiles for a cache node or an origin, and the onboarding sponsor (`sponsord` + its onramp). | [`compose/README.md`](compose/README.md) |
| **Helm** (`charts/decdn-node/`) | Kubernetes, one release per node, cache or origin. | [`charts/decdn-node/README.md`](charts/decdn-node/README.md) |

The iroh relay and DNS server a publisher may run deploy with Ansible only.

Supported hosts: Debian 12/13 and Ubuntu 24.04/26.04 on x86_64 or aarch64. Network,
disk and platform requirements, and how to choose a path:
Expand Down Expand Up @@ -110,7 +119,7 @@ This is the canonical statement; the per-path READMEs add only what is specific
| [`compose/`](compose/README.md) | The Docker Compose deploy path: the node, `sponsord`, `sponsord-onramp` and Caddy, one profile each. |
| [`charts/decdn-node/`](charts/decdn-node/README.md) | The Helm chart; it renders the node's dashboards and alert rules from `monitoring/`. |
| [`monitoring/`](monitoring/README.md) | The deCDN Grafana dashboards and Prometheus alert rules, for the node (`decdn-node/`, every deploy path), sponsord (`sponsord/`, Ansible and Compose) and the iroh relay (`iroh-relay/`, Ansible). |
| [`docs/`](docs/requirements.md) | Cross-path operator docs: requirements, lifecycle. |
| [`docs/`](docs/requirements.md) | Cross-path operator docs: the two front doors ([node operators](docs/node-operators.md), [publishers](docs/publishers.md)), requirements, lifecycle. |
| `scripts/` | Generators for the upstream mirrors (network profiles) and the release gate. |
| `Makefile` | Lint, test and security targets; CI runs the same ones. `make help` lists them. |
| `.github/workflows/` | CI (`ci.yml`, `molecule.yml`), releases (`release-collection.yml`, `release-chart.yml`), the weekly upstream drift check. |
Expand Down
9 changes: 9 additions & 0 deletions docs/lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,15 @@ steps with the `decdn` CLI, both with `--dry-run` first (use `decdn_chain` from
Delete the keystore only after the withdrawal has landed. The public `udp/4433`
firewall rule stays until baseline is re-run without it.

An **origin node** (`decdn_origin_nodes`) decommissions the same way, and the same
command covers it. Its operator also stays seated in `OriginAssignment` for its
namespaces. The namespace's publisher unseats it with `decdn publish revoke`
(`OriginAssignment.removeOrigin`; governance may too, and once the operator
deregisters anyone may prune it as inactive)
([ADR 011 § Origin Assignment Authority](https://github.com/decdn/decdn/blob/main/adr/011-content-takedown.md#origin-assignment-authority)),
ideally after another origin is seated, since a namespace with no origin left becomes
unassigned.

Silence `DecdnNodeDown` for the host before you run `make decommission`: the
[`*Down` alerts](../monitoring/README.md#the-down-alerts) also fire on a series that
went silent, so it fires for up to a day afterwards.
Expand Down
59 changes: 59 additions & 0 deletions docs/node-operators.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Node operators

You run deCDN **cache nodes**: you bond stake, serve the content clients ask for, and are
paid per megabyte. You hold no content of your own. A miss is filled from other nodes,
and an origin is the canonical source. The cache role is permissionless: any bonded
operator may re-serve any blob it holds
([ADR 001](https://github.com/decdn/decdn/blob/main/adr/001-network.md),
[ADR 002 § Publisher Identity and Namespaces](https://github.com/decdn/decdn/blob/main/adr/002-content-addressing.md#publisher-identity-and-namespaces)).

If you publish content and run its origins, read [`publishers.md`](publishers.md)
instead.

## What you deploy

One service per host: `decdn-node`, with no origin backend. Every path deploys it the same
way: a non-root daemon, public QUIC on **udp/4433**, metrics and the admin RPC on
loopback, and the chain's contract addresses taken from upstream's deployment manifest.

| Path | Start here | Entry point |
|------|------------|-------------|
| **Ansible** (VMs, bare metal, a fleet) | [`ansible/README.md`](../ansible/README.md) | `inventory/hosts-node.yml.example`, `make deploy-node` (`playbooks/node.yml`) |
| **cloud-init** (one VM, no control machine) | [`cloud-init/README.md`](../cloud-init/README.md) | `cloud-init/user-data-node.yaml` |
| **Docker Compose** (one Docker host) | [`compose/README.md`](../compose/README.md) | `COMPOSE_PROFILES=node` |
| **Helm** (Kubernetes, one release per node) | [`charts/decdn-node/README.md`](../charts/decdn-node/README.md) | the chart, with no `config.cache.origin` |

[`requirements.md`](requirements.md) has the hardware, network and platform requirements,
and compares the paths. The Galaxy collection for your own playbooks is
[`decdn.node`](../ansible/galaxy/node/README.md).

A node without an origin backend fills a miss by pulling from other nodes
(`node_to_node_pull_through_enabled`). The Ansible role and the chart turn that on for
you. On Compose, set it in `node.toml`; the daemon's default is off. A pull is paid
from the node's own PaymentPool (the node-to-node tier of
[ADR 003](https://github.com/decdn/decdn/blob/main/adr/003-payments.md)), so the node
fronts USDC for its misses. `buyer_working_deposit_micro_usdc` is not a cap: it is the
deposit the pool opens with and the balance every top-up refills it to, so keep the
node's wallet funded for as long as it pulls.

## Before it earns

The node serves paid traffic only after its on-chain stake and registration
([ADR 019, Phase 2](https://github.com/decdn/decdn/blob/main/adr/019-node-onboarding.md#phase-2--on-chain-setup)).
That is your step on every path: upstream's `decdn setup` walks it end to end, and
[`lifecycle.md`](lifecycle.md#running-on-chain-commands) shows how to run it on each
path. This repository stops at host preparation and startup.

## Day 2

- Backups, restores, host migration and decommissioning: [`lifecycle.md`](lifecycle.md).
- Dashboards and alert rules: [`monitoring/`](../monitoring/README.md). Grafana Cloud
shipping is opt-in on Ansible (`decdn_grafana_cloud_enabled`).
- Relays and discovery: a node uses n0's iroh relays and DNS discovery by default. To
move to self-hosted ones, set `decdn_relay_urls` and the `decdn_discovery_*` knobs on
Ansible (`network.relay_urls` and `[network.discovery]` in `node.toml` or the chart's
`config`). Two caveats ([`iroh_relay`](../ansible/roles/iroh_relay/README.md),
[`iroh_dns_server`](../ansible/roles/iroh_dns_server/README.md)):
a publisher's relay admits only listed node IDs by default, so ask for yours to be
added; and your own discovery settings drop the n0 leg, so clients must use the same
`dns_origin` to find your node.
83 changes: 83 additions & 0 deletions docs/publishers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Publishers

You publish content on deCDN. A **publisher** is an Ethereum address that owns at least
one namespace in the `PublisherRegistry` contract. It becomes one with its first
`createNamespace()` call. Once governance has vetted you as a publisher, you seat the
operators that serve each namespace as **origins**
([ADR 002 § Publisher Identity and Namespaces](https://github.com/decdn/decdn/blob/main/adr/002-content-addressing.md#publisher-identity-and-namespaces),
[ADR 011 § Origin Assignment Authority](https://github.com/decdn/decdn/blob/main/adr/011-content-takedown.md#origin-assignment-authority)).
You take your steps with the `decdn` CLI, not with this repository
([Appendix: Binaries](https://github.com/decdn/decdn/blob/main/adr/appendix-binaries.md)):
`decdn publish namespace create`, `decdn publish assign` (`OriginAssignment.addOrigin`),
`decdn publish revoke` (`removeOrigin`), and `decdn origin import`, which writes
content into a local directory in the layout an origin reads: point an `fs` origin at
it, or sync it to an S3 bucket (the layouts are the same). It does not write to S3 or
an HTTP store itself. The vetting itself is governance's.

This repository deploys what a publisher runs. Node operators who only run cache nodes
should read [`node-operators.md`](node-operators.md) instead.

## What you deploy

| Service | What it is | Ansible group and playbook | Other paths |
|---------|------------|----------------------------|-------------|
| **Origin node** | `decdn-node` with an origin backend (S3/R2/B2/MinIO, NFS or local disk, or an HTTP store): the canonical source of your namespace's content | `decdn_origin_nodes`, a child of `decdn_nodes`; `playbooks/origin.yml` | cloud-init `user-data-publisher.yaml`; Compose `origin` profile; the Helm chart with `config.cache.origin` |
| **sponsord** + **sponsord-onramp** | Pays for your users' downloads: the treasury wallet that owns a PaymentPool and signs capped capabilities for clients ([ADR 003 § Capability delegation](https://github.com/decdn/decdn/blob/main/adr/003-payments.md#capability-delegation)), and its public gate (Turnstile, installers) behind Caddy | `sponsord_hosts`, `sponsord_onramp_hosts`; `playbooks/sponsord.yml` | cloud-init `user-data-publisher.yaml`; Compose `sponsord`, `onramp`, `caddy` profiles |
| **iroh relay** | NAT-traversal fallback for peers that cannot hole-punch. Relays are operational infrastructure, not an incentivized role ([Architecture § Trust Assumptions](https://github.com/decdn/decdn/blob/main/adr/architecture.md#trust-assumptions)) | `iroh_relay_hosts`; `playbooks/iroh_relay.yml` | Ansible only |
| **iroh DNS server** | The pkarr relay and DNS server nodes publish and resolve their address records through, in place of n0's ([ADR 001 § Node Discovery](https://github.com/decdn/decdn/blob/main/adr/001-network.md#node-discovery-registry)) | `iroh_dns_server_hosts`; `playbooks/iroh_dns_server.yml` | Ansible only |

Only the origin is required. With Ansible, start from
`ansible/inventory/hosts-publisher.yml.example`; `make deploy-publisher`
(`playbooks/publisher.yml`) deploys all four in that order, and each has its own target
(`deploy-origin`, `deploy-sponsord`, `deploy-relay`, `deploy-dns`). The roles also ship as
the [`decdn.publisher`](../ansible/galaxy/publisher/README.md) Galaxy collection, which
depends on [`decdn.node`](../ansible/galaxy/node/README.md) for the baseline, observability
and the origin node itself.

## Origin nodes

An origin is a deCDN node like any other: it is bonded and registered
([ADR 019](https://github.com/decdn/decdn/blob/main/adr/019-node-onboarding.md)), with the
same hardening, port and day-2 tooling as a cache node. Three things differ:

- **The backend.** Set `decdn_cache_origin_kind` (`http`, `fs` or `s3`) and its fields,
or `decdn_cache_origins` for an ordered fallback list
([`decdn_node` role](../ansible/roles/decdn_node/README.md)).
S3 keys go in `/etc/decdn/decdn.env` on the host, never in inventory or user-data.
`playbooks/origin.yml` refuses a `decdn_origin_nodes` host without a backend, or one
outside `decdn_nodes`.
- **What it serves.** With a backend, the node fills misses from it rather than from
other nodes. By default it serves only what its own backend holds and declines
everything else, even a blob already in its cache (`relay_foreign_namespaces`,
[ADR 002 § Retrieval by namespace](https://github.com/decdn/decdn/blob/main/adr/002-content-addressing.md#retrieval-by-namespace),
[ADR 037 § Warming is a relay-edge mechanism](https://github.com/decdn/decdn/blob/main/adr/037-regional-proxy-warming.md#warming-is-a-relay-edge-mechanism)).
The gate is the backend's content, not the namespace's on-chain assignment.
- **Recognition is on-chain.** As the namespace's publisher, you seat the node's operator
with `decdn publish assign` (`OriginAssignment.addOrigin`). Configuring a backend without that only means the
node's bytes are served as cache
([Architecture § Origin Backends](https://github.com/decdn/decdn/blob/main/adr/architecture.md#origin-backends)).
Multiple operators may serve one namespace, so run more than one origin for
redundancy.

## Relays and the DNS server

A node's `decdn_relay_urls` replaces n0's relays, so deploy at least two relays before
pointing nodes at them. By default a relay admits only the nodes in the same inventory
(`iroh_relay_access: allowlist`): it reads their IDs from `decdn_nodes`, your origins
included. Other operators' nodes need their IDs in `iroh_relay_allowlist`, or
`everyone`. Clients can never be listed (they use a fresh key per fetch), so a node
behind NAT that serves clients needs its relays in `everyone` mode. The DNS server has
no replication: each DNS origin (`dns_origin`) is served by exactly one server, a single
point of failure for pkarr lookups. Give every node and client that uses it the same
`dns_origin`. Both roles' READMEs have the
details: [`iroh_relay`](../ansible/roles/iroh_relay/README.md),
[`iroh_dns_server`](../ansible/roles/iroh_dns_server/README.md).

## Day 2

[`lifecycle.md`](lifecycle.md) covers backup, restore, migration and decommission for
every service above. Decommissioning an origin stops the node but leaves its operator
seated on-chain, where `OriginAssignment.getOrigins` still lists it. Unseat it with
`decdn publish revoke` (`OriginAssignment.removeOrigin`); once the operator deregisters,
anyone may also prune it as inactive
([ADR 011 § Origin Assignment Authority](https://github.com/decdn/decdn/blob/main/adr/011-content-takedown.md#origin-assignment-authority)).
6 changes: 4 additions & 2 deletions docs/requirements.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
# Requirements and choosing a deploy path

What a deCDN node needs from its host and network, and which of this repo's four
deploy paths fits. Protocol facts (bond sizing, fees) are not here: they come from the
deCDN ADRs.
deploy paths fits. It applies to a node operator's cache node and a publisher's origin
node alike: an origin is the same node with an origin backend
([`node-operators.md`](node-operators.md), [`publishers.md`](publishers.md)). Protocol
facts (bond sizing, fees) are not here: they come from the deCDN ADRs.

## Choosing a path

Expand Down
Loading