Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 41 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Four ways to deploy the node, cache or origin:
| Path | For | Start here |
|------|-----|------------|
| **Docker Compose** (`compose/`), recommended | One host that already runs Docker. The quickest start, [below](#quickstart-docker-compose). Profiles for a cache node or an origin, the onboarding sponsor (`sponsord` + its onramp), an iroh relay, an iroh DNS server and Grafana Alloy, driven by the `decdn-compose` wrapper (set-up, preflight, guarded restarts). | [`compose/README.md`](compose/README.md) |
| **Ansible** (`ansible/`) | VMs and bare metal, one node or a fleet. Hardens the host too (firewall, SSH, patching). Also published on Ansible Galaxy as the [`decdn.node`](https://galaxy.ansible.com/ui/repo/published/decdn/node/) and [`decdn.publisher`](https://galaxy.ansible.com/ui/repo/published/decdn/publisher/) collections, roles and playbooks, so a fleet needs only an inventory of its own ([`examples/collection-fleet/`](examples/collection-fleet/README.md)). | [`ansible/README.md`](ansible/README.md) |
| **Ansible** (`ansible/`) | VMs and bare metal, one node or a fleet. Hardens the host too (firewall, SSH, patching). Also published on Ansible Galaxy as the [`decdn.node`](https://galaxy.ansible.com/ui/repo/published/decdn/node/) and [`decdn.publisher`](https://galaxy.ansible.com/ui/repo/published/decdn/publisher/) collections, roles and playbooks, so a fleet needs only an inventory of its own: the [quickstart below](#quickstart-ansible-from-galaxy). | [`ansible/README.md`](ansible/README.md) |
| **cloud-init** (`cloud-init/`) | One VM, no control machine: paste the user-data into your provider's "create server" form. It runs the Ansible playbook on the host itself, hardening included. One template per kind of operator: a cache node, or a publisher's origin with the onboarding sponsor (`sponsord` + its onramp). | [`cloud-init/README.md`](cloud-init/README.md) |
| **Helm** (`charts/decdn-node/`) | Kubernetes, one release per node, cache or origin. Published to `oci://ghcr.io/decdn/charts/decdn-node`, signed, and listed on [Artifact Hub](https://artifacthub.io/packages/helm/decdn-node/decdn-node). | [`charts/decdn-node/README.md`](charts/decdn-node/README.md) |

Expand Down Expand Up @@ -86,9 +86,48 @@ sudo compose/decdn-compose cli setup --mbps 100 --region DE --multiaddr "$MA"
Compose installs the node, not the host around it: firewall, SSH and patching stay
yours. The full flow (origins, the sponsor, relays, DNS, Alloy, upgrades) is in
[`compose/README.md`](compose/README.md). For a fleet, or a host you want hardened from
scratch, use [Ansible](ansible/README.md) (or [cloud-init](cloud-init/README.md) for
scratch, use [Ansible](#quickstart-ansible-from-galaxy) (or [cloud-init](cloud-init/README.md) for
one VM); on Kubernetes, the [Helm chart](charts/decdn-node/README.md).

## Quickstart (Ansible, from Galaxy)

From your workstation (ansible-core ≥ 2.15, e.g. `pipx install ansible-core`) with an
SSH key in `~/.ssh/`, and a fresh Debian or Ubuntu host. The collections bring the roles
and playbooks; you clone this repository once, only to copy the example project and
inventory templates into a private repository of your own:

```bash
git clone --depth 1 https://github.com/decdn/devops.git
cp -R devops/examples/collection-fleet my-fleet && cd my-fleet # make this your private repo
ansible-galaxy collection install -r requirements.yml -p collections
mkdir inventory && cp ../devops/ansible/inventory/hosts-node.yml.example inventory/hosts.yml
cp -R ../devops/ansible/inventory/{group_vars,host_vars} inventory/
cp inventory/host_vars/decdn-node-1/secret.yml.example inventory/host_vars/decdn-node-1/secret.yml
$EDITOR inventory/hosts.yml # ansible_host: the node's IP
$EDITOR inventory/host_vars/decdn-node-1/main.yml # decdn_region; add decdn_node_generate_keystore: true (the node makes its wallet)
$EDITOR inventory/host_vars/decdn-node-1/secret.yml # decdn_rpc_url (git-ignored)
```

Then deploy. The first run connects as root, or as your image's sudo user (`-u ubuntu`
on Ubuntu cloud images): it creates an admin account named after your local `$USER`,
keyed with your `~/.ssh/id_*` key, and then turns root and password login off. Every
later run connects as that account, so drop `-u`:

```bash
ssh root@<node-ip> true # accept the host key once (or ubuntu@…)
ansible-playbook decdn.node.node -l decdn-node-1 -u root # first run only
ansible-playbook decdn.node.node -l decdn-node-1 --check --diff # later changes: dry run,
ansible-playbook decdn.node.node -l decdn-node-1 # then apply
```

The play opens udp/4433 in the host firewall; open it in your provider's security
group too. Then fund the node's wallet and stake and register it with `decdn setup`
([`docs/lifecycle.md` § Running on-chain commands](docs/lifecycle.md#running-on-chain-commands)).
Publishers start from `hosts-publisher.yml.example`, uncomment the groups they run and
deploy with `decdn.publisher.publisher` ([`docs/publishers.md`](docs/publishers.md)). Backups, decommission and every other
`make` target in the docs:
[From `make` to `ansible-playbook`](examples/collection-fleet/README.md#from-make-to-ansible-playbook).

## Security model

This is the canonical statement; the per-path READMEs add only what is specific to them.
Expand Down
10 changes: 8 additions & 2 deletions ansible/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,14 @@ Declarative Ansible project for deploying deCDN nodes, one or a fleet, over a
hardened host baseline. The roles and these playbooks also ship as two collections on
Ansible Galaxy: [`decdn.node`](https://galaxy.ansible.com/ui/repo/published/decdn/node/) (what a node operator runs) and
[`decdn.publisher`](https://galaxy.ansible.com/ui/repo/published/decdn/publisher/) (what a publisher runs beside its
origin nodes), so a fleet can run from an inventory and the collections alone (see
[Packaging as Galaxy collections](#packaging-as-galaxy-collections-decdnnode-decdnpublisher)).
origin nodes), so a fleet can run from a private repository that holds only its
inventory, with the collections in place of this project: start from the
[root README's quickstart](../README.md#quickstart-ansible-from-galaxy), and translate the
`make` targets below with
[From `make` to `ansible-playbook`](../examples/collection-fleet/README.md#from-make-to-ansible-playbook)
(`make deploy-node LIMIT=<host>` is `ansible-playbook decdn.node.node -l <host>`). How
the collections are built:
[Packaging as Galaxy collections](#packaging-as-galaxy-collections-decdnnode-decdnpublisher).

| Playbook | Purpose | Make target |
|----------|---------|-------------|
Expand Down
8 changes: 6 additions & 2 deletions ansible/inventory/hosts-publisher.yml.example
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,12 @@
#
# Bootstrapping a fresh host, ansible_user and the Ubuntu sudo-rs workaround work as
# in hosts-node.yml.example: read its header before the first run. In short, the first
# run per host is `make deploy-publisher LIMIT=<host> ANSIBLE_ARGS='-u root'` (your
# image's bootstrap user in place of root); after it, plain `make deploy-publisher`.
# run per host is baseline only,
# make deploy-publisher LIMIT=<host> ANSIBLE_ARGS='--tags baseline -u root'
# (your image's bootstrap user in place of root); after it, plain
# `make deploy-publisher LIMIT=<host>`. Baseline only, because a relay's full run
# reads every node's ID over SSH, and under -u root it would try the nodes, whose
# root login baseline has already turned off, as root.

# Origin nodes — provisioned by playbooks/origin.yml (`make deploy-origin`) and by
# publisher.yml. An origin is a deCDN node with an origin backend: the canonical
Expand Down
16 changes: 16 additions & 0 deletions docs/lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,22 @@ Day-2 procedures for a running deCDN node, and for sponsord in
[its own section](#sponsord-the-onboarding-sponsor). The Ansible targets run from
`ansible/`; the manual commands work for any deploy path.

Deploying from the Galaxy collections instead of a clone? Each `make` target below is
an `ansible-playbook` command run from your inventory project:

```bash
ansible-playbook decdn.publisher.backup -l <host> # make backup: nodes, and sponsord's credentials
ansible-playbook decdn.publisher.decommission -l <host> # make decommission: every service on the host
ansible-playbook decdn.node.backup -l <host> # with decdn.node alone: a node's backup
ansible-playbook decdn.node.decommission -l <host> # with decdn.node alone: a host that runs only a node
```

`ANSIBLE_ARGS='…'` become plain options (`-e decdn_backup_scope=full`). Backups land in
`backups/` beside your `ansible.cfg`, not `ansible/backups/`. Relays and DNS servers
have nothing to back up. The Makefile's checks do not apply, so always pass `-l`:
[From `make` to `ansible-playbook`](../examples/collection-fleet/README.md#from-make-to-ansible-playbook)
lists them and maps every target.

## What makes a node *that* node

| File | What it is | Lose it and… |
Expand Down
35 changes: 28 additions & 7 deletions docs/node-operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,12 @@ loopback, and the chain's contract addresses taken from upstream's deployment ma
| Path | Start here | Entry point |
|------|------------|-------------|
| **Docker Compose** (one Docker host), recommended | [`compose/README.md`](../compose/README.md) | `decdn-compose init node` |
| **Ansible** (VMs, bare metal, a fleet) | [`ansible/README.md`](../ansible/README.md) | `inventory/hosts-node.yml.example`, `make deploy-node` (`playbooks/node.yml`) |
| **Ansible** (VMs, bare metal, a fleet) | [below](#deploy-with-ansible-from-galaxy), [`ansible/README.md`](../ansible/README.md) | `ansible/inventory/hosts-node.yml.example`, `ansible-playbook decdn.node.node` (or `make deploy-node` from a clone) |
| **cloud-init** (one VM, no control machine) | [`cloud-init/README.md`](../cloud-init/README.md) | `cloud-init/user-data-node.yaml` |
| **Helm** (Kubernetes, one release per node) | [`charts/decdn-node/README.md`](../charts/decdn-node/README.md) | `oci://ghcr.io/decdn/charts/decdn-node`, with no `config.cache.origin` |

[`requirements.md`](requirements.md) has the hardware, network and platform requirements,
and compares the paths. The roles and the node playbooks are also the
[`decdn.node`](../ansible/galaxy/node/README.md) collection on
[Ansible Galaxy](https://galaxy.ansible.com/ui/repo/published/decdn/node/) (`ansible-galaxy collection install decdn.node`):
with only an inventory of your own, `ansible-playbook -i inventory/hosts.yml decdn.node.node`
deploys what `make deploy-node` does, firewall hole included; or call the roles from
your own playbooks.
and compares the paths.

A node without an origin backend fills a miss by pulling from other nodes
(`node_to_node_pull_through_enabled`). The Ansible role and the chart turn that on for
Expand All @@ -40,6 +35,32 @@ fronts USDC for its misses. `buyer_working_deposit_micro_usdc` is not a cap: it
deposit the pool opens with and the balance every top-up refills it to, so keep the
node's wallet funded for as long as it pulls.

## Deploy with Ansible from Galaxy

The roles and the node playbooks are the
[`decdn.node`](../ansible/galaxy/node/README.md) collection on
[Ansible Galaxy](https://galaxy.ansible.com/ui/repo/published/decdn/node/), so you run
them from a private repository that holds only your inventory. Set it up once with the
[root README's quickstart](../README.md#quickstart-ansible-from-galaxy) (it clones this
repository only to copy [`examples/collection-fleet/`](../examples/collection-fleet/README.md)
and the inventory templates), then:

```bash
ansible-playbook decdn.node.node -l <host> -u root # a fresh host's first run (or -u ubuntu, …)
ansible-playbook decdn.node.node -l <host> --check --diff # dry run
ansible-playbook decdn.node.node -l <host> # deploy or update
ansible-playbook decdn.node.backup -l <host> # encrypted identity backup, to backups/
ansible-playbook decdn.node.decommission -l <host> # asks you to type the host; removes the node, keeps its keys
```

Without `-l`, a dry run, deploy or backup covers every cache node in the inventory;
decommission refuses more than `decdn_decommission_max_hosts` hosts (default 1). A
backup needs your public key in `decdn_backup_age_recipients`
([`lifecycle.md`](lifecycle.md#backup-make-backup)). Every `make` target in the docs,
and what the Makefile checks that a plain `ansible-playbook` run does not:
[From `make` to `ansible-playbook`](../examples/collection-fleet/README.md#from-make-to-ansible-playbook).
You can also call the roles from your own playbooks.

## Before it earns

The node serves paid traffic only after its on-chain stake and registration
Expand Down
67 changes: 49 additions & 18 deletions docs/publishers.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,22 +21,53 @@ should read [`node-operators.md`](node-operators.md) instead.

| Service | What it is | Ansible group and playbook | Other paths |
|---------|------------|----------------------------|-------------|
| **Origin node** | `decdn-node` with an origin backend (S3/R2/B2/MinIO, NFS or local disk, or an HTTP store): the canonical source of your namespace's content | `decdn_origin_nodes`, a child of `decdn_nodes`; `playbooks/origin.yml` | cloud-init `user-data-publisher.yaml`; Compose `origin` profile; the Helm chart with `config.cache.origin` (an `fs` origin also needs `origin.volume`) |
| **sponsord** + **sponsord-onramp** | Pays for your users' downloads: the treasury wallet that owns a PaymentPool and signs capped capabilities for clients ([ADR 003 § Capability delegation](https://github.com/decdn/decdn/blob/main/adr/003-payments.md#capability-delegation)), and its public gate (Turnstile, installers) behind Caddy | `sponsord_hosts`, `sponsord_onramp_hosts`; `playbooks/sponsord.yml` | cloud-init `user-data-publisher.yaml`; Compose `sponsord`, `onramp`, `caddy` profiles |
| **iroh relay** | NAT-traversal fallback for peers that cannot hole-punch. Relays are operational infrastructure, not an incentivized role ([Architecture § Trust Assumptions](https://github.com/decdn/decdn/blob/main/adr/architecture.md#trust-assumptions)) | `iroh_relay_hosts`; `playbooks/iroh_relay.yml` | Compose `relay` profile |
| **iroh DNS server** | The pkarr relay and DNS server nodes publish and resolve their address records through, in place of n0's ([ADR 001 § Node Discovery](https://github.com/decdn/decdn/blob/main/adr/001-network.md#node-discovery-registry)) | `iroh_dns_server_hosts`; `playbooks/iroh_dns_server.yml` | Compose `dns` profile |

Only the origin is required. With Ansible, start from
`ansible/inventory/hosts-publisher.yml.example`; `make deploy-publisher`
(`playbooks/publisher.yml`) deploys all four in that order, and each has its own target
(`deploy-origin`, `deploy-sponsord`, `deploy-relay`, `deploy-dns`). The roles and these
playbooks also ship as the [`decdn.publisher`](../ansible/galaxy/publisher/README.md) collection on
[Ansible Galaxy](https://galaxy.ansible.com/ui/repo/published/decdn/publisher/) (`ansible-galaxy collection install decdn.publisher`),
which depends on [`decdn.node`](../ansible/galaxy/node/README.md) for the baseline,
observability and the origin node itself: with only an inventory of your own,
`ansible-playbook -i inventory/hosts.yml decdn.publisher.publisher` (or `decdn.publisher.origin`,
`.sponsord`, `.iroh_relay`, `.iroh_dns_server`) deploys what the make targets do,
firewall holes and placement checks included.
| **Origin node** | `decdn-node` with an origin backend (S3/R2/B2/MinIO, NFS or local disk, or an HTTP store): the canonical source of your namespace's content | `decdn_origin_nodes`, a child of `decdn_nodes`; `decdn.publisher.origin` | cloud-init `user-data-publisher.yaml`; Compose `origin` profile; the Helm chart with `config.cache.origin` (an `fs` origin also needs `origin.volume`) |
| **sponsord** + **sponsord-onramp** | Pays for your users' downloads: the treasury wallet that owns a PaymentPool and signs capped capabilities for clients ([ADR 003 § Capability delegation](https://github.com/decdn/decdn/blob/main/adr/003-payments.md#capability-delegation)), and its public gate (Turnstile, installers) behind Caddy | `sponsord_hosts`, `sponsord_onramp_hosts`; `decdn.publisher.sponsord` | cloud-init `user-data-publisher.yaml`; Compose `sponsord`, `onramp`, `caddy` profiles |
| **iroh relay** | NAT-traversal fallback for peers that cannot hole-punch. Relays are operational infrastructure, not an incentivized role ([Architecture § Trust Assumptions](https://github.com/decdn/decdn/blob/main/adr/architecture.md#trust-assumptions)) | `iroh_relay_hosts`; `decdn.publisher.iroh_relay` | Compose `relay` profile |
| **iroh DNS server** | The pkarr relay and DNS server nodes publish and resolve their address records through, in place of n0's ([ADR 001 § Node Discovery](https://github.com/decdn/decdn/blob/main/adr/001-network.md#node-discovery-registry)) | `iroh_dns_server_hosts`; `decdn.publisher.iroh_dns_server` | Compose `dns` profile |

Only the origin is required.

### Deploy with Ansible from Galaxy

The roles and playbooks are the [`decdn.publisher`](../ansible/galaxy/publisher/README.md)
collection on [Ansible Galaxy](https://galaxy.ansible.com/ui/repo/published/decdn/publisher/),
which pulls in [`decdn.node`](../ansible/galaxy/node/README.md) for the baseline,
observability and the origin node itself. You run them from a private repository that
holds only your inventory. Set it up once with the
[root README's quickstart](../README.md#quickstart-ansible-from-galaxy), with two
changes:

- Start `inventory/hosts.yml` from `hosts-publisher.yml.example`. Only the origin
(`origin-1`) is live in it: uncomment the groups you run (sponsord and its onramp,
iroh relays, the DNS server).
- Edit `host_vars/origin-1/` rather than `decdn-node-1/`: `decdn_region`, the origin
backend, and `decdn_node_generate_keystore: true` if the node should make its wallet.

Then:

```bash
ansible-playbook decdn.publisher.publisher -l <host> --tags baseline -u root # a fresh host's first run (or -u ubuntu, …)
Comment thread
thiras marked this conversation as resolved.
ansible-playbook decdn.publisher.publisher -l <host> --check --diff # dry run
ansible-playbook decdn.publisher.publisher -l <host> # every service the inventory gives the host
ansible-playbook decdn.publisher.origin -l <host> # or one service: an origin node
ansible-playbook decdn.publisher.sponsord -l <host> # sponsord and its onramp
ansible-playbook decdn.publisher.iroh_relay -l <host> # an iroh relay
ansible-playbook decdn.publisher.iroh_dns_server -l <host> # the iroh DNS server
ansible-playbook decdn.publisher.backup -l <host> # node identity and sponsord credentials (relays and DNS servers hold nothing to back up)
ansible-playbook decdn.publisher.decommission -l <host> # asks you to type the host; removes every service on it, keeps the keys
```

The first run is baseline only (`--tags baseline`): it creates your admin account and
turns root login off. That matters for a relay, whose full run logs into every node in
the inventory to read its ID and would try those nodes as root too. Drop `-l` to cover
every host in the inventory. The playbooks open each service's firewall holes and refuse
a host in the wrong groups. From a clone of this repository, the same runs are
`make deploy-publisher`, `deploy-origin`, `deploy-sponsord`, `deploy-relay` and
`deploy-dns` in `ansible/`;
[From `make` to `ansible-playbook`](../examples/collection-fleet/README.md#from-make-to-ansible-playbook)
maps every target and lists what the Makefile checks that a plain `ansible-playbook`
run does not.

## Origin nodes

Expand All @@ -50,8 +81,8 @@ same hardening, port and day-2 tooling as a cache node. Three things differ:
S3 keys go in `/etc/decdn/decdn.env` on the host (or in `decdn_extra_env` in a
git-ignored `secret.yml`, beside `decdn_rpc_url`), never in tracked inventory or
user-data.
`playbooks/origin.yml` refuses a `decdn_origin_nodes` host without a backend, or one
outside `decdn_nodes`.
The origin playbook (`decdn.publisher.origin`) refuses a `decdn_origin_nodes` host
without a backend, or one outside `decdn_nodes`.
- **What it serves.** With a backend, the node fills misses from it rather than from
other nodes. By default it serves only what its own backend holds and declines
everything else, even a blob already in its cache (`relay_foreign_namespaces`,
Expand Down
Loading
Loading