Skip to content

lifecycle.md

Latest commit

 

History

History
222 lines (173 loc) · 10.4 KB

lifecycle.md

File metadata and controls

222 lines (173 loc) · 10.4 KB
Raw

Node lifecycle: backup, restore, migration, decommission

Day-2 procedures for a running deCDN node. The Ansible targets run from ansible/; the manual commands work for any deploy path.

What makes a node that node

File What it is Lose it and…
/var/lib/decdn/node.secret iroh node key (the wire identity) peers see a new node; the on-chain binding points at a key you no longer hold
/var/lib/decdn/keystore.json operator Ethereum key, encrypted you cannot unbond, withdraw or appeal a slash: the bond is stranded
/etc/decdn/keystore.password password for the keystore same as losing the keystore
/var/lib/decdn/*.redb, receipts daemon state, including vouchers not yet redeemed unredeemed revenue and settlement progress
/var/lib/decdn/cache/ cached blobs nothing lasting: it refills from origins and peers

Back up the first three the day a node is created, and again before any host change.

Backup (make backup)

Backups are encrypted on the node to public keys you choose, so plaintext key material never leaves the host and nothing secret is needed on the control machine. Generate a key pair once, somewhere that is neither the node nor this repo:

age-keygen -o ~/.config/decdn/backup-age.key     # keep this file safe and offline-capable
age-keygen -y ~/.config/decdn/backup-age.key     # prints the public recipient, age1…

Put the public key(s) in inventory (SSH public keys work too):

# group_vars/decdn_nodes.yml
decdn_backup_age_recipients:
  - age1…               # yours
  - ssh-ed25519 AAAA…   # a second custodian, optional
cd ansible
make backup LIMIT=<host>                                          # identity, hot
make backup LIMIT=<host> ANSIBLE_ARGS='-e decdn_backup_scope=full' # stops the node for the copy
Scope Archives Node downtime
identity (default) node.secret, keystore.json, keystore.password none
full the whole data dir except the cache, plus the keystore (wherever it lives) and the password file the copy (the redb stores are only consistent at rest); restarted afterwards if it was up

Add -e decdn_backup_include_env=true to include decdn.env (your RPC URL, which may embed an API key). The archive is written to /var/backups/decdn/ on the host (root 0600). make backup runs one host at a time.

  • Identity archives (kilobytes) are fetched to ansible/backups/<host>/, which is git-ignored.
  • Full archives are not fetched by default: Ansible's fetch would read the whole file into memory on both ends. The run prints a streaming copy command instead (ssh <host> sudo cat <file> > <file>). Force a fetch with -e decdn_backup_fetch=true if you know the archive is small.

If the archive step fails (a bad recipient, a full disk), the node is restarted, no partial file is left behind and the run fails. decdn_backup_leave_stopped only applies to a backup that succeeded.

Test the restore path, not just the backup:

age -d -i ~/.config/decdn/backup-age.key ansible/backups/<host>/<file>.tar.age | tar -tz

Without Ansible (Compose or a hand-built host), the same archive is one command on the node:

sudo tar -C / --numeric-owner -czf - var/lib/decdn/node.secret var/lib/decdn/keystore.json \
    etc/decdn/keystore.password | age -r age1… -o node-identity.tar.age

Restore and host migration

Run two hosts with the same node.secret and keystore and they will fight over one identity on the network and on-chain. Stop the old host first.

  1. Freeze the old host and take a full backup in the same step:

    make backup LIMIT=old-host ANSIBLE_ARGS='-e decdn_backup_scope=full -e decdn_backup_leave_stopped=true'

    Then make sure it cannot come back on a reboot. Either run make decommission LIMIT=old-host (it removes the node's service and this repo's Alloy agent and keeps the data; ignore the on-chain exit steps it prints, since the identity is moving, not leaving) or run sudo systemctl disable --now decdn-node on it.

  2. Prepare the new host with a normal deploy. Do not set decdn_node_generate_keystore: true for it. The deploy creates the decdn user and directories, then stops at the identity gate because the keys are not there yet. That is expected.

    make deploy LIMIT=new-host ANSIBLE_ARGS='-u root'   # first converge of a fresh box

  3. Restore, streaming the archive from the old host through your workstation's age straight into the new host, so the plaintext never touches a disk off the target (the full archive stays on the old host; step 1 printed its path):

    ssh old-host sudo cat /var/backups/decdn/<file>.tar.age \
  | age -d -i ~/.config/decdn/backup-age.key \
  | ssh new-host 'sudo tar -xzf - -C / --no-same-owner \
      && sudo chown -R decdn:decdn /var/lib/decdn /etc/decdn/keystore.password'

    --no-same-owner plus the chown matter: the decdn user's uid on the new host need not match the old one. The next deploy re-applies 0600 to the key files.

  4. Converge the new host: make deploy LIMIT=new-host.

  5. Update what is on-chain, if it changed. The node is registered with its multiaddr and region. A new public IP needs decdn node update-multiaddrs; a new country needs decdn node update-region. With decdn_chain from Running on-chain commands, preview first:

    decdn_chain node update-multiaddrs --multiaddr /ip4/<new-public-ip>/udp/4433/quic-v1 --dry-run

    Then the same command without --dry-run. The address set you pass replaces the one on-chain.

Decommission (make decommission)

make backup LIMIT=<host>          # first, always
make decommission LIMIT=<host>    # LIMIT is required; you type the host name to confirm

It stops decdn-node with systemctl (SIGTERM, which is the daemon's graceful drain path), disables it and removes the unit, and tears down the Grafana Alloy agent this repo installed, if any. -e decdn_decommission_purge_cache=true also deletes the cache. It refuses to run on more than one host unless you raise decdn_decommission_max_hosts, and an unanswered confirmation prompt fails after decdn_decommission_prompt_seconds (300). Both entry points refuse a decdn_cache_dir that equals or encloses the identity, so a purge can never take the keys with it.

It keeps the identity, /etc/decdn and the binaries, because the keystore is what withdraws the bond. It does not touch the chain. The exit is two operator steps with the decdn CLI, both with --dry-run first (use decdn_chain from Running on-chain commands):

  1. decdn_chain node deregister leaves the active node set. The bond is not returned: it stays deposited and slashable.
  2. decdn_chain node unbond --all starts the unbonding window. Run it again after the window to withdraw.

Delete the keystore only after the withdrawal has landed. The public udp/4433 firewall rule stays until baseline is re-run without it.

Do not use decdn node drain to take a systemd-managed node down: the unit is Restart=always, so systemd starts the drained daemon again five seconds later.

Running on-chain commands

decdn setup, node bond, node register, node update-multiaddrs, node update-region, node deregister and node unbond sign with the operator key and need the RPC endpoint. They do not read DECDN_RPC_URL from the environment, unlike the daemon: they take --rpc-url, or blockchain.rpc_url from node.toml. The Ansible role leaves rpc_url out of node.toml on purpose (it may embed an API key), and on Compose config init wrote the public endpoint there. So pass it explicitly. This helper runs the CLI as decdn with the unit's own environment file and hands the URL over as --rpc-url:

decdn_chain() {
  sudo systemd-run --pty --wait --collect -p User=decdn \
    -p EnvironmentFile=/etc/decdn/decdn.env \
    /bin/sh -c 'exec /usr/local/bin/decdn "$@" --config /etc/decdn/node.toml \
      --rpc-url "$DECDN_RPC_URL" --keystore-password-file /etc/decdn/keystore.password' \
    decdn "$@"
}

decdn_chain setup --mbps 100 --region DE \
  --multiaddr /ip4/<public-ip>/udp/4433/quic-v1 --dry-run

The URL is in the decdn process's arguments while the command runs, so other local users could read it with ps. On a shared host, write a 0600 copy of node.toml with rpc_url set under [blockchain] and pass that as --config instead. The clean fix is upstream: give CommonChainArgs.rpc_url (crates/common/src/cli/common.rs) env = "DECDN_RPC_URL", as the daemon has. Then EnvironmentFile= alone would be enough and the helper could drop --rpc-url.

Register the node's public /ip4/ multiaddr, and on a dual-stack host its /ip6/ one too (repeat --multiaddr). Since decdn/decdn#2144 (869141e9) the daemon binds QUIC on both 0.0.0.0:4433 and [::]:4433; on a host without IPv6 it starts IPv4-only and logs a warn. Check ss -ulpn shows [::]:4433 before registering an /ip6/ address. An older build binds IPv6 on a random port, so an /ip6/ address on-chain would point at a port nothing listens on: register /ip4/ only there. To add the /ip6/ address after an upgrade, run decdn_chain node update-multiaddrs with both addresses; it replaces the whole on-chain set.

To fund the wallet you need its address, and keystore.json has no plaintext address field. whoami decrypts it. It takes no --rpc-url, so run it directly rather than through decdn_chain:

sudo systemd-run --pty --wait --collect -p User=decdn \
  /usr/local/bin/decdn --config /etc/decdn/node.toml whoami \
  --keystore-password-file /etc/decdn/keystore.password

Compose and Kubernetes

  • Compose (compose/) uses the same host paths (/var/lib/decdn, /etc/decdn), so the manual backup command and the restore steps above apply unchanged; stop the node with docker compose stop.
  • Helm: the identity lives in the operator-provisioned existingSecret, which you created off-cluster and should already hold elsewhere. The daemon's state is on the PVC; snapshot it with your storage's VolumeSnapshot support after scaling the StatefulSet to zero. Never run two releases with the same identity Secret.