From c4f0de4d61729f8d6810f3bb2a06b6b6918a698b Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Tue, 1 Sep 2026 21:33:01 +0300 Subject: [PATCH 1/3] feat(decdn_node)!: re-sync config schema, contracts and install path with decdn d306cc5c MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The repo had fallen ~180 commits behind decdn/decdn and was broken in two independent ways that fail at different stages of a deploy: 1. The pinned release does not exist. `git ls-remote --tags` on decdn/decdn returns zero tags — release.yml fires on a `v[0-9]*` tag push and has never run — so `make deploy` 404'd on the first get_url. The default install method is now `manual`, and the release-mode assert names the cause. 2. The role rendered 18 node.toml keys the daemon rejects. Every upstream config section is `#[serde(deny_unknown_fields)]` with no serde aliases, so a stale key is a startup crash-loop, and `content_blacklist_address` had become required (an absent value is a fail-open compliance trap, ADR 011/031). Upstream causes: the shared payment pool replaced pairwise channels, [gossip] was deleted, the example configs were dropped in favour of `decdn config init`, and all fourteen contracts were redeployed (deployBlock 11613778). BREAKING CHANGE: `decdn_payment_channel_address` is now `decdn_payment_pool_address`, `decdn_buyer_deposit_micro_usdc` is now `decdn_buyer_working_deposit_micro_usdc`, and `decdn_content_blacklist_address` is required. Sixteen variables backing deleted upstream keys are removed (`decdn_enable_0rtt`, `decdn_delivery_ceiling`, `decdn_voucher_interval_mb`, the three `*_from_block` knobs, the settlement auto-close pair, the speculative-pull trio, and the per-tarball sha256 pins). See ansible/galaxy/CHANGELOG.md for the full upgrade note. Also in this change: - Full schema parity: [network.discovery], [cache.tinylfu], [cache.serve_economics], [cache.origin_retry], [cache.circuit_breaker], [[cache.origins]], [security], [load_shed], [dht/probe.rate_limit], [receipts] and [content] are now rendered. - `decdn config validate` runs against the installed binary after templating, so schema drift fails the deploy with the daemon's own error instead of crash-looping the service. This is the durable guard: the hand-mirrored Ansible asserts are what drifted in the first place. - Release integrity: SHA256SUMS is verified against a detached signature using a vendored keyring, parsed via --status-fd so an EXPIRED or REVOKED key is rejected (plain `gpg --verify` exits 0 for both). Each archive is matched to exactly one manifest entry rather than counting ': OK' lines. - The RPC URL is sourced from the 0600 env file rather than passed via Ansible's `environment:`, which is a shell prefix and would put the embedded API key in the target's process table. - New `molecule/schema` scenario: asserts every rendered config PATH exists upstream, from an inventory generated by files/gen-schema-keys.py. Paths, not leaf names — `sketch_bytes` is legal at cache.tinylfu.sketch_bytes and a startup failure at cache.sketch_bytes. - Validation scenario grows to 11 negative cases, including one proving the config gate itself still fails a deploy. Verified: all 5 molecule scenarios pass (default idempotent); ansible-lint production profile clean; a real decdn 0.1.1 binary validates the host_vars, maximal and multi-origin renders (exit 0); all 8 addresses match contracts/deployments/421614.json. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01JrEoHSWL8rdZTB8FuT9pAM --- .github/workflows/molecule.yml | 7 +- .gitignore | 5 + AGENTS.md | 18 +- README.md | 5 +- ansible/Makefile | 4 + ansible/README.md | 22 +- ansible/galaxy/CHANGELOG.md | 63 ++ .../inventory/host_vars/decdn-node-1/main.yml | 84 +- .../host_vars/decdn-node-1/secret.yml.example | 19 +- ansible/molecule/default/converge.yml | 34 +- .../molecule/default/files/decdn-node-stub | 47 +- ansible/molecule/default/verify.yml | 70 +- .../molecule/generate-keystore/converge.yml | 4 +- ansible/molecule/schema/README.md | 41 + ansible/molecule/schema/converge.yml | 208 +++++ .../molecule/schema/files/gen-schema-keys.py | 144 ++++ ansible/molecule/schema/files/schema-keys.txt | 173 ++++ ansible/molecule/schema/molecule.yml | 47 ++ ansible/molecule/schema/prepare.yml | 44 ++ ansible/molecule/schema/verify.yml | 191 +++++ ansible/molecule/slow-readiness/converge.yml | 4 +- ansible/molecule/validation/converge.yml | 149 +++- ansible/molecule/validation/molecule.yml | 1 + ansible/molecule/validation/prepare.yml | 44 ++ ansible/roles/decdn_node/README.md | 325 ++++++-- ansible/roles/decdn_node/defaults/main.yml | 254 ++++-- .../decdn_node/files/decdn-release-KEYS.asc | 23 + ansible/roles/decdn_node/handlers/main.yml | 10 + ansible/roles/decdn_node/tasks/install.yml | 225 +++++- ansible/roles/decdn_node/tasks/main.yml | 744 ++++++++++++++---- .../templates/decdn-node.service.j2 | 10 + .../roles/decdn_node/templates/node.toml.j2 | 417 ++++++++-- 32 files changed, 3012 insertions(+), 424 deletions(-) create mode 100644 ansible/molecule/schema/README.md create mode 100644 ansible/molecule/schema/converge.yml create mode 100755 ansible/molecule/schema/files/gen-schema-keys.py create mode 100644 ansible/molecule/schema/files/schema-keys.txt create mode 100644 ansible/molecule/schema/molecule.yml create mode 100644 ansible/molecule/schema/prepare.yml create mode 100644 ansible/molecule/schema/verify.yml create mode 100644 ansible/molecule/validation/prepare.yml create mode 100644 ansible/roles/decdn_node/files/decdn-release-KEYS.asc diff --git a/.github/workflows/molecule.yml b/.github/workflows/molecule.yml index a5d1758..ad06337 100644 --- a/.github/workflows/molecule.yml +++ b/.github/workflows/molecule.yml @@ -37,6 +37,9 @@ jobs: run: make deps - name: molecule test # --all runs every scenario under ansible/molecule/: default - # (operator-provisioned keystore), generate-keystore (opt-in host-side - # wallet generation), and slow-readiness (advisory /metrics probe timeout). + # (operator-provisioned keystore + rendered-value assertions), schema + # (config key-set drift against the upstream field list), validation + # (bad knobs must be rejected by the role's own asserts), generate-keystore + # (opt-in host-side wallet generation), and slow-readiness (advisory + # /metrics probe timeout). run: molecule test --all diff --git a/.gitignore b/.gitignore index cb837cf..9a7507c 100644 --- a/.gitignore +++ b/.gitignore @@ -28,6 +28,11 @@ ansible/build/ *.tar.gz ansible/importer_result.json +# ansible-compat scaffolding (a .lock plus empty modules/collections/roles dirs), +# created in the project root by ansible-lint and molecule — so `make lint-ansible` +# and `make molecule` both leave one behind. Regenerated on every run. +.ansible/ + # Python bytecode (e.g. from the molecule stub daemon or any local tooling) __pycache__/ *.pyc diff --git a/AGENTS.md b/AGENTS.md index e359d43..17e2f78 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,10 +48,20 @@ ansible/ # the deployment project (DevSec-hardened, lean roles) - **`ansible/`** — the declarative deployment project. **The public deCDN node** (`playbooks/site.yml` → baseline + `decdn-node`), installed from a pinned GitHub release - tarball under a hardened systemd unit; public QUIC udp/4433, loopback metrics/admin, - operator-provisioned eth keystore, required chain knobs (no baked protocol facts — sourced - from ADRs), over a shared DevSec-hardened `baseline`. See `ansible/README.md`. (On-chain - node stake/registration, ADR 019 Phase 2, is an operator step, not automated.) + tarball — verified against the release's GPG-signed `SHA256SUMS` — or, while upstream + has no release tag cut (the current default), from locally-built binaries; under a hardened + systemd unit; public QUIC udp/4433, loopback metrics/admin, operator-provisioned eth + keystore, required chain knobs (no baked protocol facts — sourced from ADRs), over a + shared DevSec-hardened `baseline`. See `ansible/README.md`. (On-chain node + stake/registration, ADR 019 Phase 2, is a manual operator step, driven by `decdn setup`.) + + **Config-schema coupling.** `roles/decdn_node/templates/node.toml.j2` renders against + `decdn/crates/common/src/config/types.rs`, where every section is + `#[serde(deny_unknown_fields)]` with **no** serde aliases — a key the role emits that + the installed binary does not know is a startup crash-loop. Two guards: the role runs + `decdn config validate` against the real binary after templating, and the + `molecule/schema` scenario checks the rendered key set against a committed inventory of + upstream field names. Re-sync both when bumping the pinned decdn version. ## Commands diff --git a/README.md b/README.md index 51a7ac6..47f52a6 100644 --- a/README.md +++ b/README.md @@ -31,11 +31,14 @@ baseline host hardening — DevSec os/ssh, nftables default-deny inbound, fail2ban, unattended-upgrades, chrony, an admin sudo user │ └─ site.yml → decdn-node public QUIC udp/4433; metrics+admin loopback; - release-tarball install; hardened systemd unit + signed-tarball or local-build install; + hardened systemd unit ``` On-chain node stake + registration (ADR 019 Phase 2) is an **operator step**, not automated here — the node serves paid traffic only after it is staked and registered. +Upstream's `decdn setup` walks that phase end to end (with `--dry-run`); this repo +stops at host prep and startup. ## Repository layout diff --git a/ansible/Makefile b/ansible/Makefile index 990c610..c1db30d 100644 --- a/ansible/Makefile +++ b/ansible/Makefile @@ -25,6 +25,10 @@ deploy: # --- Tests ------------------------------------------------------------------- # Containerised converge + idempotence + verify of the decdn_node role against a # stub daemon (needs Docker; a privileged systemd container). See molecule/. +# Scenarios: default (rendered values + idempotence), schema (config key-set drift +# against the upstream field list), validation (bad knobs must be rejected by the +# role's own asserts), generate-keystore (opt-in host-side wallet), slow-readiness +# (advisory /metrics probe timeout). # `--all` runs every scenario: `default` (operator-provisioned keystore) and # `generate-keystore` (opt-in host-side wallet generation). molecule: diff --git a/ansible/README.md b/ansible/README.md index f7ade1b..8e8ab8a 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -85,12 +85,13 @@ unless you listed one). all, use the **`manual`** install method (`decdn_node_install_method: manual` + `decdn_node_manual_bin_src` / `decdn_cli_manual_bin_src`). 2. Per-node config in `inventory/host_vars//main.yml` (committed) — - `decdn_node_version`, the three contract addresses, `decdn_region`, cache origin, … — plus - the one secret, `decdn_rpc_url`, in a sibling git-ignored `secret.yml` (copy the shipped - `host_vars/decdn-node-1/secret.yml.example`). The committed `main.yml` already carries the - Arbitrum Sepolia genesis contract addresses; edit `decdn_region` + cache origin for your - node. Contract addresses are protocol facts — source them from the deCDN contract - deployment / an ADR, never guess. + the install method + binary sources, the **four** required contract addresses, + `decdn_region`, cache origin, … — plus the one secret, `decdn_rpc_url`, in a sibling + git-ignored `secret.yml` (copy the shipped `host_vars/decdn-node-1/secret.yml.example`). + The committed `main.yml` already carries the current Arbitrum Sepolia addresses; edit + `decdn_region`, the binary paths and the cache origin for your node. Contract addresses + are protocol facts — copy them from the upstream deployment manifest + `decdn/contracts/deployments/.json`, never guess. 3. The **eth keystore + password file** provisioned on the host (operator step — the wallet must be funded + staked per the deCDN node-onboarding ADR, 019). As the `decdn` user, create the password file FIRST (`key-gen` reads it, never creates it), then generate the @@ -113,11 +114,14 @@ ss -lun | grep 4433 # public QUIC listener ss -ltn | grep -E '9090|9191' # metrics + admin — 127.0.0.1 ONLY curl -s 127.0.0.1:9090/metrics # 200 once up decdn node health # admin RPC; full readiness needs on-chain registration +decdn config validate --config /etc/decdn/node.toml # the role runs this too, on every deploy ``` -The node serves paid traffic only **after** on-chain stake + registration (the -`CapacityBond` txns of ADR 019 §2.2–2.3) — an operator action, not automated here, and -with no turnkey CLI yet (see `roles/decdn_node/README.md`). +The node serves paid traffic only **after** on-chain stake + registration (ADR 019 +§2.2–2.3) — an operator action, not automated here, but no longer a raw contract call: +`decdn setup` walks the whole of Phase 2, and `decdn node bond` / `register` are the +primitives underneath it. All take `--dry-run`. See +[`roles/decdn_node/README.md`](roles/decdn_node/README.md#on-chain-onboarding). --- diff --git a/ansible/galaxy/CHANGELOG.md b/ansible/galaxy/CHANGELOG.md index ee094d8..e981288 100644 --- a/ansible/galaxy/CHANGELOG.md +++ b/ansible/galaxy/CHANGELOG.md @@ -19,12 +19,75 @@ Not yet published to Galaxy (pre-1.0; the published shape may still change). - `decdn.node.decdn_node` — the `decdn-node` daemon, installed from a pinned GitHub Release tarball under a hardened systemd unit; public QUIC udp/4433, loopback metrics + admin RPC. +- Release-integrity verification: `release` mode fetches the release's `SHA256SUMS` + and `SHA256SUMS.asc`, verifies the detached signature against the maintainer + keyring vendored at `roles/decdn_node/files/decdn-release-KEYS.asc`, then checks + the tarballs against the manifest. Replaces the hand-pasted `decdn_node_sha256` / + `decdn_cli_sha256` pins, which are removed. `decdn_verify_release_signature` + (default `true`) and `decdn_release_keyring` control it. +- `decdn config validate` now runs against the installed binary after `node.toml` is + templated, so a config-schema mismatch fails the deploy with the daemon's own + error rather than crash-looping the service. +- `decdn_extra_env` — extra `KEY: value` pairs appended to the `0600` env file. The + supported home for environment-borne secrets, notably the AWS credentials behind + an S3 cache origin (which are deliberately never written to the `0640` `node.toml`). +- `ExecReload` on the unit plus a `Reload decdn-node` handler, for the five + hot-reloadable config sections. +- Full config-schema parity with `decdn/decdn` @ `d306cc5c`: `[network.discovery]`, + `[cache.tinylfu]`, `[cache.serve_economics]`, `[cache.origin_retry]`, + `[cache.circuit_breaker]`, `[[cache.origins]]`, `[security]`, `[load_shed]`, + `[dht.rate_limit]`, `[probe.rate_limit]`, `[receipts]` and `[content]` are now + rendered, along with the new `[blockchain]` and `[cache]` scalars. + - `decdn_node_generate_keystore` (default `false`) — opt-in host-side wallet generation: when `true` the `decdn_node` role runs `decdn key-gen` only if the keystore is absent (minting a random `0600` password file first, but only when the keystore is also absent), never overwriting an existing wallet. Funding + on-chain staking/registration remain a manual step. +### Changed (BREAKING) + +The collection has never been published, so this is not a break against any +released version — but it *is* a break against the shape earlier commits on `main` +had, and against any inventory written for it. + +- `decdn_payment_channel_address` → **`decdn_payment_pool_address`**. Upstream + replaced pairwise payment channels with a shared payment pool. +- `decdn_buyer_deposit_micro_usdc` → **`decdn_buyer_working_deposit_micro_usdc`**. + The split initial/working buyer deposits were merged into one. +- `decdn_content_blacklist_address` is now **REQUIRED**, not optional. Upstream + refuses to resolve a config without it (ADR 011/031): an absent or zero address is + a fail-open compliance trap. +- `decdn_origin_assignment_address` and `decdn_publisher_registry_address` are now + **independently** optional; the old "set both or neither" assert is gone. +- The `arbitrum-sepolia` contract addresses in `host_vars/decdn-node-1/main.yml` are + re-synced to deployBlock 11613778. Upstream redeployed all fourteen contracts, so + every previous address is dead. +- The default `decdn_node_install_method` is now `manual`, because upstream has cut + no release tag yet and `release` mode has nothing to download. + +### Removed + +Config keys upstream deleted. Every config section is `deny_unknown_fields` with no +serde aliases, so leaving any of these set would refuse the daemon's startup: + +- `decdn_relay_url` (singular — use the `decdn_relay_urls` list) and + `decdn_enable_0rtt`. +- `decdn_slash_judge_from_block`, `decdn_origin_directory_from_block`, + `decdn_content_blacklist_from_block` — the watchers no longer take a scan floor. +- `decdn_delivery_ceiling` (on-chain `PaymentPool.getRateBounds()` is authoritative; + `DECDN_DELIVERY_CEILING` is a retired env var upstream) and + `decdn_voucher_interval_mb` (voucher-interval negotiation was deleted). +- `decdn_settlement_auto_threshold_micro_usdc` and + `decdn_settlement_auto_by_voucher_nonce_span` — auto-`closeChannel` went away with + the channels. +- `decdn_pull_ahead_bytes`, `decdn_max_unrecouped_leech_bytes`, + `decdn_pull_share_ratio_percent`, `decdn_pull_through_require_authorized_origin` — + the speculative-pull accounting was replaced by + `decdn_node_pull_stall_window_sec` + `decdn_node_pull_min_throughput_bps`. +- `decdn_region_accounting_interval_sec`. +- `decdn_node_sha256` / `decdn_cli_sha256` — superseded by signed `SHA256SUMS`. + [Unreleased]: https://github.com/decdn/devops/commits/main diff --git a/ansible/inventory/host_vars/decdn-node-1/main.yml b/ansible/inventory/host_vars/decdn-node-1/main.yml index f20c6df..2993bb6 100644 --- a/ansible/inventory/host_vars/decdn-node-1/main.yml +++ b/ansible/inventory/host_vars/decdn-node-1/main.yml @@ -1,52 +1,80 @@ --- # Per-node deployment values for decdn-node-1 (COMMITTED — non-secret config only). -# Ansible merges every *.yml in this directory for host decdn-node-1. The one -# sensitive value, decdn_rpc_url, lives in the sibling secret.yml (git-ignored) — -# copy secret.yml.example to secret.yml and fill it in. The node refuses to start -# until both this file and secret.yml are set. +# Ansible merges every *.yml in this directory for host decdn-node-1. The sensitive +# values (decdn_rpc_url, and any decdn_extra_env secrets) live in the sibling +# secret.yml (git-ignored) — copy secret.yml.example to secret.yml and fill it in. +# The node refuses to start until both this file and secret.yml are set. # -# The contract addresses below are the canonical Arbitrum Sepolia (chain 421614) -# v0.1.0 genesis deploy (deployBlock 11418331) — public on-chain facts from the deCDN -# contract deployment, not invented here. If you target a different deployment, replace -# them (and chain_id) with that deployment's values. -# MUST-EDIT lines are yours to fill; the rest match the genesis deploy. +# CONTRACT ADDRESSES ARE NOT INVENTED HERE. They are a verbatim copy of the +# upstream deployment manifest decdn/contracts/deployments/421614.json (mirrored +# byte-identically at decdn/crates/cli/deployments/421614.json — a +# deployment-manifest-mirror pre-commit hook and a CI `cmp` keep the two in step). +# That manifest is what `decdn config init --chain arbitrum-sepolia` bakes in, so +# diffing this file against a fresh `config init` is the cross-check after any +# redeploy. If you target a different deployment, replace these (and chain_id) with +# that deployment's manifest values. +# +# Synced from deployBlock 11613778 (decdn/decdn @ d306cc5c, 2026-09-01). Upstream +# has redeployed three times; addresses change wholesale each time, so re-sync +# rather than patching individual lines. +# MUST-EDIT lines are yours to fill; the rest match the manifest. -# Pinned release to install (a v GitHub Release must exist). -decdn_node_version: "0.1.0" -# Recommended: pin the tarball's sha256 (no checksums file is published upstream). -decdn_node_sha256: "" +# --- Which binaries to install ------------------------------------------------ +# Upstream has cut NO release tag yet (`git ls-remote --tags decdn/decdn` is empty +# and release.yml only fires on a v[0-9]* tag push), so there is no tarball to +# download and "release" mode cannot work. Build both binaries from a decdn/decdn +# checkout and point the role at them: +# cargo build --release -p decdn-node -p decdn-cli +# Once upstream cuts a release, switch to: +# decdn_node_install_method: release +# decdn_node_version: "0.1.1" +# and the role verifies the tarballs against the GPG-signed SHA256SUMS manifest. +decdn_node_install_method: manual +decdn_node_manual_bin_src: "" # MUST-EDIT — control-machine path to decdn-node +decdn_cli_manual_bin_src: "" # MUST-EDIT — control-machine path to decdn # --- Chain (Arbitrum Sepolia, chain 421614) ----------------------------------- # The RPC endpoint (decdn_rpc_url) is the one sensitive value — set it in secret.yml. decdn_chain_id: 421614 # Arbitrum Sepolia -# Required contracts (from the chain-421614 deployment). -decdn_payment_channel_address: "0x62B911B7CdEA5eedFB7dC0E2343440516fB8cb43" # PaymentChannel -decdn_capacity_bond_address: "0x2F08aE39B0127C92d56F9E5eAd20692b217dDc58" # CapacityBond -decdn_slash_judge_address: "0xa9feAb7c1a82e32a2A52c40aD545C0111eC3A2b4" # SlashJudge +# Required contracts. All four are mandatory upstream — a node will not resolve its +# config without them. +decdn_payment_pool_address: "0x64140155a931C01c5ff008c0667F0C3657a6d357" # PaymentPool +decdn_capacity_bond_address: "0xb6262F55bf20935d22F95A1aEc5b6101A5D98B5e" # CapacityBond +decdn_slash_judge_address: "0x63Bc91195baDfc1c65a9A2cf0556Bbfd18E94A55" # SlashJudge +# ContentBlacklist became REQUIRED (ADR 011/031): an absent or zero address is a +# fail-open compliance trap, and serving a blacklisted hash past its compliance +# window is slashable. +decdn_content_blacklist_address: "0x1971CeD7Ac2c37e87a1b8Cf25f4611B0D3F2b9B0" # ContentBlacklist # --- Slash appeals (optional; ADR 028) ---------------------------------------- -decdn_slash_appeal_address: "0x4CA50D80bFcBA877eAC4cFa36FF95D7f803c6590" # SlashAppeal -decdn_slash_judge_from_block: 11418331 # SlashJudge deploy block — bounds per-restart rescan +# Only `decdn appeal slash` uses this; the daemon validates but ignores it. +decdn_slash_appeal_address: "0x2C56029cD21E5eaa325C01f3Cd22c46C6E34040D" # SlashAppeal -# --- Origin directory (optional; ADR 022 — set both or neither) --------------- -decdn_origin_assignment_address: "0x632F49B3447810eF20eee884fF6e7EAB5560b19e" # OriginAssignment -decdn_publisher_registry_address: "0xa9D014d24924E7A5BCaB9E49A68119B1e92EBCD7" # PublisherRegistry -decdn_origin_directory_from_block: 11418331 # PublisherRegistry deploy block +# --- Origin directory (optional; ADR 022) ------------------------------------- +# Independently optional since #1737 dropped the whole-directory mirror. +# OriginAssignment feeds the daemon's cache-miss pull-through fallback; +# PublisherRegistry is validate-only on the daemon and consumed by `decdn publish`. +decdn_origin_assignment_address: "0x25C872456E8BC7c1a8c698CEB3FF7Ea6667d9cd2" # OriginAssignment +decdn_publisher_registry_address: "0x5C99fAF051c1684f1b481982c7D730883b60e339" # PublisherRegistry -# --- Content blacklist compliance (optional; ADR 011/031) --------------------- -decdn_content_blacklist_address: "0x7e903dd1674EDe0596629205E55e5fe1B670dBD3" # ContentBlacklist -decdn_content_blacklist_from_block: 11418331 # ContentBlacklist deploy block +# --- CLI-only (optional) ------------------------------------------------------ +# externalDeps.usdc from the same manifest. The daemon validates but never reads +# it; `decdn setup` uses it to bond with USDC instead of the native token. +decdn_usdc_address: "0x75faf114eafb1BDbe2F0316DF893fd58CE46AA4d" # --- Cache pull-through origin (what the node fetches on a cache miss) --------- # MUST-EDIT — operator-specific backing origin (NOT a chain fact). A serving node # needs one, else cache misses fail NoOrigin. http shown; see defaults/main.yml for -# fs / s3 fields. +# fs / s3 fields, and decdn_cache_origins for an ordered fallback list. decdn_cache_origin_kind: "http" decdn_cache_origin_url: "https://your-origin.example/" # --- Node identity / locale --------------------------------------------------- decdn_region: "US" # MUST-EDIT — ISO 3166-1 alpha-2 of the node's physical location -# decdn_relay_url: "" # optional iroh relay for NAT traversal +# decdn_relay_urls: [] # optional iroh relays for NAT traversal ([] => n0 defaults) # --- Economics ---------------------------------------------------------------- +# Seller-side quote. The on-chain PaymentPool rate bounds are authoritative at +# runtime — the daemon overwrites payment.delivery_floor from getRateBounds() at +# startup — so this is the node's own asking price, not a clamp. decdn_rate_per_mb: 10 # USDC base units (6 decimals); 10 = $0.00001/MB diff --git a/ansible/inventory/host_vars/decdn-node-1/secret.yml.example b/ansible/inventory/host_vars/decdn-node-1/secret.yml.example index f44a3d0..86b2d28 100644 --- a/ansible/inventory/host_vars/decdn-node-1/secret.yml.example +++ b/ansible/inventory/host_vars/decdn-node-1/secret.yml.example @@ -1,8 +1,21 @@ --- -# Per-node SECRET for decdn-node-1. Copy to secret.yml (same directory, git-ignored) -# and fill in. Ansible merges it with main.yml for this host. This is the ONLY -# per-node value kept out of git — everything else lives in the committed main.yml. +# Per-node SECRETS for decdn-node-1. Copy to secret.yml (same directory, +# git-ignored) and fill in. Ansible merges it with main.yml for this host. These +# are the ONLY per-node values kept out of git — everything else lives in the +# committed main.yml. # # MUST-EDIT — SENSITIVE (may embed an API key). The public endpoint below works # for light use; run your own or use a provider for production reliability. decdn_rpc_url: "https://sepolia-rollup.arbitrum.io/rpc" + +# Optional. Extra KEY: value pairs appended to the 0600 /etc/decdn/decdn.env that +# systemd hands the daemon. This is where environment-borne secrets belong — most +# notably the AWS credentials backing an S3 cache origin, which the role +# deliberately does NOT write into node.toml (that file is 0640 and diffable): +# +# decdn_extra_env: +# AWS_ACCESS_KEY_ID: "AKIA..." +# AWS_SECRET_ACCESS_KEY: "..." +# AWS_SESSION_TOKEN: "..." # only for assume-role / SSO flows +# +# Pair that with decdn_cache_origin_s3_use_default_chain: true in main.yml. diff --git a/ansible/molecule/default/converge.yml b/ansible/molecule/default/converge.yml index 5d1ee62..16a217a 100644 --- a/ansible/molecule/default/converge.yml +++ b/ansible/molecule/default/converge.yml @@ -19,9 +19,12 @@ decdn_rpc_url: "https://rpc.example.invalid/" decdn_chain_id: 421614 decdn_region: "US" - decdn_payment_channel_address: "0x1111111111111111111111111111111111111111" + decdn_payment_pool_address: "0x1111111111111111111111111111111111111111" decdn_capacity_bond_address: "0x2222222222222222222222222222222222222222" decdn_slash_judge_address: "0x3333333333333333333333333333333333333333" + # REQUIRED upstream since ADR 011/031 — the role asserts it alongside the other + # three, so every scenario has to carry it. + decdn_content_blacklist_address: "0x4444444444444444444444444444444444444444" # fs origin: exercises the role's "create the fs cache-origin base dir" task # (#28) — /var/lib/decdn/origin does not pre-exist, so the run genuinely creates # it (verify.yml asserts owner/group/mode). Keep in sync with verify.yml. @@ -34,11 +37,32 @@ # verify.yml asserts each renders with the exact value AND type. Keep in sync. decdn_node_to_node_pull_through_enabled: true decdn_gc_interval_sec: 0 - decdn_pull_ahead_bytes: 2097152 - decdn_pull_share_ratio_percent: 400 - decdn_enable_0rtt: false + decdn_credit_max: 2097152 + decdn_relay_foreign_namespaces: false decdn_delivery_floor: 0 - decdn_settlement_auto_threshold_micro_usdc: 50000000 + decdn_redeem_threshold_micro_usdc: 50000000 + # A float knob: `| float` must render a bare TOML float, and verify.yml's type + # guards have to accept float (not just int) for these. Covers the one rendering + # form the old scenario had no example of. + decdn_security_per_source_rate_per_sec: 50.5 + decdn_serve_economics_discount: 0.25 + # One knob per NEW sub-table, so each table's header actually gets emitted and + # verify.yml can prove the scalar [cache] keys did not fall into it. + decdn_tinylfu_sketch_bytes: 524288 + decdn_serve_economics_n_max: 32 + decdn_origin_retry_max_retries: 5 + decdn_circuit_breaker_enabled: false + decdn_load_shed_per_client_serve_cap: 16 + decdn_receipts_retained_files: 8 + # Dict-shaped knobs: only a subset of the eight keys is set, which is the case + # the template's key loop has to get right (absent keys must not render). + decdn_dht_rate_limit: + per_peer_rate_per_sec: 10.0 + per_peer_burst: 20 + decdn_probe_rate_limit: + max_tracked_per_ip: 1024 + decdn_content_denied_hashes: + - "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc" # Two entries each so the {% for %} comma-join branch (loop.last) is exercised — # a single-element list never renders the separator. verify.yml asserts order. decdn_relay_urls: diff --git a/ansible/molecule/default/files/decdn-node-stub b/ansible/molecule/default/files/decdn-node-stub index 6196a2f..302cbcf 100755 --- a/ansible/molecule/default/files/decdn-node-stub +++ b/ansible/molecule/default/files/decdn-node-stub @@ -13,9 +13,13 @@ can be exercised end-to-end without a published release or a live chain: If the marker file NO_METRICS_MARKER exists, it instead blocks forever WITHOUT binding metrics — standing in for a healthy node whose metrics listener binds late (behind the - startup PaymentChannel bootstrap). The `slow-readiness` + startup PaymentPool bootstrap). The `slow-readiness` scenario uses this to prove the role's advisory probe warns (not fails) on timeout while the unit stays `running`. + * `config validate ...` + -> parse the rendered node.toml as TOML and exit 0. A stub has no + schema, so this checks syntax only; `molecule/schema` is what + covers key-level drift against the real upstream field list. * `key-gen ...` -> stand in for the CLI's wallet generator (exercised by the `generate-keystore` scenario). Enforces the contract the ROLE ASSUMES (not a verified real-CLI fact): the --password-file is @@ -30,6 +34,7 @@ It intentionally implements no protocol behaviour — the scenario verifies host prep, config/unit rendering, hardening, and loopback binding, not node logic. """ import os +import tomllib import sys import time from http.server import BaseHTTPRequestHandler, HTTPServer @@ -41,6 +46,9 @@ VERSION = "decdn-node 0.0.0-molecule-stub" # healthy node whose metrics listener binds late. The slow-readiness scenario # stages it (see that scenario's prepare.yml) to exercise the advisory-timeout path. NO_METRICS_MARKER = "/etc/decdn/stub-no-metrics" +# Forces `config validate` to exit 1, so a scenario can prove the role's config +# gate still fails the deploy (the gate itself runs with failed_when: false). +BAD_CONFIG_MARKER = "/etc/decdn/stub-bad-config" class _Handler(BaseHTTPRequestHandler): @@ -95,6 +103,41 @@ def key_gen(args): return 0 +def config_validate(args): + """Stub `decdn config validate`: parse the rendered node.toml and exit 0. + + The real CLI resolves the config through the daemon's own serde types, which is + what makes it the authoritative schema gate. A stub cannot do that -- it has no + schema -- so it does the one check it CAN do honestly: the file exists and is + syntactically valid TOML. Anything stronger here would be a lie that lets a + schema regression pass molecule. The `molecule/schema` scenario is what covers + key-level drift; see its README. + """ + config = _opt(args, "--config") + # Forced-failure marker, mirroring NO_METRICS_MARKER. The role's config gate + # deliberately runs with `failed_when: false` and defers to a following `fail` + # task, so a regression in THAT task's `when:` would silently no-op the whole + # gate. This lets a scenario prove the gate still fires. + if os.path.isfile(BAD_CONFIG_MARKER): + print(f"stub config validate: refusing {config} (forced by " + f"{BAD_CONFIG_MARKER})", file=sys.stderr) + return 1 + if not config: + print("stub config validate: --config is required", file=sys.stderr) + return 2 + if not os.path.isfile(config): + print(f"stub config validate: no such config file: {config}", file=sys.stderr) + return 1 + try: + with open(config, "rb") as fh: + tomllib.load(fh) + except tomllib.TOMLDecodeError as exc: + print(f"stub config validate: {config} is not valid TOML: {exc}", file=sys.stderr) + return 1 + print(f"stub config validate: {config} parsed") + return 0 + + def main(): args = sys.argv[1:] if "--version" in args: @@ -102,6 +145,8 @@ def main(): return 0 if args and args[0] == "key-gen": return key_gen(args[1:]) + if args[:2] == ["config", "validate"]: + return config_validate(args[2:]) if args and args[0] == "run": # Late-bind simulation: stay alive (Type=simple => unit stays `running`) # but never bind metrics, so the role's advisory probe times out. diff --git a/ansible/molecule/default/verify.yml b/ansible/molecule/default/verify.yml index 025c56d..0062716 100644 --- a/ansible/molecule/default/verify.yml +++ b/ansible/molecule/default/verify.yml @@ -78,9 +78,10 @@ ("observability", "metrics_bind"): "127.0.0.1", ("observability", "metrics_port"): 9090, ("blockchain", "chain_id"): 421614, - ("blockchain", "payment_channel_address"): "0x1111111111111111111111111111111111111111", + ("blockchain", "payment_pool_address"): "0x1111111111111111111111111111111111111111", ("blockchain", "capacity_bond_address"): "0x2222222222222222222222222222222222222222", ("blockchain", "slash_judge_address"): "0x3333333333333333333333333333333333333333", + ("blockchain", "content_blacklist_address"): "0x4444444444444444444444444444444444444444", } bad = { ".".join(k): (d.get(k[0], {}).get(k[1]), v) @@ -102,32 +103,83 @@ # `0 == False` and `1 == True`, so isinstance guards are load-bearing here). def _is_int(x): return isinstance(x, int) and not isinstance(x, bool) + def _is_float(x): + return isinstance(x, float) cache, net, pay, bc = (d.get(s, {}) for s in ("cache", "network", "payment", "blockchain")) + sec, shed = d.get("security", {}), d.get("load_shed", {}) + dht = d.get("dht", {}).get("rate_limit", {}) + probe = d.get("probe", {}).get("rate_limit", {}) knob_checks = [ ("cache.node_to_node_pull_through_enabled", cache.get("node_to_node_pull_through_enabled") is True), ("cache.gc_interval_sec (explicit 0 must emit)", _is_int(cache.get("gc_interval_sec")) and cache.get("gc_interval_sec") == 0), - ("cache.pull_ahead_bytes (bare-int Bytes)", - _is_int(cache.get("pull_ahead_bytes")) and cache.get("pull_ahead_bytes") == 2097152), - ("cache.pull_share_ratio_percent (bare-int Percent)", - _is_int(cache.get("pull_share_ratio_percent")) and cache.get("pull_share_ratio_percent") == 400), + ("cache.relay_foreign_namespaces (explicit false must emit)", + cache.get("relay_foreign_namespaces") is False), + ("payment.credit_max (bare-int Bytes)", + _is_int(pay.get("credit_max")) and pay.get("credit_max") == 2097152), # Two-element lists in exact order — exercises the comma-join branch. ("cache.pinned_hashes", cache.get("pinned_hashes") == ["a" * 64, "b" * 64]), ("network.relay_urls", net.get("relay_urls") == ["https://relay1.example.invalid:443", "https://relay2.example.invalid:443"]), - ("network.enable_0rtt", net.get("enable_0rtt") is False), ("payment.delivery_floor (explicit 0 must emit)", _is_int(pay.get("delivery_floor")) and pay.get("delivery_floor") == 0), - ("blockchain.settlement_auto_threshold_micro_usdc", - _is_int(bc.get("settlement_auto_threshold_micro_usdc")) - and bc.get("settlement_auto_threshold_micro_usdc") == 50000000), + ("blockchain.redeem_threshold_micro_usdc", + _is_int(bc.get("redeem_threshold_micro_usdc")) + and bc.get("redeem_threshold_micro_usdc") == 50000000), + # Floats must render as TOML floats, not ints or strings. + ("security.per_source_rate_per_sec (bare float)", + _is_float(sec.get("per_source_rate_per_sec")) + and sec.get("per_source_rate_per_sec") == 50.5), + ("cache.serve_economics.discount (bare float)", + _is_float(cache.get("serve_economics", {}).get("discount")) + and cache.get("serve_economics", {}).get("discount") == 0.25), + # One assertion per NEW sub-table: proves each header was emitted and + # that the value landed inside it. + ("cache.tinylfu", cache.get("tinylfu", {}).get("sketch_bytes") == 524288), + ("cache.serve_economics", cache.get("serve_economics", {}).get("n_max") == 32), + ("cache.origin_retry", cache.get("origin_retry", {}).get("max_retries") == 5), + ("cache.circuit_breaker", cache.get("circuit_breaker", {}).get("enabled") is False), + ("load_shed", shed.get("per_client_serve_cap") == 16), + ("receipts", d.get("receipts", {}).get("retained_files") == 8), + ("content.denied_hashes", d.get("content", {}).get("denied_hashes") == ["c" * 64]), + # Rate-limit tables render from a fixed key list: the keys converge set + # must be present with the right types, and the ones it did NOT set must + # be ABSENT (rendering a default would silently pin it). + ("dht.rate_limit set keys", + _is_float(dht.get("per_peer_rate_per_sec")) and dht.get("per_peer_rate_per_sec") == 10.0 + and _is_int(dht.get("per_peer_burst")) and dht.get("per_peer_burst") == 20), + ("dht.rate_limit unset keys stay absent", set(dht) == {"per_peer_rate_per_sec", "per_peer_burst"}), + ("probe.rate_limit unset keys stay absent", set(probe) == {"max_tracked_per_ip"}), ] + # The TOML sub-table hazard: every scalar [cache] key must be emitted BEFORE + # the first [cache.*] header, or it silently nests into that sub-table + # instead. The knobs at risk are the ones the template emits LAST among the + # scalars — relay_foreign_namespaces and the node_pull_* family sit directly + # above the first sub-table header, so they are the canaries. Assert they + # are top-level in [cache] AND that no sub-table has swallowed one. + _cache_scalars = {k for k, v in cache.items() if not isinstance(v, dict)} + _subtables = {k: v for k, v in cache.items() if isinstance(v, dict)} + _leaked = { + f"{tbl}.{key}" + for tbl, body in _subtables.items() + for key in body + if key in ("relay_foreign_namespaces", "node_to_node_pull_through_enabled", + "eviction_policy", "admission_policy", "cache_size_mb", "cache_dir") + } + knob_checks.append( + ("cache scalars did not fall into a sub-table", + not _leaked + and {"cache_dir", "cache_size_mb", "relay_foreign_namespaces", + "node_to_node_pull_through_enabled"} <= _cache_scalars), + ) knob_bad = [name for name, ok in knob_checks if not ok] if knob_bad: print("node.toml tuning-knob mismatch:", knob_bad, file=sys.stderr) print(" cache=", cache, "\n network=", net, "\n payment=", pay, file=sys.stderr) + print(" security=", sec, "\n load_shed=", shed, file=sys.stderr) + print(" leaked-into-subtable=", _leaked, file=sys.stderr) sys.exit(1) - name: Assert node.toml is valid TOML and renders the expected content diff --git a/ansible/molecule/generate-keystore/converge.yml b/ansible/molecule/generate-keystore/converge.yml index e7b7eb3..8a42d1f 100644 --- a/ansible/molecule/generate-keystore/converge.yml +++ b/ansible/molecule/generate-keystore/converge.yml @@ -19,9 +19,11 @@ decdn_rpc_url: "https://rpc.example.invalid/" decdn_chain_id: 421614 decdn_region: "US" - decdn_payment_channel_address: "0x1111111111111111111111111111111111111111" + decdn_payment_pool_address: "0x1111111111111111111111111111111111111111" decdn_capacity_bond_address: "0x2222222222222222222222222222222222222222" decdn_slash_judge_address: "0x3333333333333333333333333333333333333333" + # REQUIRED upstream (ADR 011/031) — the role asserts it with the other three. + decdn_content_blacklist_address: "0x4444444444444444444444444444444444444444" decdn_cache_origin_kind: "http" decdn_cache_origin_url: "https://origin.example.invalid/" roles: diff --git a/ansible/molecule/schema/README.md b/ansible/molecule/schema/README.md new file mode 100644 index 0000000..897f140 --- /dev/null +++ b/ansible/molecule/schema/README.md @@ -0,0 +1,41 @@ +# `schema` scenario — config key-set drift guard + +Renders `node.toml` with **every** operator-facing knob set, then asserts that +every key it emits exists in the upstream deCDN config schema. + +## Why this exists + +Upstream marks every config section `#[serde(deny_unknown_fields)]` and defines +**no** serde aliases anywhere in `crates/common/src/config/`. A key this role +renders that the daemon does not know is not a warning — it is a failed config +load and a systemd crash-loop. + +Nothing else in this repo catches that: + +- `default` asserts values and types, but only for keys it already knows about. +- The stub daemon (`molecule/default/files/decdn-node-stub`) has no schema. Its + `config validate` checks TOML syntax, which a stale key passes. +- The role's own Ansible asserts mirror upstream constraints *by hand*, which is + exactly the thing that drifted. + +When this repo last fell behind (`decdn/decdn` @ `d306cc5c`), the role was +emitting 18 keys the daemon had removed or renamed — `payment_channel_address`, +the `[gossip]` table, `voucher_interval_mb`, the three `*_from_block` knobs, and +more. Every test passed. This scenario is that regression. + +## Keeping it current + +`files/schema-keys.txt` is generated from the upstream source; its header carries +the exact regeneration command. Re-run it whenever you sync this repo against a +new `decdn/decdn` revision, **before** touching the template — the diff on that +file is the changelog of what the config schema did. + +When you add a knob to the role, add it to `converge.yml` as well. A knob that is +never set is a knob this guard cannot see; `verify.yml`'s breadth check is a +backstop against the converge quietly collapsing, not a substitute for that. + +## What it does not check + +Value ranges and cross-field constraints — `validation` covers the reject path, +and the real gate is `decdn config validate` running against an actual binary in +`tasks/main.yml`. This scenario is about key names only. diff --git a/ansible/molecule/schema/converge.yml b/ansible/molecule/schema/converge.yml new file mode 100644 index 0000000..b766f32 --- /dev/null +++ b/ansible/molecule/schema/converge.yml @@ -0,0 +1,208 @@ +--- +# Render node.toml with EVERY operator-facing knob set to a non-default value, so +# the verify step sees the maximal key set the role is capable of emitting. Values +# are chosen to be individually VALID (the role's own asserts still run and must +# pass) — this scenario is about key names, not range checking; `validation` covers +# the reject path. +# +# When you add a knob to the role, add it here too. A knob that is never set is a +# knob this guard cannot see. +- name: Converge + hosts: all + become: true + vars: + stub_bin: "{{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/../default/files/decdn-node-stub" + decdn_node_install_method: manual + decdn_node_manual_bin_src: "{{ stub_bin }}" + decdn_cli_manual_bin_src: "{{ stub_bin }}" + decdn_node_version: "0.0.0-molecule-stub" + + # --- required --- + decdn_rpc_url: "https://rpc.example.invalid/" + decdn_chain_id: 421614 + decdn_region: "DE" + decdn_payment_pool_address: "0x1111111111111111111111111111111111111111" + decdn_capacity_bond_address: "0x2222222222222222222222222222222222222222" + decdn_slash_judge_address: "0x3333333333333333333333333333333333333333" + decdn_content_blacklist_address: "0x4444444444444444444444444444444444444444" + + # --- [network] + [network.discovery] --- + decdn_relay_urls: ["https://relay1.example.invalid:443", "https://relay2.example.invalid:443"] + decdn_discovery_pkarr_url: "https://pkarr.example.invalid/" + decdn_discovery_dns_origin: "discovery.example.invalid" + # A REAL ed25519 public key, not 64 arbitrary hex chars: upstream parses this as + # an iroh NodeId and rejects a non-curve-point, so a placeholder would pass the + # role's shape regex and still fail `decdn config validate` against a real binary. + decdn_discovery_peers: + "253bad481e6371866c9f6276b2a7b3a10ca16255668e740e6fc01da1cacc4350": + relay_url: "https://relay.example.invalid/" + addrs: ["203.0.113.4:4433", "203.0.113.5:4433"] + + # --- [blockchain] optional + CLI-only --- + decdn_slash_appeal_address: "0x5555555555555555555555555555555555555555" + decdn_origin_assignment_address: "0x6666666666666666666666666666666666666666" + decdn_publisher_registry_address: "0x7777777777777777777777777777777777777777" + decdn_usdc_address: "0x8888888888888888888888888888888888888888" + decdn_swap_venue: "uniswap-v3" + decdn_swap_router_address: "0x9999999999999999999999999999999999999999" + decdn_swap_quoter_address: "0xaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaAaA" + decdn_swap_fee_tier: 3000 + decdn_swap_balancer_pool: "0xbBbBbBbBbBbBbBbBbBbBbBbBbBbBbBbBbBbBbBbB" + decdn_swap_pool_address: "0xcCcCcCcCcCcCcCcCcCcCcCcCcCcCcCcCcCcCcCcC" + decdn_origin_directory_positive_ttl_sec: 600 + decdn_origin_directory_negative_ttl_sec: 60 + decdn_origin_directory_cache_capacity: 8192 + decdn_content_blacklist_poll_interval_sec: 300 + decdn_rpc_watchdog_interval_sec: 15 + decdn_event_poll_interval_ms: 5000 + decdn_rate_bounds_poll_interval_sec: 1800 + decdn_fee_shares_poll_interval_sec: 1800 + decdn_redeem_threshold_micro_usdc: 2000000 + decdn_redeem_max_vouchers_per_tx: 250 + decdn_redeem_interval_secs: 120 + decdn_buyer_working_deposit_micro_usdc: 20000000 + decdn_buyer_max_approve: false + decdn_pool_min_remaining_deposit_micro_usdc: 500000 + decdn_pool_floor_signer_share_bps: 3000 + decdn_pool_floor_signer_max_windows: 4 + + # --- [payment] --- + decdn_rate_per_mb: 12 + decdn_delivery_floor: 0 + decdn_credit_max: 33554432 + decdn_credit_ramp_divisor: 4 + decdn_frame_target_bytes: 262144 + decdn_voucher_commit_interval_ms: 2500 + + # --- [cache] scalars --- + decdn_cache_size_mb: 20480 + decdn_max_blob_size_mb: 2048 + decdn_max_rate_per_mb: 50 + decdn_pinned_hashes: ["aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"] + decdn_cache_user_agent: "decdn-node/molecule" + decdn_gc_interval_sec: 0 + decdn_fs_rescan_interval_sec: 30 + decdn_origin_probe_ttl_sec: 30 + decdn_origin_probe_negative_ttl_sec: 4 + decdn_origin_probe_fault_ttl_sec: 8 + decdn_origin_probe_timeout_ms: 1500 + decdn_origin_probe_memo_capacity: 2048 + decdn_eviction_high_water_pct: 85 + decdn_eviction_target_pct: 70 + decdn_eviction_per_sweep_budget: 32 + decdn_eviction_tick_secs: 2 + decdn_max_probe_holds: 512 + decdn_stake_lane_reserved_holds: 16 + decdn_node_to_node_pull_through_enabled: true + decdn_relay_foreign_namespaces: false + decdn_node_pull_probe_fanout: 8 + decdn_node_pull_timeout_sec: 30 + decdn_node_pull_stall_window_sec: 15 + decdn_node_pull_min_throughput_bps: 8192 + decdn_eviction_policy: "tinylfu" + decdn_admission_policy: "tinylfu" + + # --- [cache.*] sub-tables --- + decdn_tinylfu_sketch_bytes: 524288 + decdn_tinylfu_promotion_threshold: 3 + decdn_tinylfu_probation_target_pct: 20 + decdn_tinylfu_aging_halflife_sec: 900 + decdn_serve_economics_policy: "margin" + decdn_serve_economics_discount: 0.25 + decdn_serve_economics_n_max: 32 + decdn_serve_economics_warming_budget: 1000000 + decdn_serve_economics_warming_refill: 100 + decdn_origin_retry_max_retries: 5 + decdn_origin_retry_initial_backoff_ms: 200 + decdn_origin_retry_max_backoff_ms: 20000 + decdn_origin_retry_jitter_ratio: 0.2 + decdn_origin_retry_buffered_max_bytes: 8388608 + decdn_circuit_breaker_enabled: false + decdn_circuit_breaker_failure_threshold: 10 + decdn_circuit_breaker_cooldown_ms: 60000 + decdn_circuit_breaker_half_open_max_calls: 2 + + # s3 origin: the widest [cache.origin] variant, plus the default-chain + # credential sub-table. The mutually-exclusive [[cache.origins]] form is + # covered by the second play below, which re-renders to a different path. + decdn_cache_origin_kind: "s3" + decdn_cache_origin_s3_bucket: "decdn-blobs" + decdn_cache_origin_s3_region: "us-east-1" + decdn_cache_origin_s3_endpoint_url: "https://acct.r2.cloudflarestorage.invalid" + decdn_cache_origin_s3_path_style: true + decdn_cache_origin_s3_prefix: "blobs/" + decdn_cache_origin_decompress: "strict" + decdn_cache_origin_s3_use_default_chain: true + decdn_cache_origin_s3_profile: "decdn" + + # --- [observability] / [security] / [load_shed] / [dht] / [probe] --- + decdn_otlp_endpoint: "http://localhost:4317" + decdn_security_max_concurrent_handlers: 128 + decdn_security_per_source_rate_per_sec: 50.5 + decdn_security_per_source_burst: 100 + decdn_security_max_tracked_sources: 2048 + decdn_load_shed_policy: "always-admit" + decdn_load_shed_egress_budget_mbps: 500 + decdn_load_shed_max_concurrent_serves_high: 128 + decdn_load_shed_max_concurrent_serves_low: 96 + decdn_load_shed_per_client_serve_cap: 16 + # All eight keys on both tables, so the guard sees the complete rate-limit shape. + decdn_dht_rate_limit: + per_peer_rate_per_sec: 10.0 + per_peer_burst: 20 + per_ip_rate_per_sec: 50.0 + per_ip_burst: 100 + global_rate_per_sec: 500.0 + global_burst: 1000 + max_tracked_per_ip: 2048 + max_tracked_per_peer: 2048 + decdn_probe_rate_limit: + per_peer_rate_per_sec: 2.5 + per_peer_burst: 5 + per_ip_rate_per_sec: 25.0 + per_ip_burst: 100 + global_rate_per_sec: 500.0 + global_burst: 1000 + max_tracked_per_ip: 1024 + max_tracked_per_peer: 1024 + + # --- [receipts] / [content] --- + decdn_receipts_max_file_bytes: 67108864 + decdn_receipts_retained_files: 8 + decdn_content_denied_hashes: ["bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"] + decdn_content_denied_origins: ["0x000000000000000000000000000000000000dEaD"] + roles: + - role: decdn_node + +# The [[cache.origins]] array-of-tables form cannot coexist with [cache.origin] in +# one render — the role asserts they are mutually exclusive, because upstream +# rejects a config carrying both. So it gets its own render, to a separate path, +# covering the http and fs variants the play above (s3) does not. Without this the +# template's `_multi` branch and the role's per-entry origin validation would ship +# untested, and verify.yml's array-of-tables walk would be dead code. +- name: Converge (multi-origin variant) + hosts: all + become: true + vars: + stub_bin: "{{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/../default/files/decdn-node-stub" + decdn_node_install_method: manual + decdn_node_manual_bin_src: "{{ stub_bin }}" + decdn_cli_manual_bin_src: "{{ stub_bin }}" + decdn_node_version: "0.0.0-molecule-stub" + decdn_config_file: /etc/decdn/node-origins.toml + decdn_rpc_url: "https://rpc.example.invalid/" + decdn_chain_id: 421614 + decdn_region: "DE" + decdn_payment_pool_address: "0x1111111111111111111111111111111111111111" + decdn_capacity_bond_address: "0x2222222222222222222222222222222222222222" + decdn_slash_judge_address: "0x3333333333333333333333333333333333333333" + decdn_content_blacklist_address: "0x4444444444444444444444444444444444444444" + # Empty, so the scalar form is suppressed and the list form renders. + decdn_cache_origin_kind: "" + decdn_cache_origins: + - {kind: "http", url: "https://primary.example.invalid/", decompress: "strict"} + - {kind: "fs", path: "/var/lib/decdn/origin"} + - {kind: "s3", bucket: "mirror-blobs", region: "eu-west-1", path_style: false, + endpoint_url: "https://acct.b2.example.invalid", prefix: "blobs/"} + roles: + - role: decdn_node diff --git a/ansible/molecule/schema/files/gen-schema-keys.py b/ansible/molecule/schema/files/gen-schema-keys.py new file mode 100755 index 0000000..10d91f3 --- /dev/null +++ b/ansible/molecule/schema/files/gen-schema-keys.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python3 +"""Regenerate schema-keys.txt: every FULLY-QUALIFIED config path upstream accepts. + +Run from anywhere: + + ./gen-schema-keys.py ../../../../../decdn > schema-keys.txt + +Fully-qualified, not bare leaf names. An earlier version of this inventory listed +only leaf names, which let a key sitting in the WRONG table pass -- including the +exact hazard node.toml.j2's header warns about, where a scalar [cache] key emitted +below a [cache.*] sub-table header silently nests into that sub-table. `sketch_bytes` +is legal at cache.tinylfu.sketch_bytes and a startup failure at cache.sketch_bytes, +so the path is the thing that has to be checked. + +Field names are parsed out of the upstream sources; the struct -> TOML path mapping +below is the one thing that cannot be derived from them, because it is expressed in +the *shape* of FileConfig rather than in any struct's own text. Keep it in step when +upstream adds a section. +""" + +import re +import sys +from pathlib import Path + +# struct/enum name -> the TOML path its fields live at. `None` means the type is a +# container whose own fields are other sections (FileConfig), so its field names are +# section names already covered by the entries below. +STRUCT_PATHS = { + "FileConfig": None, + "IdentityConfig": "identity", + "NetworkConfig": "network", + "DiscoveryConfig": "network.discovery", + # Keyed by NodeId, so the concrete hop is a wildcard. + "DiscoveryPeer": "network.discovery.peers.*", + "BlockchainConfig": "blockchain", + "CacheConfig": "cache", + "TinyLfuConfig": "cache.tinylfu", + "ServeEconomicsConfig": "cache.serve_economics", + "RetryPolicy": "cache.origin_retry", + "CircuitBreakerPolicy": "cache.circuit_breaker", + "PaymentConfig": "payment", + "ObservabilityConfig": "observability", + "SecurityConfig": "security", + "LoadShedConfig": "load_shed", + "DhtConfig": "dht", + "DhtRateLimitConfig": "dht.rate_limit", + "ProbeConfig": "probe", + "ProbeRateLimitConfig": "probe.rate_limit", + "ReceiptsConfig": "receipts", + "ContentConfig": "content", +} + +# The single [cache.origin] table and the [[cache.origins]] array of tables take the +# same fields, so every origin-shaped path is emitted under both prefixes. +ORIGIN_PREFIXES = ("cache.origin", "cache.origins") + +# OriginConfig is `#[serde(tag = "kind")]` over Http { url, decompress }, +# Fs { path } and S3(S3OriginConfig): the tag and the variant payloads are not +# struct fields anywhere, so they are named here. S3Credentials is likewise +# `#[serde(tag = "source")]` over Static {...} and DefaultChain { profile }; only +# the default-chain arm is templated by this role (static AWS keys must never be +# written into the 0640 node.toml), so only its fields are listed. +ORIGIN_ENUM_FIELDS = ["kind", "url", "decompress", "path"] +CREDENTIAL_FIELDS = ["source", "profile"] + +FIELD_RE = re.compile(r"^\s+pub\s+([a-z_0-9]+)\s*:") +TYPE_RE = re.compile(r"^pub\s+(?:struct|enum)\s+([A-Za-z0-9_]+)") + + +def fields_by_struct(paths): + """Map every `pub struct`/`pub enum` name to its own `pub field:` names.""" + out, current = {}, None + for path in paths: + for line in Path(path).read_text().splitlines(): + type_match = TYPE_RE.match(line) + if type_match: + current = type_match.group(1) + out.setdefault(current, []) + continue + field_match = FIELD_RE.match(line) + if field_match and current: + out[current].append(field_match.group(1)) + return out + + +def main(): + if len(sys.argv) != 2: + sys.exit(f"usage: {sys.argv[0]} ") + decdn = Path(sys.argv[1]) + sources = [ + decdn / "crates/common/src/config/types.rs", + decdn / "crates/config-types/src/retry.rs", + decdn / "crates/config-types/src/circuit_breaker.rs", + ] + missing = [str(s) for s in sources if not s.is_file()] + if missing: + sys.exit(f"not a decdn checkout -- missing: {', '.join(missing)}") + + by_struct = fields_by_struct(sources) + + unknown = sorted( + name for name in STRUCT_PATHS if name not in by_struct + ) + if unknown: + sys.exit( + "these structs are in STRUCT_PATHS but not in the upstream sources " + f"(renamed or removed upstream?): {', '.join(unknown)}" + ) + + paths = set() + for struct, prefix in STRUCT_PATHS.items(): + if prefix is None: + continue + for field in by_struct[struct]: + paths.add(f"{prefix}.{field}") + + # S3OriginConfig's fields, plus the enum tag/payload names, under both the + # singular and the array-of-tables prefix. + for prefix in ORIGIN_PREFIXES: + for field in by_struct["S3OriginConfig"] + ORIGIN_ENUM_FIELDS: + paths.add(f"{prefix}.{field}") + for field in CREDENTIAL_FIELDS: + paths.add(f"{prefix}.credentials.{field}") + + print( + "# Every FULLY-QUALIFIED config path the deCDN node accepts, one per line.\n" + "#\n" + "# GENERATED -- do not hand-edit. Regenerate with:\n" + "#\n" + "# ./gen-schema-keys.py > schema-keys.txt\n" + "#\n" + "# Paths, not bare leaf names: `sketch_bytes` is legal at\n" + "# cache.tinylfu.sketch_bytes and a startup failure at cache.sketch_bytes, and\n" + "# a leaf-name inventory cannot tell the two apart. See gen-schema-keys.py for\n" + "# the struct -> path mapping and molecule/schema/README.md for why this exists.\n" + "#\n" + f"# Synced from decdn/decdn @ d306cc5c (crate version 0.1.1): {len(paths)} paths." + ) + for path in sorted(paths): + print(path) + + +if __name__ == "__main__": + main() diff --git a/ansible/molecule/schema/files/schema-keys.txt b/ansible/molecule/schema/files/schema-keys.txt new file mode 100644 index 0000000..3364b49 --- /dev/null +++ b/ansible/molecule/schema/files/schema-keys.txt @@ -0,0 +1,173 @@ +# Every FULLY-QUALIFIED config path the deCDN node accepts, one per line. +# +# GENERATED -- do not hand-edit. Regenerate with: +# +# ./gen-schema-keys.py > schema-keys.txt +# +# Paths, not bare leaf names: `sketch_bytes` is legal at +# cache.tinylfu.sketch_bytes and a startup failure at cache.sketch_bytes, and +# a leaf-name inventory cannot tell the two apart. See gen-schema-keys.py for +# the struct -> path mapping and molecule/schema/README.md for why this exists. +# +# Synced from decdn/decdn @ d306cc5c (crate version 0.1.1): 161 paths. +blockchain.buyer_max_approve +blockchain.buyer_working_deposit_micro_usdc +blockchain.capacity_bond_address +blockchain.chain_id +blockchain.content_blacklist_address +blockchain.content_blacklist_poll_interval_sec +blockchain.eth_keystore +blockchain.event_poll_interval_ms +blockchain.fee_shares_poll_interval_sec +blockchain.origin_assignment_address +blockchain.origin_directory_cache_capacity +blockchain.origin_directory_negative_ttl_sec +blockchain.origin_directory_positive_ttl_sec +blockchain.payment_pool_address +blockchain.pool_floor_signer_max_windows +blockchain.pool_floor_signer_share_bps +blockchain.pool_min_remaining_deposit_micro_usdc +blockchain.publisher_registry_address +blockchain.rate_bounds_poll_interval_sec +blockchain.redeem_interval_secs +blockchain.redeem_max_vouchers_per_tx +blockchain.redeem_threshold_micro_usdc +blockchain.rpc_url +blockchain.rpc_watchdog_interval_sec +blockchain.slash_appeal_address +blockchain.slash_judge_address +blockchain.swap_balancer_pool +blockchain.swap_fee_tier +blockchain.swap_pool_address +blockchain.swap_quoter_address +blockchain.swap_router_address +blockchain.swap_venue +blockchain.usdc_address +cache.admission_policy +cache.cache_dir +cache.cache_size_mb +cache.circuit_breaker +cache.circuit_breaker.cooldown_ms +cache.circuit_breaker.enabled +cache.circuit_breaker.failure_threshold +cache.circuit_breaker.half_open_max_calls +cache.eviction_high_water_pct +cache.eviction_per_sweep_budget +cache.eviction_policy +cache.eviction_target_pct +cache.eviction_tick_secs +cache.fs_rescan_interval_sec +cache.gc_interval_sec +cache.max_blob_size_mb +cache.max_probe_holds +cache.max_rate_per_mb +cache.node_pull_min_throughput_bps +cache.node_pull_probe_fanout +cache.node_pull_stall_window_sec +cache.node_pull_timeout_sec +cache.node_to_node_pull_through_enabled +cache.origin +cache.origin.bucket +cache.origin.credentials +cache.origin.credentials.profile +cache.origin.credentials.source +cache.origin.decompress +cache.origin.endpoint_url +cache.origin.kind +cache.origin.path +cache.origin.path_style +cache.origin.prefix +cache.origin.region +cache.origin.url +cache.origin_probe_fault_ttl_sec +cache.origin_probe_memo_capacity +cache.origin_probe_negative_ttl_sec +cache.origin_probe_timeout_ms +cache.origin_probe_ttl_sec +cache.origin_retry +cache.origin_retry.buffered_max_bytes +cache.origin_retry.initial_backoff_ms +cache.origin_retry.jitter_ratio +cache.origin_retry.max_backoff_ms +cache.origin_retry.max_retries +cache.origins +cache.origins.bucket +cache.origins.credentials +cache.origins.credentials.profile +cache.origins.credentials.source +cache.origins.decompress +cache.origins.endpoint_url +cache.origins.kind +cache.origins.path +cache.origins.path_style +cache.origins.prefix +cache.origins.region +cache.origins.url +cache.pinned_hashes +cache.relay_foreign_namespaces +cache.serve_economics +cache.serve_economics.discount +cache.serve_economics.n_max +cache.serve_economics.policy +cache.serve_economics.warming_budget +cache.serve_economics.warming_refill +cache.stake_lane_reserved_holds +cache.tinylfu +cache.tinylfu.aging_halflife_sec +cache.tinylfu.probation_target_pct +cache.tinylfu.promotion_threshold +cache.tinylfu.sketch_bytes +cache.user_agent +content.denied_hashes +content.denied_origins +dht.rate_limit +dht.rate_limit.global_burst +dht.rate_limit.global_rate_per_sec +dht.rate_limit.max_tracked_per_ip +dht.rate_limit.max_tracked_per_peer +dht.rate_limit.per_ip_burst +dht.rate_limit.per_ip_rate_per_sec +dht.rate_limit.per_peer_burst +dht.rate_limit.per_peer_rate_per_sec +identity.data_dir +identity.region +load_shed.egress_budget_mbps +load_shed.max_concurrent_serves_high +load_shed.max_concurrent_serves_low +load_shed.per_client_serve_cap +load_shed.policy +network.bind_port +network.discovery +network.discovery.dns_origin +network.discovery.peers +network.discovery.peers.*.addrs +network.discovery.peers.*.relay_url +network.discovery.pkarr_url +network.relay_urls +observability.admin_port +observability.log_format +observability.log_level +observability.metrics_bind +observability.metrics_port +observability.otlp_endpoint +payment.credit_max +payment.credit_ramp_divisor +payment.delivery_floor +payment.frame_target_bytes +payment.rate_per_mb +payment.voucher_commit_interval_ms +probe.rate_limit +probe.rate_limit.global_burst +probe.rate_limit.global_rate_per_sec +probe.rate_limit.max_tracked_per_ip +probe.rate_limit.max_tracked_per_peer +probe.rate_limit.per_ip_burst +probe.rate_limit.per_ip_rate_per_sec +probe.rate_limit.per_peer_burst +probe.rate_limit.per_peer_rate_per_sec +receipts.max_file_bytes +receipts.retained_files +security.max_concurrent_handlers +security.max_tracked_sources +security.per_source_burst +security.per_source_rate_per_sec diff --git a/ansible/molecule/schema/molecule.yml b/ansible/molecule/schema/molecule.yml new file mode 100644 index 0000000..1c2afe2 --- /dev/null +++ b/ansible/molecule/schema/molecule.yml @@ -0,0 +1,47 @@ +--- +# Schema-drift guard: render node.toml with EVERY knob set, then assert that every +# key it emits exists in the upstream config schema. See README.md in this dir. +# +# This is the regression that would have caught the d306cc5c sync: the role had +# been rendering 18 keys the daemon no longer accepts, and nothing failed until a +# real binary tried to load the file. Every config section upstream is +# deny_unknown_fields with no serde aliases, so one stale key is a startup +# crash-loop -- and the stub daemon cannot catch that, because a stub has no schema. +# Hence the committed key inventory in files/schema-keys.txt. +# +# `baseline` is NOT exercised here, for the same reasons as the `default` scenario. +# There is no idempotence step: this scenario asserts on rendered content, and +# `default` already covers idempotence. +dependency: + name: galaxy + options: + requirements-file: ../../requirements.yml +driver: + name: docker +platforms: + - name: decdn-node-molecule-schema + # Pinned by digest for reproducible CI (repo convention — cf. the SHA-pinned + # actions/images in .github/workflows/ci.yml). Tag: :latest as of 2026-07-11 + # (this image publishes only :latest). Re-resolve the digest to bump. + image: geerlingguy/docker-debian12-ansible@sha256:4553092be2c00b1ffe580927b9ff03f3c3a0df32b7dd693a3eb02efb6c2b77b7 + pre_build_image: true + command: /usr/lib/systemd/systemd + privileged: true + cgroupns_mode: host + volumes: + - /sys/fs/cgroup:/sys/fs/cgroup:rw +provisioner: + name: ansible + env: + ANSIBLE_ROLES_PATH: "${MOLECULE_PROJECT_DIRECTORY}/roles" + ANSIBLE_COLLECTIONS_PATH: "${MOLECULE_PROJECT_DIRECTORY}/collections" +verifier: + name: ansible +scenario: + test_sequence: + - dependency + - create + - prepare + - converge + - verify + - destroy diff --git a/ansible/molecule/schema/prepare.yml b/ansible/molecule/schema/prepare.yml new file mode 100644 index 0000000..0f467c6 --- /dev/null +++ b/ansible/molecule/schema/prepare.yml @@ -0,0 +1,44 @@ +--- +# The decdn_node role requires an operator-provisioned eth keystore + password +# file to already exist before it starts the daemon (it never generates them). +# Stage placeholder files here so the role's keystore gate passes in CI. The +# stub daemon ignores them; this scenario does not exercise real wallet logic. +- name: Prepare + hosts: all + become: true + vars: + decdn_home: /var/lib/decdn + decdn_etc: /etc/decdn + tasks: + - name: Ensure data + config directories exist + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: "0755" + loop: + - "{{ decdn_home }}" + - "{{ decdn_etc }}" + + # Stage these at 0644 (deliberately looser than the target). The role locks + # them to 0600; verify.yml asserts 0600, so the assertion actually attributes + # the mode to the role rather than to what prepare pre-set. + - name: Stage a placeholder eth keystore + ansible.builtin.copy: + dest: "{{ decdn_home }}/keystore.json" + content: "{{ '{}' }}\n" + mode: "0644" + + - name: Stage a placeholder keystore password file + ansible.builtin.copy: + dest: "{{ decdn_etc }}/keystore.password" + content: "molecule-placeholder\n" + mode: "0644" + + # The role's pre-start gate also requires the node identity (node.secret), a + # co-equal key-gen output. Stage it at 0644 so verify.yml can attribute the + # 0600 lock-down to the role. + - name: Stage a placeholder node identity + ansible.builtin.copy: + dest: "{{ decdn_home }}/node.secret" + content: "molecule-placeholder-node-secret\n" + mode: "0644" diff --git a/ansible/molecule/schema/verify.yml b/ansible/molecule/schema/verify.yml new file mode 100644 index 0000000..7c8cbe1 --- /dev/null +++ b/ansible/molecule/schema/verify.yml @@ -0,0 +1,191 @@ +--- +# Assert that every key the role rendered into node.toml exists in the upstream +# config schema, and that the file is valid TOML. +# +# The `default` scenario checks VALUES; this one checks the KEY SET. They fail on +# different regressions: default catches a knob rendered with the wrong type or +# dropped entirely, schema catches a knob rendered under a name the daemon will +# reject at load. Only the second would have caught the d306cc5c drift. +- name: Verify + hosts: all + become: true + tasks: + - name: Stage the upstream schema key inventory + ansible.builtin.copy: + src: schema-keys.txt + dest: /root/schema-keys.txt + mode: "0644" + + - name: Stage the schema-drift checker + ansible.builtin.copy: + dest: /root/molecule-assert-schema.py + mode: "0755" + content: | + """Flag any config PATH in the rendered node.toml that upstream rejects.""" + import sys, tomllib + + config_path = sys.argv[1] if len(sys.argv) > 1 else "/etc/decdn/node.toml" + with open(config_path, "rb") as fh: + config = tomllib.load(fh) + with open("/root/schema-keys.txt", encoding="utf-8") as fh: + known = { + line.strip() for line in fh + if line.strip() and not line.startswith("#") + } + + def _lookup(root, dotted): + """Resolve a dotted path back to its value, for leaf/table triage.""" + node = root + for part in dotted.split("."): + if isinstance(node, list): + node = node[0] if node else {} + if not isinstance(node, dict) or part not in node: + return None + node = node[part] + return node + + def walk(node, path=""): + """Yield the dotted path of every key in every table.""" + if isinstance(node, dict): + for key, value in node.items(): + here = f"{path}.{key}" if path else key + yield here + yield from walk(value, here) + elif isinstance(node, list): + # [[cache.origins]] is an array of tables. Every element shares one + # schema, so they collapse onto the same path -- an index would make + # the inventory depend on how many origins an operator configured. + for item in node: + if isinstance(item, dict): + yield from walk(item, path) + + def normalize(dotted): + """Collapse the one path hop that is data rather than a schema field. + + [network.discovery.peers.] is keyed by a 64-char NodeId, so the + hop itself is never in the inventory; the DiscoveryPeer fields BELOW it + are (as network.discovery.peers.*). Only that single hop is rewritten, + so a bogus key beside relay_url/addrs is still caught. + """ + prefix = "network.discovery.peers." + if dotted.startswith(prefix): + rest = dotted[len(prefix):].split(".", 1) + return prefix + "*" + ("." + rest[1] if len(rest) > 1 else "") + return dotted + + # A table header is itself a path (`cache.tinylfu`), and intermediate tables + # are not fields of anything -- they are the sections the inventory is keyed + # BY. Only leaf paths are checked; a bogus TABLE surfaces as its children + # being unknown, or (if empty) as the section-set assertion below. + unknown = sorted({ + normalize(dotted) for dotted in walk(config) + if not isinstance(_lookup(config, dotted), (dict, list)) + and normalize(dotted) not in known + }) + + if unknown: + print("node.toml emits paths absent from the upstream config schema:", + file=sys.stderr) + for path in unknown: + print(f" {path}", file=sys.stderr) + print( + "\nEvery config section upstream is deny_unknown_fields with no " + "serde aliases, so each of these is a daemon startup failure. Note " + "a path can be wrong because the KEY is unknown or because a known " + "key landed in the wrong TABLE -- a scalar [cache] key emitted below " + "a [cache.*] header nests into it silently. Re-sync " + "roles/decdn_node/templates/node.toml.j2, then regenerate " + "molecule/schema/files/schema-keys.txt (see gen-schema-keys.py).", + file=sys.stderr, + ) + sys.exit(1) + + print(f"schema OK ({config_path}): {len(known)} known paths, " + f"{sum(1 for _ in walk(config))} emitted, all recognised") + + - name: Assert node.toml emits no key outside the upstream schema + ansible.builtin.command: + cmd: python3 /root/molecule-assert-schema.py + changed_when: false + + # The second converge play renders the mutually-exclusive [[cache.origins]] + # form here. Same checker, different file: this is what exercises the walker's + # array-of-tables branch and the http/fs origin variants. + - name: Assert the multi-origin render emits no key outside the schema + ansible.builtin.command: + cmd: python3 /root/molecule-assert-schema.py /etc/decdn/node-origins.toml + changed_when: false + + # A path check cannot catch this one: `rpc_url` is a legitimate [blockchain] + # field, so a template regression that emitted it would be a VALID path and + # sail through the checker above. node.toml is 0640 and diffable by design + # (AGENTS.md hard rule 1) — the RPC URL may embed an API key and belongs only + # in the 0600 env file. Assert the absence explicitly. + - name: Assert no secret leaked into the world-diffable node.toml + ansible.builtin.shell: + executable: /bin/bash + cmd: | + set -euo pipefail + leaked=0 + for pattern in '^rpc_url' '^[a-z_]*password' '^[a-z_]*secret' \ + '^access_key_id' '^secret_access_key' '^session_token'; do + if grep -qE "$pattern" /etc/decdn/node.toml; then + echo "node.toml contains a secret-bearing key matching $pattern" >&2 + leaked=1 + fi + done + exit "$leaked" + changed_when: false + + # A guard on the guard: if the converge stopped setting most knobs (someone + # trimmed it, or a rename silently dropped a block), the checks above would + # pass trivially on a nearly-empty file. + # + # The SECTION SET is asserted exactly, not counted: sections are stable (a new + # knob adds a key, not a table), so an exact set catches a whole vanished + # section — e.g. the ~27-key optional [blockchain] block, which is the shape + # the original drift actually took and which a loose count would tolerate. + - name: Assert the rendered config is as wide as this scenario intends + ansible.builtin.shell: + # /bin/sh is dash on Debian and has no `pipefail`; this needs bash. + executable: /bin/bash + cmd: | + set -euo pipefail + # `|| true`: grep -c exits 1 on zero matches, which under `set -e` would + # abort before the diagnostic below could name what was missing. + keys=$(grep -cE '^[a-z_0-9]+ = ' /etc/decdn/node.toml || true) + tables=$(grep -oE '^\[+[a-z_0-9.]+' /etc/decdn/node.toml | tr -d '[' | sort -u) + expected='blockchain + cache + cache.circuit_breaker + cache.origin + cache.origin.credentials + cache.origin_retry + cache.serve_economics + cache.tinylfu + content + dht.rate_limit + identity + load_shed + network + network.discovery + network.discovery.peers.253bad481e6371866c9f6276b2a7b3a10ca16255668e740e6fc01da1cacc4350 + observability + payment + probe.rate_limit + receipts + security' + if [ "$tables" != "$(echo "$expected" | sed 's/^ *//')" ]; then + echo "rendered table set changed. got:" >&2; echo "$tables" >&2 + echo "expected:" >&2; echo "$expected" | sed 's/^ *//' >&2 + exit 1 + fi + # Key floor stays a count: knobs are added upstream routinely, so an exact + # number would need editing on every sync. 125 is ~10 below today's render. + if [ "$keys" -lt 125 ]; then + echo "only $keys scalar keys rendered (expected >= 125) — the converge" >&2 + echo "has collapsed, so the schema check above proved little." >&2 + exit 1 + fi + echo "breadth OK: $keys keys, $(echo "$tables" | grep -c '') tables" + changed_when: false diff --git a/ansible/molecule/slow-readiness/converge.yml b/ansible/molecule/slow-readiness/converge.yml index bb7466c..6cee1cc 100644 --- a/ansible/molecule/slow-readiness/converge.yml +++ b/ansible/molecule/slow-readiness/converge.yml @@ -18,9 +18,11 @@ decdn_rpc_url: "https://rpc.example.invalid/" decdn_chain_id: 421614 decdn_region: "US" - decdn_payment_channel_address: "0x1111111111111111111111111111111111111111" + decdn_payment_pool_address: "0x1111111111111111111111111111111111111111" decdn_capacity_bond_address: "0x2222222222222222222222222222222222222222" decdn_slash_judge_address: "0x3333333333333333333333333333333333333333" + # REQUIRED upstream (ADR 011/031) — the role asserts it with the other three. + decdn_content_blacklist_address: "0x4444444444444444444444444444444444444444" decdn_cache_origin_kind: "http" decdn_cache_origin_url: "https://origin.example.invalid/" # Tiny probe window (~2s) so the advisory timeout is exercised fast. The stub diff --git a/ansible/molecule/validation/converge.yml b/ansible/molecule/validation/converge.yml index 515ce82..34c9926 100644 --- a/ansible/molecule/validation/converge.yml +++ b/ansible/molecule/validation/converge.yml @@ -10,6 +10,12 @@ # That is what makes this a real regression guard: if a bad value slipped PAST # validation, the role would run on and fail later at a differently-named task (e.g. # the keystore gate) — which is NOT counted, so the final assert catches the miss. +# +# ONE case is deliberately different: the final "config-gate" case has a valid +# config and has to reach the END of the role, because what it proves is that the +# post-template `decdn config validate` gate still fails the deploy. It therefore +# mutates the host, which is why this scenario has a prepare.yml staging wallet +# material. It runs last so it cannot interfere with the assert-only cases above. - name: Converge (expect validation failures) hosts: all become: true @@ -22,24 +28,28 @@ decdn_cli_manual_bin_src: "{{ stub_bin }}" decdn_rpc_url: "https://rpc.example.invalid/" decdn_region: "US" - decdn_payment_channel_address: "0x1111111111111111111111111111111111111111" + decdn_payment_pool_address: "0x1111111111111111111111111111111111111111" decdn_capacity_bond_address: "0x2222222222222222222222222222222222222222" decdn_slash_judge_address: "0x3333333333333333333333333333333333333333" + # REQUIRED upstream (ADR 011/031) — the role asserts it with the other three. + decdn_content_blacklist_address: "0x4444444444444444444444444444444444444444" tasks: - name: Initialize the rejected-by-validation tracker ansible.builtin.set_fact: decdn_rejected: [] # --- Case: cross-field (the regression guard for the effective-value fix) ------ - # A raised pull_ahead with the leech cap left UNSET (resolves to 256 MiB): the - # daemon would reject floor>cap, so the role must too. - - name: "Case cross-field — pull_ahead above the default (unset) leech cap" + # A raised eviction target with high-water left UNSET (resolves to 90). The + # eviction hysteresis gap is STRUCTURAL upstream — target must be <= high-water + # MINUS 5 — so 88 is rejected even though it is plainly less than 90. That + # margin is exactly what a naive `<` comparison would let through. + - name: "Case cross-field — eviction target inside the structural hysteresis gap" block: - - name: Run decdn_node with pull_ahead > default leech cap + - name: Run decdn_node with eviction_target_pct within 5 of the default high-water ansible.builtin.include_role: name: decdn_node vars: - decdn_pull_ahead_bytes: 536870912 + decdn_eviction_target_pct: 88 rescue: - name: Record cross-field rejection (only if a validation assert failed) ansible.builtin.set_fact: @@ -81,7 +91,7 @@ ansible.builtin.include_role: name: decdn_node vars: - decdn_enable_0rtt: "yes" + decdn_relay_foreign_namespaces: "yes" rescue: - name: Record bool rejection (only if a validation assert failed) ansible.builtin.set_fact: @@ -103,6 +113,70 @@ decdn_rejected: "{{ decdn_rejected + ['list'] }}" when: ansible_failed_task.name is match('^Validate optional') + # --- Case: enum (a policy value outside the daemon's closed set) -------------- + - name: "Case enum — unknown eviction policy" + block: + - name: Run decdn_node with an unknown eviction policy + ansible.builtin.include_role: + name: decdn_node + vars: + decdn_eviction_policy: "lfu" + rescue: + - name: Record enum rejection (only if a validation assert failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['enum'] }}" + when: ansible_failed_task.name is match('^Validate optional') + + # --- Case: dict (unknown key in a rate-limit table) --------------------------- + # The template renders these tables from a fixed key list, so an unknown key is + # SILENTLY DROPPED rather than rendered — the limit would stay at its default + # with nothing in the diff to show for it. Only the assert catches this. + - name: "Case dict — unknown key in a rate-limit table" + block: + - name: Run decdn_node with an unknown DHT rate-limit key + ansible.builtin.include_role: + name: decdn_node + vars: + decdn_dht_rate_limit: + per_peer_rate_per_second: 10.0 + rescue: + - name: Record dict rejection (only if a validation assert failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['dict'] }}" + when: ansible_failed_task.name is match('^Validate optional') + + # --- Case: bounded-range (below a HARD floor the daemon errors on) ------------ + # tinylfu sketch_bytes has a 16384 floor, and it bites on the stock lru/always + # policies too because the default serve_economics policy builds the same + # sketch — so a below-floor value is a startup crash-loop on a config that + # otherwise looks fine. + - name: "Case bounded-range — tinylfu sketch below the hard floor" + block: + - name: Run decdn_node with an undersized tinylfu sketch + ansible.builtin.include_role: + name: decdn_node + vars: + decdn_tinylfu_sketch_bytes: 8192 + rescue: + - name: Record bounded-range rejection (only if a validation assert failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['bounded-range'] }}" + when: ansible_failed_task.name is match('^Validate optional') + + # --- Case: float (a non-numeric would `| float` to 0.0, disabling a limit) ----- + - name: "Case float — non-numeric rate limit" + block: + - name: Run decdn_node with a non-numeric float knob + ansible.builtin.include_role: + name: decdn_node + vars: + decdn_security_per_source_rate_per_sec: "fast" + rescue: + - name: Record float rejection (only if a validation assert failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['float'] }}" + when: ansible_failed_task.name is match('^Validate optional') + # --- Case: string (double-quote breaks the rendered TOML string) -------------- - name: "Case string — user_agent containing a double-quote" block: @@ -117,6 +191,63 @@ decdn_rejected: "{{ decdn_rejected + ['string'] }}" when: ansible_failed_task.name is match('^Validate optional') + # --- Case: the config gate itself --------------------------------------------- + # Every case above trips an Ansible assert BEFORE the host is touched. This one + # proves the last line of defence still works: `decdn config validate` runs + # with `failed_when: false` and hands off to a separate `fail` task, so a + # regression in that task's `when:` would silently turn the authoritative gate + # into a no-op and start the daemon on an unvalidated config. The stub honours + # a marker file to force a non-zero validate. + - name: "Case config-gate — a rejected config must fail the deploy" + block: + # The role notifies `Restart decdn-node` when it writes the env file and + # node.toml — both of which happen BEFORE the config gate. In production + # the gate's failure aborts the play, so those handlers never flush. Here + # the rescue below lets the play complete, so they do — and they would + # fail on a unit that the aborted role never got as far as installing. + # Pre-install it so the play can flush them, keeping this case about the + # gate rather than about handler ordering. + # A placeholder, not the role's own template: that template interpolates + # role variables which are not in scope in this play. All the handler + # needs is a unit it can restart. + - name: Pre-install a placeholder unit so the pending handler can flush + ansible.builtin.copy: + dest: /etc/systemd/system/decdn-node.service + mode: "0644" + content: | + [Unit] + Description=placeholder for the molecule validation scenario + [Service] + Type=oneshot + RemainAfterExit=yes + ExecStart=/bin/true + [Install] + WantedBy=multi-user.target + + - name: Reload systemd so the placeholder unit is visible + ansible.builtin.systemd_service: + daemon_reload: true + + - name: Force the stub's config validate to refuse + ansible.builtin.copy: + dest: /etc/decdn/stub-bad-config + content: "molecule\n" + mode: "0644" + + - name: Run decdn_node against a config the binary rejects + ansible.builtin.include_role: + name: decdn_node + rescue: + - name: Record config-gate rejection (only if the gate task failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['config-gate'] }}" + when: ansible_failed_task.name is match('^Report the config-validation failure') + always: + - name: Clear the forced-failure marker + ansible.builtin.file: + path: /etc/decdn/stub-bad-config + state: absent + - name: Confirm every bad value was rejected by the role's own validation ansible.builtin.assert: that: @@ -127,4 +258,6 @@ a missing tag means that bad value slipped past validation (it would reach the daemon and crash-loop it at config load). vars: - _decdn_expected: ["cross-field", "range", "shape", "bool", "list", "string"] + _decdn_expected: + ["cross-field", "range", "bounded-range", "shape", "bool", "enum", "dict", + "float", "list", "string", "config-gate"] diff --git a/ansible/molecule/validation/molecule.yml b/ansible/molecule/validation/molecule.yml index 05aeb0b..2d702e4 100644 --- a/ansible/molecule/validation/molecule.yml +++ b/ansible/molecule/validation/molecule.yml @@ -33,5 +33,6 @@ scenario: test_sequence: - dependency - create + - prepare - converge - destroy diff --git a/ansible/molecule/validation/prepare.yml b/ansible/molecule/validation/prepare.yml new file mode 100644 index 0000000..0f467c6 --- /dev/null +++ b/ansible/molecule/validation/prepare.yml @@ -0,0 +1,44 @@ +--- +# The decdn_node role requires an operator-provisioned eth keystore + password +# file to already exist before it starts the daemon (it never generates them). +# Stage placeholder files here so the role's keystore gate passes in CI. The +# stub daemon ignores them; this scenario does not exercise real wallet logic. +- name: Prepare + hosts: all + become: true + vars: + decdn_home: /var/lib/decdn + decdn_etc: /etc/decdn + tasks: + - name: Ensure data + config directories exist + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: "0755" + loop: + - "{{ decdn_home }}" + - "{{ decdn_etc }}" + + # Stage these at 0644 (deliberately looser than the target). The role locks + # them to 0600; verify.yml asserts 0600, so the assertion actually attributes + # the mode to the role rather than to what prepare pre-set. + - name: Stage a placeholder eth keystore + ansible.builtin.copy: + dest: "{{ decdn_home }}/keystore.json" + content: "{{ '{}' }}\n" + mode: "0644" + + - name: Stage a placeholder keystore password file + ansible.builtin.copy: + dest: "{{ decdn_etc }}/keystore.password" + content: "molecule-placeholder\n" + mode: "0644" + + # The role's pre-start gate also requires the node identity (node.secret), a + # co-equal key-gen output. Stage it at 0644 so verify.yml can attribute the + # 0600 lock-down to the role. + - name: Stage a placeholder node identity + ansible.builtin.copy: + dest: "{{ decdn_home }}/node.secret" + content: "molecule-placeholder-node-secret\n" + mode: "0644" diff --git a/ansible/roles/decdn_node/README.md b/ansible/roles/decdn_node/README.md index 434c556..4d9ce60 100644 --- a/ansible/roles/decdn_node/README.md +++ b/ansible/roles/decdn_node/README.md @@ -1,10 +1,23 @@ # roles/decdn_node Provisions a **public deCDN node** (`decdn-node` daemon) under a hardened systemd -unit. Two install methods (`decdn_node_install_method`): the default `release` -pulls a pinned GitHub Release tarball, and `manual` copies locally-built binaries -from the Ansible control machine — the pre-release path for when no release exists -yet. This is the repo's deployment (`playbooks/site.yml`). +unit. Two install methods (`decdn_node_install_method`): `release` pulls a pinned +GitHub Release tarball and verifies it against the GPG-signed `SHA256SUMS` +manifest, and `manual` copies locally-built binaries from the Ansible control +machine. This is the repo's deployment (`playbooks/site.yml`). + +> **The default is `manual`, because upstream has cut no release yet.** +> `decdn/decdn`'s `release.yml` fires on a `v[0-9]*` tag push and +> `git ls-remote --tags` is empty, so there is nothing for `release` mode to +> download. Build the two binaries from a checkout until that changes. + +**Schema tracking.** This role renders `node.toml` against the config schema of +`decdn/decdn` @ `d306cc5c` (crate version 0.1.1). Upstream marks every config +section `#[serde(deny_unknown_fields)]` and defines **no** serde aliases, so a key +this role emits that your binary does not know is a startup crash-loop, not a +warning. The role runs `decdn config validate` against the installed binary after +templating, so a mismatch fails the deploy with the daemon's own error message +instead. The `molecule/schema` scenario guards the same thing in CI. ## What this role does (and does not) @@ -17,21 +30,32 @@ Per the deCDN node-onboarding ADR (019), a node only serves paid traffic after probe (see [Readiness](#readiness)). - **Operator (manual, NOT automated here):** Phase 1 **key material** (generate the node + eth keys) and Phase 2 **on-chain** (fund + stake the wallet, register the - node). See "register the node" below — there is no turnkey CLI for it yet. + node). These are no longer raw contract calls — upstream shipped a guided + onboarding CLI, `decdn setup`, over the `decdn node bond` / `register` + primitives (`unbond` / `deregister` are the exit path, which setup never calls). + See [On-chain onboarding](#on-chain-onboarding). ## Prerequisites 1. **The binaries.** Pick an install method with `decdn_node_install_method`: - - **`release`** (default) — the role downloads - `decdn-node--.tar.gz` (and the `decdn` CLI) from - `decdn_node_release_base` (default - `https://github.com/decdn/decdn/releases/download`). Set `decdn_node_version` to a real - `v` release, which must be **publicly reachable** from the target host; - override `decdn_node_release_base` to fetch from a mirror. *(No release exists yet — - either cut one with the upstream `release.yml` workflow, or use `manual` below in the - meantime.)* - - **`manual`** — the role copies the two binaries **verbatim** from the paths + - **`release`** — the role downloads `decdn-node--.tar.gz` (and the + `decdn` CLI) from `decdn_node_release_base` (default + `https://github.com/decdn/decdn/releases/download`), then **verifies both**: + it fetches the release's `SHA256SUMS` and `SHA256SUMS.asc`, checks the + detached signature against the maintainer keyring vendored at + `files/decdn-release-KEYS.asc` (a byte copy of `decdn/KEYS`; upstream's rule + is that a good signature from *any* key in that file is authentic), and only + then checks the tarballs against the manifest. `gnupg` is installed on the + target for this. Set `decdn_release_keyring` to pin your own export, or + `decdn_verify_release_signature: false` for an air-gapped mirror that strips + the signature — that takes the tarballs on trust. + + Set `decdn_node_version` to a real `v` release, publicly reachable + from the target host; override `decdn_node_release_base` for a mirror. + **No upstream release exists yet**, so this mode currently has nothing to + fetch — the role's assert says so rather than surfacing a bare 404. + - **`manual`** (current default) — the role copies the two binaries **verbatim** from the paths you give it (`decdn_node_manual_bin_src` + `decdn_cli_manual_bin_src`) on the Ansible control machine; it does *not* consult `decdn_node_target`, so you are responsible for building for the host's architecture (the default target is @@ -64,10 +88,8 @@ Per the deCDN node-onboarding ADR (019), a node only serves paid traffic after ``` This writes `/var/lib/decdn/node.secret` + `/var/lib/decdn/keystore.json`. - **Fund + stake** the printed eth address, then **register** the node on-chain — - the `CapacityBond.bond` / `declareMbps` / `registerNode` transactions of ADR 019 - §2.2–2.3. There is **no turnkey `decdn` subcommand** for registration yet (the - onboarding CLI is listed as *Deferred* in ADR 019); perform the txns out-of-band. + **Fund** the printed eth address, then bond + register it — see + [On-chain onboarding](#on-chain-onboarding). The role **refuses to start** until the keystore, node identity (`node.secret`), and password file all exist — by default it never generates wallet material @@ -82,37 +104,52 @@ Per the deCDN node-onboarding ADR (019), a node only serves paid traffic after `decdn_node_version` (`release` mode only) **or** `decdn_node_manual_bin_src` + `decdn_cli_manual_bin_src` (`manual` mode), `decdn_rpc_url` (sensitive — may embed an API key; goes in the git-ignored `secret.yml`, everything else in the committed -`main.yml`), `decdn_payment_channel_address`, `decdn_capacity_bond_address`, -`decdn_slash_judge_address` (all `0x`+40-hex; SlashJudge non-zero), -`decdn_region` (ISO 3166-1 alpha-2). Contract addresses/chain-id are protocol -facts — source them from the deployment / an ADR, never guess. +`main.yml`), `decdn_region` (ISO 3166-1 alpha-2), and **four** contract addresses +(all `0x`+40-hex, none the zero address): + +| Variable | Contract | Why required | +|---|---|---| +| `decdn_payment_pool_address` | PaymentPool | Settlement. Renamed from `decdn_payment_channel_address` — pairwise channels were replaced by a shared payment pool. | +| `decdn_capacity_bond_address` | CapacityBond | Stake + registration. | +| `decdn_slash_judge_address` | SlashJudge | EIP-712 `verifyingContract` (ADR 014). | +| `decdn_content_blacklist_address` | ContentBlacklist | **Newly required.** Upstream refuses to resolve a config without it: an absent or zero address is a fail-open compliance trap, and serving a blacklisted hash past its window is slashable (ADR 011/031). | + +Contract addresses and chain-id are protocol facts. Copy them from the upstream +deployment manifest `decdn/contracts/deployments/.json` — the same file +`decdn config init --chain arbitrum-sepolia` bakes in — and never guess. Upstream +redeploys wholesale, so re-sync the whole set rather than patching lines. Optional (omitted from `node.toml` unless set): -- `decdn_slash_appeal_address` — SlashAppeal contract (ADR 028); only needed to file - appeals with `decdn appeal slash`, and `decdn_slash_judge_from_block` — SlashJudge - deploy block; bounds the slash-detection watcher's per-restart chain rescan. -- `decdn_origin_assignment_address` + `decdn_publisher_registry_address` — the ADR 022 - chain-backed origin directory that gates DHT prefetch. **Set both or neither** (either - alone fails the deploy-time assert); `decdn_origin_directory_from_block` bounds its - per-restart log replay. -- `decdn_content_blacklist_address` — the ADR 011/031 compliance watcher (evicts - blacklisted blobs in your region scope); `decdn_content_blacklist_from_block` bounds - its per-restart log replay. Unset ⇒ no watcher (serving a blacklisted hash past its - compliance window is then slashable with no local protection). -- `decdn_cache_origin_kind` (`http`|`fs`|`s3`) + that kind's fields — the pull-through - origin the node fetches on a cache miss. **A serving node needs one:** unset ⇒ no - `[cache.origin]` and cache misses fail `NoOrigin` (the node can only serve blobs it - already holds). +- `decdn_slash_appeal_address` — SlashAppeal (ADR 028); only `decdn appeal slash` + uses it. The daemon accepts the key but never resolves or reads it. +- `decdn_origin_assignment_address` and `decdn_publisher_registry_address` — the + ADR 022 chain-backed origin directory. **Independently optional** (the old "both + or neither" rule went away with the whole-directory mirror): the daemon reads + OriginAssignment for cache-miss pull-through fallback, and only validates + PublisherRegistry, which `decdn publish` consumes. +- `decdn_usdc_address` and the `decdn_swap_*` knobs — CLI-only `[blockchain]` keys. + The daemon accepts but never parses them — only `decdn setup` reads them, to + swap USDC into the deCDN TOKEN and bond that (the bond is always TOKEN; native + ETH is only ever gas). They live in `node.toml` because the section is + `deny_unknown_fields` and the CLI shares the file, which also means nothing + downstream catches a malformed value — hence the role's own shape asserts. +- `decdn_cache_origin_kind` (`http`|`fs`|`s3`) + that kind's fields — the + pull-through origin the node fetches on a cache miss. **A serving node needs + one:** unset ⇒ no `[cache.origin]` and cache misses fail `NoOrigin`. For an + ordered fallback chain use `decdn_cache_origins` (a list of the same per-kind + dicts) instead — the two forms are mutually exclusive upstream and the role + fails loud if both are set. +- `decdn_extra_env` — extra `KEY: value` pairs appended to the 0600 + `/etc/decdn/decdn.env`. This is where environment-borne secrets belong; see + [S3 credentials](#s3-credentials). - `decdn_node_generate_keystore` (default `false`) — opt-in turnkey wallet. When `true` the role runs `decdn key-gen` on the host **only if the keystore is absent** (minting a random `0600` password file first, but only when the keystore is *also* absent) — it never overwrites an existing wallet, and it will not mint a password beside a pre-existing keystore (that password couldn't decrypt it; the gate fails loud instead so - you supply the matching one). The generated wallet is still **unfunded + unstaked** - (funding + on-chain staking/registration stay manual — see the eth-wallet step above). - Leave `false` to keep the operator-provisioned posture (the fail-loud gate then requires - you to provision the keystore, `node.secret`, and password yourself). + you supply the matching one). The generated wallet is still **unfunded + unstaked**. + Leave `false` to keep the operator-provisioned posture. - `decdn_readiness_retries` (default `30`) + `decdn_readiness_delay` (default `2`, seconds) — bound the `/metrics` readiness probe window (`retries × delay`, so ~60 s at these defaults). A **timeout** only warns; it never fails the deploy — but a non-200 *answer* @@ -121,38 +158,151 @@ Optional (omitted from `node.toml` unless set): ### Optional tuning knobs These expose daemon config fields that most operators never touch. Each defaults to -`""` (or `[]`), meaning **omit the key and use the daemon's own default** — set one only -to override. Unlike the `*_from_block` knobs above, an explicit `0`/`false` **is** emitted -(`0` is meaningful — e.g. `decdn_gc_interval_sec: 0` disables the GC sweep). A malformed -or out-of-range value fails loud at deploy time (the daemon would otherwise reject it at -load and crash-loop). See `defaults/main.yml` for every knob's upstream default and unit. - -- **Node-to-node pull-through (#29)** — `decdn_node_to_node_pull_through_enabled` (bool) - gates all serve-time origin pull from upstream nodes; the tuning knobs - (`decdn_node_pull_probe_fanout`, `decdn_node_pull_timeout_sec`, `decdn_pull_ahead_bytes`, - `decdn_max_unrecouped_leech_bytes`, `decdn_pull_share_ratio_percent`, - `decdn_pull_through_require_authorized_origin`) only take effect when it is `true`. - `*_bytes` / `*_percent` are **bare integers** (bytes; percent at scale 100, `400` = 4.0×). -- **Settlement / channels** — `decdn_redeem_threshold_micro_usdc`, - `decdn_buyer_deposit_micro_usdc`, `decdn_buyer_max_approve` (bool), and the opt-in - auto-close pair `decdn_settlement_auto_threshold_micro_usdc` / - `decdn_settlement_auto_by_voucher_nonce_span` (`""` disables — a configured `0` is - rejected upstream). All µUSDC. -- **Delivery-rate clamps** — `decdn_delivery_floor` / `decdn_delivery_ceiling` (µUSDC; - floor ≤ ceiling) and `decdn_voucher_interval_mb` (`1..=1024`). -- **Cache tuning** — `decdn_max_probe_holds`, `decdn_stake_lane_reserved_holds`, - `decdn_gc_interval_sec`, `decdn_pinned_hashes` (list of 64-char **lowercase-hex** BLAKE3 - hashes), `decdn_cache_user_agent`. +`""` (or `[]`/`{}`), meaning **omit the key and use the daemon's own default** — set one +only to override; an explicit `0`/`false` **is** emitted (`0` is meaningful — e.g. +`decdn_gc_interval_sec: 0` disables the GC sweep). A malformed or out-of-range value +fails loud at deploy time, and `decdn config validate` catches anything the role's own +asserts miss. See `defaults/main.yml` for every knob's upstream default, unit and range. + +- **Payment / credit** — `decdn_rate_per_mb`, `decdn_delivery_floor`, + `decdn_credit_max` (bare-integer bytes), `decdn_credit_ramp_divisor`, + `decdn_frame_target_bytes` (`1..=1048576`), `decdn_voucher_commit_interval_ms`. + Note `delivery_floor` is a **pre-chain seed only**: the daemon overwrites it from + `PaymentPool.getRateBounds()` at startup, so the on-chain value is authoritative. + There is no companion ceiling knob any more. +- **Settlement / payment pool** — `decdn_redeem_threshold_micro_usdc`, + `decdn_redeem_max_vouchers_per_tx`, `decdn_redeem_interval_secs`, + `decdn_buyer_working_deposit_micro_usdc`, `decdn_buyer_max_approve` (bool), + `decdn_pool_min_remaining_deposit_micro_usdc`, and the per-signer floor pair + `decdn_pool_floor_signer_share_bps` (`1..=10000`) / + `decdn_pool_floor_signer_max_windows`. All µUSDC unless noted. +- **Cache sizing + eviction** — `decdn_cache_size_mb`, `decdn_max_blob_size_mb` + (defaults to `cache_size_mb` upstream; must be `<=` it), `decdn_max_rate_per_mb` + (buyer-side rate ceiling), `decdn_gc_interval_sec`, `decdn_fs_rescan_interval_sec`, + `decdn_max_probe_holds`, `decdn_stake_lane_reserved_holds`, `decdn_pinned_hashes` + (64-char **lowercase-hex** BLAKE3), `decdn_cache_user_agent`, the + `decdn_eviction_*` family, and the `decdn_origin_probe_*` TTLs (which must satisfy + `negative <= fault <= positive`). **The eviction hysteresis gap is structural:** + `decdn_eviction_target_pct` must be at least **5** below + `decdn_eviction_high_water_pct`, not merely below it. +- **Admission / eviction policy (ADR 040)** — `decdn_eviction_policy` + (`lru`|`tinylfu`), `decdn_admission_policy` (`always`|`tinylfu`) and the + `decdn_tinylfu_*` knobs. `decdn_tinylfu_sketch_bytes` has a **hard floor of + 16384**, and the floor applies even on the stock `lru`/`always` policies because + the default serve-economics policy builds the same sketch. +- **Refuse-to-serve economics (ADR 041)** — `decdn_serve_economics_policy` + (`off`|`margin`), `decdn_serve_economics_discount` (a **float** in `(0.0, 1.0]`, + not basis points), `decdn_serve_economics_n_max`, and the warming budget/refill pair. +- **Node-to-node pull-through (#29)** — `decdn_node_to_node_pull_through_enabled` + (bool) gates all serve-time origin pull from upstream nodes; + `decdn_node_pull_probe_fanout`, `decdn_node_pull_timeout_sec`, + `decdn_node_pull_stall_window_sec` and `decdn_node_pull_min_throughput_bps` tune it. + The last two replaced a single stall timeout: a leg is now abandoned when it + delivers under the throughput floor across the window, not merely when it goes + quiet. `decdn_relay_foreign_namespaces` (origin-only node policy) has a + **role-derived** default — `false` when an origin backend is configured, `true` + when none — so leave it unset unless you mean to pin it. +- **Origin retry + circuit breaker** — the `decdn_origin_retry_*` and + `decdn_circuit_breaker_*` families. - **Blockchain watchers** — `decdn_rpc_watchdog_interval_sec` (`0` or `>= 10`), - `decdn_event_poll_interval_ms` (`>= 250`), `decdn_content_blacklist_poll_interval_sec` (`>= 1`). -- **Network** — `decdn_relay_urls` (list; when non-empty the role emits it and omits the - singular `decdn_relay_url`), `decdn_enable_0rtt` (bool). + `decdn_event_poll_interval_ms` (`>= 250`), `decdn_content_blacklist_poll_interval_sec` + (`>= 1`), `decdn_rate_bounds_poll_interval_sec`, `decdn_fee_shares_poll_interval_sec`, + and the `decdn_origin_directory_*` cache knobs. +- **Network** — `decdn_relay_urls` (list; the singular `relay_url` config key no + longer exists). Operator-run address discovery (#818) via + `decdn_discovery_pkarr_url` + `decdn_discovery_dns_origin` (**set together or not + at all**) and/or `decdn_discovery_peers` (a map of 64-char lowercase-hex NodeId to + `{relay_url, addrs}`). Setting either mechanism drops the n0 discovery leg. +- **Abuse limits + load shedding** — the `decdn_security_*` family, + `decdn_load_shed_*` (`policy` is `resource-pressure`|`always-admit`; the low-water + serve mark must be `<=` the high), and the `decdn_dht_rate_limit` / + `decdn_probe_rate_limit` **dicts** (any subset of eight keys; rate keys are floats, + the rest integers — an unknown key is rejected at deploy time, because the template + would otherwise silently drop it). +- **Receipts + local denylist** — `decdn_receipts_max_file_bytes`, + `decdn_receipts_retained_files`, and `decdn_content_denied_hashes` / + `decdn_content_denied_origins` (node-local, independent of the on-chain blacklist). - **Observability** — `decdn_otlp_endpoint` (OTLP span export; needs the node built - `--features otlp`), `decdn_region_accounting_interval_sec`. + `--features otlp`). + +### S3 credentials + +The role deliberately **never** writes static AWS keys into `node.toml`: that file +is `0640` and diffable, and committing credentials to it would break this repo's +no-secrets rule. Only `source = "default-chain"` is templated. Set +`decdn_cache_origin_s3_use_default_chain: true` (plus an optional +`decdn_cache_origin_s3_profile`) and supply the keys through the `0600` env file: + +```yaml +# host_vars//secret.yml (git-ignored) +decdn_extra_env: + AWS_ACCESS_KEY_ID: "AKIA..." + AWS_SECRET_ACCESS_KEY: "..." + AWS_SESSION_TOKEN: "..." # assume-role / SSO only +``` + +`default-chain` is the full AWS credential chain — those environment variables +first, then a `~/.aws/credentials` profile, then an IAM role or instance profile — +so an S3, R2, B2 or MinIO origin works with no secret in a tracked file, and an +instance profile needs no credentials at all, just the toggle. Note the region is +**not** taken from `AWS_REGION`: it comes from `[cache.origin].region` +(`decdn_cache_origin_s3_region`). + +### Config reload + +SIGHUP (and `decdn node reload`, which shares its mutex) re-reads the config file +and applies **five** sections: `observability.log_level`, `cache.pinned_hashes`, +all of `[security]`, all of `[content]`, and all of `[load_shed]`. Everything else +logs "requires restart" and keeps its running value — notably `payment.rate_per_mb`, +which upstream demoted to restart-required. + +The unit exposes `ExecReload`, and the role ships a `Reload decdn-node` handler, +but the `node.toml` task still notifies a **restart**: a template diff carries no +information about which sections changed, so choosing reload is an operator's +deliberate call. + +See `roles/decdn_node/defaults/main.yml` for the full knob list, defaults and units. -Source contract addresses / chain-id from the deCDN contract deployment for your target -chain, or the relevant ADR — never guess. See `roles/decdn_node/defaults/main.yml` for the -full knob list and defaults. +## On-chain onboarding + +The role stops at ADR 019 Phase 1 (host prep) and Phase 3 (startup). Phase 2 — fund, +bond, register — is the operator's, but it is **no longer a set of raw contract +calls**: upstream ships a guided CLI. Every one of these takes `--dry-run`. + +```bash +# Guided path: pre-flight checks (clock skew, gas, balances), key generation, +# bond and registration, ending in a readiness summary. Thin orchestration over +# `key-gen` / `node bond` / `node register` — it submits no transaction they do +# not. (The exit path below is NOT part of setup.) +decdn setup --mbps 100 --region US \ + --multiaddr /ip4//udp/4433/quic-v1 --yes --accept-terms + +# Or drive the primitives directly: +decdn node bond --mbps 100 # CapacityBond.bond + declareMbps (idempotent: + # tops up only the shortfall, so a re-run after a + # partial failure converges rather than over-bonding) +decdn node register --region US \ + --multiaddr /ip4//udp/4433/quic-v1 \ + --accept-terms # CapacityBond.registerNode — builds the EIP-712 + # binding + ed25519 ownership signatures locally. + # --region is REQUIRED (no default). +``` + +Exiting is the reverse, in order: `decdn node deregister` (leaves the active set +and clears the declared tier — the bond stays deposited and **fully slashable**), +then `decdn node unbond --all` (starts the unbonding window; the node is INACTIVE +for its whole duration, and a second call withdraws once it matures). + +`decdn node rotate-key --key iroh` rotates the node's iroh identity. `--key` is +required and has no default; `--key eth` is a different, much heavier operation — +it moves the whole Ethereum identity, which means deregister, a full unbond window, +and re-bonding (resetting `firstBondedAt`). + +**Running these from a playbook:** `decdn setup` and `decdn node register` prompt +for terms acceptance and abort in any non-TTY context, so both need +`--accept-terms`; `setup` additionally needs `--yes` to skip its bond-amount +confirmation. All of them read the same `node.toml` this role renders, so the +contract addresses only have to be right once. ## Network @@ -171,7 +321,7 @@ unit is in the `running` state. The probe is **advisory** on timeout: exhausting the window logs a warning but does **not** fail the deploy. The daemon can bind its loopback metrics/admin listeners well after process start on an otherwise-healthy node — they come up -behind the startup PaymentChannel buyer bootstrap (an upstream startup-ordering +behind the startup PaymentPool buyer bootstrap (an upstream startup-ordering issue tracked in `decdn/decdn`, not here), which can take many minutes. A short probe that hard-failed would therefore false-fail a healthy deploy, and stretching it to cover the worst case would hang every deploy for that whole @@ -195,8 +345,8 @@ journalctl -u decdn-node -e # look for "node runtime ready" ``` Note that neither check proves the node is serving **paid** traffic — that -additionally requires on-chain stake + registration (ADR 019 Phase 2, manual); -verify with `decdn node health`. +additionally requires on-chain stake + registration (see +[On-chain onboarding](#on-chain-onboarding)); verify with `decdn node health`. ## Files on the host @@ -212,17 +362,30 @@ verify with `decdn node health`. ```bash systemctl status decdn-node journalctl -u decdn-node -e -decdn node health # admin RPC (127.0.0.1:9191) -decdn node peers - -# If this node is slashed (surfaced via the admin RPC — admin_v1_slashes), file an -# appeal within the ADR 028 window. Needs decdn_slash_appeal_address set in node.toml -# (else pass --slash-appeal-address / DECDN_SLASH_APPEAL_ADDRESS): +# Admin RPC (127.0.0.1:9191). All of these accept --json and --timeout-ms, and +# honour DECDN_ADMIN_URL; `node top` scrapes /metrics and honours DECDN_METRICS_URL. +decdn node health # identity + process uptime +decdn node status # DHT health: routing-table fill, active stakers, republish depth +decdn node lanes # per-lane accrued claim, voucher age, redemption-threshold state +decdn node slashes # slashes detected against this operator, with appeal deadlines +decdn node top # live activity: streams, cache hit rate, bytes returned +decdn node evict # force a blob out of the local cache (takedown, corruption) +decdn node reload # re-read node.toml, apply the five hot-reloadable sections +decdn node drain --wait # graceful shutdown. Without --wait it is fire-and-forget + # (the response means "asked to stop", not "stopped"); + # --wait polls to completion (--wait-timeout-secs 30). + +# Read-only chain query, not admin RPC: lists active registered nodes and maps +# node-ids/regions to operator addresses. Loads no keystore and spends nothing. +decdn node lookup --region US --probe + +# If this node is slashed (surfaced by `decdn node slashes` / admin_v1_slashes), +# file an appeal within the ADR 028 window. Needs decdn_slash_appeal_address set in +# node.toml (else pass --slash-appeal-address / DECDN_SLASH_APPEAL_ADDRESS): decdn appeal slash ``` -Upgrades (`release` mode): bump `decdn_node_version` (+ `decdn_node_sha256`) and -re-deploy — the version stamp triggers re-install + restart; the persistent +Upgrades (`release` mode): bump `decdn_node_version` and re-deploy — the version stamp triggers re-install + restart; the persistent `node.secret` and `keystore.json` are untouched. Upgrades (`manual` mode): there is **no** version stamp — rebuild the binaries diff --git a/ansible/roles/decdn_node/defaults/main.yml b/ansible/roles/decdn_node/defaults/main.yml index 2dbe92a..97ef62d 100644 --- a/ansible/roles/decdn_node/defaults/main.yml +++ b/ansible/roles/decdn_node/defaults/main.yml @@ -7,104 +7,214 @@ # "release" (default): download the pinned GitHub Release tarballs (a v # release must exist). "manual": copy locally-built binaries from the Ansible # control machine — for pre-release / dev deploys where no release exists yet. -decdn_node_install_method: release # "release" | "manual" +# Upstream has cut NO release tag yet (release.yml fires on a v[0-9]* tag push and +# `git ls-remote --tags decdn/decdn` is empty), so "release" mode currently has +# nothing to fetch. The default is therefore "manual"; flip it to "release" and pin +# decdn_node_version once an upstream release exists. +decdn_node_install_method: manual # "release" | "manual" decdn_node_manual_bin_src: "" # manual: control-machine path to the decdn-node daemon binary decdn_cli_manual_bin_src: "" # manual: control-machine path to the decdn CLI binary # --- Release to install (REQUIRED in "release" mode — a v GitHub Release must exist) --- -decdn_node_version: "" # e.g. "0.1.0" (asserted non-empty in "release" mode) +decdn_node_version: "" # e.g. "0.1.1" (asserted non-empty in "release" mode) decdn_node_target: x86_64-unknown-linux-gnu -decdn_node_sha256: "" # optional "abc123..." to pin the daemon tarball -decdn_cli_sha256: "" # optional, for the decdn CLI tarball decdn_node_release_base: "https://github.com/decdn/decdn/releases/download" +# Release integrity. Upstream publishes a SHA256SUMS manifest covering every +# archive, GPG-signed by a maintainer key from decdn/decdn's root KEYS file +# (SHA256SUMS.asc). The role verifies the signature, then checks the tarballs +# against the manifest — this replaces the old hand-pasted per-tarball sha256 +# pins. Set to false only for an air-gapped mirror that strips the signature. +decdn_verify_release_signature: true +# Keyring used for that check. Defaults to the copy vendored with this role; +# point it at your own export of the upstream KEYS file to pin a narrower set. +decdn_release_keyring: decdn-release-KEYS.asc # --- Identity / on-chain ----------------------------------------------------- -# rpc_url, the three contract addresses and region are REQUIRED with no usable +# rpc_url, the FOUR contract addresses and region are REQUIRED with no usable # default (empty → the asserts in tasks/main.yml fail loud). Supply via host_vars. # chain_id is the one exception: it defaults to the upstream binary's own default # (Arbitrum Sepolia) rather than being a deployment-specific fact to source. decdn_rpc_url: "" # SENSITIVE; may embed an API key -decdn_payment_channel_address: "" +decdn_payment_pool_address: "" decdn_capacity_bond_address: "" decdn_slash_judge_address: "" +# REQUIRED (ADR 011/031). ContentBlacklist watcher — evicts blacklisted blobs in +# your region scope. Upstream now refuses to resolve a config without it: an +# absent or zero address is a fail-open compliance trap, and serving a +# blacklisted hash past its compliance window is slashable. +decdn_content_blacklist_address: "" # Optional (ADR 028). SlashAppeal contract — only needed to file appeals with # `decdn appeal slash`; the daemon ignores it. Empty => key omitted from node.toml. -decdn_slash_appeal_address: "" -# Optional. SlashJudge deploy block: the slash-detection watcher rescans from -# here on EVERY daemon start, so 0 (the upstream default) re-scans the full chain -# each restart — RPC-heavy on an established L2. Set to bound restart cost. -decdn_slash_judge_from_block: 0 -# Optional (ADR 022). Chain-backed origin directory that gates DHT prefetch. Set -# BOTH or NEITHER — either alone fails the deploy-time assert. Empty => both keys -# omitted and the directory is deny-all (prefetch finds no authorized origins). +decdn_slash_appeal_address: "" # accepted but never read by the daemon; `decdn appeal slash` uses it +# Optional (ADR 022). Chain-backed origin directory. These are INDEPENDENTLY +# optional upstream — the old "both or neither" rule is gone. The daemon reads +# origin_assignment_address for cache-miss pull-through fallback; it only +# validates publisher_registry_address, which `decdn publish` consumes. +# Empty origin_assignment_address => the directory is empty (no authorized origins). decdn_origin_assignment_address: "" decdn_publisher_registry_address: "" -# Optional. PublisherRegistry deploy block: bounds the origin-directory log replay -# on restart (absent/0 => scans the whole chain — RPC-heavy on an established L2). -# Only emitted when the pair above is set AND this is > 0. -decdn_origin_directory_from_block: 0 -# Optional (ADR 011/031). ContentBlacklist watcher — evicts blacklisted blobs in -# your region scope. Empty => no watcher (serving a blacklisted hash past its -# compliance window is then slashable with no local protection). -decdn_content_blacklist_address: "" -# Optional. ContentBlacklist deploy block; bounds the blacklist log replay on -# restart. Only emitted when decdn_content_blacklist_address is set AND this is > 0. -decdn_content_blacklist_from_block: 0 +# Origin-directory lookup cache ("" => daemon default). +decdn_origin_directory_positive_ttl_sec: "" # daemon dflt 300 +decdn_origin_directory_negative_ttl_sec: "" # daemon dflt 30 +decdn_origin_directory_cache_capacity: "" # daemon dflt 4096 decdn_region: "" # ISO 3166-1 alpha-2 decdn_chain_id: 421614 # Arbitrum Sepolia (matches decdn-node's --chain-id default) # --- Network ------------------------------------------------------------------ decdn_bind_port: 4433 # public QUIC (udp); opened in the baseline firewall -decdn_relay_url: "" # optional iroh relay for NAT traversal (deprecated single alias) -# iroh relay URLs; [] => n0 default relays. When this list is non-empty the template -# emits it and omits decdn_relay_url (this list supersedes the singular alias). +# iroh relay URLs; [] => n0 default relays. Reachability is probed at bring-up and +# logged but never fatal. The [network] table no longer has a singular `relay_url` +# scalar — only this list. (The --relay-url CLI flag / DECDN_RELAY_URL env var do +# still exist, and `relay_url` is still a key INSIDE each discovery peer below.) # e.g. ["https://relay1.example:443", "https://relay2.example:443"] decdn_relay_urls: [] # The optional knobs below default to "" (or []) => "omit the key, use the daemon's # own default". Set a value only to override; an explicit 0/false IS emitted. -decdn_enable_0rtt: "" # bool; daemon dflt true — QUIC 0-RTT for cdn/probe/v1 (ADR 015) +# +# Operator-run address discovery. Absent => the n0-hosted pkarr/DNS discovery. +# pkarr_url (publish this node's record) requires dns_origin (resolve peers); +# dns_origin alone is legal — a resolve-only node that does not publish. Static +# peers are an independent, combinable mechanism: a map of 64-char lowercase-hex +# NodeId => {relay_url, addrs}. Setting either mechanism drops the n0 leg. +decdn_discovery_pkarr_url: "" +decdn_discovery_dns_origin: "" +decdn_discovery_peers: {} # --- Economics ---------------------------------------------------------------- decdn_rate_per_mb: 10 # USDC base units (6 decimals) -# Delivery-rate clamps + voucher cadence (µUSDC; "" => daemon default). -decdn_delivery_floor: "" # daemon dflt 0 — rate clamp floor before signing ProbeResponse -decdn_delivery_ceiling: "" # daemon dflt 1_000_000_000_000 (protocol MAX) — rate clamp ceiling -decdn_voucher_interval_mb: "" # daemon dflt 1; range 1..=1024 — voucher cadence (ADR 003) +# PRE-CHAIN SEED ONLY: the daemon reads PaymentPool.getRateBounds() at startup and +# overwrites this before it serves anything, then tracks RateBoundsUpdated. It only +# governs the window before that read completes (a failed read refuses startup), so +# the on-chain value is authoritative. There is no companion ceiling knob any more — +# `delivery_ceiling` was removed and DECDN_DELIVERY_CEILING is a retired env var. +decdn_delivery_floor: "" # daemon dflt 0 +# Per-stream credit window (ADR 003 credit slow-start). credit_max is a bare +# integer of bytes on the wire; ramp_divisor 0 opens the full ceiling immediately. +decdn_credit_max: "" # daemon dflt 67108864 (64 MiB) +decdn_credit_ramp_divisor: "" # daemon dflt 2 +# Sender-chosen ChunkData frame size (#1804). Range 1..=1048576. +decdn_frame_target_bytes: "" # daemon dflt 1048576 (1 MiB) +decdn_voucher_commit_interval_ms: "" # daemon dflt 5000; must be > 0 -# --- Settlement / channels (all µUSDC; "" => daemon default) ------------------ -decdn_redeem_threshold_micro_usdc: "" # daemon dflt 1_000_000 (1 USDC) — accrued µUSDC to trigger withdraw -decdn_buyer_deposit_micro_usdc: "" # daemon dflt 10_000_000 (10 USDC) — escrow on a buyer-channel open -decdn_buyer_max_approve: "" # bool; daemon dflt true — one-time max USDC approval to PaymentChannel -# Auto-closeChannel triggers (OR'd). "" => off (disabled upstream); a configured 0 is rejected. -decdn_settlement_auto_threshold_micro_usdc: "" # un-redeemed µUSDC threshold -decdn_settlement_auto_by_voucher_nonce_span: "" # nonce-span companion trigger +# --- Settlement / payment pool (all µUSDC unless noted; "" => daemon default) --- +# ADR 003 shared payment pool: capped delegated signers + node-addressed cumulative +# vouchers. The old pairwise-channel knobs (auto-closeChannel triggers, the split +# initial/working buyer deposits, voucher_interval_mb negotiation) no longer exist. +decdn_redeem_threshold_micro_usdc: "" # daemon dflt 1_000_000 (1 USDC) — accrued µUSDC to trigger redeem +decdn_redeem_max_vouchers_per_tx: "" # daemon dflt 300; > 0 — gas-bounded redeemMany chunking +decdn_redeem_interval_secs: "" # daemon dflt 300; range 1..=21600 — redeemer self-tick +decdn_buyer_working_deposit_micro_usdc: "" # daemon dflt 10_000_000 (10 USDC); > 0 +# bool; profile-dependent upstream — true for the daemon, false for the `decdn` +# client (fetch / bundle pull). node.toml is deliberately shared with the CLI, so +# pinning it here changes client behaviour too. One-time max USDC approval to PaymentPool. +decdn_buyer_max_approve: "" +decdn_pool_min_remaining_deposit_micro_usdc: "" # daemon dflt 1_000_000 — solvency floor before a pool drains +# Per-signer floor isolation under the per-pool solvency ceiling (#1841). +decdn_pool_floor_signer_share_bps: "" # daemon dflt 2500; range 1..=10000 +decdn_pool_floor_signer_max_windows: "" # daemon dflt 0 (off) # --- Blockchain watcher tuning ("" => daemon default) ------------------------- decdn_rpc_watchdog_interval_sec: "" # daemon dflt 30; 0 disables, else >= 10 — RPC watchdog -decdn_event_poll_interval_ms: "" # daemon dflt 7000; >= 250 — eth_getFilterChanges poll +decdn_event_poll_interval_ms: "" # daemon dflt 7000; >= 250 — multiplexed getLogs poll decdn_content_blacklist_poll_interval_sec: "" # daemon dflt 600; >= 1 — blacklist re-scope cadence +decdn_rate_bounds_poll_interval_sec: "" # daemon dflt 3600; > 0 — PaymentPool.getRateBounds refresh +decdn_fee_shares_poll_interval_sec: "" # daemon dflt 3600; > 0 — FeeRouter share refresh + +# --- CLI-only [blockchain] keys ---------------------------------------------- +# The daemon ACCEPTS these (they are declared so a config that also drives the CLI +# still passes [blockchain]'s deny_unknown_fields) but never parses or reads them — +# only `decdn setup` does. So nothing downstream catches a malformed value, which +# is exactly why the role asserts their shape itself. +decdn_usdc_address: "" +decdn_swap_venue: "" # "uniswap-v3" | "balancer-v3" +decdn_swap_router_address: "" +decdn_swap_quoter_address: "" +decdn_swap_fee_tier: "" # e.g. 3000 +decdn_swap_balancer_pool: "" +decdn_swap_pool_address: "" # --- Cache -------------------------------------------------------------------- decdn_cache_size_mb: 10240 # 10 GB +# NOTE: this is a deliberate OVERRIDE, not the daemon default. Upstream defaults +# max_blob_size_mb to cache_size_mb; 1 GB caps any single blob at a tenth of the +# cache so one large object cannot evict everything else. Must be <= cache_size_mb. +# DO NOT set 0: despite what upstream's own `config init` template says, there is +# no zero special-case in the cache engine — the admit gate is a plain +# `advertised > max_blob_bytes`, so 0 rejects every blob with a size hint. decdn_max_blob_size_mb: 1024 # 1 GB +# Buyer-side ABSOLUTE per-MB rate ceiling for paid pulls (#1375) — what this node +# will accept a provider to quote on a cache-miss pull, on top of the always-applied +# probe-relative bound. Distinct from payment.delivery_floor, which raises this +# node's own SELLER quote. "" / 0 => unlimited. +decdn_max_rate_per_mb: "" # Cache tuning ("" / [] => omit + use the daemon default; an explicit 0 IS emitted — # 0 is meaningful here, e.g. gc_interval_sec = 0 disables the sweep and # max_unrecouped_leech_bytes = 0 turns the cap off). -decdn_max_probe_holds: "" # daemon dflt 256 — eviction-exempt holds (ADR 005); 0 = has_blob answers false +# daemon dflt 256 — eviction-exempt holds (ADR 005). 0 => has_blob answers false for +# STORE-BACKED content only; origin-servable content takes no hold and is still advertised. +decdn_max_probe_holds: "" decdn_stake_lane_reserved_holds: "" # daemon dflt 0 — holds reserved for node-to-node probes (#757) decdn_gc_interval_sec: "" # daemon dflt 300 — iroh-blobs GC sweep; 0 = off -decdn_pinned_hashes: [] # eviction-exempt pins; each a 64-char lowercase-hex BLAKE3 hash +decdn_fs_rescan_interval_sec: "" # daemon dflt 60 — fs-origin rescan; 0 = off +# eviction-exempt pins; each a 64-char BLAKE3 hash. Upstream accepts mixed case; +# this role requires lowercase, matching the canonical output form. +decdn_pinned_hashes: [] decdn_cache_user_agent: "" # daemon dflt decdn-node/ — HTTP origin User-Agent +# Origin-probe memo TTLs. Upstream requires negative <= fault <= positive. +decdn_origin_probe_ttl_sec: "" # daemon dflt 15 (positive) +decdn_origin_probe_negative_ttl_sec: "" # daemon dflt 2 +decdn_origin_probe_fault_ttl_sec: "" # daemon dflt 5 +decdn_origin_probe_timeout_ms: "" # daemon dflt 2000 +decdn_origin_probe_memo_capacity: "" # daemon dflt 4096 +# Eviction driver (#1173). The hysteresis gap is STRUCTURAL: target_pct must be +# <= high_water_pct - 5, not merely less than it. +decdn_eviction_high_water_pct: "" # daemon dflt 90; range 60..=95 +decdn_eviction_target_pct: "" # daemon dflt 80; range 40..=90 +decdn_eviction_per_sweep_budget: "" # daemon dflt 16; range 1..=256 +decdn_eviction_tick_secs: "" # daemon dflt 1; range 1..=60 +# Pluggable admission & eviction policies (ADR 040). +decdn_eviction_policy: "" # daemon dflt "lru"; "lru" | "tinylfu" +decdn_admission_policy: "" # daemon dflt "always"; "always" | "tinylfu" +# TinyLFU sketch. sketch_bytes has a HARD FLOOR of 16384 — and the floor bites even +# on the stock lru/always policies, because the default serve_economics policy +# ("margin") builds the same sketch (#1803 review). +decdn_tinylfu_sketch_bytes: "" # daemon dflt 262144; >= 16384 +decdn_tinylfu_promotion_threshold: "" # daemon dflt 2 +decdn_tinylfu_probation_target_pct: "" # daemon dflt 10; range 1..=50 +decdn_tinylfu_aging_halflife_sec: "" # daemon dflt 600 (resolved but not yet consulted) +# Refuse-to-serve economics (ADR 041) — when serving a miss is worth the egress. +decdn_serve_economics_policy: "" # daemon dflt "margin"; "off" | "margin" +decdn_serve_economics_discount: "" # daemon dflt 0.5 — a FLOAT in (0.0, 1.0], not bps +decdn_serve_economics_n_max: "" # daemon dflt 64 +decdn_serve_economics_warming_budget: "" # daemon dflt 5000000 +decdn_serve_economics_warming_refill: "" # daemon dflt 58 +# Origin fetch retry + circuit breaker ("" => daemon default). +decdn_origin_retry_max_retries: "" # daemon dflt 3 +decdn_origin_retry_initial_backoff_ms: "" # daemon dflt 100 +decdn_origin_retry_max_backoff_ms: "" # daemon dflt 10000 +decdn_origin_retry_jitter_ratio: "" # daemon dflt 0.1 (float) +decdn_origin_retry_buffered_max_bytes: "" # daemon dflt 4194304 (4 MiB); max 67108864 +decdn_circuit_breaker_enabled: "" # bool; daemon dflt true +decdn_circuit_breaker_failure_threshold: "" # daemon dflt 5 +decdn_circuit_breaker_cooldown_ms: "" # daemon dflt 30000 +decdn_circuit_breaker_half_open_max_calls: "" # daemon dflt 1 # Node-to-node pull-through (#29). The toggle gates all serve-time origin pull from # upstream nodes; the knobs below only take effect when it is true. *_bytes / *_percent # are BARE INTEGERS on the wire (bytes; percent at scale 100, so 400 = 4.0x). decdn_node_to_node_pull_through_enabled: "" # bool; daemon dflt false — enable pull from upstream nodes (#29) -decdn_node_pull_probe_fanout: "" # daemon dflt 5 — providers probed per miss; 0 disables pull +# daemon dflt 5 — providers probed per miss. 0 disables pull in practice (the +# candidate list is `.take(fanout)`), but that is a consequence, not a validated +# upstream contract — do not rely on it as an off switch. +decdn_node_pull_probe_fanout: "" decdn_node_pull_timeout_sec: "" # daemon dflt 20 — per-upstream pull timeout -decdn_pull_ahead_bytes: "" # daemon dflt 1048576 (1 MiB) — speculative pull window (ADR 037) -decdn_max_unrecouped_leech_bytes: "" # daemon dflt 268435456 (256 MiB) — speculative-spend cap; 0 = off -decdn_pull_share_ratio_percent: "" # daemon dflt 400 (=4.0x) — per-peer speculative ceiling -decdn_pull_through_require_authorized_origin: "" # bool; daemon dflt false — gate pulls on the ADR 022 directory +# Byte-progress stall detection (#1797) — replaced the old single stall timeout. +# A leg is abandoned when it delivers under min_throughput_bps across the window. +decdn_node_pull_stall_window_sec: "" # daemon dflt 20 +decdn_node_pull_min_throughput_bps: "" # daemon dflt 4096 +# Origin-only node policy (#1759): whether to relay namespaces this node is not an +# authorized origin for. Daemon default is ROLE-DERIVED — false when an origin +# backend is configured, true when none — so leave "" unless you mean to pin it. +decdn_relay_foreign_namespaces: "" # bool # Pull-through origin (#437): what the node fetches on a cache miss. Empty kind => # the [cache.origin] table is omitted and misses fail NoOrigin — a serving node # needs an origin. Pick ONE kind and set that kind's fields; the others are ignored. @@ -117,6 +227,17 @@ decdn_cache_origin_s3_region: "" # s3: AWS region (used for SigV4 even wi decdn_cache_origin_s3_endpoint_url: "" # s3, optional: custom endpoint for R2/B2/MinIO (omit for AWS) decdn_cache_origin_s3_path_style: false # s3, optional: path-style addressing (MinIO & many self-hosted) decdn_cache_origin_s3_prefix: "" # s3, optional: key prefix prepended to every object +# s3, optional: emit [cache.origin.credentials] source = "default-chain". Static +# access keys are deliberately NOT templated — node.toml is 0640 and diffable. +# Supply them as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN via +# decdn_extra_env instead; the default chain reads exactly those. +decdn_cache_origin_s3_use_default_chain: false +decdn_cache_origin_s3_profile: "" # s3, optional: ~/.aws/credentials profile override +# Ordered multi-origin fallback (#284). A list of dicts using the same per-kind keys +# as the scalars above (kind/url/decompress, kind/path, kind/bucket/region/...). +# Mutually exclusive with decdn_cache_origin_kind: when this is non-empty the role +# emits [[cache.origins]] and ignores the scalar form. Empty list => scalar form. +decdn_cache_origins: [] # --- Observability (loopback only) -------------------------------------------- decdn_log_level: info @@ -126,14 +247,41 @@ decdn_metrics_bind: "127.0.0.1" # keep loopback — Prometheus scrape is a fol decdn_admin_port: 9191 # Optional ("" => omit + use the daemon default). decdn_otlp_endpoint: "" # OTLP span export; http(s)://; needs the node built --features otlp -decdn_region_accounting_interval_sec: "" # daemon dflt 3600 — per-region bandwidth log; 0 = off + +# --- Abuse limits, load shedding, receipts, local denylist -------------------- +# [security] and [load_shed] and [content] are HOT-RELOADABLE: change them and run +# `decdn node reload` (or systemctl reload decdn-node) instead of restarting. +decdn_security_max_concurrent_handlers: "" # daemon dflt 256; 0 disables the cap +decdn_security_per_source_rate_per_sec: "" # daemon dflt 100.0 (float); 0.0 disables +decdn_security_per_source_burst: "" # daemon dflt 200; > 0 when the rate is > 0 +decdn_security_max_tracked_sources: "" # daemon dflt 4096; 0 = unbounded +# Load shedding under resource pressure (#1756). +decdn_load_shed_policy: "" # daemon dflt "resource-pressure"; also "always-admit" +decdn_load_shed_egress_budget_mbps: "" # daemon dflt 0 (no budget) +decdn_load_shed_max_concurrent_serves_high: "" # daemon dflt 256 +decdn_load_shed_max_concurrent_serves_low: "" # daemon dflt 192; must be <= the high mark +decdn_load_shed_per_client_serve_cap: "" # daemon dflt 32 +# DHT / probe rate limits. Dicts so the two identically-shaped tables share one +# schema; any subset of the eight keys may be set. Rate keys are floats, the rest +# integers: per_peer_rate_per_sec, per_ip_rate_per_sec, global_rate_per_sec, +# per_peer_burst, per_ip_burst, global_burst, max_tracked_per_ip, max_tracked_per_peer. +# Daemon defaults — dht: 20.0/40, 100.0/200, 1000.0/2000, 4096, 4096 +# probe: 5.0/5, 50.0/200, 1000.0/2000, 4096, 4096 +decdn_dht_rate_limit: {} +decdn_probe_rate_limit: {} +# Download-receipt log at /download_receipts.jsonl (path not configurable). +decdn_receipts_max_file_bytes: "" # daemon dflt 134217728 (128 MiB); floor 1 MiB, ceiling 1 GiB +decdn_receipts_retained_files: "" # daemon dflt 4; cap 100 +# Node-local denylist, independent of the on-chain ContentBlacklist. Hot-reloadable. +decdn_content_denied_hashes: [] # 64-char lowercase-hex BLAKE3 hashes +decdn_content_denied_origins: [] # 0x publisher addresses (checksum-agnostic; zero address rejected) # --- Readiness probe (advisory) ----------------------------------------------- # After start, the role probes http://127.0.0.1:/metrics as a # best-effort readiness signal. A probe TIMEOUT only WARNS (it does NOT fail the # deploy): the daemon can bind its metrics/admin listeners well after process # start on an otherwise-healthy node (they come up behind the startup -# PaymentChannel buyer bootstrap — an upstream startup-ordering issue tracked in +# PaymentPool buyer bootstrap — an upstream startup-ordering issue tracked in # decdn/decdn, not here), so a bounded probe would otherwise false-fail a healthy # deploy. A metrics endpoint that ANSWERS with a non-200 error, however, fails # loud — that's a fault, not slow startup. The systemd service-state assert @@ -152,6 +300,12 @@ decdn_home: /var/lib/decdn # data dir: node.secret + keystore.json decdn_etc: /etc/decdn decdn_config_file: /etc/decdn/node.toml decdn_env_file: /etc/decdn/decdn.env # DECDN_RPC_URL (sensitive) — 0600 +# Extra KEY: value pairs appended to the 0600 env file. This is the supported home +# for secrets the daemon reads from the environment rather than node.toml — most +# notably AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN backing an +# S3 cache origin with source = "default-chain". Keep it out of committed vars: +# set it in the git-ignored secret.yml alongside decdn_rpc_url. +decdn_extra_env: {} # Eth keystore lives in the data dir (matches `decdn key-gen --output-dir`). decdn_keystore_file: /var/lib/decdn/keystore.json # operator-provisioned (0600) decdn_keystore_password_file: /etc/decdn/keystore.password # operator-provisioned (0600) diff --git a/ansible/roles/decdn_node/files/decdn-release-KEYS.asc b/ansible/roles/decdn_node/files/decdn-release-KEYS.asc new file mode 100644 index 0000000..2ccdf53 --- /dev/null +++ b/ansible/roles/decdn_node/files/decdn-release-KEYS.asc @@ -0,0 +1,23 @@ +Public keys trusted to sign deCDN releases. Import with `gpg --import KEYS`. + +A release tag, the SHA256SUMS manifest, the SBOM and image-digest.txt are each +signed by one maintainer's personal key. A good signature from ANY key in this +file is authentic — see SECURITY.md. Adding or removing a maintainer is a +change to this file. + + Ant Somers + DA75 1570 6F18 73D2 74D8 A369 9E11 A9FF D62D AADB + +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mDMEanigZRYJKwYBBAHaRw8BAQdAXB6xg4XVGmve59MN5EUHPlEVhZuSO2V0TYQa +j0l7ASu0GkFudCBTb21lcnMgPGFudEBkZWNkbi5vcmc+iJkEExYKAEEWIQTadRVw +bxhz0nTYo2meEan/1i2q2wUCanigZQIbAwUJBaOagAULCQgHAgIiAgYVCgkICwIE +FgIDAQIeBwIXgAAKCRCeEan/1i2q2z4VAP9R8ha+nmRqgWnZh7N8VInlO71pXbeW +8GIYaZxp1FdgBQD+L+U2XomAtXAK7Yh52DOUfhQhq2ME2AXutkGeA2UEoQm4OARq +eKBlEgorBgEEAZdVAQUBAQdA0M8mYVvDIMRDYQHurFQdYhmCKjjxuLFNyyEtb82H +/VEDAQgHiHgEGBYKACAWIQTadRVwbxhz0nTYo2meEan/1i2q2wUCanigZQIbDAAK +CRCeEan/1i2q2yLLAQC/TKs0jU6nQu7oot+sAJI754t0fktoAks1dfb8k8rKOgEA +2JAAMJACUKFQArfPQrekJvVL2ZMINf8AdU7FXPt2Jwc= +=ulJ+ +-----END PGP PUBLIC KEY BLOCK----- diff --git a/ansible/roles/decdn_node/handlers/main.yml b/ansible/roles/decdn_node/handlers/main.yml index 9944251..fd30903 100644 --- a/ansible/roles/decdn_node/handlers/main.yml +++ b/ansible/roles/decdn_node/handlers/main.yml @@ -7,3 +7,13 @@ ansible.builtin.systemd: name: decdn-node state: restarted + +# Applies the five hot-reloadable config sections without dropping connections. +# Nothing in this role notifies it — the node.toml handler restarts, because a +# template diff carries no information about which sections changed. It exists so +# an operator (or a playbook that knows it only touched [content] or [load_shed]) +# can `- meta: flush_handlers` a reload instead of a restart. +- name: Reload decdn-node + ansible.builtin.systemd_service: + name: decdn-node + state: reloaded diff --git a/ansible/roles/decdn_node/tasks/install.yml b/ansible/roles/decdn_node/tasks/install.yml index b2bb6a3..2802d14 100644 --- a/ansible/roles/decdn_node/tasks/install.yml +++ b/ansible/roles/decdn_node/tasks/install.yml @@ -1,11 +1,12 @@ --- # Install the decdn-node daemon + decdn CLI. Two methods, selected by # decdn_node_install_method: -# "release" (default): download the pinned GitHub Release tarballs. Idempotent -# and upgrade-aware via a version stamp; re-extracts only on change. NOTE: -# requires a published v release (see role README). -# "manual": copy locally-built binaries from the Ansible control machine (for -# pre-release / dev deploys). copy's checksum idempotency replaces the stamp. +# "release": download the pinned GitHub Release tarballs, verify them against the +# GPG-signed SHA256SUMS manifest upstream publishes, then extract. Idempotent +# and upgrade-aware via a version stamp; re-extracts only on change. NOTE: as +# of decdn/decdn @ d306cc5c no release tag exists yet — see tasks/main.yml. +# "manual" (default today): copy locally-built binaries from the Ansible control +# machine. copy's checksum idempotency replaces the stamp. # A --version backstop runs in both modes. - name: Install from pinned GitHub Release tarballs @@ -13,9 +14,11 @@ block: - name: Build the release tarball list ansible.builtin.set_fact: + # Each archive holds exactly ONE flat binary (no directory prefix) — the + # unarchive below extracts straight into /usr/local/bin on that contract. decdn_release_tarballs: - - {archive: decdn-node, sha: "{{ decdn_node_sha256 }}"} - - {archive: decdn, sha: "{{ decdn_cli_sha256 }}"} + - decdn-node + - decdn - name: Read the installed-version stamp ansible.builtin.slurp: @@ -30,31 +33,186 @@ {{ (decdn_stamp.content is not defined) or ((decdn_stamp.content | b64decode | trim) != decdn_node_version) }} + # `ansible.builtin.tempfile` does not declare supports_check_mode, so under + # `make check` it is skipped and registers WITHOUT a `.path`; the tasks below + # (file/get_url do support check mode) would then fail on an undefined + # attribute rather than reporting anything useful. Nothing here is inspectable + # in check mode anyway — it downloads and verifies — so skip it as a unit. + - name: Report that the release download is not simulated in check mode + ansible.builtin.debug: + msg: >- + check mode: skipping the download + signature verification of + decdn {{ decdn_node_version }}. The binaries on the target are NOT + verified by this run; the config and unit below still diff normally. + when: + - decdn_need_install + - ansible_check_mode + - name: Install decdn-node + decdn CLI {{ decdn_node_version }} - when: decdn_need_install + when: + - decdn_need_install + - not ansible_check_mode block: + # A private staging directory, not fixed /tmp/.tar.gz paths. These + # bytes are about to be trusted on the strength of a signature, and a + # predictable name in a world-writable /tmp is pre-seedable by any local + # user between the download and the checksum check. + - name: Create a private download staging directory + ansible.builtin.tempfile: + state: directory + prefix: decdn-install- + register: decdn_stage + changed_when: false + + - name: Restrict the staging directory to root + ansible.builtin.file: + path: "{{ decdn_stage.path }}" + state: directory + owner: root + group: root + mode: "0700" + + - name: WARN that release signature verification is DISABLED + ansible.builtin.debug: + msg: |- + decdn_verify_release_signature is false. SHA256SUMS is being fetched + UNAUTHENTICATED from {{ decdn_node_release_base }}, so the checksum + check below proves only that the download was not corrupted — NOT + that these binaries came from deCDN. Anyone able to serve you the + tarball can serve a matching manifest. Re-enable it before this host + serves paid traffic. + when: not decdn_verify_release_signature + + - name: Ensure gnupg is available for signature verification + ansible.builtin.package: + name: gnupg + state: present + when: decdn_verify_release_signature + - name: Download release tarballs ansible.builtin.get_url: url: "{{ decdn_node_release_base }}/v{{ decdn_node_version }}/\ - {{ item.archive }}-{{ decdn_node_version }}-{{ decdn_node_target }}.tar.gz" - dest: "/tmp/{{ item.archive }}-{{ decdn_node_version }}.tar.gz" - mode: "0644" - checksum: "{{ ('sha256:' + item.sha) if (item.sha | length > 0) else omit }}" + {{ item }}-{{ decdn_node_version }}-{{ decdn_node_target }}.tar.gz" + dest: "{{ decdn_stage.path }}/{{ item }}-{{ decdn_node_version }}-{{ decdn_node_target }}.tar.gz" + owner: root + group: root + mode: "0600" loop: "{{ decdn_release_tarballs }}" - loop_control: - label: "{{ item.archive }}" + + # One manifest covers every archive in the release, so it replaces the old + # per-tarball hand-pasted sha256 pins. + - name: Download the SHA256SUMS manifest + ansible.builtin.get_url: + url: "{{ decdn_node_release_base }}/v{{ decdn_node_version }}/SHA256SUMS" + dest: "{{ decdn_stage.path }}/SHA256SUMS" + owner: root + group: root + mode: "0600" + + - name: Download the SHA256SUMS signature + ansible.builtin.get_url: + url: "{{ decdn_node_release_base }}/v{{ decdn_node_version }}/SHA256SUMS.asc" + dest: "{{ decdn_stage.path }}/SHA256SUMS.asc" + owner: root + group: root + mode: "0600" + when: decdn_verify_release_signature + + - name: Create the keyring staging directory + ansible.builtin.file: + path: "{{ decdn_stage.path }}/keyring" + state: directory + owner: root + group: root + mode: "0700" + when: decdn_verify_release_signature + + # Deliberately NOT beside the archives: `sha256sum --check` runs in the + # archive directory, and upstream publishes its KEYS file as a release + # asset — a same-named manifest entry would checksum this vendored copy + # against theirs and fail for a reason unrelated to the binaries. + - name: Stage the release signing keyring + ansible.builtin.copy: + src: "{{ decdn_release_keyring }}" + dest: "{{ decdn_stage.path }}/keyring/KEYS.asc" + owner: root + group: root + mode: "0600" + when: decdn_verify_release_signature + + # An isolated GNUPGHOME inside the staging dir, never root's own keyring: + # verification must answer "is this signed by a key from the shipped KEYS + # file", not "by any key this host happens to trust". Upstream's contract is + # that a good signature from ANY key in KEYS is authentic. + # Status-parsed, not exit-code-parsed. `gpg --verify` exits 0 for a good + # signature made by an EXPIRED or REVOKED key — it only warns on stderr, + # which `changed_when: false` then hides from the operator entirely. The + # two cases that actually matter operationally (the release key was + # revoked because a maintainer was compromised, or it quietly aged out — + # the vendored key expires 2029-08-08) would both install the tarball. + # VALIDSIG is emitted only for a good signature from a still-valid key. + - name: Verify the SHA256SUMS signature against the release keyring + ansible.builtin.shell: + chdir: "{{ decdn_stage.path }}" + # /bin/sh is dash on Debian and has no `pipefail`; this needs bash. + executable: /bin/bash + cmd: | + set -euo pipefail + export GNUPGHOME="{{ decdn_stage.path }}/gnupg" + mkdir -p "$GNUPGHOME" + chmod 700 "$GNUPGHOME" + gpg --batch --quiet --import keyring/KEYS.asc + gpg --batch --status-fd 3 --verify SHA256SUMS.asc SHA256SUMS \ + 3>gpg-status.txt + if grep -qE '^\[GNUPG:\] (EXPKEYSIG|REVKEYSIG|EXPSIG)' gpg-status.txt; then + echo "SHA256SUMS carries a good signature from an EXPIRED or REVOKED" >&2 + echo "deCDN release key. Refusing to install. Re-sync" >&2 + echo "roles/decdn_node/files/decdn-release-KEYS.asc from upstream KEYS," >&2 + echo "or verify this release out of band." >&2 + exit 1 + fi + grep -q '^\[GNUPG:\] VALIDSIG ' gpg-status.txt + register: decdn_sig_check + changed_when: false + failed_when: decdn_sig_check.rc != 0 + when: decdn_verify_release_signature + + # Each archive is checked by NAME against exactly one manifest entry, + # rather than running `sha256sum --check --ignore-missing` over the whole + # manifest and counting ': OK' lines. Two reasons the count was not enough: + # it counts OK lines rather than distinct files (a manifest listing one + # file twice satisfies a count of two while the other goes unverified), + # and with zero matches `grep -c` exits 1, which under `set -e` aborts the + # assignment before the diagnostic can say what was missing. + - name: Verify the tarballs against SHA256SUMS + ansible.builtin.shell: + chdir: "{{ decdn_stage.path }}" + executable: /bin/bash + cmd: | + set -euo pipefail + {% for archive in decdn_release_tarballs %} + f={{ (archive ~ '-' ~ decdn_node_version ~ '-' ~ decdn_node_target ~ '.tar.gz') | quote }} + n=$(grep -cF -- " $f" SHA256SUMS || true) + if [ "$n" -ne 1 ]; then + echo "SHA256SUMS has $n entries for $f (expected exactly 1)." >&2 + echo "The manifest does not cover this archive, or covers it twice." >&2 + exit 1 + fi + grep -F -- " $f" SHA256SUMS | sha256sum --check --strict - + {% endfor %} + register: decdn_sum_check + changed_when: false + failed_when: decdn_sum_check.rc != 0 - name: Extract binaries into /usr/local/bin ansible.builtin.unarchive: - src: "/tmp/{{ item.archive }}-{{ decdn_node_version }}.tar.gz" + src: "{{ decdn_stage.path }}/{{ item }}-{{ decdn_node_version }}-{{ decdn_node_target }}.tar.gz" dest: /usr/local/bin remote_src: true owner: root group: root mode: "0755" loop: "{{ decdn_release_tarballs }}" - loop_control: - label: "{{ item.archive }}" notify: Restart decdn-node - name: Ensure the stamp directory exists @@ -72,6 +230,27 @@ owner: root group: root mode: "0644" + always: + # Kept on failure: the tarballs, manifest, signature and gpg status file + # are the only evidence for "why did verification fail", and deleting them + # forces a re-run with cleanup disabled to find out. + - name: Report the retained staging directory after a failure + ansible.builtin.debug: + msg: >- + Verification failed; leaving {{ decdn_stage.path }} in place for + inspection (gpg-status.txt, SHA256SUMS, SHA256SUMS.asc, the archives). + Remove it by hand once done. + when: + - decdn_stage.path is defined + - decdn_sig_check.failed | default(false) or decdn_sum_check.failed | default(false) + + - name: Remove the download staging directory + ansible.builtin.file: + path: "{{ decdn_stage.path }}" + state: absent + when: + - decdn_stage.path is defined + - not (decdn_sig_check.failed | default(false) or decdn_sum_check.failed | default(false)) - name: Install locally-built binaries from the control machine when: decdn_node_install_method == "manual" @@ -114,3 +293,15 @@ decdn_installed_version.rc != 0 or (decdn_node_version | length > 0 and decdn_node_version not in decdn_installed_version.stdout) + +# The `decdn` CLI is what runs `config validate`, `key-gen` and every day-2 command, +# so a half-installed pair (daemon present, CLI missing) must fail here rather than +# at the config gate further down. +- name: Verify the installed decdn CLI runs (version backstop) + ansible.builtin.command: "{{ decdn_cli_bin }} --version" + register: decdn_installed_cli_version + changed_when: false + failed_when: >- + decdn_installed_cli_version.rc != 0 + or (decdn_node_version | length > 0 + and decdn_node_version not in decdn_installed_cli_version.stdout) diff --git a/ansible/roles/decdn_node/tasks/main.yml b/ansible/roles/decdn_node/tasks/main.yml index 892a32a..e771d91 100644 --- a/ansible/roles/decdn_node/tasks/main.yml +++ b/ansible/roles/decdn_node/tasks/main.yml @@ -12,13 +12,32 @@ decdn_node_install_method must be "release" (pinned GitHub Release tarballs) or "manual" (copy locally-built binaries from the control machine). +# NOTE (2026-09): upstream decdn/decdn has NO release tags yet — `git ls-remote +# --tags` is empty and .github/workflows/release.yml only fires on a v[0-9]* tag +# push. Until the first tag is cut, "release" mode has nothing to download and +# "manual" is the only working method; that is why the role default is "manual". +# This assert stays so the failure names the cause instead of surfacing a bare 404 +# from the first get_url. - name: Require a pinned release version ansible.builtin.assert: that: - decdn_node_version | length > 0 fail_msg: >- - decdn_node_version must be set to a published release (e.g. "0.1.0"). A - matching v GitHub Release must exist. Set it in host_vars. + decdn_node_install_method "release" requires decdn_node_version (e.g. "0.1.1") + and a matching v GitHub Release. As of decdn/decdn @ d306cc5c no + release tag has been cut, so there is nothing to download — build the two + binaries from a checkout and use decdn_node_install_method "manual" until an + upstream release exists. Set the version in host_vars once one does. + when: decdn_node_install_method == "release" + +- name: Validate decdn_verify_release_signature is a real boolean + ansible.builtin.assert: + that: + - decdn_verify_release_signature is boolean + fail_msg: >- + decdn_verify_release_signature must be a boolean (true/false). Leave it true + unless you are installing from a mirror that strips SHA256SUMS.asc — turning + it off means the tarballs are taken on trust. when: decdn_node_install_method == "release" - name: Require manual binary sources when install method is manual @@ -45,25 +64,31 @@ that: - decdn_rpc_url | length > 0 - decdn_chain_id | int > 0 - - decdn_payment_channel_address is match('^0x[0-9a-fA-F]{40}$') + - decdn_payment_pool_address is match('^0x[0-9a-fA-F]{40}$') - decdn_capacity_bond_address is match('^0x[0-9a-fA-F]{40}$') - decdn_slash_judge_address is match('^0x[0-9a-fA-F]{40}$') - - decdn_payment_channel_address != decdn_zero_address + - decdn_content_blacklist_address is match('^0x[0-9a-fA-F]{40}$') + - decdn_payment_pool_address != decdn_zero_address - decdn_capacity_bond_address != decdn_zero_address - decdn_slash_judge_address != decdn_zero_address + - decdn_content_blacklist_address != decdn_zero_address - decdn_region is match('^[A-Z]{2}$') fail_msg: >- - decdn_rpc_url, a positive decdn_chain_id, the three contract addresses + decdn_rpc_url, a positive decdn_chain_id, the FOUR contract addresses (0x + 40 hex, none the zero address), and an uppercase ISO-3166 2-letter - decdn_region are required. Source chain-id + contract addresses from the deCDN - contract deployment / the relevant ADR — never guess. Set them in host_vars. + decdn_region are required. decdn_content_blacklist_address joined this list + upstream (ADR 011/031): the daemon now refuses to resolve a config without a + ContentBlacklist, because an absent or zero address is a fail-open compliance + trap — serving a blacklisted hash past its window is slashable. Source + chain-id + contract addresses from the deCDN contract deployment manifest + (decdn/contracts/deployments/.json) — never guess. Set them in host_vars. vars: decdn_zero_address: "0x0000000000000000000000000000000000000000" # slash_appeal_address is optional (only `decdn appeal slash` uses it — the daemon -# ignores the key), so it is not in the hard assert above. But if an operator sets -# it, a half-filled/zero value would silently render a bad node.toml, so validate -# it the same fail-loud way as the required addresses when present. +# validates but ignores the key), so it is not in the hard assert above. But if an +# operator sets it, a half-filled/zero value would silently render a bad node.toml, +# so validate it the same fail-loud way as the required addresses when present. - name: Validate the optional SlashAppeal address when set ansible.builtin.assert: that: @@ -77,73 +102,52 @@ decdn_zero_address: "0x0000000000000000000000000000000000000000" when: decdn_slash_appeal_address | length > 0 -# slash_judge_from_block feeds a `| int > 0` gate in node.toml.j2, and Jinja's int -# filter silently coerces any unparseable value to 0 — which drops the key and -# reverts the daemon to a full-chain rescan on every restart, discarding the -# operator's input with no trace in the rendered config. Assert integer shape so a -# typo fails loud at deploy time instead. Runs unconditionally: the default 0 -# passes, so a malformed value can never slip through the coercion silently. -- name: Validate the SlashJudge scan-floor block is a non-negative integer - ansible.builtin.assert: - that: - - decdn_slash_judge_from_block | string is match('^[0-9]+$') - fail_msg: >- - decdn_slash_judge_from_block must be a non-negative integer (the SlashJudge - deploy block). Got "{{ decdn_slash_judge_from_block }}". Leave it 0 to scan - from genesis, or set the deploy block to bound per-restart rescan cost. - -# The ADR 022 origin directory needs BOTH OriginAssignment + PublisherRegistry (it -# gates prefetch on the pair); the template emits them together, so reject a -# one-sided config here. Runs when EITHER is set — an empty counterpart then fails -# the address regex, turning "both or neither" into a loud, specific failure. -- name: Validate the optional origin-directory contract pair (ADR 022) +# The ADR 022 origin-directory contracts are INDEPENDENTLY optional upstream — the +# old "both or neither" rule went away when the daemon stopped mirroring the whole +# directory (#1737). origin_assignment_address feeds the daemon's cache-miss +# pull-through fallback; publisher_registry_address is validate-only on the daemon +# and consumed by `decdn publish`. Validate each on its own when set, so a typo +# still fails loud without forcing an operator to set a contract they don't use. +- name: Validate the optional origin-directory contract addresses (ADR 022) ansible.builtin.assert: that: - - decdn_origin_assignment_address is match('^0x[0-9a-fA-F]{40}$') - - decdn_publisher_registry_address is match('^0x[0-9a-fA-F]{40}$') - - decdn_origin_assignment_address != decdn_zero_address - - decdn_publisher_registry_address != decdn_zero_address + - item.value is match('^0x[0-9a-fA-F]{40}$') + - item.value != decdn_zero_address fail_msg: >- - The origin directory (ADR 022) needs BOTH decdn_origin_assignment_address and - decdn_publisher_registry_address set to valid contract addresses (0x + 40 hex, - not the zero address) — or BOTH empty to omit it. Source them from the deCDN - contract deployment for your target chain / ADR 022. + {{ item.name }} is set but is not a valid contract address (0x + 40 hex, not + the zero address). Leave it empty to omit it, or source the deployed address + from the deCDN contract deployment manifest for your chain / ADR 022. vars: decdn_zero_address: "0x0000000000000000000000000000000000000000" - when: >- - (decdn_origin_assignment_address | length > 0) - or (decdn_publisher_registry_address | length > 0) + loop: + - {name: decdn_origin_assignment_address, value: "{{ decdn_origin_assignment_address }}"} + - {name: decdn_publisher_registry_address, value: "{{ decdn_publisher_registry_address }}"} + loop_control: + label: "{{ item.name }}" + when: item.value | length > 0 -# content_blacklist_address is optional (ADR 011/031); a half-filled/zero value -# would silently render a bad node.toml, so validate it fail-loud when present — -# same posture as the SlashAppeal address. -- name: Validate the optional ContentBlacklist address when set +# The CLI-only [blockchain] address keys (usdc + swap venue). The daemon validates +# them under deny_unknown_fields even though only `decdn setup` / `decdn pool` read +# them, so a malformed value fails the daemon's own startup — catch it here first. +- name: Validate the optional CLI-only contract addresses when set ansible.builtin.assert: that: - - decdn_content_blacklist_address is match('^0x[0-9a-fA-F]{40}$') - - decdn_content_blacklist_address != decdn_zero_address + - item.value is match('^0x[0-9a-fA-F]{40}$') + - item.value != decdn_zero_address fail_msg: >- - decdn_content_blacklist_address is set but is not a valid contract address - (0x + 40 hex, not the zero address). Leave it empty to omit it, or set the - deployed ContentBlacklist address (source it from the deployment / ADR 011). + {{ item.name }} is set but is not a valid contract address (0x + 40 hex, not + the zero address). Leave it empty to omit it. vars: decdn_zero_address: "0x0000000000000000000000000000000000000000" - when: decdn_content_blacklist_address | length > 0 - -# The *_from_block knobs feed `| int > 0` gates in node.toml.j2, and Jinja's int -# filter silently coerces an unparseable value to 0 — dropping the key and -# reverting to a full-chain rescan with no trace. Assert integer shape (like -# slash_judge_from_block) so a typo fails loud. Unconditional: the default 0 passes. -- name: Validate the origin-directory and blacklist scan-floor blocks are integers - ansible.builtin.assert: - that: - - decdn_origin_directory_from_block | string is match('^[0-9]+$') - - decdn_content_blacklist_from_block | string is match('^[0-9]+$') - fail_msg: >- - decdn_origin_directory_from_block and decdn_content_blacklist_from_block must - be non-negative integers (the respective contract deploy blocks). Got - "{{ decdn_origin_directory_from_block }}" / "{{ decdn_content_blacklist_from_block }}". - Leave them 0 to scan from genesis, or set the deploy block to bound rescan cost. + loop: + - {name: decdn_usdc_address, value: "{{ decdn_usdc_address }}"} + - {name: decdn_swap_router_address, value: "{{ decdn_swap_router_address }}"} + - {name: decdn_swap_quoter_address, value: "{{ decdn_swap_quoter_address }}"} + - {name: decdn_swap_balancer_pool, value: "{{ decdn_swap_balancer_pool }}"} + - {name: decdn_swap_pool_address, value: "{{ decdn_swap_pool_address }}"} + loop_control: + label: "{{ item.name }}" + when: item.value | length > 0 # The cache pull-through origin is a tagged [cache.origin] table: node.toml.j2 # emits only the chosen kind's fields, and the daemon denies unknown fields. Reject @@ -163,7 +167,9 @@ # leftover value for a non-selected kind is never rendered, so it must not # fail an unrelated deploy. http: DecompressMode accepts only auto|strict (a # typo would render TOML the daemon rejects at load — an opaque crash-loop). - - (decdn_cache_origin_kind != "http") or (decdn_cache_origin_decompress in ["", "auto", "strict"]) + # DecompressMode is a field on BOTH the http variant and S3OriginConfig, and + # the template emits it for both — so guard both, not just http. + - decdn_cache_origin_kind not in ["http", "s3"] or decdn_cache_origin_decompress in ["", "auto", "strict"] # s3: path_style feeds a `| bool` gate in node.toml.j2; require a REAL boolean # so a quoted "false" is rejected here, not silently coerced to path_style = true. - (decdn_cache_origin_kind != "s3") or (decdn_cache_origin_s3_path_style is boolean) @@ -176,6 +182,102 @@ kind empty to omit [cache.origin] (cache misses then fail NoOrigin). when: decdn_cache_origin_kind | length > 0 +# The inverse gate. Every decdn_cache_origin_* default is "", and the validator +# above is skipped when the kind is empty — so setting a payload and forgetting the +# kind renders NO [cache.origin] table, produces no assert, no diff line and a +# green deploy, and every cache miss then fails NoOrigin at runtime on a node that +# looks healthy to systemd and to the readiness probe. Fail loud instead. +- name: Require a cache origin kind when any origin field is set + ansible.builtin.assert: + that: + - decdn_cache_origin_kind | length > 0 + fail_msg: >- + One or more decdn_cache_origin_* values are set but decdn_cache_origin_kind is + empty, so NO [cache.origin] table is rendered and every cache miss would fail + NoOrigin. Set decdn_cache_origin_kind to http|fs|s3, or clear the leftover + fields (or use decdn_cache_origins for the multi-origin form). + when: >- + (decdn_cache_origins | length == 0) + and ((decdn_cache_origin_url | length > 0) or (decdn_cache_origin_path | length > 0) + or (decdn_cache_origin_s3_bucket | length > 0) or (decdn_cache_origin_s3_region | length > 0) + or (decdn_cache_origin_s3_endpoint_url | length > 0) + or (decdn_cache_origin_s3_prefix | length > 0) + or (decdn_cache_origin_decompress | length > 0) + or (decdn_cache_origin_s3_profile | length > 0) + or decdn_cache_origin_s3_use_default_chain | bool) + +# Same per-kind contract for the [[cache.origins]] list form. Entries are dicts, so +# the checks read from item.* rather than the scalar vars, and a missing key must be +# caught here: the template renders `o.url` unguarded for kind http, which would +# raise an undefined-variable error mid-template rather than a useful message. +- name: Validate each cache origin in the multi-origin list + ansible.builtin.assert: + that: + - item.kind | default('') in ["http", "fs", "s3"] + - (item.kind != "http") or (item.url | default('') | length > 0) + - (item.kind != "fs") or (item.path | default('') | length > 0) + - >- + (item.kind != "s3") + or (item.bucket | default('') | length > 0 and item.region | default('') | length > 0) + - item.decompress | default('') in ["", "auto", "strict"] + - item.path_style is not defined or item.path_style is boolean + # The template renders a FIXED field list per kind, so any other key is + # silently dropped — `path_stile: true` would leave the node talking to + # MinIO in virtual-host mode with nothing in the diff to show for it. + # Mirror the rate-limit tables' subset check. + - item.keys() | list | difference(decdn_origin_keys[item.kind]) | length == 0 + fail_msg: >- + decdn_cache_origins[{{ ansible_loop.index0 }}] is not a valid origin: each entry + needs kind http|fs|s3 plus that kind's required fields — http: url; + fs: path; s3: bucket + region. Optional decompress must be auto|strict and + path_style must be a real boolean, and no key outside + {{ decdn_origin_keys[item.kind | default('http')] | join(', ') }} (unknown keys + are silently dropped by the template). Got {{ item }}. + vars: + # Per-kind field lists, matching what node.toml.j2 actually emits. `credentials` + # is absent on purpose: the template only renders the credentials sub-table for + # the single-origin form, so accepting it here would silently drop it. + decdn_origin_keys: + http: [kind, url, decompress] + fs: [kind, path] + s3: [kind, bucket, region, endpoint_url, path_style, prefix, decompress] + loop: "{{ decdn_cache_origins }}" + loop_control: + label: "{{ item.kind | default('?') }}" + extended: true + when: decdn_cache_origins | length > 0 + +# The S3 default-chain credential toggle. Static access keys are deliberately NOT +# templated (node.toml is 0640 and diffable) — they belong in the 0600 env file via +# decdn_extra_env, which is where the AWS default chain reads them from. +- name: Validate the S3 credential-source knobs + ansible.builtin.assert: + that: + - decdn_cache_origin_s3_use_default_chain is boolean + - >- + decdn_cache_origin_s3_profile in ["", none] + or decdn_cache_origin_s3_profile is match('^[^\"\\s\\\\]+$') + fail_msg: >- + decdn_cache_origin_s3_use_default_chain must be a real boolean, and + decdn_cache_origin_s3_profile (when set) must contain no quote, whitespace or + backslash. Static AWS keys are never written to node.toml — put + AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN in + decdn_extra_env (git-ignored secret.yml) and leave the default chain to read them. + +- name: Reject S3 credential knobs alongside the multi-origin form + ansible.builtin.assert: + that: + - not (decdn_cache_origin_s3_use_default_chain | bool + or decdn_cache_origin_s3_profile | length > 0) + fail_msg: >- + decdn_cache_origin_s3_use_default_chain / decdn_cache_origin_s3_profile apply + only to the single [cache.origin] table — the template does not render a + credentials sub-table for [[cache.origins]] entries, so setting them with + decdn_cache_origins would SILENTLY drop them. Omit the credentials block (the + AWS default chain is used anyway when it is absent) and supply keys via + decdn_extra_env, or switch to the single-origin form. + when: decdn_cache_origins | length > 0 + # --- Optional tuning knobs ("" => omit + use the daemon default) -------------- # These template with a `!= ""` gate (NOT `| int > 0`) because 0/false are # meaningful values here. A bad value would render TOML the daemon rejects at @@ -186,18 +288,19 @@ - name: Validate optional boolean tuning knobs ansible.builtin.assert: that: - - decdn_enable_0rtt in ["", none] or decdn_enable_0rtt is boolean - - decdn_buyer_max_approve in ["", none] or decdn_buyer_max_approve is boolean - - decdn_node_to_node_pull_through_enabled in ["", none] or decdn_node_to_node_pull_through_enabled is boolean - - >- - decdn_pull_through_require_authorized_origin in ["", none] - or decdn_pull_through_require_authorized_origin is boolean + - item.value in ["", none] or item.value is boolean + quiet: true fail_msg: >- - decdn_enable_0rtt, decdn_buyer_max_approve, - decdn_node_to_node_pull_through_enabled and - decdn_pull_through_require_authorized_origin must each be a real boolean - (true/false), not a quoted string — they template through `| lower` into a - bare TOML bool. Leave a knob "" (or null) to omit it (daemon default applies). + {{ item.name }} must be a real boolean (true/false), not a quoted string — it + templates through `| bool | lower` into a bare TOML bool. Leave it "" (or + null) to omit the key and take the daemon default. + loop: + - {name: decdn_buyer_max_approve, value: "{{ decdn_buyer_max_approve }}"} + - {name: decdn_node_to_node_pull_through_enabled, value: "{{ decdn_node_to_node_pull_through_enabled }}"} + - {name: decdn_relay_foreign_namespaces, value: "{{ decdn_relay_foreign_namespaces }}"} + - {name: decdn_circuit_breaker_enabled, value: "{{ decdn_circuit_breaker_enabled }}"} + loop_control: + label: "{{ item.name }}" # Integer shape first (a separate task): if a value is non-numeric this fails loud # BEFORE the range task below runs its `| int` comparisons, so `| int` there never @@ -213,30 +316,136 @@ quiet: true fail_msg: >- "{{ item }}" is not a non-negative integer <= i64::MAX (9223372036854775807). - Every optional numeric knob (delivery/settlement/watcher/cache-tuning/ - pull-through) must be a bare u64 in that range, or "" / null to omit it. See - roles/decdn_node/defaults/main.yml for the full list. + Every optional numeric knob (payment/settlement/watcher/cache-tuning/ + pull-through/limits) must be a bare u64 in that range, or "" / null to omit + it. See roles/decdn_node/defaults/main.yml for the full list. # A list EXPRESSION (not a list of "{{ }}" strings) so each item keeps its type — # a null stays None (caught by the `in [..., none]` guard) instead of becoming the # string "None", which would fail the shape regex. loop: >- - {{ [decdn_delivery_floor, decdn_delivery_ceiling, decdn_voucher_interval_mb, - decdn_redeem_threshold_micro_usdc, decdn_buyer_deposit_micro_usdc, - decdn_settlement_auto_threshold_micro_usdc, decdn_settlement_auto_by_voucher_nonce_span, - decdn_rpc_watchdog_interval_sec, decdn_event_poll_interval_ms, - decdn_content_blacklist_poll_interval_sec, decdn_max_probe_holds, - decdn_stake_lane_reserved_holds, decdn_gc_interval_sec, decdn_node_pull_probe_fanout, - decdn_node_pull_timeout_sec, decdn_pull_ahead_bytes, decdn_max_unrecouped_leech_bytes, - decdn_pull_share_ratio_percent, decdn_region_accounting_interval_sec] }} + {{ [decdn_delivery_floor, decdn_credit_max, decdn_credit_ramp_divisor, + decdn_frame_target_bytes, decdn_voucher_commit_interval_ms, + decdn_redeem_threshold_micro_usdc, decdn_redeem_max_vouchers_per_tx, + decdn_redeem_interval_secs, decdn_buyer_working_deposit_micro_usdc, + decdn_pool_min_remaining_deposit_micro_usdc, decdn_pool_floor_signer_share_bps, + decdn_pool_floor_signer_max_windows, decdn_rpc_watchdog_interval_sec, + decdn_event_poll_interval_ms, decdn_content_blacklist_poll_interval_sec, + decdn_rate_bounds_poll_interval_sec, decdn_fee_shares_poll_interval_sec, + decdn_origin_directory_positive_ttl_sec, decdn_origin_directory_negative_ttl_sec, + decdn_origin_directory_cache_capacity, decdn_swap_fee_tier, + decdn_max_blob_size_mb, decdn_max_rate_per_mb, decdn_max_probe_holds, + decdn_stake_lane_reserved_holds, decdn_gc_interval_sec, + decdn_fs_rescan_interval_sec, decdn_origin_probe_ttl_sec, + decdn_origin_probe_negative_ttl_sec, decdn_origin_probe_fault_ttl_sec, + decdn_origin_probe_timeout_ms, decdn_origin_probe_memo_capacity, + decdn_eviction_high_water_pct, decdn_eviction_target_pct, + decdn_eviction_per_sweep_budget, decdn_eviction_tick_secs, + decdn_node_pull_probe_fanout, decdn_node_pull_timeout_sec, + decdn_node_pull_stall_window_sec, decdn_node_pull_min_throughput_bps, + decdn_tinylfu_sketch_bytes, decdn_tinylfu_promotion_threshold, + decdn_tinylfu_probation_target_pct, decdn_tinylfu_aging_halflife_sec, + decdn_serve_economics_n_max, decdn_serve_economics_warming_budget, + decdn_serve_economics_warming_refill, decdn_origin_retry_max_retries, + decdn_origin_retry_initial_backoff_ms, decdn_origin_retry_max_backoff_ms, + decdn_origin_retry_buffered_max_bytes, decdn_circuit_breaker_failure_threshold, + decdn_circuit_breaker_cooldown_ms, decdn_circuit_breaker_half_open_max_calls, + decdn_security_max_concurrent_handlers, decdn_security_per_source_burst, + decdn_security_max_tracked_sources, decdn_load_shed_egress_budget_mbps, + decdn_load_shed_max_concurrent_serves_high, decdn_load_shed_max_concurrent_serves_low, + decdn_load_shed_per_client_serve_cap, decdn_receipts_max_file_bytes, + decdn_receipts_retained_files] }} + +# Float-shaped knobs render through `| float` into a bare TOML float. A non-numeric +# value would make `| float` coerce to 0.0 and silently disable a rate limit, so +# assert the shape (accepting ints, which Jinja widens) before that happens. +- name: Validate optional float tuning knobs are non-negative numbers + ansible.builtin.assert: + that: + - item.value in ["", none] or (item.value is number and item.value | float >= 0.0) + quiet: true + fail_msg: >- + {{ item.name }} must be a non-negative number (it templates through `| float` + into a bare TOML float). Got "{{ item.value }}". Leave it "" to omit the key. + loop: + - {name: decdn_serve_economics_discount, value: "{{ decdn_serve_economics_discount }}"} + - {name: decdn_origin_retry_jitter_ratio, value: "{{ decdn_origin_retry_jitter_ratio }}"} + - {name: decdn_security_per_source_rate_per_sec, value: "{{ decdn_security_per_source_rate_per_sec }}"} + loop_control: + label: "{{ item.name }}" + +# The two rate-limit tables share one shape. Reject unknown keys here rather than +# letting them render: [dht.rate_limit] / [probe.rate_limit] are deny_unknown_fields +# upstream, and the template's key loop would silently DROP anything not on its +# list — so a typo'd knob would leave the limit at its default with no trace. +- name: Validate optional DHT + probe rate-limit tables + ansible.builtin.assert: + that: + - item.value | type_debug in ["dict", "AnsibleMapping"] + - item.value.keys() | difference(decdn_rate_limit_keys) | length == 0 + - item.value.values() | select('string') | list | length == 0 + - item.value.values() | reject('number') | list | length == 0 + # Jinja's `number` test is isinstance(v, numbers.Number) and Python's bool + # subclasses int, so `true` passes it and renders as `1` — a burst of one + # rather than a config error. Reject booleans explicitly. + - item.value.values() | select('boolean') | list | length == 0 + - item.value.values() | select('lt', 0) | list | length == 0 + fail_msg: >- + {{ item.name }} must be a mapping whose keys are a subset of + {{ decdn_rate_limit_keys | join(', ') }}, with non-negative, non-boolean, + unquoted numeric values. + Got {{ item.value }}. Unknown keys are silently dropped by the template and + rejected by the daemon, so they fail here instead. + vars: + decdn_rate_limit_keys: + - per_peer_rate_per_sec + - per_peer_burst + - per_ip_rate_per_sec + - per_ip_burst + - global_rate_per_sec + - global_burst + - max_tracked_per_ip + - max_tracked_per_peer + loop: + - {name: decdn_dht_rate_limit, value: "{{ decdn_dht_rate_limit }}"} + - {name: decdn_probe_rate_limit, value: "{{ decdn_probe_rate_limit }}"} + loop_control: + label: "{{ item.name }}" + when: item.value | length > 0 + +# Enum-valued knobs. The daemon matches these against a closed set and refuses to +# start on anything else, so a typo is a crash-loop rather than a fallback. +- name: Validate optional enum knobs + ansible.builtin.assert: + that: + - item.value in ["", none] or item.value in item.allowed + quiet: true + fail_msg: >- + {{ item.name }} must be one of {{ item.allowed | join(' | ') }} + (got "{{ item.value }}"), or "" to take the daemon default. + # Every allowed value is QUOTED. YAML 1.1 parses a bare `off` (and `on`, `no`, + # `yes`) as a boolean, so `allowed: [off, margin]` would become [False, "margin"] + # and the string "off" could never match — silently rejecting a valid policy. + loop: + - {name: decdn_eviction_policy, value: "{{ decdn_eviction_policy }}", allowed: ["lru", "tinylfu"]} + - {name: decdn_admission_policy, value: "{{ decdn_admission_policy }}", allowed: ["always", "tinylfu"]} + - {name: decdn_serve_economics_policy, value: "{{ decdn_serve_economics_policy }}", + allowed: ["off", "margin"]} + - {name: decdn_load_shed_policy, value: "{{ decdn_load_shed_policy }}", + allowed: ["resource-pressure", "always-admit"]} + - {name: decdn_swap_venue, value: "{{ decdn_swap_venue }}", allowed: ["uniswap-v3", "balancer-v3"]} + - {name: decdn_log_format, value: "{{ decdn_log_format }}", allowed: ["pretty", "json"]} + - {name: decdn_log_level, value: "{{ decdn_log_level }}", + allowed: ["trace", "debug", "info", "warn", "error"]} + loop_control: + label: "{{ item.name }}" # Range / nonzero constraints the daemon enforces at load (out-of-range => the node # refuses to start). Values are integer-shaped by the task above, so `| int` is safe. +# ("" and null both mean "unset" and skip the check.) - name: Validate optional knob ranges the daemon rejects out-of-range ansible.builtin.assert: that: # rpc_watchdog: 0 disables, otherwise must be >= 10 (1..9 rejected upstream). - # ("" and null both mean "unset" and skip the check — the shape task above - # already rejected any non-numeric, non-null value.) - >- decdn_rpc_watchdog_interval_sec in ["", none] or (decdn_rpc_watchdog_interval_sec | int == 0 or decdn_rpc_watchdog_interval_sec | int >= 10) @@ -244,58 +453,113 @@ - >- decdn_content_blacklist_poll_interval_sec in ["", none] or (decdn_content_blacklist_poll_interval_sec | int >= 1) - - decdn_redeem_threshold_micro_usdc in ["", none] or (decdn_redeem_threshold_micro_usdc | int >= 1) - - decdn_buyer_deposit_micro_usdc in ["", none] or (decdn_buyer_deposit_micro_usdc | int >= 1) - # settlement auto-close: 0 is rejected (omit to disable), so require >= 1. - - >- - decdn_settlement_auto_threshold_micro_usdc in ["", none] - or (decdn_settlement_auto_threshold_micro_usdc | int >= 1) - - >- - decdn_settlement_auto_by_voucher_nonce_span in ["", none] - or (decdn_settlement_auto_by_voucher_nonce_span | int >= 1) - - >- - decdn_delivery_ceiling in ["", none] - or (decdn_delivery_ceiling | int >= 1 and decdn_delivery_ceiling | int <= 1000000000000) - - >- - decdn_voucher_interval_mb in ["", none] - or (decdn_voucher_interval_mb | int >= 1 and decdn_voucher_interval_mb | int <= 1024) + # Knobs the daemon requires strictly positive (a configured 0 is rejected — + # omit the knob instead to take the default). + - item in ["", none] or (item | int >= 1) + quiet: true fail_msg: >- An optional knob is out of the range the daemon accepts: decdn_rpc_watchdog_interval_sec (0 or >= 10), decdn_event_poll_interval_ms (>= 250), - decdn_content_blacklist_poll_interval_sec (>= 1), - decdn_redeem_threshold_micro_usdc / decdn_buyer_deposit_micro_usdc (>= 1), - decdn_settlement_auto_threshold_micro_usdc / - decdn_settlement_auto_by_voucher_nonce_span (>= 1 — leave "" to disable, 0 is - rejected), decdn_delivery_ceiling (1..=1_000_000_000_000), - decdn_voucher_interval_mb (1..=1024). + decdn_content_blacklist_poll_interval_sec (>= 1); and each of + decdn_redeem_threshold_micro_usdc, decdn_redeem_max_vouchers_per_tx, + decdn_redeem_interval_secs, decdn_buyer_working_deposit_micro_usdc, + decdn_rate_bounds_poll_interval_sec, decdn_fee_shares_poll_interval_sec, + decdn_voucher_commit_interval_ms must be >= 1 when set ("{{ item }}" is not). + loop: >- + {{ [decdn_redeem_threshold_micro_usdc, decdn_redeem_max_vouchers_per_tx, + decdn_redeem_interval_secs, decdn_buyer_working_deposit_micro_usdc, + decdn_rate_bounds_poll_interval_sec, decdn_fee_shares_poll_interval_sec, + decdn_voucher_commit_interval_ms] }} + +# Bounded ranges. Each is a closed interval the daemon range-checks at load, so an +# out-of-range value is a startup refusal rather than a clamp. +- name: Validate optional bounded-range knobs + ansible.builtin.assert: + that: + - item.value in ["", none] or (item.value | int >= item.min and item.value | int <= item.max) + quiet: true + fail_msg: >- + {{ item.name }} must be in {{ item.min }}..={{ item.max }} when set + (got "{{ item.value }}"). Leave it "" to take the daemon default. + loop: + - {name: decdn_eviction_high_water_pct, value: "{{ decdn_eviction_high_water_pct }}", min: 60, max: 95} + - {name: decdn_eviction_target_pct, value: "{{ decdn_eviction_target_pct }}", min: 40, max: 90} + - {name: decdn_eviction_per_sweep_budget, value: "{{ decdn_eviction_per_sweep_budget }}", min: 1, max: 256} + - {name: decdn_eviction_tick_secs, value: "{{ decdn_eviction_tick_secs }}", min: 1, max: 60} + - {name: decdn_frame_target_bytes, value: "{{ decdn_frame_target_bytes }}", min: 1, max: 1048576} + - {name: decdn_pool_floor_signer_share_bps, value: "{{ decdn_pool_floor_signer_share_bps }}", + min: 1, max: 10000} + - {name: decdn_tinylfu_probation_target_pct, value: "{{ decdn_tinylfu_probation_target_pct }}", + min: 1, max: 50} + # sketch_bytes has a HARD FLOOR: below it the daemon errors out at config load. + # The floor bites on the stock lru/always policies too, because the default + # serve_economics policy ("margin") builds the same sketch (#1803 review). + - {name: decdn_tinylfu_sketch_bytes, value: "{{ decdn_tinylfu_sketch_bytes }}", + min: 16384, max: 9223372036854775807} + - {name: decdn_origin_retry_buffered_max_bytes, value: "{{ decdn_origin_retry_buffered_max_bytes }}", + min: 1, max: 67108864} + - {name: decdn_receipts_max_file_bytes, value: "{{ decdn_receipts_max_file_bytes }}", + min: 1048576, max: 1073741824} + - {name: decdn_receipts_retained_files, value: "{{ decdn_receipts_retained_files }}", min: 1, max: 100} + - {name: decdn_redeem_interval_secs, value: "{{ decdn_redeem_interval_secs }}", min: 1, max: 21600} + loop_control: + label: "{{ item.name }}" - name: Validate optional cross-field knob constraints ansible.builtin.assert: that: # The daemon fills each UNSET side with its own default and THEN enforces the # constraint, so compare effective (default-substituted) values — NOT "skip if - # either side is unset". Skipping would let a raised pull_ahead with an unset - # leech cap (which resolves to 256 MiB), or a floor above the default ceiling, - # slip through here and crash-loop the daemon at config load. - - _eff_dfloor | int <= _eff_dceil | int - # leech cap: a RESOLVED 0 disables it; otherwise pull_ahead <= leech. - - _eff_leech | int == 0 or (_eff_pull | int <= _eff_leech | int) + # either side is unset". Skipping would let a lowered high-water mark with an + # unset target (which resolves to 80) slip through here and crash-loop the + # daemon at config load. + # + # The eviction hysteresis gap is STRUCTURAL: target must sit at least 5 points + # below high-water, not merely below it. + - _eff_evict_target | int <= (_eff_evict_high | int - 5) + # Origin-probe memo TTLs must be ordered negative <= fault <= positive. + - _eff_probe_neg | int <= _eff_probe_fault | int + - _eff_probe_fault | int <= _eff_probe_pos | int + # A single blob may not exceed the whole cache (0 = unlimited, so exempt). + - _eff_max_blob | int == 0 or (_eff_max_blob | int <= decdn_cache_size_mb | int) + # Load-shed low-water mark may not exceed the high-water mark. + - _eff_shed_low | int <= _eff_shed_high | int fail_msg: >- Cross-field constraint violated (unset sides compared against the daemon - default): decdn_delivery_floor must be <= decdn_delivery_ceiling (default - 1_000_000_000_000), and decdn_pull_ahead_bytes must be <= - decdn_max_unrecouped_leech_bytes (default 268435456) unless that cap is 0 (off). + default): decdn_eviction_target_pct (dflt 80) must be <= + decdn_eviction_high_water_pct (dflt 90) MINUS 5 — the gap is structural; + decdn_origin_probe_negative_ttl_sec (dflt 2) <= decdn_origin_probe_fault_ttl_sec + (dflt 5) <= decdn_origin_probe_ttl_sec (dflt 15); + decdn_max_blob_size_mb must be <= decdn_cache_size_mb (0 = unlimited); + decdn_load_shed_max_concurrent_serves_low (dflt 192) must be <= + decdn_load_shed_max_concurrent_serves_high (dflt 256). vars: # "" and null both mean "unset" => substitute the daemon default. - _eff_dfloor: "{{ decdn_delivery_floor if decdn_delivery_floor not in ['', none] else 0 }}" - _eff_dceil: "{{ decdn_delivery_ceiling if decdn_delivery_ceiling not in ['', none] else 1000000000000 }}" - _eff_pull: "{{ decdn_pull_ahead_bytes if decdn_pull_ahead_bytes not in ['', none] else 1048576 }}" - _eff_leech: >- - {{ decdn_max_unrecouped_leech_bytes if decdn_max_unrecouped_leech_bytes not in ['', none] - else 268435456 }} + _eff_evict_high: >- + {{ decdn_eviction_high_water_pct if decdn_eviction_high_water_pct not in ['', none] else 90 }} + _eff_evict_target: >- + {{ decdn_eviction_target_pct if decdn_eviction_target_pct not in ['', none] else 80 }} + _eff_probe_pos: >- + {{ decdn_origin_probe_ttl_sec if decdn_origin_probe_ttl_sec not in ['', none] else 15 }} + _eff_probe_neg: >- + {{ decdn_origin_probe_negative_ttl_sec + if decdn_origin_probe_negative_ttl_sec not in ['', none] else 2 }} + _eff_probe_fault: >- + {{ decdn_origin_probe_fault_ttl_sec + if decdn_origin_probe_fault_ttl_sec not in ['', none] else 5 }} + # max_blob_size_mb defaults to cache_size_mb upstream (#1685), so an unset side + # trivially satisfies the constraint. + _eff_max_blob: >- + {{ decdn_max_blob_size_mb if decdn_max_blob_size_mb not in ['', none] else decdn_cache_size_mb }} + _eff_shed_high: >- + {{ decdn_load_shed_max_concurrent_serves_high + if decdn_load_shed_max_concurrent_serves_high not in ['', none] else 256 }} + _eff_shed_low: >- + {{ decdn_load_shed_max_concurrent_serves_low + if decdn_load_shed_max_concurrent_serves_low not in ['', none] else 192 }} -- name: Validate optional list + singular-relay knobs +- name: Validate optional list knobs ansible.builtin.assert: that: # Fully anchored (^...$) with [^"\s\\]: a start-anchored '^https?://' prefix @@ -304,15 +568,80 @@ # in a TOML basic string it starts an escape, and a trailing '\' escapes the # closing quote. A null list is treated as unset (is none => pass). - decdn_relay_urls is none or decdn_relay_urls | reject('match', '^https?://[^\"\\s\\\\]+$') | list | length == 0 - # The singular relay_url is templated too (when relay_urls is empty), so guard it - # with the same rule — a bad value there breaks node.toml just the same. - - decdn_relay_url in ["", none] or (decdn_relay_url is match('^https?://[^\"\\s\\\\]+$')) - decdn_pinned_hashes is none or decdn_pinned_hashes | reject('match', '^[0-9a-f]{64}$') | list | length == 0 + # The node-local [content] denylist (hot-reloadable). Hashes are bare 64-char + # lowercase hex; denied origins are 0x publisher addresses, checksum-agnostic + # upstream but never the zero address. + - >- + decdn_content_denied_hashes is none + or decdn_content_denied_hashes | reject('match', '^[0-9a-f]{64}$') | list | length == 0 + - >- + decdn_content_denied_origins is none + or decdn_content_denied_origins | reject('match', '^0x[0-9a-fA-F]{40}$') | list | length == 0 + - >- + decdn_content_denied_origins is none + or '0x0000000000000000000000000000000000000000' not in decdn_content_denied_origins + fail_msg: >- + decdn_relay_urls entries must each be an http(s):// URL with no quote, + whitespace or backslash; decdn_pinned_hashes and decdn_content_denied_hashes + entries must each be a 64-char lowercase-hex BLAKE3 hash (uppercase is + rejected upstream); decdn_content_denied_origins entries must each be a + 0x + 40 hex address and none may be the zero address. Leave a list empty ([]). + +# Address discovery (#818): pkarr_url and dns_origin are a PAIR upstream — a custom +# pkarr relay with no DNS origin cannot resolve peers. Static peers are independent +# and combinable, so they are validated separately. +- name: Validate the optional address-discovery configuration + ansible.builtin.assert: + that: + # pkarr_url PUBLISHES this node's record; dns_origin RESOLVES peers. A + # publish-only config cannot find anyone, so pkarr requires dns. The reverse + # is legal upstream — dns_origin alone is a resolve-only node that does not + # publish — so it is deliberately NOT required to carry a pkarr_url. + - decdn_discovery_pkarr_url | length == 0 or decdn_discovery_dns_origin | length > 0 + - >- + decdn_discovery_pkarr_url | length == 0 + or decdn_discovery_pkarr_url is match('^https?://[^\"\\s\\\\]+$') + - >- + decdn_discovery_dns_origin | length == 0 + or decdn_discovery_dns_origin is match('^[A-Za-z0-9.-]+$') fail_msg: >- - decdn_relay_urls / decdn_relay_url entries must each be an http(s):// URL with - no quote, whitespace or backslash, and decdn_pinned_hashes entries must each be - a 64-char lowercase-hex BLAKE3 hash (uppercase is rejected upstream). Leave a - list empty ([]) or a scalar "" to omit it. + decdn_discovery_pkarr_url requires decdn_discovery_dns_origin (a custom pkarr + relay with no DNS origin publishes this node's address but cannot resolve + anyone else's). dns_origin alone is fine — that is a resolve-only node. The + pkarr URL must be http(s):// with no quote/whitespace/backslash and the DNS + origin a bare domain. Setting either drops the n0 discovery leg entirely. + +- name: Validate the optional static discovery peers + ansible.builtin.assert: + that: + - item.key is match('^[0-9a-f]{64}$') + - item.value.addrs is defined and item.value.addrs | length > 0 + - item.value.addrs | reject('match', '^[^\"\\s\\\\]+:[0-9]+$') | list | length == 0 + - >- + item.value.relay_url is not defined or item.value.relay_url in ["", none] + or item.value.relay_url is match('^https?://[^\"\\s\\\\]+$') + fail_msg: >- + Each decdn_discovery_peers key must be a 64-char lowercase-hex NodeId, and its + value a mapping with a non-empty `addrs` list of host:port entries plus an + optional http(s) `relay_url`. Offending entry: {{ item.key }}. + loop: "{{ decdn_discovery_peers | default({}, true) | dict2items }}" + loop_control: + label: "{{ item.key }}" + +# cache.origin and cache.origins are mutually exclusive upstream, and an empty +# `origins` array is rejected outright. The template already prefers the list form +# and never emits both, but that would silently IGNORE a scalar origin an operator +# thought was in effect — so make the ambiguity fail loud instead. +- name: Validate the cache origin forms are not both configured + ansible.builtin.assert: + that: + - not (decdn_cache_origins | length > 0 and decdn_cache_origin_kind | length > 0) + fail_msg: >- + decdn_cache_origins (the ordered [[cache.origins]] fallback list) and + decdn_cache_origin_kind (the single [cache.origin] table) are mutually + exclusive — upstream rejects a config carrying both. Clear + decdn_cache_origin_kind to use the list, or empty the list to use the scalars. # user_agent + otlp_endpoint interpolate into TOML basic strings ("..."). An # unescaped '"', backslash or newline breaks the rendered node.toml (a daemon @@ -331,6 +660,69 @@ decdn_otlp_endpoint must be an http(s):// URL with no quote, whitespace or backslash (both are interpolated verbatim into node.toml). Leave either "" to omit. +# The REQUIRED scalars render raw (no `not in ["", none]` guard), so unlike every +# optional knob they reach node.toml unchecked. `decdn_cache_size_mb: "20 GB"` +# renders `cache_size_mb = 20 GB` — invalid TOML — and also slips the cross-field +# check above, because Jinja's `| int` coerces it to 0 and the `== 0` exemption +# then short-circuits. Same shape guard the optional knobs already get. +- name: Validate the required numeric knobs are integers in range + ansible.builtin.assert: + that: + - item.value | string is match('^[0-9]+$') + - item.value | int >= item.min + - item.value | int <= item.max + quiet: true + fail_msg: >- + {{ item.name }} must be an integer in {{ item.min }}..={{ item.max }} + (got "{{ item.value }}"). It is rendered into node.toml unquoted, so a + non-numeric value produces invalid TOML and a daemon crash-loop. + loop: + - {name: decdn_bind_port, value: "{{ decdn_bind_port }}", min: 1, max: 65535} + - {name: decdn_metrics_port, value: "{{ decdn_metrics_port }}", min: 1, max: 65535} + # 0 is meaningful: it disables the admin server entirely. + - {name: decdn_admin_port, value: "{{ decdn_admin_port }}", min: 0, max: 65535} + - {name: decdn_cache_size_mb, value: "{{ decdn_cache_size_mb }}", min: 1, max: 9223372036854775807} + - {name: decdn_rate_per_mb, value: "{{ decdn_rate_per_mb }}", min: 0, max: 9223372036854775807} + loop_control: + label: "{{ item.name }}" + +# AGENTS.md hard rule 2 is specifically about this knob: service daemons bind +# loopback, and only the QUIC port gets a firewall hole. Nothing else enforced it — +# `decdn_metrics_bind: "0.0.0.0"` was a clean deploy and a publicly-bound metrics +# endpoint with only the baseline firewall behind it. +- name: Require the metrics listener to stay on loopback + ansible.builtin.assert: + that: + - decdn_metrics_bind is match('^(127\.[0-9.]+|::1|localhost)$') + fail_msg: >- + decdn_metrics_bind must be a loopback address (got "{{ decdn_metrics_bind }}"). + Metrics carry operational detail and the endpoint has no auth; exposing it is + a deliberate architecture change, not a config tweak. Front it with a reverse + proxy that terminates auth + TLS instead (AGENTS.md hard rule 2). The admin + RPC is hardcoded to 127.0.0.1 upstream and cannot be moved at all. + +# The one knob with no shape validation, and the one documented as the home for +# secrets. systemd's EnvironmentFile parser is shell-like: it applies POSIX-ish +# unquoting and SILENTLY SKIPS a line it cannot parse, so a secret containing a +# quote or backslash would simply be absent at runtime — surfacing hours later as +# an S3 403, with no_log guaranteeing nothing in the deploy output shows why. +- name: Validate decdn_extra_env keys and values + ansible.builtin.assert: + that: + - item.key is match('^[A-Za-z_][A-Za-z0-9_]*$') + - item.value | string is match('^[^\r\n]*$') + - item.key != 'DECDN_RPC_URL' + fail_msg: >- + decdn_extra_env keys must be POSIX environment-variable names and values must + contain no newline (systemd silently DROPS an unparseable EnvironmentFile + line, so the variable would be absent at runtime rather than wrong). + DECDN_RPC_URL is set from decdn_rpc_url and must not be overridden here. + Offending entry: {{ item.key }}. + loop: "{{ decdn_extra_env | default({}, true) | dict2items }}" + loop_control: + label: "{{ item.key }}" + no_log: true # values are secrets by design + # --- User & directories ------------------------------------------------------- - name: Create decdn system group ansible.builtin.group: @@ -526,7 +918,11 @@ owner: "{{ decdn_user }}" group: "{{ decdn_group }}" mode: "0600" - content: "DECDN_RPC_URL={{ decdn_rpc_url }}\n" + content: | + DECDN_RPC_URL={{ decdn_rpc_url }} + {% for k, v in (decdn_extra_env | default({}, true)) | dictsort %} + {{ k }}="{{ v | string | replace('\\', '\\\\') | replace('"', '\\"') }}" + {% endfor %} no_log: true # the RPC URL may embed an API key notify: Restart decdn-node @@ -539,6 +935,73 @@ mode: "0640" notify: Restart decdn-node +# The role's own asserts mirror upstream constraints by hand and will drift again; +# this is the authoritative check. `decdn config validate` resolves the config the +# same way `decdn-node run` does — every section is deny_unknown_fields with no +# serde aliases, so a key this role renders but the installed binary does not know +# is a startup crash-loop. Validating here turns that into a failed deploy with the +# daemon's own error message, and it neither binds ports nor dials the RPC endpoint. +# +# Runs as decdn_user because it reads the 0600 keystore password; DECDN_RPC_URL is +# supplied from the same value systemd will hand the daemon. Skipped in check mode +# (node.toml has not been written yet) — the asserts above are the check-mode gate. +# The RPC URL is sourced from the 0600 env file rather than passed via Ansible's +# `environment:`. `environment:` is implemented as a shell prefix on the module +# command (`ENV=val /usr/bin/python3 ...`), so it lands in the target host's +# process table and any local user can read the embedded API key out of `ps` for +# the duration of the task — `no_log` hides it from Ansible's output, not from +# /proc. Sourcing the file systemd already reads keeps the secret on disk at 0600. +- name: Validate the rendered node.toml with the installed binary + ansible.builtin.shell: + executable: /bin/bash + cmd: | + set -euo pipefail + set -a + . {{ decdn_env_file | quote }} + set +a + {{ decdn_cli_bin | quote }} config validate \ + --config {{ decdn_config_file | quote }} \ + --keystore-password-file {{ decdn_keystore_password_file | quote }} + become_user: "{{ decdn_user }}" + become: true + register: decdn_config_validate + changed_when: false + # Deliberately non-fatal: the next task turns a failure into an actionable + # message. `failed_when: rc != 0` here would abort the play first and leave the + # operator with a bare "non-zero return code". + failed_when: false + when: not ansible_check_mode + +# `failed_when: false` above suppresses EVERY failure verdict, including ones that +# never produce an `rc` at all — a become_user failure (Debian without `acl`), a +# privilege-escalation timeout, a missing interpreter. Defaulting a missing rc to 0 +# would treat "the gate could not run" as "the gate passed" and start the daemon on +# an unvalidated config, so an absent rc defaults to 1 and is reported distinctly. +- name: Report the config-validation failure + ansible.builtin.fail: + msg: >- + {% if decdn_config_validate.rc is not defined %} + `decdn config validate` could not be RUN + ({{ decdn_config_validate.msg | default('no rc and no msg — re-run with -vvv', true) }}). + {{ decdn_config_file }} is therefore UNVALIDATED and the daemon must not be + started on it. A common cause is that becoming the unprivileged + {{ decdn_user }} user needs the `acl` package on this host. + {% else %} + `decdn config validate` rejected the rendered {{ decdn_config_file }}: + {{ decdn_config_validate.stderr | default('', true) + or decdn_config_validate.stdout | default('', true) + or decdn_config_validate.msg | default('(no output)', true) }} + — this role renders a key the installed decdn + {{ decdn_node_version | default('(manual)', true) }} does not accept, or a + value outside its range. Config sections are deny_unknown_fields with no + aliases upstream, so re-sync roles/decdn_node/templates/node.toml.j2 against + the binary's schema. + {% endif %} + when: + - not ansible_check_mode + - decdn_config_validate is defined + - decdn_config_validate.rc | default(1) != 0 + - name: Install decdn-node systemd unit ansible.builtin.template: src: decdn-node.service.j2 @@ -572,6 +1035,15 @@ # still report `running` (see that task) — a persistent timeout here is therefore # not provably benign. Tune the window with # decdn_readiness_retries × decdn_readiness_delay (≈60s at the defaults). +# KICS reports this task under "Communication Over HTTP" (MEDIUM). It is a false +# positive and is left unsuppressed deliberately — KICS's inline ignore directives +# have no effect on this query in the pinned engine version, and a no-op directive +# that reads like a suppression is worse than none. The target is the loopback +# interface: the daemon serves metrics over plain HTTP and upstream binds that +# listener to 127.0.0.1 (the admin listener is hardcoded there), so there is no TLS +# to require. "Require the metrics listener to stay on loopback" above enforces that +# the address cannot be moved off loopback. `make security` gates on --fail-on high, +# so this does not fail the build. - name: Wait for the node metrics endpoint (advisory) ansible.builtin.uri: url: "http://127.0.0.1:{{ decdn_metrics_port }}/metrics" diff --git a/ansible/roles/decdn_node/templates/decdn-node.service.j2 b/ansible/roles/decdn_node/templates/decdn-node.service.j2 index cbe82c8..5feda3a 100644 --- a/ansible/roles/decdn_node/templates/decdn-node.service.j2 +++ b/ansible/roles/decdn_node/templates/decdn-node.service.j2 @@ -24,6 +24,16 @@ ExecStart={{ decdn_bin }} run \ --config {{ decdn_config_file }} \ --keystore-password-file {{ decdn_keystore_password_file }} +# SIGHUP re-reads the config file and applies the hot-reloadable sections only: +# observability.log_level, cache.pinned_hashes, all of [security], all of [content] +# and all of [load_shed]. Every other field logs "requires restart" and keeps its +# running value — notably payment.rate_per_mb, which upstream demoted to +# restart-required. `decdn node reload` drives the same path over the loopback +# admin RPC and shares its mutex, so the two queue rather than race. The role's +# node.toml handler still notifies a RESTART: it cannot tell which sections a given +# diff touched, so reload is an operator's deliberate call. +ExecReload=/bin/kill -HUP $MAINPID + # Graceful stop so the node can drain/flush. SIGTERM is its shutdown signal. KillSignal=SIGTERM TimeoutStopSec=30 diff --git a/ansible/roles/decdn_node/templates/node.toml.j2 b/ansible/roles/decdn_node/templates/node.toml.j2 index 1486b3c..3ecfe99 100644 --- a/ansible/roles/decdn_node/templates/node.toml.j2 +++ b/ansible/roles/decdn_node/templates/node.toml.j2 @@ -1,6 +1,15 @@ # MANAGED BY the decdn-node role — do not edit by hand. -# Field names match decdn/crates/common/src/config/types.rs. The [cache] table -# uses deny_unknown_fields upstream, so only documented keys are emitted. +# +# Field names track decdn/crates/common/src/config/types.rs. EVERY section there +# carries #[serde(deny_unknown_fields)] and the config crate defines NO serde +# aliases, so a stale or misspelled key is a hard startup failure, not a warning. +# The reference rendering is the DEFAULT_CONFIG template `decdn config init` +# writes (decdn/crates/cli/src/commands/config.rs) — upstream CI asserts it covers +# every wired field, so it is the thing to diff against when re-syncing. +# +# TOML ordering hazard: every scalar [cache] key MUST be emitted before the first +# [cache.*] sub-table header, or it silently nests into that sub-table instead. +# # Note: rpc_url is NOT here — it's the one sensitive value and is supplied via # DECDN_RPC_URL from the 0600 env file, so this file stays non-secret/diffable. @@ -12,37 +21,51 @@ region = "{{ decdn_region }}" bind_port = {{ decdn_bind_port }} {% if decdn_relay_urls is not none and decdn_relay_urls | length > 0 %} relay_urls = [{% for u in decdn_relay_urls %}"{{ u }}"{% if not loop.last %}, {% endif %}{% endfor %}] -{% elif decdn_relay_url not in ["", none] %} -relay_url = "{{ decdn_relay_url }}" {% endif %} -{% if decdn_enable_0rtt not in ["", none] %} -enable_0rtt = {{ decdn_enable_0rtt | lower }} +{% if decdn_discovery_pkarr_url not in ["", none] or (decdn_discovery_peers is not none and decdn_discovery_peers | length > 0) %} + +[network.discovery] +{% if decdn_discovery_pkarr_url not in ["", none] %} +pkarr_url = "{{ decdn_discovery_pkarr_url }}" +dns_origin = "{{ decdn_discovery_dns_origin }}" +{% endif %} +{% for node_id, peer in (decdn_discovery_peers | default({}, true)).items() %} + +[network.discovery.peers.{{ node_id }}] +{% if peer.relay_url is defined and peer.relay_url not in ["", none] %} +relay_url = "{{ peer.relay_url }}" +{% endif %} +addrs = [{% for a in peer.addrs %}"{{ a }}"{% if not loop.last %}, {% endif %}{% endfor %}] +{% endfor %} {% endif %} [blockchain] chain_id = {{ decdn_chain_id }} eth_keystore = "{{ decdn_keystore_file }}" -payment_channel_address = "{{ decdn_payment_channel_address }}" +payment_pool_address = "{{ decdn_payment_pool_address }}" capacity_bond_address = "{{ decdn_capacity_bond_address }}" slash_judge_address = "{{ decdn_slash_judge_address }}" -{% if decdn_slash_appeal_address | length > 0 %} -slash_appeal_address = "{{ decdn_slash_appeal_address }}" -{% endif %} -{% if decdn_slash_judge_from_block | int > 0 %} -slash_judge_from_block = {{ decdn_slash_judge_from_block | int }} -{% endif %} +content_blacklist_address = "{{ decdn_content_blacklist_address }}" {% if decdn_origin_assignment_address | length > 0 %} origin_assignment_address = "{{ decdn_origin_assignment_address }}" +{% endif %} +{% if decdn_publisher_registry_address | length > 0 %} publisher_registry_address = "{{ decdn_publisher_registry_address }}" -{% if decdn_origin_directory_from_block | int > 0 %} -origin_directory_from_block = {{ decdn_origin_directory_from_block | int }} {% endif %} +{% if decdn_slash_appeal_address | length > 0 %} +slash_appeal_address = "{{ decdn_slash_appeal_address }}" {% endif %} -{% if decdn_content_blacklist_address | length > 0 %} -content_blacklist_address = "{{ decdn_content_blacklist_address }}" -{% if decdn_content_blacklist_from_block | int > 0 %} -content_blacklist_from_block = {{ decdn_content_blacklist_from_block | int }} +{% if decdn_origin_directory_positive_ttl_sec not in ["", none] %} +origin_directory_positive_ttl_sec = {{ decdn_origin_directory_positive_ttl_sec | int }} +{% endif %} +{% if decdn_origin_directory_negative_ttl_sec not in ["", none] %} +origin_directory_negative_ttl_sec = {{ decdn_origin_directory_negative_ttl_sec | int }} {% endif %} +{% if decdn_origin_directory_cache_capacity not in ["", none] %} +origin_directory_cache_capacity = {{ decdn_origin_directory_cache_capacity | int }} +{% endif %} +{% if decdn_content_blacklist_poll_interval_sec not in ["", none] %} +content_blacklist_poll_interval_sec = {{ decdn_content_blacklist_poll_interval_sec | int }} {% endif %} {% if decdn_rpc_watchdog_interval_sec not in ["", none] %} rpc_watchdog_interval_sec = {{ decdn_rpc_watchdog_interval_sec | int }} @@ -50,23 +73,56 @@ rpc_watchdog_interval_sec = {{ decdn_rpc_watchdog_interval_sec | int }} {% if decdn_event_poll_interval_ms not in ["", none] %} event_poll_interval_ms = {{ decdn_event_poll_interval_ms | int }} {% endif %} -{% if decdn_content_blacklist_poll_interval_sec not in ["", none] %} -content_blacklist_poll_interval_sec = {{ decdn_content_blacklist_poll_interval_sec | int }} +{% if decdn_rate_bounds_poll_interval_sec not in ["", none] %} +rate_bounds_poll_interval_sec = {{ decdn_rate_bounds_poll_interval_sec | int }} +{% endif %} +{% if decdn_fee_shares_poll_interval_sec not in ["", none] %} +fee_shares_poll_interval_sec = {{ decdn_fee_shares_poll_interval_sec | int }} {% endif %} {% if decdn_redeem_threshold_micro_usdc not in ["", none] %} redeem_threshold_micro_usdc = {{ decdn_redeem_threshold_micro_usdc | int }} {% endif %} -{% if decdn_buyer_deposit_micro_usdc not in ["", none] %} -buyer_deposit_micro_usdc = {{ decdn_buyer_deposit_micro_usdc | int }} +{% if decdn_redeem_max_vouchers_per_tx not in ["", none] %} +redeem_max_vouchers_per_tx = {{ decdn_redeem_max_vouchers_per_tx | int }} +{% endif %} +{% if decdn_redeem_interval_secs not in ["", none] %} +redeem_interval_secs = {{ decdn_redeem_interval_secs | int }} +{% endif %} +{% if decdn_buyer_working_deposit_micro_usdc not in ["", none] %} +buyer_working_deposit_micro_usdc = {{ decdn_buyer_working_deposit_micro_usdc | int }} {% endif %} {% if decdn_buyer_max_approve not in ["", none] %} -buyer_max_approve = {{ decdn_buyer_max_approve | lower }} +buyer_max_approve = {{ decdn_buyer_max_approve | bool | lower }} +{% endif %} +{% if decdn_pool_min_remaining_deposit_micro_usdc not in ["", none] %} +pool_min_remaining_deposit_micro_usdc = {{ decdn_pool_min_remaining_deposit_micro_usdc | int }} +{% endif %} +{% if decdn_pool_floor_signer_share_bps not in ["", none] %} +pool_floor_signer_share_bps = {{ decdn_pool_floor_signer_share_bps | int }} +{% endif %} +{% if decdn_pool_floor_signer_max_windows not in ["", none] %} +pool_floor_signer_max_windows = {{ decdn_pool_floor_signer_max_windows | int }} +{% endif %} +{% if decdn_usdc_address | length > 0 %} +usdc_address = "{{ decdn_usdc_address }}" +{% endif %} +{% if decdn_swap_venue not in ["", none] %} +swap_venue = "{{ decdn_swap_venue }}" {% endif %} -{% if decdn_settlement_auto_threshold_micro_usdc not in ["", none] %} -settlement_auto_threshold_micro_usdc = {{ decdn_settlement_auto_threshold_micro_usdc | int }} +{% if decdn_swap_router_address | length > 0 %} +swap_router_address = "{{ decdn_swap_router_address }}" {% endif %} -{% if decdn_settlement_auto_by_voucher_nonce_span not in ["", none] %} -settlement_auto_by_voucher_nonce_span = {{ decdn_settlement_auto_by_voucher_nonce_span | int }} +{% if decdn_swap_quoter_address | length > 0 %} +swap_quoter_address = "{{ decdn_swap_quoter_address }}" +{% endif %} +{% if decdn_swap_fee_tier not in ["", none] %} +swap_fee_tier = {{ decdn_swap_fee_tier | int }} +{% endif %} +{% if decdn_swap_balancer_pool | length > 0 %} +swap_balancer_pool = "{{ decdn_swap_balancer_pool }}" +{% endif %} +{% if decdn_swap_pool_address | length > 0 %} +swap_pool_address = "{{ decdn_swap_pool_address }}" {% endif %} [payment] @@ -74,25 +130,29 @@ rate_per_mb = {{ decdn_rate_per_mb }} {% if decdn_delivery_floor not in ["", none] %} delivery_floor = {{ decdn_delivery_floor | int }} {% endif %} -{% if decdn_delivery_ceiling not in ["", none] %} -delivery_ceiling = {{ decdn_delivery_ceiling | int }} +{% if decdn_credit_max not in ["", none] %} +credit_max = {{ decdn_credit_max | int }} +{% endif %} +{% if decdn_credit_ramp_divisor not in ["", none] %} +credit_ramp_divisor = {{ decdn_credit_ramp_divisor | int }} {% endif %} -{% if decdn_voucher_interval_mb not in ["", none] %} -voucher_interval_mb = {{ decdn_voucher_interval_mb | int }} +{% if decdn_frame_target_bytes not in ["", none] %} +frame_target_bytes = {{ decdn_frame_target_bytes | int }} +{% endif %} +{% if decdn_voucher_commit_interval_ms not in ["", none] %} +voucher_commit_interval_ms = {{ decdn_voucher_commit_interval_ms | int }} {% endif %} +# --- [cache] -------------------------------------------------------------- +# Every scalar below MUST stay above the first [cache.*] sub-table header. [cache] cache_dir = "{{ decdn_cache_dir }}" cache_size_mb = {{ decdn_cache_size_mb }} -max_blob_size_mb = {{ decdn_max_blob_size_mb }} -{% if decdn_max_probe_holds not in ["", none] %} -max_probe_holds = {{ decdn_max_probe_holds | int }} +{% if decdn_max_blob_size_mb not in ["", none] %} +max_blob_size_mb = {{ decdn_max_blob_size_mb | int }} {% endif %} -{% if decdn_stake_lane_reserved_holds not in ["", none] %} -stake_lane_reserved_holds = {{ decdn_stake_lane_reserved_holds | int }} -{% endif %} -{% if decdn_gc_interval_sec not in ["", none] %} -gc_interval_sec = {{ decdn_gc_interval_sec | int }} +{% if decdn_max_rate_per_mb not in ["", none] %} +max_rate_per_mb = {{ decdn_max_rate_per_mb | int }} {% endif %} {% if decdn_pinned_hashes is not none and decdn_pinned_hashes | length > 0 %} pinned_hashes = [{% for h in decdn_pinned_hashes %}"{{ h }}"{% if not loop.last %}, {% endif %}{% endfor %}] @@ -100,8 +160,50 @@ pinned_hashes = [{% for h in decdn_pinned_hashes %}"{{ h }}"{% if not loop.last {% if decdn_cache_user_agent not in ["", none] %} user_agent = "{{ decdn_cache_user_agent }}" {% endif %} +{% if decdn_gc_interval_sec not in ["", none] %} +gc_interval_sec = {{ decdn_gc_interval_sec | int }} +{% endif %} +{% if decdn_fs_rescan_interval_sec not in ["", none] %} +fs_rescan_interval_sec = {{ decdn_fs_rescan_interval_sec | int }} +{% endif %} +{% if decdn_origin_probe_ttl_sec not in ["", none] %} +origin_probe_ttl_sec = {{ decdn_origin_probe_ttl_sec | int }} +{% endif %} +{% if decdn_origin_probe_negative_ttl_sec not in ["", none] %} +origin_probe_negative_ttl_sec = {{ decdn_origin_probe_negative_ttl_sec | int }} +{% endif %} +{% if decdn_origin_probe_fault_ttl_sec not in ["", none] %} +origin_probe_fault_ttl_sec = {{ decdn_origin_probe_fault_ttl_sec | int }} +{% endif %} +{% if decdn_origin_probe_timeout_ms not in ["", none] %} +origin_probe_timeout_ms = {{ decdn_origin_probe_timeout_ms | int }} +{% endif %} +{% if decdn_origin_probe_memo_capacity not in ["", none] %} +origin_probe_memo_capacity = {{ decdn_origin_probe_memo_capacity | int }} +{% endif %} +{% if decdn_eviction_high_water_pct not in ["", none] %} +eviction_high_water_pct = {{ decdn_eviction_high_water_pct | int }} +{% endif %} +{% if decdn_eviction_target_pct not in ["", none] %} +eviction_target_pct = {{ decdn_eviction_target_pct | int }} +{% endif %} +{% if decdn_eviction_per_sweep_budget not in ["", none] %} +eviction_per_sweep_budget = {{ decdn_eviction_per_sweep_budget | int }} +{% endif %} +{% if decdn_eviction_tick_secs not in ["", none] %} +eviction_tick_secs = {{ decdn_eviction_tick_secs | int }} +{% endif %} +{% if decdn_max_probe_holds not in ["", none] %} +max_probe_holds = {{ decdn_max_probe_holds | int }} +{% endif %} +{% if decdn_stake_lane_reserved_holds not in ["", none] %} +stake_lane_reserved_holds = {{ decdn_stake_lane_reserved_holds | int }} +{% endif %} {% if decdn_node_to_node_pull_through_enabled not in ["", none] %} -node_to_node_pull_through_enabled = {{ decdn_node_to_node_pull_through_enabled | lower }} +node_to_node_pull_through_enabled = {{ decdn_node_to_node_pull_through_enabled | bool | lower }} +{% endif %} +{% if decdn_relay_foreign_namespaces not in ["", none] %} +relay_foreign_namespaces = {{ decdn_relay_foreign_namespaces | bool | lower }} {% endif %} {% if decdn_node_pull_probe_fanout not in ["", none] %} node_pull_probe_fanout = {{ decdn_node_pull_probe_fanout | int }} @@ -109,42 +211,153 @@ node_pull_probe_fanout = {{ decdn_node_pull_probe_fanout | int }} {% if decdn_node_pull_timeout_sec not in ["", none] %} node_pull_timeout_sec = {{ decdn_node_pull_timeout_sec | int }} {% endif %} -{% if decdn_pull_ahead_bytes not in ["", none] %} -pull_ahead_bytes = {{ decdn_pull_ahead_bytes | int }} +{% if decdn_node_pull_stall_window_sec not in ["", none] %} +node_pull_stall_window_sec = {{ decdn_node_pull_stall_window_sec | int }} +{% endif %} +{% if decdn_node_pull_min_throughput_bps not in ["", none] %} +node_pull_min_throughput_bps = {{ decdn_node_pull_min_throughput_bps | int }} +{% endif %} +{% if decdn_eviction_policy not in ["", none] %} +eviction_policy = "{{ decdn_eviction_policy }}" +{% endif %} +{% if decdn_admission_policy not in ["", none] %} +admission_policy = "{{ decdn_admission_policy }}" +{% endif %} +{% if decdn_tinylfu_sketch_bytes not in ["", none] or decdn_tinylfu_promotion_threshold not in ["", none] or decdn_tinylfu_probation_target_pct not in ["", none] or decdn_tinylfu_aging_halflife_sec not in ["", none] %} + +[cache.tinylfu] +{% if decdn_tinylfu_sketch_bytes not in ["", none] %} +sketch_bytes = {{ decdn_tinylfu_sketch_bytes | int }} +{% endif %} +{% if decdn_tinylfu_promotion_threshold not in ["", none] %} +promotion_threshold = {{ decdn_tinylfu_promotion_threshold | int }} +{% endif %} +{% if decdn_tinylfu_probation_target_pct not in ["", none] %} +probation_target_pct = {{ decdn_tinylfu_probation_target_pct | int }} {% endif %} -{% if decdn_max_unrecouped_leech_bytes not in ["", none] %} -max_unrecouped_leech_bytes = {{ decdn_max_unrecouped_leech_bytes | int }} +{% if decdn_tinylfu_aging_halflife_sec not in ["", none] %} +aging_halflife_sec = {{ decdn_tinylfu_aging_halflife_sec | int }} {% endif %} -{% if decdn_pull_share_ratio_percent not in ["", none] %} -pull_share_ratio_percent = {{ decdn_pull_share_ratio_percent | int }} {% endif %} -{% if decdn_pull_through_require_authorized_origin not in ["", none] %} -pull_through_require_authorized_origin = {{ decdn_pull_through_require_authorized_origin | lower }} +{% if decdn_serve_economics_policy not in ["", none] or decdn_serve_economics_discount not in ["", none] or decdn_serve_economics_n_max not in ["", none] or decdn_serve_economics_warming_budget not in ["", none] or decdn_serve_economics_warming_refill not in ["", none] %} + +[cache.serve_economics] +{% if decdn_serve_economics_policy not in ["", none] %} +policy = "{{ decdn_serve_economics_policy }}" +{% endif %} +{% if decdn_serve_economics_discount not in ["", none] %} +discount = {{ decdn_serve_economics_discount | float }} +{% endif %} +{% if decdn_serve_economics_n_max not in ["", none] %} +n_max = {{ decdn_serve_economics_n_max | int }} +{% endif %} +{% if decdn_serve_economics_warming_budget not in ["", none] %} +warming_budget = {{ decdn_serve_economics_warming_budget | int }} {% endif %} -{% if decdn_cache_origin_kind | length > 0 %} +{% if decdn_serve_economics_warming_refill not in ["", none] %} +warming_refill = {{ decdn_serve_economics_warming_refill | int }} +{% endif %} +{% endif %} +{% if decdn_origin_retry_max_retries not in ["", none] or decdn_origin_retry_initial_backoff_ms not in ["", none] or decdn_origin_retry_max_backoff_ms not in ["", none] or decdn_origin_retry_jitter_ratio not in ["", none] or decdn_origin_retry_buffered_max_bytes not in ["", none] %} -[cache.origin] -kind = "{{ decdn_cache_origin_kind }}" -{% if decdn_cache_origin_kind == "http" %} -url = "{{ decdn_cache_origin_url }}" -{% if decdn_cache_origin_decompress | length > 0 %} -decompress = "{{ decdn_cache_origin_decompress }}" +[cache.origin_retry] +{% if decdn_origin_retry_max_retries not in ["", none] %} +max_retries = {{ decdn_origin_retry_max_retries | int }} +{% endif %} +{% if decdn_origin_retry_initial_backoff_ms not in ["", none] %} +initial_backoff_ms = {{ decdn_origin_retry_initial_backoff_ms | int }} {% endif %} -{% elif decdn_cache_origin_kind == "fs" %} -path = "{{ decdn_cache_origin_path }}" -{% elif decdn_cache_origin_kind == "s3" %} -bucket = "{{ decdn_cache_origin_s3_bucket }}" -region = "{{ decdn_cache_origin_s3_region }}" -{% if decdn_cache_origin_s3_endpoint_url | length > 0 %} -endpoint_url = "{{ decdn_cache_origin_s3_endpoint_url }}" +{% if decdn_origin_retry_max_backoff_ms not in ["", none] %} +max_backoff_ms = {{ decdn_origin_retry_max_backoff_ms | int }} {% endif %} -{% if decdn_cache_origin_s3_path_style | bool %} +{% if decdn_origin_retry_jitter_ratio not in ["", none] %} +jitter_ratio = {{ decdn_origin_retry_jitter_ratio | float }} +{% endif %} +{% if decdn_origin_retry_buffered_max_bytes not in ["", none] %} +buffered_max_bytes = {{ decdn_origin_retry_buffered_max_bytes | int }} +{% endif %} +{% endif %} +{% if decdn_circuit_breaker_enabled not in ["", none] or decdn_circuit_breaker_failure_threshold not in ["", none] or decdn_circuit_breaker_cooldown_ms not in ["", none] or decdn_circuit_breaker_half_open_max_calls not in ["", none] %} + +[cache.circuit_breaker] +{% if decdn_circuit_breaker_enabled not in ["", none] %} +enabled = {{ decdn_circuit_breaker_enabled | bool | lower }} +{% endif %} +{% if decdn_circuit_breaker_failure_threshold not in ["", none] %} +failure_threshold = {{ decdn_circuit_breaker_failure_threshold | int }} +{% endif %} +{% if decdn_circuit_breaker_cooldown_ms not in ["", none] %} +cooldown_ms = {{ decdn_circuit_breaker_cooldown_ms | int }} +{% endif %} +{% if decdn_circuit_breaker_half_open_max_calls not in ["", none] %} +half_open_max_calls = {{ decdn_circuit_breaker_half_open_max_calls | int }} +{% endif %} +{% endif %} +{# + Pull-through origin. `cache.origin` (single) and `cache.origins` (ordered + fallback list) are mutually exclusive upstream and an empty `origins` array is + rejected, so exactly one form is ever emitted. decdn_cache_origins wins when + set; the scalar decdn_cache_origin_* vars render the single-table form. +#} +{% set _multi = decdn_cache_origins is not none and decdn_cache_origins | length > 0 %} +{% if _multi %} +{% set _origins = decdn_cache_origins %} +{% elif decdn_cache_origin_kind | length > 0 %} +{% set _origins = [{ + "kind": decdn_cache_origin_kind, + "url": decdn_cache_origin_url, + "decompress": decdn_cache_origin_decompress, + "path": decdn_cache_origin_path, + "bucket": decdn_cache_origin_s3_bucket, + "region": decdn_cache_origin_s3_region, + "endpoint_url": decdn_cache_origin_s3_endpoint_url, + "path_style": decdn_cache_origin_s3_path_style, + "prefix": decdn_cache_origin_s3_prefix, + }] %} +{% else %} +{% set _origins = [] %} +{% endif %} +{% for o in _origins %} + +{{ '[[cache.origins]]' if _multi else '[cache.origin]' }} +kind = "{{ o.kind }}" +{% if o.kind == 'http' %} +url = "{{ o.url }}" +{% if o.decompress | default('', true) | length > 0 %} +decompress = "{{ o.decompress }}" +{% endif %} +{% elif o.kind == 'fs' %} +path = "{{ o.path }}" +{% elif o.kind == 's3' %} +bucket = "{{ o.bucket }}" +region = "{{ o.region }}" +{% if o.endpoint_url | default('', true) | length > 0 %} +endpoint_url = "{{ o.endpoint_url }}" +{% endif %} +{% if o.path_style | default(false, true) | bool %} path_style = true {% endif %} -{% if decdn_cache_origin_s3_prefix | length > 0 %} -prefix = "{{ decdn_cache_origin_s3_prefix }}" +{% if o.prefix | default('', true) | length > 0 %} +prefix = "{{ o.prefix }}" +{% endif %} +{% if o.decompress | default('', true) | length > 0 %} +decompress = "{{ o.decompress }}" {% endif %} {% endif %} +{% endfor %} +{# + S3 credentials. Only the `default-chain` source is templated: `static` would put + an access key into this 0640, diffable file. Supply static keys as + AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN in the 0600 + {{ decdn_env_file }} instead — the default chain reads exactly those. +#} +{% if not _multi and decdn_cache_origin_kind == 's3' and decdn_cache_origin_s3_use_default_chain | bool %} + +[cache.origin.credentials] +source = "default-chain" +{% if decdn_cache_origin_s3_profile not in ["", none] %} +profile = "{{ decdn_cache_origin_s3_profile }}" +{% endif %} {% endif %} [observability] @@ -156,6 +369,74 @@ admin_port = {{ decdn_admin_port }} {% if decdn_otlp_endpoint not in ["", none] %} otlp_endpoint = "{{ decdn_otlp_endpoint }}" {% endif %} -{% if decdn_region_accounting_interval_sec not in ["", none] %} -region_accounting_interval_sec = {{ decdn_region_accounting_interval_sec | int }} +{% if decdn_security_max_concurrent_handlers not in ["", none] or decdn_security_per_source_rate_per_sec not in ["", none] or decdn_security_per_source_burst not in ["", none] or decdn_security_max_tracked_sources not in ["", none] %} + +[security] +{% if decdn_security_max_concurrent_handlers not in ["", none] %} +max_concurrent_handlers = {{ decdn_security_max_concurrent_handlers | int }} +{% endif %} +{% if decdn_security_per_source_rate_per_sec not in ["", none] %} +per_source_rate_per_sec = {{ decdn_security_per_source_rate_per_sec | float }} +{% endif %} +{% if decdn_security_per_source_burst not in ["", none] %} +per_source_burst = {{ decdn_security_per_source_burst | int }} +{% endif %} +{% if decdn_security_max_tracked_sources not in ["", none] %} +max_tracked_sources = {{ decdn_security_max_tracked_sources | int }} +{% endif %} +{% endif %} +{% if decdn_load_shed_policy not in ["", none] or decdn_load_shed_egress_budget_mbps not in ["", none] or decdn_load_shed_max_concurrent_serves_high not in ["", none] or decdn_load_shed_max_concurrent_serves_low not in ["", none] or decdn_load_shed_per_client_serve_cap not in ["", none] %} + +[load_shed] +{% if decdn_load_shed_policy not in ["", none] %} +policy = "{{ decdn_load_shed_policy }}" +{% endif %} +{% if decdn_load_shed_egress_budget_mbps not in ["", none] %} +egress_budget_mbps = {{ decdn_load_shed_egress_budget_mbps | int }} +{% endif %} +{% if decdn_load_shed_max_concurrent_serves_high not in ["", none] %} +max_concurrent_serves_high = {{ decdn_load_shed_max_concurrent_serves_high | int }} +{% endif %} +{% if decdn_load_shed_max_concurrent_serves_low not in ["", none] %} +max_concurrent_serves_low = {{ decdn_load_shed_max_concurrent_serves_low | int }} +{% endif %} +{% if decdn_load_shed_per_client_serve_cap not in ["", none] %} +per_client_serve_cap = {{ decdn_load_shed_per_client_serve_cap | int }} +{% endif %} +{% endif %} +{% for _sec, _rl in [("dht", decdn_dht_rate_limit), ("probe", decdn_probe_rate_limit)] %} +{% if _rl is not none and _rl | length > 0 %} + +[{{ _sec }}.rate_limit] +{% for _k in ["per_peer_rate_per_sec", "per_ip_rate_per_sec", "global_rate_per_sec"] %} +{% if _k in _rl %} +{{ _k }} = {{ _rl[_k] | float }} +{% endif %} +{% endfor %} +{% for _k in ["per_peer_burst", "per_ip_burst", "global_burst", "max_tracked_per_ip", "max_tracked_per_peer"] %} +{% if _k in _rl %} +{{ _k }} = {{ _rl[_k] | int }} +{% endif %} +{% endfor %} +{% endif %} +{% endfor %} +{% if decdn_receipts_max_file_bytes not in ["", none] or decdn_receipts_retained_files not in ["", none] %} + +[receipts] +{% if decdn_receipts_max_file_bytes not in ["", none] %} +max_file_bytes = {{ decdn_receipts_max_file_bytes | int }} +{% endif %} +{% if decdn_receipts_retained_files not in ["", none] %} +retained_files = {{ decdn_receipts_retained_files | int }} +{% endif %} +{% endif %} +{% if (decdn_content_denied_hashes is not none and decdn_content_denied_hashes | length > 0) or (decdn_content_denied_origins is not none and decdn_content_denied_origins | length > 0) %} + +[content] +{% if decdn_content_denied_hashes is not none and decdn_content_denied_hashes | length > 0 %} +denied_hashes = [{% for h in decdn_content_denied_hashes %}"{{ h }}"{% if not loop.last %}, {% endif %}{% endfor %}] +{% endif %} +{% if decdn_content_denied_origins is not none and decdn_content_denied_origins | length > 0 %} +denied_origins = [{% for a in decdn_content_denied_origins %}"{{ a }}"{% if not loop.last %}, {% endif %}{% endfor %}] +{% endif %} {% endif %} From 1cd052bf01cc4800501d73bd0f6fd1f2dc54dbc6 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Tue, 1 Sep 2026 21:49:20 +0300 Subject: [PATCH 2/3] ci: run KICS from the pinned engine image instead of the broken action MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `kics` job has been failing on every PR even though the scan itself is clean (0 CRITICAL / 0 HIGH). The failure is in Checkmarx/kics-github-action's wrapper, not in our Ansible tree. Root cause: the action's entrypoint runs the scan, discards its exit code into `KICS_EXIT_CODE`, then unconditionally does apk add --update nodejs npm && npm ci && npm run build && node dist/index.js and the step's exit status is that last `node` invocation. That `apk add` is an *unpinned* fetch into a *digest-pinned* wolfi-base image, so the two have now drifted apart: Chainguard's current nodejs-26 needs GLIBC_2.44, the pinned base ships older, and node dies with node: /usr/lib/libm.so.6: version `GLIBC_2.44' not found (required by node) three times over (npm is a node script too). The step therefore fails no matter what the scan found. Upstream has no fix — the only commit past our pinned SHA just removes a Dependabot config — and the inputs can't disable that code path. Fix: drop the action and drive the KICS *engine* image directly via `make security`, which the repo already used for local scans. CI and local runs are now the same command, there is no Node layer, and the unpinned runtime `apk add` is gone — one digest-pinned artifact (`KICS_IMAGE`) instead of a pinned image that fetches unpinned packages. The gate is unchanged in substance: the engine's own `--fail-on high` exit code (verified: exit 0 with 0 HIGH, exit 40 when a finding meets the threshold). Also: - `-w /repo` so findings carry repo-relative paths (`ansible/...`) rather than `../../repo/ansible/...`; this is what the summary prints and what the (still-commented) SARIF code-scanning upload would need. - `--report-formats json,sarif` in the Makefile so the local run produces the same artifacts CI uploads. - A "Summarise KICS findings" step replaces the action's `enable_jobs_summary`. - Drop the now-dead Dependabot ignore for the action; refresh the supply-chain notes in ci.yml, CONTRIBUTING.md and the Makefile. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DEQihoRTxzVdi2wpPb9Apq --- .github/dependabot.yml | 6 ---- .github/workflows/ci.yml | 59 ++++++++++++++++++++++++++-------------- CONTRIBUTING.md | 20 ++++++++------ Makefile | 18 ++++++------ 4 files changed, 60 insertions(+), 43 deletions(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index bfaa5a0..50270ec 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -23,9 +23,3 @@ updates: groups: actions: patterns: ["*"] - ignore: - # Pinned to the post-remediation hardened HEAD (see ci.yml). The newest - # RELEASE tag (v2.1.20) points at an older commit that predates the April - # 2026 base-image digest-pinning, so an automated bump would DOWNGRADE - # security. Re-pin manually only after verifying a newer clean commit/tag. - - dependency-name: "Checkmarx/kics-github-action" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cd41a8c..8b1b4fa 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -96,17 +96,23 @@ jobs: path: ansible/build/decdn-node-*.tar.gz if-no-files-found: ignore - # Dedicated IaC security scan of the Ansible tree via the official KICS action. - # KICS severities are HIGH/MEDIUM/LOW/INFO (no "critical"); we gate on HIGH. + # Dedicated IaC security scan of the Ansible tree, driven straight from the + # digest-pinned KICS *engine* image by `make security` — the exact command + # developers run locally, so CI and local results cannot drift. KICS severities + # are CRITICAL/HIGH/MEDIUM/LOW/INFO; the engine's own `--fail-on high` exit + # code is the gate. # - # SUPPLY-CHAIN NOTE: this action's git tags were hijacked in the March 2026 - # TeamPCP attack (CISA KEV). It has since been remediated — tags restored to - # their legitimate pre-hijack commits and explicit hardening applied (base - # images digest-pinned, workflows SHA-pinned, StepSecurity best practices). We - # pin to the post-remediation hardened HEAD by SHA; the `v2.1.20` *tag* points - # at the older Mar-04 commit that predates the April base-image digest-pinning, - # so we deliberately do NOT use the tag (and Dependabot is told not to bump it - # — see .github/dependabot.yml). Re-verify the SHA before any change. + # SUPPLY-CHAIN NOTE: we deliberately do NOT use Checkmarx/kics-github-action. + # Its git tags were hijacked in the March 2026 TeamPCP attack (CISA KEV), and + # beyond that history its entrypoint `apk add`s nodejs/npm at *run* time inside + # its digest-pinned base image and then executes the result — an unpinned fetch + # that defeats the pinning it advertises. That fetch also breaks the action + # outright today: Chainguard's current nodejs wants a newer glibc than the + # pinned base ships, so `node dist/index.js` dies and the step exits non-zero + # no matter what the scan found. Driving the engine image ourselves drops the + # Node layer entirely and leaves one pinned artifact — `KICS_IMAGE` in the + # Makefile, pinned by Docker Hub digest (a different artifact from the + # hijacked action). See CONTRIBUTING.md. kics: needs: changes if: needs.changes.outputs.ansible == 'true' @@ -114,16 +120,29 @@ jobs: steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: KICS Ansible security scan (fail on HIGH) - # master @ 2026-05-22 "[StepSecurity] Apply security best practices (#157)" - uses: Checkmarx/kics-github-action@7117906d8779ecaf5180f34c4931a774f10d7625 - with: - path: ansible - platform_type: Ansible - exclude_paths: ansible/collections - fail_on: high - output_formats: json,sarif - output_path: kics-results - enable_jobs_summary: true + run: make security + # Replaces the action's `enable_jobs_summary`. + - name: Summarise KICS findings + if: always() + run: | + results=kics-results/results.json + { + echo "### KICS IaC scan" + echo + if [ -f "$results" ]; then + jq -r '.severity_counters + | "| CRITICAL | HIGH | MEDIUM | LOW | INFO |", + "|---|---|---|---|---|", + "| \(.CRITICAL) | \(.HIGH) | \(.MEDIUM) | \(.LOW) | \(.INFO) |"' "$results" + echo + jq -r 'if (.total_counter // 0) == 0 then "No findings." + else (.queries[] | .query_name as $q | .severity as $s + | .files[] | "- **\($s)** \($q) — `\(.file_name):\(.line)`") + end' "$results" + else + echo "No results file — the scan did not complete." + fi + } >> "$GITHUB_STEP_SUMMARY" - name: Upload KICS results if: always() uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 059c9d9..c6000d2 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,7 +23,7 @@ tree), and `markdownlint`. |--------|--------------| | `make lint` | run all pre-commit hooks on every file (the full local hygiene gate) | | `make lint-ansible` | install Galaxy collections + run `ansible-lint` (its production profile includes the Ansible security rules) | -| `make security` | KICS IaC security scan of `ansible/` (engine image; CI uses the official KICS action) | +| `make security` | KICS IaC security scan of `ansible/` (digest-pinned engine image — CI runs this same target) | `ansible-lint` is **not** a per-commit hook (it needs the collections installed). Run it on demand with `make lint-ansible`, or `pre-commit run ansible-lint --hook-stage manual`. @@ -41,14 +41,16 @@ Run it on demand with `make lint-ansible`, or `pre-commit run ansible-lint --hoo - **Third-party actions are pinned to a full commit SHA** with a version comment — a mutable tag can be re-pointed to malicious code. -- **`Checkmarx/kics-github-action`** was hijacked in the March 2026 TeamPCP attack - (CISA KEV) and has since been remediated. It is pinned to the **post-remediation - hardened HEAD** by SHA — *not* a release tag, because the newest tag (`v2.1.20`) - predates the April hardening. Dependabot is told **not** to bump it - (`.github/dependabot.yml`); re-pin manually only after verifying a newer clean - commit. The KICS engine image used locally (`make security`) is the **Docker Hub** - engine (a different artifact than the hijacked action) and is pinned by a - digest verified against Docker Hub — currently `v2.1.20`. +- **`Checkmarx/kics-github-action` is deliberately not used.** Its git tags were + hijacked in the March 2026 TeamPCP attack (CISA KEV), and even post-remediation + its entrypoint `apk add`s `nodejs`/`npm` at *run* time inside its digest-pinned + base image and executes the result — an unpinned fetch that defeats the pinning. + (That fetch also breaks it outright today: Chainguard's current nodejs needs a + newer glibc than the pinned base ships, so the action's Node reporter dies and + the step fails regardless of findings.) CI instead runs `make security`, which + drives the **Docker Hub** KICS engine image — a different artifact from the + hijacked action — pinned by a digest verified against Docker Hub, currently + `v2.1.20`. The engine's `--fail-on high` exit code is the gate. - **Dependabot** (`.github/dependabot.yml`) bumps the other action SHAs weekly. - **Bump manually** (Dependabot can't): the `KICS_IMAGE` digest in the `Makefile`, and the pre-commit hook revs via `pre-commit autoupdate`. diff --git a/Makefile b/Makefile index ccc8e5f..1466e09 100644 --- a/Makefile +++ b/Makefile @@ -3,11 +3,11 @@ .PHONY: help hooks lint lint-ansible security molecule galaxy-build galaxy-check SHELL := /bin/bash -# Local KICS runs use the engine image pinned by digest. CI runs the official -# Checkmarx/kics-github-action instead (a GitHub Action can't run outside CI). -# Pinning by digest means a re-pointed tag can't ship malicious code (cf. the -# March 2026 KICS action compromise); the digest is verified against Docker Hub -# on each bump. v2.1.20 (March 2026). +# KICS runs straight from the engine image, pinned by digest. This target IS the +# CI security gate (.github/workflows/ci.yml calls it), so local and CI runs are +# byte-identical. Pinning by digest means a re-pointed tag can't ship malicious +# code (cf. the March 2026 KICS action compromise); the digest is verified +# against Docker Hub on each bump. v2.1.20 (March 2026). KICS_IMAGE := checkmarx/kics:v2.1.20-alpine@sha256:990ae994fbbe59760c8e4f7e89b1193a39a0c2968909058ec29335cb6d80efc1 help: ## list targets @@ -24,12 +24,14 @@ lint-ansible: ## full ansible-lint locally (installs collections first) $(MAKE) -C ansible deps $(MAKE) -C ansible lint -security: ## KICS IaC security scan of ansible/ (CI runs the official action) +# -w /repo so findings carry repo-relative paths (not ../../repo/...), which is +# what the CI job summary prints and what SARIF code-scanning uploads need. +security: ## KICS IaC security scan of ansible/ (pinned engine image; CI runs this) mkdir -p kics-results - docker run --rm --user $(shell id -u):$(shell id -g) -v "$(CURDIR):/repo" $(KICS_IMAGE) \ + docker run --rm --user $(shell id -u):$(shell id -g) -w /repo -v "$(CURDIR):/repo" $(KICS_IMAGE) \ scan --path /repo/ansible --type Ansible \ --exclude-paths /repo/ansible/collections \ - --report-formats json --output-path /repo/kics-results \ + --report-formats json,sarif --output-path /repo/kics-results \ --no-progress --fail-on high molecule: ## containerised converge/verify of the decdn_node role (needs Docker) From 62c201105215a1c0cec4f5ca1e988aeb3c558e1f Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Tue, 1 Sep 2026 22:02:51 +0300 Subject: [PATCH 3/3] fix(decdn_node): exact SHA256SUMS matching and reachable resolve-only discovery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects found in review of this PR's own changes. 1. install.yml matched manifest entries with `grep -F " $f"`, a substring match on the filename. Verified against a scratch manifest: - a sibling entry that merely STARTS with $f (e.g. "$f.sig") makes the count 2, so a perfectly good manifest aborts with the misleading "covers it twice"; - the `hash *name` binary-mode form (`sha256sum -b`) matches nothing at all, count 0, "does not cover this archive". Now matches the filename FIELD exactly via awk string equality, skipping the hash and the two-character " " / " *" separator. Tested across six manifest shapes (normal, .sig sibling, binary mode, missing tarball, genuine duplicate, wrong hash) — each now lands on the right verdict. Note this was never a verification hole: SHA256SUMS is GPG-verified before this step, and every wrong path already failed closed (the count guard, or sha256sum --check on an absent file). The defect is availability — a legitimate upstream manifest could block the install. 2. node.toml.j2 nested `dns_origin` inside the `pkarr_url` branch, so the resolve-only mode that defaults/main.yml and the assert in tasks/main.yml both call legal ("dns_origin alone is fine — that is a resolve-only node") rendered to nothing: no [network.discovery] table, and the daemon silently fell back to n0 discovery. dns_origin was also dropped when combined with static peers. Both keys are now emitted independently, and the table is emitted when any of the three mechanisms is set. Guarded by a new "resolve-only discovery variant" play in the schema scenario, following the multi-origin variant's pattern: a separate render to its own path, plus explicit presence/absence assertions — the schema checker alone cannot see a key that was never emitted. Verified red/green: with the old template the scenario fails on "Assert dns_origin renders without pkarr_url"; with the fix, schema passes end to end. The role README said the pair must be "set together or not at all", which contradicted the role's own assert. Corrected. No CHANGELOG entry: both defects were introduced in this PR's unreleased 0.1.0 content and never shipped. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DEQihoRTxzVdi2wpPb9Apq --- ansible/molecule/schema/converge.yml | 29 +++++++++++++++++++ ansible/molecule/schema/verify.yml | 28 ++++++++++++++++++ ansible/roles/decdn_node/README.md | 6 ++-- ansible/roles/decdn_node/tasks/install.yml | 11 +++++-- .../roles/decdn_node/templates/node.toml.j2 | 8 ++++- 5 files changed, 77 insertions(+), 5 deletions(-) diff --git a/ansible/molecule/schema/converge.yml b/ansible/molecule/schema/converge.yml index b766f32..4ab2754 100644 --- a/ansible/molecule/schema/converge.yml +++ b/ansible/molecule/schema/converge.yml @@ -206,3 +206,32 @@ endpoint_url: "https://acct.b2.example.invalid", prefix: "blobs/"} roles: - role: decdn_node + +# Resolve-only discovery: dns_origin WITHOUT pkarr_url. The role's own assert calls +# this legal (pkarr publishes, dns resolves; a node may resolve without publishing), +# but the maximal play above sets both, so the dns-only path is invisible to it. +# It rendered to nothing until this guard existed — dns_origin used to live inside +# the pkarr branch, so the documented resolve-only mode silently produced no +# [network.discovery] table at all. Separate render, separate file. +- name: Converge (resolve-only discovery variant) + hosts: all + become: true + vars: + stub_bin: "{{ lookup('ansible.builtin.env', 'MOLECULE_SCENARIO_DIRECTORY') }}/../default/files/decdn-node-stub" + decdn_node_install_method: manual + decdn_node_manual_bin_src: "{{ stub_bin }}" + decdn_cli_manual_bin_src: "{{ stub_bin }}" + decdn_node_version: "0.0.0-molecule-stub" + decdn_config_file: /etc/decdn/node-resolve-only.toml + decdn_rpc_url: "https://rpc.example.invalid/" + decdn_chain_id: 421614 + decdn_region: "DE" + decdn_payment_pool_address: "0x1111111111111111111111111111111111111111" + decdn_capacity_bond_address: "0x2222222222222222222222222222222222222222" + decdn_slash_judge_address: "0x3333333333333333333333333333333333333333" + decdn_content_blacklist_address: "0x4444444444444444444444444444444444444444" + # The point of this play: dns_origin set, pkarr_url deliberately left empty. + decdn_discovery_pkarr_url: "" + decdn_discovery_dns_origin: "resolve-only.example.invalid" + roles: + - role: decdn_node diff --git a/ansible/molecule/schema/verify.yml b/ansible/molecule/schema/verify.yml index 7c8cbe1..99841d5 100644 --- a/ansible/molecule/schema/verify.yml +++ b/ansible/molecule/schema/verify.yml @@ -116,6 +116,34 @@ cmd: python3 /root/molecule-assert-schema.py /etc/decdn/node-origins.toml changed_when: false + - name: Assert the resolve-only render emits no key outside the schema + ansible.builtin.command: + cmd: python3 /root/molecule-assert-schema.py /etc/decdn/node-resolve-only.toml + changed_when: false + + # The schema checker above only proves every emitted key is LEGAL. It cannot + # see a key that was never emitted, which is exactly how the resolve-only mode + # broke: dns_origin sat inside the pkarr_url branch, so setting dns_origin + # alone rendered no [network.discovery] table and the daemon silently fell + # back to n0 discovery. Assert presence and absence explicitly. + - name: Read the resolve-only render + ansible.builtin.slurp: + src: /etc/decdn/node-resolve-only.toml + register: decdn_resolve_only_toml + + - name: Assert dns_origin renders without pkarr_url + vars: + rendered: "{{ decdn_resolve_only_toml.content | b64decode }}" + ansible.builtin.assert: + that: + - "'[network.discovery]' in rendered" + - "'dns_origin = \"resolve-only.example.invalid\"' in rendered" + - "'pkarr_url' not in rendered" + fail_msg: >- + The resolve-only discovery render is wrong. dns_origin alone is legal + (the role's own assert says so) and must produce a [network.discovery] + table carrying dns_origin and no pkarr_url. + # A path check cannot catch this one: `rpc_url` is a legitimate [blockchain] # field, so a template regression that emitted it would be a VALID path and # sail through the checker above. node.toml is 0640 and diffable by design diff --git a/ansible/roles/decdn_node/README.md b/ansible/roles/decdn_node/README.md index 4d9ce60..be2631d 100644 --- a/ansible/roles/decdn_node/README.md +++ b/ansible/roles/decdn_node/README.md @@ -210,8 +210,10 @@ asserts miss. See `defaults/main.yml` for every knob's upstream default, unit an and the `decdn_origin_directory_*` cache knobs. - **Network** — `decdn_relay_urls` (list; the singular `relay_url` config key no longer exists). Operator-run address discovery (#818) via - `decdn_discovery_pkarr_url` + `decdn_discovery_dns_origin` (**set together or not - at all**) and/or `decdn_discovery_peers` (a map of 64-char lowercase-hex NodeId to + `decdn_discovery_pkarr_url` (publishes this node's record — **requires** + `decdn_discovery_dns_origin`, which resolves peers) and/or `decdn_discovery_dns_origin` + **on its own**, a resolve-only node that never publishes; and/or + `decdn_discovery_peers` (a map of 64-char lowercase-hex NodeId to `{relay_url, addrs}`). Setting either mechanism drops the n0 discovery leg. - **Abuse limits + load shedding** — the `decdn_security_*` family, `decdn_load_shed_*` (`policy` is `resource-pressure`|`always-admit`; the low-water diff --git a/ansible/roles/decdn_node/tasks/install.yml b/ansible/roles/decdn_node/tasks/install.yml index 2802d14..8feb289 100644 --- a/ansible/roles/decdn_node/tasks/install.yml +++ b/ansible/roles/decdn_node/tasks/install.yml @@ -192,13 +192,20 @@ set -euo pipefail {% for archive in decdn_release_tarballs %} f={{ (archive ~ '-' ~ decdn_node_version ~ '-' ~ decdn_node_target ~ '.tar.gz') | quote }} - n=$(grep -cF -- " $f" SHA256SUMS || true) + # Match the filename FIELD exactly, not a substring. `grep -F " $f"` + # collides with any sibling entry that merely STARTS with $f (a + # "$f.sig"/"$f.sha256" line makes the count 2 and aborts a perfectly + # good manifest) and misses the `hash *name` binary-mode form outright + # (count 0). index($0," ")+2 skips the hash and the two-character + # " " / " *" separator, leaving the name — spaces in it included. + entry=$(awk -v f="$f" 'substr($0, index($0, " ") + 2) == f' SHA256SUMS) + n=$(printf '%s\n' "$entry" | grep -c . || true) if [ "$n" -ne 1 ]; then echo "SHA256SUMS has $n entries for $f (expected exactly 1)." >&2 echo "The manifest does not cover this archive, or covers it twice." >&2 exit 1 fi - grep -F -- " $f" SHA256SUMS | sha256sum --check --strict - + printf '%s\n' "$entry" | sha256sum --check --strict - {% endfor %} register: decdn_sum_check changed_when: false diff --git a/ansible/roles/decdn_node/templates/node.toml.j2 b/ansible/roles/decdn_node/templates/node.toml.j2 index 3ecfe99..b0715f5 100644 --- a/ansible/roles/decdn_node/templates/node.toml.j2 +++ b/ansible/roles/decdn_node/templates/node.toml.j2 @@ -22,11 +22,17 @@ bind_port = {{ decdn_bind_port }} {% if decdn_relay_urls is not none and decdn_relay_urls | length > 0 %} relay_urls = [{% for u in decdn_relay_urls %}"{{ u }}"{% if not loop.last %}, {% endif %}{% endfor %}] {% endif %} -{% if decdn_discovery_pkarr_url not in ["", none] or (decdn_discovery_peers is not none and decdn_discovery_peers | length > 0) %} +{% if decdn_discovery_pkarr_url not in ["", none] or decdn_discovery_dns_origin not in ["", none] or (decdn_discovery_peers is not none and decdn_discovery_peers | length > 0) %} [network.discovery] +{# pkarr_url PUBLISHES, dns_origin RESOLVES. They are emitted independently: the + assert in tasks/main.yml requires dns_origin whenever pkarr_url is set, but + dns_origin ALONE is legal (a resolve-only node), so it must not be nested in + the pkarr branch or that mode renders to nothing. #} {% if decdn_discovery_pkarr_url not in ["", none] %} pkarr_url = "{{ decdn_discovery_pkarr_url }}" +{% endif %} +{% if decdn_discovery_dns_origin not in ["", none] %} dns_origin = "{{ decdn_discovery_dns_origin }}" {% endif %} {% for node_id, peer in (decdn_discovery_peers | default({}, true)).items() %}