From da83303e97e986f3208a82d2570f6280ce83651a Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 3 Sep 2026 03:30:15 +0300 Subject: [PATCH 1/3] feat(decdn_node): allow a host-provisioned /etc/decdn/decdn.env (#39) AGENTS.md 1 says secrets are generated on -- or operator-provisioned to -- the target host. The host-generated wallet material honoured that; the one operator-supplied secret did not. decdn_rpc_url (may embed a provider API key) and decdn_extra_env (AWS credentials behind an S3 origin) sat in plaintext in a git-ignored host_vars//secret.yml on every operator's control machine and were pushed over SSH on each converge, so one compromised laptop yielded the whole fleet's RPC keys. The env file may now be written by the operator on the target instead. The path is inferred from host state: * decdn_rpc_url set -> the role authors the file from inventory, as before * decdn_rpc_url empty -> the operator owns the file; the role never reads its contents back, enforcing only 0600 decdn:decdn and grepping for a DECDN_RPC_URL= line with a real value * neither -> fail loud, with the provisioning commands decdn_extra_env against a host-provisioned file is rejected rather than silently dropped: the file is authored all-or-nothing, and a dropped AWS key surfaces hours later as an S3 403 behind a green deploy. Because stat.exists cannot tell an operator's file from one the role wrote last week, decdn_env_checksum_file (/etc/decdn/.decdn.env.sha256, 0600 root) records the hash after each successful converge and serves as a provenance marker. It drives three behaviours: * an out-of-band edit restarts the unit on the next converge. The record is written after the daemon is started, not next to the comparison -- the handler does not flush until after the config-validate gate, so a record written earlier would say "already handled" after any aborted run and permanently burn the one signal that the secret had been rotated * an empty decdn_rpc_url against a file the role itself wrote fails loud. A missing secret.yml (fresh clone, CI runner, undecrypted vault) is not the same as handing the file to the host, and adopting it would deploy green while keeping a stale endpoint * an inventory decdn_rpc_url against a file someone else wrote fails loud rather than discarding it; decdn_env_overwrite_host_file: true confirms. With no record (a pre-#39 host) provenance is unknowable, so the role warns and starts tracking -- and that first converge deliberately does not restart The env file is also required to be a regular file. The role chowns and chmods that path as root inside a decdn-owned 0750 directory, so following a symlink would apply 0600 decdn ownership to whatever it points at. config validate now exits 3 when it cannot source the env file, so a quoting typo in decdn.env no longer reports as a node.toml schema mismatch. Tests: new host-env molecule scenario (host-provisioned file, S3 default-chain origin, no decdn_rpc_url) asserting the file is byte-identical to what the operator wrote, locked to 0600 decdn:decdn, that node.toml leaks no env value, and -- via a side_effect rotation and a second converge -- that the daemon is actually restarted. idempotence pins the converse. Three negative cases in the validation scenario (no endpoint anywhere, extra_env against a host file, a host file with no usable DECDN_RPC_URL). default keeps the inventory path and now also covers the decdn_extra_env accept branch and its quote/backslash escaping, the sidecar's hash, and EnvironmentFile= in the unit. build_ignore is dropped from galaxy.yml: "*.example" excluded nothing real (build.sh stages only roles/ plus the overlay) while silently dropping roles/decdn_node/files/decdn.env.example, which external operators need. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WaE3ZP2xQdjd9juzRYKLwi --- .github/workflows/molecule.yml | 5 +- AGENTS.md | 6 +- ansible/README.md | 39 +- ansible/galaxy/CHANGELOG.md | 22 ++ ansible/galaxy/galaxy.yml | 8 +- ansible/inventory/group_vars/decdn_nodes.yml | 5 +- .../inventory/host_vars/decdn-node-1/main.yml | 11 +- .../host_vars/decdn-node-1/secret.yml.example | 31 +- ansible/molecule/default/converge.yml | 9 + ansible/molecule/default/verify.yml | 61 +++- ansible/molecule/host-env/converge.yml | 33 ++ ansible/molecule/host-env/molecule.yml | 51 +++ ansible/molecule/host-env/prepare.yml | 56 +++ ansible/molecule/host-env/side_effect.yml | 52 +++ ansible/molecule/host-env/verify.yml | 182 +++++++++ ansible/molecule/validation/converge.yml | 112 +++++- ansible/roles/decdn_node/README.md | 105 +++++- ansible/roles/decdn_node/defaults/main.yml | 39 +- .../roles/decdn_node/files/decdn.env.example | 41 +++ ansible/roles/decdn_node/tasks/main.yml | 344 +++++++++++++++++- .../roles/decdn_node/templates/node.toml.j2 | 4 +- 21 files changed, 1152 insertions(+), 64 deletions(-) create mode 100644 ansible/molecule/host-env/converge.yml create mode 100644 ansible/molecule/host-env/molecule.yml create mode 100644 ansible/molecule/host-env/prepare.yml create mode 100644 ansible/molecule/host-env/side_effect.yml create mode 100644 ansible/molecule/host-env/verify.yml create mode 100644 ansible/roles/decdn_node/files/decdn.env.example diff --git a/.github/workflows/molecule.yml b/.github/workflows/molecule.yml index ad06337..b0b0a31 100644 --- a/.github/workflows/molecule.yml +++ b/.github/workflows/molecule.yml @@ -40,6 +40,7 @@ jobs: # (operator-provisioned keystore + rendered-value assertions), schema # (config key-set drift against the upstream field list), validation # (bad knobs must be rejected by the role's own asserts), generate-keystore - # (opt-in host-side wallet generation), and slow-readiness (advisory - # /metrics probe timeout). + # (opt-in host-side wallet generation), host-env (host-provisioned + # /etc/decdn/decdn.env — no secret on the control machine), and + # slow-readiness (advisory /metrics probe timeout). run: molecule test --all diff --git a/AGENTS.md b/AGENTS.md index e37bd97..a90c3d1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -51,8 +51,10 @@ ansible/ # the deployment project (DevSec-hardened, lean roles) tarball — verified against the release's GPG-signed `SHA256SUMS` — or, while upstream has no release tag cut (the current default), from locally-built binaries; under a hardened systemd unit; public QUIC udp/4433, loopback metrics/admin, operator-provisioned eth - keystore, required chain knobs (no baked protocol facts — sourced from ADRs), over a - shared DevSec-hardened `baseline`. See `ansible/README.md`. (On-chain node + keystore, operator-provisionable secret env file, and required chain knobs (no baked + protocol facts — sourced from ADRs), over a shared DevSec-hardened `baseline`. + Leaving `decdn_rpc_url` empty means the operator wrote `0600 /etc/decdn/decdn.env` + on the host and the role only gates on it, so no secret transits the control machine. See `ansible/README.md`. (On-chain node stake/registration, ADR 019 Phase 2, is a manual operator step, driven by `decdn setup`.) **Config-schema coupling.** `roles/decdn_node/templates/node.toml.j2` renders against diff --git a/ansible/README.md b/ansible/README.md index 0d95176..94dc86d 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -25,9 +25,11 @@ baseline host hardening — DevSec os/ssh, nftables default-deny inbound, operators must not install or enable ufw, which would replace the nftables ruleset and drop QUIC/SSH. - **No secrets in the repo.** The node's eth keystore is **operator-provisioned** and never - generated here; its `rpc_url` (which may embed an API key) lives in a git-ignored - `host_vars//secret.yml` (the rest of `host_vars` is committed, non-secret config) and - is rendered to a `0600` config. + generated here; its `rpc_url` (which may embed an API key) is provisioned the same way — + a `0600 /etc/decdn/decdn.env` written on the target host, which the role gates on but + never reads back — or, if you prefer, carried in a git-ignored + `host_vars//secret.yml` (the rest of `host_vars` is committed, non-secret config) + and rendered to that same `0600` file. - **Host hardening via DevSec** (`os_hardening` + `ssh_hardening`): key-only SSH, no root login, kernel/sysctl/PAM hardening — applied last, after the admin key is in place. baseline overrides a few `os_hardening` sysctls so hardening can't sever node @@ -54,11 +56,29 @@ make deps # vendor pinned collections into ./ cp inventory/hosts.yml.example inventory/hosts.yml $EDITOR inventory/hosts.yml # set hosts for decdn_nodes $EDITOR inventory/group_vars/all.yml # optional: override admin user/keys, allowlists -# Per-node RPC secret (the rest of host_vars//main.yml is committed config): +``` + +The per-node **secret** (`decdn_rpc_url`, which may embed a provider API key, plus any +environment-borne secrets like AWS keys for an S3 origin) has two homes — pick one. +**Preferred:** provision a `0600 /etc/decdn/decdn.env` on the target host and leave +`decdn_rpc_url` empty, so nothing sensitive is stored on or transits this machine. The role +then leaves that file's contents alone, enforcing only `0600 decdn:decdn`. Full commands and +the file format are in +[`roles/decdn_node/README.md` § Secrets](roles/decdn_node/README.md#secrets) — the single +canonical copy; don't duplicate them here. + +**Or** carry it in inventory, and the role authors `decdn.env` from those values on every run: + +```bash cp inventory/host_vars/decdn-node-1/secret.yml.example inventory/host_vars/decdn-node-1/secret.yml $EDITOR inventory/host_vars/decdn-node-1/secret.yml # set decdn_rpc_url ``` +The role fails loud when neither exists, and — once it has recorded a checksum for the file — +when the two would collide: an inventory value that would discard a host-side edit, or an +empty `decdn_rpc_url` that turns out to mean "`secret.yml` went missing" rather than "the host +owns this file". + Your `hosts.yml` is no longer force-ignored — commit it in your fork if you want, or keep it local. @@ -86,8 +106,10 @@ unless you listed one). `decdn_node_manual_bin_src` / `decdn_cli_manual_bin_src`). 2. Per-node config in `inventory/host_vars//main.yml` (committed) — the install method + binary sources, the **four** required contract addresses, - `decdn_region`, cache origin, … — plus the one secret, `decdn_rpc_url`, in a sibling - git-ignored `secret.yml` (copy the shipped `host_vars/decdn-node-1/secret.yml.example`). + `decdn_region`, cache origin, … — plus the one secret, `decdn_rpc_url`, either + provisioned as `/etc/decdn/decdn.env` on the host (preferred — see + [Setup](#setup)) or in a sibling git-ignored `secret.yml` (copy the shipped + `host_vars/decdn-node-1/secret.yml.example`). The committed `main.yml` already carries the current Arbitrum Sepolia addresses; edit `decdn_region`, the binary paths and the cache origin for your node. Contract addresses are protocol facts — copy them from the upstream deployment manifest @@ -188,7 +210,7 @@ a non-mutating dry run. Defaults live in each role (`roles/*/defaults/main.yml`); override in `group_vars` (shared) or `host_vars` (per node; `main.yml` committed config, `secret.yml` git-ignored for -the RPC URL). Highlights: +the RPC URL when it is not provisioned on the host instead). Highlights: | Var | Default | Notes | |-----|---------|-------| @@ -198,8 +220,9 @@ the RPC URL). Highlights: | `baseline_preserve_ipv6_autoconf` | `true` | Keep IPv6 RA/autoconf under hardening; set `false` for static-IPv6 hosts. | | `baseline_rp_filter_loose` | `false` | `true` loosens reverse-path filtering (`rp_filter=2`) for multi-homed nodes. | | `decdn_node_version` | `""` | **required**; a `v` release must exist. | -| `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node — `rpc_url` in `host_vars//secret.yml`, addresses in `main.yml`; sourced from an ADR/deployment. | +| `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node — `rpc_url` from a host-provisioned `0600 /etc/decdn/decdn.env` (preferred) *or* `host_vars//secret.yml`, addresses in `main.yml`; sourced from an ADR/deployment. | | `decdn_region` / `decdn_bind_port` / `decdn_rate_per_mb` | `""` / `4433` / `10` | node identity, QUIC port, USDC base units/MB. | +| `decdn_env_checksum_file` / `decdn_env_overwrite_host_file` | `/etc/decdn/.decdn.env.sha256` / `false` | Provenance record for the secret env file (`0600 root`), and the opt-in that lets an inventory `decdn_rpc_url` overwrite a host-edited one. | --- diff --git a/ansible/galaxy/CHANGELOG.md b/ansible/galaxy/CHANGELOG.md index e981288..14c8585 100644 --- a/ansible/galaxy/CHANGELOG.md +++ b/ansible/galaxy/CHANGELOG.md @@ -31,6 +31,28 @@ Not yet published to Galaxy (pre-1.0; the published shape may still change). - `decdn_extra_env` — extra `KEY: value` pairs appended to the `0600` env file. The supported home for environment-borne secrets, notably the AWS credentials behind an S3 cache origin (which are deliberately never written to the `0640` `node.toml`). +- **Host-provisioned secrets.** `decdn_rpc_url` may now be left empty when the + operator has written a `0600 /etc/decdn/decdn.env` on the target host: the role + then leaves that file's *content* alone (it never reads it back to the control + machine), enforcing only `0600 decdn:decdn` and checking that a non-empty + `DECDN_RPC_URL=` line is present. Setting `decdn_rpc_url` keeps the previous + behaviour and overwrites the host file from inventory. The role fails loud, with + the provisioning commands, when neither is present — and rejects + `decdn_extra_env` against a host-provisioned file rather than silently dropping + it. A template for the host file ships at + `roles/decdn_node/files/decdn.env.example`. +- `decdn_env_checksum_file` (default `/etc/decdn/.decdn.env.sha256`, `0600 root`) — + a role-managed record of the env file's hash, written on both authoring paths + after the daemon is running. It restarts the unit when an **out-of-band** edit of + a host-provisioned `decdn.env` is detected (instead of leaving the daemon on stale + values behind a green deploy), and doubles as the file's provenance marker: an + empty `decdn_rpc_url` against a file the role itself wrote fails loud (a missing + `secret.yml` is not the same as handing the file to the host), and an inventory + `decdn_rpc_url` against a file someone else wrote fails loud rather than + discarding it — override with `decdn_env_overwrite_host_file: true`. + **Upgrade note:** hosts deployed before this have no record, so the first + converge seeds one. That first run deliberately does *not* restart the daemon, + and the role warns rather than fails if it rewrites an untracked file. - `ExecReload` on the unit plus a `Reload decdn-node` handler, for the five hot-reloadable config sections. - Full config-schema parity with `decdn/decdn` @ `d306cc5c`: `[network.discovery]`, diff --git a/ansible/galaxy/galaxy.yml b/ansible/galaxy/galaxy.yml index 97d1ef9..be01a3b 100644 --- a/ansible/galaxy/galaxy.yml +++ b/ansible/galaxy/galaxy.yml @@ -38,5 +38,9 @@ repository: https://github.com/decdn/devops documentation: https://github.com/decdn/devops/tree/main/ansible homepage: https://decdn.org issues: https://github.com/decdn/devops/issues -build_ignore: - - "*.example" +# No build_ignore. It used to carry "*.example" as a guard against shipping an +# inventory secret template, but build.sh stages ONLY roles/ + this overlay — the +# inventory never enters the tree, so the pattern excluded nothing real while +# silently dropping roles/decdn_node/files/decdn.env.example, which external +# operators need (it is the template for the host-provisioned 0600 env file). +# Add a NARROW, path-anchored pattern here if something must ever be excluded. diff --git a/ansible/inventory/group_vars/decdn_nodes.yml b/ansible/inventory/group_vars/decdn_nodes.yml index 9c7f791..ea63981 100644 --- a/ansible/inventory/group_vars/decdn_nodes.yml +++ b/ansible/inventory/group_vars/decdn_nodes.yml @@ -1,7 +1,8 @@ --- # Shared config for public deCDN node hosts (playbooks/site.yml). -# Per-node values live in host_vars//: addresses/region/version in main.yml, -# the sensitive rpc_url in a git-ignored secret.yml. +# Per-node values live in host_vars//: addresses/region/version in main.yml. +# The sensitive rpc_url is provisioned as /etc/decdn/decdn.env on the host, or (if +# you would rather the role authored that file) in a git-ignored secret.yml. # Open the node's public QUIC listener in the baseline firewall. Metrics (9090) # and admin RPC (9191) stay loopback and get NO hole. diff --git a/ansible/inventory/host_vars/decdn-node-1/main.yml b/ansible/inventory/host_vars/decdn-node-1/main.yml index 2993bb6..4558c86 100644 --- a/ansible/inventory/host_vars/decdn-node-1/main.yml +++ b/ansible/inventory/host_vars/decdn-node-1/main.yml @@ -1,9 +1,11 @@ --- # Per-node deployment values for decdn-node-1 (COMMITTED — non-secret config only). # Ansible merges every *.yml in this directory for host decdn-node-1. The sensitive -# values (decdn_rpc_url, and any decdn_extra_env secrets) live in the sibling -# secret.yml (git-ignored) — copy secret.yml.example to secret.yml and fill it in. -# The node refuses to start until both this file and secret.yml are set. +# values (decdn_rpc_url, and any environment-borne secrets) live OFF this file: +# either in a 0600 /etc/decdn/decdn.env provisioned on the target host (preferred — +# nothing sensitive on the control machine), or in the sibling git-ignored +# secret.yml (copy secret.yml.example). The role refuses to start the node until +# this file is set and ONE of those two secret sources exists. # # CONTRACT ADDRESSES ARE NOT INVENTED HERE. They are a verbatim copy of the # upstream deployment manifest decdn/contracts/deployments/421614.json (mirrored @@ -34,7 +36,8 @@ decdn_node_manual_bin_src: "" # MUST-EDIT — control-machine path to decdn-no decdn_cli_manual_bin_src: "" # MUST-EDIT — control-machine path to decdn # --- Chain (Arbitrum Sepolia, chain 421614) ----------------------------------- -# The RPC endpoint (decdn_rpc_url) is the one sensitive value — set it in secret.yml. +# The RPC endpoint is the one sensitive value — provision it as DECDN_RPC_URL in +# /etc/decdn/decdn.env on the host, or set decdn_rpc_url in secret.yml. decdn_chain_id: 421614 # Arbitrum Sepolia # Required contracts. All four are mandatory upstream — a node will not resolve its # config without them. diff --git a/ansible/inventory/host_vars/decdn-node-1/secret.yml.example b/ansible/inventory/host_vars/decdn-node-1/secret.yml.example index 86b2d28..edf08b0 100644 --- a/ansible/inventory/host_vars/decdn-node-1/secret.yml.example +++ b/ansible/inventory/host_vars/decdn-node-1/secret.yml.example @@ -1,17 +1,28 @@ --- -# Per-node SECRETS for decdn-node-1. Copy to secret.yml (same directory, -# git-ignored) and fill in. Ansible merges it with main.yml for this host. These -# are the ONLY per-node values kept out of git — everything else lives in the -# committed main.yml. +# Per-node SECRETS for decdn-node-1 — the INVENTORY path, which is optional. # -# MUST-EDIT — SENSITIVE (may embed an API key). The public endpoint below works -# for light use; run your own or use a provider for production reliability. +# You only need this file if you want the role to author /etc/decdn/decdn.env for +# you. The preferred posture is the other way round: provision that 0600 file on +# the target host (see roles/decdn_node/files/decdn.env.example) and skip this +# file entirely — the secret then never sits in plaintext on your control machine +# and never transits it on a converge. The role fails loud if NEITHER exists. +# +# If you do use it: copy to secret.yml (same directory, git-ignored) and fill in. +# Ansible merges it with main.yml for this host. These are the ONLY per-node +# values kept out of git — everything else lives in the committed main.yml. +# +# SENSITIVE (may embed an API key). Setting this makes the role REWRITE +# /etc/decdn/decdn.env from these values on every run, discarding any lines added +# on the host. The public endpoint below works for light use; run your own or use +# a provider for production reliability. decdn_rpc_url: "https://sepolia-rollup.arbitrum.io/rpc" -# Optional. Extra KEY: value pairs appended to the 0600 /etc/decdn/decdn.env that -# systemd hands the daemon. This is where environment-borne secrets belong — most -# notably the AWS credentials backing an S3 cache origin, which the role -# deliberately does NOT write into node.toml (that file is 0640 and diffable): +# Optional, and only meaningful alongside decdn_rpc_url above (the role writes the +# env file all-or-nothing, and rejects extra_env against a host-provisioned file +# rather than silently dropping it). Extra KEY: value pairs appended to the 0600 +# /etc/decdn/decdn.env that systemd hands the daemon — most notably the AWS +# credentials backing an S3 cache origin, which the role deliberately does NOT +# write into node.toml (that file is 0640 and diffable): # # decdn_extra_env: # AWS_ACCESS_KEY_ID: "AKIA..." diff --git a/ansible/molecule/default/converge.yml b/ansible/molecule/default/converge.yml index 16a217a..e19b691 100644 --- a/ansible/molecule/default/converge.yml +++ b/ansible/molecule/default/converge.yml @@ -17,6 +17,15 @@ # stay in sync with the version string printed by files/decdn-node-stub. decdn_node_version: "0.0.0-molecule-stub" decdn_rpc_url: "https://rpc.example.invalid/" + # The ACCEPTED side of the decdn_extra_env gate (#39): with an inventory + # decdn_rpc_url the role authors the whole env file, so extra_env is rendered + # rather than rejected. Nothing else in any scenario sets it alongside a + # non-empty rpc_url, which left both this branch AND the template's + # backslash/quote escaping untested. The value carries a double-quote and a + # backslash on purpose — systemd SKIPS a line it cannot parse, so a broken + # escape means the variable is silently absent at runtime. + decdn_extra_env: + DECDN_MOLECULE_EXTRA: 'a "quoted" \ value' decdn_chain_id: 421614 decdn_region: "US" decdn_payment_pool_address: "0x1111111111111111111111111111111111111111" diff --git a/ansible/molecule/default/verify.yml b/ansible/molecule/default/verify.yml index 0062716..47f312f 100644 --- a/ansible/molecule/default/verify.yml +++ b/ansible/molecule/default/verify.yml @@ -30,6 +30,7 @@ - "{{ decdn_etc }}/node.toml" - "{{ decdn_etc }}/decdn.env" - /etc/systemd/system/decdn-node.service + - "{{ decdn_etc }}/.decdn.env.sha256" - name: Assert node.toml (0640) and the secret env file (0600) modes ansible.builtin.assert: @@ -39,7 +40,62 @@ - decdn_files.results[0].stat.exists and decdn_files.results[0].stat.mode == '0640' - decdn_files.results[1].stat.exists and decdn_files.results[1].stat.mode == '0600' - decdn_files.results[2].stat.exists - fail_msg: "node.toml / decdn.env / unit missing or wrong mode" + # The env-file checksum record (#39) — 0600 root:root. It is what makes an + # out-of-band edit of decdn.env notify a restart on the next converge, so + # it must be written on the inventory-authored path too. + - >- + decdn_files.results[3].stat.exists + and decdn_files.results[3].stat.mode == '0600' + and decdn_files.results[3].stat.pw_name == 'root' + and decdn_files.results[3].stat.gr_name == 'root' + fail_msg: "node.toml / decdn.env / unit / env-checksum missing or wrong mode/owner" + + # Existence and mode say nothing about the VALUE. A record holding a constant, + # an empty string, or the pre-write hash would pass the stat above while + # breaking the restart signal in one direction or the other, so compare it + # against the file it claims to describe. + # The role authored this file from inventory, so unlike the host-env scenario + # its exact content IS the role's output and can be asserted directly. + - name: Read the role-authored env file + ansible.builtin.slurp: + src: "{{ decdn_etc }}/decdn.env" + register: decdn_env_body + + - name: Assert the env file renders the RPC URL and an escaped decdn_extra_env + ansible.builtin.assert: + that: + - "'DECDN_RPC_URL=https://rpc.example.invalid/' in env_body" + # Exact escaped form: the template wraps extra_env values in double + # quotes and escapes backslash-then-quote. A regression here is silent — + # systemd drops the unparseable line and the variable is simply absent. + - env_expected in env_body + fail_msg: >- + {{ decdn_etc }}/decdn.env is missing the RPC URL or did not escape the + decdn_extra_env value correctly. Got: {{ env_body }} + vars: + env_body: "{{ decdn_env_body.content | b64decode }}" + env_expected: 'DECDN_MOLECULE_EXTRA="a \"quoted\" \\ value"' + + - name: Compute the env file's current sha256 + ansible.builtin.stat: + path: "{{ decdn_etc }}/decdn.env" + get_checksum: true + checksum_algorithm: sha256 + register: decdn_env_hash + + - name: Read the recorded env-file checksum + ansible.builtin.slurp: + src: "{{ decdn_etc }}/.decdn.env.sha256" + register: decdn_env_recorded + + - name: Assert the recorded checksum describes the env file the role wrote + ansible.builtin.assert: + that: + - (decdn_env_recorded.content | b64decode | trim) == decdn_env_hash.stat.checksum + fail_msg: >- + The recorded env-file checksum does not match {{ decdn_etc }}/decdn.env on + the inventory-authored path, so the next converge would either restart the + daemon for no reason or miss a real change." # prepare.yml stages the keystore + password at 0644; the role is responsible # for locking them to 0600. Asserting 0600 here therefore verifies the role's @@ -229,6 +285,9 @@ - "'RestrictAddressFamilies=AF_INET AF_INET6 AF_NETLINK' in unit" - "'SystemCallFilter=@system-service' in unit" - "'User=decdn' in unit" + # Without this the daemon never receives DECDN_RPC_URL at all, and every + # other assertion in this scenario would still pass. + - "'EnvironmentFile=/etc/decdn/decdn.env' in unit" fail_msg: "hardened unit is missing an expected directive" vars: unit: "{{ decdn_unit.content | b64decode }}" diff --git a/ansible/molecule/host-env/converge.yml b/ansible/molecule/host-env/converge.yml new file mode 100644 index 0000000..e580d6a --- /dev/null +++ b/ansible/molecule/host-env/converge.yml @@ -0,0 +1,33 @@ +--- +# Converge with NO decdn_rpc_url and NO decdn_extra_env: every secret the daemon +# needs is already on the host (staged by prepare.yml), so the role must gate on +# host state instead of demanding the value from inventory. The remaining +# chain/contract knobs mirror ../default/converge.yml (placeholder but well-formed, +# so the role's fail-loud asserts still run). baseline is not exercised here — see +# molecule.yml. +- name: Converge + hosts: all + become: true + vars: + # Reuse the single shared stub from the default scenario (do not duplicate it). + stub_bin: "{{ lookup('ansible.builtin.env', 'MOLECULE_PROJECT_DIRECTORY') }}/molecule/default/files/decdn-node-stub" + decdn_node_install_method: manual + decdn_node_manual_bin_src: "{{ stub_bin }}" + decdn_cli_manual_bin_src: "{{ stub_bin }}" + decdn_node_version: "0.0.0-molecule-stub" + # decdn_rpc_url is deliberately UNSET — that is the feature under test. + decdn_chain_id: 421614 + decdn_region: "US" + decdn_payment_pool_address: "0x1111111111111111111111111111111111111111" + decdn_capacity_bond_address: "0x2222222222222222222222222222222222222222" + decdn_slash_judge_address: "0x3333333333333333333333333333333333333333" + # REQUIRED upstream (ADR 011/031) — the role asserts it with the other three. + decdn_content_blacklist_address: "0x4444444444444444444444444444444444444444" + # An S3 origin reading its credentials from the host env file — the exact + # combination the old inventory-only path forced onto the control machine. + decdn_cache_origin_kind: "s3" + decdn_cache_origin_s3_bucket: "molecule-bucket" + decdn_cache_origin_s3_region: "us-east-1" + decdn_cache_origin_s3_use_default_chain: true + roles: + - role: decdn_node diff --git a/ansible/molecule/host-env/molecule.yml b/ansible/molecule/host-env/molecule.yml new file mode 100644 index 0000000..858d2c6 --- /dev/null +++ b/ansible/molecule/host-env/molecule.yml @@ -0,0 +1,51 @@ +--- +# Containerised converge + idempotence + verify for the HOST-PROVISIONED secret +# path (#39): /etc/decdn/decdn.env is written by the operator on the target and +# decdn_rpc_url is left empty in inventory, so no secret transits the control +# machine. The role must then NOT rewrite the file, must lock it to 0600 +# decdn:decdn, and must still start the daemon. `idempotence` is load-bearing +# here — it proves the role neither rewrites the file nor restart-loops on it. +# +# `baseline` is NOT exercised here for the same reasons as the `default` scenario +# (host-level hardening is meaningless in a throwaway systemd container) — see +# ../default/molecule.yml. This scenario reuses the shared stub daemon under +# ../default/files/decdn-node-stub. +dependency: + name: galaxy + options: + requirements-file: ../../requirements.yml +driver: + name: docker +platforms: + - name: decdn-node-hostenv-molecule + # Same image as the default scenario, pinned by digest for reproducible CI. + # Re-resolve the digest to bump (keep in sync with ../default/molecule.yml). + image: geerlingguy/docker-debian12-ansible@sha256:4553092be2c00b1ffe580927b9ff03f3c3a0df32b7dd693a3eb02efb6c2b77b7 + pre_build_image: true + command: /usr/lib/systemd/systemd + privileged: true + cgroupns_mode: host + volumes: + - /sys/fs/cgroup:/sys/fs/cgroup:rw +provisioner: + name: ansible + env: + ANSIBLE_ROLES_PATH: "${MOLECULE_PROJECT_DIRECTORY}/roles" + ANSIBLE_COLLECTIONS_PATH: "${MOLECULE_PROJECT_DIRECTORY}/collections" +verifier: + name: ansible +scenario: + # converge -> idempotence proves a STABLE host file neither gets rewritten nor + # restarts the unit; side_effect then edits that file out of band and the second + # converge must restart the daemon (verify.yml compares MainPID across it). The + # order matters: idempotence has to run while the file is still untouched. + test_sequence: + - dependency + - create + - prepare + - converge + - idempotence + - side_effect + - converge + - verify + - destroy diff --git a/ansible/molecule/host-env/prepare.yml b/ansible/molecule/host-env/prepare.yml new file mode 100644 index 0000000..dc03af7 --- /dev/null +++ b/ansible/molecule/host-env/prepare.yml @@ -0,0 +1,56 @@ +--- +# Stage what an OPERATOR would have provisioned on the host: the eth keystore + +# password + node identity (as ../default/prepare.yml does), and — the point of +# this scenario — a /etc/decdn/decdn.env the role must not author or rewrite. +# +# Both the env file and the wallet material are staged at 0644, deliberately +# looser than the target: the role is responsible for locking every one of them +# to 0600, so verify.yml's mode assertions attribute the mode to the role rather +# than to what prepare pre-set. +- name: Prepare + hosts: all + become: true + vars: + decdn_home: /var/lib/decdn + decdn_etc: /etc/decdn + tasks: + - name: Ensure data + config directories exist + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: "0755" + loop: + - "{{ decdn_home }}" + - "{{ decdn_etc }}" + + - name: Stage a placeholder eth keystore + ansible.builtin.copy: + dest: "{{ decdn_home }}/keystore.json" + content: "{{ '{}' }}\n" + mode: "0644" + + - name: Stage a placeholder keystore password file + ansible.builtin.copy: + dest: "{{ decdn_etc }}/keystore.password" + content: "molecule-placeholder\n" + mode: "0644" + + - name: Stage a placeholder node identity + ansible.builtin.copy: + dest: "{{ decdn_home }}/node.secret" + content: "molecule-placeholder-node-secret\n" + mode: "0644" + + # The host-provisioned secret env file. It carries a second KEY=value the role + # has no inventory var for at all (converge passes no decdn_extra_env), so + # verify.yml's byte-identical assertion proves the role preserved operator + # content it could not have reproduced. In production that second line is + # typically AWS_ACCESS_KEY_ID for an S3 origin; a neutral marker name is used + # here so the KICS hardcoded-secret query has nothing to flag in a fixture. + - name: Stage the operator-provisioned secret environment file + ansible.builtin.copy: + dest: "{{ decdn_etc }}/decdn.env" + content: | + DECDN_RPC_URL=https://rpc.example.invalid/ + DECDN_MOLECULE_HOST_LINE=written-by-the-operator + mode: "0644" diff --git a/ansible/molecule/host-env/side_effect.yml b/ansible/molecule/host-env/side_effect.yml new file mode 100644 index 0000000..6eaa611 --- /dev/null +++ b/ansible/molecule/host-env/side_effect.yml @@ -0,0 +1,52 @@ +--- +# Edit the host-provisioned env file OUT OF BAND, exactly as an operator rotating +# an RPC key would, and record the daemon's current MainPID. molecule.yml runs a +# SECOND converge after this, and verify.yml then asserts the PID changed. +# +# This is the only test of the feature's headline behaviour. Every other assertion +# about /etc/decdn/.decdn.env.sha256 checks a precondition (it exists, it is 0600 +# root, it matches the file) — all of which stay true if the restart signal is +# removed entirely. `idempotence` proves the converse (an UNCHANGED file must not +# restart), so the two together pin the behaviour from both sides. +- name: Side effect — rotate the secret out of band + hosts: all + become: true + vars: + decdn_etc: /etc/decdn + tasks: + # pgrep, not `systemctl show --property=MainPID`: ansible-lint's + # command-instead-of-module rule rejects any `systemctl` invocation, and there + # is no module that returns a unit's PID. This queries the process table + # instead, which is a different job from managing the service. + - name: Record the daemon's PID before the edit + ansible.builtin.command: + cmd: pgrep -f '/usr/local/bin/decdn-node run' + register: decdn_pid_before + changed_when: false + failed_when: false + + - name: Assert the daemon was actually running before the edit + ansible.builtin.assert: + that: + - decdn_pid_before.stdout_lines | length == 1 + - decdn_pid_before.stdout | trim | int > 0 + fail_msg: >- + Expected exactly one running decdn-node before the out-of-band edit + (got {{ decdn_pid_before.stdout_lines | default([]) }}), so a PID change + after it would prove nothing. + + - name: Save the pre-edit MainPID for verify.yml + ansible.builtin.copy: + dest: "{{ decdn_etc }}/molecule-pid-before" + content: "{{ decdn_pid_before.stdout | trim }}\n" + mode: "0644" + + # Append/replace in place rather than rewriting the file: this must look like + # an operator edit, and it also proves the role preserves the line it did not + # write (DECDN_MOLECULE_HOST_LINE) across the restart path. + - name: Rotate the secret out of band (operator edits the file directly) + ansible.builtin.lineinfile: + path: "{{ decdn_etc }}/decdn.env" + regexp: '^DECDN_RPC_URL=' + line: "DECDN_RPC_URL=https://rotated.example.invalid/" + mode: "0600" diff --git a/ansible/molecule/host-env/verify.yml b/ansible/molecule/host-env/verify.yml new file mode 100644 index 0000000..8c443bb --- /dev/null +++ b/ansible/molecule/host-env/verify.yml @@ -0,0 +1,182 @@ +--- +# Verify the host-provisioned secret path (#39): the role must have left the +# operator's /etc/decdn/decdn.env byte-for-byte intact, locked it to 0600 +# decdn:decdn, recorded its checksum, and started the daemon on it. +- name: Verify + hosts: all + become: true + vars: + decdn_etc: /etc/decdn + # What the host file holds by the time verify runs: prepare.yml's content with + # side_effect.yml's rotated DECDN_RPC_URL line. Comparing against a literal + # (rather than re-reading and trusting whatever is there) is what makes this an + # assertion about the ROLE's behaviour and not a tautology — and the untouched + # DECDN_MOLECULE_HOST_LINE is a key the role has no inventory var for, so a + # role that rewrote the file could not have reproduced it. + expected_env: | + DECDN_RPC_URL=https://rotated.example.invalid/ + DECDN_MOLECULE_HOST_LINE=written-by-the-operator + tasks: + - name: Read back the host-provisioned env file + ansible.builtin.slurp: + src: "{{ decdn_etc }}/decdn.env" + register: decdn_env_content + + - name: Assert the role did not rewrite the operator's env file + ansible.builtin.assert: + that: + - (decdn_env_content.content | b64decode) == expected_env + fail_msg: >- + {{ decdn_etc }}/decdn.env does not hold the operator's content. With + decdn_rpc_url unset the role must leave it byte-for-byte alone — a + rewrite would have dropped DECDN_MOLECULE_HOST_LINE (converge passes no + decdn_extra_env), which in production is the AWS credential line behind + the S3 origin: silently broken at runtime, green at deploy time. It must + also have survived the out-of-band rotation in side_effect.yml. + Got: {{ decdn_env_content.content | b64decode }} + + # prepare.yml stages the env file (and the wallet material) at 0644; the role + # is responsible for locking them to 0600. Asserting 0600 here therefore + # verifies the role's lock-down ran, not a mode prepare pre-set. + - name: Stat the env file, its checksum record, and the wallet material + ansible.builtin.stat: + path: "{{ item }}" + register: decdn_files + loop: + - "{{ decdn_etc }}/decdn.env" + - "{{ decdn_etc }}/.decdn.env.sha256" + - /var/lib/decdn/keystore.json + - /var/lib/decdn/node.secret + - "{{ decdn_etc }}/keystore.password" + + - name: Assert the role locked the host-provisioned env file to 0600 decdn:decdn + ansible.builtin.assert: + that: + # exists-and-mode-and-owner in one expression: a missing file + # short-circuits to the fail_msg instead of raising on the absent stat. + - >- + decdn_files.results[0].stat.exists + and decdn_files.results[0].stat.mode == '0600' + and decdn_files.results[0].stat.pw_name == 'decdn' + and decdn_files.results[0].stat.gr_name == 'decdn' + fail_msg: "decdn.env missing, not 0600, or not owned by decdn (role lock-down did not run)" + + - name: Assert the env-file checksum record exists at 0600 root:root + ansible.builtin.assert: + that: + - >- + decdn_files.results[1].stat.exists + and decdn_files.results[1].stat.mode == '0600' + and decdn_files.results[1].stat.pw_name == 'root' + fail_msg: >- + {{ decdn_etc }}/.decdn.env.sha256 missing or not 0600 root:root — without + it an out-of-band edit of decdn.env would never notify a restart. + + - name: Assert the wallet material is locked to 0600 + ansible.builtin.assert: + that: + - decdn_files.results[2].stat.exists and decdn_files.results[2].stat.mode == '0600' + - decdn_files.results[3].stat.exists and decdn_files.results[3].stat.mode == '0600' + - decdn_files.results[4].stat.exists and decdn_files.results[4].stat.mode == '0600' + fail_msg: "keystore / node.secret / password file missing or not 0600" + + # The recorded hash must be the hash of the file as it stands, or the next + # converge would restart the daemon for no reason (or, worse, not restart it + # after a genuine edit). + - name: Compute the env file's current sha256 + ansible.builtin.stat: + path: "{{ decdn_etc }}/decdn.env" + get_checksum: true + checksum_algorithm: sha256 + register: decdn_env_hash + + - name: Read the recorded checksum + ansible.builtin.slurp: + src: "{{ decdn_etc }}/.decdn.env.sha256" + register: decdn_env_recorded + + - name: Assert the recorded checksum matches the env file on disk + ansible.builtin.assert: + that: + - (decdn_env_recorded.content | b64decode | trim) == decdn_env_hash.stat.checksum + fail_msg: >- + The recorded env-file checksum does not match {{ decdn_etc }}/decdn.env, + so the restart-on-out-of-band-edit signal is wrong in both directions. + + # node.toml must carry the S3 origin with source = "default-chain" and NO + # static keys: the credentials live only in the 0600 env file. + - name: Read the rendered node.toml + ansible.builtin.slurp: + src: "{{ decdn_etc }}/node.toml" + register: decdn_node_toml + + - name: Assert node.toml delegates S3 credentials to the env file + ansible.builtin.assert: + that: + - "'source = \"default-chain\"' in node_toml" + # No VALUE from the 0600 env file may appear in this 0640, diffable file. + # (The template's header comment names DECDN_RPC_URL to explain where the + # endpoint lives, so match on the values, not the variable names.) + - "'written-by-the-operator' not in node_toml" + - "'rpc.example.invalid' not in node_toml" + fail_msg: "node.toml is missing the default-chain credential source, or leaked an env-file value" + vars: + node_toml: "{{ decdn_node_toml.content | b64decode }}" + + # THE headline behaviour: side_effect.yml rotated DECDN_RPC_URL out of band and + # molecule.yml re-converged. The daemon must have been restarted, or it is + # still serving with the old endpoint behind a green deploy. Comparing MainPID + # is what makes this behavioural — the checksum assertions above all remain + # true even if the restart signal is deleted outright. + - name: Read the pre-edit MainPID recorded by side_effect + ansible.builtin.slurp: + src: "{{ decdn_etc }}/molecule-pid-before" + register: decdn_pid_before + + # pgrep rather than `systemctl show` — see the note in side_effect.yml. + - name: Read the daemon's PID after the re-converge + ansible.builtin.command: + cmd: pgrep -f '/usr/local/bin/decdn-node run' + register: decdn_pid_after + changed_when: false + failed_when: false + + - name: Assert the out-of-band edit restarted decdn-node + ansible.builtin.assert: + that: + - decdn_pid_after.stdout_lines | length == 1 + - decdn_pid_after.stdout | trim | int > 0 + - (decdn_pid_after.stdout | trim) != (decdn_pid_before.content | b64decode | trim) + fail_msg: >- + decdn-node still runs as PID {{ decdn_pid_after.stdout | trim }} after + {{ decdn_etc }}/decdn.env was edited out of band and the role re-converged + — the daemon is on the PRE-rotation secret behind a green deploy. The + env-file checksum record did not drive the restart handler. + + # Without EnvironmentFile the daemon never receives the host-provisioned + # secret at all, and every other assertion in this scenario would still pass. + - name: Read the rendered systemd unit + ansible.builtin.slurp: + src: /etc/systemd/system/decdn-node.service + register: decdn_unit + + - name: Assert the unit actually loads the secret env file + ansible.builtin.assert: + that: + - "'EnvironmentFile=' ~ decdn_etc ~ '/decdn.env' in (decdn_unit.content | b64decode)" + fail_msg: >- + decdn-node.service does not carry EnvironmentFile={{ decdn_etc }}/decdn.env, + so the host-provisioned secret never reaches the daemon. + + - name: Gather service facts + ansible.builtin.service_facts: + + - name: Assert decdn-node is running on the host-provisioned secrets + ansible.builtin.assert: + that: + # Membership-and-state in one expression so an absent service + # short-circuits to the fail_msg instead of raising on .state. + - >- + 'decdn-node.service' in ansible_facts.services + and ansible_facts.services['decdn-node.service'].state == 'running' + fail_msg: "decdn-node is not running — check: journalctl -u decdn-node -e" diff --git a/ansible/molecule/validation/converge.yml b/ansible/molecule/validation/converge.yml index 34c9926..c13d716 100644 --- a/ansible/molecule/validation/converge.yml +++ b/ansible/molecule/validation/converge.yml @@ -3,13 +3,20 @@ # REJECTS bad values (AGENTS.md hard-rule 4), so a regression that weakens an assert # fails CI. Each case runs decdn_node with a VALID required config plus ONE bad # optional knob, wrapped in block/rescue. The role aborts at its validation asserts -# (which run before any host mutation), so cases don't interfere and no service starts. +# (which for the tuning-knob cases run before any host mutation), so those cases +# don't interfere and no service starts. The env-file cases below are the +# exception — see the ordering note. # -# The rescue records a case as "rejected" ONLY when the failing task is one of the -# role's own "Validate optional ..." asserts (matched on ansible_failed_task.name). -# That is what makes this a real regression guard: if a bad value slipped PAST -# validation, the role would run on and fail later at a differently-named task (e.g. -# the keystore gate) — which is NOT counted, so the final assert catches the miss. +# The rescue records a case as "rejected" ONLY when the failing task is the SPECIFIC +# assert that case targets (matched on ansible_failed_task.name — mostly the role's +# "Validate optional ..." family, plus the two secret-source gates). That is what +# makes this a real regression guard: if a bad value slipped PAST validation, the +# role would run on and fail later at a differently-named task (e.g. the keystore +# gate) — which is NOT counted, so the final assert catches the miss. +# +# Two cases touch host state on purpose: "env-extra" stages a host-provisioned +# /etc/decdn/decdn.env (it has to exist for that gate to fire), so it must run +# AFTER "env-gate", which proves the absent-file case still fails. # # ONE case is deliberately different: the final "config-gate" case has a valid # config and has to reach the END of the role, because what it proves is that the @@ -191,8 +198,96 @@ decdn_rejected: "{{ decdn_rejected + ['string'] }}" when: ansible_failed_task.name is match('^Validate optional') + # --- Case: no RPC endpoint anywhere (#39) ------------------------------------- + # decdn_rpc_url may now be empty — but ONLY when the operator has provisioned + # /etc/decdn/decdn.env on the host. Nothing has staged that file yet at this + # point in the play, so "empty and absent" must still fail loud. A regression + # that relaxed the gate to "always optional" would start a daemon with no chain + # endpoint. This case must run BEFORE the one below, which stages the file. + - name: "Case env-gate — no rpc_url and no host-provisioned env file" + block: + - name: Run decdn_node with neither an inventory rpc_url nor a host env file + ansible.builtin.include_role: + name: decdn_node + vars: + decdn_rpc_url: "" + rescue: + - name: Record env-gate rejection (only if the RPC gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['env-gate'] }}" + when: ansible_failed_task.name is match('^Require an RPC endpoint') + + # --- Case: decdn_extra_env against a host-provisioned env file (#39) ---------- + # The role writes the env file ALL-OR-NOTHING from inventory, so extra_env with + # an empty rpc_url would be silently dropped — the S3 origin would 403 hours + # later with nothing in the deploy output to explain it. Must fail loud instead. + - name: "Case env-extra — decdn_extra_env alongside a host-provisioned env file" + block: + - name: Stage a host-provisioned env file so the role takes the host path + ansible.builtin.copy: + dest: /etc/decdn/decdn.env + content: "DECDN_RPC_URL=https://rpc.example.invalid/\n" + mode: "0600" + + - name: Run decdn_node with extra_env but no rpc_url + ansible.builtin.include_role: + name: decdn_node + vars: + decdn_rpc_url: "" + decdn_extra_env: + DECDN_MOLECULE_EXTRA: "would-be-silently-dropped" + rescue: + - name: Record env-extra rejection (only if the extra_env assert failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['env-extra'] }}" + when: ansible_failed_task.name is match('^Reject decdn_extra_env') + always: + # Each env-file case owns its fixture end to end rather than relying on the + # next case to tolerate what it left behind — the ordering dependency was + # already implicit and a third case made it fragile. + - name: Remove the staged env file so later cases start clean + ansible.builtin.file: + path: /etc/decdn/decdn.env + state: absent + + # --- Case: a host env file with no usable DECDN_RPC_URL (#39) ----------------- + # The RPC gate accepts the host path on stat.exists ALONE, so this grep is the + # only thing standing between a file that merely exists and a daemon started + # with no chain endpoint. Without a case here, weakening the pattern (dropping + # the value match, un-anchoring it) or neutering the report task's `when:` + # would be an invisible regression — exactly what the config-gate case below + # exists to prevent for its own gate. The staged file is commented out, which + # is the realistic operator error (and `DECDN_RPC_URL= ` with only whitespace + # must fail too — that is what the [^[:space:]] in the pattern buys). + - name: "Case env-content — host env file carrying no usable DECDN_RPC_URL" + block: + - name: Stage a host env file whose DECDN_RPC_URL is commented out + ansible.builtin.copy: + dest: /etc/decdn/decdn.env + content: | + #DECDN_RPC_URL=https://rpc.example.invalid/ + DECDN_MOLECULE_OTHER=present + mode: "0600" + + - name: Run decdn_node against an env file with no usable endpoint + ansible.builtin.include_role: + name: decdn_node + vars: + decdn_rpc_url: "" + rescue: + - name: Record env-content rejection (only if the content gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['env-content'] }}" + when: ansible_failed_task.name is match('^Report a host-provisioned env file') + always: + - name: Remove the staged env file so later cases start clean + ansible.builtin.file: + path: /etc/decdn/decdn.env + state: absent + # --- Case: the config gate itself --------------------------------------------- - # Every case above trips an Ansible assert BEFORE the host is touched. This one + # Every tuning-knob case above trips an Ansible assert before the host is + # touched (env-extra is the exception — it stages a file first). This one # proves the last line of defence still works: `decdn config validate` runs # with `failed_when: false` and hands off to a separate `fail` task, so a # regression in that task's `when:` would silently turn the authoritative gate @@ -260,4 +355,5 @@ vars: _decdn_expected: ["cross-field", "range", "bounded-range", "shape", "bool", "enum", "dict", - "float", "list", "string", "config-gate"] + "float", "list", "string", "env-gate", "env-extra", "env-content", + "config-gate"] diff --git a/ansible/roles/decdn_node/README.md b/ansible/roles/decdn_node/README.md index be2631d..036068f 100644 --- a/ansible/roles/decdn_node/README.md +++ b/ansible/roles/decdn_node/README.md @@ -102,10 +102,10 @@ Per the deCDN node-onboarding ADR (019), a node only serves paid traffic after ## Required variables (set in `host_vars//`) `decdn_node_version` (`release` mode only) **or** `decdn_node_manual_bin_src` + -`decdn_cli_manual_bin_src` (`manual` mode), `decdn_rpc_url` (sensitive — may embed an -API key; goes in the git-ignored `secret.yml`, everything else in the committed -`main.yml`), `decdn_region` (ISO 3166-1 alpha-2), and **four** contract addresses -(all `0x`+40-hex, none the zero address): +`decdn_cli_manual_bin_src` (`manual` mode), an RPC endpoint (sensitive — may embed an +API key; provision it on the host or set `decdn_rpc_url` in the git-ignored +`secret.yml` — see [Secrets](#secrets)), `decdn_region` (ISO 3166-1 alpha-2), and +**four** contract addresses (all `0x`+40-hex, none the zero address): | Variable | Contract | Why required | |---|---|---| @@ -141,8 +141,10 @@ Optional (omitted from `node.toml` unless set): dicts) instead — the two forms are mutually exclusive upstream and the role fails loud if both are set. - `decdn_extra_env` — extra `KEY: value` pairs appended to the 0600 - `/etc/decdn/decdn.env`. This is where environment-borne secrets belong; see - [S3 credentials](#s3-credentials). + `/etc/decdn/decdn.env` **when the role authors that file** (i.e. `decdn_rpc_url` + is set). With a host-provisioned env file the role rejects it rather than + silently dropping it — put those lines in the host file instead. See + [Secrets](#secrets) and [S3 credentials](#s3-credentials). - `decdn_node_generate_keystore` (default `false`) — opt-in turnkey wallet. When `true` the role runs `decdn key-gen` on the host **only if the keystore is absent** (minting a random `0600` password file first, but only when the keystore is *also* absent) — it @@ -227,13 +229,97 @@ asserts miss. See `defaults/main.yml` for every knob's upstream default, unit an - **Observability** — `decdn_otlp_endpoint` (OTLP span export; needs the node built `--features otlp`). +## Secrets + +Every secret this role needs ends up in exactly one place on the host: `0600` +files under `/etc/decdn` and `/var/lib/decdn`, owned by the `decdn` user. The +wallet material (`keystore.json`, `node.secret`, `keystore.password`) is +provisioned — or, with `decdn_node_generate_keystore`, minted — **on the host**, +and never touches the control machine. The one +operator-supplied secret — the chain RPC URL, which may embed a provider API key, +plus anything else the daemon reads from its environment — has two authoring +paths: + +| | Host-provisioned (preferred) | Inventory-carried | +|---|---|---| +| Set | `decdn_rpc_url: ""` (the default) | `decdn_rpc_url` in `host_vars//secret.yml` | +| Who writes `/etc/decdn/decdn.env` | you, on the target | the role, on every converge | +| Secret on the control machine | never | plaintext, git-ignored | +| Extra env vars | further `KEY=value` lines in that file | `decdn_extra_env` | + +On the host path the role never reads the file's contents back — that would put +the secret on the control machine, defeating the point. It reads only a **sha256** +of the file (which does cross to the control machine, to be compared and written +back as the record below), and on that path it confines itself to: + +- enforcing `0600 decdn:decdn` on the file, content untouched, after refusing to + touch it at all unless it is a regular file (a symlink would have root apply + that ownership to whatever it points at); +- grepping for a `DECDN_RPC_URL=` line with a non-whitespace value, failing loud + if there is none. + +**Provisioning it.** Do this on the target as **root** — on a host that has never +been converged the `decdn` account does not exist yet (this role creates it, and +chowns the file on the next run), so `install -o decdn` / `chown decdn:decdn` +would fail. The full template, including the S3 credential block, is +[`files/decdn.env.example`](files/decdn.env.example): + +```bash +(umask 077; sudo mkdir -p /etc/decdn) +echo 'DECDN_RPC_URL=https://your-endpoint.example/rpc' | sudo tee /etc/decdn/decdn.env >/dev/null +sudo chmod 600 /etc/decdn/decdn.env +``` + +systemd's `EnvironmentFile` parser is shell-*like* but not a shell: one `KEY=value` +per line, no `export`, no expansion — and it **skips** a line it cannot parse, +logging a warning to `journalctl -u decdn-node` that is easy to miss, so the +variable is simply absent at runtime. Single-quote any value containing a space, +quote, backslash, `$` or backtick. (`decdn config validate` sources this file with +bash, which *would* expand `$VAR` and execute `$(…)`; quoting avoids both.) On the +inventory path the role applies that escaping to `decdn_extra_env` values for you — +but **not** to `decdn_rpc_url` itself, which is rendered raw. + +### The provenance record + +`/etc/decdn/.decdn.env.sha256` (`0600 root`, `decdn_env_checksum_file`) holds the +env file's sha256 as of the last converge that got the daemon running. It is +written on **both** paths, and it is what lets the role tell the two apart — a +bare `stat` cannot, since a file the role wrote last week exists exactly as much +as an operator-provisioned one. From it the role gets three behaviours: + +- **Restart on an out-of-band edit.** Record ≠ file on the host path ⇒ the unit is + restarted so the daemon picks up the new values. The record is written *after* + the restart, so a run that aborts in between leaves the old value in place and + the next converge re-detects — the signal is never burned by a failed deploy. +- **"`secret.yml` went missing" ≠ "the host owns this now."** An empty + `decdn_rpc_url` against a file the role itself wrote is treated as a lost + inventory value and fails loud, instead of silently deploying a stale endpoint. + To migrate a node to the host path deliberately, `sudo rm` the record once. +- **No silent clobbering.** An inventory `decdn_rpc_url` against a file the record + says someone else wrote fails loud rather than discarding those lines; set + `decdn_env_overwrite_host_file: true` to confirm. With **no** record (a host + that predates this) the role cannot tell, so it warns and starts tracking. + +If neither authoring path is satisfied the role fails before touching the host, +with the provisioning commands above in the failure message. + ### S3 credentials The role deliberately **never** writes static AWS keys into `node.toml`: that file is `0640` and diffable, and committing credentials to it would break this repo's no-secrets rule. Only `source = "default-chain"` is templated. Set `decdn_cache_origin_s3_use_default_chain: true` (plus an optional -`decdn_cache_origin_s3_profile`) and supply the keys through the `0600` env file: +`decdn_cache_origin_s3_profile`) and supply the keys through the `0600` env file. +On the host, as further lines in `/etc/decdn/decdn.env`: + +```ini +AWS_ACCESS_KEY_ID=AKIA... +AWS_SECRET_ACCESS_KEY=... +AWS_SESSION_TOKEN=... # assume-role / SSO only +``` + +…or, if you are carrying the RPC URL in inventory anyway, through `decdn_extra_env` +(the role then renders both into the same file): ```yaml # host_vars//secret.yml (git-ignored) @@ -250,7 +336,7 @@ instance profile needs no credentials at all, just the toggle. Note the region i **not** taken from `AWS_REGION`: it comes from `[cache.origin].region` (`decdn_cache_origin_s3_region`). -### Config reload +## Config reload SIGHUP (and `decdn node reload`, which shares its mutex) re-reads the config file and applies **five** sections: `observability.log_level`, `cache.pinned_hashes`, @@ -354,7 +440,8 @@ additionally requires on-chain stake + registration (see - `/usr/local/bin/decdn-node`, `/usr/local/bin/decdn` — daemon + CLI - `/etc/decdn/node.toml` (`0640 decdn`) — rendered config; non-secret (no `rpc_url`), so it stays `--check`-diffable -- `/etc/decdn/decdn.env` (`0600 decdn`) — the one sensitive value, `DECDN_RPC_URL`, read by the unit via `EnvironmentFile` +- `/etc/decdn/decdn.env` (`0600 decdn`) — the one operator-supplied secret, `DECDN_RPC_URL` (+ any extra env vars), read by the unit via `EnvironmentFile`; role-authored or host-provisioned ([Secrets](#secrets)) +- `/etc/decdn/.decdn.env.sha256` (`0600 root`) — role-managed hash of the above: provenance marker, and the trigger for a restart after an out-of-band edit. Not a secret store - `/etc/decdn/keystore.password` (`0600 decdn`) — operator-provisioned - `/var/lib/decdn/` (`0700 decdn`) — `node.secret`, `keystore.json`, `cache/`, state - `/etc/systemd/system/decdn-node.service` — hardened unit diff --git a/ansible/roles/decdn_node/defaults/main.yml b/ansible/roles/decdn_node/defaults/main.yml index 97ef62d..b34b8d1 100644 --- a/ansible/roles/decdn_node/defaults/main.yml +++ b/ansible/roles/decdn_node/defaults/main.yml @@ -1,7 +1,10 @@ --- # decdn-node role defaults. Per-node deployment values belong in host_vars//: -# non-secret config (contract addresses, region, version) in the committed main.yml, -# and the sensitive rpc_url in a git-ignored secret.yml (see secret.yml.example). +# non-secret config (contract addresses, region, version) in the committed main.yml. +# The sensitive rpc_url does NOT have to live on the control machine at all — +# provision /etc/decdn/decdn.env on the target (files/decdn.env.example) and leave +# decdn_rpc_url empty; set it in a git-ignored secret.yml only if you want the role +# to author that file for you. # --- Install method ----------------------------------------------------------- # "release" (default): download the pinned GitHub Release tarballs (a v @@ -30,11 +33,18 @@ decdn_verify_release_signature: true decdn_release_keyring: decdn-release-KEYS.asc # --- Identity / on-chain ----------------------------------------------------- -# rpc_url, the FOUR contract addresses and region are REQUIRED with no usable -# default (empty → the asserts in tasks/main.yml fail loud). Supply via host_vars. +# The FOUR contract addresses and region are REQUIRED with no usable default +# (empty → the asserts in tasks/main.yml fail loud). Supply via host_vars. # chain_id is the one exception: it defaults to the upstream binary's own default # (Arbitrum Sepolia) rather than being a deployment-specific fact to source. -decdn_rpc_url: "" # SENSITIVE; may embed an API key +# +# rpc_url is required too, but has TWO sources. Leave it empty here and provision +# a 0600 /etc/decdn/decdn.env on the target host instead (the preferred posture — +# the secret then never touches the control machine); set it here only to have the +# role author that file for you, in which case it OVERWRITES any host-provisioned +# one. The role fails loud when NEITHER is present. See roles/decdn_node/README.md +# ("Secrets") and roles/decdn_node/files/decdn.env.example. +decdn_rpc_url: "" # SENSITIVE; may embed an API key — or provision decdn.env on the host decdn_payment_pool_address: "" decdn_capacity_bond_address: "" decdn_slash_judge_address: "" @@ -300,11 +310,28 @@ decdn_home: /var/lib/decdn # data dir: node.secret + keystore.json decdn_etc: /etc/decdn decdn_config_file: /etc/decdn/node.toml decdn_env_file: /etc/decdn/decdn.env # DECDN_RPC_URL (sensitive) — 0600 +# Role-managed record of the env file's sha256 as of the last converge that got +# the daemon running. NOT a secret store (it holds one hash, 0600 root:root). It +# does two jobs: it lets an OUT-OF-BAND edit of a host-provisioned decdn.env +# notify the restart handler, and it is the file's provenance marker — matching +# means this role authored it, differing means someone else did, and absent means +# the host predates this mechanism. Written on BOTH authoring paths. +decdn_env_checksum_file: /etc/decdn/.decdn.env.sha256 +# Confirm that rendering the env file from inventory may DISCARD a host-side edit. +# Only consulted when the recorded checksum proves the file on the host is not the +# one this role last wrote; with no record (a pre-#39 host) the role cannot tell +# and warns instead. Leave false so an operator-written AWS credential line cannot +# be silently overwritten by a converge that carries decdn_rpc_url. +decdn_env_overwrite_host_file: false # Extra KEY: value pairs appended to the 0600 env file. This is the supported home # for secrets the daemon reads from the environment rather than node.toml — most # notably AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN backing an # S3 cache origin with source = "default-chain". Keep it out of committed vars: -# set it in the git-ignored secret.yml alongside decdn_rpc_url. +# set it in the git-ignored secret.yml alongside decdn_rpc_url — or, better, skip +# the control machine entirely and write them straight into the host's decdn.env +# (see decdn_rpc_url above). The two forms are mutually exclusive: the role writes +# the env file ALL-OR-NOTHING from inventory, so decdn_extra_env alongside a +# host-provisioned file is rejected rather than silently dropped. decdn_extra_env: {} # Eth keystore lives in the data dir (matches `decdn key-gen --output-dir`). decdn_keystore_file: /var/lib/decdn/keystore.json # operator-provisioned (0600) diff --git a/ansible/roles/decdn_node/files/decdn.env.example b/ansible/roles/decdn_node/files/decdn.env.example new file mode 100644 index 0000000..f2ad528 --- /dev/null +++ b/ansible/roles/decdn_node/files/decdn.env.example @@ -0,0 +1,41 @@ +# /etc/decdn/decdn.env — the deCDN node's secret environment file. +# +# Provision this ON THE TARGET HOST, not on the control machine: systemd reads it +# as root (EnvironmentFile=) before dropping to the `decdn` user, and the decdn_node +# Ansible role deliberately never reads its contents back. Leave decdn_rpc_url empty +# in inventory and the role leaves this file alone, enforcing only 0600 ownership — +# so the RPC key (and any cloud credentials below) never sit on a laptop. +# +# This file is a TEMPLATE that lives on the control machine; copy its contents to +# the host by hand (or paste the two commands below). Create it as ROOT: on a host +# that has never been converged the `decdn` account does not exist yet — the role +# creates it, and chowns this file to decdn:decdn on the converge that follows. +# +# (umask 077; sudo mkdir -p /etc/decdn) +# sudo tee /etc/decdn/decdn.env >/dev/null # then paste the lines, Ctrl-D +# sudo chmod 600 /etc/decdn/decdn.env +# +# After editing it out of band, re-run `make deploy` — the role notices the change +# and restarts the unit — or restart by hand: `sudo systemctl restart decdn-node`. +# +# FORMAT: systemd's EnvironmentFile parser is shell-LIKE but is not a shell — no +# expansion, no `export`, one KEY=value per line. It SKIPS a line it cannot parse +# (logging a warning to `journalctl -u decdn-node` that is easy to miss), so the +# variable is simply absent at runtime — surfacing much later as an auth failure. +# Single-quote any value containing a space, quote, backslash, `$` or backtick. +# The role's `decdn config validate` step sources this file with bash, which — +# unlike systemd — WOULD expand `$VAR` and execute `$(...)`; quoting avoids both. + +# REQUIRED. The chain RPC endpoint — may embed a provider API key, which is why +# this file exists at all. Public endpoints work for light use; run your own or use +# a provider for production reliability. +DECDN_RPC_URL=https://your-endpoint.example/rpc + +# OPTIONAL. Static AWS credentials for an S3 cache pull-through origin. The role +# never writes these into node.toml (that file is 0640 and diffable) — it only +# templates `source = "default-chain"` there (decdn_cache_origin_s3_use_default_chain: +# true), and the AWS default credential chain reads exactly these variables. +# Unnecessary on an instance with an IAM role / instance profile. +#AWS_ACCESS_KEY_ID=AKIA... +#AWS_SECRET_ACCESS_KEY=... +#AWS_SESSION_TOKEN=... # assume-role / SSO flows only diff --git a/ansible/roles/decdn_node/tasks/main.yml b/ansible/roles/decdn_node/tasks/main.yml index e771d91..ddcdeb8 100644 --- a/ansible/roles/decdn_node/tasks/main.yml +++ b/ansible/roles/decdn_node/tasks/main.yml @@ -59,10 +59,52 @@ decdn_node_generate_keystore must be a boolean (true/false), not a quoted string — it gates a `| bool` conditional on host-side wallet generation. +# The RPC URL has TWO authoring paths (#39) and the choice is inferred from host +# state, so it has to be established BEFORE the asserts that depend on it. Either +# the operator supplies decdn_rpc_url from inventory (the role then authors the +# env file, overwriting whatever is on the host), or they provision a 0600 +# /etc/decdn/decdn.env on the target themselves and the secret never touches the +# control machine at all. +# +# `stat.exists` alone cannot tell those apart — a file the role itself wrote on a +# previous converge exists just as much as an operator-provisioned one. That +# ambiguity is dangerous in BOTH directions, so the checksum this role records +# after every successful converge is used as a provenance marker: +# record == file -> this role authored the file +# record missing -> provenance unknown (pre-#39 host, or first converge) +# record != file -> the file was changed by someone other than this role +# get_checksum is what makes that comparison possible; the value is a sha256 of a +# secret-bearing file, not the secret, and it never leaves the 0600 record. +- name: Check for a host-provisioned secret environment file + ansible.builtin.stat: + path: "{{ decdn_env_file }}" + get_checksum: true + checksum_algorithm: sha256 + register: decdn_env_pre_stat + +# Deliberately not `slurp`-ing the env file itself anywhere in this role — only +# this 0600 record of its hash, which is not sensitive. A missing record is the +# normal state on a host that has never run a post-#39 converge, so a read failure +# is an expected outcome rather than an error. +- name: Read the env-file checksum this role last recorded + ansible.builtin.slurp: + src: "{{ decdn_env_checksum_file }}" + register: decdn_env_recorded + failed_when: false + changed_when: false + +- name: Resolve the env file's provenance + ansible.builtin.set_fact: + # "" when no record exists — provenance unknown. + decdn_env_recorded_sum: "{{ decdn_env_recorded.content | default('') | b64decode | trim }}" + decdn_env_role_authored: >- + {{ (decdn_env_pre_stat.stat.checksum | default('') | length > 0) + and ((decdn_env_recorded.content | default('') | b64decode | trim) + == (decdn_env_pre_stat.stat.checksum | default(''))) }} + - name: Require chain + contract configuration ansible.builtin.assert: that: - - decdn_rpc_url | length > 0 - decdn_chain_id | int > 0 - decdn_payment_pool_address is match('^0x[0-9a-fA-F]{40}$') - decdn_capacity_bond_address is match('^0x[0-9a-fA-F]{40}$') @@ -74,7 +116,7 @@ - decdn_content_blacklist_address != decdn_zero_address - decdn_region is match('^[A-Z]{2}$') fail_msg: >- - decdn_rpc_url, a positive decdn_chain_id, the FOUR contract addresses + A positive decdn_chain_id, the FOUR contract addresses (0x + 40 hex, none the zero address), and an uppercase ISO-3166 2-letter decdn_region are required. decdn_content_blacklist_address joined this list upstream (ADR 011/031): the daemon now refuses to resolve a config without a @@ -85,6 +127,128 @@ vars: decdn_zero_address: "0x0000000000000000000000000000000000000000" +# The RPC endpoint gate. Mirrors the pre-start keystore gate below: the role does +# not insist on carrying the secret itself, it insists the secret EXISTS — either +# in inventory, or already on the host at 0600. Only "neither" is a failure. +# +# The provisioning commands below are deliberately heredoc-free and reference NO +# `decdn` user: this assert fires ~700 tasks before "Create decdn system user", so +# on a never-converged host that account does not exist yet and `install -o decdn` +# / `chown decdn:decdn` would fail. Creating the file as root is correct — "Lock +# down a host-provisioned env file" chowns it to {{ decdn_user }} on the converge +# that follows. (A heredoc cannot survive this message either: an indented `EOF` +# does not terminate `<<'EOF'`, and the block's own indentation would prefix the +# written line with spaces, which the DECDN_RPC_URL grep below then rejects.) +- name: Require an RPC endpoint — inventory value or a host-provisioned env file + ansible.builtin.assert: + that: + - decdn_rpc_url | length > 0 or decdn_env_pre_stat.stat.exists + fail_msg: |- + No RPC endpoint: decdn_rpc_url is empty and {{ decdn_env_file }} does not + exist on this host. The URL may embed a provider API key, so the preferred + posture is to provision it on the host and keep it off the control machine + entirely. Run this on the TARGET, as root (the {{ decdn_user }} account may + not exist yet — this role creates it, and chowns the file on the next run): + sudo mkdir -p {{ decdn_etc }} + echo 'DECDN_RPC_URL=https://…' | sudo tee {{ decdn_env_file }} >/dev/null + sudo chmod 600 {{ decdn_env_file }} + Add any other environment-borne secrets (e.g. AWS_ACCESS_KEY_ID / + AWS_SECRET_ACCESS_KEY for an S3 cache origin) as further KEY=value lines in + that same file — see roles/decdn_node/files/decdn.env.example. This role + then leaves the file's CONTENT alone and only enforces 0600 ownership. + Alternatively set decdn_rpc_url (plus any decdn_extra_env) in the git-ignored + host_vars//secret.yml and the role will author the file for you. + +# stat.exists is true for a directory or a symlink. Both are hazards, not merely +# unsupported: "Lock down a host-provisioned env file" runs `file` as root, and +# with the module's default follow:true a symlink planted by a compromised +# {{ decdn_user }} account (which owns the 0750 {{ decdn_etc }}) would have root +# chown+chmod the TARGET — /etc/shadow, say. A symlink also has no stat.checksum, +# which would surface much later as an undefined-variable crash. Refuse early. +- name: Require the env file to be a regular file + ansible.builtin.assert: + that: + - decdn_env_pre_stat.stat.isreg | default(false) + fail_msg: >- + {{ decdn_env_file }} exists but is not a regular file + (symlink/directory/device). This role chowns and chmods that path as root, + so following a link out of {{ decdn_etc }} would apply 0600 {{ decdn_user }} + ownership to whatever it points at. Replace it with a real file. To keep the + secret on other storage, bind-mount it there rather than symlinking. + when: decdn_env_pre_stat.stat.exists + +# The dangerous half of inferring the path from host state. An empty decdn_rpc_url +# is meant to say "the operator provisioned this file" — but it also happens when +# the git-ignored secret.yml is simply ABSENT: a fresh clone, a new control +# machine, a CI runner, an undecrypted vault. Without this check the role would +# then adopt its own previously-written file as though it were operator-supplied, +# and a ROTATED url in the missing secret.yml would silently never land: the grep +# finds the role's own stale DECDN_RPC_URL line, the checksum is unchanged, no +# restart is queued, and the deploy is green. Before #39 that state failed loud at +# "Require chain + contract configuration"; this restores that. +- name: Refuse to adopt a role-authored env file as host-provisioned + ansible.builtin.assert: + that: + - not decdn_env_role_authored | bool + fail_msg: >- + decdn_rpc_url is empty, but {{ decdn_env_file }} is byte-for-byte the file + THIS ROLE last wrote from inventory — so the value has gone missing from the + control machine (an absent host_vars//secret.yml, a fresh clone, an + undecrypted vault) rather than being deliberately handed to the host. + Treating it as host-provisioned would deploy green while silently keeping the + old endpoint. Restore decdn_rpc_url, or — if you really are migrating this + node to the host-provisioned path — hand the file over explicitly by + discarding the role's provenance record: + sudo rm {{ decdn_env_checksum_file }} + when: decdn_rpc_url | length == 0 + +# The mirror direction, and the reason it is an assert rather than the `debug` it +# started as: the role now actively tells operators to keep AWS credentials in +# this file, and an inventory decdn_rpc_url rewrites it WHOLESALE. Losing those +# lines produces exactly the outcome "Reject decdn_extra_env" exists to prevent — +# an S3 403 hours later behind a green deploy — so it gets the same treatment. +# Scoped to a file whose provenance is KNOWN to be foreign: with no record (a +# pre-#39 host's first converge) this cannot distinguish anything, so it stays +# quiet and the run seeds the record for next time. +- name: Refuse to overwrite an env file this role did not write + ansible.builtin.assert: + that: + - decdn_env_overwrite_host_file | bool + fail_msg: >- + decdn_rpc_url is set, but {{ decdn_env_file }} on this host does not match + the checksum this role last recorded — it was written or edited by someone + else. Rendering it from inventory would DISCARD those lines (any + AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY behind an S3 origin included), and + because this role never reads the file it cannot tell you what would be + lost. Clear decdn_rpc_url to keep the host's copy, fold the host's extra + lines into decdn_extra_env, or set decdn_env_overwrite_host_file: true to + confirm the file is the role's to author. + when: + - decdn_rpc_url | length > 0 + - decdn_env_pre_stat.stat.exists + - decdn_env_recorded_sum | length > 0 + - not decdn_env_role_authored | bool + +# The env file is authored ALL-OR-NOTHING from inventory: the copy below renders +# DECDN_RPC_URL *and* decdn_extra_env in one go, and it is skipped entirely when +# decdn_rpc_url is empty. So extra_env alongside a host-provisioned file would be +# silently dropped — surfacing hours later as an S3 403 with nothing in the deploy +# output to explain it (AGENTS.md hard-rule 4). Fail loud instead. +- name: Reject decdn_extra_env when the env file is host-provisioned + ansible.builtin.assert: + that: + - decdn_extra_env | default({}, true) | length == 0 + fail_msg: >- + decdn_extra_env is set but decdn_rpc_url is empty, so this role will NOT + write the host-provisioned {{ decdn_env_file }} and those variables would + never reach the daemon. Append them as KEY=value lines to + {{ decdn_env_file }} on the host instead, or move BOTH decdn_rpc_url and + decdn_extra_env into host_vars//secret.yml so the role authors the + whole file. + when: + - decdn_rpc_url | length == 0 + - decdn_env_pre_stat.stat.exists + # slash_appeal_address is optional (only `decdn appeal slash` uses it — the daemon # validates but ignores the key), so it is not in the hard assert above. But if an # operator sets it, a half-filled/zero value would silently render a bad node.toml, @@ -261,8 +425,9 @@ decdn_cache_origin_s3_use_default_chain must be a real boolean, and decdn_cache_origin_s3_profile (when set) must contain no quote, whitespace or backslash. Static AWS keys are never written to node.toml — put - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN in - decdn_extra_env (git-ignored secret.yml) and leave the default chain to read them. + AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN in the 0600 + {{ decdn_env_file }} on the host (or in decdn_extra_env, if the role authors + that file) and leave the default chain to read them. - name: Reject S3 credential knobs alongside the multi-origin form ansible.builtin.assert: @@ -274,8 +439,8 @@ only to the single [cache.origin] table — the template does not render a credentials sub-table for [[cache.origins]] entries, so setting them with decdn_cache_origins would SILENTLY drop them. Omit the credentials block (the - AWS default chain is used anyway when it is absent) and supply keys via - decdn_extra_env, or switch to the single-origin form. + AWS default chain is used anyway when it is absent) and supply keys via the + 0600 {{ decdn_env_file }}, or switch to the single-origin form. when: decdn_cache_origins | length > 0 # --- Optional tuning knobs ("" => omit + use the daemon default) -------------- @@ -912,7 +1077,16 @@ # The RPC URL is the one sensitive value (it may embed an API key), so it lives # in a 0600 EnvironmentFile (DECDN_RPC_URL, read by the binary's clap env) rather # than node.toml. That keeps node.toml non-secret and diffable under `--check`. -- name: Write the secret environment file +# +# Two authoring paths, decided by the gate near the top of this file (#39): +# * decdn_rpc_url set -> the role authors the file from inventory (below). The +# value transits the control machine on every converge and OVERWRITES any +# host-provisioned file. +# * decdn_rpc_url empty -> the operator wrote the file on the host. The role +# never reads its content back (that would put the secret on the control +# machine, which is the whole point of the host path); it only enforces +# ownership + 0600 and greps for a non-empty DECDN_RPC_URL line. +- name: Write the secret environment file (inventory-supplied) ansible.builtin.copy: dest: "{{ decdn_env_file }}" owner: "{{ decdn_user }}" @@ -924,7 +1098,120 @@ {{ k }}="{{ v | string | replace('\\', '\\\\') | replace('"', '\\"') }}" {% endfor %} no_log: true # the RPC URL may embed an API key + register: decdn_env_copy notify: Restart decdn-node + when: decdn_rpc_url | length > 0 + +# The copy above is no_log, so a run that replaced an existing file with +# control-machine values shows a bare "changed" and nothing else. Say what was +# lost. A file with a KNOWN-foreign provenance already failed the assert at the +# top of this file, so what is left here is the unknowable case: no record yet +# (a pre-#39 host's first converge). Gated on `is changed` as well, so it speaks +# up only when the contents actually moved rather than nagging every converge. +- name: Warn that an inventory decdn_rpc_url overwrote an untracked env file + ansible.builtin.debug: + msg: >- + decdn_rpc_url is set in inventory, so {{ decdn_env_file }} was REWRITTEN + from control-machine values, and this host carried no provenance record for + it — any KEY=value lines added on the host that are not in decdn_extra_env + are gone. From this converge on, that file is tracked: an edit made outside + Ansible will fail the deploy instead of being silently discarded. Clear + decdn_rpc_url (and decdn_extra_env) to hand the file back to the host. + when: + - decdn_rpc_url | length > 0 + - decdn_env_pre_stat.stat.exists + - decdn_env_recorded_sum | length == 0 + - decdn_env_copy is changed + +# Content untouched (no `content:`/`src:`) — this only asserts the file is owned +# by the daemon user and unreadable by anyone else, the same lock-down the +# operator-provisioned keystore gets above. +- name: Lock down a host-provisioned env file (ownership + mode only) + ansible.builtin.file: + path: "{{ decdn_env_file }}" + owner: "{{ decdn_user }}" + group: "{{ decdn_group }}" + mode: "0600" + when: decdn_rpc_url | length == 0 + +# `decdn config validate` further down sources this file, so a missing +# DECDN_RPC_URL would eventually surface — but as an opaque daemon error about an +# unresolvable config. Name the actual cause here instead. grep -q keeps the +# secret out of stdout AND out of argv; deliberately NOT slurp, which would copy +# the secret to the control machine and defeat the point of the host path. +- name: Confirm a host-provisioned env file carries a non-empty DECDN_RPC_URL + ansible.builtin.command: + cmd: grep -qE '^[[:space:]]*DECDN_RPC_URL=[^[:space:]]' {{ decdn_env_file | quote }} + register: decdn_env_grep + changed_when: false + # Read-only, so it must still run under `make check` — otherwise the one gate + # that can catch a malformed host file is exactly the gate a dry run skips. + check_mode: false + # Non-fatal here so the next task can turn rc into an actionable message + # (mirrors the config-validation pair below). + failed_when: false + when: decdn_rpc_url | length == 0 + +- name: Report a host-provisioned env file with no usable DECDN_RPC_URL + ansible.builtin.fail: + msg: >- + {% if decdn_env_grep.rc is not defined %} + The {{ decdn_env_file }} content check could not be RUN + ({{ decdn_env_grep.msg | default('no rc and no msg — re-run with -vvv', true) }}), + so the host-provisioned env file is UNVERIFIED and the daemon must not be + started on it. + {% elif decdn_env_grep.rc | int > 1 %} + {{ decdn_env_file }} could not be read (grep exited + {{ decdn_env_grep.rc }}: {{ decdn_env_grep.stderr | default('no stderr', true) }}). + {% else %} + {{ decdn_env_file }} exists but carries no `DECDN_RPC_URL=` line with a + non-whitespace value, + so the daemon would start with no chain endpoint. Add one — note systemd + SKIPS a line its shell-like parser cannot read, logging a warning to + `journalctl -u decdn-node` that is easy to miss, so single-quote a value + containing a space, quote, backslash or `$`: + echo 'DECDN_RPC_URL=https://your-endpoint.example/rpc' | sudo tee -a + {{ decdn_env_file }} >/dev/null + {% endif %} + when: + - decdn_rpc_url | length == 0 + - decdn_env_grep.rc | default(1) != 0 + +# Re-stat after the file is final (the inventory path may just have rewritten it) +# so the comparison below, and the record written at the very end of this role, +# both describe what the daemon is about to be started on. +# +# Not no_log: the value is a sha256, not the secret, and hiding it would censor a +# module failure here into an unactionable "output has been hidden" — while the +# `changed` it drives is the only explanation an operator gets for a restart. +- name: Re-stat the env file now that it is final + ansible.builtin.stat: + path: "{{ decdn_env_file }}" + get_checksum: true + checksum_algorithm: sha256 + register: decdn_env_post_stat + +# Restart on an OUT-OF-BAND edit. The copy above already notifies when the role +# itself authors a changed file, so this covers only the host-provisioned path, +# where nothing else would notice that the operator edited decdn.env between +# converges — the daemon would keep running on the previous values behind a fully +# green deploy. Handlers dedupe and flush before the daemon is started, so this +# costs at most one restart. +# +# An ABSENT record is deliberately not a change: it means this host has never run +# a post-#39 converge, and treating "not tracked yet" as "edited" would bounce +# every node in the fleet on the upgrade run for no functional reason. +- name: Note an out-of-band change to the host-provisioned env file + ansible.builtin.debug: + msg: >- + {{ decdn_env_file }} has changed since the last converge recorded it — + restarting decdn-node so the daemon picks the new values up. + changed_when: true + notify: Restart decdn-node + when: + - decdn_rpc_url | length == 0 + - decdn_env_recorded_sum | length > 0 + - (decdn_env_post_stat.stat.checksum | default('')) != decdn_env_recorded_sum - name: Template node.toml (non-secret — RPC URL is in the env file) ansible.builtin.template: @@ -956,8 +1243,13 @@ executable: /bin/bash cmd: | set -euo pipefail + # Exit 3 is reserved for "the env file could not be sourced" so the report + # task below does not blame node.toml for a quoting typo in decdn.env. NOTE + # this is bash, not systemd's parser: it expands $VAR and executes $(...) + # and backticks, which systemd would pass through literally. Operators are + # told to single-quote values (files/decdn.env.example). set -a - . {{ decdn_env_file | quote }} + . {{ decdn_env_file | quote }} || { echo "env-file could not be sourced" >&2; exit 3; } set +a {{ decdn_cli_bin | quote }} config validate \ --config {{ decdn_config_file | quote }} \ @@ -980,7 +1272,13 @@ - name: Report the config-validation failure ansible.builtin.fail: msg: >- - {% if decdn_config_validate.rc is not defined %} + {% if decdn_config_validate.rc | default(0) == 3 %} + {{ decdn_env_file }} could not be sourced, so the config was never + validated: {{ decdn_config_validate.stderr | default('(no output)', true) }}. + This is a shell-syntax fault in the ENV FILE, not a problem with + {{ decdn_config_file }} — most often an unbalanced quote, or an unquoted + value containing a space. Single-quote the value and re-run. + {% elif decdn_config_validate.rc is not defined %} `decdn config validate` could not be RUN ({{ decdn_config_validate.msg | default('no rc and no msg — re-run with -vvv', true) }}). {{ decdn_config_file }} is therefore UNVALIDATED and the daemon must not be @@ -1022,6 +1320,34 @@ enabled: true state: started +# Record the env file's hash LAST, and only here — deliberately after the handler +# flush and the start above, so the record means "the daemon is running on this +# content", not "Ansible saw this content". +# +# Writing it earlier (next to the comparison) looks equivalent and is not: the +# handler that acts on a detected change does not flush until well after the +# node.toml template and the `decdn config validate` gate, both of which can abort +# the play. A record written before that abort would say "already handled" on the +# NEXT converge — the comparison would find no change, no restart would be queued, +# and `state: started` is a no-op on a running unit — so a single failed run would +# permanently burn the one signal that the secret had been rotated. Writing it +# here means an aborted run simply leaves the old record in place and the next +# converge re-detects the change. +# +# No notify: by the time this runs, any restart this converge owed has happened. +- name: Record the env-file checksum (the daemon is now running on it) + ansible.builtin.copy: + dest: "{{ decdn_env_checksum_file }}" + owner: root + group: root + mode: "0600" + content: "{{ decdn_env_post_stat.stat.checksum }}\n" + # Under `--check` on an unprovisioned host the env-file copy only SIMULATES the + # write, so there is nothing to hash and stat.checksum is undefined. Requiring + # the checksum itself (not merely stat.exists) also keeps a non-regular file — + # which stat reports as existing but never checksums — from templating undefined. + when: (decdn_env_post_stat.stat.checksum | default('')) | length > 0 + # --- Readiness ---------------------------------------------------------------- # ADVISORY probe: the daemon binds its loopback /metrics listener only after its # startup bootstrap completes, which on a healthy node can lag process start by a diff --git a/ansible/roles/decdn_node/templates/node.toml.j2 b/ansible/roles/decdn_node/templates/node.toml.j2 index b0715f5..1d3bcb8 100644 --- a/ansible/roles/decdn_node/templates/node.toml.j2 +++ b/ansible/roles/decdn_node/templates/node.toml.j2 @@ -355,7 +355,9 @@ decompress = "{{ o.decompress }}" S3 credentials. Only the `default-chain` source is templated: `static` would put an access key into this 0640, diffable file. Supply static keys as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN in the 0600 - {{ decdn_env_file }} instead — the default chain reads exactly those. + {{ decdn_env_file }} instead — the default chain reads exactly those. Either + write them there on the host, or (when the role authors that file, i.e. + decdn_rpc_url is set) pass them as decdn_extra_env. #} {% if not _multi and decdn_cache_origin_kind == 's3' and decdn_cache_origin_s3_use_default_chain | bool %} From 7df856ac3e85d3d237743030dbc8bccc680a664b Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 3 Sep 2026 03:51:38 +0300 Subject: [PATCH 2/3] fix(decdn_node): correct env-file escaping and make the provenance record explicit Two bugs, both found by the tests added in the previous commit. 1. The env file's escaping was inverted, and had been since before this branch. Inside a YAML literal block Jinja reads '\\' as TWO literal backslashes, so the role's replace('\\', '\\\\') | replace('"', '\\"') chain never matched a lone backslash and rendered a double quote as \\" -- an escaped backslash followed by a CLOSING quote. Any value containing a quote was truncated there, any backslash passed through unescaped, and systemd then skipped the line it could not parse, leaving the variable absent at runtime. An S3 secret key containing either character would 403 on the first cache miss with nothing in the deploy output to explain it. Both decdn_extra_env values and DECDN_RPC_URL (previously interpolated raw and unquoted) now render through to_json(ensure_ascii=false), which emits exactly the double-quoted, C-escaped form EnvironmentFile expects. ensure_ascii=false matters: systemd understands \xNN, not \uNNNN. Verified round-tripping quotes, backslashes, spaces, '#' and UTF-8. This was invisible because no molecule scenario had ever set decdn_extra_env; the default scenario now does, with a value carrying both a quote and a backslash. 2. The provenance record could not actually establish provenance. It held only a hash, and it is written on BOTH authoring paths -- so "record matches file" means "unchanged since the last converge", not "this role authored it". The new adopt guard therefore fired on the second converge of a legitimately host-provisioned node: exactly the configuration it exists to protect. host-env's idempotence step caught it. The record is now " ", source being host or inventory, which makes both guards precise: an empty decdn_rpc_url against a file recorded as inventory and unchanged since means the value went missing from the control machine, and a set decdn_rpc_url against a file recorded as host (or one whose hash has moved) means an operator edit is about to be discarded. Both verifies now assert the source token, not just the hash. Also fixes Jinja filter precedence in the record's ternary -- filters bind tighter than comparisons, so an unbracketed `decdn_rpc_url | length > 0 | ternary(...)` parses as (decdn_rpc_url | length) > (0 | ternary(...)) and would have written a nonsense source token. molecule test --all: all six scenarios pass, failed=0. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WaE3ZP2xQdjd9juzRYKLwi --- ansible/galaxy/CHANGELOG.md | 22 ++++++- ansible/molecule/default/verify.yml | 30 +++++++--- ansible/molecule/host-env/verify.yml | 18 ++++-- ansible/roles/decdn_node/README.md | 37 ++++++++---- ansible/roles/decdn_node/defaults/main.yml | 12 ++-- ansible/roles/decdn_node/tasks/main.yml | 70 +++++++++++++++------- 6 files changed, 134 insertions(+), 55 deletions(-) diff --git a/ansible/galaxy/CHANGELOG.md b/ansible/galaxy/CHANGELOG.md index 14c8585..9a8a90f 100644 --- a/ansible/galaxy/CHANGELOG.md +++ b/ansible/galaxy/CHANGELOG.md @@ -42,8 +42,8 @@ Not yet published to Galaxy (pre-1.0; the published shape may still change). it. A template for the host file ships at `roles/decdn_node/files/decdn.env.example`. - `decdn_env_checksum_file` (default `/etc/decdn/.decdn.env.sha256`, `0600 root`) — - a role-managed record of the env file's hash, written on both authoring paths - after the daemon is running. It restarts the unit when an **out-of-band** edit of + a role-managed ` ` record for the env file, written on both + authoring paths after the daemon is running. It restarts the unit when an **out-of-band** edit of a host-provisioned `decdn.env` is detected (instead of leaving the daemon on stale values behind a green deploy), and doubles as the file's provenance marker: an empty `decdn_rpc_url` against a file the role itself wrote fails loud (a missing @@ -67,6 +67,24 @@ Not yet published to Galaxy (pre-1.0; the published shape may still change). keystore is also absent), never overwriting an existing wallet. Funding + on-chain staking/registration remain a manual step. +### Fixed + +- **`decdn_extra_env` values were escaped incorrectly** in the `0600` env file, and + `DECDN_RPC_URL` was not escaped at all. Inside a YAML literal block Jinja reads + `'\\'` as *two* literal backslashes, so the role's hand-rolled + `replace('\\', '\\\\') | replace('"', '\\"')` chain never matched a lone backslash + and rendered a double quote as `\\"` — an escaped backslash followed by a + *closing* quote. Any value containing a quote was truncated there and any + backslash was passed through unescaped; systemd then skipped the line it could + not parse and the variable was simply **absent** at runtime (an S3 origin whose + secret key contained a quote or backslash would 403 with nothing to show why). + Both are now rendered with `to_json(ensure_ascii=false)`, which emits exactly the + double-quoted, C-escaped form `EnvironmentFile=` expects. Values containing a + space, `#`, quote, backslash or non-ASCII character now survive intact. + **Upgrade note:** this changes the rendered file (`DECDN_RPC_URL` gains quotes), + so the first converge on the inventory path rewrites it and restarts the daemon + once. + ### Changed (BREAKING) The collection has never been published, so this is not a break against any diff --git a/ansible/molecule/default/verify.yml b/ansible/molecule/default/verify.yml index 47f312f..fc6641a 100644 --- a/ansible/molecule/default/verify.yml +++ b/ansible/molecule/default/verify.yml @@ -64,16 +64,19 @@ - name: Assert the env file renders the RPC URL and an escaped decdn_extra_env ansible.builtin.assert: that: - - "'DECDN_RPC_URL=https://rpc.example.invalid/' in env_body" - # Exact escaped form: the template wraps extra_env values in double - # quotes and escapes backslash-then-quote. A regression here is silent — - # systemd drops the unparseable line and the variable is simply absent. + - env_rpc_expected in env_body + # Exact escaped form. This caught a real inversion in the original + # hand-rolled replace() chain, which rendered a quote as \\" — an escaped + # backslash plus a CLOSING quote — truncating the value at the first + # quote and leaving backslashes unescaped. The failure is silent: + # systemd drops the line it cannot parse and the variable is absent. - env_expected in env_body fail_msg: >- {{ decdn_etc }}/decdn.env is missing the RPC URL or did not escape the decdn_extra_env value correctly. Got: {{ env_body }} vars: env_body: "{{ decdn_env_body.content | b64decode }}" + env_rpc_expected: 'DECDN_RPC_URL="https://rpc.example.invalid/"' env_expected: 'DECDN_MOLECULE_EXTRA="a \"quoted\" \\ value"' - name: Compute the env file's current sha256 @@ -88,14 +91,23 @@ src: "{{ decdn_etc }}/.decdn.env.sha256" register: decdn_env_recorded - - name: Assert the recorded checksum describes the env file the role wrote + - name: Assert the record names the inventory path and the file's real hash ansible.builtin.assert: that: - - (decdn_env_recorded.content | b64decode | trim) == decdn_env_hash.stat.checksum + - record_fields | length == 2 + # The source token is not derivable from the hash and is what stops a + # later converge with an empty decdn_rpc_url from adopting this + # role-authored file as though the operator owned it. + - record_fields[0] == 'inventory' + - record_fields[1] == decdn_env_hash.stat.checksum fail_msg: >- - The recorded env-file checksum does not match {{ decdn_etc }}/decdn.env on - the inventory-authored path, so the next converge would either restart the - daemon for no reason or miss a real change." + Expected "inventory " in + {{ decdn_etc }}/.decdn.env.sha256, got + "{{ decdn_env_recorded.content | b64decode | trim }}". A wrong hash makes + the next converge restart for no reason or miss a real change; a wrong + source token breaks the provenance guards. + vars: + record_fields: "{{ (decdn_env_recorded.content | b64decode | trim).split() }}" # prepare.yml stages the keystore + password at 0644; the role is responsible # for locking them to 0600. Asserting 0600 here therefore verifies the role's diff --git a/ansible/molecule/host-env/verify.yml b/ansible/molecule/host-env/verify.yml index 8c443bb..bb08305 100644 --- a/ansible/molecule/host-env/verify.yml +++ b/ansible/molecule/host-env/verify.yml @@ -95,13 +95,23 @@ src: "{{ decdn_etc }}/.decdn.env.sha256" register: decdn_env_recorded - - name: Assert the recorded checksum matches the env file on disk + - name: Assert the record names the HOST path and the file's real hash ansible.builtin.assert: that: - - (decdn_env_recorded.content | b64decode | trim) == decdn_env_hash.stat.checksum + - record_fields | length == 2 + # Must say "host": recorded as "inventory" here, the next converge would + # read an empty decdn_rpc_url plus a matching hash as "the value went + # missing from inventory" and fail a perfectly good node. + - record_fields[0] == 'host' + - record_fields[1] == decdn_env_hash.stat.checksum fail_msg: >- - The recorded env-file checksum does not match {{ decdn_etc }}/decdn.env, - so the restart-on-out-of-band-edit signal is wrong in both directions. + Expected "host " in + {{ decdn_etc }}/.decdn.env.sha256, got + "{{ decdn_env_recorded.content | b64decode | trim }}". A wrong hash makes + the restart-on-out-of-band-edit signal wrong in both directions; a wrong + source token makes the next converge fail this node. + vars: + record_fields: "{{ (decdn_env_recorded.content | b64decode | trim).split() }}" # node.toml must carry the S3 origin with source = "default-chain" and NO # static keys: the credentials live only in the 0600 env file. diff --git a/ansible/roles/decdn_node/README.md b/ansible/roles/decdn_node/README.md index 036068f..61fc34d 100644 --- a/ansible/roles/decdn_node/README.md +++ b/ansible/roles/decdn_node/README.md @@ -281,24 +281,35 @@ but **not** to `decdn_rpc_url` itself, which is rendered raw. ### The provenance record -`/etc/decdn/.decdn.env.sha256` (`0600 root`, `decdn_env_checksum_file`) holds the -env file's sha256 as of the last converge that got the daemon running. It is -written on **both** paths, and it is what lets the role tell the two apart — a -bare `stat` cannot, since a file the role wrote last week exists exactly as much -as an operator-provisioned one. From it the role gets three behaviours: +`/etc/decdn/.decdn.env.sha256` (`0600 root`, `decdn_env_checksum_file`) holds +` ` — which path wrote the file, and what its contents hashed to — +as of the last converge that got the daemon running: + +```text +host 3b1f…c7 # the operator owns /etc/decdn/decdn.env +inventory 9a4e…21 # this role rendered it from decdn_rpc_url +``` + +It is written on **both** paths, which is exactly why the `source` token is needed +and cannot be inferred from the hash: a matching hash only means "unchanged since +the last converge". A bare `stat` tells you even less — a file the role wrote last +week exists exactly as much as an operator-provisioned one. From this the role gets +three behaviours: - **Restart on an out-of-band edit.** Record ≠ file on the host path ⇒ the unit is restarted so the daemon picks up the new values. The record is written *after* the restart, so a run that aborts in between leaves the old value in place and the next converge re-detects — the signal is never burned by a failed deploy. - **"`secret.yml` went missing" ≠ "the host owns this now."** An empty - `decdn_rpc_url` against a file the role itself wrote is treated as a lost - inventory value and fails loud, instead of silently deploying a stale endpoint. - To migrate a node to the host path deliberately, `sudo rm` the record once. -- **No silent clobbering.** An inventory `decdn_rpc_url` against a file the record - says someone else wrote fails loud rather than discarding those lines; set - `decdn_env_overwrite_host_file: true` to confirm. With **no** record (a host - that predates this) the role cannot tell, so it warns and starts tracking. + `decdn_rpc_url` against a file recorded as `inventory` *and* unchanged since is + treated as a lost inventory value and fails loud, instead of silently deploying a + stale endpoint. To migrate a node to the host path deliberately, `sudo rm` the + record once — that is the explicit hand-over. +- **No silent clobbering.** An inventory `decdn_rpc_url` against a file recorded as + `host`, or one whose hash no longer matches, fails loud rather than discarding + those lines; set `decdn_env_overwrite_host_file: true` to confirm. With **no** + record (a host that predates this) the role cannot tell, so it warns and starts + tracking. If neither authoring path is satisfied the role fails before touching the host, with the provisioning commands above in the failure message. @@ -441,7 +452,7 @@ additionally requires on-chain stake + registration (see - `/usr/local/bin/decdn-node`, `/usr/local/bin/decdn` — daemon + CLI - `/etc/decdn/node.toml` (`0640 decdn`) — rendered config; non-secret (no `rpc_url`), so it stays `--check`-diffable - `/etc/decdn/decdn.env` (`0600 decdn`) — the one operator-supplied secret, `DECDN_RPC_URL` (+ any extra env vars), read by the unit via `EnvironmentFile`; role-authored or host-provisioned ([Secrets](#secrets)) -- `/etc/decdn/.decdn.env.sha256` (`0600 root`) — role-managed hash of the above: provenance marker, and the trigger for a restart after an out-of-band edit. Not a secret store +- `/etc/decdn/.decdn.env.sha256` (`0600 root`) — role-managed ` ` record of the above: provenance marker, and the trigger for a restart after an out-of-band edit. Not a secret store - `/etc/decdn/keystore.password` (`0600 decdn`) — operator-provisioned - `/var/lib/decdn/` (`0700 decdn`) — `node.secret`, `keystore.json`, `cache/`, state - `/etc/systemd/system/decdn-node.service` — hardened unit diff --git a/ansible/roles/decdn_node/defaults/main.yml b/ansible/roles/decdn_node/defaults/main.yml index b34b8d1..2687cc4 100644 --- a/ansible/roles/decdn_node/defaults/main.yml +++ b/ansible/roles/decdn_node/defaults/main.yml @@ -310,12 +310,12 @@ decdn_home: /var/lib/decdn # data dir: node.secret + keystore.json decdn_etc: /etc/decdn decdn_config_file: /etc/decdn/node.toml decdn_env_file: /etc/decdn/decdn.env # DECDN_RPC_URL (sensitive) — 0600 -# Role-managed record of the env file's sha256 as of the last converge that got -# the daemon running. NOT a secret store (it holds one hash, 0600 root:root). It -# does two jobs: it lets an OUT-OF-BAND edit of a host-provisioned decdn.env -# notify the restart handler, and it is the file's provenance marker — matching -# means this role authored it, differing means someone else did, and absent means -# the host predates this mechanism. Written on BOTH authoring paths. +# Role-managed " " record for the env file, as of the last +# converge that got the daemon running. NOT a secret store (0600 root:root, one +# hash and one word). It does two jobs: it lets an OUT-OF-BAND edit of a +# host-provisioned decdn.env notify the restart handler, and it records WHICH PATH +# wrote the file. Written on BOTH paths, which is why the source token exists — a +# matching hash alone only proves the file is unchanged, not who authored it. decdn_env_checksum_file: /etc/decdn/.decdn.env.sha256 # Confirm that rendering the env file from inventory may DISCARD a host-side edit. # Only consulted when the recorded checksum proves the file on the host is not the diff --git a/ansible/roles/decdn_node/tasks/main.yml b/ansible/roles/decdn_node/tasks/main.yml index ddcdeb8..c4a3d6c 100644 --- a/ansible/roles/decdn_node/tasks/main.yml +++ b/ansible/roles/decdn_node/tasks/main.yml @@ -68,13 +68,17 @@ # # `stat.exists` alone cannot tell those apart — a file the role itself wrote on a # previous converge exists just as much as an operator-provisioned one. That -# ambiguity is dangerous in BOTH directions, so the checksum this role records -# after every successful converge is used as a provenance marker: -# record == file -> this role authored the file -# record missing -> provenance unknown (pre-#39 host, or first converge) -# record != file -> the file was changed by someone other than this role -# get_checksum is what makes that comparison possible; the value is a sha256 of a -# secret-bearing file, not the secret, and it never leaves the 0600 record. +# ambiguity is dangerous in BOTH directions, so after every converge that gets the +# daemon running the role records WHICH PATH wrote the file and what its contents +# hashed to, as " " in decdn_env_checksum_file. The source token is +# load-bearing and not derivable from the hash: the record is written on both +# paths, so a matching hash means only "unchanged since the last converge". +# source=host -> the operator owns this file +# source=inventory -> this role rendered it from decdn_rpc_url +# no record -> provenance unknown (a host predating this mechanism) +# hash mismatch -> the file changed since that converge +# get_checksum is what makes the hash comparison possible; the value is a sha256 of +# a secret-bearing file, not the secret. - name: Check for a host-provisioned secret environment file ansible.builtin.stat: path: "{{ decdn_env_file }}" @@ -93,14 +97,16 @@ failed_when: false changed_when: false +# Both fields default to "" — treated exactly like "no record", which is also the +# right answer for a truncated or older single-field record: warn, do not fail. - name: Resolve the env file's provenance ansible.builtin.set_fact: - # "" when no record exists — provenance unknown. - decdn_env_recorded_sum: "{{ decdn_env_recorded.content | default('') | b64decode | trim }}" - decdn_env_role_authored: >- - {{ (decdn_env_pre_stat.stat.checksum | default('') | length > 0) - and ((decdn_env_recorded.content | default('') | b64decode | trim) - == (decdn_env_pre_stat.stat.checksum | default(''))) }} + decdn_env_record_source: >- + {{ ((decdn_env_recorded.content | default('') | b64decode | trim).split() | length == 2) + | ternary((decdn_env_recorded.content | default('') | b64decode | trim).split()[0], '') }} + decdn_env_recorded_sum: >- + {{ ((decdn_env_recorded.content | default('') | b64decode | trim).split() | length == 2) + | ternary((decdn_env_recorded.content | default('') | b64decode | trim).split()[1], '') }} - name: Require chain + contract configuration ansible.builtin.assert: @@ -189,7 +195,8 @@ - name: Refuse to adopt a role-authored env file as host-provisioned ansible.builtin.assert: that: - - not decdn_env_role_authored | bool + - decdn_env_record_source != 'inventory' + or decdn_env_recorded_sum != (decdn_env_pre_stat.stat.checksum | default('')) fail_msg: >- decdn_rpc_url is empty, but {{ decdn_env_file }} is byte-for-byte the file THIS ROLE last wrote from inventory — so the value has gone missing from the @@ -215,9 +222,11 @@ that: - decdn_env_overwrite_host_file | bool fail_msg: >- - decdn_rpc_url is set, but {{ decdn_env_file }} on this host does not match - the checksum this role last recorded — it was written or edited by someone - else. Rendering it from inventory would DISCARD those lines (any + decdn_rpc_url is set, but {{ decdn_env_file }} on this host is not a file + this role authored — the provenance record reads + "{{ decdn_env_record_source | default('', true) }}" and/or its checksum no + longer matches, so it was written or edited by someone else. Rendering it + from inventory would DISCARD those lines (any AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY behind an S3 origin included), and because this role never reads the file it cannot tell you what would be lost. Clear decdn_rpc_url to keep the host's copy, fold the host's extra @@ -227,7 +236,9 @@ - decdn_rpc_url | length > 0 - decdn_env_pre_stat.stat.exists - decdn_env_recorded_sum | length > 0 - - not decdn_env_role_authored | bool + - >- + decdn_env_record_source != 'inventory' + or decdn_env_recorded_sum != (decdn_env_pre_stat.stat.checksum | default('')) # The env file is authored ALL-OR-NOTHING from inventory: the copy below renders # DECDN_RPC_URL *and* decdn_extra_env in one go, and it is skipped entirely when @@ -1092,10 +1103,20 @@ owner: "{{ decdn_user }}" group: "{{ decdn_group }}" mode: "0600" + # to_json, NOT hand-rolled replace() chains. Inside a YAML literal block Jinja + # takes '\\' as TWO literal backslashes, so the obvious + # `replace('\\', '\\\\') | replace('"', '\\"')` silently escapes the wrong + # thing: it never matches a lone backslash, and it renders a quote as \\" — + # an escaped backslash followed by a CLOSING quote, truncating the value. + # to_json emits exactly the double-quoted, C-escaped form systemd's + # EnvironmentFile parser expects, and ensure_ascii=false keeps UTF-8 literal + # (systemd understands \xNN, not \uNNNN). It supplies the quotes itself, so a + # value with a space, '#', quote or backslash survives intact — the RPC URL + # included, which used to be interpolated raw. content: | - DECDN_RPC_URL={{ decdn_rpc_url }} + DECDN_RPC_URL={{ decdn_rpc_url | string | to_json(ensure_ascii=false) }} {% for k, v in (decdn_extra_env | default({}, true)) | dictsort %} - {{ k }}="{{ v | string | replace('\\', '\\\\') | replace('"', '\\"') }}" + {{ k }}={{ v | string | to_json(ensure_ascii=false) }} {% endfor %} no_log: true # the RPC URL may embed an API key register: decdn_env_copy @@ -1341,7 +1362,14 @@ owner: root group: root mode: "0600" - content: "{{ decdn_env_post_stat.stat.checksum }}\n" + # " " — the source token is what a later converge uses to tell + # "the operator owns this file" from "I rendered this from inventory". + # Parenthesised: filters bind tighter than comparisons, so an unbracketed + # `decdn_rpc_url | length > 0 | ternary(...)` parses as + # (decdn_rpc_url | length) > (0 | ternary(...)). + content: > + {{ (decdn_rpc_url | length > 0) | ternary('inventory', 'host') }} + {{ decdn_env_post_stat.stat.checksum }} # Under `--check` on an unprovisioned host the env-file copy only SIMULATES the # write, so there is nothing to hash and stat.checksum is undefined. Requiring # the checksum itself (not merely stat.exists) also keeps a non-regular file — From 440a582efa4e86913438e73698aeb518cd7a6138 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Thu, 3 Sep 2026 03:53:56 +0300 Subject: [PATCH 3/3] fix(decdn_node): umask before writing decdn.env, and name the opt-in assert honestly Two review findings on #41. `sudo tee` followed by `chmod 600` leaves a window in which a secret-bearing file carries the caller's umask, so every copy of the provisioning recipe now sets `umask 077` first (fail_msg, role README, decdn.env.example). The trailing chmod stays as belt-and-braces: sudo may apply its own umask, so the umask alone is not a guarantee of the end state. "Refuse to overwrite an env file this role did not write" printed on every run of a task that conditionally PERMITS the overwrite when decdn_env_overwrite_host_file is true. Renamed to "Require confirmation to overwrite ..." so the play output is accurate in both states. No molecule rescue matches that name, so no test changes. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01WaE3ZP2xQdjd9juzRYKLwi --- ansible/roles/decdn_node/README.md | 5 +++-- ansible/roles/decdn_node/files/decdn.env.example | 5 +++-- ansible/roles/decdn_node/tasks/main.yml | 7 +++++-- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/ansible/roles/decdn_node/README.md b/ansible/roles/decdn_node/README.md index 61fc34d..e35a316 100644 --- a/ansible/roles/decdn_node/README.md +++ b/ansible/roles/decdn_node/README.md @@ -265,9 +265,10 @@ would fail. The full template, including the S3 credential block, is [`files/decdn.env.example`](files/decdn.env.example): ```bash -(umask 077; sudo mkdir -p /etc/decdn) +umask 077 # so the file is never briefly world-readable +sudo mkdir -p /etc/decdn echo 'DECDN_RPC_URL=https://your-endpoint.example/rpc' | sudo tee /etc/decdn/decdn.env >/dev/null -sudo chmod 600 /etc/decdn/decdn.env +sudo chmod 600 /etc/decdn/decdn.env # belt-and-braces: sudo may apply its own umask ``` systemd's `EnvironmentFile` parser is shell-*like* but not a shell: one `KEY=value` diff --git a/ansible/roles/decdn_node/files/decdn.env.example b/ansible/roles/decdn_node/files/decdn.env.example index f2ad528..135645c 100644 --- a/ansible/roles/decdn_node/files/decdn.env.example +++ b/ansible/roles/decdn_node/files/decdn.env.example @@ -11,9 +11,10 @@ # that has never been converged the `decdn` account does not exist yet — the role # creates it, and chowns this file to decdn:decdn on the converge that follows. # -# (umask 077; sudo mkdir -p /etc/decdn) +# umask 077 # never briefly world-readable +# sudo mkdir -p /etc/decdn # sudo tee /etc/decdn/decdn.env >/dev/null # then paste the lines, Ctrl-D -# sudo chmod 600 /etc/decdn/decdn.env +# sudo chmod 600 /etc/decdn/decdn.env # sudo may apply its own umask # # After editing it out of band, re-run `make deploy` — the role notices the change # and restarts the unit — or restart by hand: `sudo systemctl restart decdn-node`. diff --git a/ansible/roles/decdn_node/tasks/main.yml b/ansible/roles/decdn_node/tasks/main.yml index c4a3d6c..21b2e3f 100644 --- a/ansible/roles/decdn_node/tasks/main.yml +++ b/ansible/roles/decdn_node/tasks/main.yml @@ -154,7 +154,10 @@ exist on this host. The URL may embed a provider API key, so the preferred posture is to provision it on the host and keep it off the control machine entirely. Run this on the TARGET, as root (the {{ decdn_user }} account may - not exist yet — this role creates it, and chowns the file on the next run): + not exist yet — this role creates it, and chowns the file on the next run). + umask FIRST, so the file is never briefly world-readable; the explicit chmod + is the belt-and-braces, since sudo may apply its own umask: + umask 077 sudo mkdir -p {{ decdn_etc }} echo 'DECDN_RPC_URL=https://…' | sudo tee {{ decdn_env_file }} >/dev/null sudo chmod 600 {{ decdn_env_file }} @@ -217,7 +220,7 @@ # Scoped to a file whose provenance is KNOWN to be foreign: with no record (a # pre-#39 host's first converge) this cannot distinguish anything, so it stays # quiet and the run seeds the record for next time. -- name: Refuse to overwrite an env file this role did not write +- name: Require confirmation to overwrite an env file this role did not write ansible.builtin.assert: that: - decdn_env_overwrite_host_file | bool