diff --git a/.github/workflows/molecule.yml b/.github/workflows/molecule.yml index ad06337..b0b0a31 100644 --- a/.github/workflows/molecule.yml +++ b/.github/workflows/molecule.yml @@ -40,6 +40,7 @@ jobs: # (operator-provisioned keystore + rendered-value assertions), schema # (config key-set drift against the upstream field list), validation # (bad knobs must be rejected by the role's own asserts), generate-keystore - # (opt-in host-side wallet generation), and slow-readiness (advisory - # /metrics probe timeout). + # (opt-in host-side wallet generation), host-env (host-provisioned + # /etc/decdn/decdn.env — no secret on the control machine), and + # slow-readiness (advisory /metrics probe timeout). run: molecule test --all diff --git a/AGENTS.md b/AGENTS.md index e37bd97..a90c3d1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -51,8 +51,10 @@ ansible/ # the deployment project (DevSec-hardened, lean roles) tarball — verified against the release's GPG-signed `SHA256SUMS` — or, while upstream has no release tag cut (the current default), from locally-built binaries; under a hardened systemd unit; public QUIC udp/4433, loopback metrics/admin, operator-provisioned eth - keystore, required chain knobs (no baked protocol facts — sourced from ADRs), over a - shared DevSec-hardened `baseline`. See `ansible/README.md`. (On-chain node + keystore, operator-provisionable secret env file, and required chain knobs (no baked + protocol facts — sourced from ADRs), over a shared DevSec-hardened `baseline`. + Leaving `decdn_rpc_url` empty means the operator wrote `0600 /etc/decdn/decdn.env` + on the host and the role only gates on it, so no secret transits the control machine. See `ansible/README.md`. (On-chain node stake/registration, ADR 019 Phase 2, is a manual operator step, driven by `decdn setup`.) **Config-schema coupling.** `roles/decdn_node/templates/node.toml.j2` renders against diff --git a/ansible/README.md b/ansible/README.md index 0d95176..94dc86d 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -25,9 +25,11 @@ baseline host hardening — DevSec os/ssh, nftables default-deny inbound, operators must not install or enable ufw, which would replace the nftables ruleset and drop QUIC/SSH. - **No secrets in the repo.** The node's eth keystore is **operator-provisioned** and never - generated here; its `rpc_url` (which may embed an API key) lives in a git-ignored - `host_vars//secret.yml` (the rest of `host_vars` is committed, non-secret config) and - is rendered to a `0600` config. + generated here; its `rpc_url` (which may embed an API key) is provisioned the same way — + a `0600 /etc/decdn/decdn.env` written on the target host, which the role gates on but + never reads back — or, if you prefer, carried in a git-ignored + `host_vars//secret.yml` (the rest of `host_vars` is committed, non-secret config) + and rendered to that same `0600` file. - **Host hardening via DevSec** (`os_hardening` + `ssh_hardening`): key-only SSH, no root login, kernel/sysctl/PAM hardening — applied last, after the admin key is in place. baseline overrides a few `os_hardening` sysctls so hardening can't sever node @@ -54,11 +56,29 @@ make deps # vendor pinned collections into ./ cp inventory/hosts.yml.example inventory/hosts.yml $EDITOR inventory/hosts.yml # set hosts for decdn_nodes $EDITOR inventory/group_vars/all.yml # optional: override admin user/keys, allowlists -# Per-node RPC secret (the rest of host_vars//main.yml is committed config): +``` + +The per-node **secret** (`decdn_rpc_url`, which may embed a provider API key, plus any +environment-borne secrets like AWS keys for an S3 origin) has two homes — pick one. +**Preferred:** provision a `0600 /etc/decdn/decdn.env` on the target host and leave +`decdn_rpc_url` empty, so nothing sensitive is stored on or transits this machine. The role +then leaves that file's contents alone, enforcing only `0600 decdn:decdn`. Full commands and +the file format are in +[`roles/decdn_node/README.md` § Secrets](roles/decdn_node/README.md#secrets) — the single +canonical copy; don't duplicate them here. + +**Or** carry it in inventory, and the role authors `decdn.env` from those values on every run: + +```bash cp inventory/host_vars/decdn-node-1/secret.yml.example inventory/host_vars/decdn-node-1/secret.yml $EDITOR inventory/host_vars/decdn-node-1/secret.yml # set decdn_rpc_url ``` +The role fails loud when neither exists, and — once it has recorded a checksum for the file — +when the two would collide: an inventory value that would discard a host-side edit, or an +empty `decdn_rpc_url` that turns out to mean "`secret.yml` went missing" rather than "the host +owns this file". + Your `hosts.yml` is no longer force-ignored — commit it in your fork if you want, or keep it local. @@ -86,8 +106,10 @@ unless you listed one). `decdn_node_manual_bin_src` / `decdn_cli_manual_bin_src`). 2. Per-node config in `inventory/host_vars//main.yml` (committed) — the install method + binary sources, the **four** required contract addresses, - `decdn_region`, cache origin, … — plus the one secret, `decdn_rpc_url`, in a sibling - git-ignored `secret.yml` (copy the shipped `host_vars/decdn-node-1/secret.yml.example`). + `decdn_region`, cache origin, … — plus the one secret, `decdn_rpc_url`, either + provisioned as `/etc/decdn/decdn.env` on the host (preferred — see + [Setup](#setup)) or in a sibling git-ignored `secret.yml` (copy the shipped + `host_vars/decdn-node-1/secret.yml.example`). The committed `main.yml` already carries the current Arbitrum Sepolia addresses; edit `decdn_region`, the binary paths and the cache origin for your node. Contract addresses are protocol facts — copy them from the upstream deployment manifest @@ -188,7 +210,7 @@ a non-mutating dry run. Defaults live in each role (`roles/*/defaults/main.yml`); override in `group_vars` (shared) or `host_vars` (per node; `main.yml` committed config, `secret.yml` git-ignored for -the RPC URL). Highlights: +the RPC URL when it is not provisioned on the host instead). Highlights: | Var | Default | Notes | |-----|---------|-------| @@ -198,8 +220,9 @@ the RPC URL). Highlights: | `baseline_preserve_ipv6_autoconf` | `true` | Keep IPv6 RA/autoconf under hardening; set `false` for static-IPv6 hosts. | | `baseline_rp_filter_loose` | `false` | `true` loosens reverse-path filtering (`rp_filter=2`) for multi-homed nodes. | | `decdn_node_version` | `""` | **required**; a `v` release must exist. | -| `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node — `rpc_url` in `host_vars//secret.yml`, addresses in `main.yml`; sourced from an ADR/deployment. | +| `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node — `rpc_url` from a host-provisioned `0600 /etc/decdn/decdn.env` (preferred) *or* `host_vars//secret.yml`, addresses in `main.yml`; sourced from an ADR/deployment. | | `decdn_region` / `decdn_bind_port` / `decdn_rate_per_mb` | `""` / `4433` / `10` | node identity, QUIC port, USDC base units/MB. | +| `decdn_env_checksum_file` / `decdn_env_overwrite_host_file` | `/etc/decdn/.decdn.env.sha256` / `false` | Provenance record for the secret env file (`0600 root`), and the opt-in that lets an inventory `decdn_rpc_url` overwrite a host-edited one. | --- diff --git a/ansible/galaxy/CHANGELOG.md b/ansible/galaxy/CHANGELOG.md index e981288..9a8a90f 100644 --- a/ansible/galaxy/CHANGELOG.md +++ b/ansible/galaxy/CHANGELOG.md @@ -31,6 +31,28 @@ Not yet published to Galaxy (pre-1.0; the published shape may still change). - `decdn_extra_env` — extra `KEY: value` pairs appended to the `0600` env file. The supported home for environment-borne secrets, notably the AWS credentials behind an S3 cache origin (which are deliberately never written to the `0640` `node.toml`). +- **Host-provisioned secrets.** `decdn_rpc_url` may now be left empty when the + operator has written a `0600 /etc/decdn/decdn.env` on the target host: the role + then leaves that file's *content* alone (it never reads it back to the control + machine), enforcing only `0600 decdn:decdn` and checking that a non-empty + `DECDN_RPC_URL=` line is present. Setting `decdn_rpc_url` keeps the previous + behaviour and overwrites the host file from inventory. The role fails loud, with + the provisioning commands, when neither is present — and rejects + `decdn_extra_env` against a host-provisioned file rather than silently dropping + it. A template for the host file ships at + `roles/decdn_node/files/decdn.env.example`. +- `decdn_env_checksum_file` (default `/etc/decdn/.decdn.env.sha256`, `0600 root`) — + a role-managed ` ` record for the env file, written on both + authoring paths after the daemon is running. It restarts the unit when an **out-of-band** edit of + a host-provisioned `decdn.env` is detected (instead of leaving the daemon on stale + values behind a green deploy), and doubles as the file's provenance marker: an + empty `decdn_rpc_url` against a file the role itself wrote fails loud (a missing + `secret.yml` is not the same as handing the file to the host), and an inventory + `decdn_rpc_url` against a file someone else wrote fails loud rather than + discarding it — override with `decdn_env_overwrite_host_file: true`. + **Upgrade note:** hosts deployed before this have no record, so the first + converge seeds one. That first run deliberately does *not* restart the daemon, + and the role warns rather than fails if it rewrites an untracked file. - `ExecReload` on the unit plus a `Reload decdn-node` handler, for the five hot-reloadable config sections. - Full config-schema parity with `decdn/decdn` @ `d306cc5c`: `[network.discovery]`, @@ -45,6 +67,24 @@ Not yet published to Galaxy (pre-1.0; the published shape may still change). keystore is also absent), never overwriting an existing wallet. Funding + on-chain staking/registration remain a manual step. +### Fixed + +- **`decdn_extra_env` values were escaped incorrectly** in the `0600` env file, and + `DECDN_RPC_URL` was not escaped at all. Inside a YAML literal block Jinja reads + `'\\'` as *two* literal backslashes, so the role's hand-rolled + `replace('\\', '\\\\') | replace('"', '\\"')` chain never matched a lone backslash + and rendered a double quote as `\\"` — an escaped backslash followed by a + *closing* quote. Any value containing a quote was truncated there and any + backslash was passed through unescaped; systemd then skipped the line it could + not parse and the variable was simply **absent** at runtime (an S3 origin whose + secret key contained a quote or backslash would 403 with nothing to show why). + Both are now rendered with `to_json(ensure_ascii=false)`, which emits exactly the + double-quoted, C-escaped form `EnvironmentFile=` expects. Values containing a + space, `#`, quote, backslash or non-ASCII character now survive intact. + **Upgrade note:** this changes the rendered file (`DECDN_RPC_URL` gains quotes), + so the first converge on the inventory path rewrites it and restarts the daemon + once. + ### Changed (BREAKING) The collection has never been published, so this is not a break against any diff --git a/ansible/galaxy/galaxy.yml b/ansible/galaxy/galaxy.yml index 97d1ef9..be01a3b 100644 --- a/ansible/galaxy/galaxy.yml +++ b/ansible/galaxy/galaxy.yml @@ -38,5 +38,9 @@ repository: https://github.com/decdn/devops documentation: https://github.com/decdn/devops/tree/main/ansible homepage: https://decdn.org issues: https://github.com/decdn/devops/issues -build_ignore: - - "*.example" +# No build_ignore. It used to carry "*.example" as a guard against shipping an +# inventory secret template, but build.sh stages ONLY roles/ + this overlay — the +# inventory never enters the tree, so the pattern excluded nothing real while +# silently dropping roles/decdn_node/files/decdn.env.example, which external +# operators need (it is the template for the host-provisioned 0600 env file). +# Add a NARROW, path-anchored pattern here if something must ever be excluded. diff --git a/ansible/inventory/group_vars/decdn_nodes.yml b/ansible/inventory/group_vars/decdn_nodes.yml index 9c7f791..ea63981 100644 --- a/ansible/inventory/group_vars/decdn_nodes.yml +++ b/ansible/inventory/group_vars/decdn_nodes.yml @@ -1,7 +1,8 @@ --- # Shared config for public deCDN node hosts (playbooks/site.yml). -# Per-node values live in host_vars//: addresses/region/version in main.yml, -# the sensitive rpc_url in a git-ignored secret.yml. +# Per-node values live in host_vars//: addresses/region/version in main.yml. +# The sensitive rpc_url is provisioned as /etc/decdn/decdn.env on the host, or (if +# you would rather the role authored that file) in a git-ignored secret.yml. # Open the node's public QUIC listener in the baseline firewall. Metrics (9090) # and admin RPC (9191) stay loopback and get NO hole. diff --git a/ansible/inventory/host_vars/decdn-node-1/main.yml b/ansible/inventory/host_vars/decdn-node-1/main.yml index 2993bb6..4558c86 100644 --- a/ansible/inventory/host_vars/decdn-node-1/main.yml +++ b/ansible/inventory/host_vars/decdn-node-1/main.yml @@ -1,9 +1,11 @@ --- # Per-node deployment values for decdn-node-1 (COMMITTED — non-secret config only). # Ansible merges every *.yml in this directory for host decdn-node-1. The sensitive -# values (decdn_rpc_url, and any decdn_extra_env secrets) live in the sibling -# secret.yml (git-ignored) — copy secret.yml.example to secret.yml and fill it in. -# The node refuses to start until both this file and secret.yml are set. +# values (decdn_rpc_url, and any environment-borne secrets) live OFF this file: +# either in a 0600 /etc/decdn/decdn.env provisioned on the target host (preferred — +# nothing sensitive on the control machine), or in the sibling git-ignored +# secret.yml (copy secret.yml.example). The role refuses to start the node until +# this file is set and ONE of those two secret sources exists. # # CONTRACT ADDRESSES ARE NOT INVENTED HERE. They are a verbatim copy of the # upstream deployment manifest decdn/contracts/deployments/421614.json (mirrored @@ -34,7 +36,8 @@ decdn_node_manual_bin_src: "" # MUST-EDIT — control-machine path to decdn-no decdn_cli_manual_bin_src: "" # MUST-EDIT — control-machine path to decdn # --- Chain (Arbitrum Sepolia, chain 421614) ----------------------------------- -# The RPC endpoint (decdn_rpc_url) is the one sensitive value — set it in secret.yml. +# The RPC endpoint is the one sensitive value — provision it as DECDN_RPC_URL in +# /etc/decdn/decdn.env on the host, or set decdn_rpc_url in secret.yml. decdn_chain_id: 421614 # Arbitrum Sepolia # Required contracts. All four are mandatory upstream — a node will not resolve its # config without them. diff --git a/ansible/inventory/host_vars/decdn-node-1/secret.yml.example b/ansible/inventory/host_vars/decdn-node-1/secret.yml.example index 86b2d28..edf08b0 100644 --- a/ansible/inventory/host_vars/decdn-node-1/secret.yml.example +++ b/ansible/inventory/host_vars/decdn-node-1/secret.yml.example @@ -1,17 +1,28 @@ --- -# Per-node SECRETS for decdn-node-1. Copy to secret.yml (same directory, -# git-ignored) and fill in. Ansible merges it with main.yml for this host. These -# are the ONLY per-node values kept out of git — everything else lives in the -# committed main.yml. +# Per-node SECRETS for decdn-node-1 — the INVENTORY path, which is optional. # -# MUST-EDIT — SENSITIVE (may embed an API key). The public endpoint below works -# for light use; run your own or use a provider for production reliability. +# You only need this file if you want the role to author /etc/decdn/decdn.env for +# you. The preferred posture is the other way round: provision that 0600 file on +# the target host (see roles/decdn_node/files/decdn.env.example) and skip this +# file entirely — the secret then never sits in plaintext on your control machine +# and never transits it on a converge. The role fails loud if NEITHER exists. +# +# If you do use it: copy to secret.yml (same directory, git-ignored) and fill in. +# Ansible merges it with main.yml for this host. These are the ONLY per-node +# values kept out of git — everything else lives in the committed main.yml. +# +# SENSITIVE (may embed an API key). Setting this makes the role REWRITE +# /etc/decdn/decdn.env from these values on every run, discarding any lines added +# on the host. The public endpoint below works for light use; run your own or use +# a provider for production reliability. decdn_rpc_url: "https://sepolia-rollup.arbitrum.io/rpc" -# Optional. Extra KEY: value pairs appended to the 0600 /etc/decdn/decdn.env that -# systemd hands the daemon. This is where environment-borne secrets belong — most -# notably the AWS credentials backing an S3 cache origin, which the role -# deliberately does NOT write into node.toml (that file is 0640 and diffable): +# Optional, and only meaningful alongside decdn_rpc_url above (the role writes the +# env file all-or-nothing, and rejects extra_env against a host-provisioned file +# rather than silently dropping it). Extra KEY: value pairs appended to the 0600 +# /etc/decdn/decdn.env that systemd hands the daemon — most notably the AWS +# credentials backing an S3 cache origin, which the role deliberately does NOT +# write into node.toml (that file is 0640 and diffable): # # decdn_extra_env: # AWS_ACCESS_KEY_ID: "AKIA..." diff --git a/ansible/molecule/default/converge.yml b/ansible/molecule/default/converge.yml index 16a217a..e19b691 100644 --- a/ansible/molecule/default/converge.yml +++ b/ansible/molecule/default/converge.yml @@ -17,6 +17,15 @@ # stay in sync with the version string printed by files/decdn-node-stub. decdn_node_version: "0.0.0-molecule-stub" decdn_rpc_url: "https://rpc.example.invalid/" + # The ACCEPTED side of the decdn_extra_env gate (#39): with an inventory + # decdn_rpc_url the role authors the whole env file, so extra_env is rendered + # rather than rejected. Nothing else in any scenario sets it alongside a + # non-empty rpc_url, which left both this branch AND the template's + # backslash/quote escaping untested. The value carries a double-quote and a + # backslash on purpose — systemd SKIPS a line it cannot parse, so a broken + # escape means the variable is silently absent at runtime. + decdn_extra_env: + DECDN_MOLECULE_EXTRA: 'a "quoted" \ value' decdn_chain_id: 421614 decdn_region: "US" decdn_payment_pool_address: "0x1111111111111111111111111111111111111111" diff --git a/ansible/molecule/default/verify.yml b/ansible/molecule/default/verify.yml index 0062716..fc6641a 100644 --- a/ansible/molecule/default/verify.yml +++ b/ansible/molecule/default/verify.yml @@ -30,6 +30,7 @@ - "{{ decdn_etc }}/node.toml" - "{{ decdn_etc }}/decdn.env" - /etc/systemd/system/decdn-node.service + - "{{ decdn_etc }}/.decdn.env.sha256" - name: Assert node.toml (0640) and the secret env file (0600) modes ansible.builtin.assert: @@ -39,7 +40,74 @@ - decdn_files.results[0].stat.exists and decdn_files.results[0].stat.mode == '0640' - decdn_files.results[1].stat.exists and decdn_files.results[1].stat.mode == '0600' - decdn_files.results[2].stat.exists - fail_msg: "node.toml / decdn.env / unit missing or wrong mode" + # The env-file checksum record (#39) — 0600 root:root. It is what makes an + # out-of-band edit of decdn.env notify a restart on the next converge, so + # it must be written on the inventory-authored path too. + - >- + decdn_files.results[3].stat.exists + and decdn_files.results[3].stat.mode == '0600' + and decdn_files.results[3].stat.pw_name == 'root' + and decdn_files.results[3].stat.gr_name == 'root' + fail_msg: "node.toml / decdn.env / unit / env-checksum missing or wrong mode/owner" + + # Existence and mode say nothing about the VALUE. A record holding a constant, + # an empty string, or the pre-write hash would pass the stat above while + # breaking the restart signal in one direction or the other, so compare it + # against the file it claims to describe. + # The role authored this file from inventory, so unlike the host-env scenario + # its exact content IS the role's output and can be asserted directly. + - name: Read the role-authored env file + ansible.builtin.slurp: + src: "{{ decdn_etc }}/decdn.env" + register: decdn_env_body + + - name: Assert the env file renders the RPC URL and an escaped decdn_extra_env + ansible.builtin.assert: + that: + - env_rpc_expected in env_body + # Exact escaped form. This caught a real inversion in the original + # hand-rolled replace() chain, which rendered a quote as \\" — an escaped + # backslash plus a CLOSING quote — truncating the value at the first + # quote and leaving backslashes unescaped. The failure is silent: + # systemd drops the line it cannot parse and the variable is absent. + - env_expected in env_body + fail_msg: >- + {{ decdn_etc }}/decdn.env is missing the RPC URL or did not escape the + decdn_extra_env value correctly. Got: {{ env_body }} + vars: + env_body: "{{ decdn_env_body.content | b64decode }}" + env_rpc_expected: 'DECDN_RPC_URL="https://rpc.example.invalid/"' + env_expected: 'DECDN_MOLECULE_EXTRA="a \"quoted\" \\ value"' + + - name: Compute the env file's current sha256 + ansible.builtin.stat: + path: "{{ decdn_etc }}/decdn.env" + get_checksum: true + checksum_algorithm: sha256 + register: decdn_env_hash + + - name: Read the recorded env-file checksum + ansible.builtin.slurp: + src: "{{ decdn_etc }}/.decdn.env.sha256" + register: decdn_env_recorded + + - name: Assert the record names the inventory path and the file's real hash + ansible.builtin.assert: + that: + - record_fields | length == 2 + # The source token is not derivable from the hash and is what stops a + # later converge with an empty decdn_rpc_url from adopting this + # role-authored file as though the operator owned it. + - record_fields[0] == 'inventory' + - record_fields[1] == decdn_env_hash.stat.checksum + fail_msg: >- + Expected "inventory " in + {{ decdn_etc }}/.decdn.env.sha256, got + "{{ decdn_env_recorded.content | b64decode | trim }}". A wrong hash makes + the next converge restart for no reason or miss a real change; a wrong + source token breaks the provenance guards. + vars: + record_fields: "{{ (decdn_env_recorded.content | b64decode | trim).split() }}" # prepare.yml stages the keystore + password at 0644; the role is responsible # for locking them to 0600. Asserting 0600 here therefore verifies the role's @@ -229,6 +297,9 @@ - "'RestrictAddressFamilies=AF_INET AF_INET6 AF_NETLINK' in unit" - "'SystemCallFilter=@system-service' in unit" - "'User=decdn' in unit" + # Without this the daemon never receives DECDN_RPC_URL at all, and every + # other assertion in this scenario would still pass. + - "'EnvironmentFile=/etc/decdn/decdn.env' in unit" fail_msg: "hardened unit is missing an expected directive" vars: unit: "{{ decdn_unit.content | b64decode }}" diff --git a/ansible/molecule/host-env/converge.yml b/ansible/molecule/host-env/converge.yml new file mode 100644 index 0000000..e580d6a --- /dev/null +++ b/ansible/molecule/host-env/converge.yml @@ -0,0 +1,33 @@ +--- +# Converge with NO decdn_rpc_url and NO decdn_extra_env: every secret the daemon +# needs is already on the host (staged by prepare.yml), so the role must gate on +# host state instead of demanding the value from inventory. The remaining +# chain/contract knobs mirror ../default/converge.yml (placeholder but well-formed, +# so the role's fail-loud asserts still run). baseline is not exercised here — see +# molecule.yml. +- name: Converge + hosts: all + become: true + vars: + # Reuse the single shared stub from the default scenario (do not duplicate it). + stub_bin: "{{ lookup('ansible.builtin.env', 'MOLECULE_PROJECT_DIRECTORY') }}/molecule/default/files/decdn-node-stub" + decdn_node_install_method: manual + decdn_node_manual_bin_src: "{{ stub_bin }}" + decdn_cli_manual_bin_src: "{{ stub_bin }}" + decdn_node_version: "0.0.0-molecule-stub" + # decdn_rpc_url is deliberately UNSET — that is the feature under test. + decdn_chain_id: 421614 + decdn_region: "US" + decdn_payment_pool_address: "0x1111111111111111111111111111111111111111" + decdn_capacity_bond_address: "0x2222222222222222222222222222222222222222" + decdn_slash_judge_address: "0x3333333333333333333333333333333333333333" + # REQUIRED upstream (ADR 011/031) — the role asserts it with the other three. + decdn_content_blacklist_address: "0x4444444444444444444444444444444444444444" + # An S3 origin reading its credentials from the host env file — the exact + # combination the old inventory-only path forced onto the control machine. + decdn_cache_origin_kind: "s3" + decdn_cache_origin_s3_bucket: "molecule-bucket" + decdn_cache_origin_s3_region: "us-east-1" + decdn_cache_origin_s3_use_default_chain: true + roles: + - role: decdn_node diff --git a/ansible/molecule/host-env/molecule.yml b/ansible/molecule/host-env/molecule.yml new file mode 100644 index 0000000..858d2c6 --- /dev/null +++ b/ansible/molecule/host-env/molecule.yml @@ -0,0 +1,51 @@ +--- +# Containerised converge + idempotence + verify for the HOST-PROVISIONED secret +# path (#39): /etc/decdn/decdn.env is written by the operator on the target and +# decdn_rpc_url is left empty in inventory, so no secret transits the control +# machine. The role must then NOT rewrite the file, must lock it to 0600 +# decdn:decdn, and must still start the daemon. `idempotence` is load-bearing +# here — it proves the role neither rewrites the file nor restart-loops on it. +# +# `baseline` is NOT exercised here for the same reasons as the `default` scenario +# (host-level hardening is meaningless in a throwaway systemd container) — see +# ../default/molecule.yml. This scenario reuses the shared stub daemon under +# ../default/files/decdn-node-stub. +dependency: + name: galaxy + options: + requirements-file: ../../requirements.yml +driver: + name: docker +platforms: + - name: decdn-node-hostenv-molecule + # Same image as the default scenario, pinned by digest for reproducible CI. + # Re-resolve the digest to bump (keep in sync with ../default/molecule.yml). + image: geerlingguy/docker-debian12-ansible@sha256:4553092be2c00b1ffe580927b9ff03f3c3a0df32b7dd693a3eb02efb6c2b77b7 + pre_build_image: true + command: /usr/lib/systemd/systemd + privileged: true + cgroupns_mode: host + volumes: + - /sys/fs/cgroup:/sys/fs/cgroup:rw +provisioner: + name: ansible + env: + ANSIBLE_ROLES_PATH: "${MOLECULE_PROJECT_DIRECTORY}/roles" + ANSIBLE_COLLECTIONS_PATH: "${MOLECULE_PROJECT_DIRECTORY}/collections" +verifier: + name: ansible +scenario: + # converge -> idempotence proves a STABLE host file neither gets rewritten nor + # restarts the unit; side_effect then edits that file out of band and the second + # converge must restart the daemon (verify.yml compares MainPID across it). The + # order matters: idempotence has to run while the file is still untouched. + test_sequence: + - dependency + - create + - prepare + - converge + - idempotence + - side_effect + - converge + - verify + - destroy diff --git a/ansible/molecule/host-env/prepare.yml b/ansible/molecule/host-env/prepare.yml new file mode 100644 index 0000000..dc03af7 --- /dev/null +++ b/ansible/molecule/host-env/prepare.yml @@ -0,0 +1,56 @@ +--- +# Stage what an OPERATOR would have provisioned on the host: the eth keystore + +# password + node identity (as ../default/prepare.yml does), and — the point of +# this scenario — a /etc/decdn/decdn.env the role must not author or rewrite. +# +# Both the env file and the wallet material are staged at 0644, deliberately +# looser than the target: the role is responsible for locking every one of them +# to 0600, so verify.yml's mode assertions attribute the mode to the role rather +# than to what prepare pre-set. +- name: Prepare + hosts: all + become: true + vars: + decdn_home: /var/lib/decdn + decdn_etc: /etc/decdn + tasks: + - name: Ensure data + config directories exist + ansible.builtin.file: + path: "{{ item }}" + state: directory + mode: "0755" + loop: + - "{{ decdn_home }}" + - "{{ decdn_etc }}" + + - name: Stage a placeholder eth keystore + ansible.builtin.copy: + dest: "{{ decdn_home }}/keystore.json" + content: "{{ '{}' }}\n" + mode: "0644" + + - name: Stage a placeholder keystore password file + ansible.builtin.copy: + dest: "{{ decdn_etc }}/keystore.password" + content: "molecule-placeholder\n" + mode: "0644" + + - name: Stage a placeholder node identity + ansible.builtin.copy: + dest: "{{ decdn_home }}/node.secret" + content: "molecule-placeholder-node-secret\n" + mode: "0644" + + # The host-provisioned secret env file. It carries a second KEY=value the role + # has no inventory var for at all (converge passes no decdn_extra_env), so + # verify.yml's byte-identical assertion proves the role preserved operator + # content it could not have reproduced. In production that second line is + # typically AWS_ACCESS_KEY_ID for an S3 origin; a neutral marker name is used + # here so the KICS hardcoded-secret query has nothing to flag in a fixture. + - name: Stage the operator-provisioned secret environment file + ansible.builtin.copy: + dest: "{{ decdn_etc }}/decdn.env" + content: | + DECDN_RPC_URL=https://rpc.example.invalid/ + DECDN_MOLECULE_HOST_LINE=written-by-the-operator + mode: "0644" diff --git a/ansible/molecule/host-env/side_effect.yml b/ansible/molecule/host-env/side_effect.yml new file mode 100644 index 0000000..6eaa611 --- /dev/null +++ b/ansible/molecule/host-env/side_effect.yml @@ -0,0 +1,52 @@ +--- +# Edit the host-provisioned env file OUT OF BAND, exactly as an operator rotating +# an RPC key would, and record the daemon's current MainPID. molecule.yml runs a +# SECOND converge after this, and verify.yml then asserts the PID changed. +# +# This is the only test of the feature's headline behaviour. Every other assertion +# about /etc/decdn/.decdn.env.sha256 checks a precondition (it exists, it is 0600 +# root, it matches the file) — all of which stay true if the restart signal is +# removed entirely. `idempotence` proves the converse (an UNCHANGED file must not +# restart), so the two together pin the behaviour from both sides. +- name: Side effect — rotate the secret out of band + hosts: all + become: true + vars: + decdn_etc: /etc/decdn + tasks: + # pgrep, not `systemctl show --property=MainPID`: ansible-lint's + # command-instead-of-module rule rejects any `systemctl` invocation, and there + # is no module that returns a unit's PID. This queries the process table + # instead, which is a different job from managing the service. + - name: Record the daemon's PID before the edit + ansible.builtin.command: + cmd: pgrep -f '/usr/local/bin/decdn-node run' + register: decdn_pid_before + changed_when: false + failed_when: false + + - name: Assert the daemon was actually running before the edit + ansible.builtin.assert: + that: + - decdn_pid_before.stdout_lines | length == 1 + - decdn_pid_before.stdout | trim | int > 0 + fail_msg: >- + Expected exactly one running decdn-node before the out-of-band edit + (got {{ decdn_pid_before.stdout_lines | default([]) }}), so a PID change + after it would prove nothing. + + - name: Save the pre-edit MainPID for verify.yml + ansible.builtin.copy: + dest: "{{ decdn_etc }}/molecule-pid-before" + content: "{{ decdn_pid_before.stdout | trim }}\n" + mode: "0644" + + # Append/replace in place rather than rewriting the file: this must look like + # an operator edit, and it also proves the role preserves the line it did not + # write (DECDN_MOLECULE_HOST_LINE) across the restart path. + - name: Rotate the secret out of band (operator edits the file directly) + ansible.builtin.lineinfile: + path: "{{ decdn_etc }}/decdn.env" + regexp: '^DECDN_RPC_URL=' + line: "DECDN_RPC_URL=https://rotated.example.invalid/" + mode: "0600" diff --git a/ansible/molecule/host-env/verify.yml b/ansible/molecule/host-env/verify.yml new file mode 100644 index 0000000..bb08305 --- /dev/null +++ b/ansible/molecule/host-env/verify.yml @@ -0,0 +1,192 @@ +--- +# Verify the host-provisioned secret path (#39): the role must have left the +# operator's /etc/decdn/decdn.env byte-for-byte intact, locked it to 0600 +# decdn:decdn, recorded its checksum, and started the daemon on it. +- name: Verify + hosts: all + become: true + vars: + decdn_etc: /etc/decdn + # What the host file holds by the time verify runs: prepare.yml's content with + # side_effect.yml's rotated DECDN_RPC_URL line. Comparing against a literal + # (rather than re-reading and trusting whatever is there) is what makes this an + # assertion about the ROLE's behaviour and not a tautology — and the untouched + # DECDN_MOLECULE_HOST_LINE is a key the role has no inventory var for, so a + # role that rewrote the file could not have reproduced it. + expected_env: | + DECDN_RPC_URL=https://rotated.example.invalid/ + DECDN_MOLECULE_HOST_LINE=written-by-the-operator + tasks: + - name: Read back the host-provisioned env file + ansible.builtin.slurp: + src: "{{ decdn_etc }}/decdn.env" + register: decdn_env_content + + - name: Assert the role did not rewrite the operator's env file + ansible.builtin.assert: + that: + - (decdn_env_content.content | b64decode) == expected_env + fail_msg: >- + {{ decdn_etc }}/decdn.env does not hold the operator's content. With + decdn_rpc_url unset the role must leave it byte-for-byte alone — a + rewrite would have dropped DECDN_MOLECULE_HOST_LINE (converge passes no + decdn_extra_env), which in production is the AWS credential line behind + the S3 origin: silently broken at runtime, green at deploy time. It must + also have survived the out-of-band rotation in side_effect.yml. + Got: {{ decdn_env_content.content | b64decode }} + + # prepare.yml stages the env file (and the wallet material) at 0644; the role + # is responsible for locking them to 0600. Asserting 0600 here therefore + # verifies the role's lock-down ran, not a mode prepare pre-set. + - name: Stat the env file, its checksum record, and the wallet material + ansible.builtin.stat: + path: "{{ item }}" + register: decdn_files + loop: + - "{{ decdn_etc }}/decdn.env" + - "{{ decdn_etc }}/.decdn.env.sha256" + - /var/lib/decdn/keystore.json + - /var/lib/decdn/node.secret + - "{{ decdn_etc }}/keystore.password" + + - name: Assert the role locked the host-provisioned env file to 0600 decdn:decdn + ansible.builtin.assert: + that: + # exists-and-mode-and-owner in one expression: a missing file + # short-circuits to the fail_msg instead of raising on the absent stat. + - >- + decdn_files.results[0].stat.exists + and decdn_files.results[0].stat.mode == '0600' + and decdn_files.results[0].stat.pw_name == 'decdn' + and decdn_files.results[0].stat.gr_name == 'decdn' + fail_msg: "decdn.env missing, not 0600, or not owned by decdn (role lock-down did not run)" + + - name: Assert the env-file checksum record exists at 0600 root:root + ansible.builtin.assert: + that: + - >- + decdn_files.results[1].stat.exists + and decdn_files.results[1].stat.mode == '0600' + and decdn_files.results[1].stat.pw_name == 'root' + fail_msg: >- + {{ decdn_etc }}/.decdn.env.sha256 missing or not 0600 root:root — without + it an out-of-band edit of decdn.env would never notify a restart. + + - name: Assert the wallet material is locked to 0600 + ansible.builtin.assert: + that: + - decdn_files.results[2].stat.exists and decdn_files.results[2].stat.mode == '0600' + - decdn_files.results[3].stat.exists and decdn_files.results[3].stat.mode == '0600' + - decdn_files.results[4].stat.exists and decdn_files.results[4].stat.mode == '0600' + fail_msg: "keystore / node.secret / password file missing or not 0600" + + # The recorded hash must be the hash of the file as it stands, or the next + # converge would restart the daemon for no reason (or, worse, not restart it + # after a genuine edit). + - name: Compute the env file's current sha256 + ansible.builtin.stat: + path: "{{ decdn_etc }}/decdn.env" + get_checksum: true + checksum_algorithm: sha256 + register: decdn_env_hash + + - name: Read the recorded checksum + ansible.builtin.slurp: + src: "{{ decdn_etc }}/.decdn.env.sha256" + register: decdn_env_recorded + + - name: Assert the record names the HOST path and the file's real hash + ansible.builtin.assert: + that: + - record_fields | length == 2 + # Must say "host": recorded as "inventory" here, the next converge would + # read an empty decdn_rpc_url plus a matching hash as "the value went + # missing from inventory" and fail a perfectly good node. + - record_fields[0] == 'host' + - record_fields[1] == decdn_env_hash.stat.checksum + fail_msg: >- + Expected "host " in + {{ decdn_etc }}/.decdn.env.sha256, got + "{{ decdn_env_recorded.content | b64decode | trim }}". A wrong hash makes + the restart-on-out-of-band-edit signal wrong in both directions; a wrong + source token makes the next converge fail this node. + vars: + record_fields: "{{ (decdn_env_recorded.content | b64decode | trim).split() }}" + + # node.toml must carry the S3 origin with source = "default-chain" and NO + # static keys: the credentials live only in the 0600 env file. + - name: Read the rendered node.toml + ansible.builtin.slurp: + src: "{{ decdn_etc }}/node.toml" + register: decdn_node_toml + + - name: Assert node.toml delegates S3 credentials to the env file + ansible.builtin.assert: + that: + - "'source = \"default-chain\"' in node_toml" + # No VALUE from the 0600 env file may appear in this 0640, diffable file. + # (The template's header comment names DECDN_RPC_URL to explain where the + # endpoint lives, so match on the values, not the variable names.) + - "'written-by-the-operator' not in node_toml" + - "'rpc.example.invalid' not in node_toml" + fail_msg: "node.toml is missing the default-chain credential source, or leaked an env-file value" + vars: + node_toml: "{{ decdn_node_toml.content | b64decode }}" + + # THE headline behaviour: side_effect.yml rotated DECDN_RPC_URL out of band and + # molecule.yml re-converged. The daemon must have been restarted, or it is + # still serving with the old endpoint behind a green deploy. Comparing MainPID + # is what makes this behavioural — the checksum assertions above all remain + # true even if the restart signal is deleted outright. + - name: Read the pre-edit MainPID recorded by side_effect + ansible.builtin.slurp: + src: "{{ decdn_etc }}/molecule-pid-before" + register: decdn_pid_before + + # pgrep rather than `systemctl show` — see the note in side_effect.yml. + - name: Read the daemon's PID after the re-converge + ansible.builtin.command: + cmd: pgrep -f '/usr/local/bin/decdn-node run' + register: decdn_pid_after + changed_when: false + failed_when: false + + - name: Assert the out-of-band edit restarted decdn-node + ansible.builtin.assert: + that: + - decdn_pid_after.stdout_lines | length == 1 + - decdn_pid_after.stdout | trim | int > 0 + - (decdn_pid_after.stdout | trim) != (decdn_pid_before.content | b64decode | trim) + fail_msg: >- + decdn-node still runs as PID {{ decdn_pid_after.stdout | trim }} after + {{ decdn_etc }}/decdn.env was edited out of band and the role re-converged + — the daemon is on the PRE-rotation secret behind a green deploy. The + env-file checksum record did not drive the restart handler. + + # Without EnvironmentFile the daemon never receives the host-provisioned + # secret at all, and every other assertion in this scenario would still pass. + - name: Read the rendered systemd unit + ansible.builtin.slurp: + src: /etc/systemd/system/decdn-node.service + register: decdn_unit + + - name: Assert the unit actually loads the secret env file + ansible.builtin.assert: + that: + - "'EnvironmentFile=' ~ decdn_etc ~ '/decdn.env' in (decdn_unit.content | b64decode)" + fail_msg: >- + decdn-node.service does not carry EnvironmentFile={{ decdn_etc }}/decdn.env, + so the host-provisioned secret never reaches the daemon. + + - name: Gather service facts + ansible.builtin.service_facts: + + - name: Assert decdn-node is running on the host-provisioned secrets + ansible.builtin.assert: + that: + # Membership-and-state in one expression so an absent service + # short-circuits to the fail_msg instead of raising on .state. + - >- + 'decdn-node.service' in ansible_facts.services + and ansible_facts.services['decdn-node.service'].state == 'running' + fail_msg: "decdn-node is not running — check: journalctl -u decdn-node -e" diff --git a/ansible/molecule/validation/converge.yml b/ansible/molecule/validation/converge.yml index 34c9926..c13d716 100644 --- a/ansible/molecule/validation/converge.yml +++ b/ansible/molecule/validation/converge.yml @@ -3,13 +3,20 @@ # REJECTS bad values (AGENTS.md hard-rule 4), so a regression that weakens an assert # fails CI. Each case runs decdn_node with a VALID required config plus ONE bad # optional knob, wrapped in block/rescue. The role aborts at its validation asserts -# (which run before any host mutation), so cases don't interfere and no service starts. +# (which for the tuning-knob cases run before any host mutation), so those cases +# don't interfere and no service starts. The env-file cases below are the +# exception — see the ordering note. # -# The rescue records a case as "rejected" ONLY when the failing task is one of the -# role's own "Validate optional ..." asserts (matched on ansible_failed_task.name). -# That is what makes this a real regression guard: if a bad value slipped PAST -# validation, the role would run on and fail later at a differently-named task (e.g. -# the keystore gate) — which is NOT counted, so the final assert catches the miss. +# The rescue records a case as "rejected" ONLY when the failing task is the SPECIFIC +# assert that case targets (matched on ansible_failed_task.name — mostly the role's +# "Validate optional ..." family, plus the two secret-source gates). That is what +# makes this a real regression guard: if a bad value slipped PAST validation, the +# role would run on and fail later at a differently-named task (e.g. the keystore +# gate) — which is NOT counted, so the final assert catches the miss. +# +# Two cases touch host state on purpose: "env-extra" stages a host-provisioned +# /etc/decdn/decdn.env (it has to exist for that gate to fire), so it must run +# AFTER "env-gate", which proves the absent-file case still fails. # # ONE case is deliberately different: the final "config-gate" case has a valid # config and has to reach the END of the role, because what it proves is that the @@ -191,8 +198,96 @@ decdn_rejected: "{{ decdn_rejected + ['string'] }}" when: ansible_failed_task.name is match('^Validate optional') + # --- Case: no RPC endpoint anywhere (#39) ------------------------------------- + # decdn_rpc_url may now be empty — but ONLY when the operator has provisioned + # /etc/decdn/decdn.env on the host. Nothing has staged that file yet at this + # point in the play, so "empty and absent" must still fail loud. A regression + # that relaxed the gate to "always optional" would start a daemon with no chain + # endpoint. This case must run BEFORE the one below, which stages the file. + - name: "Case env-gate — no rpc_url and no host-provisioned env file" + block: + - name: Run decdn_node with neither an inventory rpc_url nor a host env file + ansible.builtin.include_role: + name: decdn_node + vars: + decdn_rpc_url: "" + rescue: + - name: Record env-gate rejection (only if the RPC gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['env-gate'] }}" + when: ansible_failed_task.name is match('^Require an RPC endpoint') + + # --- Case: decdn_extra_env against a host-provisioned env file (#39) ---------- + # The role writes the env file ALL-OR-NOTHING from inventory, so extra_env with + # an empty rpc_url would be silently dropped — the S3 origin would 403 hours + # later with nothing in the deploy output to explain it. Must fail loud instead. + - name: "Case env-extra — decdn_extra_env alongside a host-provisioned env file" + block: + - name: Stage a host-provisioned env file so the role takes the host path + ansible.builtin.copy: + dest: /etc/decdn/decdn.env + content: "DECDN_RPC_URL=https://rpc.example.invalid/\n" + mode: "0600" + + - name: Run decdn_node with extra_env but no rpc_url + ansible.builtin.include_role: + name: decdn_node + vars: + decdn_rpc_url: "" + decdn_extra_env: + DECDN_MOLECULE_EXTRA: "would-be-silently-dropped" + rescue: + - name: Record env-extra rejection (only if the extra_env assert failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['env-extra'] }}" + when: ansible_failed_task.name is match('^Reject decdn_extra_env') + always: + # Each env-file case owns its fixture end to end rather than relying on the + # next case to tolerate what it left behind — the ordering dependency was + # already implicit and a third case made it fragile. + - name: Remove the staged env file so later cases start clean + ansible.builtin.file: + path: /etc/decdn/decdn.env + state: absent + + # --- Case: a host env file with no usable DECDN_RPC_URL (#39) ----------------- + # The RPC gate accepts the host path on stat.exists ALONE, so this grep is the + # only thing standing between a file that merely exists and a daemon started + # with no chain endpoint. Without a case here, weakening the pattern (dropping + # the value match, un-anchoring it) or neutering the report task's `when:` + # would be an invisible regression — exactly what the config-gate case below + # exists to prevent for its own gate. The staged file is commented out, which + # is the realistic operator error (and `DECDN_RPC_URL= ` with only whitespace + # must fail too — that is what the [^[:space:]] in the pattern buys). + - name: "Case env-content — host env file carrying no usable DECDN_RPC_URL" + block: + - name: Stage a host env file whose DECDN_RPC_URL is commented out + ansible.builtin.copy: + dest: /etc/decdn/decdn.env + content: | + #DECDN_RPC_URL=https://rpc.example.invalid/ + DECDN_MOLECULE_OTHER=present + mode: "0600" + + - name: Run decdn_node against an env file with no usable endpoint + ansible.builtin.include_role: + name: decdn_node + vars: + decdn_rpc_url: "" + rescue: + - name: Record env-content rejection (only if the content gate failed) + ansible.builtin.set_fact: + decdn_rejected: "{{ decdn_rejected + ['env-content'] }}" + when: ansible_failed_task.name is match('^Report a host-provisioned env file') + always: + - name: Remove the staged env file so later cases start clean + ansible.builtin.file: + path: /etc/decdn/decdn.env + state: absent + # --- Case: the config gate itself --------------------------------------------- - # Every case above trips an Ansible assert BEFORE the host is touched. This one + # Every tuning-knob case above trips an Ansible assert before the host is + # touched (env-extra is the exception — it stages a file first). This one # proves the last line of defence still works: `decdn config validate` runs # with `failed_when: false` and hands off to a separate `fail` task, so a # regression in that task's `when:` would silently turn the authoritative gate @@ -260,4 +355,5 @@ vars: _decdn_expected: ["cross-field", "range", "bounded-range", "shape", "bool", "enum", "dict", - "float", "list", "string", "config-gate"] + "float", "list", "string", "env-gate", "env-extra", "env-content", + "config-gate"] diff --git a/ansible/roles/decdn_node/README.md b/ansible/roles/decdn_node/README.md index be2631d..e35a316 100644 --- a/ansible/roles/decdn_node/README.md +++ b/ansible/roles/decdn_node/README.md @@ -102,10 +102,10 @@ Per the deCDN node-onboarding ADR (019), a node only serves paid traffic after ## Required variables (set in `host_vars//`) `decdn_node_version` (`release` mode only) **or** `decdn_node_manual_bin_src` + -`decdn_cli_manual_bin_src` (`manual` mode), `decdn_rpc_url` (sensitive — may embed an -API key; goes in the git-ignored `secret.yml`, everything else in the committed -`main.yml`), `decdn_region` (ISO 3166-1 alpha-2), and **four** contract addresses -(all `0x`+40-hex, none the zero address): +`decdn_cli_manual_bin_src` (`manual` mode), an RPC endpoint (sensitive — may embed an +API key; provision it on the host or set `decdn_rpc_url` in the git-ignored +`secret.yml` — see [Secrets](#secrets)), `decdn_region` (ISO 3166-1 alpha-2), and +**four** contract addresses (all `0x`+40-hex, none the zero address): | Variable | Contract | Why required | |---|---|---| @@ -141,8 +141,10 @@ Optional (omitted from `node.toml` unless set): dicts) instead — the two forms are mutually exclusive upstream and the role fails loud if both are set. - `decdn_extra_env` — extra `KEY: value` pairs appended to the 0600 - `/etc/decdn/decdn.env`. This is where environment-borne secrets belong; see - [S3 credentials](#s3-credentials). + `/etc/decdn/decdn.env` **when the role authors that file** (i.e. `decdn_rpc_url` + is set). With a host-provisioned env file the role rejects it rather than + silently dropping it — put those lines in the host file instead. See + [Secrets](#secrets) and [S3 credentials](#s3-credentials). - `decdn_node_generate_keystore` (default `false`) — opt-in turnkey wallet. When `true` the role runs `decdn key-gen` on the host **only if the keystore is absent** (minting a random `0600` password file first, but only when the keystore is *also* absent) — it @@ -227,13 +229,109 @@ asserts miss. See `defaults/main.yml` for every knob's upstream default, unit an - **Observability** — `decdn_otlp_endpoint` (OTLP span export; needs the node built `--features otlp`). +## Secrets + +Every secret this role needs ends up in exactly one place on the host: `0600` +files under `/etc/decdn` and `/var/lib/decdn`, owned by the `decdn` user. The +wallet material (`keystore.json`, `node.secret`, `keystore.password`) is +provisioned — or, with `decdn_node_generate_keystore`, minted — **on the host**, +and never touches the control machine. The one +operator-supplied secret — the chain RPC URL, which may embed a provider API key, +plus anything else the daemon reads from its environment — has two authoring +paths: + +| | Host-provisioned (preferred) | Inventory-carried | +|---|---|---| +| Set | `decdn_rpc_url: ""` (the default) | `decdn_rpc_url` in `host_vars//secret.yml` | +| Who writes `/etc/decdn/decdn.env` | you, on the target | the role, on every converge | +| Secret on the control machine | never | plaintext, git-ignored | +| Extra env vars | further `KEY=value` lines in that file | `decdn_extra_env` | + +On the host path the role never reads the file's contents back — that would put +the secret on the control machine, defeating the point. It reads only a **sha256** +of the file (which does cross to the control machine, to be compared and written +back as the record below), and on that path it confines itself to: + +- enforcing `0600 decdn:decdn` on the file, content untouched, after refusing to + touch it at all unless it is a regular file (a symlink would have root apply + that ownership to whatever it points at); +- grepping for a `DECDN_RPC_URL=` line with a non-whitespace value, failing loud + if there is none. + +**Provisioning it.** Do this on the target as **root** — on a host that has never +been converged the `decdn` account does not exist yet (this role creates it, and +chowns the file on the next run), so `install -o decdn` / `chown decdn:decdn` +would fail. The full template, including the S3 credential block, is +[`files/decdn.env.example`](files/decdn.env.example): + +```bash +umask 077 # so the file is never briefly world-readable +sudo mkdir -p /etc/decdn +echo 'DECDN_RPC_URL=https://your-endpoint.example/rpc' | sudo tee /etc/decdn/decdn.env >/dev/null +sudo chmod 600 /etc/decdn/decdn.env # belt-and-braces: sudo may apply its own umask +``` + +systemd's `EnvironmentFile` parser is shell-*like* but not a shell: one `KEY=value` +per line, no `export`, no expansion — and it **skips** a line it cannot parse, +logging a warning to `journalctl -u decdn-node` that is easy to miss, so the +variable is simply absent at runtime. Single-quote any value containing a space, +quote, backslash, `$` or backtick. (`decdn config validate` sources this file with +bash, which *would* expand `$VAR` and execute `$(…)`; quoting avoids both.) On the +inventory path the role applies that escaping to `decdn_extra_env` values for you — +but **not** to `decdn_rpc_url` itself, which is rendered raw. + +### The provenance record + +`/etc/decdn/.decdn.env.sha256` (`0600 root`, `decdn_env_checksum_file`) holds +` ` — which path wrote the file, and what its contents hashed to — +as of the last converge that got the daemon running: + +```text +host 3b1f…c7 # the operator owns /etc/decdn/decdn.env +inventory 9a4e…21 # this role rendered it from decdn_rpc_url +``` + +It is written on **both** paths, which is exactly why the `source` token is needed +and cannot be inferred from the hash: a matching hash only means "unchanged since +the last converge". A bare `stat` tells you even less — a file the role wrote last +week exists exactly as much as an operator-provisioned one. From this the role gets +three behaviours: + +- **Restart on an out-of-band edit.** Record ≠ file on the host path ⇒ the unit is + restarted so the daemon picks up the new values. The record is written *after* + the restart, so a run that aborts in between leaves the old value in place and + the next converge re-detects — the signal is never burned by a failed deploy. +- **"`secret.yml` went missing" ≠ "the host owns this now."** An empty + `decdn_rpc_url` against a file recorded as `inventory` *and* unchanged since is + treated as a lost inventory value and fails loud, instead of silently deploying a + stale endpoint. To migrate a node to the host path deliberately, `sudo rm` the + record once — that is the explicit hand-over. +- **No silent clobbering.** An inventory `decdn_rpc_url` against a file recorded as + `host`, or one whose hash no longer matches, fails loud rather than discarding + those lines; set `decdn_env_overwrite_host_file: true` to confirm. With **no** + record (a host that predates this) the role cannot tell, so it warns and starts + tracking. + +If neither authoring path is satisfied the role fails before touching the host, +with the provisioning commands above in the failure message. + ### S3 credentials The role deliberately **never** writes static AWS keys into `node.toml`: that file is `0640` and diffable, and committing credentials to it would break this repo's no-secrets rule. Only `source = "default-chain"` is templated. Set `decdn_cache_origin_s3_use_default_chain: true` (plus an optional -`decdn_cache_origin_s3_profile`) and supply the keys through the `0600` env file: +`decdn_cache_origin_s3_profile`) and supply the keys through the `0600` env file. +On the host, as further lines in `/etc/decdn/decdn.env`: + +```ini +AWS_ACCESS_KEY_ID=AKIA... +AWS_SECRET_ACCESS_KEY=... +AWS_SESSION_TOKEN=... # assume-role / SSO only +``` + +…or, if you are carrying the RPC URL in inventory anyway, through `decdn_extra_env` +(the role then renders both into the same file): ```yaml # host_vars//secret.yml (git-ignored) @@ -250,7 +348,7 @@ instance profile needs no credentials at all, just the toggle. Note the region i **not** taken from `AWS_REGION`: it comes from `[cache.origin].region` (`decdn_cache_origin_s3_region`). -### Config reload +## Config reload SIGHUP (and `decdn node reload`, which shares its mutex) re-reads the config file and applies **five** sections: `observability.log_level`, `cache.pinned_hashes`, @@ -354,7 +452,8 @@ additionally requires on-chain stake + registration (see - `/usr/local/bin/decdn-node`, `/usr/local/bin/decdn` — daemon + CLI - `/etc/decdn/node.toml` (`0640 decdn`) — rendered config; non-secret (no `rpc_url`), so it stays `--check`-diffable -- `/etc/decdn/decdn.env` (`0600 decdn`) — the one sensitive value, `DECDN_RPC_URL`, read by the unit via `EnvironmentFile` +- `/etc/decdn/decdn.env` (`0600 decdn`) — the one operator-supplied secret, `DECDN_RPC_URL` (+ any extra env vars), read by the unit via `EnvironmentFile`; role-authored or host-provisioned ([Secrets](#secrets)) +- `/etc/decdn/.decdn.env.sha256` (`0600 root`) — role-managed ` ` record of the above: provenance marker, and the trigger for a restart after an out-of-band edit. Not a secret store - `/etc/decdn/keystore.password` (`0600 decdn`) — operator-provisioned - `/var/lib/decdn/` (`0700 decdn`) — `node.secret`, `keystore.json`, `cache/`, state - `/etc/systemd/system/decdn-node.service` — hardened unit diff --git a/ansible/roles/decdn_node/defaults/main.yml b/ansible/roles/decdn_node/defaults/main.yml index 97ef62d..2687cc4 100644 --- a/ansible/roles/decdn_node/defaults/main.yml +++ b/ansible/roles/decdn_node/defaults/main.yml @@ -1,7 +1,10 @@ --- # decdn-node role defaults. Per-node deployment values belong in host_vars//: -# non-secret config (contract addresses, region, version) in the committed main.yml, -# and the sensitive rpc_url in a git-ignored secret.yml (see secret.yml.example). +# non-secret config (contract addresses, region, version) in the committed main.yml. +# The sensitive rpc_url does NOT have to live on the control machine at all — +# provision /etc/decdn/decdn.env on the target (files/decdn.env.example) and leave +# decdn_rpc_url empty; set it in a git-ignored secret.yml only if you want the role +# to author that file for you. # --- Install method ----------------------------------------------------------- # "release" (default): download the pinned GitHub Release tarballs (a v @@ -30,11 +33,18 @@ decdn_verify_release_signature: true decdn_release_keyring: decdn-release-KEYS.asc # --- Identity / on-chain ----------------------------------------------------- -# rpc_url, the FOUR contract addresses and region are REQUIRED with no usable -# default (empty → the asserts in tasks/main.yml fail loud). Supply via host_vars. +# The FOUR contract addresses and region are REQUIRED with no usable default +# (empty → the asserts in tasks/main.yml fail loud). Supply via host_vars. # chain_id is the one exception: it defaults to the upstream binary's own default # (Arbitrum Sepolia) rather than being a deployment-specific fact to source. -decdn_rpc_url: "" # SENSITIVE; may embed an API key +# +# rpc_url is required too, but has TWO sources. Leave it empty here and provision +# a 0600 /etc/decdn/decdn.env on the target host instead (the preferred posture — +# the secret then never touches the control machine); set it here only to have the +# role author that file for you, in which case it OVERWRITES any host-provisioned +# one. The role fails loud when NEITHER is present. See roles/decdn_node/README.md +# ("Secrets") and roles/decdn_node/files/decdn.env.example. +decdn_rpc_url: "" # SENSITIVE; may embed an API key — or provision decdn.env on the host decdn_payment_pool_address: "" decdn_capacity_bond_address: "" decdn_slash_judge_address: "" @@ -300,11 +310,28 @@ decdn_home: /var/lib/decdn # data dir: node.secret + keystore.json decdn_etc: /etc/decdn decdn_config_file: /etc/decdn/node.toml decdn_env_file: /etc/decdn/decdn.env # DECDN_RPC_URL (sensitive) — 0600 +# Role-managed " " record for the env file, as of the last +# converge that got the daemon running. NOT a secret store (0600 root:root, one +# hash and one word). It does two jobs: it lets an OUT-OF-BAND edit of a +# host-provisioned decdn.env notify the restart handler, and it records WHICH PATH +# wrote the file. Written on BOTH paths, which is why the source token exists — a +# matching hash alone only proves the file is unchanged, not who authored it. +decdn_env_checksum_file: /etc/decdn/.decdn.env.sha256 +# Confirm that rendering the env file from inventory may DISCARD a host-side edit. +# Only consulted when the recorded checksum proves the file on the host is not the +# one this role last wrote; with no record (a pre-#39 host) the role cannot tell +# and warns instead. Leave false so an operator-written AWS credential line cannot +# be silently overwritten by a converge that carries decdn_rpc_url. +decdn_env_overwrite_host_file: false # Extra KEY: value pairs appended to the 0600 env file. This is the supported home # for secrets the daemon reads from the environment rather than node.toml — most # notably AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN backing an # S3 cache origin with source = "default-chain". Keep it out of committed vars: -# set it in the git-ignored secret.yml alongside decdn_rpc_url. +# set it in the git-ignored secret.yml alongside decdn_rpc_url — or, better, skip +# the control machine entirely and write them straight into the host's decdn.env +# (see decdn_rpc_url above). The two forms are mutually exclusive: the role writes +# the env file ALL-OR-NOTHING from inventory, so decdn_extra_env alongside a +# host-provisioned file is rejected rather than silently dropped. decdn_extra_env: {} # Eth keystore lives in the data dir (matches `decdn key-gen --output-dir`). decdn_keystore_file: /var/lib/decdn/keystore.json # operator-provisioned (0600) diff --git a/ansible/roles/decdn_node/files/decdn.env.example b/ansible/roles/decdn_node/files/decdn.env.example new file mode 100644 index 0000000..135645c --- /dev/null +++ b/ansible/roles/decdn_node/files/decdn.env.example @@ -0,0 +1,42 @@ +# /etc/decdn/decdn.env — the deCDN node's secret environment file. +# +# Provision this ON THE TARGET HOST, not on the control machine: systemd reads it +# as root (EnvironmentFile=) before dropping to the `decdn` user, and the decdn_node +# Ansible role deliberately never reads its contents back. Leave decdn_rpc_url empty +# in inventory and the role leaves this file alone, enforcing only 0600 ownership — +# so the RPC key (and any cloud credentials below) never sit on a laptop. +# +# This file is a TEMPLATE that lives on the control machine; copy its contents to +# the host by hand (or paste the two commands below). Create it as ROOT: on a host +# that has never been converged the `decdn` account does not exist yet — the role +# creates it, and chowns this file to decdn:decdn on the converge that follows. +# +# umask 077 # never briefly world-readable +# sudo mkdir -p /etc/decdn +# sudo tee /etc/decdn/decdn.env >/dev/null # then paste the lines, Ctrl-D +# sudo chmod 600 /etc/decdn/decdn.env # sudo may apply its own umask +# +# After editing it out of band, re-run `make deploy` — the role notices the change +# and restarts the unit — or restart by hand: `sudo systemctl restart decdn-node`. +# +# FORMAT: systemd's EnvironmentFile parser is shell-LIKE but is not a shell — no +# expansion, no `export`, one KEY=value per line. It SKIPS a line it cannot parse +# (logging a warning to `journalctl -u decdn-node` that is easy to miss), so the +# variable is simply absent at runtime — surfacing much later as an auth failure. +# Single-quote any value containing a space, quote, backslash, `$` or backtick. +# The role's `decdn config validate` step sources this file with bash, which — +# unlike systemd — WOULD expand `$VAR` and execute `$(...)`; quoting avoids both. + +# REQUIRED. The chain RPC endpoint — may embed a provider API key, which is why +# this file exists at all. Public endpoints work for light use; run your own or use +# a provider for production reliability. +DECDN_RPC_URL=https://your-endpoint.example/rpc + +# OPTIONAL. Static AWS credentials for an S3 cache pull-through origin. The role +# never writes these into node.toml (that file is 0640 and diffable) — it only +# templates `source = "default-chain"` there (decdn_cache_origin_s3_use_default_chain: +# true), and the AWS default credential chain reads exactly these variables. +# Unnecessary on an instance with an IAM role / instance profile. +#AWS_ACCESS_KEY_ID=AKIA... +#AWS_SECRET_ACCESS_KEY=... +#AWS_SESSION_TOKEN=... # assume-role / SSO flows only diff --git a/ansible/roles/decdn_node/tasks/main.yml b/ansible/roles/decdn_node/tasks/main.yml index e771d91..21b2e3f 100644 --- a/ansible/roles/decdn_node/tasks/main.yml +++ b/ansible/roles/decdn_node/tasks/main.yml @@ -59,10 +59,58 @@ decdn_node_generate_keystore must be a boolean (true/false), not a quoted string — it gates a `| bool` conditional on host-side wallet generation. +# The RPC URL has TWO authoring paths (#39) and the choice is inferred from host +# state, so it has to be established BEFORE the asserts that depend on it. Either +# the operator supplies decdn_rpc_url from inventory (the role then authors the +# env file, overwriting whatever is on the host), or they provision a 0600 +# /etc/decdn/decdn.env on the target themselves and the secret never touches the +# control machine at all. +# +# `stat.exists` alone cannot tell those apart — a file the role itself wrote on a +# previous converge exists just as much as an operator-provisioned one. That +# ambiguity is dangerous in BOTH directions, so after every converge that gets the +# daemon running the role records WHICH PATH wrote the file and what its contents +# hashed to, as " " in decdn_env_checksum_file. The source token is +# load-bearing and not derivable from the hash: the record is written on both +# paths, so a matching hash means only "unchanged since the last converge". +# source=host -> the operator owns this file +# source=inventory -> this role rendered it from decdn_rpc_url +# no record -> provenance unknown (a host predating this mechanism) +# hash mismatch -> the file changed since that converge +# get_checksum is what makes the hash comparison possible; the value is a sha256 of +# a secret-bearing file, not the secret. +- name: Check for a host-provisioned secret environment file + ansible.builtin.stat: + path: "{{ decdn_env_file }}" + get_checksum: true + checksum_algorithm: sha256 + register: decdn_env_pre_stat + +# Deliberately not `slurp`-ing the env file itself anywhere in this role — only +# this 0600 record of its hash, which is not sensitive. A missing record is the +# normal state on a host that has never run a post-#39 converge, so a read failure +# is an expected outcome rather than an error. +- name: Read the env-file checksum this role last recorded + ansible.builtin.slurp: + src: "{{ decdn_env_checksum_file }}" + register: decdn_env_recorded + failed_when: false + changed_when: false + +# Both fields default to "" — treated exactly like "no record", which is also the +# right answer for a truncated or older single-field record: warn, do not fail. +- name: Resolve the env file's provenance + ansible.builtin.set_fact: + decdn_env_record_source: >- + {{ ((decdn_env_recorded.content | default('') | b64decode | trim).split() | length == 2) + | ternary((decdn_env_recorded.content | default('') | b64decode | trim).split()[0], '') }} + decdn_env_recorded_sum: >- + {{ ((decdn_env_recorded.content | default('') | b64decode | trim).split() | length == 2) + | ternary((decdn_env_recorded.content | default('') | b64decode | trim).split()[1], '') }} + - name: Require chain + contract configuration ansible.builtin.assert: that: - - decdn_rpc_url | length > 0 - decdn_chain_id | int > 0 - decdn_payment_pool_address is match('^0x[0-9a-fA-F]{40}$') - decdn_capacity_bond_address is match('^0x[0-9a-fA-F]{40}$') @@ -74,7 +122,7 @@ - decdn_content_blacklist_address != decdn_zero_address - decdn_region is match('^[A-Z]{2}$') fail_msg: >- - decdn_rpc_url, a positive decdn_chain_id, the FOUR contract addresses + A positive decdn_chain_id, the FOUR contract addresses (0x + 40 hex, none the zero address), and an uppercase ISO-3166 2-letter decdn_region are required. decdn_content_blacklist_address joined this list upstream (ADR 011/031): the daemon now refuses to resolve a config without a @@ -85,6 +133,136 @@ vars: decdn_zero_address: "0x0000000000000000000000000000000000000000" +# The RPC endpoint gate. Mirrors the pre-start keystore gate below: the role does +# not insist on carrying the secret itself, it insists the secret EXISTS — either +# in inventory, or already on the host at 0600. Only "neither" is a failure. +# +# The provisioning commands below are deliberately heredoc-free and reference NO +# `decdn` user: this assert fires ~700 tasks before "Create decdn system user", so +# on a never-converged host that account does not exist yet and `install -o decdn` +# / `chown decdn:decdn` would fail. Creating the file as root is correct — "Lock +# down a host-provisioned env file" chowns it to {{ decdn_user }} on the converge +# that follows. (A heredoc cannot survive this message either: an indented `EOF` +# does not terminate `<<'EOF'`, and the block's own indentation would prefix the +# written line with spaces, which the DECDN_RPC_URL grep below then rejects.) +- name: Require an RPC endpoint — inventory value or a host-provisioned env file + ansible.builtin.assert: + that: + - decdn_rpc_url | length > 0 or decdn_env_pre_stat.stat.exists + fail_msg: |- + No RPC endpoint: decdn_rpc_url is empty and {{ decdn_env_file }} does not + exist on this host. The URL may embed a provider API key, so the preferred + posture is to provision it on the host and keep it off the control machine + entirely. Run this on the TARGET, as root (the {{ decdn_user }} account may + not exist yet — this role creates it, and chowns the file on the next run). + umask FIRST, so the file is never briefly world-readable; the explicit chmod + is the belt-and-braces, since sudo may apply its own umask: + umask 077 + sudo mkdir -p {{ decdn_etc }} + echo 'DECDN_RPC_URL=https://…' | sudo tee {{ decdn_env_file }} >/dev/null + sudo chmod 600 {{ decdn_env_file }} + Add any other environment-borne secrets (e.g. AWS_ACCESS_KEY_ID / + AWS_SECRET_ACCESS_KEY for an S3 cache origin) as further KEY=value lines in + that same file — see roles/decdn_node/files/decdn.env.example. This role + then leaves the file's CONTENT alone and only enforces 0600 ownership. + Alternatively set decdn_rpc_url (plus any decdn_extra_env) in the git-ignored + host_vars//secret.yml and the role will author the file for you. + +# stat.exists is true for a directory or a symlink. Both are hazards, not merely +# unsupported: "Lock down a host-provisioned env file" runs `file` as root, and +# with the module's default follow:true a symlink planted by a compromised +# {{ decdn_user }} account (which owns the 0750 {{ decdn_etc }}) would have root +# chown+chmod the TARGET — /etc/shadow, say. A symlink also has no stat.checksum, +# which would surface much later as an undefined-variable crash. Refuse early. +- name: Require the env file to be a regular file + ansible.builtin.assert: + that: + - decdn_env_pre_stat.stat.isreg | default(false) + fail_msg: >- + {{ decdn_env_file }} exists but is not a regular file + (symlink/directory/device). This role chowns and chmods that path as root, + so following a link out of {{ decdn_etc }} would apply 0600 {{ decdn_user }} + ownership to whatever it points at. Replace it with a real file. To keep the + secret on other storage, bind-mount it there rather than symlinking. + when: decdn_env_pre_stat.stat.exists + +# The dangerous half of inferring the path from host state. An empty decdn_rpc_url +# is meant to say "the operator provisioned this file" — but it also happens when +# the git-ignored secret.yml is simply ABSENT: a fresh clone, a new control +# machine, a CI runner, an undecrypted vault. Without this check the role would +# then adopt its own previously-written file as though it were operator-supplied, +# and a ROTATED url in the missing secret.yml would silently never land: the grep +# finds the role's own stale DECDN_RPC_URL line, the checksum is unchanged, no +# restart is queued, and the deploy is green. Before #39 that state failed loud at +# "Require chain + contract configuration"; this restores that. +- name: Refuse to adopt a role-authored env file as host-provisioned + ansible.builtin.assert: + that: + - decdn_env_record_source != 'inventory' + or decdn_env_recorded_sum != (decdn_env_pre_stat.stat.checksum | default('')) + fail_msg: >- + decdn_rpc_url is empty, but {{ decdn_env_file }} is byte-for-byte the file + THIS ROLE last wrote from inventory — so the value has gone missing from the + control machine (an absent host_vars//secret.yml, a fresh clone, an + undecrypted vault) rather than being deliberately handed to the host. + Treating it as host-provisioned would deploy green while silently keeping the + old endpoint. Restore decdn_rpc_url, or — if you really are migrating this + node to the host-provisioned path — hand the file over explicitly by + discarding the role's provenance record: + sudo rm {{ decdn_env_checksum_file }} + when: decdn_rpc_url | length == 0 + +# The mirror direction, and the reason it is an assert rather than the `debug` it +# started as: the role now actively tells operators to keep AWS credentials in +# this file, and an inventory decdn_rpc_url rewrites it WHOLESALE. Losing those +# lines produces exactly the outcome "Reject decdn_extra_env" exists to prevent — +# an S3 403 hours later behind a green deploy — so it gets the same treatment. +# Scoped to a file whose provenance is KNOWN to be foreign: with no record (a +# pre-#39 host's first converge) this cannot distinguish anything, so it stays +# quiet and the run seeds the record for next time. +- name: Require confirmation to overwrite an env file this role did not write + ansible.builtin.assert: + that: + - decdn_env_overwrite_host_file | bool + fail_msg: >- + decdn_rpc_url is set, but {{ decdn_env_file }} on this host is not a file + this role authored — the provenance record reads + "{{ decdn_env_record_source | default('', true) }}" and/or its checksum no + longer matches, so it was written or edited by someone else. Rendering it + from inventory would DISCARD those lines (any + AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY behind an S3 origin included), and + because this role never reads the file it cannot tell you what would be + lost. Clear decdn_rpc_url to keep the host's copy, fold the host's extra + lines into decdn_extra_env, or set decdn_env_overwrite_host_file: true to + confirm the file is the role's to author. + when: + - decdn_rpc_url | length > 0 + - decdn_env_pre_stat.stat.exists + - decdn_env_recorded_sum | length > 0 + - >- + decdn_env_record_source != 'inventory' + or decdn_env_recorded_sum != (decdn_env_pre_stat.stat.checksum | default('')) + +# The env file is authored ALL-OR-NOTHING from inventory: the copy below renders +# DECDN_RPC_URL *and* decdn_extra_env in one go, and it is skipped entirely when +# decdn_rpc_url is empty. So extra_env alongside a host-provisioned file would be +# silently dropped — surfacing hours later as an S3 403 with nothing in the deploy +# output to explain it (AGENTS.md hard-rule 4). Fail loud instead. +- name: Reject decdn_extra_env when the env file is host-provisioned + ansible.builtin.assert: + that: + - decdn_extra_env | default({}, true) | length == 0 + fail_msg: >- + decdn_extra_env is set but decdn_rpc_url is empty, so this role will NOT + write the host-provisioned {{ decdn_env_file }} and those variables would + never reach the daemon. Append them as KEY=value lines to + {{ decdn_env_file }} on the host instead, or move BOTH decdn_rpc_url and + decdn_extra_env into host_vars//secret.yml so the role authors the + whole file. + when: + - decdn_rpc_url | length == 0 + - decdn_env_pre_stat.stat.exists + # slash_appeal_address is optional (only `decdn appeal slash` uses it — the daemon # validates but ignores the key), so it is not in the hard assert above. But if an # operator sets it, a half-filled/zero value would silently render a bad node.toml, @@ -261,8 +439,9 @@ decdn_cache_origin_s3_use_default_chain must be a real boolean, and decdn_cache_origin_s3_profile (when set) must contain no quote, whitespace or backslash. Static AWS keys are never written to node.toml — put - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN in - decdn_extra_env (git-ignored secret.yml) and leave the default chain to read them. + AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN in the 0600 + {{ decdn_env_file }} on the host (or in decdn_extra_env, if the role authors + that file) and leave the default chain to read them. - name: Reject S3 credential knobs alongside the multi-origin form ansible.builtin.assert: @@ -274,8 +453,8 @@ only to the single [cache.origin] table — the template does not render a credentials sub-table for [[cache.origins]] entries, so setting them with decdn_cache_origins would SILENTLY drop them. Omit the credentials block (the - AWS default chain is used anyway when it is absent) and supply keys via - decdn_extra_env, or switch to the single-origin form. + AWS default chain is used anyway when it is absent) and supply keys via the + 0600 {{ decdn_env_file }}, or switch to the single-origin form. when: decdn_cache_origins | length > 0 # --- Optional tuning knobs ("" => omit + use the daemon default) -------------- @@ -912,19 +1091,151 @@ # The RPC URL is the one sensitive value (it may embed an API key), so it lives # in a 0600 EnvironmentFile (DECDN_RPC_URL, read by the binary's clap env) rather # than node.toml. That keeps node.toml non-secret and diffable under `--check`. -- name: Write the secret environment file +# +# Two authoring paths, decided by the gate near the top of this file (#39): +# * decdn_rpc_url set -> the role authors the file from inventory (below). The +# value transits the control machine on every converge and OVERWRITES any +# host-provisioned file. +# * decdn_rpc_url empty -> the operator wrote the file on the host. The role +# never reads its content back (that would put the secret on the control +# machine, which is the whole point of the host path); it only enforces +# ownership + 0600 and greps for a non-empty DECDN_RPC_URL line. +- name: Write the secret environment file (inventory-supplied) ansible.builtin.copy: dest: "{{ decdn_env_file }}" owner: "{{ decdn_user }}" group: "{{ decdn_group }}" mode: "0600" + # to_json, NOT hand-rolled replace() chains. Inside a YAML literal block Jinja + # takes '\\' as TWO literal backslashes, so the obvious + # `replace('\\', '\\\\') | replace('"', '\\"')` silently escapes the wrong + # thing: it never matches a lone backslash, and it renders a quote as \\" — + # an escaped backslash followed by a CLOSING quote, truncating the value. + # to_json emits exactly the double-quoted, C-escaped form systemd's + # EnvironmentFile parser expects, and ensure_ascii=false keeps UTF-8 literal + # (systemd understands \xNN, not \uNNNN). It supplies the quotes itself, so a + # value with a space, '#', quote or backslash survives intact — the RPC URL + # included, which used to be interpolated raw. content: | - DECDN_RPC_URL={{ decdn_rpc_url }} + DECDN_RPC_URL={{ decdn_rpc_url | string | to_json(ensure_ascii=false) }} {% for k, v in (decdn_extra_env | default({}, true)) | dictsort %} - {{ k }}="{{ v | string | replace('\\', '\\\\') | replace('"', '\\"') }}" + {{ k }}={{ v | string | to_json(ensure_ascii=false) }} {% endfor %} no_log: true # the RPC URL may embed an API key + register: decdn_env_copy + notify: Restart decdn-node + when: decdn_rpc_url | length > 0 + +# The copy above is no_log, so a run that replaced an existing file with +# control-machine values shows a bare "changed" and nothing else. Say what was +# lost. A file with a KNOWN-foreign provenance already failed the assert at the +# top of this file, so what is left here is the unknowable case: no record yet +# (a pre-#39 host's first converge). Gated on `is changed` as well, so it speaks +# up only when the contents actually moved rather than nagging every converge. +- name: Warn that an inventory decdn_rpc_url overwrote an untracked env file + ansible.builtin.debug: + msg: >- + decdn_rpc_url is set in inventory, so {{ decdn_env_file }} was REWRITTEN + from control-machine values, and this host carried no provenance record for + it — any KEY=value lines added on the host that are not in decdn_extra_env + are gone. From this converge on, that file is tracked: an edit made outside + Ansible will fail the deploy instead of being silently discarded. Clear + decdn_rpc_url (and decdn_extra_env) to hand the file back to the host. + when: + - decdn_rpc_url | length > 0 + - decdn_env_pre_stat.stat.exists + - decdn_env_recorded_sum | length == 0 + - decdn_env_copy is changed + +# Content untouched (no `content:`/`src:`) — this only asserts the file is owned +# by the daemon user and unreadable by anyone else, the same lock-down the +# operator-provisioned keystore gets above. +- name: Lock down a host-provisioned env file (ownership + mode only) + ansible.builtin.file: + path: "{{ decdn_env_file }}" + owner: "{{ decdn_user }}" + group: "{{ decdn_group }}" + mode: "0600" + when: decdn_rpc_url | length == 0 + +# `decdn config validate` further down sources this file, so a missing +# DECDN_RPC_URL would eventually surface — but as an opaque daemon error about an +# unresolvable config. Name the actual cause here instead. grep -q keeps the +# secret out of stdout AND out of argv; deliberately NOT slurp, which would copy +# the secret to the control machine and defeat the point of the host path. +- name: Confirm a host-provisioned env file carries a non-empty DECDN_RPC_URL + ansible.builtin.command: + cmd: grep -qE '^[[:space:]]*DECDN_RPC_URL=[^[:space:]]' {{ decdn_env_file | quote }} + register: decdn_env_grep + changed_when: false + # Read-only, so it must still run under `make check` — otherwise the one gate + # that can catch a malformed host file is exactly the gate a dry run skips. + check_mode: false + # Non-fatal here so the next task can turn rc into an actionable message + # (mirrors the config-validation pair below). + failed_when: false + when: decdn_rpc_url | length == 0 + +- name: Report a host-provisioned env file with no usable DECDN_RPC_URL + ansible.builtin.fail: + msg: >- + {% if decdn_env_grep.rc is not defined %} + The {{ decdn_env_file }} content check could not be RUN + ({{ decdn_env_grep.msg | default('no rc and no msg — re-run with -vvv', true) }}), + so the host-provisioned env file is UNVERIFIED and the daemon must not be + started on it. + {% elif decdn_env_grep.rc | int > 1 %} + {{ decdn_env_file }} could not be read (grep exited + {{ decdn_env_grep.rc }}: {{ decdn_env_grep.stderr | default('no stderr', true) }}). + {% else %} + {{ decdn_env_file }} exists but carries no `DECDN_RPC_URL=` line with a + non-whitespace value, + so the daemon would start with no chain endpoint. Add one — note systemd + SKIPS a line its shell-like parser cannot read, logging a warning to + `journalctl -u decdn-node` that is easy to miss, so single-quote a value + containing a space, quote, backslash or `$`: + echo 'DECDN_RPC_URL=https://your-endpoint.example/rpc' | sudo tee -a + {{ decdn_env_file }} >/dev/null + {% endif %} + when: + - decdn_rpc_url | length == 0 + - decdn_env_grep.rc | default(1) != 0 + +# Re-stat after the file is final (the inventory path may just have rewritten it) +# so the comparison below, and the record written at the very end of this role, +# both describe what the daemon is about to be started on. +# +# Not no_log: the value is a sha256, not the secret, and hiding it would censor a +# module failure here into an unactionable "output has been hidden" — while the +# `changed` it drives is the only explanation an operator gets for a restart. +- name: Re-stat the env file now that it is final + ansible.builtin.stat: + path: "{{ decdn_env_file }}" + get_checksum: true + checksum_algorithm: sha256 + register: decdn_env_post_stat + +# Restart on an OUT-OF-BAND edit. The copy above already notifies when the role +# itself authors a changed file, so this covers only the host-provisioned path, +# where nothing else would notice that the operator edited decdn.env between +# converges — the daemon would keep running on the previous values behind a fully +# green deploy. Handlers dedupe and flush before the daemon is started, so this +# costs at most one restart. +# +# An ABSENT record is deliberately not a change: it means this host has never run +# a post-#39 converge, and treating "not tracked yet" as "edited" would bounce +# every node in the fleet on the upgrade run for no functional reason. +- name: Note an out-of-band change to the host-provisioned env file + ansible.builtin.debug: + msg: >- + {{ decdn_env_file }} has changed since the last converge recorded it — + restarting decdn-node so the daemon picks the new values up. + changed_when: true notify: Restart decdn-node + when: + - decdn_rpc_url | length == 0 + - decdn_env_recorded_sum | length > 0 + - (decdn_env_post_stat.stat.checksum | default('')) != decdn_env_recorded_sum - name: Template node.toml (non-secret — RPC URL is in the env file) ansible.builtin.template: @@ -956,8 +1267,13 @@ executable: /bin/bash cmd: | set -euo pipefail + # Exit 3 is reserved for "the env file could not be sourced" so the report + # task below does not blame node.toml for a quoting typo in decdn.env. NOTE + # this is bash, not systemd's parser: it expands $VAR and executes $(...) + # and backticks, which systemd would pass through literally. Operators are + # told to single-quote values (files/decdn.env.example). set -a - . {{ decdn_env_file | quote }} + . {{ decdn_env_file | quote }} || { echo "env-file could not be sourced" >&2; exit 3; } set +a {{ decdn_cli_bin | quote }} config validate \ --config {{ decdn_config_file | quote }} \ @@ -980,7 +1296,13 @@ - name: Report the config-validation failure ansible.builtin.fail: msg: >- - {% if decdn_config_validate.rc is not defined %} + {% if decdn_config_validate.rc | default(0) == 3 %} + {{ decdn_env_file }} could not be sourced, so the config was never + validated: {{ decdn_config_validate.stderr | default('(no output)', true) }}. + This is a shell-syntax fault in the ENV FILE, not a problem with + {{ decdn_config_file }} — most often an unbalanced quote, or an unquoted + value containing a space. Single-quote the value and re-run. + {% elif decdn_config_validate.rc is not defined %} `decdn config validate` could not be RUN ({{ decdn_config_validate.msg | default('no rc and no msg — re-run with -vvv', true) }}). {{ decdn_config_file }} is therefore UNVALIDATED and the daemon must not be @@ -1022,6 +1344,41 @@ enabled: true state: started +# Record the env file's hash LAST, and only here — deliberately after the handler +# flush and the start above, so the record means "the daemon is running on this +# content", not "Ansible saw this content". +# +# Writing it earlier (next to the comparison) looks equivalent and is not: the +# handler that acts on a detected change does not flush until well after the +# node.toml template and the `decdn config validate` gate, both of which can abort +# the play. A record written before that abort would say "already handled" on the +# NEXT converge — the comparison would find no change, no restart would be queued, +# and `state: started` is a no-op on a running unit — so a single failed run would +# permanently burn the one signal that the secret had been rotated. Writing it +# here means an aborted run simply leaves the old record in place and the next +# converge re-detects the change. +# +# No notify: by the time this runs, any restart this converge owed has happened. +- name: Record the env-file checksum (the daemon is now running on it) + ansible.builtin.copy: + dest: "{{ decdn_env_checksum_file }}" + owner: root + group: root + mode: "0600" + # " " — the source token is what a later converge uses to tell + # "the operator owns this file" from "I rendered this from inventory". + # Parenthesised: filters bind tighter than comparisons, so an unbracketed + # `decdn_rpc_url | length > 0 | ternary(...)` parses as + # (decdn_rpc_url | length) > (0 | ternary(...)). + content: > + {{ (decdn_rpc_url | length > 0) | ternary('inventory', 'host') }} + {{ decdn_env_post_stat.stat.checksum }} + # Under `--check` on an unprovisioned host the env-file copy only SIMULATES the + # write, so there is nothing to hash and stat.checksum is undefined. Requiring + # the checksum itself (not merely stat.exists) also keeps a non-regular file — + # which stat reports as existing but never checksums — from templating undefined. + when: (decdn_env_post_stat.stat.checksum | default('')) | length > 0 + # --- Readiness ---------------------------------------------------------------- # ADVISORY probe: the daemon binds its loopback /metrics listener only after its # startup bootstrap completes, which on a healthy node can lag process start by a diff --git a/ansible/roles/decdn_node/templates/node.toml.j2 b/ansible/roles/decdn_node/templates/node.toml.j2 index b0715f5..1d3bcb8 100644 --- a/ansible/roles/decdn_node/templates/node.toml.j2 +++ b/ansible/roles/decdn_node/templates/node.toml.j2 @@ -355,7 +355,9 @@ decompress = "{{ o.decompress }}" S3 credentials. Only the `default-chain` source is templated: `static` would put an access key into this 0640, diffable file. Supply static keys as AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN in the 0600 - {{ decdn_env_file }} instead — the default chain reads exactly those. + {{ decdn_env_file }} instead — the default chain reads exactly those. Either + write them there on the host, or (when the role authors that file, i.e. + decdn_rpc_url is set) pass them as decdn_extra_env. #} {% if not _multi and decdn_cache_origin_kind == 's3' and decdn_cache_origin_s3_use_default_chain | bool %}