feat(decdn_node): allow a host-provisioned /etc/decdn/decdn.env (#39)

.github/workflows/molecule.yml

@@ -40,6 +40,7 @@ jobs:
         # (operator-provisioned keystore + rendered-value assertions), schema
         # (config key-set drift against the upstream field list), validation
         # (bad knobs must be rejected by the role's own asserts), generate-keystore
-        # (opt-in host-side wallet generation), and slow-readiness (advisory
-        # /metrics probe timeout).
+        # (opt-in host-side wallet generation), host-env (host-provisioned
+        # /etc/decdn/decdn.env — no secret on the control machine), and
+        # slow-readiness (advisory /metrics probe timeout).
         run: molecule test --all

AGENTS.md

@@ -51,8 +51,10 @@ ansible/                # the deployment project (DevSec-hardened, lean roles)
   tarball — verified against the release's GPG-signed `SHA256SUMS` — or, while upstream
   has no release tag cut (the current default), from locally-built binaries; under a hardened
   systemd unit; public QUIC udp/4433, loopback metrics/admin, operator-provisioned eth
-  keystore, required chain knobs (no baked protocol facts — sourced from ADRs), over a
-  shared DevSec-hardened `baseline`. See `ansible/README.md`. (On-chain node
+  keystore, operator-provisionable secret env file, and required chain knobs (no baked
+  protocol facts — sourced from ADRs), over a shared DevSec-hardened `baseline`.
+  Leaving `decdn_rpc_url` empty means the operator wrote `0600 /etc/decdn/decdn.env`
+  on the host and the role only gates on it, so no secret transits the control machine. See `ansible/README.md`. (On-chain node
   stake/registration, ADR 019 Phase 2, is a manual operator step, driven by `decdn setup`.)
 
   **Config-schema coupling.** `roles/decdn_node/templates/node.toml.j2` renders against

ansible/README.md

@@ -25,9 +25,11 @@ baseline   host hardening — DevSec os/ssh, nftables default-deny inbound,
   operators must not install or enable ufw, which would replace the nftables ruleset and
   drop QUIC/SSH.
 - **No secrets in the repo.** The node's eth keystore is **operator-provisioned** and never
-  generated here; its `rpc_url` (which may embed an API key) lives in a git-ignored
-  `host_vars/<node>/secret.yml` (the rest of `host_vars` is committed, non-secret config) and
-  is rendered to a `0600` config.
+  generated here; its `rpc_url` (which may embed an API key) is provisioned the same way —
+  a `0600 /etc/decdn/decdn.env` written on the target host, which the role gates on but
+  never reads back — or, if you prefer, carried in a git-ignored
+  `host_vars/<node>/secret.yml` (the rest of `host_vars` is committed, non-secret config)
+  and rendered to that same `0600` file.
 - **Host hardening via DevSec** (`os_hardening` + `ssh_hardening`): key-only SSH, no root
   login, kernel/sysctl/PAM hardening — applied last, after the admin key is in place.
   baseline overrides a few `os_hardening` sysctls so hardening can't sever node
@@ -54,11 +56,29 @@ make deps                                    # vendor pinned collections into ./
 cp inventory/hosts.yml.example inventory/hosts.yml
 $EDITOR inventory/hosts.yml                   # set hosts for decdn_nodes
 $EDITOR inventory/group_vars/all.yml          # optional: override admin user/keys, allowlists
-# Per-node RPC secret (the rest of host_vars/<node>/main.yml is committed config):
+```
+
+The per-node **secret** (`decdn_rpc_url`, which may embed a provider API key, plus any
+environment-borne secrets like AWS keys for an S3 origin) has two homes — pick one.
+**Preferred:** provision a `0600 /etc/decdn/decdn.env` on the target host and leave
+`decdn_rpc_url` empty, so nothing sensitive is stored on or transits this machine. The role
+then leaves that file's contents alone, enforcing only `0600 decdn:decdn`. Full commands and
+the file format are in
+[`roles/decdn_node/README.md` § Secrets](roles/decdn_node/README.md#secrets) — the single
+canonical copy; don't duplicate them here.
+
+**Or** carry it in inventory, and the role authors `decdn.env` from those values on every run:
+
+```bash
 cp inventory/host_vars/decdn-node-1/secret.yml.example inventory/host_vars/decdn-node-1/secret.yml
 $EDITOR inventory/host_vars/decdn-node-1/secret.yml   # set decdn_rpc_url
 ```
 
+The role fails loud when neither exists, and — once it has recorded a checksum for the file —
+when the two would collide: an inventory value that would discard a host-side edit, or an
+empty `decdn_rpc_url` that turns out to mean "`secret.yml` went missing" rather than "the host
+owns this file".
+
 Your `hosts.yml` is no longer force-ignored — commit it in your fork if you want, or keep
 it local.
 
@@ -86,8 +106,10 @@ unless you listed one).
    `decdn_node_manual_bin_src` / `decdn_cli_manual_bin_src`).
 2. Per-node config in `inventory/host_vars/<node>/main.yml` (committed) —
    the install method + binary sources, the **four** required contract addresses,
-   `decdn_region`, cache origin, … — plus the one secret, `decdn_rpc_url`, in a sibling
-   git-ignored `secret.yml` (copy the shipped `host_vars/decdn-node-1/secret.yml.example`).
+   `decdn_region`, cache origin, … — plus the one secret, `decdn_rpc_url`, either
+   provisioned as `/etc/decdn/decdn.env` on the host (preferred — see
+   [Setup](#setup)) or in a sibling git-ignored `secret.yml` (copy the shipped
+   `host_vars/decdn-node-1/secret.yml.example`).
    The committed `main.yml` already carries the current Arbitrum Sepolia addresses; edit
    `decdn_region`, the binary paths and the cache origin for your node. Contract addresses
    are protocol facts — copy them from the upstream deployment manifest
@@ -188,7 +210,7 @@ a non-mutating dry run.
 
 Defaults live in each role (`roles/*/defaults/main.yml`); override in `group_vars`
 (shared) or `host_vars` (per node; `main.yml` committed config, `secret.yml` git-ignored for
-the RPC URL). Highlights:
+the RPC URL when it is not provisioned on the host instead). Highlights:
 
 | Var | Default | Notes |
 |-----|---------|-------|
@@ -198,8 +220,9 @@ the RPC URL). Highlights:
 | `baseline_preserve_ipv6_autoconf` | `true` | Keep IPv6 RA/autoconf under hardening; set `false` for static-IPv6 hosts. |
 | `baseline_rp_filter_loose` | `false` | `true` loosens reverse-path filtering (`rp_filter=2`) for multi-homed nodes. |
 | `decdn_node_version` | `""` | **required**; a `v<version>` release must exist. |
-| `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node — `rpc_url` in `host_vars/<node>/secret.yml`, addresses in `main.yml`; sourced from an ADR/deployment. |
+| `decdn_rpc_url` + 3 contract addresses | `""` | **required** per node — `rpc_url` from a host-provisioned `0600 /etc/decdn/decdn.env` (preferred) *or* `host_vars/<node>/secret.yml`, addresses in `main.yml`; sourced from an ADR/deployment. |
 | `decdn_region` / `decdn_bind_port` / `decdn_rate_per_mb` | `""` / `4433` / `10` | node identity, QUIC port, USDC base units/MB. |
+| `decdn_env_checksum_file` / `decdn_env_overwrite_host_file` | `/etc/decdn/.decdn.env.sha256` / `false` | Provenance record for the secret env file (`0600 root`), and the opt-in that lets an inventory `decdn_rpc_url` overwrite a host-edited one. |
 
 ---
 

ansible/galaxy/CHANGELOG.md

@@ -31,6 +31,28 @@ Not yet published to Galaxy (pre-1.0; the published shape may still change).
 - `decdn_extra_env` — extra `KEY: value` pairs appended to the `0600` env file. The
   supported home for environment-borne secrets, notably the AWS credentials behind
   an S3 cache origin (which are deliberately never written to the `0640` `node.toml`).
+- **Host-provisioned secrets.** `decdn_rpc_url` may now be left empty when the
+  operator has written a `0600 /etc/decdn/decdn.env` on the target host: the role
+  then leaves that file's *content* alone (it never reads it back to the control
+  machine), enforcing only `0600 decdn:decdn` and checking that a non-empty
+  `DECDN_RPC_URL=` line is present. Setting `decdn_rpc_url` keeps the previous
+  behaviour and overwrites the host file from inventory. The role fails loud, with
+  the provisioning commands, when neither is present — and rejects
+  `decdn_extra_env` against a host-provisioned file rather than silently dropping
+  it. A template for the host file ships at
+  `roles/decdn_node/files/decdn.env.example`.
+- `decdn_env_checksum_file` (default `/etc/decdn/.decdn.env.sha256`, `0600 root`) —
+  a role-managed `<source> <sha256>` record for the env file, written on both
+  authoring paths after the daemon is running. It restarts the unit when an **out-of-band** edit of
+  a host-provisioned `decdn.env` is detected (instead of leaving the daemon on stale
+  values behind a green deploy), and doubles as the file's provenance marker: an
+  empty `decdn_rpc_url` against a file the role itself wrote fails loud (a missing
+  `secret.yml` is not the same as handing the file to the host), and an inventory
+  `decdn_rpc_url` against a file someone else wrote fails loud rather than
+  discarding it — override with `decdn_env_overwrite_host_file: true`.
+  **Upgrade note:** hosts deployed before this have no record, so the first
+  converge seeds one. That first run deliberately does *not* restart the daemon,
+  and the role warns rather than fails if it rewrites an untracked file.
 - `ExecReload` on the unit plus a `Reload decdn-node` handler, for the five
   hot-reloadable config sections.
 - Full config-schema parity with `decdn/decdn` @ `d306cc5c`: `[network.discovery]`,
@@ -45,6 +67,24 @@ Not yet published to Galaxy (pre-1.0; the published shape may still change).
   keystore is also absent), never overwriting an existing wallet. Funding + on-chain
   staking/registration remain a manual step.
 
+### Fixed
+
+- **`decdn_extra_env` values were escaped incorrectly** in the `0600` env file, and
+  `DECDN_RPC_URL` was not escaped at all. Inside a YAML literal block Jinja reads
+  `'\\'` as *two* literal backslashes, so the role's hand-rolled
+  `replace('\\', '\\\\') | replace('"', '\\"')` chain never matched a lone backslash
+  and rendered a double quote as `\\"` — an escaped backslash followed by a
+  *closing* quote. Any value containing a quote was truncated there and any
+  backslash was passed through unescaped; systemd then skipped the line it could
+  not parse and the variable was simply **absent** at runtime (an S3 origin whose
+  secret key contained a quote or backslash would 403 with nothing to show why).
+  Both are now rendered with `to_json(ensure_ascii=false)`, which emits exactly the
+  double-quoted, C-escaped form `EnvironmentFile=` expects. Values containing a
+  space, `#`, quote, backslash or non-ASCII character now survive intact.
+  **Upgrade note:** this changes the rendered file (`DECDN_RPC_URL` gains quotes),
+  so the first converge on the inventory path rewrites it and restarts the daemon
+  once.
+
 ### Changed (BREAKING)
 
 The collection has never been published, so this is not a break against any

ansible/galaxy/galaxy.yml

@@ -38,5 +38,9 @@ repository: https://github.com/decdn/devops
 documentation: https://github.com/decdn/devops/tree/main/ansible
 homepage: https://decdn.org
 issues: https://github.com/decdn/devops/issues
-build_ignore:
-  - "*.example"
+# No build_ignore. It used to carry "*.example" as a guard against shipping an
+# inventory secret template, but build.sh stages ONLY roles/ + this overlay — the
+# inventory never enters the tree, so the pattern excluded nothing real while
+# silently dropping roles/decdn_node/files/decdn.env.example, which external
+# operators need (it is the template for the host-provisioned 0600 env file).
+# Add a NARROW, path-anchored pattern here if something must ever be excluded.

ansible/inventory/group_vars/decdn_nodes.yml

@@ -1,7 +1,8 @@
 ---
 # Shared config for public deCDN node hosts (playbooks/site.yml).
-# Per-node values live in host_vars/<node>/: addresses/region/version in main.yml,
-# the sensitive rpc_url in a git-ignored secret.yml.
+# Per-node values live in host_vars/<node>/: addresses/region/version in main.yml.
+# The sensitive rpc_url is provisioned as /etc/decdn/decdn.env on the host, or (if
+# you would rather the role authored that file) in a git-ignored secret.yml.
 
 # Open the node's public QUIC listener in the baseline firewall. Metrics (9090)
 # and admin RPC (9191) stay loopback and get NO hole.

ansible/inventory/host_vars/decdn-node-1/main.yml

@@ -1,9 +1,11 @@
 ---
 # Per-node deployment values for decdn-node-1 (COMMITTED — non-secret config only).
 # Ansible merges every *.yml in this directory for host decdn-node-1. The sensitive
-# values (decdn_rpc_url, and any decdn_extra_env secrets) live in the sibling
-# secret.yml (git-ignored) — copy secret.yml.example to secret.yml and fill it in.
-# The node refuses to start until both this file and secret.yml are set.
+# values (decdn_rpc_url, and any environment-borne secrets) live OFF this file:
+# either in a 0600 /etc/decdn/decdn.env provisioned on the target host (preferred —
+# nothing sensitive on the control machine), or in the sibling git-ignored
+# secret.yml (copy secret.yml.example). The role refuses to start the node until
+# this file is set and ONE of those two secret sources exists.
 #
 # CONTRACT ADDRESSES ARE NOT INVENTED HERE. They are a verbatim copy of the
 # upstream deployment manifest decdn/contracts/deployments/421614.json (mirrored
@@ -34,7 +36,8 @@ decdn_node_manual_bin_src: ""   # MUST-EDIT — control-machine path to decdn-no
 decdn_cli_manual_bin_src: ""    # MUST-EDIT — control-machine path to decdn
 
 # --- Chain (Arbitrum Sepolia, chain 421614) -----------------------------------
-# The RPC endpoint (decdn_rpc_url) is the one sensitive value — set it in secret.yml.
+# The RPC endpoint is the one sensitive value — provision it as DECDN_RPC_URL in
+# /etc/decdn/decdn.env on the host, or set decdn_rpc_url in secret.yml.
 decdn_chain_id: 421614                                   # Arbitrum Sepolia
 # Required contracts. All four are mandatory upstream — a node will not resolve its
 # config without them.

ansible/inventory/host_vars/decdn-node-1/secret.yml.example

@@ -1,17 +1,28 @@
 ---
-# Per-node SECRETS for decdn-node-1. Copy to secret.yml (same directory,
-# git-ignored) and fill in. Ansible merges it with main.yml for this host. These
-# are the ONLY per-node values kept out of git — everything else lives in the
-# committed main.yml.
+# Per-node SECRETS for decdn-node-1 — the INVENTORY path, which is optional.
 #
-# MUST-EDIT — SENSITIVE (may embed an API key). The public endpoint below works
-# for light use; run your own or use a provider for production reliability.
+# You only need this file if you want the role to author /etc/decdn/decdn.env for
+# you. The preferred posture is the other way round: provision that 0600 file on
+# the target host (see roles/decdn_node/files/decdn.env.example) and skip this
+# file entirely — the secret then never sits in plaintext on your control machine
+# and never transits it on a converge. The role fails loud if NEITHER exists.
+#
+# If you do use it: copy to secret.yml (same directory, git-ignored) and fill in.
+# Ansible merges it with main.yml for this host. These are the ONLY per-node
+# values kept out of git — everything else lives in the committed main.yml.
+#
+# SENSITIVE (may embed an API key). Setting this makes the role REWRITE
+# /etc/decdn/decdn.env from these values on every run, discarding any lines added
+# on the host. The public endpoint below works for light use; run your own or use
+# a provider for production reliability.
 decdn_rpc_url: "https://sepolia-rollup.arbitrum.io/rpc"
 
-# Optional. Extra KEY: value pairs appended to the 0600 /etc/decdn/decdn.env that
-# systemd hands the daemon. This is where environment-borne secrets belong — most
-# notably the AWS credentials backing an S3 cache origin, which the role
-# deliberately does NOT write into node.toml (that file is 0640 and diffable):
+# Optional, and only meaningful alongside decdn_rpc_url above (the role writes the
+# env file all-or-nothing, and rejects extra_env against a host-provisioned file
+# rather than silently dropping it). Extra KEY: value pairs appended to the 0600
+# /etc/decdn/decdn.env that systemd hands the daemon — most notably the AWS
+# credentials backing an S3 cache origin, which the role deliberately does NOT
+# write into node.toml (that file is 0640 and diffable):
 #
 # decdn_extra_env:
 #   AWS_ACCESS_KEY_ID: "AKIA..."

ansible/molecule/default/converge.yml

@@ -17,6 +17,15 @@
     # stay in sync with the version string printed by files/decdn-node-stub.
     decdn_node_version: "0.0.0-molecule-stub"
     decdn_rpc_url: "https://rpc.example.invalid/"
+    # The ACCEPTED side of the decdn_extra_env gate (#39): with an inventory
+    # decdn_rpc_url the role authors the whole env file, so extra_env is rendered
+    # rather than rejected. Nothing else in any scenario sets it alongside a
+    # non-empty rpc_url, which left both this branch AND the template's
+    # backslash/quote escaping untested. The value carries a double-quote and a
+    # backslash on purpose — systemd SKIPS a line it cannot parse, so a broken
+    # escape means the variable is silently absent at runtime.
+    decdn_extra_env:
+      DECDN_MOLECULE_EXTRA: 'a "quoted" \ value'
     decdn_chain_id: 421614
     decdn_region: "US"
     decdn_payment_pool_address: "0x1111111111111111111111111111111111111111"