feat: close the sponsord operational gaps (top-up hold guard, monitoring, lifecycle, Compose gate page)

.claude/skills/grafana-dashboards/SKILL.md

@@ -40,7 +40,16 @@ their only home.
   `decdn-delivery`. Every rule carries a `component` label. Only rules with a matching
   section in upstream's `docs/runbook.md` (20 of 51) also carry a `runbook_url`.
 
-Add a row to an existing dashboard before starting a fifth one. The Helm chart renders
+- `sponsord/` — the onboarding sponsor's own pair: `dashboard-sponsord.json` (uid
+  `decdn-sponsord`) and `prometheus-alerts.yml` (group `sponsord`, `job="sponsord"`).
+  The chart does not render this subdirectory (sponsord has no Kubernetes path). Its
+  `sponsord_*` names come from upstream `decdn/sponsord` `crates/sponsord/src/metrics.rs`
+  (check them with the recipe after the node one below). Its logs are plain text,
+  not JSON: select `{unit="sponsord.service"}` and filter on the `level` label the
+  `grafana_alloy` role parses. The rules have promtool unit tests in
+  `charts/decdn-node/tests/sponsord-alerts_test.yml`: keep them in step.
+
+Add a row to an existing dashboard before starting a fifth node one. The Helm chart renders
 these files as-is (`templates/prometheusrule.yaml`, `templates/dashboards-configmap.yaml`),
 so a new `*.json` becomes a new ConfigMap and `charts/decdn-node/tests/render-test.sh`
 checks the rule and dashboard counts against the directory. Never pass them through
@@ -69,6 +78,18 @@ curl -s http://127.0.0.1:9090/metrics | grep -v '^#' | grep -oE '^decdn_[a-z0-9_
 comm -23 /tmp/used /tmp/exported   # must print nothing
 ```
 
+For sponsord, from the same directory (`sponsord_log_level` in a panel description is an
+Ansible variable, not a series):
+
+```bash
+grep -hv '^\s*#' sponsord/*.json sponsord/*.yml | grep -oE 'sponsord_[a-z0-9_]+' | sort -u > /tmp/used-sd
+curl -s http://127.0.0.1:8090/metrics | grep -v '^#' | grep -oE '^sponsord_[a-z0-9_]+' | sort -u > /tmp/exported-sd
+comm -23 /tmp/used-sd /tmp/exported-sd   # prints only sponsord_log_level
+```
+
+A code's `sponsord_request_errors_total{code}` series appears only after that code's first
+error, so a fresh daemon may not export it yet; check against upstream `metrics.rs` then.
+
 `decdn_iroh_*` series appear only once the node's endpoint is up, and a labelled family
 with no child (e.g. `decdn_staker_set_active_by_region` before any staker declares a
 region) emits nothing — read a stray hit with that in mind. `decdn_health` is a Prometheus

AGENTS.md

@@ -58,7 +58,7 @@ mirror (`vars/main/networks.yml`), never hand-copied into inventory or docs.
 
 ```
 ansible/                # the deployment project (DevSec-hardened, lean roles)
-  playbooks/            # site.yml (decdn node + sponsord), sponsord.yml (+ onramp), backup.yml, decommission.yml
+  playbooks/            # site.yml (decdn node + sponsord), sponsord.yml (+ onramp), backup.yml, decommission.yml (node + sponsord)
   roles/                # baseline, decdn_node, grafana_alloy, sponsord, sponsord_onramp
   inventory/ galaxy/ molecule/    # see ansible/README.md
 cloud-init/             # user-data{,-sponsord}.yaml + on-host bootstrap.sh; pinned ansible-core/collections (see its README.md)
@@ -88,7 +88,10 @@ holds the deCDN Grafana dashboards and Prometheus alert rules; upstream ships no
 `decdn_*` series a panel or rule names must be one `decdn-node` exports — check the
 node's `/metrics` and upstream's `adr/appendix-observability.md` registry (a `planned`
 row emits nothing). Nothing in CI checks the names, so a typo renders `(no data)` or a
-rule that never fires. See `.claude/skills/grafana-dashboards/SKILL.md`.
+rule that never fires. See `.claude/skills/grafana-dashboards/SKILL.md`. sponsord's pair
+lives in its `sponsord/` subdirectory, which the chart does not render; its `sponsord_*`
+names must be ones upstream `decdn/sponsord` `crates/sponsord/src/metrics.rs` exports,
+and its rules have promtool unit tests (`charts/decdn-node/tests/sponsord-alerts_test.yml`).
 
 ## Current services
 
@@ -125,7 +128,11 @@ rule that never fires. See `.claude/skills/grafana-dashboards/SKILL.md`.
   `playbooks/backup.yml` / `decommission.yml` (`make backup` / `make decommission`) are
   role entry points (`tasks_from: backup|decommission`): backups are encrypted on the host
   to operator public keys; decommission needs `LIMIT` + typed confirmation, keeps the
-  identity and never touches the chain.
+  identity and never touches the chain. Both also cover `sponsord_hosts` (and the onramp),
+  with one confirmation per run, in the first play that reaches a host, whose prompt
+  lists every service it covers (`_decdn_decommission_confirmed`; the
+  `decdn_decommission_*` cap and timeout are copied into all three roles' defaults,
+  `make test-scripts` checks).
   Leaving `decdn_rpc_url` empty means the operator wrote `0600 /etc/decdn/decdn.env`
   on the host and the role only gates on it, so no secret transits the control machine. See `ansible/README.md`. (On-chain node
   stake/registration, ADR 019 Phase 2, is a manual operator step, driven by `decdn setup`.)
@@ -186,6 +193,16 @@ rule that never fires. See `.claude/skills/grafana-dashboards/SKILL.md`.
   - **Config gate:** sponsord has no `config validate` and only env/flag config, so
     `/healthz` after start is the gate, and it is **fatal**. The daemon binds only
     after the keystore decrypts and the on-chain pool-owner check passes.
+  - **Top-up hold guard:** a restart or stop forgets a held, unconfirmed pool top-up
+    (`sponsord_pool_topup_unconfirmed_since_unix` > 0), and the pool can be refilled
+    twice. `tasks/topup-hold.yml` reads `/metrics` (at the address the running
+    process listens on) before the role queues a restart and before decommission
+    stops the daemon, and fails while a hold is on or `/metrics` does not answer.
+    **The restart-inputs comparison is the role's only restart trigger:** never add
+    `notify: Restart sponsord` to another task; make the file a hashed input
+    instead, or the restart skips the guard. `sponsord_restart_ignore_topup_hold`
+    overrides it (pass it as JSON). Compose cannot guard itself; its README has the
+    manual check.
   - **Alloy toggles:** `playbooks/group_vars/all.yml` derives
     `grafana_alloy_node_enabled` / `grafana_alloy_sponsord_enabled` from group
     membership. They are host-scoped so a co-located host's two plays render one
@@ -264,6 +281,11 @@ rule that never fires. See `.claude/skills/grafana-dashboards/SKILL.md`.
     read-only one by one into `/run/secrets/`. Listeners and secret paths are set in
     `environment:`, which beats env files, because the release images default to
     `0.0.0.0`.
+  - **Onramp entrypoint:** upstream serves whatever `ONRAMP_GATE_TEMPLATE` names, and
+    the container also mounts the daemon token and the Turnstile secret, so the
+    onramp starts through a `/bin/sh` check that the path resolves into
+    `/etc/sponsord/onramp-gate/`, then a flagless `exec`. `invariants.jq` pins that
+    script byte for byte (`onramp_entrypoint`): change both together.
   - **Caddy:** the official image by digest, non-root, keeping only
     `NET_BIND_SERVICE`. `compose/Caddyfile` is a hand-kept copy of the role's
     `Caddyfile.j2`: change one, change both.

Makefile

@@ -148,6 +148,7 @@ test-scripts:        ## test the Makefile and molecule-driver guards, release ga
 lint-helm:           ## helm lint + render tests + kubeconform + promtool + shared schema-key check (needs helm, yq, python3>=3.11, docker)
 	KUBECONFORM="docker run --rm -i $(KUBECONFORM_IMAGE)" \
 	PROMTOOL="docker run --rm -i --entrypoint promtool $(PROMTOOL_IMAGE)" \
+	PROMTOOL_IMAGE="$(PROMTOOL_IMAGE)" \
 	$(CHART)/tests/render-test.sh
 
 # The molecule grafana-cloud scenario runs the role against a stub that exits 0

ansible/Makefile

@@ -128,7 +128,8 @@ decommission:
 # install-method and wallet side effects run in sponsord-install), sponsord-onramp
 # and sponsord-onramp-caddy (the onramp beside its daemon, behind the real distro
 # Caddy; the proxy side effects run in sponsord-onramp-caddy), sponsord-onramp-source
-# (that converge, then both sponsord roles built from source), source-build (the
+# (that converge, then both sponsord roles built from source), sponsord-onramp-lifecycle
+# (that converge, then sponsord's backup and decommission), source-build (the
 # node built from source on the host: sandbox, stamps, rollback), and cloud-init and
 # cloud-init-sponsord (the cloud-init/ templates, booted for real). The glob in
 # scripts/molecule.sh is the authority; this list is a guide.
@@ -164,7 +165,7 @@ decommission:
 SCENARIOS ?= all
 JOBS ?=
 SLOW_FIRST ?= cloud-init-sponsord cloud-init sponsord-onramp validation \
-	validation-sponsord sponsord-onramp-caddy sponsord-install sponsord \
+	validation-sponsord sponsord-onramp-caddy sponsord-onramp-lifecycle sponsord-install sponsord \
 	validation-alloy os-matrix grafana-cloud
 MOLECULE_LOCK_PREFIX ?= /tmp/decdn-devops-molecule
 molecule_env = SCENARIOS='$(SCENARIOS)' JOBS='$(JOBS)' SLOW_FIRST='$(SLOW_FIRST)' \

ansible/README.md

@@ -6,8 +6,8 @@ hardened host baseline. The roles also ship as the `decdn.node` Galaxy collectio
 | Playbook | Purpose | Make target |
 |----------|---------|-------------|
 | **`site.yml`** | Harden the host and deploy the public **deCDN node** (`decdn-node`). | `make check` / `make deploy` |
-| `backup.yml` | Encrypted backup of a node's identity or full state. | `make backup` |
-| `decommission.yml` | Stop a node and remove its service (keeps the identity; no on-chain steps). | `make decommission` |
+| `backup.yml` | Encrypted backup of a node's identity or full state, and of sponsord's credentials. | `make backup` |
+| `decommission.yml` | Stop a node, the onramp and sponsord and remove their services (keeps the keys; no on-chain steps). | `make decommission` |
 | `sponsord.yml` | Harden the host and deploy **sponsord**, the onboarding sponsor, on hosts in `sponsord_hosts`, with or without a node; then its public **sponsord-onramp** (behind Caddy) on hosts in `sponsord_onramp_hosts`. Also run by `site.yml`. | `make check-sponsord` / `make deploy-sponsord` |
 
 ```
@@ -301,8 +301,11 @@ make backup LIMIT=<host> ANSIBLE_ARGS='-e decdn_backup_scope=full'  # full state
 make decommission LIMIT=<host>                                     # typed confirmation; keeps the identity
 ```
 
-Backups need `decdn_backup_age_recipients` (public keys). Restore, host migration and
-the on-chain exit: [`docs/lifecycle.md`](../docs/lifecycle.md).
+Backups need `decdn_backup_age_recipients` (public keys); on `sponsord_hosts` they
+archive the treasury keystore and password, the API token and the Turnstile secret.
+On a sponsord host, decommission refuses while sponsord holds an unconfirmed pool
+top-up. Restore, host migration and the on-chain exit:
+[`docs/lifecycle.md`](../docs/lifecycle.md).
 
 ---
 
@@ -333,15 +336,17 @@ git-ignored inventory instead: role-authored env file + provenance record),
 itself is Debian 12), `lifecycle` (a `decdn_network` profile with an override, both
 backup scopes decrypted and checked, a rejected and a real decommission),
 `sponsord` (`playbooks/sponsord.yml` on a host with no node: generated token, credential
-rotation restart, the fatal `/healthz` gate, Alloy scraping sponsord and not the node;
+rotation restart, the fatal `/healthz` gate, the top-up hold guard, Alloy scraping sponsord and not the node;
 `grafana-cloud` covers sponsord co-located with a node), `sponsord-install` (the same
 converge, then release mode against a signed loopback mirror and treasury wallet
 generation), `sponsord-onramp` (the onramp beside its daemon, behind the real distro
 Caddy with `tls internal`: the https chain, the client address Caddy passes on, a
 Turnstile rotation restart, the fatal gates), `sponsord-onramp-caddy` (the same
 converge, then ACME and `none` modes and a refused foreign Caddyfile),
 `sponsord-onramp-source` (the same converge, then both sponsord roles built from a
-fixture repo with `install_method: source`), `source-build` (the node built from source
+fixture repo with `install_method: source`), `sponsord-onramp-lifecycle` (the same converge, then
+sponsord's backup decrypted and checked, decommission refused for a held top-up and a
+bad confirmation, then a real one), `source-build` (the node built from source
 on the host: the build sandbox, stamps, rollback, an untrusted earlier build home), and
 `cloud-init` and `cloud-init-sponsord` (the `cloud-init/` templates booted for real).
 A slow scenario is split along its side effects rather than allowed to set the wall

ansible/galaxy/CHANGELOG.md

@@ -184,6 +184,33 @@ collection adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
   and on Debian Ansible needs ACL support to hand the temp module file to that user.
   Without it both fail without an `rc`, which the role can only report after the fact
   ("becoming the unprivileged decdn user needs the `acl` package on this host").
+- `sponsord`: a top-up hold guard. When the keeper broadcasts a pool top-up and
+  cannot read its receipt, it holds further top-ups until that transaction mines
+  (upstream sponsord #41); a restart forgets the hold and the pool can be refilled
+  twice. Before the role restarts sponsord it reads `/metrics` at the address the
+  running daemon listens on, and fails while
+  `sponsord_pool_topup_unconfirmed_since_unix` is non-zero or `/metrics` does not
+  answer, before anything is queued, leaving the restart for the next run. A daemon
+  that has not bound yet holds nothing and passes. `sponsord_restart_ignore_topup_hold`
+  (passed as JSON) overrides it.
+- `sponsord`, `sponsord_onramp`: `sponsord_log_level` / `sponsord_onramp_log_level`,
+  rendered as `RUST_LOG` (`""` keeps the daemon's errors-only and the onramp's info
+  default). Each comma-separated part must be a level or `target=level`: tracing
+  reads any other bare word as a target name, and the onramp (built without
+  tracing's env-filter) logs nothing at all on a value it cannot parse.
+- `sponsord`: `tasks_from: backup` (`sponsord_backup_*`): the treasury keystore and
+  password, the API token and the onramp's Turnstile secret (`secret.env` opt-in),
+  taken hot and encrypted on the host to `sponsord_backup_age_recipients` (default
+  `decdn_backup_age_recipients`), then fetched. Each must be a regular file: tar
+  would store a symlink as the link, not the key.
+- `sponsord`, `sponsord_onramp`: `tasks_from: decommission`. The onramp's stops it,
+  removes its unit, and stops and disables the role's Caddy (marker-gated).
+  sponsord's refuses while a top-up is held, then stops sponsord and removes its
+  unit, checking for a held top-up again right before the stop. Both keep
+  `/etc/sponsord` and never touch the chain. They share
+  `decdn_decommission_max_hosts` / `_prompt_seconds` / `_confirm` with `decdn_node`;
+  the first play to reach a host asks once, and its prompt lists every service the
+  run stops there.
 
 ### Removed
 
@@ -198,6 +225,19 @@ collection adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
 
 ### Changed
 
+- `sponsord`: the restart-inputs comparison is now the only thing that restarts the
+  daemon. The tasks that write secret.env, sponsord.env, the unit or the binary no
+  longer notify the handler themselves (each is a hashed input already), so a write
+  that lands back on the recorded content restarts nothing, and no restart can skip
+  the hold guard. Under `--check`, no restart is reported.
+- `playbooks/backup.yml` and `playbooks/decommission.yml` also cover
+  `sponsord_hosts` (and `sponsord_onramp_hosts`): a backup of sponsord's
+  credentials; a decommission that first checks every sponsord host for a held
+  top-up, then removes the node, the onramp and sponsord, and tears down Alloy on
+  either kind of host. Its first play enforces `decdn_decommission_max_hosts`
+  against every host the run touches: each role checks only its own play, so one
+  node-only and one sponsord-only host would otherwise both pass.
+
 - **Breaking:** `decdn_node_install_method`, `sponsord_install_method` and
   `sponsord_onramp_install_method` now default to `release` (was `manual`). An
   inventory that relied on the old default must set `manual` explicitly. Upstream

ansible/galaxy/README.md

@@ -9,8 +9,8 @@ repository — five roles and nothing else:
 | `decdn.node.baseline` | Debian/Ubuntu host baseline — nftables default-deny inbound, fail2ban, unattended-upgrades, chrony, an admin sudo user, then DevSec OS + SSH hardening (applied last). |
 | `decdn.node.decdn_node` | The `decdn-node` daemon under a hardened systemd unit — from a GPG-verified release tarball (the default), a build on the host from a git ref, or locally built binaries; public QUIC udp/4433, loopback metrics + admin RPC. `decdn_network` sets the chain from upstream's manifest; `tasks_from: backup` / `decommission` for day 2. |
 | `decdn.node.grafana_alloy` | Opt-in Grafana Cloud observability agent — loopback-only Alloy receiver and hardened telemetry export; scrapes the node, `sponsord`, or both (`grafana_alloy_node_enabled` / `grafana_alloy_sponsord_enabled`), and labels the onramp's journal (`grafana_alloy_sponsord_onramp_enabled`). |
-| `decdn.node.sponsord` | The `sponsord` onboarding sponsor (treasury signer + PaymentPool keeper), standalone or beside a node — local binary or GPG-verified `sponsord-v*` release, `DynamicUser` unit with the API token and treasury wallet as systemd credentials, loopback-only API, `/healthz` deploy gate. `sponsord_network` sets the chain and PaymentPool from upstream's manifest. |
-| `decdn.node.sponsord_onramp` | `sponsord-onramp`, sponsord's public side (Turnstile gate, installers, CLI API), on the daemon's host — local binary or GPG-verified `sponsord-onramp-v*` release, `DynamicUser` unit with the daemon token and Turnstile secret as systemd credentials, loopback listener behind Caddy (distro package, ACME TLS, admin API off; or your own proxy), `/healthz` deploy gate. |
+| `decdn.node.sponsord` | The `sponsord` onboarding sponsor (treasury signer + PaymentPool keeper), standalone or beside a node — local binary or GPG-verified `sponsord-v*` release, `DynamicUser` unit with the API token and treasury wallet as systemd credentials, loopback-only API, `/healthz` deploy gate. `sponsord_network` sets the chain and PaymentPool from upstream's manifest. Restarts refuse while a pool top-up is held; `tasks_from: backup` / `decommission` for day 2. |
+| `decdn.node.sponsord_onramp` | `sponsord-onramp`, sponsord's public side (Turnstile gate, installers, CLI API), on the daemon's host — local binary or GPG-verified `sponsord-onramp-v*` release, `DynamicUser` unit with the daemon token and Turnstile secret as systemd credentials, loopback listener behind Caddy (distro package, ACME TLS, admin API off; or your own proxy), `/healthz` deploy gate; `tasks_from: decommission`. |
 
 ## Requirements
 

ansible/galaxy/galaxy.yml

@@ -16,8 +16,8 @@ authors:
   - deCDN Contributors
 description: >-
   Deploy and harden a public deCDN node: DevSec host baseline, the decdn_node
-  daemon (with backup and decommission), the sponsord onboarding sponsor and its
-  public onramp, and opt-in Grafana Cloud observability.
+  daemon and the sponsord onboarding sponsor with its public onramp (each with
+  backup and decommission), and opt-in Grafana Cloud observability.
 license:
   - MIT
 # Galaxy tags must match ^[a-z0-9]+$ (no hyphens/underscores).

ansible/molecule/sponsord-install/molecule.yml

@@ -66,6 +66,7 @@ provisioner:
         sponsord_rpc_url: "https://rpc.example.invalid/?key=$$HOME$$(touch /tmp/sponsord-rpc-expanded)&q=o'brien \"x\""
         sponsord_max_ttl_secs: 3600
         sponsord_pool_low_water_micro_usdc: 0
+        sponsord_log_level: "sponsord=debug,info"
         sponsord_readiness_retries: 10
         sponsord_readiness_delay: 1
         decdn_grafana_cloud_enabled: true