Repository navigation
feat: close the sponsord operational gaps (top-up hold guard, monitoring, lifecycle, Compose gate page) - #102
Merged
Merged
Conversation
…sord_log_level A restart forgets sponsord's hold on an unconfirmed top-up (upstream #41), and the pool can then be refilled twice. The role now reads /metrics before it flushes a pending restart and fails while the hold is on, leaving the restart for the next run; sponsord_restart_ignore_topup_hold overrides it. Also renders sponsord_log_level as RUST_LOG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
backup.yml archives sponsord's credentials (treasury keystore and password, API token, Turnstile secret), encrypted on the host. decommission.yml checks every sponsord host for a held top-up, then removes the node, the onramp (and the role's Caddy) and sponsord, with one typed confirmation per host, and tears down Alloy on either kind of host. Covered by molecule/sponsord-onramp-lifecycle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
files/monitoring/sponsord/ holds a dashboard (uid decdn-sponsord) and rules for keeper failures, a stale pool read, a held top-up, an empty pool and request errors, with promtool unit tests in lint-helm. The chart does not render them: sponsord has no Kubernetes path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arts The onramp mounts /etc/sponsord/onramp-gate/ read-only for ONRAMP_GATE_TEMPLATE (a fixed directory, so no setting can mount a credential into the page server). The README adds the manual top-up hold check before a restart, a backup command, and notes the onramp ignores RUST_LOG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- sponsord: the restart-inputs comparison is the only restart trigger, and the top-up hold guard runs before it queues anything, so a write landing back on the recorded content can no longer restart past the guard (nor can --force-handlers). The guard probes the address the running process listens on, passes a daemon that has not bound yet, requires sponsord's keeper series, accepts a labelled gauge, retries briefly, and words its refusal for a restart or a stop. The documented override is the JSON form the boolean check accepts. - decommission: check the hold again right before sponsord's stop (the prompt can wait minutes); each prompt lists every service the run stops on its hosts. - backup: the Turnstile secret at the onramp role's path, required on onramp hosts. - sponsord_onramp_log_level: the onramp does read RUST_LOG (as Targets: a level or target=level pairs); correct the docs that said it ignored it. - alerts: SponsordRequestFailures fires on a code's first appearance (upstream exports it from its first error, already non-zero); a pool never read since start is SponsordPoolNeverChecked (3h), not "stale" after 15m. Tests model the exporter as it is, with internal-code and resolve cases. - tests: a revert during a hold, an unreadable /metrics, backup and log-level negatives, the decommission play order, a second decommission run, no confirmation left behind by a refusal. - docs: co-located caveats in lifecycle.md, the gate page's path rule, safer manual check and backup in the Compose README, stale scenario and galaxy text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Secret exposure, host-cap enforcement, backup integrity, and validation issues remain unresolved.
5 open findings
What changed in this PR
Adds sponsord operational safeguards, lifecycle automation, monitoring assets, log controls, and Compose gate-page support.
Changes:
- Guards sponsord restarts/decommissioning during unconfirmed top-ups.
- Adds encrypted backups, multi-service decommissioning, and log-level configuration.
- Adds sponsord dashboards, alerts, tests, and Compose documentation.
| File | Description |
|---|---|
tests/scripts-test.sh |
Extends lifecycle and Compose checks. |
Makefile |
Passes the Prometheus image to chart tests. |
docs/lifecycle.md |
Documents sponsord lifecycle procedures. |
compose/tests/invariants.jq |
Allows the gate-page mount. |
compose/sponsord-onramp.env.example |
Documents gate templates and logging. |
compose/README.md |
Adds hold, backup, and gate-page guidance. |
compose/compose.yaml |
Mounts the custom gate-page directory. |
charts/decdn-node/tests/sponsord-alerts_test.yml |
Tests sponsord alert behavior. |
charts/decdn-node/tests/render-test.sh |
Validates sponsord monitoring assets. |
charts/decdn-node/files/monitoring/sponsord/prometheus-alerts.yml |
Adds sponsord alert rules. |
charts/decdn-node/files/monitoring/sponsord/dashboard-sponsord.json |
Adds the sponsord dashboard. |
charts/decdn-node/files/monitoring/README.md |
Documents sponsord monitoring. |
charts/decdn-node/CHANGELOG.md |
Records monitoring additions. |
ansible/roles/sponsord/templates/sponsord.env.j2 |
Renders RUST_LOG. |
ansible/roles/sponsord/tasks/topup-hold.yml |
Implements the top-up hold guard. |
ansible/roles/sponsord/tasks/source.yml |
Removes direct restart notification. |
ansible/roles/sponsord/tasks/main.yml |
Centralizes guarded restart decisions. |
ansible/roles/sponsord/tasks/install.yml |
Removes installer restart notifications. |
ansible/roles/sponsord/tasks/decommission.yml |
Adds sponsord decommissioning. |
ansible/roles/sponsord/tasks/backup.yml |
Adds encrypted credential backups. |
ansible/roles/sponsord/README.md |
Documents new sponsord operations. |
ansible/roles/sponsord/defaults/main.yml |
Adds lifecycle, logging, and guard defaults. |
ansible/roles/sponsord_onramp/templates/sponsord-onramp.env.j2 |
Renders onramp RUST_LOG. |
ansible/roles/sponsord_onramp/tasks/main.yml |
Validates onramp logging configuration. |
ansible/roles/sponsord_onramp/tasks/decommission.yml |
Adds onramp/Caddy decommissioning. |
ansible/roles/sponsord_onramp/README.md |
Documents logging and removal. |
ansible/roles/sponsord_onramp/defaults/main.yml |
Adds logging and lifecycle defaults. |
ansible/roles/decdn_node/tasks/decommission.yml |
Shares confirmation across services. |
ansible/roles/decdn_node/README.md |
Notes co-located sponsord removal. |
ansible/README.md |
Updates lifecycle and scenario documentation. |
ansible/playbooks/decommission.yml |
Decommissions all deployed services. |
ansible/playbooks/backup.yml |
Backs up sponsord credentials. |
ansible/molecule/validation-sponsord/converge.yml |
Adds negative validation cases. |
ansible/molecule/sponsord/verify.yml |
Verifies sponsord logging output. |
ansible/molecule/sponsord/side_effect.yml |
Exercises the hold guard. |
ansible/molecule/sponsord/molecule.yml |
Configures log-level coverage. |
ansible/molecule/sponsord/files/sponsord-stub |
Simulates hold metrics. |
ansible/molecule/sponsord-onramp/side_effect.yml |
References lifecycle coverage. |
ansible/molecule/sponsord-onramp/molecule.yml |
Documents shared scenarios. |
ansible/molecule/sponsord-onramp-lifecycle/verify.yml |
Verifies backup and decommission results. |
ansible/molecule/sponsord-onramp-lifecycle/side_effect.yml |
Exercises lifecycle operations. |
ansible/molecule/sponsord-onramp-lifecycle/molecule.yml |
Defines the lifecycle scenario. |
ansible/molecule/sponsord-install/molecule.yml |
Mirrors logging test configuration. |
ansible/Makefile |
Registers the lifecycle scenario. |
ansible/galaxy/README.md |
Documents collection lifecycle features. |
ansible/galaxy/galaxy.yml |
Updates collection description. |
ansible/galaxy/CHANGELOG.md |
Records sponsord operational changes. |
AGENTS.md |
Adds sponsord operational invariants. |
.claude/skills/grafana-dashboards/SKILL.md |
Adds sponsord monitoring guidance. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
… logging - decommission.yml: the first play enforces decdn_decommission_max_hosts against every host the run touches (each role checks only its own play, so a node-only and a sponsord-only host both passed), then checks for a held top-up. - compose: the onramp starts through a pinned /bin/sh check that refuses an ONRAMP_GATE_TEMPLATE resolving outside /etc/sponsord/onramp-gate/; upstream would serve any file, and /run/secrets holds the daemon token and the Turnstile secret. invariants.jq pins the script byte for byte (no flags). - backup: every archived credential must be a regular file; tar stores a symlink as the link, not the key. - log levels: each comma-separated part must be a level or target=level; a misspelled level or a bare non-level word silences the onramp (and reads as a target for sponsord). - render-test.sh: the local promtool test step honours PROMTOOL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An audit against upstream
decdn/sponsord@ce104e7found every config setting already wired, but several operational gaps. This closes all of them except a Kubernetes path for sponsord, which is out of scope.Changes
Top-up hold guard (
ansible/roles/sponsord). Upstream #41 makes the keeper hold further pool top-ups while one is unconfirmed (sponsord_pool_topup_unconfirmed_since_unix> 0). A restart forgets the hold, and the pool can then be refilled twice.tasks/topup-hold.ymlreads/metricsat the address the running process listens on./metricsdoes not answer.ANSIBLE_ARGS='-e {"sponsord_restart_ignore_topup_hold":true}'.Log levels.
sponsord_log_levelandsponsord_onramp_log_levelare rendered asRUST_LOG. The onramp is built without tracing's env-filter, so it accepts only a level ortarget=levelpairs, and the role refuses anything else.Backup and decommission.
make backuparchives sponsord's credentials, encrypted on the host: the treasury keystore and password, the API token and the Turnstile secret.make decommissionchecks every sponsord host for a held top-up first, then removes the node, the onramp (stopping and disabling the role's Caddy) and sponsord, and tears down Alloy./etc/sponsordis kept, and nothing on-chain is touched.Monitoring.
charts/decdn-node/files/monitoring/sponsord/adds a dashboard and 8 alert rules:The rules have promtool unit tests, run from
lint-helmagainst a copy with the annotations stripped. The chart does not render this subdirectory. The treasury wallet's own balance is not covered, because sponsord does not export it.Compose. The onramp mounts
/etc/sponsord/onramp-gate/read-only forONRAMP_GATE_TEMPLATE. The README adds the manual hold check before a restart, and a manual backup command.Known gap
A restart this repo does not cause, such as a crash or a package upgrade restarting services, still forgets a hold. It is documented, and
SponsordTopupHeldis how an operator learns of a hold. A "hold lost to a restart" alert and a needrestart exclusion could follow.Verification
make lint,make lint-ansible,make lint-helm,make lint-compose,make test-scripts.make security: no new high findings. The one new medium is the loopback HTTP/metricsprobe, the same accepted finding as the existing/healthzprobe.make molecule SCENARIOS='sponsord sponsord-install validation-sponsord sponsord-onramp sponsord-onramp-lifecycle lifecycle'all pass. They cover:/metrics;sponsord_*name in the rules and the dashboard checked against upstreamcrates/sponsord/src/metrics.rs.🤖 Generated with Claude Code