feat: operator toolkit for deCDN nodes worldwide - #69
Conversation
…ection .claude/settings.json enabled a third-party Claude Code plugin for everyone who clones this public repo; that is a per-developer preference and belongs in the git-ignored settings.local.json. CONTRIBUTING's Solidity section described a forge fmt hook and a CI solidity job that do not exist. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nt out The fleet overlay template carried the deCDN team's own launch topology (seed/edge and metered/unmetered groups, Hetzner/OVHcloud hosts, model-catalogue sizing) and ansible/docs/launch-runbook.md, which pointed at files in the private decdn/internal repo. Both now live in decdn/internal-devops. The public template keeps one decdn_nodes group, takes the chain from decdn_network, and explains how to add child groups of your own. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rofiles decdn_node_target now follows the host's gathered architecture, so aarch64 hosts get the aarch64 release and ELF check with no inventory change. An unsupported or mismatched triple fails loud instead of silently falling back to x86-64; grafana_alloy's arch mapping gets the assert its comment promised. decdn_network selects the role's mirror of upstream's deployment manifest (vars/main/networks.yml, generated by scripts/sync-network-profiles.py from contracts/deployments/<chainId>.json and KNOWN_CHAINS). The address defaults read it lazily, so inventory values still win and are reported. The example host_vars now use it, which also fixes its eight addresses: they were three upstream redeploys stale (PaymentPool 0x6414... vs 0x351E... at deployBlock 11744904). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ly what is tested The roles' meta claimed Ubuntu jammy/noble while every molecule scenario ran on Debian 12 alone. A new os-matrix scenario runs the default scenario's plays (converge, idempotence, verify) on Debian 13, Ubuntu 24.04 and Ubuntu 26.04; with default on Debian 12 that is exactly the set the meta now lists. jammy is dropped. baseline is documented as host-verified, since it cannot run in a container. The Debian 12 image is re-pinned to current :latest alongside. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
New role entry points, playbooks and make targets for the day-2 steps that had no tooling: - make backup: tars the node's identity (hot) or its full state minus the cache (stopping the node for a consistent copy, then restarting it) and encrypts it ON THE HOST to operator-supplied age/SSH public keys. Only ciphertext is fetched to the git-ignored ansible/backups/. - make decommission: needs LIMIT and a typed confirmation, is capped at one host by default, stops the node with systemctl, removes the unit and our Alloy agent, and keeps the identity, since the keystore withdraws the bond. It prints the on-chain exit steps (deregister, then unbond --all) and touches nothing on-chain itself. docs/lifecycle.md covers restore and host migration. The role README no longer presents 'decdn node drain' as the way down: under Restart=always systemd starts the drained daemon again. A new lifecycle molecule scenario exercises the network profile, both backup scopes, a rejected confirmation and the real decommission. CI syntax-checks every playbook. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/sync-monitoring.sh vendors decdn/decdn's monitoring/ (four Grafana dashboards, 39 reference alert rules) into charts/decdn-node/files/monitoring with a SOURCE record of the upstream commit and file hashes. The chart can now render them: metrics.prometheusRule (a PrometheusRule, rules passed through unmodified, optional extra labels) and metrics.grafanaDashboards (one sidecar-labelled ConfigMap per dashboard). The ServiceMonitor gains the target labels they select on (job, region, deployment_environment, instance), matching what grafana_alloy already stamps on the Ansible path. The render tests check rule count and label merge, one dashboard per vendored file, and that nothing renders when disabled; the node.toml extraction now filters on the key, since the chart can render more than one ConfigMap. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
compose/ runs the upstream daemon image with the Ansible role's host layout (/etc/decdn read-only, /var/lib/decdn), the systemd unit's start command, and its hardening: dedicated host uid/gid, read-only rootfs, no capabilities, no-new-privileges, SIGTERM with a 300 s grace. It uses host networking, so the daemon's loopback metrics and hard-wired loopback admin RPC stay loopback and Docker publishes no ports. The RPC URL stays in a root-only env file. make lint-compose (new CI job) renders it with the example env and asserts those invariants; make security now scans compose/ too, with the /etc/decdn mount exclusion justified in the Makefile. Smoke-tested with an image built from the upstream Dockerfile: the node reaches 'runtime ready' on the read-only rootfs, the host CLI reaches the admin RPC, and stop drains cleanly. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- A pre-commit job runs every hook on every file, so shellcheck, markdownlint and yamllint gate outside contributions too, not only machines with the hook. - Dependabot now bumps pre-commit hook revs; its header says honestly which ecosystems do not apply and what stays manual. - upstream-drift.yml (weekly, non-blocking) regenerates the network profiles, the vendored monitoring assets and the config schema-key inventory from decdn/decdn main and fails when a committed mirror is stale. - Issue forms ask for deploy path, install method, versions and OS/arch, remind reporters to redact RPC URLs and tokens, and route security reports and daemon bugs to the right place. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…switch release.yml runs on vX.Y.Z tags. build always runs: it checks that the tag, galaxy.yml, Chart.yaml and both changelogs agree (scripts/check-release-version.sh), re-runs galaxy-check and lint-helm, and uploads both packages with SHA256SUMS. publish runs only when the PUBLISH_ENABLED repository variable is 'true', behind the 'release' environment: chart to oci://ghcr.io/decdn/charts with a keyless cosign signature, collection to Galaxy, then the GitHub Release. Until the switch is flipped, a tag push is a dry run. RELEASING.md covers the steps, the first-release setup (Galaxy namespace, GHCR visibility, environment) and recovery when Galaxy's non-idempotent publish fails part-way. The collection README no longer tells people to install from Galaxy before anything is there; the chart gets a CHANGELOG. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- README becomes the operator landing page: a chooser for the three deploy paths, what every path provides, and the canonical security model. The per-path READMEs keep only what is specific to them and link back. - New docs/requirements.md: platforms and how each is tested, network and firewall (including cloud security groups, NAT, IPv6), a disk formula from the role's knobs, and a path comparison. It quotes no CPU/RAM minimum, because upstream publishes none. - CONTRIBUTING is the single home for make targets, CI and the upstream mirrors; AGENTS keeps the hard rules and a short command block, and both now cover compose/, the lifecycle playbooks, releases and the generated mirrors. - ansible/README: the manual install method is presented as the default it is, the chain comes from decdn_network, and the Grafana section defers to its role README. - Stale claims fixed: 'two roles' (three ship), hard-coded scenario counts, the release tarball as the default install, the site.yml diagram missing grafana_alloy, Galaxy publishing described as manual-only. - Collection CHANGELOG gains this PR's entries. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved release, Ansible variable loading, Compose secret exposure, chart validation, and monitoring synchronization issues remain.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 4
Open (7)
Tag filter incorrectly uses regex quantifier syntax · New Architecture mapping vars file is not auto-loaded · New Network profiles vars file is not loaded · New Avoid mounting host RPC secret into the container · New Recursively list monitoring files for synchronization · New Check vendored file contents for drift · New Document Compose-specific secret file ownership · New
What changed in this PR
Expands the repository into an operator toolkit spanning Ansible, Docker Compose, Helm, lifecycle operations, monitoring, and releases.
Changes:
- Adds Compose deployment and backup/decommission workflows.
- Adds generated network profiles, monitoring assets, and expanded CI coverage.
- Adds Galaxy/OCI release automation and operator documentation.
| File | Reviewed change |
|---|---|
SECURITY.md |
Documents security guidance and chart verification. |
scripts/sync-network-profiles.py |
Generates network profiles. |
scripts/sync-monitoring.sh |
Vendors monitoring assets. |
scripts/check-release-version.sh |
Validates release versions. |
RELEASING.md |
Documents artifact publishing. |
README.md |
Updates deployment paths and security guidance. |
Makefile |
Adds Compose linting and security scans. |
docs/requirements.md |
Documents platform requirements. |
docs/lifecycle.md |
Documents lifecycle operations. |
CONTRIBUTING.md |
Updates contributor and CI guidance. |
compose/README.md |
Documents Compose operations. |
compose/decdn.env.example |
Adds Compose secret template. |
compose/compose.yaml |
Adds the Compose service definition. |
compose/.env.example |
Adds Compose configuration template. |
charts/decdn-node/values.yaml |
Adds monitoring configuration. |
charts/decdn-node/values.schema.json |
Schemas monitoring values. |
charts/decdn-node/tests/render-test.sh |
Tests chart rendering. |
charts/decdn-node/templates/servicemonitor.yaml |
Adds monitoring labels. |
charts/decdn-node/templates/prometheusrule.yaml |
Renders alert rules. |
charts/decdn-node/templates/dashboards-configmap.yaml |
Renders dashboard ConfigMaps. |
charts/decdn-node/README.md |
Documents chart monitoring and OCI usage. |
charts/decdn-node/files/monitoring/SOURCE |
Records monitoring provenance. |
charts/decdn-node/files/monitoring/README.md |
Documents vendored monitoring assets. |
charts/decdn-node/ci/ci-values.yaml |
Enables monitoring in chart tests. |
charts/decdn-node/CHANGELOG.md |
Records chart changes. |
ansible/roles/grafana_alloy/tasks/install.yml |
Validates Alloy architectures. |
ansible/roles/grafana_alloy/meta/main.yml |
Updates platform support. |
ansible/roles/decdn_node/vars/main/networks.yml |
Adds generated network data. |
ansible/roles/decdn_node/vars/main/arch.yml |
Maps architectures to targets. |
ansible/roles/decdn_node/tasks/main.yml |
Adds architecture and network validation. |
ansible/roles/decdn_node/tasks/decommission.yml |
Adds guarded decommissioning. |
ansible/roles/decdn_node/tasks/backup.yml |
Adds encrypted backups. |
ansible/roles/decdn_node/README.md |
Documents profiles and lifecycle operations. |
ansible/roles/decdn_node/meta/main.yml |
Updates role platform metadata. |
ansible/roles/decdn_node/defaults/main.yml |
Adds profile and lifecycle defaults. |
ansible/roles/baseline/README.md |
Updates platform documentation. |
ansible/roles/baseline/meta/main.yml |
Updates platform metadata. |
ansible/README.md |
Documents Ansible workflows. |
ansible/playbooks/site.yml |
Documents Alloy ordering. |
ansible/playbooks/decommission.yml |
Adds decommission entry point. |
ansible/playbooks/backup.yml |
Adds backup entry point. |
ansible/molecule/validation/molecule.yml |
Updates validation scenario. |
ansible/molecule/validation/converge.yml |
Tests architecture and profile validation. |
ansible/molecule/slow-readiness/molecule.yml |
Updates scenario configuration. |
ansible/molecule/schema/molecule.yml |
Updates schema scenario metadata. |
ansible/molecule/os-matrix/molecule.yml |
Adds OS matrix coverage. |
ansible/molecule/lifecycle/verify.yml |
Verifies lifecycle behavior. |
ansible/molecule/lifecycle/prepare.yml |
Prepares lifecycle tests. |
ansible/molecule/lifecycle/molecule.yml |
Defines lifecycle scenario. |
ansible/molecule/lifecycle/converge.yml |
Exercises lifecycle workflows. |
ansible/molecule/host-env/molecule.yml |
Updates scenario configuration. |
ansible/molecule/grafana-cloud/molecule.yml |
Updates scenario configuration. |
ansible/molecule/grafana-cloud-token/molecule.yml |
Updates scenario configuration. |
ansible/molecule/generate-keystore/molecule.yml |
Updates scenario configuration. |
ansible/molecule/default/molecule.yml |
Updates default scenario metadata. |
ansible/molecule/default/converge.yml |
Reuses project test fixtures. |
ansible/Makefile |
Adds lifecycle targets. |
ansible/inventory/host_vars/decdn-node-1/main.yml |
Uses generated network profiles. |
ansible/inventory/fleet.example/hosts.yml |
Simplifies the fleet template. |
ansible/inventory/fleet.example/host_vars/node-a/secret.yml.example |
Adds secret template guidance. |
ansible/inventory/fleet.example/group_vars/decdn_unmetered.yml |
Removes team-specific settings. |
ansible/inventory/fleet.example/group_vars/decdn_seed.yml |
Removes team-specific settings. |
ansible/inventory/fleet.example/group_vars/decdn_nodes.yml |
Adds generic fleet defaults. |
ansible/inventory/fleet.example/group_vars/decdn_metered.yml |
Removes team-specific settings. |
ansible/inventory/fleet.example/group_vars/decdn_edge.yml |
Removes team-specific settings. |
ansible/galaxy/README.md |
Updates collection documentation. |
ansible/galaxy/galaxy.yml |
Updates collection metadata. |
ansible/galaxy/CHANGELOG.md |
Records collection changes. |
ansible/docs/launch-runbook.md |
Removes the team-specific runbook. |
ansible/.gitignore |
Ignores fetched backups. |
AGENTS.md |
Updates repository guidance. |
.pre-commit-config.yaml |
Excludes generated monitoring assets. |
.gitignore |
Ignores backups and release artifacts. |
.github/workflows/upstream-drift.yml |
Adds scheduled mirror checks. |
.github/workflows/release.yml |
Adds collection and chart publishing. |
.github/workflows/molecule.yml |
Updates Molecule workflow guidance. |
.github/workflows/ci.yml |
Adds Compose and pre-commit CI jobs. |
.github/ISSUE_TEMPLATE/feature_request.yml |
Adds feature request form. |
.github/ISSUE_TEMPLATE/config.yml |
Configures issue routing. |
.github/ISSUE_TEMPLATE/bug_report.yml |
Adds bug report form. |
.github/dependabot.yml |
Adds pre-commit updates. |
.claude/settings.json |
Removes forced plugin configuration. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…ecret ownership - sync-monitoring.sh --check compared only SOURCE, so a hand-edited vendored dashboard or rule file with an untouched SOURCE passed. It now byte-compares every file with upstream as well. - The README's security model said every secret file is owned by the service account; Compose's decdn.env is root:root 0600 because Docker reads it before the container starts. The sentence now names the owner per path. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ound - A full backup now always carries the keystore, even when decdn_keystore_file lives outside decdn_home; before, a migration backup could lack the wallet that withdraws the bond. - New path guards (validate-lifecycle-paths.yml) run first in both entry points: a decdn_cache_dir equal to or enclosing the identity (which emptied every backup and let the cache purge delete the keys) and a decdn_backup_dir in or around the node's directories are refused. - Full backups stop the node whenever the unit exists and decide the restart from ActiveState, so a crash-looping node is not restarted mid-copy. A failed archive step restarts the node and removes its .partial; leave_stopped only applies to a backup that succeeded. - Full archives are no longer fetched by default: fetch under become reads the whole file into memory on both ends. The report prints a streaming copy. - Decommission's prompt times out (decdn_decommission_prompt_seconds, 300), failing closed, and its report no longer implies the CLI reads decdn.env. Tests: the lifecycle scenario now proves the stop/restart (InvocationID), the failed-backup recovery, leave_stopped, full-archive contents (daemon state in, cache out), the decommission host cap, an unanswered confirmation, and the cache purge. The validation scenario adds nine negatives: bad backup recipients, the path guards on both entry points, an unset target triple, and a mistyped network with no addresses. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Dashboard ConfigMaps truncated their names to 63 characters, so with a fullname of ~52+ characters three of them collided. ConfigMap names may be 253 characters; the truncation is gone, and a render test checks every object name is unique under a 63-character fullnameOverride. - grafanaDashboards.enabled with no vendored dashboards now fails the render instead of rendering nothing; the render test covers it. - The ruleLabels precedence (a rule's own labels win) is now actually tested: ci-values sets ruleLabels.severity, which no rule may take. - The chart CHANGELOG collects changes under [Unreleased], like the collection's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… guard tests - release.yml: log cosign into ghcr.io too (it does not read Helm's registry credentials, so the first publish would have pushed the chart and failed to sign it), and print helm push output when it fails. - Upstream mirrors: sync-network-profiles.py now parses the manifest-key mapping from upstream's KnownChain::addresses() instead of a hand-written table, refuses KNOWN_CHAINS entries it cannot read and any change to the address fields, and keeps git's stderr. Both generators exit 1 for stale and 2 for could-not-run; sync-monitoring also refuses a layout with no dashboards or no rule file. The drift job reports stale and broken separately. - compose.yaml builds the image as DECDN_IMAGE_REPO@DECDN_IMAGE_DIGEST, so no .env can pin a mutable tag. lint-compose tells render failures from invariant violations. - tests/scripts-test.sh (make test-scripts, CI job scripts): the ansible Makefile's scoping guards, the release gate, lint-compose's negatives, and the generators' exit codes with UPSTREAM set. - check-release-version.sh rejects arguments it does not recognise. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ings - The on-chain commands (setup, bond, register, update-multiaddrs, update-region, deregister, unbond) do not read DECDN_RPC_URL; only the daemon does. docs/lifecycle.md gains a decdn_chain helper that passes it as --rpc-url from decdn.env, and the role and Compose READMEs point at it. - Coverage claims: decdn_node runs on all four distros, grafana_alloy's install path on Debian 12 only. - Helm takes a digest or a tag; only Compose enforces a digest. - Makefile comments, the Helm pin count, RELEASING's changelog procedure for the first release, CONTRIBUTING's generator notes, the --start-at-task example (a task name that did not exist), and stale inventory comments. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Pushed fixes for the internal multi-agent review (code, tests, comment accuracy, silent failures), in four commits: Safety (
Chart (
CI/release (
Docs (
New tests
Local run: all 10 molecule scenarios, lint, lint-ansible, lint-helm, lint-alloy, lint-compose, security, test-scripts, galaxy-check and actionlint pass. 🤖 Generated with Claude Code |



Summary
This PR makes the repo usable by node operators outside the team, in one PR of 11 reviewable commits:
release.ymlpublishes thedecdn.nodecollection (Galaxy) and the chart (oci://ghcr.io/decdn/charts, keyless cosign) from avX.Y.Ztag. It only publishes when thePUBLISH_ENABLEDvariable istrue; until then every tag is a dry run. The steps are inRELEASING.md.os-matrix) as well as Debian 12, and the meta claims only those four. jammy is dropped because it was never tested.compose/.decdn_network: arbitrum-sepoliatakes the contract addresses from a generated mirror of upstream's manifest (scripts/sync-network-profiles.py).0x6414…vs0x351E…at deployBlock 11744904.make backupencrypts on the host to age/SSH public keys.identityscope runs hot;fullscope stops the node for a consistent copy.make decommissionneedsLIMITand a typed confirmation. It keeps the identity and prints the on-chain exit steps.docs/lifecycle.mdcovers restore and host migration.decdn node drainunderRestart=alwaysgets restarted.scripts/sync-monitoring.sh) and render optionally asPrometheusRule/ sidecar ConfigMaps. The ServiceMonitor gains the labels they select on.pre-commitjob, so the hooks are no longer local-only.composejob; KICS now also coverscompose/.upstream-driftjob for the three generated mirrors..claude/settings.json(it forced a third-party plugin on every cloner).docs/requirements.md.Things to know
decdnnamespace plusGALAXY_API_KEY, thereleaseenvironment, and making the GHCR package public after the first push. SeeRELEASING.md.node runtime readyon the read-only rootfs;/etc/decdnmount. The justification is in theMakefile. Two MEDIUMs remain by design: host network and no healthcheck.JOBS=3, with 8 → 10 scenarios. The 45-minute timeout is unchanged.networkshortcut yet; its README points todecdn config init --chain.Test plan
make lint(every pre-commit hook),make lint-ansible(production profile, 0 findings)make molecule JOBS=3: all 10 scenarios pass, including the newos-matrix,lifecycle(profile + override, both backup scopes decrypted and checked, rejected and real decommission) and the newvalidationcases (arch mismatch, unknown network, chain-id mismatch)make lint-helm: rendered rules match the vendored file rule-for-rule, plus the new negativesmake lint-alloy,make lint-compose(including a negative check),make security(no HIGH)make galaxy-check: importer clean; the new task and vars files ship in the collectionactionlintclean; release gate tested for both success (notes written) and mismatchsync-network-profiles.py --checkandsync-monitoring.sh --checkare current against upstream mainworkflow_dispatchdry run ofrelease.yml; firstupstream-driftrun🤖 Generated with Claude Code