Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions .agents/review/decisions.md

This file was deleted.

2 changes: 2 additions & 0 deletions .agents/skills/ci-pipelines/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: Use when changing continuous integration, GitHub Actions workflows,

# CI pipelines

Canonical workflows use maintained major-version tags for external actions.

## Billing shapes the job graph

- Jobs bill per minute, rounded up, minimum one. Fold sub-minute checks into an existing job on the same trust level instead of giving them their own.
Expand Down
2 changes: 2 additions & 0 deletions .agents/skills/declarative-infra/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,5 @@ Before pushing, run `nix flake check`, build the host toplevel, run `tofu fmt -c
- For SOPS, development environments, CI secrets, or provider credentials, read [Secrets bootstrap](references/secrets.md).
- For preview environments, read [Pull request previews](references/pr-previews.md). They are the default for web-host adoption; omitting them requires a recorded decision in the host repository.
- For image promotion into a dedicated infrastructure repository, read [Image promotion](references/image-promotion.md).

- For third-party container updates whose publisher is outside the infrastructure owner's control, read [Vendor image promotion](references/vendor-image-promotion.md).
25 changes: 24 additions & 1 deletion .agents/skills/declarative-infra/references/bootstrap.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,26 @@ in {
email = acmeEmail; # required: ACME registration contact
};

systemd.services.podman-image-prune = {
description = "Remove unused container images older than seven days";
serviceConfig = {
Type = "oneshot";
Nice = 19;
IOSchedulingClass = "idle";
};
script = ''
${pkgs.podman}/bin/podman image prune --all --force --filter until=168h
'';
};
systemd.timers.podman-image-prune = {
wantedBy = [ "timers.target" ];
timerConfig = {
OnCalendar = "daily";
Persistent = true;
RandomizedDelaySec = "30m";
};
};

virtualisation.podman = {
enable = true;
defaultNetwork.settings.dns_enabled = true; # containers resolve each other by name
Expand All @@ -162,6 +182,9 @@ Beyond the module:

- Only Caddy publishes services; nothing else opens 80/443, and every additional firewall port is a documented decision in the host repo.
- Pin container images by digest.
- Every Podman host needs image retention for production as well as previews. Daily image-only pruning of unused images created more than seven days ago is the default. Podman protects images referenced by any container, including stopped containers. Do not use `podman system prune`, volume pruning, or external-container removal as a substitute. Nix garbage collection and journal retention do not reclaim OCI images.
- The age filter uses the image creation timestamp, not its pull or last-use time. Retired images may need to be pulled again for rollback; keep registry access available. Where a deployment or preview controller protects staged or rollback images beyond container references, coordinate cleanup with that controller or exclude its labels and retain its own collector. A preview-only collector does not replace production retention.
- Verify the timer and a successful cleanup on each host after deployment. During a disk-full incident, check PostgreSQL recovery and every service sharing the filesystem after reclaiming unused images; never remove database files or volumes to make room.
- A host running PostgreSQL also runs a local dump timer with retention (hourly `pg_dump --format=custom` into a `postgres`-owned directory is the norm); shape the unit however reads best.
- A host running GitHub Actions jobs uses `services.github-runners.<name>` with a SOPS-provided token, `programs.nix-ld.enable = true` plus `NIX_LD`/`NIX_LD_LIBRARY_PATH` in the runner environment so downloaded tooling executes, and systemd resource caps (`CPUQuota`, `MemoryHigh`/`MemoryMax`) so jobs cannot starve the host's services.

Expand Down Expand Up @@ -267,7 +290,7 @@ A repo whose trusted CI builds and deploys NixOS system closures uses a private,

A deploy that changes an app's image restarts `podman-<app>.service` in place: the old container stops before the new one starts, and Caddy has no upstream for that app until the new container is up. That brief interruption is the accepted default for this profile. The health readback in `image-promotion.md` verifies that the deploy completed; it does not keep the old container serving through the switch.

Keep the window to container startup, not registry pull: after the deploy job's identity and registry-access checks and immediately before `nix run .#deploy-rs`, the workflow pre-pulls every gated image reference on the target over the deploy SSH connection. Set `REGISTRY_AUTH_FILE` and pass `--authfile` on every Podman registry command. Public entries use the root-owned empty `/run/containers/auth/anonymous.json`; private entries use only `/run/containers/auth/ghcr-private.json`, atomically created by the host's SOPS-backed `podman-ghcr-login` unit. Give every container unit the matching `REGISTRY_AUTH_FILE` too, so implicit pulls cannot fall back to ambient Podman or Docker auth. A private pre-pull always contacts the registry even when the digest is cached, so missing or expired credentials fail before activation; a public cached digest may skip its network pull after the drift detector has independently proved anonymous access. Pulling by digest is idempotent and additive, and any failed pre-pull fails the deploy before activation touches the running system.
Keep the window to container startup, not registry pull: after the deploy job's identity and registry-access checks and immediately before `nix run .#deploy-rs`, the workflow pre-pulls every gated image reference on the target over the deploy SSH connection. Set `REGISTRY_AUTH_FILE` and pass `--authfile` on every Podman registry command. Public entries use the root-owned empty `/run/containers/auth/anonymous.json`; private entries use only `/run/containers/auth/ghcr-private.json`, atomically created by the host's SOPS-backed `podman-ghcr-login` unit. Give every container unit the matching `REGISTRY_AUTH_FILE` too, so implicit pulls cannot fall back to ambient Podman or Docker auth. A private pre-pull always contacts the registry even when the digest is cached, so missing or expired credentials fail before activation; a public cached digest may skip its network pull after the same deployment has independently proved anonymous access to that exact digest. Pulling by digest is idempotent and additive, and any failed pre-pull fails the deploy before activation touches the running system.

Private-image migration and container units require and order after `podman-ghcr-login.service`. Public-image units must not depend on that service. The SOPS secret restarts the oneshot when its decrypted value changes. The unit reads `registry.github_token` through stdin, writes a same-directory `0600` temporary auth file, and atomically renames it to the private path only after `podman login` succeeds. The complete credential, two-stage adoption, auth-file, and rotation contract is in `image-promotion.md#private-ghcr-host-access`.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ lifecycle: [announced, branch, open, merged, deploy-failed, completed, supersede
<!-- contract:metadata-transition -->
```yaml
imagesPath: infra/images.json
metadataFields: [sourceRepository, sourceRef, sourceWorkflow, imageRepository, registryAccess, trackedTag, promotionLatencyMinutes]
metadataFields: [sourceRepository, sourceRef, sourceWorkflow, imageRepository, registryAccess]
pinFields: [promotionEnabled, digest, promotedSourceSha]
disabledPin: { promotionEnabled: false, digest: null, promotedSourceSha: null }
operations:
Expand All @@ -80,14 +80,14 @@ operations:
<!-- contract:registry-access -->
```yaml
public:
detectorCredential: none
detectorProof: anonymously-readable
workflowCredential: none
workflowProof: anonymously-readable
hostCredential: none
hostAuthFile: /run/containers/auth/anonymous.json
private:
detectorCredential: github-actions-token
detectorPermissions: { contents: read, packages: read }
detectorProof: exact-private-visibility-then-anonymous-denied-then-authenticated-readable
workflowCredential: github-actions-token
workflowPermissions: { contents: read, packages: read }
workflowProof: exact-private-visibility-then-anonymous-denied-then-authenticated-readable
hostCredential: sops-classic-pat
hostCredentialScopes: [read:packages]
hostCredentialAuthority: all-packages-readable-by-token-owner
Expand All @@ -99,25 +99,6 @@ private:
forbiddenDesiredStateFields: [credential, secretPath, username, authFile]
```

<!-- contract:registry-resolution -->
```sh
set -euo pipefail
case "$REGISTRY_ACCESS" in
public)
resolve-anonymous-tag
;;
private)
require-exact-private-visibility
reject-anonymous-readable
resolve-authenticated-tag
;;
*)
printf 'unsupported registry access mode: %s\n' "$REGISTRY_ACCESS" >&2
exit 1
;;
esac
```

<!-- contract:registry-access-proof -->
```sh
set -euo pipefail
Expand Down Expand Up @@ -231,8 +212,13 @@ digest_hex=${DIGEST#sha256:}
branch="image-bump/${APP}/${SOURCE_SHA:0:12}-${digest_hex:0:12}"
prs=$(gh pr list --repo example/infra --state all --search "\"$marker\" in:body" --json number,body,state)
pr=$(jq -er --arg marker "$marker" '[.[] | select(.state == "MERGED" and (.body | contains($marker)))] | if length == 1 then .[0].number else error("expected one merged promotion PR") end' <<<"$prs")
view=$(gh pr view "$pr" --repo example/infra --json state,mergeCommit,author,headRefName,headRepository,files,statusCheckRollup)
merge_sha=$(jq -er --arg branch "$branch" 'if .state == "MERGED" and .author.login == "promotion-bot[bot]" and .headRefName == $branch and .headRepository.nameWithOwner == "example/infra" and [.files[].path] == ["infra/images.json"] and ([.statusCheckRollup[] | select(.name == "trusted-promotion-provenance" and .conclusion == "SUCCESS")] | length) == 1 then .mergeCommit.oid else error("merged promotion PR is not trusted") end' <<<"$view")
view=$(gh pr view "$pr" --repo example/infra --json state,mergeCommit,author,headRefName,headRepository,headRefOid,files)
merge_sha=$(jq -er --arg branch "$branch" 'if .state == "MERGED" and .author.login == "promotion-bot[bot]" and .headRefName == $branch and .headRepository.nameWithOwner == "example/infra" and [.files[].path] == ["infra/images.json"] then .mergeCommit.oid else error("merged promotion PR is not trusted") end' <<<"$view")
head=$(jq -er '.headRefOid | select(test("^[0-9a-f]{40}$"))' <<<"$view")
checks=$(gh api "repos/example/infra/commits/$head/check-runs?check_name=trusted-promotion-provenance&filter=latest&per_page=100" --paginate --slurp)
jq -e --arg head "$head" '[.[].check_runs[]] | length == 1 and all(.[];
.name == "trusted-promotion-provenance" and .head_sha == $head and
.app.slug == "github-actions" and .status == "completed" and .conclusion == "success")' <<<"$checks" >/dev/null
encoded=$(gh api "repos/example/infra/contents/infra/images.json?ref=$merge_sha")
images=$(jq -er '.content' <<<"$encoded" | base64 --decode)
jq -er --arg app "$APP" --arg digest "$DIGEST" --arg sha "$SOURCE_SHA" 'select(.[$app].promotionEnabled == true and .[$app].digest == $digest and .[$app].promotedSourceSha == $sha) | true' <<<"$images" >/dev/null
Expand All @@ -242,19 +228,3 @@ gh run watch "$run_id" --repo example/infra --exit-status
result=$(gh run view "$run_id" --repo example/infra --json headSha,conclusion,jobs)
jq -er --arg sha "$merge_sha" 'if .headSha == $sha and .conclusion == "success" and ([.jobs[] | select(.name == "deploy" and .conclusion == "success")] | length) == 1 and ([.jobs[] | select(.name == "deploy")] | length) == 1 then true else error("exact deploy did not complete successfully") end' <<<"$result" >/dev/null
```

<!-- contract:drift-detector -->
```sh
set -euo pipefail
window=0
while :; do
initial_desired=$(read-desired-digest "$window" initial)
initial_observed=$(resolve-tracked-tag "$window" initial)
test "$initial_desired" != "$initial_observed" || exit 0
wait-promotion-window "$window"
current_desired=$(read-desired-digest "$window" current)
current_observed=$(resolve-tracked-tag "$window" current)
if test "$current_desired" != "$initial_desired" || test "$current_observed" != "$initial_observed"; then window=$((window + 1)); continue; fi
exit 1
done
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Coordinated image releases

Use a release group when images from one source repository share a release boundary, such as a UI and worker that share a database and API. Grouped promotion is required once the infrastructure owner declares that boundary. It is not required for unrelated applications.

## Declaration and proof

Keep group membership in a reviewed infrastructure-owned manifest alongside `images.json`. A group has one primary app and a nonempty set of unique members that includes that primary. Every member exists in `images.json`, belongs to at most one group, and has a distinct image repository. All members share the source repository, ref, authorized workflow, enablement state, and promoted source SHA. Validate the whole document before using any member. Membership and source metadata changes use the disabled metadata transition for the whole group.

For example, `{"mail-ui":["mail-ui","mail-worker"]}` declares `mail-ui` as the announcement entrypoint. A companion cannot independently request promotion. The trusted writer reads membership from its trusted base, never from an announcement or candidate branch.

One successful source push run builds all members from the same source commit. Its unique successful build job emits one immutable `IMAGE_PROMOTION_RECORD` per member. Announce only after the entire source workflow has completed successfully. The infrastructure verifier authenticates the configured workflow and run, normalizes its build log, and requires exactly the declared set of records. Every record must bind the same repository, ref, source SHA, and run ID to its configured image and a valid digest. Missing, duplicate, additional, or conflicting records reject the whole candidate. Prove registry access independently for every member.

The primary announcement supplies the existing scalar payload. Companion digests come exclusively from the verified records. A primary identity is bound to the complete proven member-to-digest map. Repeated announcements may add successful run evidence only when that map is identical. A different companion digest is a conflicting release, even when the primary digest is unchanged; reject it and publish a new source commit.

## Desired state and lifecycle

The trusted writer updates every member's digest and source SHA in one infrastructure PR. An unchanged member digest is allowed, but its source SHA must still record the new coordinated release. The provenance gate compares the whole group against the exact proof and rejects partial transitions, unrelated app changes, or metadata edits. The PR changes only `images.json`.

Apply duplicate detection, branch reuse, terminal supersession, current-main ancestry, and rollback to the whole group. Comparing only the primary pin cannot prove a duplicate. Reusing an existing branch requires its complete group map to match. A superseded group operation cannot be reopened through a companion announcement. A rollback selects and verifies the whole previous release as a new approved operation.

## Deployment boundary

A single PR prevents inconsistent desired pins; it does not make container replacement atomic. Choose and document a rollout that respects the application's compatibility boundary. If old and new processes cannot safely share the database or API, stop every group member before migration, run migrations from the selected release, and restart the complete group. Otherwise explicitly establish compatibility across the transition. Do not claim that grouped pins alone prevent temporarily mixed running versions.

Pre-pull and verify all group images before stopping services. Migration failure stops the rollout and leaves completion failed. Do not restart an old member against a changed database without a separately approved recovery decision.

Completion requires the exact infrastructure merge SHA and deploy attempt, matching image readback and health for every member, and successful shared migrations and infrastructure postconditions. A healthy UI with a failed or old worker is incomplete. Record that result on the coordinated PR with a link to the deployment evidence.
Loading