Skip to content

Add reliability signals, notifications and recovery coverage - #150

Merged
kasparsd merged 6 commits into
mainfrom
codex/sre-reliability
Nov 16, 2026
Merged

kasparsd merged 6 commits into
mainfrom
codex/sre-reliability

Conversation

@kasparsd

@kasparsd kasparsd commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Operators can monitor control readiness, scrape project-filtered metrics, receive signed best-effort notifications and check external application reachability without adding required services.

  • Add authenticated readiness and metrics, configurable writer admission with Retry-After, and nonblocking signed notifications with bounded retries and delivery counters.
  • Skip never-seen stale alerts and allow a startup reconnect grace period. Invalid optional webhook configuration disables delivery while control remains available.
  • Add structured control/agent logs with quiet routine agent requests, aborted-request logging, safe default classifications and opt-in sanitized debug causes.
  • Preserve existing cross-node host ordering and replacement semantics; the sorting change was removed to avoid upgrade churn and resolver-order changes.
  • Add recovery, ENOSPC, overdue-writer, notification shutdown, authorization and convergence coverage. The external probe follows bounded same-host redirects and rejects HTTPS downgrades.
  • Consolidate changesets into observability, notifications and recovery documentation/tests.

Validation: 1,176 local tests passed; typecheck, build, lint and documentation checks passed (three pre-existing lint warnings). Code commit fa1a609 passed the complete CI matrix, including the damaged-repository Docker drill, fake-VPS convergence and all Node versions. Latest commit 9bbf18d only records verified runtime evidence; its CI rerun is pending.

The source-built linux/arm64 agent ran in a disposable Ubuntu 24.04.5 guest, emitted JSON startup logs, remained online using its existing state volume, and passed debug-cause checks against synthetic bearer/basic/cookie/token/URL secrets. The verified logger hash matches the pushed source. Guest/control processes and temporary credentials/build artifacts were cleaned up. Canonical evidence is in docs/real-host-evaluation.md.

CI investigation: the repository-permission fixture was corrected; the subsequent fake-VPS inspect gap was narrowed to discovery convergence and now retries only the exact transient missing-container response within the existing deadline. The intervening commit changed only the backup integration fixture, not product runtime. Restore assertions now include response bodies if the intermittent 400 recurs; all checks passed on the review code.

Limits: notification queues/cursors are in memory and can lose events across outages or shutdown; retain external availability and scheduled backup checks. Real endpoint changes still recreate clients to update hosts entries. Debug sanitization covers common patterns, not arbitrary application-secret recognition. Injected faults do not establish behavior on a physically full production host.

@kasparsd kasparsd changed the title Add control readiness and reliability acceptance criteria Add reliability signals, notifications and recovery coverage Nov 16, 2026
@kasparsd
kasparsd merged commit d4b995d into main Nov 16, 2026
9 checks passed
@kasparsd
kasparsd deleted the codex/sre-reliability branch November 16, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant