Skip to content

One monitor, slice 3: the evidence watchdog pages when a nightly gate fails (SCN-2026-008) - #572

Merged
sam-aydlette merged 1 commit into
mainfrom
feat/watchdog-gates
Nov 17, 2026
Merged

sam-aydlette merged 1 commit into
mainfrom
feat/watchdog-gates

Conversation

@sam-aydlette

Copy link
Copy Markdown
Owner

SCN-Type: adaptive
SCN-Rationale: One monitor, slice 3: the evidence watchdog compares the nightly status beacon's gate results with the gate register and pages, through one new alarm, when any nightly gate did not pass, including a detect-and-recover gate that fails inside a successful run.
SCN-Record: SCN-2026-008

SIA:

  • monitoring: yes (a new FailingGates metric and alarm on the existing watchdog, paging through the existing SNS topic)
  • new-component: yes (one CloudWatch alarm, aws_cloudwatch_metric_alarm.evidence["failing-gates"], classified by the existing watchdog group in the boundary classification)
  • access: no (no IAM change: the watchdog reads the same S3 objects and writes to the same metric namespace; the register ships inside the function package)
  • compliance-engine: no (no gate, status or publishing logic changes; the watchdog only reads the published beacon)
  • pipeline: no (no workflow changes)
  • external-service: no (no new service or interconnection)

Changed

  • infrastructure/watchdog/logic.mjs.
    • New parseRegister: checks only the register fields the watchdog reads, and throws on anything else.
    • New failingGates(register, beacon): every nightly check that has a run script and did not pass in the beacon's gate_results (failed, not run, or absent). A beacon without gate results counts every such check. Checks with no run script can't report, so they aren't counted.
    • computeMetrics adds FailingGates when a register is given.
  • infrastructure/watchdog/index.mjs. Reads the packaged gate-modes.json and fails closed without it: nothing is published, so every alarm reads missing data as a breach. Publishes FailingGates. The log line adds the register version and each failing check with its owner and result.
  • infrastructure/watchdog.tf.
    • Packages schemas/gate-modes.json with the function.
    • Adds the failing-gates alarm: threshold 0, 2 of 3 hourly periods, missing data counts as breaching, existing topic.
    • Cost comment updated: five alarms and five custom metrics.
  • docs/runbooks/evidence-sla-alarm.md. New row and section for failing-gates.
  • tests/test_evidence_watchdog.py:
    • every gate passing counts none;
    • a failed, a not-run and an absent gate each count, with owners;
    • a gate failing inside a successful run counts while NightlyFailed stays 0;
    • checks not on the runner aren't counted;
    • no beacon counts every runnable gate;
    • an invalid register is rejected;
    • the handler publishes FailingGates and names the gate in its log;
    • a missing or invalid register publishes nothing;
    • the alarm and the packaging are pinned.

Effect

After deploy, the watchdog publishes a fifth metric and the account gains one alarm, samaydlette-com-evidence-watchdog-failing-gates. With the current beacon (21 checks passing, 5 not on the runner), FailingGates is 0 and the alarm reads OK. Recurring cost rises about $0.40 a month: one custom metric and one standard alarm.

Verified

  • make check passes. The watchdog suite passes 26 tests, 10 of them new.

  • The new alarm is covered by the existing boundary classification match aws::metric_alarm::evidence*.

  • A local terraform plan was not usable. The make -C infrastructure plan target calls a script that is not in the repository, and a direct plan from the workstation runs with stale local variables and a read-only identity that cannot read every resource. This PR's CI plan is the plan of record.

  • CodeGuard (manual pass over the rules for JavaScript, HCL and IaC against this diff):

    • no secrets;
    • no IAM or network change;
    • the register is parsed as data and validated, never evaluated;
    • the alarm uses the existing CMK-encrypted topic.

    No findings.

Residual / needs human

  • The deploy applies the Lambda code update and the new alarm, behind the prod approval.

Couldn't verify

  • The live alarm state and the watchdog's first log line with FailingGates. I'll check both after deploy.

Follow-ups noticed

  • make -C infrastructure plan calls ./terraform-plan.sh, which does not exist, so the target always fails. The skill and CLAUDE.md name it as the pre-apply check.

🤖 Generated with Claude Code

… fails

The watchdog reads the gate register packaged with it and counts every
nightly check with a run script that did not pass in the status beacon's
gate_results (failed, not run, or absent). A new FailingGates metric and
alarm page on it, so a detect-and-recover gate that fails inside a
successful nightly reaches the operator from AWS's clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sam-aydlette
sam-aydlette merged commit 58be45e into main Nov 17, 2026
9 checks passed
@sam-aydlette
sam-aydlette deleted the feat/watchdog-gates branch November 17, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant