Repository navigation
One monitor, slice 3: the evidence watchdog pages when a nightly gate fails (SCN-2026-008) - #572
Merged
Merged
Conversation
… fails The watchdog reads the gate register packaged with it and counts every nightly check with a run script that did not pass in the status beacon's gate_results (failed, not run, or absent). A new FailingGates metric and alarm page on it, so a detect-and-recover gate that fails inside a successful nightly reaches the operator from AWS's clock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SCN-Type: adaptive
SCN-Rationale: One monitor, slice 3: the evidence watchdog compares the nightly status beacon's gate results with the gate register and pages, through one new alarm, when any nightly gate did not pass, including a detect-and-recover gate that fails inside a successful run.
SCN-Record: SCN-2026-008
SIA:
aws_cloudwatch_metric_alarm.evidence["failing-gates"], classified by the existing watchdog group in the boundary classification)Changed
infrastructure/watchdog/logic.mjs.parseRegister: checks only the register fields the watchdog reads, and throws on anything else.failingGates(register, beacon): every nightly check that has a run script and did not pass in the beacon'sgate_results(failed, not run, or absent). A beacon without gate results counts every such check. Checks with no run script can't report, so they aren't counted.computeMetricsaddsFailingGateswhen a register is given.infrastructure/watchdog/index.mjs. Reads the packagedgate-modes.jsonand fails closed without it: nothing is published, so every alarm reads missing data as a breach. PublishesFailingGates. The log line adds the register version and each failing check with its owner and result.infrastructure/watchdog.tf.schemas/gate-modes.jsonwith the function.failing-gatesalarm: threshold 0, 2 of 3 hourly periods, missing data counts as breaching, existing topic.docs/runbooks/evidence-sla-alarm.md. New row and section forfailing-gates.tests/test_evidence_watchdog.py:Effect
After deploy, the watchdog publishes a fifth metric and the account gains one alarm,
samaydlette-com-evidence-watchdog-failing-gates. With the current beacon (21 checks passing, 5 not on the runner), FailingGates is 0 and the alarm reads OK. Recurring cost rises about $0.40 a month: one custom metric and one standard alarm.Verified
make checkpasses. The watchdog suite passes 26 tests, 10 of them new.The new alarm is covered by the existing boundary classification match
aws::metric_alarm::evidence*.A local
terraform planwas not usable. Themake -C infrastructure plantarget calls a script that is not in the repository, and a direct plan from the workstation runs with stale local variables and a read-only identity that cannot read every resource. This PR's CI plan is the plan of record.CodeGuard (manual pass over the rules for JavaScript, HCL and IaC against this diff):
No findings.
Residual / needs human
prodapproval.Couldn't verify
Follow-ups noticed
make -C infrastructure plancalls./terraform-plan.sh, which does not exist, so the target always fails. The skill and CLAUDE.md name it as the pre-apply check.🤖 Generated with Claude Code