Skip to content

One monitor, slice 2: one self-closing alert issue per failing gate, labelled with its owners (SCN-2026-008) - #570

Merged
sam-aydlette merged 1 commit into
mainfrom
feat/gate-alerts
Nov 16, 2026
Merged

sam-aydlette merged 1 commit into
mainfrom
feat/gate-alerts

Conversation

@sam-aydlette

Copy link
Copy Markdown
Owner

SCN-Type: adaptive
SCN-Rationale: One monitor, slice 2: the deploy and the nightly keep one self-closing alert issue per failing gate, labelled with the owners the gate register names, in place of one deployment-failure issue and one stale-evidence issue.
SCN-Record: SCN-2026-008

SIA:

  • monitoring: yes (the deploy and nightly alert jobs now raise one issue per failing gate, close it when the gate passes, and keep one fallback issue for a run that failed outside any gate)
  • pipeline: yes (both workflows upload the gate results they record and their alert jobs run on every main run, not only failed ones)
  • access: no (no new permission; the alert jobs keep issues: write and contents: read, and the nightly alert drops pull-requests: read)
  • security-controls: no (no gate changes; every gate runs with the same logic and arguments and still blocks)
  • compliance-engine: no (how status is decided and published is unchanged)
  • external-service: no (the upload, download and checkout actions are ones the workflows already use, at the same pinned SHAs)

Changed

  • scripts/gate-alerts.py (new). Reads the results the gates recorded (GATE_RESULTS_FILE) and, through _gates.stage_results, every gate the register lists for the given stages. One issue per gate command:
    • fail: open or reopen [deploy|nightly] gate failing: <command> and comment with the run, its checks and their owners;
    • pass: close it if open;
    • not run, or not on the runner: no change.
      Checks that share one command (the reconcile.* invariants) pass or fail together, so they share one issue labelled with every owner they name. A run that failed while no gate failed opens [<context>] run failed outside any gate; the next successful run closes it. Issues carry gate-alert plus a role:* label per owner and are assigned to the repository owner. --dry-run prints the actions without calling GitHub.
  • .github/workflows/deploy-with-opa.yml.
    • The pr-stage gates step and the deploy job record results; each job uploads them (gate-results-pr, gate-results-deploy) even when it fails.
    • notify becomes Gate alerts: needs the test job too, runs on every main run that was not cancelled (pull requests never touch the issues), and calls the script over stages pr, pre-apply, post-apply and pre-publish. The queue watchdog's dependent-job list follows the rename.
  • .github/workflows/evidence-nightly.yml.
    • The refresh job uploads its results (gate-results-nightly).
    • alert becomes Gate alerts: it runs on every run that was not deliberately dropped, so passing gates close their issues. The published-evidence age report stays.
    • The refresh job's outputs and the vulnerability step's findings export, used only by the old issue text, are gone; the step runs the runner directly and its log names each finding.
  • docs/runbooks/evidence-sla-alarm.md. Points to the per-gate issues.
  • tests/test_gate_alerts.py (new). A failing gate opens one issue with every owner; a pass closes it and unreported gates are left alone; a run that failed outside any gate opens the fallback and success closes it; each workflow keeps its own issues; a dry run calls nothing.

Effect

After deploy, each deploy run on main and each nightly run opens, updates or closes gate-alert issues. With today's results nothing opens: the post-deploy nightly's gates are all passing (21 checks under 5 commands), so the first runs only close. The old single "Deployment failing" and "Evidence nightly failed" issues are no longer updated.

Verified

  • make check passes.
  • A dry run over the post-deploy nightly's recorded results gives 5 closes and 0 opens.
  • A test with an injected failure shows the reconcile command's issue would open, labelled with every owner its checks name.
  • CodeGuard (manual pass over the rules for Python, shell and YAML against this diff): gh is called with an argument list and no shell; issue titles come from the register's run values, which its loader restricts to scripts/<name> [subcommand]; job results reach the step through the environment; actions pinned by SHA; the token is the job's own, with no new permission. No findings.

Residual / needs human

  • The 99 open "Deployment Failed" issues and the old "Deployment failing" and "Evidence nightly failed" issues are closed separately.

Couldn't verify

  • The real issue create, close and label calls run only on main. The post-deploy nightly is their first real run.

Follow-ups noticed

  • The old nightly issue matched each vulnerability finding to its advisory and any open Dependabot PR; the new one links the run log. Carrying the findings into the vuln gate's issue would restore that.

🤖 Generated with Claude Code

…labelled with its owners

The deploy and the nightly each keep one issue per gate command, from the
results the gates recorded: a failure opens or reopens it, a pass closes it.
A run that failed outside any gate keeps one fallback issue per workflow.
Replaces the deployment-failure and stale-evidence issues.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sam-aydlette
sam-aydlette merged commit d1d2b53 into main Nov 16, 2026
9 checks passed
@sam-aydlette
sam-aydlette deleted the feat/gate-alerts branch November 16, 2026 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant