Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -261,3 +261,51 @@ jobs:
- name: Run tests
run: npm run test
continue-on-error: true

# Red-main alarm for the DISPATCHED path.
#
# Every merge here is an auto-merge made with GITHUB_TOKEN, which fires no
# push event — so the sweep re-arms this workflow by dispatch, and a
# dispatched run emits no `workflow_run` event for main-red-alert.yml to
# catch. That workflow is therefore silent on almost every main CI run this
# repo produces (observed 2026-08-06: main went red on a66baa55 and no issue
# was filed). A dispatched run does the handoff itself; the push path keeps
# using main-red-alert.yml. One trigger per path, one shared policy script.
post-main:
name: Main Red Alert (dispatched)
needs: [quality, auth-smoke, inventory-smoke, e2e-local]
if: always() && github.ref == 'refs/heads/main' && github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
issues: write
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPO: ${{ github.repository }}
steps:
- name: Checkout code
uses: actions/checkout@v7

- name: Resolve this run's verdict
id: verdict
env:
R_QUALITY: ${{ needs.quality.result }}
R_AUTH: ${{ needs.auth-smoke.result }}
R_INVENTORY: ${{ needs.inventory-smoke.result }}
R_E2E: ${{ needs.e2e-local.result }}
run: |
set -euo pipefail
# `skipped` is not a failure — jobs here are conditional on event type.
results="$R_QUALITY $R_AUTH $R_INVENTORY $R_E2E"
conclusion=success
case "$results" in *cancelled*) conclusion=cancelled ;; esac
case "$results" in *failure*) conclusion=failure ;; esac
echo "conclusion=$conclusion" >> "$GITHUB_OUTPUT"
echo "verdict: $conclusion (from: $results)"

- name: File or resolve the main-red issue
env:
CONCLUSION: ${{ steps.verdict.outputs.conclusion }}
RUN_SHA: ${{ github.sha }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: bash scripts/ci/main-red-alert.sh
49 changes: 12 additions & 37 deletions .github/workflows/main-red-alert.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,13 @@ name: Main Red Alert
# "the base is broken", which is the expensive part: the symptom points away
# from the cause.
#
# This files (or updates) ONE issue the moment CI fails on main, and closes it
# when main is green again — so "is main broken?" is answerable without
# watching Actions, and the merge train never stalls unnoticed again.
# This workflow covers the PUSH path only. A CI run dispatched by the auto-merge
# sweep emits no `workflow_run` event (GitHub suppresses it for runs created with
# GITHUB_TOKEN), so that path calls the same script from ci.yml's post-main job
# instead. That second path is the COMMON one here — every merge is an
# auto-merge — so a workflow_run-only alarm is silent on almost every main CI
# run this repo produces. The policy lives in scripts/ci/main-red-alert.sh so
# the two paths cannot drift.
#
# Fixing a red main means re-running MAIN's run (`gh run rerun <id> --failed`),
# not just your branch's. A green branch on a red base merges nothing.
Expand All @@ -31,44 +35,15 @@ permissions:
jobs:
alert:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4

- name: File or resolve the main-red issue
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPO: ${{ github.repository }}
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
RUN_URL: ${{ github.event.workflow_run.html_url }}
RUN_SHA: ${{ github.event.workflow_run.head_sha }}
REPO: ${{ github.repository }}
run: |
set -euo pipefail
TITLE="🔴 main is red — CI failing on main"

# A cancelled run means a newer push superseded this one (the CI
# concurrency group cancels in-progress runs). That is not a failure,
# and treating it as one is how "cancelled" gets misread as "broken".
if [ "$CONCLUSION" != "failure" ]; then
existing=$(gh issue list -R "$REPO" --state open --search "$TITLE in:title" --json number --jq '.[0].number // empty')
if [ "$CONCLUSION" = "success" ] && [ -n "$existing" ]; then
gh issue close "$existing" -R "$REPO" \
--comment "main is green again as of ${RUN_SHA:0:8} — $RUN_URL"
fi
exit 0
fi

BODY=$(printf '%s\n' \
"CI failed on \`main\` at commit \`${RUN_SHA:0:8}\`." \
"" \
"Run: $RUN_URL" \
"" \
"**Every PR branched from this commit inherits the failure**, and auto-merge refuses to merge onto a broken base — so nothing ships until this is green." \
"" \
"Fix or revert, then re-run **main's** run (\`gh run rerun <run-id> --failed\`). Re-running your own branch does not clear this." \
"" \
"This issue closes itself when a CI run on main succeeds.")

existing=$(gh issue list -R "$REPO" --state open --search "$TITLE in:title" --json number --jq '.[0].number // empty')
if [ -n "$existing" ]; then
gh issue comment "$existing" -R "$REPO" --body "$BODY"
else
gh issue create -R "$REPO" --title "$TITLE" --body "$BODY"
fi
run: bash scripts/ci/main-red-alert.sh
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,11 @@ tina/__generated__
# Uploaded files (development)
public/uploads/
*.sh
# CI scripts are part of the build contract — workflows call them by path, so a
# missing one breaks CI rather than merely being absent. Negate the whole
# directory instead of listing them one by one; the per-file negations below
# predate this and are kept because those scripts live outside scripts/ci/.
!scripts/ci/*.sh
!scripts/selfhost-deploy-revampit.sh
!scripts/e2e-inventory-prod.sh
!scripts/db/apply-migrations-ci.sh
Expand Down
64 changes: 64 additions & 0 deletions scripts/ci/main-red-alert.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#!/usr/bin/env bash
#
# File (or resolve) the single "main is red" issue.
#
# WHY THIS IS A SCRIPT AND NOT INLINE YAML
# ----------------------------------------
# Two different workflows have to reach this same verdict:
# * main-red-alert.yml, from the `workflow_run` event, when CI ran on a push;
# * ci.yml's post-main job, when CI was dispatched by the auto-merge sweep and
# GitHub therefore emits no `workflow_run` event at all.
#
# The second path is the COMMON one here: every merge is an auto-merge, made
# with GITHUB_TOKEN, which fires no push event — so the sweep re-arms ci.yml by
# dispatch, and a dispatched run cascades nothing. A workflow_run-only alarm is
# silent on exactly the path that produces almost every main CI run in this repo.
# (Observed 2026-08-06: main went red on a66baa55 and no issue was filed.)
#
# One policy, one file. Duplicating it in YAML is how the two paths would drift.
#
# Inputs (env):
# REPO owner/name
# CONCLUSION success | failure | cancelled | ...
# RUN_SHA commit the run was for
# RUN_URL link to the run

set -euo pipefail

: "${REPO:?REPO is required}"
: "${CONCLUSION:?CONCLUSION is required}"
RUN_SHA="${RUN_SHA:-unknown}"
RUN_URL="${RUN_URL:-}"

TITLE="🔴 main is red — CI failing on main"

existing=$(gh issue list -R "$REPO" --state open --search "$TITLE in:title" \
--json number --jq '.[0].number // empty')

# A cancelled run means a newer push superseded this one (the CI concurrency
# group cancels in-progress runs). That is not a failure, and treating it as one
# is how "cancelled" gets misread as "broken".
if [ "$CONCLUSION" != "failure" ]; then
if [ "$CONCLUSION" = "success" ] && [ -n "$existing" ]; then
gh issue close "$existing" -R "$REPO" \
--comment "main is green again as of ${RUN_SHA:0:8} — $RUN_URL"
fi
exit 0
fi

BODY=$(printf '%s\n' \
"CI failed on \`main\` at commit \`${RUN_SHA:0:8}\`." \
"" \
"Run: $RUN_URL" \
"" \
"**Every PR branched from this commit inherits the failure**, and auto-merge refuses to merge onto a broken base — so nothing ships until this is green." \
"" \
"Fix or revert, then re-run **main's** run (\`gh run rerun <run-id> --failed\`). Re-running your own branch does not clear this." \
"" \
"This issue closes itself when a CI run on main succeeds.")

if [ -n "$existing" ]; then
gh issue comment "$existing" -R "$REPO" --body "$BODY"
else
gh issue create -R "$REPO" --title "$TITLE" --body "$BODY"
fi
Loading