Skip to content

CI Health Triage: a red nightly streak must reach a human (the DMG was broken 4 nights; the digest knew, nobody read it) #390

Description

@kpmmmurphy

What happened

The Nightly DMG failed four nights running (2026-09-16 → 09-19). CI Health Triage diagnosed it correctly every night — the #11 digest named the exact cause (release-macos.sh requires Xcode 27, nightly-dmg.yml pinned 26) and the exact fix. Nobody saw it: the digest is comment N on a rolling 74-comment issue, and a red scheduled run notifies no one. It was found on 09-19 by a session that happened to run gh run list. Fixed in #389.

Detection worked. Delivery did not.

Proposals (pick one or more)

  1. Escalate on a streak. When the same workflow has been red ≥ 2 consecutive scheduled runs, the triage opens (or re-opens) a dedicated issue titled for that workflow, assigned to Kev — instead of only appending to 🩺 CI Health #11. One issue per streak; closed by the first green run.
  2. Widen the conclusion filter. The prompt inspects only conclusion "failure". A job whose runner label no longer resolves (now a real risk: ci(nightly): build the DMG on GitHub's Xcode 27 image — the nightly has failed since 09-16 #389 moved the nightly to GitHub's preview xcode-27 label) never starts, sits queued, and ends cancelled — outside the filter. Include cancelled and timed_out for scheduled workflows, and flag "no Nightly DMG run in the last 26 h" as its own signal.
  3. A second channel. pulse-weekly (Monday digest → M1K3 remember) could read gh run list --workflow "Nightly DMG" and pin a red streak to the top, like the COPPA verdict.

Also carried from #389's review

  • ci.yml:59-61,68 says Xcode 27 "is not on the runner image yet". True of macos-26, but xcode-27 exists now — reword so nobody waits on macos-26 to catch up. (Comment-only; ci.yml's pins themselves are fine: CI does not invoke the release script.)
  • When a macos-27 image goes GA, move nightly-dmg.yml off the preview label.
  • Watch the nightly's wall clock against timeout-minutes: 60 for the first week on the new image.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions