[auxesis]

Summary

This adds the failsafe. It is a test that fails when a workflow's selector job does not decide whether a job runs. The selector job is the first job of a workflow. It compares the files a pull request changes with each check's trigger paths, and each job's if: reads its answer. A job that does not read the answer runs on every change. That costs CI money, and nothing showed it until now.

ADR 0003 in #1416 describes the design. .github/CONTEXT.md on that branch defines "failsafe", "selector job" and "summary job".

Changes

• scripts/__tests__/failsafe.test.mjs: the new test. pnpm run test:scripts runs it, and the lint job in tests.yml runs that on every pull request.
• It reads every workflow that runs on pull_request or merge_group. In each one, a job passes when:
  • it is the selector job (id selector) or the summary job (the last job, with if: always() and every other job in its needs:);
  • it needs the selector job and its if: reads an output the selector job declares;
  • it needs a job that passes by the rule above, and its if: calls no always(), failure() or cancelled(), so GitHub skips it whenever that job skips; or
  • it has a written reason in EXEMPT_JOBS.
• Three lists hold the exemptions. Each entry needs a reason:
  • NOT_CHECKS: claude-review.yml, code-review.yml and impact.yml, for good. The first two are review bots and the third is an advisory report. None is a check, and no branch rule requires them.
  • NOT_YET_CONVERTED: the 24 workflows with no selector job yet. Each conversion removes its entry, so this list shrinks to empty.
  • EXEMPT_JOBS: jobs the selector job does not decide, on purpose. Today these are four jobs in test-eql.yml that run on every change: setup, docs-static, known-failures and doc-anchors. A job that never runs on a pull request, such as a scheduled fuzz campaign, goes here too.
• A stale exemption fails the test. That covers a workflow or job that no longer exists, a workflow listed as not yet converted that has a selector job, and a job the selector job already decides.

Verification

• pnpm run test:scripts: 82 files passed, 1434 tests passed, 3 skipped.
• pnpm exec biome check on the new file: no findings.
• I changed the tree one way at a time, ran the failsafe, and restored the files. It failed for each of these:
  • an exemption removed from EXEMPT_JOBS;
  • an exemption added for a job the selector already decides (test-eql.yml#schema);
  • an exemption with a blank reason;
  • test-eql.yml added to NOT_YET_CONVERTED;
  • a NOT_YET_CONVERTED entry for a workflow that does not exist;
  • an entry removed from NOT_YET_CONVERTED;
  • the if: removed from test-eql.yml#schema;
  • a job's if: reading needs.selector.outputs.eqll, an output the selector does not declare;
  • a new job with no needs: or if:;
  • a new job that needs schema but has if: ${{ !cancelled() }}.
• It passed for a new job that needs schema and has no if:, with the job added to the summary job's needs:.
• I merged each selector conversion pull request (ci(mutants): decide in a selector job whether the gate runs #1420, ci(crates): decide in selector jobs which crate checks run #1421, ci(golang): decide in a selector job which Go jobs run #1423, ci(js): decide in selector jobs which JavaScript checks run #1424, ci(integration): decide in selector jobs which suites run #1425, ci: decide in selector jobs whether the remaining checks run #1426) into this branch one at a time, without committing, and ran the failsafe. Each time it failed only because the converted workflows were still in NOT_YET_CONVERTED. After removing those entries and adding the two job exemptions those pull requests describe (tests-auth-refresh.yml#short and fuzz.yml#fuzz-campaign), it passed for all six.

Related

• Stacked on ci(test-eql): run the Postgres suite only for changes to EQL #1417, which adds the selector job to test-eql.yml. Merge that first.
• Refs docs(ci): record the decisions that bring CI spending under budget #1416 (ADR 0003).
• The pull requests that convert the other workflows, such as ci(mutants): decide in a selector job whether the gate runs #1420 for Mutants, branch from ci(test-eql): run the Postgres suite only for changes to EQL #1417 and do not contain this file. Whichever lands second removes that workflow's NOT_YET_CONVERTED entry.

Review notes

• The failsafe checks pull request decisions only. In test-eql.yml, a gated job runs on every event other than pull_request, including merge_group. ADR 0003 says the selector should decide in the merge queue too. When it does, this test can check merge_group as well.
• ci(mutants): decide in a selector job whether the gate runs #1420 adds scripts/__tests__/lib/selector.mjs. Once both land, the failsafe could read selector outputs through selectorOutputsRead from there. I kept its helper names out of this file, so the guard in shared-test-helpers.test.mjs will not fail when they meet.
• A job added after the summary job, or left out of the summary job's needs:, makes the failsafe report the summary job as undecided. The failure message says how to fix that.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 5eb7542

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}