Skip to content

Run CI nightly on main, and start it by hand - #126

Merged
NSchatz merged 1 commit into
mainfrom
harness/40-nightly-run-of-ci
Oct 20, 2026
Merged

NSchatz merged 1 commit into
mainfrom
harness/40-nightly-run-of-ci

Conversation

@NSchatz

@NSchatz NSchatz commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

CI on main now also runs every night, and can be started by hand.

Harness-Task: 40

What changes

.github/workflows/ci.yml gets two more triggers. push and pull_request stay as they are:

  schedule:
    # The nightly run on main, 05:27 UTC. An odd minute: GitHub delays scheduled runs at the start of every hour.
    - cron: "27 5 * * *"
  workflow_dispatch:

A scheduled run tests the latest commit on the default branch. So if main goes red with no push behind it (a dependency, a runner image, an expired date), we find out the next morning.

The one guard: the actionlint job's reporter

actionlint runs reviewdog/action-actionlint@v1 (image v1.80.4, reviewdog 0.21.2) with its default reporter, github-pr-check. That reporter would fail every scheduled run:

  • reviewdog reads the owner and repository from the event payload's repository. A schedule payload has none, so it creates its check run at repos///check-runs and gets a 404.
  • Its fallback to log annotations (service/github/check.go, Flush) covers only a 403, so it returns failed to create check: ... 404 Not Found and exits 1. The action's entrypoint.sh runs under set -eu with reviewdog last in the pipeline, so the step fails even with fail_level: none. This is Github-Check gives POST 404 error when used inside a cron job reviewdog/reviewdog#832 and #1263, both still open.

The step now passes reporter: ${{ github.event_name == 'pull_request' && 'github-pr-check' || 'github-annotations' }}:

  • On a pull request nothing changes. It is still github-pr-check with filter_mode: file, so a PR run checks exactly what it did before.
  • On every other event, github-annotations writes the annotations straight to the log, with no API call. A push already produced that same output: this workflow's token has no checks: write, so the check reporter got a 403 and fell back to the same writer. Commit 8fc7faf's check runs show this. There is one actionlint run (the job) and no separate check run from reviewdog.
  • workflow_dispatch gets the same reporter as schedule, so the dispatched run below goes down the nightly path.

Local reproduction with the reviewdog 0.21.2 release binary (checksum verified) and the flags the action passes. The GitHub API was a stub on localhost (GITHUB_API): it answers 404 to an empty owner or repository and 403 to a check-run create, like a token without checks: write:

event reporter API calls exit
schedule github-pr-check POST /repos///check-runs -> 404 1 (failed to create check)
push github-pr-check POST /repos/cosyte/config/check-runs -> 403, fallback 0
workflow_dispatch github-pr-check POST /repos/cosyte/config/check-runs -> 403, fallback 0
schedule github-annotations none 0
workflow_dispatch github-annotations none 0
push github-annotations none 0

The annotation line was the same in every 0-exit case.

Why no other step needed a guard

We read every job and step. None reads github.event.pull_request.*, github.head_ref, github.base_ref, github.event.before or github.event_name, and none diffs against a PR base. The scripts verify runs (changeset-guard, release-notes prepare, credential-surface, install-hardening, readme-check, drift-check, check-no-emdash) read no event variables. release-notes prepare compares HEAD with HEAD^, which on a scheduled run is the tip of main and its parent, the same shape a push to main has. release.yml triggers only on push, so a nightly CI run cannot start a release.

No job was renamed or dropped, so the required checks (actionlint, verify (22), verify (24)) are unchanged.

Proof of the non-PR path

gh workflow run ci.yml -R cosyte/config --ref harness/40-nightly-run-of-ci was accepted and ran on the head commit 5c5bca2: https://github.com/cosyte/config/actions/runs/39356352654. It concluded success, with verify (22), verify (24), release-dry-run and actionlint all passing on the first attempt.

In that run the actionlint step shows reporter: github-annotations in its inputs and prints ##[error][actionlint] reported by reviewdog. That is the existing node20 finding in perf-calibration.yml, which fail_level: none reports without failing. So on the nightly path the job lints every workflow file and does not skip.

Gate run locally

All passed, in verify's order, with pnpm 10.34.5: changeset-guard, release-notes prepare, credential-surface, install-hardening, readme-check, drift-check, pnpm install --frozen-lockfile, check:no-emdash, format:check, typecheck, lint, test, build, attw. Also actionlint 1.7.7 over .github/workflows/*.yml. We ran drift-check from a copy laid out as CI lays it out (<dir>/config), because it grades the config checkout beside the umbrella root. In this worktree's path it finds none and exits 2 by design.

Notes

  • The concurrency group is ${{ github.workflow }}-${{ github.ref }} with cancel-in-progress: true. A scheduled run and a push run on main share one group, so whichever starts second cancels the other. Both test the same main, and a cancelled run reports no result either way. We left it alone because it is outside this change's scope.
  • One limit the nightly shares with a push today: reviewdog's log-annotation writer fails its Flush once it has written more than 9 annotations (MaxLoggingAnnotationsPerStep, service/github/githubutils/comment_writer.go), whatever fail_level says. A push already reached that writer through the 403 fallback, so this change does not add the limit. It means a main with 10 or more actionlint findings goes red on the nightly, as it would on a push. Today there is one finding.
  • No changeset: nothing under a package's src/ or in a published tarball moves.

🤖 Generated with Claude Code

ci.yml gets a schedule trigger, cron "27 5 * * *" (05:27 UTC, an odd
minute because GitHub delays scheduled runs at the start of every
hour), and workflow_dispatch. push and pull_request stay as they are.

The actionlint job now passes reviewdog's reporter by event: the
action's default github-pr-check on a pull request, github-annotations
on every other event. A schedule event's payload has no repository, so
the check reporter in reviewdog 0.21.2 posts to repos///check-runs,
gets a 404 and exits 1 (its log-annotation fallback covers a 403
only), which would make every nightly run red. github-annotations
writes the same annotations without the API call; a push already
produced that output through the 403 fallback, since this token has no
checks: write.

Harness-Task: 40

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@NSchatz
NSchatz merged commit 5c1d29d into main Oct 20, 2026
13 checks passed
@NSchatz
NSchatz deleted the harness/40-nightly-run-of-ci branch October 20, 2026 02:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant