You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Collapse monorepo CI into one affected-tests job per OS #5052
ci.yaml is ~3.9k lines with 108 jobs, a ~1,130-line path-filter job and a hand-kept aggregate list; ~2.8k more lines of scripts keep that wiring consistent. It changed in 156 commits in 60 days, mostly to wire a new script's job.
What
Run one job per OS that executes run-affected-tests.sh (or emits a dynamic matrix), selecting tests by the <name>.sh / <name>.test.sh convention. A job skipped by if: reports success to required checks (docs).
Keep the ruleset-bound ci-aggregate-contract.yaml admission workflow and a single required check. Heavy suites may keep their own jobs.
What is lost
Nothing, as long as the ruleset-bound admission workflow stays.
First step
Add a --list-json mode, run the new job next to the old jobs for a week and compare selections, then delete the old jobs.
🤖 Generated with Claude Code (interactive session)
Step one is merged in #5060. Tests declare their inputs in # test-inputs: headers (Go modules use // test-inputs: lines in go.mod), and run-affected-tests.sh --convention selects by them. A non-blocking job per OS runs that selection beside the per-script jobs and records the comparison in its job summary.
First results: on Ubuntu nothing is selected only by the filters, and all 123 selected tests pass. On macOS, bugbot-request-marker.test.sh hangs past a 900-second deadline; its per-script job runs only on Linux. That has to be fixed or explained before macOS can gate.
Remaining, from 2026-11-22 once a week of summaries shows the filter-only list empty: delete the per-script jobs, the changes filter job and their status wiring. Make the convention job blocking and the single required check, and keep the ruleset-bound ci-aggregate-contract.yaml admission workflow. Retire ci-job-wiring.sh and contract-test-invocation.sh where nothing needs them, and update the scripts guide's "Wiring a test into CI" section.
The non-blocking macOS job takes about 60% of each PR's macOS runner time. It runs its selected tests one after another for 60 to 75 minutes, while the required macOS jobs of the same run wait up to 30 minutes for a runner. In the first run, the required checks completed 31 minutes after the run started; the critical path was Test worktree cleanup (macos-latest), which waited 21 minutes to start and then ran for 8.5 minutes. This feeds directly into the macOS queue time tracked in #3817 and the 15-minute feedback target in #4394.
I have not proved cause and effect. The org's runner queue is not visible to this identity, and other repositories share the pool. But the job holds a macOS runner the whole time that would otherwise run required work.
Two cheap options that keep the comparison:
Keep the --list comparison step on macOS (it takes seconds) and run the full convention run on Ubuntu only, until the macOS hang in bugbot-request-marker.test.sh is fixed.
Or run the macOS convention run only on main pushes (one run per merge instead of one per PR head).
Changing the job is your call during the comparison week, so I have not opened a PR.
Activity
docs(scripts): keep the scripts guide within the instruction size budget
fix(ci): compare and select Go modules too, and harden the convention…
fix(scripts): refuse a malformed test-inputs line instead of ignoring it
fix(ci): keep the convention job green while it gathers evidence
feat(ci): select affected tests by convention in one job per OS (#5060)
devantler commentedon Nov 15, 2026
🤖 Generated with Claude Code (interactive session)
Step one is merged in #5060. Tests declare their inputs in
# test-inputs:headers (Go modules use// test-inputs:lines ingo.mod), andrun-affected-tests.sh --conventionselects by them. A non-blocking job per OS runs that selection beside the per-script jobs and records the comparison in its job summary.First results: on Ubuntu nothing is selected only by the filters, and all 123 selected tests pass. On macOS,
bugbot-request-marker.test.shhangs past a 900-second deadline; its per-script job runs only on Linux. That has to be fixed or explained before macOS can gate.Remaining, from 2026-11-22 once a week of summaries shows the filter-only list empty: delete the per-script jobs, the
changesfilter job and theirstatuswiring. Make the convention job blocking and the single required check, and keep the ruleset-boundci-aggregate-contract.yamladmission workflow. Retireci-job-wiring.shandcontract-test-invocation.shwhere nothing needs them, and update the scripts guide's "Wiring a test into CI" section.devantler commentedon Nov 15, 2026
A cost of the comparison week, measured from the job timings of three of today's pull-request CI runs (#5071's and two other
claude/*script PRs):The non-blocking macOS job takes about 60% of each PR's macOS runner time. It runs its selected tests one after another for 60 to 75 minutes, while the required macOS jobs of the same run wait up to 30 minutes for a runner. In the first run, the required checks completed 31 minutes after the run started; the critical path was
Test worktree cleanup (macos-latest), which waited 21 minutes to start and then ran for 8.5 minutes. This feeds directly into the macOS queue time tracked in #3817 and the 15-minute feedback target in #4394.I have not proved cause and effect. The org's runner queue is not visible to this identity, and other repositories share the pool. But the job holds a macOS runner the whole time that would otherwise run required work.
Two cheap options that keep the comparison:
--listcomparison step on macOS (it takes seconds) and run the full convention run on Ubuntu only, until the macOS hang inbugbot-request-marker.test.shis fixed.mainpushes (one run per merge instead of one per PR head).Changing the job is your call during the comparison week, so I have not opened a PR.