Skip to content

Collapse monorepo CI into one affected-tests job per OS #5052

Description

@devantler

Why

ci.yaml is ~3.9k lines with 108 jobs, a ~1,130-line path-filter job and a hand-kept aggregate list; ~2.8k more lines of scripts keep that wiring consistent. It changed in 156 commits in 60 days, mostly to wire a new script's job.

What

Run one job per OS that executes run-affected-tests.sh (or emits a dynamic matrix), selecting tests by the <name>.sh / <name>.test.sh convention. A job skipped by if: reports success to required checks (docs).

Keep the ruleset-bound ci-aggregate-contract.yaml admission workflow and a single required check. Heavy suites may keep their own jobs.

What is lost

Nothing, as long as the ruleset-bound admission workflow stays.

First step

Add a --list-json mode, run the new job next to the old jobs for a week and compare selections, then delete the old jobs.

Part of #5047.

🤖 Generated with Claude Code

Activity

  1. moved this to 📥 Backlog in 🌊 Project Boardon Nov 14, 2026
  2. devantler commented on Nov 15, 2026

    @devantler
    ContributorAuthor

    🤖 Generated with Claude Code (interactive session)

    Step one is merged in #5060. Tests declare their inputs in # test-inputs: headers (Go modules use // test-inputs: lines in go.mod), and run-affected-tests.sh --convention selects by them. A non-blocking job per OS runs that selection beside the per-script jobs and records the comparison in its job summary.

    First results: on Ubuntu nothing is selected only by the filters, and all 123 selected tests pass. On macOS, bugbot-request-marker.test.sh hangs past a 900-second deadline; its per-script job runs only on Linux. That has to be fixed or explained before macOS can gate.

    Remaining, from 2026-11-22 once a week of summaries shows the filter-only list empty: delete the per-script jobs, the changes filter job and their status wiring. Make the convention job blocking and the single required check, and keep the ruleset-bound ci-aggregate-contract.yaml admission workflow. Retire ci-job-wiring.sh and contract-test-invocation.sh where nothing needs them, and update the scripts guide's "Wiring a test into CI" section.

  3. devantler commented on Nov 15, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    A cost of the comparison week, measured from the job timings of three of today's pull-request CI runs (#5071's and two other claude/* script PRs):

    Run macOS convention job (runtime) All other macOS jobs (runtime, total) Longest wait for a macOS runner
    43672938524 4,195 s 2,777 s 1,560 s
    43668385614 4,476 s 3,026 s 922 s
    43663221770 3,687 s 2,433 s 1,799 s

    The non-blocking macOS job takes about 60% of each PR's macOS runner time. It runs its selected tests one after another for 60 to 75 minutes, while the required macOS jobs of the same run wait up to 30 minutes for a runner. In the first run, the required checks completed 31 minutes after the run started; the critical path was Test worktree cleanup (macos-latest), which waited 21 minutes to start and then ran for 8.5 minutes. This feeds directly into the macOS queue time tracked in #3817 and the 15-minute feedback target in #4394.

    I have not proved cause and effect. The org's runner queue is not visible to this identity, and other repositories share the pool. But the job holds a macOS runner the whole time that would otherwise run required work.

    Two cheap options that keep the comparison:

    1. Keep the --list comparison step on macOS (it takes seconds) and run the full convention run on Ubuntu only, until the macOS hang in bugbot-request-marker.test.sh is fixed.
    2. Or run the macOS convention run only on main pushes (one run per merge instead of one per PR head).

    Changing the job is your call during the comparison week, so I have not opened a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Fields

    No fields configured for Refactor.

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions