Skip to content

CI on one server: all parts in parallel, merge-queue and 6-hour runs - #36

Open
ice-zeus wants to merge 2 commits into
mainfrom
task/25-ci-on-one-server
Open

ice-zeus wants to merge 2 commits into
mainfrom
task/25-ci-on-one-server

Conversation

@ice-zeus

@ice-zeus ice-zeus commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Closes #25

Replaces #31. That branch's commits carried AI-attribution trailers, which maestro-policy rejects, and pushed history is never rewritten. So this branch holds the same change as #31's last head (ee24bef) as clean commits on top of current main, which now includes #30. The proof runs below ran on #31's heads; tier1 is unchanged since then.

CI becomes one job, tier1, on one self-hosted server per run, with every part running at the same time through .github/scripts/ci-run.sh (parts runner ci-parts.sh from #22). It runs on pull requests (fast mode), merge-queue groups, every 6 hours on main and on dispatch (full mode); pushes to main run nothing. A red scheduled run opens one issue through report. rust-toolchain.toml pins 1.94.1 with rustfmt and clippy; .config/nextest.toml fails a test at 90 s and never retries. ENGINEERING.md "Continuous integration" describes it all, and docs/engineering.html is re-rendered.

Size: about 445 changed production lines (ci.yml +121/−193, ci-run.sh 93, nextest.toml 27, rust-toolchain.toml 5, .gitignore 2, repo-policy.sh 2). It replaces one workflow; splitting it would leave main with two CI designs at once. Tests (tests/ci-run.test.mjs, 8 tests) and docs are extra.

Changes beyond the issue text, each found by a proof run:

  • report installs the GitHub CLI 2.103.3 through .github/scripts/install-tool.sh, with the same pin as maestro-nightly (Maestro checks: maestro-policy, maestro-queue and a nightly cargo and npm audit #30). No CI image has gh, so report-red.sh failed with gh: command not found (run 42387373242). Adding gh to the ciod-std image would be the lasting fix, and that belongs to Maestro.
  • nextest writes JUnit under its store folder, not under --target-dir. .config/nextest.toml pins [store] dir = "target/nextest", so the reports are at target/nextest/<profile>/junit.xml.
  • The skip rule for Rust parts on a PR that changes JS but no Rust input is "no Rust input changed (.github/scripts/changed-scope.sh)". "Only documentation changed" would be wrong there.
  • repo-policy.sh: dropped a || true inside a process substitution, where its exit status is ignored anyway, so the no-escape-hatch grep is empty.
  • Part logs are kept in the ci-timing artifact. A passing part's log is not printed, and the nextest "skipped" count is only in that log.

Measurement

The 9 runs used head e157f78, warm, after a PR run of that head. The job time runs from job start to job end. Lifetime and sccache come from the ciod line. Cost is lifetime × the hourly price.

Run Size Server Wait for server Job Parts step sccache Lifetime Cost Result
42386335254 ci-8cpu c-8 113 s 33 s 22 s 100% 0m29s $0.0020 success
42386766612 ci-8cpu c-8 102 s 34 s 24 s 100% 0m30s $0.0021 success
42386775712 ci-8cpu c-8 98 s 33 s 23 s 100% 0m30s $0.0021 success
42386338304 ci-16cpu c-16 96 s 25 s 15 s 100% 0m22s $0.0031 success
42386769657 ci-16cpu c-16 99 s 22 s 13 s 100% 0m19s $0.0026 success
42386778585 ci-16cpu c-16 93 s 23 s 13 s 100% 0m20s $0.0028 success
42386341188 ci-32cpu c-32 91 s 19 s 9 s 100% 0m15s $0.0042 success
42386772899 ci-32cpu c-32 98 s 21 s 10 s 100% 0m17s $0.0047 success
42386781074 ci-32cpu c-32 94 s 21 s 11 s 100% 0m18s $0.0050 success

<size> = ci-8cpu. It is the smallest size whose three runs all have the parts step under 90 s (22–24 s) and the job under 4 minutes (33–34 s). Most of the wall time is the 90–115 s wait for a fresh server, and that wait does not shrink with a bigger size.

sccache: warm PR runs show 100% (e.g. 42385595834: ciod: c-8 · … · sccache 100% · 0m32s). The very first run with the new --target-dir folders showed 32% (cold), and every run since shows 100%.

Same coverage

The old CI is run 42371471477. The new CI is the final head ee24bef: PR run 42387982599 and the stability runs below.

Part Old CI New CI
policy repo-policy.sh (6 checks) same
docs docs/readme.html and docs/engineering.html fresh same, in its own worktree
actionlint 1.7.12, all workflows same
toolchain RUSTUP_TOOLCHAIN = Dockerfile rust-toolchain.toml = Dockerfile
fmt cargo fmt --check same
clippy cargo clippy --all-targets --locked -- -D warnings same, --target-dir target/clippy
Rust tests cargo test: 31 passed, 11 ignored (need REDIS_TEST_URL) nextest: 31 passed, 11 skipped
Node version major 24 same
npm ci --ignore-scripts same, --prefer-offline (warm image npm cache)
JS tests node --test: 14 files, 152 passed 20 files, 242 passed (see below)
vendor node scripts/vendor.mjs same, after the JS tests
npm audit PRs changing package files or .github; every push, schedule and dispatch PRs changing package files or .github; dispatch; nightly in maestro-nightly (#24)

The JS difference comes only from new tests. Run on a server, the 14 baseline files give exactly 152 passed. The 5 files added before this rebase give 80: ci-parts, ci-timing and report-red (#22), repo-policy (#23), and ci-run (this PR). #30, now on main, adds 10 more: install-tool.test.mjs (9) and one repo-policy test. 152 + 80 + 10 = 242. The "CI time" summary table shows fewer, because ci-timing.py keys tests by name and some names repeat across files. node's own count in the part log is the real one.

Every part can fail the run

Each proof branch was a PR head plus one breaking commit, dispatched with gh workflow run ci.yml --ref proof/<case> and then deleted. The first 13 cases ran on aa5d180. kill -9 and report ran on ee24bef, which only adds the report job's gh step; tier1 is the same on both heads.

Case Run Result
repo policy: a tracked home-directory path 42387370705 red: policy failed ("tracked text contains an absolute home-directory path")¹
stale docs 42387352543 red: docs failed ("…is stale")¹
actionlint error 42387346648 red: only actionlint failed
fmt 42387354729 red: only fmt failed
clippy warning (needless_return) 42387350081 red: only clippy failed (exit 101)
failing Rust test 42387375671 red: only rust-test failed (exit 100)
failing JS test 42387357052 red: only node-test failed; vendor not run (depends on it)
vendor error 42387383131 red: only vendor failed
npm ci error (package.json and lock out of sync) 42387368104 red: only npm-ci failed; node-test and vendor not run (depend on it)
npm audit (minimist 1.2.5 in the lock, 1 critical) 42387364324 red: only npm-audit failed
toolchain differs from the Dockerfile (1.94.0) 42387380669 red: Tools step, "rust-toolchain.toml channel '1.94.0' differs from the Dockerfile's rust image"
Node version check 42387361790 red: Tools step, the Node major check
a Rust and a JS test that each sleep 91 s 42387378279 red: rust-test (killed at 90 s) and node-test (timed out at 90 s) failed, and "CI time" failed on both ("took 90.006 s / 90.002 s: 90 s or more fails")
a part killed with kill -9 42388120434 red: only policy: "no result (crashed or killed)"²
report opens one issue 42387985053 (#31's head); 42388920768 (this PR's head a1fdcf5, gh through install-tool.sh) both times tier1 red (failing JS test) and report green: it opened #35, then #37, and each was closed as not planned with no comment

¹ node-test also failed in these two cases, each time in a test that deliberately re-checks the repository's own tree: repo-policy.test.mjs "passes on the repository's own tree" and render-readme.test.mjs "D2 … committed HTML is current". Those are existing guards of the same rule. Weakening them to make the proof look cleaner would be wrong.
² The first kill -9 attempt (42387359526) edited ci-run.sh unconditionally, so tests/ci-run.test.mjs (which runs that script) failed too. The second attempt kills only on the CI server.

Slow set

The throwaway branch added the Rust test slow_proof and tests/slow/proof.test.mjs, in PR #33 (closed, branch deleted).

  • PR run 42387406290 (fast) left both out: nextest "31 passed, 12 skipped (including 1 test via profile.ci.default-filter)", node 232.
  • Dispatch 42387414414 (full) ran both: nextest "32 passed, 11 skipped", node 233.

Docs-only

Throwaway PR #34 (AGENTS.md only; closed, branch deleted), run 42387409969: tier1 green, with policy, docs and actionlint passed. Every other part reads "skipped: only documentation changed (.github/scripts/changed-scope.sh)".

Drafts

This PR opened as a draft. Run 42384711204 skipped the job, so no tier1 result was reported, and marking the PR ready ran tier1. GitHub does not evaluate a skipped job's name: expression, so the skipped job shows the raw expression text rather than "tier1 (skipped on drafts)". Either way it is never a tier1 check.

No escape hatches

$ git grep -nE 'continue-on-error|\|\|[[:space:]]*(true|:)|set \+e' -- .github
$

(empty, exit 1)

Stable

5 dispatch runs of the final head a1fdcf5 ran at the same time, with the same result and counts:

Run Result Rust JS
42389489868 success 31 passed, 11 skipped 242 passed
42389493721 success 31 passed, 11 skipped 242 passed
42389497013 success 31 passed, 11 skipped 242 passed
42389500088 success 31 passed, 11 skipped 242 passed
42389503405 success 31 passed, 11 skipped 242 passed

No test was flaky. #31's head ee24bef gave the same result across 5 runs earlier (42388181296, 42388183881, 42388186222, 42388188773, 42388191300: 31/11 and 232).

Merge

Pending: after review and QA, and once #24 has merged, I send the orchestrator the maestro-init settings dry run and apply it only after they confirm. Then this PR merges through the queue, and the time from enqueue to merge is recorded here.

Replace the six CI jobs (changes, policy, rust, node, npm-audit, gate) with one job, tier1, on one
self-hosted server per run. .github/scripts/ci-run.sh starts every part at the same time through
ci-parts.sh; the job summary shows the parts table. CI runs on pull requests (fast mode, slow tests
left out), merge-queue groups, every 6 hours on main and on dispatch (full mode); pushes to main run
nothing. A red scheduled run opens one issue; the report job gets a pinned GitHub CLI, which the CI
images lack. rust-toolchain.toml pins Rust 1.94.1 and .config/nextest.toml fails a test at 90 s
with no retries.
The previous commit copied files from an older base and reverted #30's changes to repo-policy.sh,
its test and ENGINEERING.md. Restore them. The report job now installs gh through
.github/scripts/install-tool.sh with the same pin as maestro-nightly, instead of its own download.

@ice-zeus ice-zeus left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of a1fdcf5 against #25.

Blocking

None.

The diff matches the design in #25: one tier1 job plus report, the four triggers with no push, concurrency, the draft guard, Scope/Tools/parts/CI time/upload steps, the parts table and skip rules in ci-run.sh, the nextest ci/full profiles (90 s kill, no retries, slow_ filter on PRs), rust-toolchain.toml with its Dockerfile check, and .gitignore. No CARGO_*/RUSTUP_TOOLCHAIN/npm_config_cache per-run env. Untrusted values (BASE_SHA, event name) reach run: only through env:. tests/ci-run.test.mjs covers each branch of ci-run.sh (full/fast, docs-only, Rust-only skip, package skip, dispatch audit, failed dependency chain, stale docs, missing variable). The repo-policy.sh change is safe: the exit status of a process substitution is ignored. Size (~445 production lines) is explained in the body and stays under 800.

Notes

  1. ci.yml header and ENGINEERING.md say a draft's job "is named tier1 (skipped on drafts)", but the PR body found that GitHub shows the raw expression for a skipped job. A short "(GitHub may show the raw expression)" would keep the docs accurate. The behavior itself is correct: no tier1 result on drafts.
  2. On a docs-only PR the "CI time" step is skipped, so "Upload ci-timing" is skipped too, and the logs of policy/docs/actionlint are not kept. This matches the issue ("once step 5 ran"), but the PR body's "Part logs are kept in the ci-timing artifact" holds only when a test part ran.
  3. vendor rewrites web/runtime/vendor/ while policy (and other parts) may still read the working tree. It is harmless while the vendored output equals the committed files, and the 5 stable runs agree.

Existing

None found in the code this PR touches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI on one server: all parts in parallel, merge-queue and 6-hour runs, size picked by measurement

1 participant