Repository navigation
ci: split memory-all tests on PRs by develop's timings - #24207
Merged
Merged
Conversation
CircleCI splits tests by the timings of the branch's own last green run of the job. After a rebase that run can predate long tests added on develop, so TestExecuteStep_CannonKona (about 400 s) weighs nothing and lands on a full node, which then runs 10 to 12 minutes instead of about 7.5. Develop's memory-all jobs now save the timings CircleCI gave their split to a cache. On PR branches, go-binaries-for-sysgo, which both memory-all jobs already wait for, restores the latest copy once and persists it, so every node splits from the same file. shard-tests.sh passes it with --timings-file when present. No file means the split works as before. Develop, tags and the merge queue never restore it, so trusted runs read nothing a branch could have written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restored timings dir is persisted to the workspace, which go-tests attaches at the repo root and tests with `go list ./...`. A cache any branch can write could put a _test.go there. Name the dir _shard-timings, which Go skips, and keep only plain .json files after the restore. Bump the key to v2 for the new layout. Also skip the restore on proposal branches, which run post-merge like develop, and note that the restore keys follow acceptance_l1_forks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A restored cache can write ~/bin, which comes first in PATH on cimg/base, so the cleanup after the restore names root-owned binaries directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The orb's BASH_ENV runs ~/.local/bin/mise at the start of every bash step, and a restored cache can write that file. sh never reads BASH_ENV. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
go-binaries-for-sysgo holds the GitHub token context, so restoring a cache any branch can write there needed a hardened cleanup. Move the restore into memory-all-shard-timings, which has no context and runs no command after the restore: the persist glob keeps only the .json files. Project env vars still reach every job, which is why nothing runs after the restore. go-tests no longer gets the directory at all, since it does not require this job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
persist_to_workspace runs tar and gzip from the image PATH, which starts with ~/bin and ~/.local/bin. Make both root-owned before the restore. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A branch that edits its CI config can write any cache, so restoring one needed hardening. The project's API is public, so fetch the latest green develop run's test results instead and convert them to the split's format. An optional CIRCLE_TIMINGS_TOKEN keeps it working if the repo goes private. Any failure writes no file and the split works as today. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fixes: memory-all requires the timings job as terminal, so any failure there falls back to today's split. --retry-max-time stops a long Retry-After from stalling the fetch. v2 needs a personal token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mds1
enabled auto-merge
November 17, 2026 01:10
joshklop
approved these changes
Nov 17, 2026
joshklop
left a comment
Contributor
There was a problem hiding this comment.
LGTM. I ran the fetch step locally against the live API: 1 page, 893 raw items, 281 top-level tests per job, CannonKona at 406 s, done in 3 s. Summing the 7 names that exist in more than one package matches how shard-tests.sh deduplicates them, and the runner never retries a test, so retries can't inflate the sums. Wiring checks out too: the kona job gets the dependency through *memory_all_requires, shipped-memory-all-* has its own requires, and ci-gate is unchanged. The comments below are non-blocking.
mds1
added a commit
that referenced
this pull request
Nov 17, 2026
Follow-up to review on #24207: shard-tests.sh logs when the timings dir exists but has no file for the job, so a stale JOBS list shows up in the log. Comments point a future token at a restricted context and note that branches into proposal/* still fetch develop's timings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR memory-all splits used stale timings: CircleCI times the split from the branch's own last green run, which can predate slow tests added on develop since. After #23787 re-enabled
TestExecuteStep_CannonKona(about 400 s), it weighed about 0 on such branches and landed late on an already full node. That set the end of 12 of the 30 slowest PR pipelines since 11-13 (slowest node 603 to 709 s, typical 433 to 445 s, measured), and it happened again on 11-16 under the 4 x 16 layout from #24145. Modeled on 118 PR pipelines: PR p90 18.8 to 15.9 min.Change
memory-all-shard-timingsjob (no context,small.gen2). On PR branches it reads develop's latest green memory-all results from CircleCI's API, keeps top-level tests (a parent's time includes its subtests), and writes one timings file per memory-all job. Both memory-all jobs require it, so every node splits from the same file and each test still runs exactly once.shard-tests.shpasses that file tocircleci tests split --timings-filewhen it exists. Without it the split works exactly as today, and a test missing from the file keeps the existing 1m default.proposal/*skip the fetch and keep CircleCI's own timings.terminal, so even a failed or cancelled timings job never blocks it.CIRCLE_TIMINGS_TOKEN(a personal API token, the v2 API rejects project tokens) as a project env var.Checks
circleci config validate --nexton the merged config,test-decision-tree.sh25 passed.🤖 Generated with Claude Code