Skip to content

ci: split memory-all tests on PRs by develop's timings - #24207

Merged
mds1 merged 9 commits into
developfrom
mds/ci-shard-timings-2
Nov 17, 2026
Merged

mds1 merged 9 commits into
developfrom
mds/ci-shard-timings-2

Conversation

@mds1

@mds1 mds1 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

PR memory-all splits used stale timings: CircleCI times the split from the branch's own last green run, which can predate slow tests added on develop since. After #23787 re-enabled TestExecuteStep_CannonKona (about 400 s), it weighed about 0 on such branches and landed late on an already full node. That set the end of 12 of the 30 slowest PR pipelines since 11-13 (slowest node 603 to 709 s, typical 433 to 445 s, measured), and it happened again on 11-16 under the 4 x 16 layout from #24145. Modeled on 118 PR pipelines: PR p90 18.8 to 15.9 min.

Change

  • New memory-all-shard-timings job (no context, small.gen2). On PR branches it reads develop's latest green memory-all results from CircleCI's API, keeps top-level tests (a parent's time includes its subtests), and writes one timings file per memory-all job. Both memory-all jobs require it, so every node splits from the same file and each test still runs exactly once.
  • shard-tests.sh passes that file to circleci tests split --timings-file when it exists. Without it the split works exactly as today, and a test missing from the file keeps the existing 1m default.
  • No cache. The data comes from CircleCI's record of develop runs, which no branch can write. Develop, tags, the merge queue and proposal/* skip the fetch and keep CircleCI's own timings.
  • Any API failure (timeout, rate limit, outage) logs one line, writes no file, and the split runs as today. The fetch gives up within about 80 s at worst (no new call after 60 s, each call capped including retry waits). memory-all requires the job as terminal, so even a failed or cancelled timings job never blocks it.
  • The API needs no token while the project is public. If it goes private, set CIRCLE_TIMINGS_TOKEN (a personal API token, the v2 API rejects project tokens) as a project env var.

Checks

  • circleci config validate --next on the merged config, test-decision-tree.sh 25 passed.
  • Against live develop data: 281 top-level tests per job, the same count each memory-all job runs, with CannonKona at 406 s.
  • An earlier cache-based version of the same split was benched in 140318: CannonKona ran first, slowest node 426 s and 481 s against develop's 452 s and 446 s, 281 tests per job, none missing or doubled. It was dropped because a branch can write any cache.
  • Bench of this API version, 140329: the fetch took 4 s, all 4 nodes of both jobs used the same file, CannonKona ran first, 281 tests per job with none missing or doubled, all green. Slowest node 443 s (op-node) and 441 s (kona) against 452 s and 446 s on develop at the same base (140312, n=1). memory-all started no later than on develop. A bad host, unreachable address, 404 and 401 each exit 0 and write no file (local).

🤖 Generated with Claude Code

mds1 and others added 8 commits November 16, 2026 15:10
CircleCI splits tests by the timings of the branch's own last green run of
the job. After a rebase that run can predate long tests added on develop,
so TestExecuteStep_CannonKona (about 400 s) weighs nothing and lands on a
full node, which then runs 10 to 12 minutes instead of about 7.5.

Develop's memory-all jobs now save the timings CircleCI gave their split to
a cache. On PR branches, go-binaries-for-sysgo, which both memory-all jobs
already wait for, restores the latest copy once and persists it, so every
node splits from the same file. shard-tests.sh passes it with
--timings-file when present. No file means the split works as before.

Develop, tags and the merge queue never restore it, so trusted runs read
nothing a branch could have written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restored timings dir is persisted to the workspace, which go-tests
attaches at the repo root and tests with `go list ./...`. A cache any branch
can write could put a _test.go there. Name the dir _shard-timings, which Go
skips, and keep only plain .json files after the restore. Bump the key to v2
for the new layout. Also skip the restore on proposal branches, which run
post-merge like develop, and note that the restore keys follow
acceptance_l1_forks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A restored cache can write ~/bin, which comes first in PATH on cimg/base,
so the cleanup after the restore names root-owned binaries directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The orb's BASH_ENV runs ~/.local/bin/mise at the start of every bash step,
and a restored cache can write that file. sh never reads BASH_ENV.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
go-binaries-for-sysgo holds the GitHub token context, so restoring a cache
any branch can write there needed a hardened cleanup. Move the restore into
memory-all-shard-timings, which has no context and runs no command after the
restore: the persist glob keeps only the .json files. Project env vars still
reach every job, which is why nothing runs after the restore. go-tests no
longer gets the directory at all, since it does not require this job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
persist_to_workspace runs tar and gzip from the image PATH, which starts
with ~/bin and ~/.local/bin. Make both root-owned before the restore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A branch that edits its CI config can write any cache, so restoring one
needed hardening. The project's API is public, so fetch the latest green
develop run's test results instead and convert them to the split's format.
An optional CIRCLE_TIMINGS_TOKEN keeps it working if the repo goes private.
Any failure writes no file and the split works as today.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mds1
mds1 requested a review from a team as a code owner November 17, 2026 00:56
Review fixes: memory-all requires the timings job as terminal, so any
failure there falls back to today's split. --retry-max-time stops a long
Retry-After from stalling the fetch. v2 needs a personal token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mds1
mds1 enabled auto-merge November 17, 2026 01:10

@joshklop joshklop left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I ran the fetch step locally against the live API: 1 page, 893 raw items, 281 top-level tests per job, CannonKona at 406 s, done in 3 s. Summing the 7 names that exist in more than one package matches how shard-tests.sh deduplicates them, and the runner never retries a test, so retries can't inflate the sums. Wiring checks out too: the kona job gets the dependency through *memory_all_requires, shipped-memory-all-* has its own requires, and ci-gate is unchanged. The comments below are non-blocking.

Comment thread .circleci/continue/main.yml Resolved
Comment thread ops/scripts/shard-tests.sh Resolved
Comment thread .circleci/continue/main.yml Resolved
Comment thread .circleci/continue/main.yml Resolved
@mds1
mds1 added this pull request to the merge queue Nov 17, 2026
Merged via the queue into develop with commit d002119 Nov 17, 2026
115 of 116 checks passed
@mds1
mds1 deleted the mds/ci-shard-timings-2 branch November 17, 2026 02:12
mds1 added a commit that referenced this pull request Nov 17, 2026
Follow-up to review on #24207: shard-tests.sh logs when the timings dir
exists but has no file for the job, so a stale JOBS list shows up in the
log. Comments point a future token at a restricted context and note that
branches into proposal/* still fetch develop's timings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants