Skip to content

ci: add an optional RWX shadow that leaves CircleCI unchanged - #23783

Open
teddyknox wants to merge 55 commits into
developfrom
claude/rwx-additive-shadow
Open

teddyknox wants to merge 55 commits into
developfrom
claude/rwx-additive-shadow

Conversation

@teddyknox

@teddyknox teddyknox commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Adds RWX as an optional shadow of CircleCI's PR workloads, so the two providers can be compared on the same commits before any decision to migrate.

It changes nothing CircleCI runs, none of its checks is required, and a one-line change turns it off.

The PR adds 15,483 lines and removes none. It is a complete second CI, but it is purely additive. Review it by its boundaries, which the sections below point to. The changes that touch code CircleCI runs are separate PRs: #23772, #23775 and #23780 are merged, and #23776 is open. This PR does not depend on #23776.

Why it is safe to approve

  1. CircleCI is unchanged. git diff origin/develop...HEAD -- .circleci .github justfile justfiles op-acceptance-tests rust packages op-deployer is empty. Outside .rwx/ and ops/ci/, the PR adds the operator guide docs/ai/rwx-shadow-ci.md, its index line in AGENTS.md, and seven lines in ops/book/src/ci/pr-authorization.md for people who authorize fork PRs. It does not change CODEOWNERS: .rwx/ and ops/ci/ stay with the default owners (monorepo-reviewers), like .circleci/, at cloud-security's request.
  2. Nothing is required. No ruleset names an RWX check. A red RWX run blocks no merge.
  3. It has an off switch. Set the if: of the on.github.push trigger in .rwx/pr-gates.yml to false and merge. For an immediate stop, a GitHub org owner suspends the RWX GitHub App or removes this repository from it. Deleting .rwx/, ops/ci/, the guide and its index line removes it completely.
  4. It cannot drift silently. RWX selects workloads by running CircleCI's own routing scripts read-only, so both providers select the same workloads. circle-alignment.py fails an RWX task when the requires: list of one of CircleCI's four gates stops matching ops/ci/runtime/pr-gates.json. On develop, the route task fails when CircleCI's policy no longer enables one of the four shadowed workflows, for example after a rename of its flag.

What happens after merge

  • Every branch push to this repository starts one RWX run. .rwx/pr-gates.yml calls the 14 shadowed workloads and mirrors CircleCI's four gates: ci-gate, required-contracts-ci, required-rust-ci and required-rust-e2e. The run posts 19 optional checks. Tag pushes start no run. A new push to a branch cancels that branch's older run; develop keeps every run.
  • An authorized fork PR starts an RWX run. A fork PR cannot push here. After /ci authorize, bailiff pushes the commit to an external-fork/<hash> branch, and that push starts an RWX run that can read the test vault, next to CircleCI's run. An authorizer must therefore review a fork PR's changes under .rwx/ and ops/ci/ as well as .circleci/; ops/book/src/ci/pr-authorization.md now says so.
  • This costs money. Every internal PR push gets a full shadow run, of about 3,500 CPU-minutes when every workload runs and the builds are cached. Teddy holds the cost figures privately.
  • Credentials. Branches with up to two slashes can read the test-only vault optimism-go-tests-rpc-shadow (archive RPC URLs for fork tests). Only develop can write the cache vault optimism-op-reth-shadow. RWX holds no production publishing credentials.

Speed and cost

The shadow is tuned for RWX: runner sizes, cache keys and task layout come from measurements on RWX, not from CircleCI's resource classes. The commands and tests are CircleCI's. Each column is one full run of all 14 workloads on a new commit.

Before tuning Normal push Build inputs changed
Whole run 38:38 16:11 24:51
ci-gate mirror 16:00 9:06 14:57
required-rust-ci mirror 39:09 9:46 24:51
required-rust-e2e mirror 13:46 11:14 12:55
required-contracts-ci mirror 15:59 16:11, now about 5:00 16:15
CPU-minutes 4,715 3,501 5,320
  • Before tuning: the run at ac8137c. On that commit CircleCI's main workflow, which ci-gate gates, took 14:53; its other three gates took their skip paths.
  • Normal push: the run at 82947d4, a commit that changes no build input. Every dependency build and Rust baseline is a cache hit.
  • Contracts mirror, now about 5:00: the three full runs predate the contracts tuning. On a commit that changes no compile input, standalone runs of the three contract workloads now take 4:59 (L2 fork), 4:47 (standard suites) and 3:52 (upgrade variants), with the same 27,268 contract tests.
  • Build inputs changed: the run at 0dfc407, in which every build and tool task ran again. It uses more compute than before, because the Rust baselines now have 16 CPUs.

What changed:

  1. A build is keyed on what it is built from. op-reth, kona, the SP1 executor, the prestate, the contracts, the Go dependency and the compiler baselines build from the commit that last changed their inputs, so later commits and the other workloads of a run reuse them. Before, most were rebuilt for every push, and op-reth three times in one run.
  2. Runner sizes are measured. For example clippy takes 9:12 on 4 CPUs and 2:50 on 16, and the L2 fork compile is one solc process, so it went from 16 CPUs to 4.
  3. The E2E release build is split as on CircleCI, so the E2E suites wait only for the four binaries they run.
  4. A CI-only change rebuilds only what it touches. A build's inputs name the scripts that it runs, and a test follows the references from each build's entry script.
  5. The contract compiles are keyed on what they read, and 13 of them became one. The five standard suites and the eight upgrade variants compiled the same thing separately: the compiler reads the Foundry profile, and only the tests read the feature flags and the fork variables. The Go module download is reused in the same way.

The guide's "Speed and cost" section has the rules and the measurements.

Where to look

Area Added lines Suggested review
.rwx/*.yml, .rwx/packages 6,215 Read the on: block of pr-gates.yml and each vault use; skim the task bodies
ops/ci/runtime 3,890 Spot-check: scripts and data that run only inside RWX tasks
ops/ci/tests, ops/ci/migration/tests 4,615 Skim: unit tests for those scripts
ops/ci/migration 440 Skim: the gate alignment check and the CI result comparer
Guide, AGENTS.md, authorization page 323 Read fully

A task that mirrors a CircleCI job runs that job's command, and its exit status is the verdict. The guide lists each place where a task departs from that, under "Known differences from CircleCI".

Read closely: the on: block of pr-gates.yml, each vault use in .rwx/, the routing adapter ops/ci/runtime/rwx-metadata.sh, the gate policy ops/ci/runtime/pr-gates.json with ops/ci/migration/circle-alignment.py, and the guide.

Evidence

  • A full hosted run at 82947d4 passed. In the run all 451 tasks passed at the first attempt, and all four gate mirrors are green. After a merge of develop, the run at 8ecddfc79d also passed, with one retry of the op-reth integration task for the teardown crash below. The alignment check changed with that merge: develop's ci-gate now requires a new ci-gate-proposals gate, and the check reads that gate as the jobs it requires. The contracts tuning came after. Standalone runs of the seven workloads that it touches passed, and the run that the push of this head started is the first full run on it.
  • Two known flakes appeared in the tuning runs. In three runs, one op-reth integration test passed and then crashed with SIGSEGV during teardown (flake: reth-optimism-node integration test peers_negotiate_eth_69 SIGSEGVs on teardown #20973; rust: point the teardown-SIGSEGV retry at the renamed p2p test #24263 repairs its retry). One Cannon environment build timed out on a PyPI request. Neither comes from this PR.
  • The tuning lost no test. The run before it and the run after it executed the same 49,194 tests by RWX's count, workload by workload.
  • The simplification lost no test. The previous head of this PR was twice the size (30,624 added lines). The hosted run at that head, 7db6600, and the run above executed the same number of tests: 11,982 Go tests, 893 acceptance tests for each of the two variants, 2,881 Cannon tests, 4,427 Rust unit tests and 93 op-reth integration tests.
  • Unit tests. The 228 unit tests pass locally, apart from the live-toolchain fixtures, which skip on macOS and run inside their RWX tasks.
  • Static checks. rwx lint --dir .rwx reports no errors. circle-alignment.py passes against this branch's CircleCI config.
  • Reviews. The earlier head had a CI-config review and a code review. The current head implements a full audit of that head for size and correctness. No review agent has run on the current head yet.
  • George Knee's review at ac8137c asks for four things before merge. This head does the second, and the route check on develop from his non-blocking list. Tracker: RWX shadow CI evaluation #24044 still says that a push starts three RWX runs; it starts one.
    1. RWX-only archive RPC keys, separate from CircleCI's: open, outside the diff.
    2. Correct the fork PR statement and tell authorizers what to review: done in this head.
    3. Pin the required ci/circleci: checks of enforce-ci-checks-develop to the CircleCI app: open, a ruleset change by a repository admin.
    4. The experiment's duration and decision criteria in Tracker: RWX shadow CI evaluation #24044: open.
  • Dismissed findings. Wiz SAST raised 15 findings on this PR. Seven were on code that the PR no longer contains. The other eight are dismissed as false positives, each with its reason in the review thread. Five are "Command Injection via Subprocess" (go-compiled-tests.py twice, rust-cannon-report.py, go-artifacts.py, sp1-guest.py): each call passes an argument list with no shell, and its arguments are fixed commands or values from the repository itself. Three are "Unsafe XML Parsing" (ci-test-results.py, op-reth-report.py, rust-workspace-report.py): each parses JUnit reports from this repository's own CI runs.

Not tested on RWX: a run in which the route does not select a workload. Every run of this branch selects all of them, because the branch changes .rwx/ and ops/ci/; the unit tests cover that path. No develop push has carried these definitions, so the develop path (cache writes, no cancellation) runs for the first time after merge.

Known limits, please confirm

  • Squash-merge only (the repository default). The first commit imports ci: cover CircleCI PR workloads with native RWX shadows #23151 verbatim and still has its unconditional push triggers. Never push that commit as a branch head.
  • Push no unreviewed code to a branch of this repository: RWX runs it with that branch's vault access. /ci authorize is the reviewed way to run a fork PR's head.
  • Branches with more than two slashes get no RPC vault access. In RWX branch patterns * matches one path segment and ** is not supported, so the vault grants refs/heads/*, refs/heads/*/* and refs/heads/*/*/*.
  • The routing adapter inherits CircleCI's fixed /tmp/pipeline-parameters.json. RWX tasks are isolated; only two concurrent local runs of test_rwx_metadata.py would collide. Accepted without a code change.
  • The L2 fork tests use the public OP Mainnet RPC. No RWX vault holds CircleCI's private L2 RPC URL, so RWX sends its requests through a local pacer at two per second. Without it the public RPC answers HTTP 429 and the tests fail. A private URL in a vault removes the pacer.
  • A run from the CLI cancels the branch's run in progress. rwx run .rwx/pr-gates.yml enters the same concurrency pool as a push, also with --target.
  • Package versions. rwx lint warns that newer mise/install and git/clone package versions exist; this PR keeps the versions that its runs used.
  • RWX differs from CircleCI in a few documented ways (for example acceptance sharding and uncached Go test runs); the guide lists them. The flaky-test report and the selector upload are not shadowed.
  • After a change to a compile input, the contracts gate mirror takes 16 minutes. The L2 fork task then compiles for nine minutes in one solc process, and runs four minutes of paced tests. After any other change the compile is reused, and the gate takes about five minutes.
  • RWX starts no runner above 16 CPUs for this organization, although its documentation lists larger ones.
  • RWX has no mirror of ci-gate-proposals, the gate that release branches require. The ci-gate mirror covers the same jobs.
  • No tool cache has been used. Only develop can write one, so every time above is without tool caches.
  • Nothing collects CircleCI's results yet. compare-ci.py compares RWX results with CircleCI's normal JUnit, Go JSON and test-metadata output, from files that an operator downloads.

This PR replaces #23151, which rerouted about 66 CircleCI jobs through evidence-collecting wrappers and edited justfiles and product code. Here RWX calls the unchanged recipes and scripts that CircleCI calls.

Tracked in #24044.

🤖 Generated with Claude Code

teddyknox and others added 4 commits October 29, 2026 13:37
Baseline for review: later commits on this branch remove what only
CircleCI used and decouple RWX from the pilot's CircleCI and
product-code hooks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Make the imported RWX workloads work against develop's unchanged
justfiles, scripts and product code, and remove what only CircleCI used:

- Route RWX through CircleCI's own routing data and policy, read-only,
  instead of a forked copy; a change under .rwx/ or ops/ci/ selects
  every workload.
- Restrict every GitHub push trigger to develop and opt-in rwx/*
  branches, and remove every reference to the pilot branch.
- Replace the #23151 hooks with the unchanged recipes or RWX's own
  wrappers: acceptance sharding through a circleci tests split shim,
  the deployer CLI test's own build, cargo-hack's existing recipes,
  nextest extraction at the compile-time path, and exit-status verdicts
  for NUT pre-fork, NUT provenance, Kontrol and the interop diff.
- Remove the forked routing scripts, the CircleCI parity comparers, the
  CircleCI adapter tests, orphaned contract scripts, the coverage replay
  option and the flaky-report shadow, which needs a CircleCI API token.
- Fix verdicts that reported false failures: all-skipped contract
  selections, compile_fail doctests and should_panic tests, Go 1.24+
  build events, and Semgrep findings hidden by --json.
- Run the shared ops/ci unit tests in pr-gates.yml, and fail when no RWX
  task runs a test file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add an operator guide for the optional RWX shadow (when it runs, how to
turn it off, routing, gates, credentials and how to compare it with
CircleCI), index it in AGENTS.md, and give cloud-security review of
.rwx/ and ops/ci/ as #23151 did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CircleCI's routing policy writes only the workflows it enables. RWX
route tasks and gate aggregates read c-run_main, c-run_rust_ci,
c-run_rust_e2e_ci and c-run_contracts_feature_tests as booleans, so an
opt-in rwx/* push without contract, Rust or CI changes failed its gates.
The adapter now writes false for each flag the policy leaves out.

Also: correct the pilot.yml header, say which gates circle-alignment.py
checks, and warn against pushing unreviewed commits to rwx/* branches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wiz-0f98cca50a

wiz-0f98cca50a Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities -
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations -
SAST Finding SAST Findings 5 Low
Software Management Finding Software Management Findings -
Total 5 Low

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.

Comment thread ops/ci/migration/ci-test-results.py Resolved
Comment thread ops/ci/runtime/contract-upgrades.py Resolved
Comment thread ops/ci/runtime/op-reth-report.py Resolved
Comment thread ops/ci/runtime/rust-e2e.py Resolved
Comment thread ops/ci/runtime/rust-workspace-report.py Resolved
teddyknox and others added 2 commits November 6, 2026 16:04
Port what #23151 mirrored after this branch's import, adapted to the
unchanged recipes this branch calls:

- Acceptance runs one go test per test (#23974): the shim accepts the
  deduplicated name list that shard-tests.sh now passes, and the report
  checks one JSON and one JUnit per assigned test.
- Contract suites run the Amsterdam and default-EVM passes that
  `just test` runs (#23788); INTEROP replaces OPTIMISM_PORTAL_INTEROP
  (#23779); the ZK_DISPUTE_GAME variants are retired (#23868); the
  coverage shadow is retired with CircleCI's coverage job (#23879).
- The upgrade trace rerun runs without JUNIT_TEST_PATH (#23930). When
  test-upgrade's own retry replaces its JUnit with the rerun's cases,
  the verdict accepts that subset instead of reporting a false failure.
- NUT pre-fork accepts and skips <fork>_multichain states (#23897).
- shell-check runs develop's find | xargs shellcheck with the pinned
  ShellCheck (#23905); the vendored orb script and its license go.
- From #23151: the archive RPC proxy keeps up to eight fork reads in
  flight, RWX tool installs retry downloads, and the live forge-version
  fixture covers the forge package test (#23643).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread ops/ci/migration/selector-registry.py Resolved
Comment thread ops/ci/migration/selector-registry.py Resolved
Comment thread ops/ci/migration/selector-registry.py Resolved
Comment thread ops/ci/runtime/go-compiled-tests.py Resolved
Comment thread ops/ci/runtime/go-compiled-tests.py Resolved
Comment thread ops/ci/runtime/rust-cannon-report.py Resolved
Comment thread ops/ci/runtime/rust-e2e.py Resolved
teddyknox and others added 4 commits November 9, 2026 12:16
A hosted RWX run of pr-gates.yml at 1b4923d failed in three tasks.

- cannon-offline: Cannon writes rust/kona/out.bin.gz, and git does not
  ignore that file (#23772 adds the ignore line). The Rust target cache
  commit saw it as a source change. The wrapper now moves the final
  state into its report, and the cache error names the changed files.
- contracts-gate-receipt: the five heavy-fuzz-modified verdicts were
  skipped because the commit changes no contract test files, and the
  receipt counts a skip as a failure. Circle's job halts with success on
  an empty test list. The verdicts now always run, and restore accepts
  an empty preparation after it proves that the selection is still
  empty. The unused eligible value is removed.
- main-helper-tests: the l2-chains-sync live fixture wrote a forge
  version file by hand, but the check now runs go test (#23643). The
  fixture installs the forge package like the other main-check fixtures.

The contract verdict's test-results path also matches the per-EVM-pass
JUnit reports again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CircleCI runs nothing from ops/ci, so the runtime scripts' CircleCI
provider mode was dead code kept from the pilot. RWX behavior is now the
only behavior: the --provider choices, the CI_*_PROVIDER switches and the
CIRCLE_SHA1/CIRCLE_BRANCH/CIRCLE_NODE_* fallbacks are gone, and .rwx tasks
no longer pass them. Kept: the read-only use of CircleCI's routing
scripts, the acceptance `circleci tests split` shim, and CIRCLE_BRANCH
where develop's code reads it. Cached tasks miss their cache once.

The Cannon offline job now removes stale guest state before the Rust
source snapshot, so a restored state file cannot fail the cache commit.

circle-alignment.py now also fails when CircleCI's required-rust-e2e
stops requiring exactly the release build and the circle_job of each RWX
E2E job, so that list is no longer kept in step by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RWX ran only on develop and on opt-in rwx/* branches, so it never ran
next to CircleCI on a PR. Every branch push to this repository now starts
the pr-gates, rust-e2e and selector-upload runs, as CircleCI does; tag
pushes start none. Fork PRs cannot push here, so they still start no RWX
run. Routing still reuses CircleCI's scripts, so RWX selects the same
workloads.

A per-definition, per-branch concurrency pool cancels the older run when
a branch gets a new push. develop keeps every run.

Branches other than develop need a refs/heads/* grant on the test-only
optimism-go-tests-rpc-shadow vault before their RPC tasks pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RWX branch patterns match one path segment per `*` and do not support
`**`, so the test RPC vault grants one, two and three segment patterns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
teddyknox and others added 3 commits November 9, 2026 17:46
CircleCI now splits the release workspace build across rust-e2e-binaries
and rust-workspace-release (#23997), and required-rust-e2e requires both.
RWX still builds the whole workspace in one task, which covers both jobs,
so the alignment check now maps that task to the two CircleCI jobs.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…times

The parallel-slots test required arrivals at the test upstream to be at
least 45 ms apart, against a 50 ms paced start. An arrival adds connection
and thread latency that differs per request, so the test failed about one
run in five on a loaded machine. It now records the start times that the
transport schedules, which the transport owns, and still fails when the
pacing interval is halved.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
falcorocks
falcorocks previously approved these changes Nov 10, 2026
cloud-security asked not to own `.rwx/` and `ops/ci/` while RWX is an
optional shadow: every change to the shadow would otherwise need their
approval. The paths fall back to monorepo-reviewers, like `.circleci/`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
teddyknox and others added 7 commits November 18, 2026 12:02
- The guide said that RWX calls the unchanged recipes and listed seven
  differences. List the ones that exist: the jobs CircleCI halts with
  `check-changed`, the base branch, compiled Go and E2E suites, nextest
  archives, acceptance sharding by position, the L2 fork RPC, the SP1 build
  paths, and the two jobs that are not shadowed.
- Say that the two providers select the same workloads only for a PR into
  develop.
- Describe the receipt and status tasks, the comparer's new usage, and the
  task shape to follow for a new check.
- Remove the selector-upload, go-rollup and circle-gates.json references.

Delete rwx-source-archive.sh, whose two callers are gone.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The RWX shadow forks the public https://mainnet.optimism.io, because no RWX
vault holds CircleCI's private L2 RPC URL. In a hosted run forge sent about
325 requests in under a minute. The RPC then answered HTTP 429 for longer than
forge's ten retries, and the tests failed in setUp.

ops/ci/runtime/rpc-pacer.py is a local endpoint that forwards each request and
starts at most two per second. An earlier hosted run made the 420 requests of
these tests at that rate without one HTTP 429 and without one retry, so the
pacer does not retry: forge receives each upstream answer unchanged and owns
the retries.

With the rate bounded, the task runs CircleCI's recipe without the
`--threads 1 --compute-units-per-second 100` arguments. The second argument
only lengthens forge's backoff after a rate limit answer.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
36 `use` lists in the Go and Rust definitions named the checkout after a task
that itself uses the checkout, for example `use: [tools, code]`. RWX printed a
warning for each such task in every run.

The listed order had no effect. A probe on RWX showed that
`use: [seed, dependent, checkout]` produces the same files as
`use: [seed, checkout, dependent]`: RWX applies a task above the tasks it uses.
Each list now states that order, so the resolved definitions differ only in the
order of these lists.

The Go compile tasks keep their seed before the head checkout. That order does
have an effect: the seed holds an older checkout, and a later entry replaces
the files of an earlier one.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
RWX showed no tests for the contract suites and the upgrade tests, although
each task wrote a valid JUnit report (1433 test cases for one upgrade variant).
The two `test-results` paths named the report directory with a
`${{ params.* }}` expression. RWX does not expand an expression in that path.
It finds no file, and it reports no tests without a warning. A probe on RWX
with three packages that differ only in this path showed 0 tests for the
expression and 2 for a literal path and for a directory glob.

Each of these tasks has the report directory of one variant only, so the paths
now use a glob for the directory. The test that keeps expressions out of the
compile packages' output filters, for the same RWX behaviour, now also covers
every `test-results` path.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A branch push started two RWX runs: the coordinator, which calls 13 workload
definitions, and `.rwx/rust-e2e.yml`, which had its own push trigger, its own
concurrency pool and its own check. The coordinator now calls Rust E2E as a
14th workload, `native-rust-e2e`. A push starts one run, and one trigger turns
the shadow off.

The check keeps its name, `optimism-rust-e2e-shadow`, and is now a custom check
of the coordinator on the result of that call. The definition keeps its CLI and
cache-rebuild triggers.

`rwx lint` accepts a status check that names no task, and a run from the CLI
reports no checks. A test now requires one check for each workload call, each
naming a task of the coordinator.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Rust E2E was the one gate without a receipt: its check was the result of the
run. Its 14 test tasks share one `if:` condition, so a wrong route value would
skip all of them and leave the check green with no test run.

The gate now has the shape of the other three. `pr-gates.json` names its
prerequisites, `.rwx/rust-e2e.yml` has a receipt that reads the release build
and every test task, and the coordinator has a status task with the check
`optimism-rust-e2e-gate-shadow`. All four gates write the same record.

CircleCI splits the release build across two jobs, and RWX builds it in one
task. One policy row names both jobs: the first in `name`, the second in a new
`also` field, so a task still belongs to one row. The alignment check reads all
four gates from the policy; its Rust E2E special case and the unread
`circle_job` field are gone.

The release build is now a gated task, so it runs in every run, as every other
gated task does. It already ran for each new commit. A second run of the same
commit now builds again, about six minutes on 16 CPUs.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The coordinator's concurrency pool holds one run for each branch and cancels
the older one. The guide said so for a new push only. A run from the CLI enters
the same pool: a run of two tasks with `--target` cancelled a full run of this
branch at 680 of 700 tasks.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Comment thread ops/ci/runtime/sp1-guest.py Resolved
teddyknox and others added 8 commits November 18, 2026 15:55
…e-shadow

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
CircleCI's rust-tests job runs `just test`. On develop that recipe gained a
fourth part, `test-custom-embeds` (#24147), so CircleCI runs it at once. RWX
runs the parts of `just test` by name, and would have skipped the new one
without any failure.

The tests-compiler job now runs `just test-custom-embeds` too. Its unit test
reads the parts of `just test` from the justfile and fails when RWX does not
name one of them.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The acceptance, NUT pre-fork, Kontrol, Cannon and Rust E2E workloads built
op-reth, kona, the prestate, the contracts and the Go dependency from HEAD,
so each push rebuilt them even when nothing they are built from had changed.
In one run op-reth was built three times and kona twice.

Every workload now builds a dependency from the commit that last changed
its build inputs, as the Go test workload already did. The shared
dependency-source package holds that checkout, and each caller passes the
same tool tasks, so the build has one cache key for all workloads and for
later commits.

`.rwx` is no longer a build input: RWX's own cache key covers a task's
definition, so a changed definition rebuilds from the same commit and an
unrelated one rebuilds nothing.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The runner sizes copied CircleCI's resource classes. Measured on RWX:

- clippy: 9:12 on 4 CPUs, 2:50 on 16
- udeps: 9:16 on 4 CPUs, 2:43 on 16
- workspace build: 9:57 on 8 CPUs, 6:14 on 16
- slowest feature baseline: 32:12 on 4 CPUs, 17:52 on 16

These four set the time of the Rust CI gate, so they move to 16 CPUs, as do
the Cannon build and offline run and the op-reth, kona and SP1 executor
release builds. docs and no-std stay where they are: they are not on the
critical path, and no-std was no faster on 16 CPUs than on 8.

The L2 fork compile spends eight of its nine minutes in one solc process,
so it moves from 16 CPUs to 4.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Kontrol workload passed its own timeout to the Go tool package, so its
tool task, and with it its contracts build, had a cache key of its own.
The package now has one fixed timeout.

test_rwx_local_packages.py fails when a workload builds a dependency from
another checkout or with other tool tasks than the rest.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…lease build

A task passes on every layer that it uses. The dependency builds took their
checkout from a `use` layer, so each task that used a build also received
the build's older checkout: the acceptance tasks then ran on the wrong
commit and failed. Each build package now checks out its commit itself.
The files that a task writes stay out of its filtered output, so a
consumer gets the build outputs only. This removes the dependency-source
package and its clone tasks.

The Go compile tasks list the tool task before the compiler seed, which
now uses it. The L2 fork compile needs 32 GB: one solc process was killed
at 16 GB.

CircleCI splits the release build that the Rust E2E jobs need, so that
they wait only for the four binaries they run. RWX had merged the two
jobs. `binaries` now takes 4:18 against 6:16 for the whole workspace, and
`release` builds the rest from its output on a smaller runner. Each
CircleCI job has its own policy row again, so the `also` field is gone.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A task passes on every layer that it uses, and the tool tasks carry the
checkout of the run. A dependency build that used them therefore had the
run's commit in its cache key, and found a `.git` directory where it
writes its own checkout.

Each build package now has an input filter that leaves it `mise.toml` and
its own outputs. The cache key then holds the tools, the command and the
planned commit only.

The Go compiler seeds follow the same contract. Before, a compile task
inherited the seed's older checkout under the head checkout, so a file
that the head commit had deleted was still on disk. Now the preparation
task and each seed check the planned commit out themselves and pass on
only what they built.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@geoknee

geoknee commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Thanks for this @teddyknox !

Claude and I gave it a grilling, results are here https://claude.ai/artifact/Cc6RoHP3yQNL1kriX3byUy?sk=lONfwhS4K7eLxNEFk1FOTw.

Happy to approve when we address the blockers in that doc.

teddyknox and others added 13 commits November 19, 2026 09:22
A build's inputs included all of ops/ci/runtime, so a change to any CI
script, even a formatting change, rebuilt op-reth, kona, the prestate, the
contracts and every Rust baseline. The scopes now name the scripts that a
build runs. test_go_dependency_source.py follows the references from each
build's entry script and fails when one is not listed. RWX's own cache key
covers the tool scripts through the tool tasks.

The acceptance task excluded a directory that holds one tracked file.
With a build from an older commit, go-artifacts.py read that as a modified
build input. The task now excludes no tracked file, and a test checks it.

rust-e2e-release.sh has the here-string spacing that shfmt-check expects.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The Go tool package had no input filter, unlike the Rust and Foundry ones.
Its task had the whole checkout in its cache key, so it ran again for each
commit, and each task built on it got a new cache key: the Go dependency,
the contracts, the prestate and the Go compiler seeds were rebuilt for
every commit. The task now sees mise.toml and its script only, and a test
checks that every tool package has such a filter.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
One cargo-hack partition failed once on the 8 GB runner with ENOMEM while
sccache opened a file, and passed when retried. The ten tasks run for
about a minute each, so the larger runner costs about a cent per run.

The guide states why each tool package has an input filter.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Eight tool tasks that workloads define themselves had no input filter, so
each ran again for every commit. The one that installs geth for the
acceptance tests takes two minutes, and all eight acceptance shards wait
for it. Each now sees mise.toml and the script it runs only.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e-shadow

Conflict in AGENTS.md: keep develop's new description of the CI config
review guide, and this branch's line for the RWX shadow guide.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
On develop, `ci-gate` now requires `ci-gate-proposals`, the gate of the
release branches, which requires five jobs that `ci-gate` named itself
before. The alignment check reported those five as RWX-only.

The check now reads a required job of the gate's own kind as that job's
`requires:` list. RWX's policy for `ci-gate` is unchanged: it names the
same jobs that the gate waits for. RWX has no separate mirror of
`ci-gate-proposals`.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The contracts gate mirror took 16 minutes on every push: the L2 fork task
compiled for nine minutes for each commit. The compile tasks had `.git`
and the whole checkout among their inputs, and they waited for a Go module
download that also ran for each commit.

- The Go module set is a dependency build like op-reth or kona: it builds
  from the commit that last changed go.mod or go.sum, and seven workloads
  share it.
- The contract compiles have an input filter that excludes `.git` and the
  trees that a compile cannot read.
- The five standard suites and the eight upgrade variants share one
  build. They compiled the same thing 13 times: the compiler reads the
  Foundry profile, and only the tests read the feature flags and the fork
  variables. The L2 fork compile uses another profile and stays apart.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The compiles excluded `.git` and the trees that they cannot read. A probe
showed that two commits then gave the task the same 6,667 file hashes and
modes, and still a different cache key: RWX keys a task on file contents
only for a list of files to include. Each compile now names the contracts,
the Go module, the Go trees of the FFI helper and its scripts. A test
follows the helper's imports and fails when a package is outside the list.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The guide said that fork PRs never start an RWX run. They do: after
`/ci authorize`, bailiff pushes the commit to an `external-fork/<hash>`
branch of this repository, and that push starts an RWX run that can read
the test vault. The guide now says so, and the authorization page tells an
authorizer to review a fork PR's changes under `.rwx/` and `ops/ci/` too.

Found in George Knee's review of #23783.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…bled

The routing adapter writes false for a flag that CircleCI's policy does
not write. After a rename of a flag in the policy, every branch would read
"not selected", each receipt would pass with its tasks skipped, and RWX
would be green without a test. On develop the policy enables every
workflow, so the route task now fails there when one of the four is not
enabled.

Found in George Knee's review of #23783.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants