You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
RWX runs as an optional shadow of CircleCI's pull-request workloads. The goal is to compare the two providers on the same commits, then decide whether to migrate. CircleCI stays the only required CI during the evaluation.
The operator guide is docs/ai/rwx-shadow-ci.md. It lands with #23783.
End: four weeks after the start, and not before 300 PR pushes and 150 develop commits have run on both providers.
Decision: within one week after the end. Decider: to be named.
Extension: one, of at most two weeks, with a stated reason.
Two rules for the comparison
One clock, from the trigger. Time is wall time on GitHub's clock, from the push to a gate turning green. No duration that a provider reports about itself is used. Queueing, CircleCI's setup step and RWX's start-up all count.
Each provider is measured as it runs. If a provider runs fewer jobs for a push, through a cache hit, a narrower route or a check that it skips, its lower time and cost count in its favour. No job is left out to make the two sides match.
What we measure, and the bar for a switch
Element
Measure
Bar
Correctness
compare-ci.py on a weekly sample of 20 develop commits, for each workload.
No unexplained missing or differing test case. Each known difference is in the guide's list.
Speed
For each PR push: the time from the push to each required gate green, and to all four green. Start: the time GitHub records the push, taken as the creation time of the commit's first check suite. End: the creation time of the gate's first success status. p50 and p90 over every PR push in the window.
RWX's p50 from push to all four gates green is at least 25% lower, and its p90 is no higher.
Cost
List-price cost of everything each provider ran, for each PR push and for each month. RWX: vCPU-seconds times list price. CircleCI: credits times list price. The comp is ignored.
RWX costs no more than CircleCI.
Work done
For each push: the jobs or tasks that ran, and those that a cache hit or the route left out.
Reported. It explains the time and the cost. No bar of its own.
Reliability
Share of runs that fail for a reason other than the code: a runner or cache fault, a provider outage, a network failure.
RWX's share is no higher than CircleCI's.
Maintenance
Hours spent to keep the shadow aligned, and the number of CircleCI changes that needed a hand-made RWX change.
Reported. No bar.
Developer experience
At least three engineers who change CI use rwx run --target on real changes and report.
Reported. No bar.
Before the window starts
RWX-only RPC keys. RWX reads archive RPC keys of its own, not CircleCI's. The same step gives RWX a private OP Mainnet RPC URL for the L2 fork test. Without it RWX paces its requests to the public RPC, and its contracts gate carries about 3.5 minutes that CircleCI's does not.
Preconditions for a switch
These do not block the shadow. They block a decision to switch.
Cloud security's written conclusion on RWX task-cache isolation between branches. The shadow reuses builds across commits and branches, so this question decides whether a branch can influence a develop run.
A list price for both providers.
The required checks: RWX prefixes every status with RWX: , so it cannot post a ci/circleci: name. Any other app with permission to write statuses still can, because CircleCI's statuses come from a user account and cannot be pinned to an app. That needs its own issue.
Data collection
Owner: to be named. A summary goes on this issue each week.
Time: GitHub's API gives both ends of the clock: commits/<sha>/check-suites for the start, commits/<sha>/statuses for each gate.
Cost and work done:rwx results <run> --format json gives each task's duration, runner size and cache hit. CircleCI's API gives each job's duration and credits.
Test results:compare-ci.py needs CircleCI's JUnit, Go JSON and test-metadata files for each sampled commit. Nothing collects them yet. Build that in the first week, or download the sample by hand.
Outcomes
Switch: every bar is met and every precondition is done. The first step is a migration plan: which workloads move first, and how the required checks move.
Stay on CircleCI: remove .rwx/, ops/ci/ and the guide, and suspend the RWX GitHub App.
Extend: once, with the reason and with the result that would end the extension.
Starting point
The last push to #23783 (8d74357), measured by the two rules above. The third column is CircleCI on the last 14 develop commits, for the commits where it ran the workflow. Those are other commits, so the columns are not matched.
Gate
CircleCI, this push
RWX mirror, this push
CircleCI when it runs the workflow
ci-gate
13:35
10:03
median 14.0 minutes
required-contracts-ci
0:43, skipped
5:54
3 to 13 minutes
required-rust-ci
0:55, skipped
10:45
median 12.6 minutes
required-rust-e2e
0:43, skipped
11:51
median 12.5 minutes
All four
13:35
11:51
CircleCI's first status arrives 40 seconds after the push, RWX's 11 seconds after.
On this push CircleCI did not run three of the four workflows. Its routing runs the contract tests only when a contracts path changed, and Rust CI and Rust E2E only when a Rust path changed. Otherwise it runs a one-job workflow that reports the gate as passed: contracts-feature-tests-short, rust-ci-gate-short and rust-e2e-gate-skip. The PR changes no such path.
RWX uses the same routing, and adds one rule: a change under .rwx/ or ops/ci/ selects every workload, as a change under .circleci/ does on CircleCI. Every push of this PR changes those files, so RWX ran everything. Rule 2 counts that in CircleCI's favour here. On a PR that changes none of the shadow's own files, RWX skips the same workflows as CircleCI. No such run exists yet, because the definitions are only on this PR's branch.
Correction to this issue
"Every branch push to this repository starts three RWX runs" is out of date. A push starts one RWX run, which calls the 14 workloads.
Activity
teddyknox commentedon Nov 19, 2026
Experiment plan
Question. Does RWX run this repository's pull-request CI faster and at a lower cost than CircleCI, with the same test results and an acceptable risk?
CircleCI stays the only required CI for the whole window. Nothing below changes what gates a merge.
Window
developcommits have run on both providers.Two rules for the comparison
What we measure, and the bar for a switch
compare-ci.pyon a weekly sample of 20developcommits, for each workload.successstatus. p50 and p90 over every PR push in the window.rwx run --targeton real changes and report.Before the window starts
Preconditions for a switch
These do not block the shadow. They block a decision to switch.
developrun.RWX:, so it cannot post aci/circleci:name. Any other app with permission to write statuses still can, because CircleCI's statuses come from a user account and cannot be pinned to an app. That needs its own issue.Data collection
commits/<sha>/check-suitesfor the start,commits/<sha>/statusesfor each gate.rwx results <run> --format jsongives each task's duration, runner size and cache hit. CircleCI's API gives each job's duration and credits.compare-ci.pyneeds CircleCI's JUnit, Go JSON and test-metadata files for each sampled commit. Nothing collects them yet. Build that in the first week, or download the sample by hand.Outcomes
.rwx/,ops/ci/and the guide, and suspend the RWX GitHub App.Starting point
The last push to #23783 (8d74357), measured by the two rules above. The third column is CircleCI on the last 14
developcommits, for the commits where it ran the workflow. Those are other commits, so the columns are not matched.ci-gaterequired-contracts-cirequired-rust-cirequired-rust-e2eCircleCI's first status arrives 40 seconds after the push, RWX's 11 seconds after.
On this push CircleCI did not run three of the four workflows. Its routing runs the contract tests only when a contracts path changed, and Rust CI and Rust E2E only when a Rust path changed. Otherwise it runs a one-job workflow that reports the gate as passed:
contracts-feature-tests-short,rust-ci-gate-shortandrust-e2e-gate-skip. The PR changes no such path.RWX uses the same routing, and adds one rule: a change under
.rwx/orops/ci/selects every workload, as a change under.circleci/does on CircleCI. Every push of this PR changes those files, so RWX ran everything. Rule 2 counts that in CircleCI's favour here. On a PR that changes none of the shadow's own files, RWX skips the same workflows as CircleCI. No such run exists yet, because the definitions are only on this PR's branch.Correction to this issue
"Every branch push to this repository starts three RWX runs" is out of date. A push starts one RWX run, which calls the 14 workloads.