Skip to content

Tracker: RWX shadow CI evaluation #24044

Open
0 of 7 issues completed
Open

Description

@teddyknox

RWX runs as an optional shadow of CircleCI's pull-request workloads. The goal is to compare the two providers on the same commits, then decide whether to migrate. CircleCI stays the only required CI during the evaluation.

The operator guide is docs/ai/rwx-shadow-ci.md. It lands with #23783.

Pull requests

The first three change code that CircleCI runs. #23783 adds the RWX shadow and changes nothing that CircleCI runs.

Work

The sub-issues list the work. Two of them wait for the decision.

Standing work while the shadow runs

  • Review RWX spend regularly. Every branch push to this repository starts one RWX run, which calls the 14 workloads.
  • If the shadow is not worth its noise or its spend, use the off switch in the operator guide.

Activity

  1. self-assigned this
    on Nov 10, 2026
  2. teddyknox commented on Nov 19, 2026

    @teddyknox
    ContributorAuthor

    Experiment plan

    Question. Does RWX run this repository's pull-request CI faster and at a lower cost than CircleCI, with the same test results and an acceptable risk?

    CircleCI stays the only required CI for the whole window. Nothing below changes what gates a merge.

    Window

    Two rules for the comparison

    1. One clock, from the trigger. Time is wall time on GitHub's clock, from the push to a gate turning green. No duration that a provider reports about itself is used. Queueing, CircleCI's setup step and RWX's start-up all count.
    2. Each provider is measured as it runs. If a provider runs fewer jobs for a push, through a cache hit, a narrower route or a check that it skips, its lower time and cost count in its favour. No job is left out to make the two sides match.

    What we measure, and the bar for a switch

    Element Measure Bar
    Correctness compare-ci.py on a weekly sample of 20 develop commits, for each workload. No unexplained missing or differing test case. Each known difference is in the guide's list.
    Speed For each PR push: the time from the push to each required gate green, and to all four green. Start: the time GitHub records the push, taken as the creation time of the commit's first check suite. End: the creation time of the gate's first success status. p50 and p90 over every PR push in the window. RWX's p50 from push to all four gates green is at least 25% lower, and its p90 is no higher.
    Cost List-price cost of everything each provider ran, for each PR push and for each month. RWX: vCPU-seconds times list price. CircleCI: credits times list price. The comp is ignored. RWX costs no more than CircleCI.
    Work done For each push: the jobs or tasks that ran, and those that a cache hit or the route left out. Reported. It explains the time and the cost. No bar of its own.
    Reliability Share of runs that fail for a reason other than the code: a runner or cache fault, a provider outage, a network failure. RWX's share is no higher than CircleCI's.
    Maintenance Hours spent to keep the shadow aligned, and the number of CircleCI changes that needed a hand-made RWX change. Reported. No bar.
    Developer experience At least three engineers who change CI use rwx run --target on real changes and report. Reported. No bar.

    Before the window starts

    • RWX-only RPC keys. RWX reads archive RPC keys of its own, not CircleCI's. The same step gives RWX a private OP Mainnet RPC URL for the L2 fork test. Without it RWX paces its requests to the public RPC, and its contracts gate carries about 3.5 minutes that CircleCI's does not.

    Preconditions for a switch

    These do not block the shadow. They block a decision to switch.

    • Cloud security's written conclusion on RWX task-cache isolation between branches. The shadow reuses builds across commits and branches, so this question decides whether a branch can influence a develop run.
    • A list price for both providers.
    • The required checks: RWX prefixes every status with RWX: , so it cannot post a ci/circleci: name. Any other app with permission to write statuses still can, because CircleCI's statuses come from a user account and cannot be pinned to an app. That needs its own issue.

    Data collection

    • Owner: to be named. A summary goes on this issue each week.
    • Time: GitHub's API gives both ends of the clock: commits/<sha>/check-suites for the start, commits/<sha>/statuses for each gate.
    • Cost and work done: rwx results <run> --format json gives each task's duration, runner size and cache hit. CircleCI's API gives each job's duration and credits.
    • Test results: compare-ci.py needs CircleCI's JUnit, Go JSON and test-metadata files for each sampled commit. Nothing collects them yet. Build that in the first week, or download the sample by hand.

    Outcomes

    • Switch: every bar is met and every precondition is done. The first step is a migration plan: which workloads move first, and how the required checks move.
    • Stay on CircleCI: remove .rwx/, ops/ci/ and the guide, and suspend the RWX GitHub App.
    • Extend: once, with the reason and with the result that would end the extension.

    Starting point

    The last push to #23783 (8d74357), measured by the two rules above. The third column is CircleCI on the last 14 develop commits, for the commits where it ran the workflow. Those are other commits, so the columns are not matched.

    Gate CircleCI, this push RWX mirror, this push CircleCI when it runs the workflow
    ci-gate 13:35 10:03 median 14.0 minutes
    required-contracts-ci 0:43, skipped 5:54 3 to 13 minutes
    required-rust-ci 0:55, skipped 10:45 median 12.6 minutes
    required-rust-e2e 0:43, skipped 11:51 median 12.5 minutes
    All four 13:35 11:51

    CircleCI's first status arrives 40 seconds after the push, RWX's 11 seconds after.

    On this push CircleCI did not run three of the four workflows. Its routing runs the contract tests only when a contracts path changed, and Rust CI and Rust E2E only when a Rust path changed. Otherwise it runs a one-job workflow that reports the gate as passed: contracts-feature-tests-short, rust-ci-gate-short and rust-e2e-gate-skip. The PR changes no such path.

    RWX uses the same routing, and adds one rule: a change under .rwx/ or ops/ci/ selects every workload, as a change under .circleci/ does on CircleCI. Every push of this PR changes those files, so RWX ran everything. Rule 2 counts that in CircleCI's favour here. On a PR that changes none of the shadow's own files, RWX skips the same workflows as CircleCI. No such run exists yet, because the definitions are only on this PR's branch.

    Correction to this issue

    "Every branch push to this repository starts three RWX runs" is out of date. A push starts one RWX run, which calls the 14 workloads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

M-ciMeta: ci related workM-trackingMeta: tracking issue

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions