Skip to content

ci: cover CircleCI PR workloads with native RWX shadows - #23151

Closed
teddyknox wants to merge 180 commits into
developfrom
codex/rwx-ci-pilot
Closed

teddyknox wants to merge 180 commits into
developfrom
codex/rwx-ci-pilot

Conversation

@teddyknox

@teddyknox teddyknox commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Representative source-change comparison — 21 October 2026

One measured source-edit sample per service and language. Go now uses the corrected module/SDK cache configuration, with its parent warmed and verified before the new source edit was created. Historical Rust and Solidity observations are retained at their original commits.

Affected workload CircleCI RWX Cache qualification
Go fixture 17m 01s 13m 36s Verified warm parent, new source edit
Rust fixture 17m 35s 7m 03s Verified warm parent, new source edit
Solidity fixture 15m 35s 9m 56s Normal PR cache observation; caches not uniformly warm

Go measured commit: 3c99ebf34eb00d5478e2ebb50493ccb08524b8cc; warmed parent: e2d7e980154cb31ee38ce6f055d91e9556de3bae. The edit preserves the original RPC error when its revert-data method returns nil; two new regression cases pass on both providers. Original worker artifacts verify the same source, Go 1.26.9, tags/settings, package assignments, 12 workers and test parallelism 32. All 12 Circle workers restore their per-node caches from the exact warm-parent job; all 12 RWX compiler seeds hit cache. Complete captured worker logs show no Go SDK/module downloads or standard-library/external-dependency compiler invocations. Repository and generated-test compilation is recorded separately. Fresh verdicts and complete package/test-attempt accounting are verified. Helper builds and runtime subprocesses are outside the captured per-package compiler audit.

Selected Go scope includes its full unit-test job, all seven prerequisite jobs and setup/control work. Advertised build/test CPU and RAM match per worker/phase (Go 8 CPU / 16 GiB); RWX uses normal PR cache permissions with only develop able to write the protected tool cache. No changed-commit prewarming, cache-epoch changes, private benchmark cache, protected-branch spoofing or measured-run retry was used. Shared scheduling improvements apply on both services. Parent preparation/probes are excluded from the measured pair.

These times include setup, queueing, compilation/cache work and selected verdict publication; they are not full-repository PR completion times. Per-phase CPU/RAM matching does not equal identical hardware, usable memory or aggregate peak concurrency. Rust includes workspace tests, beacon/doctests and feature compilation checks; feature checks compile tests without executing them. Solidity covers standard main only, had nonuniform cache warmth, and includes 170 seconds of Circle queueing. The Rust/Solidity fixture heads are historical measurement snapshots; the shared harness has since advanced. One observation per language cannot establish typical or tail latency, fleet savings or reliability.

Current migration head 5552a61cccd569961d9ae100f896c28bc440ea9d retains 183 successful checks and one neutral check, including successful dependency review. Updated harness and Go fixture checks are all terminal and successful; exact validation states and native background runs are recorded below in the final-validation note. Previous Go results that downloaded the SDK remain qualified history.

Go SDK-corrected benchmark validation: #23151: 184 checks (183 SUCCESS, 1 NEUTRAL), #23533: 113 checks (112 SUCCESS, 1 SKIPPED), #23513: 25 checks (25 SUCCESS). Dependency review passed on all three heads. The first measured Circle/RWX attempts are retained; unrelated native checks are excluded from the benchmark time and compute scope.

Circle module-cache path correction

Current head: 5552a61cccd569961d9ae100f896c28bc440ea9d (cache fix d54fa1c6de7b5ac2f410dcd7b9860b67b9505720, merged develop f649ad865f4eb1384b2dfa022fc027f257650a86). The failed acceptance job restored a 2.1GiB module cache into /home/circleci/go/pkg/mod, while the machine image's Go processes read /home/circleci/.go_workspace/pkg/mod. Caching was enabled but bypassed, causing repeated network downloads and HTTP/2 stream errors during discovery. The shared helper now exports GOMODCACHE to the restored/saved directory through BASH_ENV for subsequent steps. Its immutable key also includes go.mod and the toolchain pins so the producer can save newly required modules/SDKs. Existing download retries, checksum verification, resources, gates and permissions are preserved.

An offline real-Go regression reproduces the different-GOPATH cache miss, then succeeds through fresh shells and a nested working directory using the YAML's actual setup command. Both Circle schemas validate; shared routing and adapter fixtures each pass 45 cases; CI and test-integrity reviews cleared the fix. The regression also covers spaces in the cache path and matching restore/save keys.

The first fix-only pipeline (#137085) proved current-producer restores on all 16 machine acceptance workers and all 12 Docker Go workers; its completed Go workers no longer downloaded the SDK. The Circle contract and Rust workflows passed before the remaining work was superseded by the develop merge. RWX interop recovered on attempt 3 after an aborted task and a Go SDK TLS timeout; original failures are retained.

Resolving the develop conflict preserves the SP1 log-commitment/source-marker checks and adds both new BRIDGE_HOOK feature combinations to native standard, modified, coverage and upgrade tasks. Native receipts cover all 29 current Circle contract-gate dependencies (eight additional occurrences). Both flags are enabled for the combined variant; host-target no_std completions are also verified. Historical benchmark evidence below is unchanged and does not establish coverage of newly added upstream Circle-only workloads.

The merge-head fetcher check caught compiler AST/type ID renumbering in four generated artifacts. Pinned local compilation exactly reproduced all four sealed RWX outputs; the refresh changes no ABI or creation/deployed bytecode. The full clean fetcher runner passed on 5552a61cccd569961d9ae100f896c28bc440ea9d with unchanged source inputs and no report errors; independent review verified the originals. The prior merge-head run was superseded after retaining that failure.

Final validation on 5552a61cccd569961d9ae100f896c28bc440ea9d: passed. Circle pipeline #137089, all four required Circle gates, dependency review, and the native RWX coordinator completed successfully. All reported checks finished without failures (183 success, 1 neutral). No manual retries were needed on the final head; superseded-run failures above remain recorded.

Go tests, OP Node acceptance, Kona acceptance and NUT pre-fork passed. Original logs verify the corrected canonical cache restore on all 28 Go/acceptance workers plus NUT, with complete successful test output, no fresh SDK/module downloads, and no HTTP/2 stream failures. They reused the corrected-key producer cache from job 5731686; go.mod, go.sum and mise.toml are unchanged across the fix and final head. Cache permissions and runner resources were preserved. The complete clean local fetcher rebuild and its original-file seals also passed on this exact head. Earlier benchmark results remain bound to their recorded commits; this validation is not another measured benchmark sample.

Go dependency reuse and shard split: recorded observation

Cache qualification corrected after log audit: all 12 Circle workers downloaded the Go 1.26.9 SDK during this sample. Module and per-worker compiler caches were reused, but the SDK was not warm: the old immutable module-cache key omitted toolchain inputs. The later fix at d54fa1c6de7b5ac2f410dcd7b9860b67b9505720 corrects that key as well as the separate machine-executor cache-path mismatch. The recorded times remain factual, but this sample is not eligible as a fully warm-toolchain comparison, and the 23.6% difference must not be presented as one. The new corrected-parent sample above supersedes this historical observation for the warm-cache comparison.

Implementation: 028834cb8713ee8e0fc9b45c20f7ceeb4f8084c7. Reuse helper builds only when ancestor inputs, toolchain/settings and artifact hashes match; keep their actual producer revision and bind the current consumer separately. Retain native compiler seeds, remove repeated module-cache tar handoffs, and restore sealed current-source discovery/binaries after filesystem assembly. Split the deployer CLI package's 34 top-level tests into four groups across the same 12 workers on both providers. Circle now retains each worker's compiler objects with per-node cache keys.

One new Go edit at 4f65e67b2c5a5da841b670ff1a9c0a6fa86595ac, after successful parent tests and module/compiler-cache verification on both services; SDK warmth was not established. No changed-commit prewarming or measured CI retry/replacement:

Affected workload CircleCI RWX
Full Go unit suite + all seven prerequisites + setup, through the last test verdict 19m29s 14m54s

RWX wall time was 23.6% lower (4m36s saved) in this sample. Full completion through control/gates was 19m41s on Circle and 15m05s on RWX. This measures affected jobs, not whole-repository PR latency or GitHub status-publication delay. Go fixture #23513, harness #23533.

Both providers passed with identical 462-package discovery, 495 work units, 3,188 unique observed top-level cases and final outcomes; the two new wrapped/nil-error regression cases passed on both. Original failure-event counts were zero. Fresh settings match (-count=1, -parallel=32, -p=4, timeout/retry policy and Go version). All 43 build/compiler/verdict allocation comparisons match advertised CPU/RAM; Go uses 12 workers at 8 CPU / 16 GB. Circle's fresh-test flag and RWX's parallelism 32 override are explicit benchmark controls. Hardware/OS/storage, usable RAM and phase overlap differ; matching is per worker/phase, not an aggregate CPU cap. Shared organization load is not isolated.

Cache evidence: all 12 Circle workers restored their own caches from the verified parent job 5731060; all 12 RWX compiler seeds were result-cache hits. Normal PR permissions were unchanged, including the sole RWX tool-cache writer on protected refs/heads/develop. Parent 0ce451245b686d9169878f0cd3da05924e46120e: Circle #137074, RWX, 12-seed cache probe. Unchanged contract/Kona/op-reth producers were reused; changed Go/prestate inputs rebuilt. Package go test -c logs show 67 Go compiler commands on RWX versus 252 on Circle, with 313 linker commands each; helper/tool builds are outside those counters.

Remaining overhead: the unsplit integration worker executes for about 8m26s. RWX still downloads a 6,483MiB Docker layer per runtime worker. Logged downloads total about 81.6GB/210 worker-seconds; published filesystem layers total about 18.8GB/77 worker-seconds. Circle cache restore/save steps total 656/78 worker-seconds and workspace attach/persist 71/18. These are overlapping aggregate telemetry with different coverage, not network-only or critical-path measurements; RWX lazy filesystem reads are not fully logged. Preparation and earlier failed/superseded development runs are excluded from the measured pair and retained separately.

Validation: Go fixture 26/26 checks green, harness 104/104 green, and dependency review green on all three PRs. All RWX checks on migration head 028834cb8713ee8e0fc9b45c20f7ceeb4f8084c7 passed. At that recorded Go-optimization revision, the main PR remained red on Circle: memory job 5731178, retried from 5730984, failed before tests during Go package discovery. Workers 1/4 received HTTP/2 INTERNAL_ERROR fetching go-bip39/go-multiaddr from proxy.golang.org; the dependent Circle gate/workflow also fail. The earlier NUT module-download failure recovered on retry. Those failures are outside the successful measured Go scope; the cache-path correction and current-head validation are above. Original artifacts/logs are retained.

Local validation: 63 Go helper tests passed (one opt-in contract fixture skipped), pinned Go fixture tests passed, Go lint zero issues, shellcheck and Circle schema validation passed. Independent implementation reviews cleared the changes. RWX lint retains the existing mise package-version warning. The projected artifact restore check and sealed discovery restoration check passed. Parent preparation's original GitHub HTTP500 and successful failed-work retry remain separate from this first-attempt measurement.

Rust compiler reuse and handoff fix: validated

The recorded Rust implementation at 8368c4c35ef1f63ff629d49ddd33187389966ffc fixes compiler/dependency cache invalidation across source edits and removes the full-target archive handoff. Seed preparation settles generated superchain inputs before publication; the unchanged parent test build logs zero recompilations. Current-source Cargo workers and runtime tests still execute freshly.

One new quantity diagnostic change at de695f31eb719bbc6ca924184a5dc1b0fcf29209 was measured after warming parent 8dd70d3246a7b9adade2df998dbb2bbd6046247e. Proof: CircleCI pipeline 137046, RWX run, detailed fixture report #23517.

Affected job group CircleCI RWX
Workspace tests, beacon and doctests 17m 29s 7m 03s
Ten feature compilation partitions 17m 35s 3m 23s
All selected verdicts 17m 35s 7m 03s

RWX selected-job wall time was 59.9% lower in this single sample. Both providers used matching per-worker CPU/RAM allocations and non-incremental compilation. Normal PR cache permissions remained in place: all twelve RWX dependency/compiler results reused proven preparation, and Circle's parent Rust hit rate was 96.3%. Exact tools, inputs, feature commands and test outcomes match; all three new regressions passed without retries. Feature checks compile test targets; unit tests, the beacon case and doctests run separately.

At that recorded Rust revision, validation was terminal: migration 175 successful, 1 neutral, parent 104 successful, fixture 20 successful. Local helper/cache/gate tests, the pinned live Rust fixture and independent reviews passed. The separate native E2E shadow also passed; its time is outside this selected-job comparison. Hardware/OS/storage and usable memory differ; split RWX test phases can overlap. Seeds follow the last build-input ancestor and this parent has the same Rust tree; accumulated deltas and cold preparation can change later performance. Full qualifications, cache counters and preserved preparation failures/cancellations are in the linked fixture report. The Rust sample remains bound to that recorded revision; current Go results are above.

Run all 88 selected CircleCI PR-workload occurrences as optional native RWX shadows: Main 33, Contracts 23, Rust 23 and Rust E2E 9. CircleCI retains its four required gates and dependency review remains required.

Implementation

  • Separate reusable compilation from fresh verdicts across Go, Rust, acceptance and contract matrices. Preserve original commands, discovery, selections, settings, failures and report bytes. Bind producer/consumer artifacts to source, toolchain and checksums; selected runtime verdicts execute freshly.
  • Fresh Rust feature checks previously recompiled without a writable baseline. Their compiler preparation now has ten producers tied to the source. Fresh feature consumers repeat the original library-feature and test-target compilation checks at 4 CPU / 8 GB, reuse compiler outputs, and publish reports without another compiler filesystem layer. These are compilation/type checks, not executed tests.
  • Restore unchanged Rust source timestamps to whole seconds, stamp changed inputs in the next second, and wait past those timestamps before Cargo. Keep incremental profiles and sccache; runtime nextest workers consume verified test archives separately from compiler workers.
  • Materialize the image's native GCC/glibc linker inputs into fresh local inodes before RWX WASM compilation. Preserve original bytes and metadata, retry only timed-out input reads at most three times, and seal the required preparation evidence. Cargo and verdicts retain their original failure semantics.
  • Route contract preflight and Forge RPC requests through a paced relay with bounded transient HTTP retries. Preserve permanent HTTP/JSON-RPC errors, assertion failures and FFI failures; redact attempts before sealing reports.
  • Keep runtime execution, permanent tests and removable migration tooling separate. Native gate policy preserves workload assignment, legitimate skips, fresh verdicts and original failures. Shared routing retains upstream publication and schedule restrictions.

Develop integration

Merged develop at 9c14635375e22608458da1d3ec4830af491009dd. Both providers include the new rust-deny and opgeth-ratchet requirements. SP1 retains all three ELF builds and checks both standalone vkey crates through the original recipe. Four fetcher compiler exports refresh upstream source/AST identifiers; ABI, creation/runtime bytecode and metadata are unchanged. The strict artifact comparison remains intact.

Historical develop integration validation

Recorded integration head: fd94c2c4b6e8b2b829a447b3492cae094d851c5b, merging develop 237ca52732f4845ea39b2c4e16250a731bd05524. This resolves the upstream retirement of issue-close automation in the shared routing source while retaining the Circle adapters, and preserves upstream serialization of the scheduled SP1 smoke test. The measured core workloads, RWX definitions, runner resources, cache permissions and linker materialization helper are unchanged from the frozen benchmark source below.

Native CI completed with 175 successful checks and 1 neutral check, including dependency review. All four CircleCI gates and native RWX shadows pass. The native RWX coordinator and native Rust E2E run both succeeded on this integration commit. Both WASM checks passed on their first attempt; each materialized 24 linker inputs (7.7 MB) with zero read retries and verified all 20 sealed original-file hashes. Preparation took 0.31s and 0.16s. Local checks pass: 45 shared routing cases, 45 Circle adapter cases, both Circle configuration validations, and 239 migration tests (226 passed, 13 skipped). Two independent automated reviews found no remaining issues. The integration commit is outside the frozen benchmark cohort.

Benchmark-source validation

Frozen source: 4d2451779159b802ccf506f35d99d266f8d405f6. Native push validation completed with 175 successful checks and one neutral check, including dependency review. Both native RWX WASM jobs pass on their first attempt. Their preparation copies 24 host linker inputs (7.7 MB per task) with zero read retries; sealed original-file hashes and source/run/first-attempt provenance verify. Preparation takes 0.30s and 0.86s. The strict original-report comparer verifies all seven additional Rust checks across providers, retaining the RWX preparation difference while requiring identical shared commands and coverage.

Local validation: 392 runtime tests (349 passed, 43 skipped) and 239 migration tests (226 passed, 13 skipped). Fault-injection coverage verifies bounded timeout recovery, immediate propagation of other I/O failures, corrupt-copy rejection, atomic replacement cleanup and metadata preservation. Missing, failed or corrupt preparation cannot count as successful parity. Independent automated CI and integrity reviews pass after addressing their findings. RWX lint passes with the existing package-version warning. Earlier live Rust, SP1 and Forge/Anvil fixtures remain dated validation evidence.

Matched benchmark with normal PR cache permissions

Three sequential pairs use the same frozen source, committed definitions, matched advertised CPU/memory and normal PR cache permissions. CircleCI passes 1/3; RWX passes 3/3. 3/3 pairs pass allocation, fresh-verdict, all-ten-feature-producer reuse and host-preparation audits. In the sole pair passing both services, full verdict time is 29m58s on CircleCI and 13m56s on RWX. This is one jointly passing observation. Every measured failure remains in the dataset; measured runs are not retried or replaced.

Pair CircleCI RWX coordinator RWX Rust E2E
1 136958 Run Run
2 136961 Run Run
3 136964 Run Run

The retained trial-1 CircleCI failure was in memory-all-kona-op-reth-fusaka. TestConductorClusterStartsWithOneActiveSequencer could not build its active L2 endpoint provider: the leader proxy returned EOF while other proxies rejected non-leader requests. Original logs and available discovery/test artifacts are retained with hashes and source binding. The underlying cause remains unconfirmed; this sample was not retried or replaced.

The retained trial-2 CircleCI failure was in memory-all-opn-op-reth-fusaka. Three OPN acceptance shards failed package discovery after module downloads from proxy.golang.org returned HTTP/2 stream INTERNAL_ERROR for go-libp2p-pubsub v0.17.0 or gnark-crypto v0.18.1. Their package discovery artifacts preserve the dependency errors. Original logs and available discovery/test artifacts are retained with hashes and source binding. The underlying cause remains unconfirmed; this sample was not retried or replaced.

Full-workflow warm-up and confirmation of all ten source-bound feature compiler producers finish before measurement. Fresh consumers repeat the original library-feature and test-target compilation checks at 4 CPU / 8 GB; these checks compile but do not execute tests. Cached producer historical runtimes are excluded from current phase/cost totals. Fresh feature and WASM reports bind the current run and first attempt.

The cache vault still permits writes only from protected develop; PRs restore it read-only. No private writable benchmark cache, user writer grant or spoofed protected branch is used. This measures new runs at unchanged source after workflow warming and observed feature reuse, rather than claiming every protected compiler baseline is warm. Changed-source PRs, cold starts and full-run RWX standard retries are different experiments. Hardware, OS/executors, disk behavior and task-visible RAM differ; RWX reserves 2 GB. Existing incremental compiler policies differ. Organization concurrency and shared load are not isolated; queueing remains in end-to-end verdict time. Cache-phase categories overlap; RWX download logs omit lazy filesystem reads during execution. Cost analysis and sanitized evidence remain private.

The preceding benchmark at 8251ed2385 remains historical evidence: CircleCI passed 3/3 and RWX 2/3, with one measured WASM failure opening crti.o and crtbeginS.o with Connection timed out. That failed attempt was never retried or replaced. A native preparation attempt also failed opening libgcc.a. The new local materialization addresses these host linker reads; the provider-side filesystem cause remains unconfirmed. The new CLI warm-up separately encountered the same Git gitlink lookup failure before Cargo; four probes verified the expected commit/tree objects and gitlink, and an unchanged targeted preparation retry is retained separately. Its cause remains unconfirmed. The confirmation also had a deny task stop after sccache startup, before Cargo/report settings, without an exit signal or OOM reason in available metadata. Its unchanged targeted preparation retry passed, with all 12 original-file hashes and source/run/second-attempt binding verified. Standalone Git probes exposed slow lookup, but an experimental Git-object arm was terminated; no production Git workaround was adopted. Both preparation failures and targeted retries remain separate from the measured cohort. Their causes remain unconfirmed.

Changed-source benchmark fixtures

Three draft source-fixture PRs stack on the shared benchmark harness #23512, which stacks on this migration PR. Each case uses one controlled first-attempt pair at the same source commit on both services. Wall time below runs from provider start through the selected verdicts, including setup, queues, compiler/cache work, prerequisites and output publication.

Source fixture / proof Selected existing jobs CircleCI RWX
Go #23513, 9cb1eea4d96e go-tests, 12 workers / 462 packages, plus seven producers 23m14s 21m35s
Rust #23517, bf1efa4fa197 rust-tests and ten rust-cargo-hack partitions 19m30s 38m15s
Solidity #23519, 13a35ccd4af7 contracts-bedrock-tests main and module preparation 15m35s 9m56s

All selected jobs passed. Original reports verify the corresponding coverage and effective workload settings; provider metadata verifies 31 Go, 23 Rust and three Solidity CPU/RAM comparisons. Matching is per worker/phase, with different hardware, OS/storage, usable RAM and phase overlap/aggregate peak allocation. RWX reserves 2 GB. Rust feature checks compile libraries and test targets; unit/compiler tasks separately execute the tests.

Normal PR compiler-cache permissions remain in place: RWX PRs restore caches read-only, with only protected develop allowed to write. Cache state evolves and is not uniformly warm; a Rust feature producer restored zero target files and had zero sccache hits. Its workspace producer spent 7m12s in post-processing, and the final feature producer spent about three minutes uploading its layer. Solidity included 2m50s of Circle queueing; its test command took 6m35s versus RWX's 5m43s.

Go uses explicit fresh -count=1 verdicts on both services, overriding Circle's usual fresh-tests=false default only for the fixture. RWX also uses CLI test-parallel=32 to match Circle's observed setting; the committed RWX default remains eight. The earlier Go diagnostic with different parallelism is retained in #23513 and excluded from the controlled comparison. Its ordinary cache writes can influence the follow-up cache state. No measured attempt was retried or replaced, and the changed commits were not prewarmed.

Go's 26 PR checks and Solidity's 20 PR checks passed. Rust's selected jobs passed, but its separate automatic native E2E shadow failed in runtime-tools, installing geth after Go checksum fetches returned HTTP/2 INTERNAL_ERROR, before E2E execution. That original failure is retained; upstream versus RWX network-path cause remains unconfirmed. That recorded fd94c2c4b6e8 integration snapshot had 175 successful and one neutral check; current-head validation is above. These single source-change observations cover the selected jobs; broader PR performance and reliability remain rollout work. Cost analysis stays private.

Evidence and rollout

The October 15 benchmark at 77ca8e52bd and archived 86-workload original-report parity at a1aa49aaf3713a8172f3f615f094488fd8e39c3d are historical evidence, separate from the current 88-workload run. The controlled earlier feature-shard probe at 77ca8e52bd demonstrated fresh compiler reuse in 2m03s versus the earlier 31m08s median execution; it is not part of the new measured cohort. Operation and retirement conditions remain in docs/ai/rwx-migration.md and docs/ai/rwx-parity-todos.md.

CircleCI retains ci-gate, required-contracts-ci, required-rust-ci, required-rust-e2e and existing review requirements. A small passing series does not establish platform-wide filesystem reliability. Protected-develop warming, broader changed-source PR coverage, fork/merge-queue events, production credential ownership, and cutover/rollback rehearsals remain rollout work. Retire migration tooling and Circle adapters only after approved cutover and closure of the rollback window.

@wiz-0f98cca50a

wiz-0f98cca50a Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities -
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations -
SAST Finding SAST Findings 18 Low
Software Management Finding Software Management Findings -
Total 18 Low

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.

Comment thread ops/ci/runtime/op-reth-report.py Resolved
teddyknox and others added 7 commits October 30, 2026 15:18
…docs-20261027

# Conflicts:
#	.circleci/continue/main.yml
develop's v9 merge made coverage-lcov call a new coverage recipe that
splits Amsterdam and default-EVM passes. The live coverage fixture copied
only the old recipe list and cut recipes at the first blank line, so
just reported a missing coverage recipe. Copy coverage, test,
_test-single, the DEFAULT_EVM_TESTS variable and apply-match-path.sh,
and end each recipe at the next unindented line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Forge records no failure cache in JUnit mode, so develop's test-rerun
falls back to rerunning cache/test-files. The suite runner now writes
that file list and always reruns a failed pass, as CircleCI's on_fail
step does. The live fixture puts its L2 source under src/L2, which the
test-validation path rule requires, and copies the test recipes that
the rerun fallback calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
develop's coverage recipe splits ordinary tests by DEFAULT_EVM_TESTS and
writes each pass's LCOV to its own --report-file. The coverage runner
also requests the attribution report, and Foundry ignores --report-file
when more than one report is requested, so the recipe could not find its
per-pass file. Both passes would also overwrite one attribution report
and share one engine summary.

The runner now executes each EVM pass as its own sealed phase
(ordinary-amsterdam, ordinary-l2, upgrade), with FOUNDRY_HARDFORK in the
recorded argv. It records the routing in evm-passes.json, keeps each
pass's failure cache for test-rerun, and concatenates both passes into
the public lcov.info as the recipe does. The comparator derives the
expected discovery and coverage commands from the sealed routing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… into codex/rwx-handoff-docs-20261027

# Conflicts:
#	packages/contracts-bedrock/test/L2/L1Block.t.sol
… into codex/rwx-handoff-docs-20261027

# Conflicts:
#	.circleci/continue/main.yml
develop removed contracts-bedrock-coverage and its dispatch parameter
(#23879). The RWX coverage workload existed only to establish same-SHA
parity with that job, so it has no counterpart left.

Remove the native coverage graph, its compile and verdict packages, the
coverage runner and comparator, its shadow check, and its
required-contracts-ci mappings. Remove the c-contract_coverage_replay
parameter and the go-ffi trie replay seed, which only served coverage
replay, and restore go-ffi to develop. The gate workspace option served
only the coverage checkout, so receipts now always run at the checkout
root. The parity checklist records the retirement and the October 30
INTEROP rename, and keeps baseline job names as recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread ops/ci/migration/selector-registry.py Resolved
Comment thread ops/ci/migration/selector-registry.py Resolved
Comment thread ops/ci/migration/selector-registry.py Resolved
Comment thread ops/ci/runtime/go-compiled-tests.py Resolved
Comment thread ops/ci/runtime/go-compiled-tests.py Resolved
Comment thread ops/ci/runtime/rust-cannon-report.py Resolved
Comment thread ops/ci/runtime/rust-e2e.py Resolved
teddyknox and others added 14 commits November 5, 2026 08:09
…docs-20261027

# Conflicts:
#	.circleci/continue/main.yml
develop replaced the circleci/shellcheck orb with a local shell-check job
that runs find | xargs shellcheck with the mise-pinned ShellCheck 0.10.0
(#23905). The merge points that job at static-checks.py so it keeps
sealing parity reports.

static-checks.py now runs develop's exact find | xargs command, discovers
files with the same find predicates, and reads the ShellCheck version
from mise.toml when it executes. The RWX static-tools layer installs the
same pin. A clean run prints nothing, so the runner and comparator
require exit 0 and empty output, not the orb's success banner. The
vendored orb script is no longer used and is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…docs-20261027

# Conflicts:
#	.circleci/continue/main.yml
develop's shell-check runs ShellCheck through xargs. GNU xargs, as on
the CI runners, reports a failing ShellCheck as exit 123; BSD xargs
reports 1. The live fixtures asserted the orb's exit code 1, so they
passed on macOS and failed on RWX.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
develop removed the ZK_DISPUTE_GAME dev feature and its two contract
matrix variants, ZK_DISPUTE_GAME and BRIDGE_HOOK,ZK_DISPUTE_GAME
(#23868). Remove the matching native compile and verdict tasks for the
standard, changed-file and upgrade suites, their required-contracts-ci
mappings and their feature environment entries. The parity checklist
records the retirement and develop's shell-check change, and keeps the
baseline job names as recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
develop's test-upgrade now retries failed tests once with --rerun when
JUNIT_TEST_PATH is set (#23930). Circle's trace step runs
test-upgrade-rerun without JUNIT_TEST_PATH, so it takes the plain path.
The upgrade runner set a diagnostic JUnit path for that step, which sent
it through the retry and passed --rerun to Forge twice. The runner now
clears JUNIT_TEST_PATH for the trace rerun, and the live fixture checks
the rerun's recorded command and failure output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
develop added op-core/nuts/state/lagoon_multichain_state.json (#23897).
Its _check-nut-prefork-states recipe regenerates that file with lagoon
and skips *_multichain names, then checks every committed state for
drift. The pre-fork runner rejected the underscore in the fork name, so
the CircleCI job and its RWX shadow failed before generation.

The runner now accepts <fork>_multichain state names, keeps them in the
drift check, and excludes them from execution with the recipe's reason.
The live fixture commits a multi-chain state and confirms that only the
three base forks run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The upgrade runner routes Forge's fork reads through the recording RPC
proxy. The proxy held one upstream request at a time, so a run's
throughput was one round trip, about 0.22s, per request. The op-mainnet
CUSTOM_GAS_TOKEN upgrade suite now makes about 4,000 serialized reads.
Forge stayed silent past Circle's 15-minute no-output limit and the job
timed out. develop's job, which calls the archive directly, runs the
same tests in about two minutes.

Each attempt still takes its start time from the global requests-per-
second budget, and a request keeps its slot across retries. A new
max_in_flight policy key bounds concurrent upstream requests; it
defaults to one, so the L2 fork proxy is unchanged. The archive policy
allows eight in flight under its 20 requests per second.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… into codex/rwx-handoff-docs-20261027

# Conflicts:
#	.circleci/continue/main.yml
develop removed the rust-binaries-for-sysgo aggregator and pinned
SOURCE_DATE_EPOCH in Circle's sccache setup (#23948). The aggregator had
no native task or gate mapping, and RWX does not use that sccache path,
so only the parity checklist changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RWX tool preparation ran each mise install once with mise's 30-second
HTTP timeout. Go's download from dl.google.com and several GitHub release
downloads timed out in recent runs, which failed whole RWX shadows.

rwx-prepare.sh now sets MISE_HTTP_RETRIES=5 and MISE_HTTP_TIMEOUT=90s
for its own installs and exports both through RWX_ENV, so later mise
installs in tasks that use the prepared tools retry too. This follows
docs/ai/docker.md: every external fetch retries with the tool's own
mechanism. A test runs the script against a stub mise and checks the
policy and its exports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…docs-20261027

# Conflicts:
#	op-acceptance-tests/justfile
…list

develop runs each acceptance test in its own go test, eight at a time
(#23974). The merge keeps the recorded RWX shard selection in the shared
recipe, runs the same per-test commands, and joins the per-test event
logs in the acceptance report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
develop's check-forge-version now also runs the forge package's
release-archive test, TestStandardBinary_ForgeBins, with go test
(#23643). The live main-checks fixtures copy the recipe into a bare
repository without a Go module, so go test failed and the RWX
pr-checks helper task failed.

A shared fixture helper adds a minimal module and a stand-in test that
passes only when the recipe sets OP_DEPLOYER_TEST_FORGE_RELEASES=1. The
fixtures still run the real recipe and a real go test, without
downloading the release archives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teddyknox

Copy link
Copy Markdown
Contributor Author

Closing in favor of #23783, which adds the RWX shadow without changing any CircleCI job.

This PR changed 66 CircleCI job definitions to collect parity evidence. #23783 is additive: .rwx/ and the ops/ci/ code it uses, so reviewing it means checking that nothing CircleCI runs has changed.

The standalone fixes from this branch are in their own PRs: #23772, #23775, #23780, and #23776 (draft). The branch codex/rwx-ci-pilot stays available for reference.

🤖 Closed by Claude Code

@teddyknox teddyknox closed this Nov 6, 2026
teddyknox added a commit that referenced this pull request Nov 6, 2026
Port what #23151 mirrored after this branch's import, adapted to the
unchanged recipes this branch calls:

- Acceptance runs one go test per test (#23974): the shim accepts the
  deduplicated name list that shard-tests.sh now passes, and the report
  checks one JSON and one JUnit per assigned test.
- Contract suites run the Amsterdam and default-EVM passes that
  `just test` runs (#23788); INTEROP replaces OPTIMISM_PORTAL_INTEROP
  (#23779); the ZK_DISPUTE_GAME variants are retired (#23868); the
  coverage shadow is retired with CircleCI's coverage job (#23879).
- The upgrade trace rerun runs without JUNIT_TEST_PATH (#23930). When
  test-upgrade's own retry replaces its JUnit with the rerun's cases,
  the verdict accepts that subset instead of reporting a false failure.
- NUT pre-fork accepts and skips <fork>_multichain states (#23897).
- shell-check runs develop's find | xargs shellcheck with the pinned
  ShellCheck (#23905); the vendored orb script and its license go.
- From #23151: the archive RPC proxy keeps up to eight fork reads in
  flight, RWX tool installs retry downloads, and the live forge-version
  fixture covers the forge package test (#23643).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant