Repository navigation
Description
Sub-issue under #20257.
When the monorepo's main go.mod flips to upstream go-ethereum (#20266), op-e2e/actions tests still need an OP-flavored execution layer. Today those tests import op-geth's core.BlockChain, core/state, core/vm, eth.Ethereum, and friends in-process and drive a custom L2EngineAPI for tx-by-tx control. We can't replace those imports with upstream go-ethereum (silently wrong EVM behaviour); we don't want to keep op-geth as a global dep just for tests.
Strategy: build a small, dedicated Rust component (call it op-reth-test-engine) that runs the OP-flavored state transition for tests, with two consumers:
- Go side — Go action tests spawn a subprocess and talk to it over Unix-socket JSON-RPC.
- Rust side — Rust action tests, when they're written, import the same component in-process as a library (no socket). The same Rust API powers fuzzers, deterministic replays, and any future Rust test harness.
This means the component must be library-first: a clean Rust API at the core, with the binary + socket wrapper being a thin shim over the library.
Layered design
┌─────────────────────────────────────────────────────────────────┐
│ Rust action tests Go action tests │
│ └─ direct lib calls └─ JSON-RPC over Unix socket │
│ │ │ │
│ ▼ ▼ │
│ ┌───────────────────────────────────┐ │
│ │ op-reth-test-engine (binary) │ │
│ │ - jsonrpsee server on UDS │ │
│ │ - thin translation: RPC → lib │ │
│ └────────────────┬──────────────────┘ │
│ │ │
│ ┌──────────▼────────────┐ │
│ │ op-reth-test-engine │ ← library crate │
│ │ (lib, no_std-ish) │ │
│ │ - in-memory chain │ │
│ │ - OP EVM exec │ │
│ │ - stateful payload │ │
│ │ builder │ │
│ └───────────────────────┘ │
│ ▲ │
│ reuses op-reth crates │
└─────────────────────────────────────────────────────────────────┘
Library API surface (Rust)
The fixed contract; everything else is a wrapper around this.
pub struct TestEngine { /* … */ }
impl TestEngine {
pub fn new(genesis: ChainGenesis, rollup_cfg: RollupConfig) -> Self;
// Standard Engine API equivalents
pub fn forkchoice_updated(&mut self, state: ForkchoiceState, attrs: Option<PayloadAttributes>)
-> Result<ForkchoiceUpdatedResult>;
pub fn new_payload(&mut self, payload: ExecutionPayload, …) -> Result<PayloadStatus>;
pub fn get_payload(&mut self, id: PayloadId) -> Result<ExecutionPayloadEnvelope>;
// Test-only extensions
pub fn include_tx(&mut self, id: PayloadId, raw_tx: Bytes) -> Result<Receipt>;
pub fn remaining_block_gas(&self, id: PayloadId) -> Result<u64>;
// Standard read-only chain queries (for the eth_* RPC surface)
pub fn block_by_number(&self, n: BlockNumberOrTag) -> Option<Block>;
pub fn block_by_hash(&self, h: B256) -> Option<Block>;
pub fn header_by_number(&self, n: BlockNumberOrTag) -> Option<Header>;
pub fn receipts_by_block_hash(&self, h: B256) -> Option<Vec<Receipt>>;
}That's it. Two test-only methods, plus standard surface.
Binary wrapper (op-reth-test-engine)
- One crate; depends on the library crate.
- Spins up a
jsonrpseeserver bound to a Unix socket (path passed via CLI). - Registers three RPC namespaces:
engine_*(standard Engine API)eth_*(read-only chain queries: getBlockByNumber/Hash, getHeaderBy…, getBlockReceipts)optest_*(optest_includeTx,optest_remainingBlockGas)
- No P2P, no txpool, no real RPC server, no persistence.
- Lifecycle: parent process kills it on test cleanup; OS reclaims the socket.
What's stripped from a full op-reth
| Component | Status |
|---|---|
| OP EVM, OP hardforks, deposit-tx handling | Keep (via existing op-reth crates) |
| Engine API state machine | Keep, but adapted to support stateful in-flight payload (see below) |
| In-memory chain database | Keep |
| P2P / discovery / sync | Remove |
| Txpool / gossipsub | Remove |
| Persistent storage (MDBX/RocksDB) | Remove (in-memory only — fast, deterministic, no cleanup races) |
| HTTP RPC server | Remove (UDS only) |
| Production RPC namespaces (admin, debug full, txpool, net, web3) | Remove |
| Rollup-node-side pieces | Out of scope |
The one architectural change
Today op-reth's payload builder fires once on FCU and getPayload returns whatever popped out. The test engine needs the builder to keep a partial-block state alive across:
forkchoice_updated(attrs)— start a partial block, applyattrs.transactions(deposits), park.- N ×
include_tx(payload_id, raw_tx)— append to the in-flight block, run the EVM on the tx, persist the receipt, error on gas-limit/invalidity. get_payload(payload_id)— seal what's there, return the executionPayload.
Indexing in-flight state by PayloadId is the natural key. This is the bulk of the actual implementation effort.
Non-goals (kept simple deliberately)
- No
pendingIndices,setForceEmpty,forcedEmpty,txpoolSync. These exist in the current Go in-process L2EngineAPI to bridge against a real txpool; in this design there is no txpool. The test injects signed txs directly intooptest_includeTx, and chooses when (and whether) to call it. Empty blocks are produced by simply not calling include_tx. - No
fromargument oninclude_tx. Recover the sender from the signature. - No persistence. New process per test or per top-level helper instance; in-memory chain only.
- No real time. Block timestamps come from FCU
attrs.timestamplike in production; no wall-clock.
Go-side migration — tracked in #21196
Now a dedicated sub-issue: #21196 (blocked by this issue; soft-after #20264; blocks the #20266 cutover). The accurate scope there is 101 L2Chain() sites across ~34 files (the counts below were the original rough estimate). Once the Rust component is callable:
- Replace
op-e2e/actions/helpers/l2_engine.go::newBackendwithexec.Command(\"op-reth-test-engine\", \"--socket\", path)+ an RPC client. - Replace direct
engine.L2Chain().*calls (~100 sites across ~14 test files) withEthClientcalls — purely mechanical. - Replace
engine.EngineApi.IncludeTx/RemainingBlockGascalls withoptest_*client calls — concentrated inhelpers/l2_engine.goplus a handful of proofs/sequencer tests. - Replace
engine.Eth.TxPool().Add(rawTx)witheth_sendRawTransaction— except wait, there's no txpool. The few tests that use this (reorg_test,safedb_test) submit L1 batcher txs to the L1 miner, which is unaffected by this issue. L2-side txpool usage (dencun_fork_test) needs to be reframed: sign and pass tooptest_includeTxdirectly.
Effort estimation
| Piece | Estimate |
|---|---|
Rust library: stateful payload builder, include_tx, in-memory chain, OP EVM wiring |
weeks |
| Rust binary: jsonrpsee + UDS + namespace registration | days |
| Go client + helpers/l2_engine.go rewrite | days |
| Mechanical chain-query migration in test files | days |
| Total | a few weeks of focused work |
Open questions (for discussion before implementation)
- Crate location: under
rust/op-reth-test-engine/(peer to op-reth) or as a sub-crate of op-reth? - Genesis injection: pass via CLI flag (path to genesis JSON) or via a startup
engine_initRPC call? - Logging: forward Rust
tracingto the Go test's stderr automatically (subprocess stderr → `t.Log`)? - Rollup-config path: tests pass it explicitly per-instance, or expose it via the lib API?
Dependencies & independence
This is the most independent item in the op-core extraction epic (#20257). The Rust deliverable (library + thin UDS binary) lives entirely in the rust/ Cargo workspace and reuses existing op-reth crates. None of the other sub-issues (#20259–#20266) touch Rust — zero file/module/import overlap, no ordering constraint. It can start and merge in parallel today.
- Not blocked by anything in the epic. It does not depend on the
op-core/*Go package extractions (opgeth-decoupling: create op-core/params (OptimismConfig, ChainConfig, LoadChainConfig) #20260/opgeth-decoupling: create op-core/fees (L1 cost + operator-fee math) #20261/opgeth-decoupling: create op-core/types (DepositTx and PostExecTx with differential tests, Receipt) #20262), the call-site swaps (opgeth-decoupling: swap call-sites to op-core/* across op-node, op-service, op-batcher #20263), or the OP-aware ethclient (opgeth-decoupling: make op-service/sources the OP-aware eth client #20264). - It blocks the final cutover, not the reverse. Until
op-e2e/actionsstops embedding op-geth in-process (core.BlockChain,eth.Ethereum,core/vm, plusL2EngineAPIfrom the out-of-scopeop-program),go.modcannot flip to upstream go-ethereum without silently-wrong EVM behaviour. opgeth-decoupling: final cutover (flip go.mod to upstream go-ethereum) #20266 has been updated to list this issue as a blocker. - Soft sequencing for the (separately-tracked) Go-side wiring. The actions-test rewrite that swaps
engine.L2Chain().*→eth_*client calls is cleaner after opgeth-decoupling: make op-service/sources the OP-aware eth client #20264 lands, so the Goeth_*decode reuses the canonical OP-aware client rather than a throwaway decoder. Soft preference, not a hard block. The Rust deliverable here has no such dependency.
Spike-validated assumptions (against develop, reth rev 81c0261)
- UDS transport is free via
reth-ipc:server::Builder::default().build(socket).start(RpcModule).await -> ServerHandle. Standard jsonrpseeRpcModuleregistration — soengine_*/eth_*/optest_*are#[rpc(server, namespace=…)]traits merged in. Wire framing is newline-delimited JSON-RPC (StreamCodecdefaultSeparator::Byte(b'\n')), compatible with go-ethereum's IPC client. - OP execution reuse confirmed:
OpChainSpec::from_genesis,OpEvmConfig/OpBlockExecutor, trie state roots viastate_root_with_updates, and Isthmus+ post-exec (is_sdm_active_at_timestamp/begin_post_exec_tx) all exist and arepub. - Build-loop reuse anchors on the in-workspace
reth-optimism-payload-builder(OpPayloadBuilderCtx), notop-rbuilder(a separate nested workspace withpub(super)helpers). - Open-question leanings: crate at top-level
rust/op-reth-test-engine/; genesis via--genesisCLI flag (not anengine_initRPC); in-memory chain backed by an ephemeral reth DB (correct OP state roots for free) rather than a literal RAM map.
Out of scope
- Production op-reth changes — this is a separate binary/crate, not a feature flag on the production binary.
- Migration of action tests to op-devstack-style — that's a parallel effort and not blocked by this.
🤖 Generated by Claude Code
Activity
sebastianst commentedon Jun 4, 2026
🤖 Claude says: Implementation underway as a stacked series of PRs (plain-git stack off
develop).PR 1/5 — open: #21216 —
op-reth-test-enginefoundation: anEphemeralChain(a rethBlockchainProviderover an ephemeral tmpfs DB, genesis-initialized viainit_genesis) + theTestEnginewrapper + read-only block/header/receipt queries + a typedError/Result. Security review: APPROVE.Approach confirmed by a reuse pass: we reuse the real op-reth stack —
OpChainSpec::from(Genesis), theOpNodeprovider factory,init_genesis,BlockchainProvider, and (in later PRs)OpExecutionPayloadValidator::ensure_well_formed_payload/OpEngineValidatorand theConfigurePostExecEvmblock builder — rather than reimplementing storage, validation, or block building. Noop-rbuilderand noreth-optimism-flashblocksdependency (thetake_bundlepark/resume primitive is generic revm). One in-memory design note: the "in-memory only" intent is met via an ephemeral reth DB under a tmpfs temp dir (gives correct OP trie state roots for free).Remaining stack:
new_payloadimport path) + attrs-lessforkchoice_updated.forkchoice_updated(attrs)→optest_includeTx×N →getPayload), keyed byPayloadId.engine_*/eth_*/optest_*overreth-ipc, newline-delimited JSON-RPC).🤖 Generated by Claude Code
feat(op-reth-test-engine): ephemeral chain + read-only queries (1/5) (e…
op-reth-test-engine: stateful in-flight payload builder
op-reth-test-engine: UDS JSON-RPC server + shared engineipc Go client
sebastianst commentedon Jul 15, 2026
🤖 Claude Fable 5 says: The engine is complete and the full track (this issue + #21196) is implemented as a draft stack: #21242 (engine) ← #21786 (op-e2e/actions switch, both-EL gate: identical 231-test pass-set) ← #21787 (deletes the in-process op-geth L2 EL, −2588 lines).
Two corrections to this issue's design text, from implementation:
optest_includeTx, gas-limit checks mirroring the old engine-API helper); the "returns Unsupported until the builder lands" note is obsolete.newPayloadvalidates+stores without moving the head; only forkchoice canonicalizes (including alternate-fork reorgs viaNewCanonicalChain::Reorg). The original design's immediate-commit sketch breaks reorg tests and violates Engine API semantics; this contract should be considered part of the design.Also of note: the generic subprocess-IPC client plumbing (spawn, UDS dial, stderr drain, death-signal, binary resolution) now lives in a shared
op-service/engineipcpackage, intended for reuse by other Rust-engine Go clients.op-reth-test-engine: stateful in-flight payload builder
op-reth-test-engine: UDS JSON-RPC server + shared engineipc Go client
op-reth-test-engine: serve full-transaction blocks over eth_getBlock*
op-reth-test-engine: serve historical eth account-state reads
op-reth-test-engine: stateful in-flight payload builder
op-reth-test-engine: UDS JSON-RPC server + shared engineipc Go client
op-reth-test-engine: serve full-transaction blocks over eth_getBlock*
op-reth-test-engine: serve historical eth account-state reads
16 remaining items
op-reth-test-engine: serve full-transaction blocks over eth_getBlock*
op-reth-test-engine: serve historical eth account-state reads
feat(op-reth-test-engine): ephemeral chain + read-only queries (1/5) (e…
op-reth-test-engine: stateful in-flight payload builder
op-reth-test-engine: UDS JSON-RPC server + shared engineipc Go client
op-reth-test-engine: serve full-transaction blocks over eth_getBlock*
op-reth-test-engine: serve historical eth account-state reads
op-reth-test-engine: UDS JSON-RPC server + shared engineipc Go client
op-reth-test-engine: UDS JSON-RPC server + shared engineipc Go client
op-reth-test-engine: serve full-transaction blocks over eth_getBlock*
op-reth-test-engine: serve historical eth account-state reads
op-reth-test-engine: UDS JSON-RPC server + shared engineipc Go client
op-reth-test-engine: serve full-transaction blocks over eth_getBlock*
op-reth-test-engine: serve historical eth account-state reads