feat: upgrade reth sdk to v2.4.0

[calbera]

Upgrade to Reth SDK v2.4.0 (Storage V2)

Reviewer Outline

Suggested review order, from highest to lowest risk. Cargo.lock (~4,300 lines) can be skimmed — the hand-written diff is roughly 1,700 lines.

Area | Code Files | What to scrutinize

1. Consensus-critical: payload ID | src/engine/payload.rs | The sha256 payload-ID derivation moved into PayloadAttributes::payload_id. Must produce byte-identical IDs; check the preserved unit-test vectors.
2. On-disk format stability | src/primitives/header.rs, src/transaction/mod.rs | Golden byte-vector tests (mod db_format) pin the Compact encoding. Confirm no codec change slipped in — existing datadirs depend on it.
3. Engine API surface | src/engine/rpc.rs, src/engine/mod.rs, src/engine/builder.rs | beacon-kit ≥ v1.4.1 treats EL HTTP 4xx as fatal; the custom getPayloadV4P11 / pre-Osaka getPayloadV5 → UnsupportedFork behavior must survive.
4. Mechanical API migration | src/node/evm/*, src/pool/, src/rpc/, remaining src/ | Trait-signature churn dictated by upstream; verify against reth v2.4.0 patterns rather than line-by-line.
5. Dependencies & features | Cargo.toml, build.rs, deny.toml | default-features = false on reth — confirm the explicit feature list loses nothing vs v1.11.4.
6. Tests | tests/e2e/storage_v2_test.rs, tests/e2e/gas_limit_regression_test.rs, tests/e2e/* | New coverage; check assertions actually pin the behaviors above.
7. Docs & CI | README.md, docs/storage-v2.md, .github/, Dockerfiles, CLAUDE.md | Operator guidance accuracy; hive assets flagged for re-tuning.

Points worth reviewer attention:

• The reth dependency is now default-features = false with an explicit feature list. Reth v2.4.0's defaults include jit (revmc, requires a system LLVM toolchain) and gmp; the explicit list reproduces the v1.11.4-equivalent set (otlp, otlp-logs, js-tracer, keccak-cache-global, asm-keccak, min-debug-logs) plus reth-revm with portable.
• rust-version = "1.95" (reth v2.4.0's MSRV).
• [profile.dev] gains debug = "line-tables-only" + split-debuginfo = "unpacked", matching upstream reth — debug test binaries otherwise exceed multiple GB each.

---

Storage V2

No node wiring was needed: the v2.4.0 launcher defaults handle everything.

• Fresh datadirs initialize as Storage V2 — RocksDB for history indices and tx lookups, static files for changesets/receipts, MDBX for hashed state.
• Existing V1 datadirs keep working unchanged: the layout persisted in DB metadata always wins over the --storage.v2 flag (which only applies at datadir creation).
• In-place conversion is bera-reth db migrate-v2 --chain <genesis>, confirmed generic over Berachain's custom primitives.
• Since no Berachain snapshots exist, migrate-v2 or resync are the only paths to V2 for existing nodes — docs/storage-v2.md says so explicitly.

Operator-facing docs: docs/storage-v2.md (what changed, the three operator situations, migration procedure, --storage.v2 semantics, node modes) plus a new "Storage" section in README.md.

---

API migration, file by file

• src/primitives/header.rs — upstream removed RlpBincode/SerdeBincodeCompat (impls deleted); HeaderMut gained set_mix_hash/set_extra_data/set_parent_beacon_block_root; Decompress now returns reth_codecs::DecompressError; BlockHeader gained block_access_list_hash()/slot_number(), both None (Berachain headers don't adopt EIP-7928 BAL fields). The on-disk Compact encoding is intentionally untouched, locked by golden test HEADER_V2_4_0_GOLDEN plus a pre-Prague1 vector.
• src/transaction/mod.rs, txtype.rs — SignedTransaction is now blanket-implemented upstream, so the manual impl is deleted; same DecompressError change; golden vector POL_ENVELOPE_V2_4_0_GOLDEN locks the PoL envelope's on-disk format.
• src/engine/payload.rs — the structural pivot of this PR. PayloadTypes lost type PayloadBuilderAttributes, so BerachainPayloadBuilderAttributes is deleted and the custom sha256 payload-ID derivation moved into the new required PayloadAttributes::payload_id(&self, parent_hash) (src/engine/payload.rs:37). Consensus-critical; the pre-existing berachain_payload_id unit-test vectors are preserved and pass. Also adds the now-required From<BerachainBuiltPayload> for BerachainExecutionData.
• src/engine/mod.rs — block_to_payload gained a bal: Option<Bytes> parameter; ExecutionPayload gained gas_limit()/slot_number().
• src/engine/builder.rs — PayloadConfig now carries payload_id; BuildArguments gained execution_cache/state_root_handle (unused here); execute_transaction returns GasOutput; finish(state_provider, None) takes the new precomputed-state-root argument; mark_invalid takes owned errors.
• src/engine/rpc.rs — EngineApi::new takes reth_tasks::Runtime instead of a boxed TaskSpawner. v2.4.0's default engine capabilities flow through; the existing engine_getPayloadV5 removal (pre-Osaka UnsupportedFork) is retained because beacon-kit ≥ v1.4.1 treats EL HTTP 4xx as fatal.
• src/evm/mod.rs — revm 41: Evm gained the required cfg_env(); the system-call path uses MainnetHandler::run_system_call/inspect_run_system_call; ExecutionResult::Success now carries ResultGas (EIP-8037 gas split), and the PoL transact_raw path zeroes it via ResultGas::default().
• src/node/evm/executor.rs, config.rs, builder.rs — commit_transaction returns GasOutput; BlockExecutorFactory gained type TxExecutionResult and the Executor GAT with a StateDB bound; BlockEnv gained slot_num; BuildPendingEnv gained a BlockOverrides parameter.
• src/consensus/mod.rs — FullConsensus::validate_block_post_execution gained a fourth parameter (block_access_list_hash, forwarded to the inner Ethereum validator); ConsensusError::Other is now Arc<dyn Error>, so all string sites use ConsensusError::msg(...).
• src/pool/transaction.rs — implements the new PoolTransaction::consensus_ref(); the pool tx now stores Recovered<BerachainTxEnvelope>.
• src/rpc/api.rs — alloy 2.x orphan rules: the alloy_network::Network/TransactionBuilder impls are replaced by impl reth_rpc_convert::RpcTypes for BerachainNetwork (src/rpc/api.rs:101); TaskSpawner → Runtime; adds send_pool_transaction and the fully-defaulted GetBlockAccessList/EthSubscriptions marker impls.

---

Tests

169 tests, all passing. New and changed coverage:

• tests/e2e/storage_v2_test.rs (new) — five binary-driven tests via CARGO_BIN_EXE:
  • test_fresh_datadir_defaults_to_storage_v2
  • test_storage_v2_flag_opts_new_datadir_into_v1
  • test_v1_datadir_remains_readable
  • test_migrate_v2_converts_v1_datadir_in_place
  • test_migrate_v2_is_idempotent
• Golden db-format tests in src/primitives/header.rs and src/transaction/mod.rs pin the exact on-disk bytes, so any future codec drift fails loudly.
• tests/e2e/mod.rs, osaka_engine_api_test.rs — migrated to the v2 e2e harness (BerachainPayloadAttributes generator, Runtime::test(), BuildNewPayload/next_attributes()).

---

CI / ops

• deny.toml — drops five ignores resolved by the upgrade (hickory-proto ×2, git2 ×2 via vergen-git2 10, core2); adds dev-only RUSTSEC-2023-0089 (atomic-polyfill via test-fuzz).
• .github/workflows/sync.yml — beacon-kit genesis/peer URLs pinned to v1.4.2-rc.0; adds a db settings step, making the nightly fresh-datadir sync a Storage V2 canary.
• Hive assets (.github/assets/hive/) — re-synced from reth v2.4.0 with the Berachain deltas re-applied; expected_failures.yaml/ignored_tests.yaml headers note they were last tuned against v1.11.4 and need re-validation once a v2.4.0-based nightly image exists.
• Dockerfiles ×3, Cross.toml — cargo-chef pinned to a Rust ≥ 1.95 tag with an MSRV comment.

---

Documentation

• README.md — Rust 1.95+ prerequisite; new "Running with BeaconKit" section (v1.4.2 two-terminal flow, JWT/genesis wiring, why --engine.persistence-threshold 0 and --engine.memory-block-buffer-target 0); new "Storage" section.
• docs/storage-v2.md (new) — full operator guide for the V1 → V2 transition.
• CLAUDE.md — refreshed reference versions (reth v2.4.0, beacon-kit v1.4.2), corrected file paths, Storage V2 note.

---

TODOs

Still needs operator validation outside this environment:

1. A real bepolia/mainnet datadir migrate-v2 run.
2. beacon-kit pairing via scripts/test-block-progression.sh.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5ac5c7b4-0cb4-4ccb-899b-8feb7a23b7e2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch reth-v2-upgrade

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[fridrik01]

As mentioned on our last sync, I would prefer we try to keep reth 2.0 state compatible so we can roll this out without an EL hard fork. We can consider updating the gaslimit in future more lightweight el hf.

[calbera]

yup, removed here 88cbc6c

[fridrik01]

@calbera update on this?

[calbera]

@calbera update on this?

Still testing.

migrate-v2 seems to be working (migrates a reth storage v1 node) on both bepolia and mainnet. nodes are syncing after the migration as normal

@camembera found an issue with taking snapshots of a v2 node (or rather downloading that snapshot) but its not bera-reth specific. filed an upstream bug here.

Next steps are running this live on a devnet and migrating validator nodes from v1 to v2 incrementally and ensure network stays live.

Would definitely help to start code review.. I'll open this PR once these tests are done.

[fridrik01]

Quick review, looks good, will wait for devnet testing and verification for a more detailed review.