diff --git a/CHANGELOG.md b/CHANGELOG.md index c3229e8..6808c8f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,96 @@ pre-release development phases (see [`docs/ROADMAP.md`](docs/ROADMAP.md)). - **Freshness primitives on `EvmCache`** — `verify_slots`, `purge_account`, `set_storage_batch_fetcher`; `EvmOverlay::call_raw_with_access_list` and `override_slot` for read-set capture and corrected re-runs. +- **State-update vocabulary & apply primitive** (`state_update` module, Phase 3, + Pillar B.1) — a generic `StateUpdate` enum (`Slot` / partial-`AccountPatch` + `Account` / `Purge` by `PurgeScope`) plus `EvmCache::apply_update` / + `apply_updates`, the single dual-layer write-through primitive (backend always, + overlay-if-present, no new overlay account materialized), returning a structured + `StateDiff` (`SlotChange`s, `AccountChange`s, `PurgeRecord`s) that records only + actual changes. The existing `inject_storage_batch_fresh` / `purge_account` / + `purge_pool_storage` / `purge_pool_slots` writers and the freshness + correction-drain are refolded onto it (signatures unchanged); generic, builds + with `--no-default-features`. +- **Relative / read-modify-write state updates** (`state_update`, Phase 3 §15) — + a saturating `SlotDelta` (`Add`/`Sub`, clamping at `U256::MAX`/`U256::ZERO`), a + `StateUpdate::SlotDelta { address, slot, delta }` variant (with the + `StateUpdate::slot_delta` constructor) so deltas flow through `apply_updates`, + and `EvmCache::modify_slot(address, slot, |Option| -> Option)` as + the general closure escape hatch. Relative application is **cold-aware**: a + delta against a slot absent from both layers is not applied (it would corrupt an + unknown value) but surfaced in the new `StateDiff.skipped: Vec` + field for the caller to fetch+seed and retry. `skipped` is informational + metadata and does not affect `StateDiff::is_empty` / `len` (changes-only). + Adding the `StateDiff.skipped` field is a struct change permitted under the + pre-1.0 break policy. Generic core (builds `--no-default-features`). +- **Post-audit state-update remediation** (`state_update`, Phase 3 §16): + - **`serde`** — `Serialize`/`Deserialize` derived (unconditionally) on the whole + vocabulary (`SlotDelta`, `StateUpdate`, `AccountPatch`, `PurgeScope`) and the + diff (`StateDiff`, `AccountChange`, `PurgeRecord`, `SkippedDelta`, + `SkippedBalanceDelta`) plus `freshness::SlotChange`, so updates can be shipped + over the wire and diffs persisted. + - **`#[non_exhaustive]`** on `StateDiff` and `AccountPatch` (both + `Default`/builder-constructed), so future field additions are non-breaking. The + leaf record types (`SlotChange`/`AccountChange`/`PurgeRecord`/`SkippedDelta`/ + `SkippedBalanceDelta`) are deliberately left exhaustive — they are routinely + built as struct literals in equality assertions. + - **Relative native-balance updates** — a `StateUpdate::BalanceDelta { address, + delta: SlotDelta }` variant (with `StateUpdate::balance_delta`), the + `EvmCache::modify_account_balance(addr, |Option| -> Option)` + closure escape hatch, a new `StateDiff.skipped_balances: + Vec` field, and `SkippedBalanceDelta`. Cold-aware: a delta + on an account absent from both layers is skipped and surfaced (never + materialized). Adding `skipped_balances` is a struct change permitted under the + pre-1.0 break policy. + - **Discoverable skip accessors** — `StateDiff::has_skipped()` / `skipped_len()` + / `is_fully_applied()`, counting **both** `skipped` and `skipped_balances`, so a + silently-dropped cold relative update is easy to detect (the changes-only + `is_empty()`/`len()` do not reflect skips). + - **Constructor symmetry** — `StateUpdate::nonce(addr, u64)`, + `StateUpdate::code(addr, Bytes)`, `StateUpdate::account(addr, AccountPatch)`. +- **Batched single-lock fast-path for `apply_updates`** (Phase 3 §16.9): a run of + consecutive `Slot`/`SlotDelta` writes now holds the backend storage write-guard + once for the run (the guard is dropped before any `Account`/`BalanceDelta`/ + `Purge` update to avoid deadlocking the non-reentrant `RwLock`, then re-acquired), + and the `SlotDelta` double-read of the old value is eliminated. The result is + byte-identical to folding `apply_update` over the batch (pinned by the + batched==sequential equivalence test). Generic core. +- **Event → state pipeline** (`events` module, Phase 4, Pillar B.2 — the *reader + half* of the event pipeline) — turn on-chain logs into the Phase 3 `StateUpdate` + vocabulary and drive them through the cache for reactive freshness: + - **`EventDecoder` / `StateView`** — a decoder is a pure function of + `(log, pre-state)` returning `Vec`; the narrow read-only + `StateView` (implemented by `EvmCache` via `cached_storage_value`) lets + stateful adapters read current cached state without RPC. Generic core. + - **`DecoderRegistry`** — dispatches a log to the decoders registered for its + emitting address (plus globals) and concatenates their output. Generic core. + - **`Erc20TransferDecoder`** — decodes ERC-20 `Transfer` logs into relative + balance `SlotDelta`s (skipping the zero-address mint/burn leg), with per-token + balance-slot config. The reactive-balance case from Phase 3 §15, now + log-driven. Generic core. + - **`UniswapV3Decoder` / `UniswapV3Layout`** (`protocols`) — `Swap` → a masked + `slot0` write (new `sqrtPriceX96` + `tick`, **preserving** the + observation/fee/`unlocked` bits — a clobbered `unlocked` would make a quote + revert `LOK`) plus an absolute `liquidity` write; `Mint`/`Burn` → per-tick + `liquidityGross`/`liquidityNet`, the `initialized` flag, the `tickBitmap` + word bit, and the in-range global `liquidity`, computed against the + `StateView` and cold-aware. Uniswap and PancakeSwap layouts. + - **`EventPipeline`** — `ingest_logs` decodes + applies a block's logs + **log-by-log in order** (so a later log sees earlier applies) and returns a + `BlockDigest`; `reorg_to` purges (purge-and-resync) the addresses touched + after a new head; `reconcile` re-reads sampled event-derived slots against + chain truth (correct **and** alarm) via the new `EvmCache::reconcile_slots`. A + thin async `drive`/`LogSource` convenience layers the synchronous core over a + stream. Generic core. +- **`StateUpdate::SlotMasked`** (`state_update`, Phase 4) — a cold-aware + read-modify-write *masked* slot write (`new = (old & !mask) | (value & mask)`) + with the `StateUpdate::slot_masked` constructor, so a pure decoder can update + selected bits of a **packed** storage word (e.g. V3 `slot0`) without clobbering + the rest. A masked write to a cold slot is skipped and surfaced in the new + `StateDiff.skipped_masks: Vec` (counted by `has_skipped` / + `skipped_len`, not by the changes-only `is_empty`/`len`); `serde` on + `SkippedMask`. Adding the variant and the field is permitted under the pre-1.0 + break policy (`StateUpdate`/`StateDiff` are `#[non_exhaustive]`). Generic core. - **Configurable transaction & block environment** — `TxConfig` (value, gas limit, gas price, nonce, access list) threaded through `call_raw_with`; block context setters (`set_coinbase`, `set_prevrandao`, `set_block_gas_limit`). @@ -52,12 +142,170 @@ pre-release development phases (see [`docs/ROADMAP.md`](docs/ROADMAP.md)). - **`protocols` feature** (default-on) gating the Uniswap V2/V3 storage layouts, V3 tick snapshots, and `inject_v3_*` / `inject_v2_pool_metadata` helpers, so the generic engine builds with `--no-default-features`. +- **Copy-on-write snapshots** (Phase 5, Pillar A) — `create_snapshot` is now a + two-tier copy-on-write view instead of an O(total state) deep clone. The cold + `BlockchainDb` index (layer 2) is flattened once into an internal, immutable, + `Arc`-shared base (`Arc` per account storage map, structural sharing — no new + dependency, Decision D1), memoized across snapshots and rebuilt copy-on-write + only for the addresses that changed; each `create_snapshot` then folds just the + hot CacheDB delta (layer 1) over a cheap `Arc::clone` of that base. Reads stay + O(1) and lock-free and are bit-for-bit identical to the deep clone (pinned by + the `tests/cow_snapshot.rs` differential-equivalence gate). The retained + `EvmCache::create_snapshot_deep_clone()` (`#[doc(hidden)] pub`, Decision D3) is + the equivalence reference and the A/B benchmark baseline. `EvmSnapshot` stays + `Send + Sync` and `EvmOverlay` stays `Send`. +- **`EvmOverlay::reset()`** (Phase 5, Pillar A.2) — recycle one overlay across + many simulations against the same snapshot without reallocating: it clears the + per-simulation dirty layer (keeping the snapshot `Arc`, `ext_db`, and the + reusable shared-memory buffer), reading the pristine snapshot again and behaving + exactly like a freshly-built overlay. The 64 KiB shared-memory buffer is also + recycled across the build→transact→revert call methods (stored as a plain + `Vec`, so the overlay stays `Send`). +- **Configurable EVM shared-memory pre-allocation** — `SharedMemoryCapacity` + (`Fixed(usize)` / `Auto`, default `Fixed(64 * 1024)` / 65,536 bytes) set via + `EvmCacheBuilder::shared_memory_capacity`. `Fixed` pins the per-context working- + memory buffer (general users running wide fan-outs of small simulations can lower + it to cut per-overlay memory; the previous behavior is the default); `Auto` sizes + it from the chain state loaded at build time (e.g. a bincode state file), clamped + to a 64 KiB floor / 4 MiB ceiling. The resolved size is readable via + `EvmCache::shared_memory_capacity()` and is propagated to every snapshot so + snapshot-backed overlays pre-allocate the same amount. `with_cache_capacity` is + the lower-level constructor behind the builder setter. +- **Explicit cold-account materialization** — `StateUpdate::AccountUpsert` and + `StateUpdate::account_upsert(...)` intentionally materialize an account absent + from both layers. Normal `StateUpdate::Account` patches are now cold-aware and + surface skipped cold patches through `StateDiff.skipped_accounts: + Vec`. +- **Invalidating layer-2 mutation wrapper** — `EvmCache::with_blockchain_db_mut` + runs a synchronous direct `BlockchainDb` mutation and invalidates the Phase 5 + memoized COW base automatically after the closure returns. +- **Exact access-list RLP data-gas helper** — + `access_list::access_list_rlp_data_gas(&AccessList)` returns the EIP-2930 RLP + calldata gas for an access list and backs the L2 profitability calculation. +- **Versioned on-disk cache envelope** — binary EVM state, bytecode, + `ImmutableDataCache`, and V3 tick snapshot cache files now start with + crate-specific magic bytes plus a `u32` version before the bincode payload. ### Changed +- **`EvmCache::create_snapshot` is now `&mut self`** (Phase 5, Decision D5) — + taking a snapshot memoizes/refreshes the cold copy-on-write base, which requires + a mutable borrow. All callers (the freshness controller, tests, examples, + benches) are updated; the return type (`Arc`) is unchanged. + Permitted under the pre-1.0 break policy. +- **`EvmCache::inject_storage_batch` is now `&mut self`** (Phase 5) — the + layer-2 bulk write now marks the touched addresses dirty for the memoized + copy-on-write base. The write itself is still a direct backend (layer-2) write + with the same semantics; only the receiver mutability changed. +- **Raw layer-2 handles were renamed to unchecked accessors** (Phase 5) — + `EvmCache::blockchain_db()` is now `unchecked_blockchain_db()` and + `EvmCache::backend()` is now `unchecked_backend()`. The rename makes the + bypass explicit; use `with_blockchain_db_mut` for synchronous direct writes that + should automatically invalidate the snapshot base. +- **Persistence APIs now return `Result<()>`** — `cache::save_binary_state`, + `PrefetchRegistry::save`, and `EvmCache::flush` report serialization, + directory-creation, and write failures to explicit callers. `Drop` remains + best-effort and logs `flush()` errors. +- **Block re-pins clear stale context** — `set_block` sets `block_number` only + for concrete numeric pins, clears it for tag/hash/`None` pins, and clears stale + `basefee` on block changes and on non-concrete pin calls that can drift under + the same tag. `repin_to_block` follows the same no-stale-basefee rule; callers + refresh `NUMBER`/`BASEFEE` via `set_block_context` after fetching the new + header. +- **Legacy raw-bincode cache files are treated as misses** — the versioned cache + envelope intentionally rejects unversioned `evm_state.bin`, `bytecodes.bin`, + `immutable_data.bin`, and `v3_tick_snapshots.bin` payloads rather than trying + to deserialize ambiguous layouts. - Simulation entry points that distinguish failure modes return `SimulationResult` (`Result`), separating decoded reverts, EVM halts, and host errors. `SimulationErrorKind` remains as a deprecated alias. +- **`inject_v2_pool_metadata` / `inject_v3_tick_bitmap*` / `inject_v3_ticks*` + (`protocols`) now write through both cache layers** (Phase 3, Decision 2). + Previously these wrote only the CacheDB overlay (layer 1); they are now folded + onto the write-through `StateUpdate::Slot` primitive, so the injected slots also + land in the BlockchainDb backend (layer 2). Signatures and return values are + unchanged and the visible `token0()`/`tickBitmap()`/`ticks()` reads are the + same; only the slot *placement* across layers changed. See + [`docs/KNOWN_ISSUES.md`](docs/KNOWN_ISSUES.md). (The cold-backfill + `inject_storage_batch` keeps its layer-2-only intent and is unchanged.) + +### Fixed + +- **Cold absolute account patches no longer mask on-chain accounts.** + `StateUpdate::Account` on an account absent from both layers now skips instead + of writing `AccountInfo::default()` fields through the shared backend. The + skipped patch is visible in `StateDiff.skipped_accounts`; intentional cold + creation uses `StateUpdate::AccountUpsert`. +- **Access-list profitability no longer conflates provider failures with + unprofitable lists.** `SmartAccessList::into_access_list_if_profitable` and + `access_list_if_profitable` now propagate provider/pricing failures as `Err` + and reserve `Ok(None)` for empty, zero-priced, or genuinely unprofitable lists. +- **`simulate_call_with_balance_deltas` now reports a real access list.** It + extracts the EIP-2930 touched account/slot list from the EVM journal before + commit/revert, including the pre/post `balanceOf` reads and the simulated call, + instead of returning `AccessList::default()`. +- **`cached_storage_value` silent-corruption bug** (Phase 3 §16.0, audit HIGH + + MED). For a storage slot absent from an overlay account whose revm + `account_state` is `StorageCleared` or `NotExisting`, the accessor now returns + `Some(U256::ZERO)` — mirroring what the live EVM `SLOAD`s + (`CacheDB::storage_ref`) — instead of falling through to the BlockchainDb + backend and returning a *shadowed* backend value the EVM never sees. The old + behavior let a `SlotDelta` / `modify_slot` compute a relative update against a + base the EVM never reads (silent state corruption) and mis-recorded + `apply_slot`'s `SlotChange.old` / change predicate. This also closes the + same-root mismatch shared by `verify_slots` / `inject_storage_batch_fresh`. +- **No-op `Account` patch no longer materializes a backend account** (Phase 3 + §16.1, audit LOW). `apply_account_patch` now computes the field change first and + **skips both layer writes** (returning an empty diff) when no field actually + changes, instead of unconditionally inserting `AccountInfo::default()` into the + shared backend for an all-`None` (or value-unchanged) patch on an absent address. + Phase 5 later tightened this further: real field changes on cold accounts now + skip through `StateDiff.skipped_accounts` unless the caller uses + `StateUpdate::AccountUpsert`. +- **`account_state`-awareness extended to the snapshot + account-info paths** + (Phase 3 fix-review, HIGH + MED). A follow-up adversarial review found the §16.0 + `cached_storage_value` fix had not been propagated to two sibling read paths: + - `create_snapshot` now mirrors the live read: a `StorageCleared`/`NotExisting` + account's storage is captured as **only** its overlay slots (shadowed backend + slots dropped) and recorded in a new `EvmSnapshot.storage_cleared` set, so + `EvmSnapshot::storage_value` and snapshot-backed `EvmOverlay`s read such a + slot as ZERO instead of the shadowed backend value (which also kept the + background freshness validator's `old` consistent with `verify_slots`). The + `EvmOverlay` storage read honors the set and does **not** fall through to its + `ext_db` for a cleared account. + - `loaded_account_info` now mirrors revm `DbAccount::info()`: a `NotExisting` + overlay account is treated as absent (returns `None`), so a `BalanceDelta` / + partial `Account` patch skips rather than computing against a stale `info`. + - `write_account_info_through` normalizes a `ZERO` `code_hash` to `KECCAK_EMPTY` + so both cache layers store an identical hash (matching revm's `insert_contract`). +- **`account_state`-awareness completed on the account (`basic`) axis** (round-2 + review, HIGH). The snapshot path still leaked a `NotExisting` account's stale + info: `create_snapshot` inserted it into `accounts`, so `EvmOverlay::basic` + returned a phantom existing account where live revm / `loaded_account_info` + return `None`. Now `create_snapshot` excludes `NotExisting` accounts from + `accounts`/`code_by_hash` and records them in a new + `EvmSnapshot.accounts_not_existing` set; `EvmOverlay::basic` returns `None` for + them (no `ext_db` fall-through). `target_account_info` (deploy path) and + `loaded_account_info` (code_hash normalized at load) were brought into line too. +- **Freshness validator trust contract hardened** (Phase 2 review). The + background validator no longer returns a *trusted* verdict on incomplete or + ambiguous verification: + - **Fixed-point round cap → `Unverified`.** Exceeding `MAX_VALIDATION_ROUNDS` + (corrections kept opening new volatile slots) now returns + `Validation::Unverified` and queues no corrections, instead of a best-effort + `Corrected` resting on un-verified state. + - **Corrected re-run host error → `Unverified`.** A failed corrected re-run + (a `transact` error, not a revert/halt) returns `Unverified` rather than + silently keeping the stale optimistic result. + - **Missing fetcher results → `Unverified`.** A new `collect_fetch_results` + helper requires the batch fetcher to return *every* requested slot; an omitted + slot yields `Unverified` instead of defaulting to zero (which could produce a + false confirmation/correction with a custom fetcher). +- **`call_raw_with_access_list*` reverts its checkpoint on transact errors** + (Phase 2 review; `EvmCache` + `EvmOverlay`). Previously the host-error path + `?`-returned before `checkpoint_revert`, leaving the journal checkpoint + un-reverted; both methods now revert on every path. (See `docs/KNOWN_ISSUES.md` + #9, now resolved.) ### Notes diff --git a/Cargo.lock b/Cargo.lock index db4da26..dc8eaf5 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1852,6 +1852,7 @@ dependencies = [ "alloy-node-bindings", "alloy-primitives", "alloy-provider", + "alloy-rlp", "alloy-rpc-client", "alloy-rpc-types-eth", "alloy-sol-types", diff --git a/Cargo.toml b/Cargo.toml index 6ccbb62..5f7cf9b 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -38,6 +38,7 @@ alloy-eips = "1.0.38" alloy-network = "1.0.38" alloy-primitives = { version = "1.4", features = ["map"] } alloy-provider = "1.0.38" +alloy-rlp = "0.3" alloy-rpc-client = "1.0.38" alloy-rpc-types-eth = "1.0.38" alloy-sol-types = "1.4" @@ -86,6 +87,14 @@ harness = false name = "freshness" harness = false +[[bench]] +name = "state_update" +harness = false + +[[bench]] +name = "event_pipeline" +harness = false + # RPC-gated real-contract benchmarks. Skipped (not failed) when RPC_URL is unset, # so `cargo bench` stays offline by default. [[bench]] diff --git a/README.md b/README.md index 3921531..2d97a5c 100644 --- a/README.md +++ b/README.md @@ -168,6 +168,8 @@ and inject all state directly: | `prefetch_registry` | Advanced | Record and persist storage touch sets for cross-cycle prefetch. | | `freshness_optimistic` | Advanced | Optimistic verify-and-rerun loop: a `Corrected` validation via a stub fetcher. | | `freshness_multi_sim` | Advanced | Many sims with selective re-run, plus classification and `ValidThrough` aging. | +| `state_update_apply` | Advanced | Apply a mixed `StateUpdate` batch (`Slot`/`Account`/`Purge`) and inspect the returned `StateDiff`. | +| `reactive_cache` | Advanced | Decode logs (ERC-20 `Transfer` + UniswapV3 `Swap`) into `StateUpdate`s, ingest a block, reconcile drift, and purge on a reorg. | **RPC examples** fork real mainnet state. Set `RPC_URL` to an Ethereum RPC endpoint (they print instructions and exit if it is unset): @@ -215,15 +217,17 @@ println!("installed {} bytes at {}", etched.code_size, etched.target_address); ## Benchmarks -Criterion benchmarks live in [`benches/`](benches). The offline benches are the -baseline against which the planned copy-on-write snapshot rewrite (roadmap -Pillar A) will be measured, so they exercise the real hot paths at a range of -cache sizes: +Criterion benchmarks live in [`benches/`](benches). The offline benches exercise +the current hot paths at a range of cache sizes, including the Phase 5 +copy-on-write snapshot implementation and retained deep-clone baselines where +useful for A/B comparison: | Bench | Measures | | --- | --- | | `simulation` | `create_snapshot` across cache sizes (100 → 10k accounts), overlay fan-out, `call_raw` throughput, sequential bundle execution, batched storage injection. | | `freshness` | The optimistic loop end-to-end (CPU and latency-hiding), `verify_slots` at scale (1 → 1000 slots), and multi-sim fan-out. | +| `state_update` | `apply_updates` throughput across batch sizes (1 → 1000 `Slot`s) and per-variant apply cost (`Slot` vs `Account` vs `Purge`). | +| `event_pipeline` | Per-event decode cost (ERC-20 `Transfer`, V3 `Swap`/`Mint`), `ingest_logs` decode+apply throughput (1 → 1000 logs), and `reorg_to` purge cost. | | `access_list` | Touch-set merge and EIP-2930 list construction. | | `revert_decoding` | Built-in and custom revert decoding, including decoder dispatch with many registered errors. | | `storage_keys` | Mapping/array storage-key derivation. | diff --git a/benches/event_pipeline.rs b/benches/event_pipeline.rs new file mode 100644 index 0000000..9ddf859 --- /dev/null +++ b/benches/event_pipeline.rs @@ -0,0 +1,240 @@ +//! Phase 4 benchmarks: the event → state pipeline (Pillar B.2). +//! +//! Measures three things, all offline (mocked provider, in-memory logs): +//! - **decode** cost per event kind (ERC-20 `Transfer`, UniswapV3 `Swap`/`Mint`), +//! isolating the pure `EventDecoder::decode` work (no apply); +//! - **ingest** throughput — [`EventPipeline::ingest_logs`] decoding **and** +//! applying a block of logs, across batch sizes (1 → 1000); +//! - **reorg** purge cost — [`EventPipeline::reorg_to`] over a touched set of +//! 1 → 1000 addresses. +//! +//! A current-thread runtime drives only the async cache constructor; the pipeline +//! itself is synchronous and never touches the network. + +use std::collections::HashMap; +use std::hint::black_box; +use std::sync::Arc; + +use alloy_primitives::aliases::{I24, U160}; +use alloy_primitives::{Address, Bytes, I256, Log, U256, hex, keccak256}; +use alloy_provider::RootProvider; +use alloy_provider::network::AnyNetwork; +use alloy_rpc_client::RpcClient; +use alloy_sol_types::{SolEvent, sol}; +use alloy_transport::mock::Asserter; +use criterion::{BenchmarkId, Criterion, Throughput, criterion_group, criterion_main}; +use evm_fork_cache::cache::{EvmCache, V3_SLOT0_SLOT, v3_tick_info_storage_keys_with_base}; +use evm_fork_cache::events::{DecoderRegistry, EventDecoder, EventPipeline, StateView}; +use evm_fork_cache::{Erc20TransferDecoder, StateUpdate, UniswapV3Decoder, UniswapV3Layout}; +use revm::state::{AccountInfo, Bytecode}; +use tokio::runtime::{Builder, Runtime}; + +const MOCK_ERC20_RUNTIME_HEX: &str = include_str!("../fixtures/mock_erc20_runtime.hex"); +const TOKEN: Address = Address::repeat_byte(0xAA); +const POOL: Address = Address::repeat_byte(0xBB); + +fn current_thread_rt() -> Runtime { + Builder::new_current_thread().enable_all().build().unwrap() +} + +/// A cache with `TOKEN` and `POOL` installed as storage-cleared accounts (so +/// unseeded slots read as zero — no RPC fallthrough). +fn seeded_cache(rt: &Runtime) -> EvmCache { + let provider = RootProvider::::new(RpcClient::mocked(Asserter::new())); + let mut cache = rt.block_on(EvmCache::new(Arc::new(provider), None)); + let runtime = Bytecode::new_raw(Bytes::from( + hex::decode(MOCK_ERC20_RUNTIME_HEX.trim()).unwrap(), + )); + let code_hash = runtime.hash_slow(); + for addr in [TOKEN, POOL] { + cache.db_mut().insert_account_info( + addr, + AccountInfo { + balance: U256::ZERO, + nonce: 0, + code: Some(runtime.clone()), + code_hash, + account_id: None, + }, + ); + cache + .db_mut() + .replace_account_storage(addr, Default::default()) + .unwrap(); + } + cache +} + +sol! { + event Swap(address indexed sender, address indexed recipient, int256 amount0, int256 amount1, uint160 sqrtPriceX96, uint128 liquidity, int24 tick); + event Mint(address sender, address indexed owner, int24 indexed tickLower, int24 indexed tickUpper, uint128 amount, uint256 amount0, uint256 amount1); +} + +fn transfer_log(token: Address, from: Address, to: Address, value: U256) -> Log { + let sig = keccak256(b"Transfer(address,address,uint256)"); + Log::new_unchecked( + token, + vec![sig, from.into_word(), to.into_word()], + Bytes::copy_from_slice(&value.to_be_bytes::<32>()), + ) +} + +fn swap_log(pool: Address, sqrt_price: u128, liquidity: u128, tick: i32) -> Log { + let ev = Swap { + sender: Address::repeat_byte(0x01), + recipient: Address::repeat_byte(0x02), + amount0: I256::try_from(-1i64).unwrap(), + amount1: I256::try_from(1i64).unwrap(), + sqrtPriceX96: U160::from(sqrt_price), + liquidity, + tick: I24::try_from(tick).unwrap(), + }; + Log { + address: pool, + data: ev.encode_log_data(), + } +} + +fn mint_log(pool: Address, lower: i32, upper: i32, amount: u128) -> Log { + let ev = Mint { + sender: Address::repeat_byte(0x03), + owner: Address::repeat_byte(0x04), + tickLower: I24::try_from(lower).unwrap(), + tickUpper: I24::try_from(upper).unwrap(), + amount, + amount0: U256::from(1), + amount1: U256::from(1), + }; + Log { + address: pool, + data: ev.encode_log_data(), + } +} + +/// A bench-local read-only [`StateView`] over a fixed map (for the V3 `Mint` +/// decode, which reads the current tick word). +struct MapView(HashMap<(Address, U256), U256>); +impl StateView for MapView { + fn storage(&self, address: Address, slot: U256) -> Option { + self.0.get(&(address, slot)).copied() + } +} + +/// A bench-local decoder that emits one absolute `Slot` write per log, keyed by +/// the log's address — so repeated ingest is idempotent (stable across iters). +struct AbsDecoder; +impl EventDecoder for AbsDecoder { + fn decode(&self, log: &Log, _view: &dyn StateView) -> Vec { + vec![StateUpdate::slot(log.address, U256::from(0), U256::from(1))] + } +} + +/// Pure `decode` cost per event kind (no apply). +fn bench_decode(c: &mut Criterion) { + let mut group = c.benchmark_group("decode"); + + let erc20 = Erc20TransferDecoder::new(U256::from(3)); + let tlog = transfer_log( + TOKEN, + Address::repeat_byte(0x21), + Address::repeat_byte(0x22), + U256::from(100), + ); + let empty = MapView(HashMap::new()); + group.bench_function("erc20_transfer", |b| { + b.iter(|| black_box(erc20.decode(black_box(&tlog), &empty))) + }); + + let v3 = UniswapV3Decoder::new().with_pool(POOL, UniswapV3Layout::uniswap(60)); + let slog = swap_log(POOL, 2_000_000, 7_500, 120); + let mut slot0_view = HashMap::new(); + slot0_view.insert( + (POOL, V3_SLOT0_SLOT), + (U256::from(1u64) << 240) | U256::from(1_000_000u64), + ); + // Seed the tick words the Mint reads (lower/upper) so it computes (not skips). + let lo = v3_tick_info_storage_keys_with_base(60, evm_fork_cache::cache::V3_TICKS_BASE_SLOT)[0]; + let hi = v3_tick_info_storage_keys_with_base(120, evm_fork_cache::cache::V3_TICKS_BASE_SLOT)[0]; + slot0_view.insert((POOL, lo), U256::ZERO); + slot0_view.insert((POOL, hi), U256::ZERO); + let view = MapView(slot0_view); + group.bench_function("v3_swap", |b| { + b.iter(|| black_box(v3.decode(black_box(&slog), &view))) + }); + let mlog = mint_log(POOL, 60, 120, 1_000); + group.bench_function("v3_mint", |b| { + b.iter(|| black_box(v3.decode(black_box(&mlog), &view))) + }); + + group.finish(); +} + +/// `ingest_logs` decode+apply throughput as the per-block log batch grows. +fn bench_ingest_batch(c: &mut Criterion) { + let rt = current_thread_rt(); + let mut cache = seeded_cache(&rt); + + let mut group = c.benchmark_group("ingest_logs"); + for &n in &[1usize, 10, 100, 1_000] { + let logs: Vec = (0..n) + .map(|i| { + Log::new_unchecked( + Address::repeat_byte((i % 251 + 1) as u8), + vec![], + Bytes::new(), + ) + }) + .collect(); + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(AbsDecoder)); + let mut pipeline = EventPipeline::new(registry); + + group.throughput(Throughput::Elements(n as u64)); + group.bench_with_input(BenchmarkId::from_parameter(n), &logs, |b, logs| { + let mut block = 0u64; + b.iter(|| { + block += 1; + black_box(pipeline.ingest_logs(&mut cache, block, black_box(logs))) + }) + }); + } + group.finish(); +} + +/// `reorg_to` purge cost over a touched set of N distinct addresses. +fn bench_reorg(c: &mut Criterion) { + let rt = current_thread_rt(); + + let mut group = c.benchmark_group("reorg_to"); + for &n in &[10usize, 100, 1_000] { + group.throughput(Throughput::Elements(n as u64)); + group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| { + b.iter_batched( + || { + // Setup: a cache + pipeline with N addresses touched at block 1. + let mut cache = seeded_cache(&rt); + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(AbsDecoder)); + let mut pipeline = EventPipeline::new(registry); + let logs: Vec = (0..n) + .map(|i| { + let mut bytes = [0u8; 20]; + bytes[0..8].copy_from_slice(&(i as u64).to_be_bytes()); + Log::new_unchecked(Address::from(bytes), vec![], Bytes::new()) + }) + .collect(); + pipeline.ingest_logs(&mut cache, 1, &logs); + (pipeline, cache) + }, + |(mut pipeline, mut cache)| { + black_box(pipeline.reorg_to(&mut cache, 0)); + }, + criterion::BatchSize::SmallInput, + ) + }); + } + group.finish(); +} + +criterion_group!(benches, bench_decode, bench_ingest_batch, bench_reorg); +criterion_main!(benches); diff --git a/benches/simulation.rs b/benches/simulation.rs index 10d0d2d..aca01e1 100644 --- a/benches/simulation.rs +++ b/benches/simulation.rs @@ -3,14 +3,25 @@ //! bundle simulation, and batched storage injection. //! //! These run fully offline (mocked provider) so they're reproducible. They -//! establish the baseline for the Pillar A (copy-on-write snapshot) rewrite: -//! `create_snapshot` is currently an O(total state) deep clone, so its cost -//! scales with the populated cache size (the `create_snapshot` group sweeps -//! 100 → 10,000 accounts to show that slope). Once Pillar A lands, the same -//! sweep should flatten toward O(changed state) — re-run this group before and -//! after to quantify the win. The `overlay_fanout` group measures the other -//! half of the value proposition: how cheaply one frozen snapshot fans out into -//! many isolated simulations. +//! quantify the Pillar A (copy-on-write snapshot) win. +//! +//! Expected shapes: +//! - **`create_snapshot` group (A/B).** The cold index is seeded into **layer 2** +//! via `inject_storage_batch` (`populated_cache_layer2`), the way a fork cache +//! actually holds it. For each size it benches both the COW `create_snapshot` +//! and the retained `create_snapshot_deep_clone`. The deep clone is an O(total +//! state) copy, so its cost slopes up with the index size; the COW path shares +//! the memoized base and avoids cloning total storage slots. It still scans +//! accounts and new layer-1 entries, so it should be much flatter than the deep +//! clone, especially as slots/account grows, but not strictly flat by account +//! count. +//! - **`resnapshot_hot_loop`.** Warms the base with one snapshot, applies a small +//! `apply_updates` layer-1 mutation, then measures `create_snapshot`. This is +//! the memoization win: the COW path avoids cloning cold storage slots but +//! remains sensitive to account scans and new layer-1 entries. +//! - **`overlay_fanout`.** Measures fanning one frozen snapshot out into many +//! isolated simulations, comparing a fresh `EvmOverlay::new` per sim against a +//! single `reset()`-recycled overlay (Pillar A.2). use std::hint::black_box; use std::sync::Arc; @@ -49,34 +60,35 @@ fn offline_cache(rt: &Runtime) -> EvmCache { rt.block_on(EvmCache::new(Arc::new(provider), None)) } -/// A cache populated with `accounts` accounts, each holding `slots_per` slots. -fn populated_cache(rt: &Runtime, accounts: usize, slots_per: usize) -> EvmCache { +/// A cache whose cold index lives in **layer 2** — seeded via +/// `inject_storage_batch`, the path a fork cache actually uses to bulk-load its +/// cold state. This is what the COW `create_snapshot` memoizes into its base. +fn populated_cache_layer2(rt: &Runtime, accounts: usize, slots_per: usize) -> EvmCache { let mut cache = offline_cache(rt); + let mut batch: Vec<(Address, U256, U256)> = Vec::with_capacity(accounts * slots_per); for a in 0..accounts { let address = addr(a); - cache - .db_mut() - .insert_account_info(address, AccountInfo::default()); for s in 0..slots_per { - cache - .db_mut() - .insert_account_storage( - address, - U256::from(s as u64), - U256::from((a * 31 + s) as u64), - ) - .unwrap(); + batch.push(( + address, + U256::from(s as u64), + U256::from((a * 31 + s) as u64), + )); } } + cache.inject_storage_batch(&batch); cache } +/// A/B snapshot creation across cold-index sizes: the COW `create_snapshot` vs. +/// the retained `create_snapshot_deep_clone`, both over a layer-2-seeded index. +/// +/// The deep clone slopes up with total slots; the COW path, after a warm-up +/// snapshot has memoized the base, avoids cloning those slots but still pays the +/// O(accounts) growth scan. fn bench_create_snapshot(c: &mut Criterion) { let rt = Runtime::new().unwrap(); let mut group = c.benchmark_group("create_snapshot"); - // Sweep from a small pool up to a production-scale index (10k contracts) so - // the O(total state) slope of the current deep clone is visible. Pillar A - // (copy-on-write) should flatten this curve. for &(accounts, slots) in &[ (100usize, 8usize), (1_000, 8), @@ -84,12 +96,50 @@ fn bench_create_snapshot(c: &mut Criterion) { (5_000, 16), (10_000, 16), ] { - let cache = populated_cache(&rt, accounts, slots); + let mut cache = populated_cache_layer2(&rt, accounts, slots); + // Warm the memoized base once so the COW measurement reflects the + // steady-state (reuse) cost, not the first full build. + black_box(cache.create_snapshot()); + group.throughput(criterion::Throughput::Elements((accounts * slots) as u64)); + group.bench_with_input( + BenchmarkId::new("cow", format!("{accounts}acct_x{slots}slot")), + &accounts, + |b, _| b.iter(|| black_box(cache.create_snapshot())), + ); + group.bench_with_input( + BenchmarkId::new("deep_clone", format!("{accounts}acct_x{slots}slot")), + &accounts, + |b, _| b.iter(|| black_box(cache.create_snapshot_deep_clone())), + ); + } + group.finish(); +} + +/// The memoization win: a hot re-snapshot loop. Warm the base once, apply a +/// *small* layer-1 mutation, then measure `create_snapshot`. Cost should track +/// account scanning plus new layer-1 entries, staying much flatter than the deep +/// clone as cold storage grows. +fn bench_resnapshot_hot_loop(c: &mut Criterion) { + let rt = Runtime::new().unwrap(); + let mut group = c.benchmark_group("resnapshot_hot_loop"); + for &(accounts, slots) in &[(1_000usize, 8usize), (5_000, 16), (10_000, 16)] { + let mut cache = populated_cache_layer2(&rt, accounts, slots); + // Warm the base. + black_box(cache.create_snapshot()); + // A handful of layer-1 writes (does not dirty the memoized base). + let target = addr(0); group.throughput(criterion::Throughput::Elements((accounts * slots) as u64)); group.bench_with_input( BenchmarkId::from_parameter(format!("{accounts}acct_x{slots}slot")), - &cache, - |b, cache| b.iter(|| black_box(cache.create_snapshot())), + &accounts, + |b, _| { + b.iter(|| { + cache + .db_mut() + .insert_account_info(target, AccountInfo::default()); + black_box(cache.create_snapshot()); + }) + }, ); } group.finish(); @@ -125,20 +175,31 @@ fn bench_overlay_fanout(c: &mut Criterion) { let mut group = c.benchmark_group("overlay_fanout"); for &k in &[1usize, 8, 32] { - group.bench_with_input( - BenchmarkId::from_parameter(format!("{k}way")), - &k, - |b, &k| { - b.iter(|| { - for _ in 0..k { - let mut overlay = EvmOverlay::new(snapshot.clone(), None); - let result = overlay.call_raw(owner, token, calldata.clone()).unwrap(); - debug_assert!(matches!(result, ExecutionResult::Success { .. })); - black_box(result); - } - }) - }, - ); + // Baseline: a fresh `EvmOverlay::new` (+ dirty maps + Arc clone + buffer) + // per simulation. + group.bench_with_input(BenchmarkId::new("new_per_sim", k), &k, |b, &k| { + b.iter(|| { + for _ in 0..k { + let mut overlay = EvmOverlay::new(snapshot.clone(), None); + let result = overlay.call_raw(owner, token, calldata.clone()).unwrap(); + debug_assert!(matches!(result, ExecutionResult::Success { .. })); + black_box(result); + } + }) + }); + // Pillar A.2: one overlay built once, `reset()` between sims (reuses the + // dirty maps, the snapshot Arc, and the shared-memory buffer). + group.bench_with_input(BenchmarkId::new("reset_recycled", k), &k, |b, &k| { + b.iter(|| { + let mut overlay = EvmOverlay::new(snapshot.clone(), None); + for _ in 0..k { + let result = overlay.call_raw(owner, token, calldata.clone()).unwrap(); + debug_assert!(matches!(result, ExecutionResult::Success { .. })); + black_box(result); + overlay.reset(); + } + }) + }); } group.finish(); } @@ -253,7 +314,7 @@ fn bench_sim_bundle(c: &mut Criterion) { /// Batched direct storage injection (the bypass-RPC write path) across sizes. fn bench_inject_storage_batch(c: &mut Criterion) { let rt = Runtime::new().unwrap(); - let cache = offline_cache(&rt); + let mut cache = offline_cache(&rt); let mut group = c.benchmark_group("inject_storage_batch"); for &n in &[100usize, 1_000, 10_000] { @@ -271,6 +332,7 @@ fn bench_inject_storage_batch(c: &mut Criterion) { criterion_group!( benches, bench_create_snapshot, + bench_resnapshot_hot_loop, bench_overlay_fanout, bench_cache_call_raw, bench_sim_bundle, diff --git a/benches/state_update.rs b/benches/state_update.rs new file mode 100644 index 0000000..8bf74e3 --- /dev/null +++ b/benches/state_update.rs @@ -0,0 +1,318 @@ +//! Phase 3 benchmarks: the targeted state-update apply primitive. +//! +//! Measures [`EvmCache::apply_updates`] throughput across batch sizes +//! (1 → 1000 `Slot` writes) and the per-variant cost of a single apply (`Slot` +//! vs `Account` patch vs `Purge`). The cache is built once per group; each +//! iteration re-uses it (the writes are idempotent / additive in-memory). +//! +//! Fully offline (mocked provider, state injected directly), so reproducible. +//! A current-thread runtime is used only to drive the async cache constructor; +//! `apply_updates` itself is synchronous and never touches the network. + +use std::hint::black_box; +use std::sync::Arc; + +use alloy_primitives::{Address, Bytes, U256, hex}; +use alloy_provider::RootProvider; +use alloy_provider::network::AnyNetwork; +use alloy_rpc_client::RpcClient; +use alloy_transport::mock::Asserter; +use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main}; +use evm_fork_cache::cache::EvmCache; +use evm_fork_cache::{AccountPatch, PurgeScope, SlotDelta, StateUpdate}; +use revm::state::{AccountInfo, Bytecode}; +use tokio::runtime::{Builder, Runtime}; + +const MOCK_ERC20_RUNTIME_HEX: &str = include_str!("../fixtures/mock_erc20_runtime.hex"); +const POOL: Address = Address::repeat_byte(0xAA); + +fn current_thread_rt() -> Runtime { + Builder::new_current_thread().enable_all().build().unwrap() +} + +/// A cache with `POOL` installed as a MockERC20 (overlay account present, so slot +/// writes exercise the overlay write-through branch too). +fn pool_cache(rt: &Runtime) -> EvmCache { + let provider = RootProvider::::new(RpcClient::mocked(Asserter::new())); + let mut cache = rt.block_on(EvmCache::new(Arc::new(provider), None)); + let runtime = Bytecode::new_raw(Bytes::from( + hex::decode(MOCK_ERC20_RUNTIME_HEX.trim()).unwrap(), + )); + let code_hash = runtime.hash_slow(); + cache.db_mut().insert_account_info( + POOL, + AccountInfo { + balance: U256::ZERO, + nonce: 0, + code: Some(runtime), + code_hash, + account_id: None, + }, + ); + // Mark storage local so unseeded slots read as zero (no RPC fallthrough). + cache + .db_mut() + .replace_account_storage(POOL, Default::default()) + .unwrap(); + cache +} + +/// `apply_updates` throughput as the `Slot` batch grows (1 → 1000). +fn bench_apply_slots_batch(c: &mut Criterion) { + let rt = current_thread_rt(); + let mut cache = pool_cache(&rt); + + let mut group = c.benchmark_group("apply_slots_batch"); + for &n in &[1usize, 10, 100, 1_000] { + // A fresh value each iteration is unnecessary; alternate two values so + // every apply records a real change (the worst case: full diff). + let updates_a: Vec = (0..n) + .map(|i| StateUpdate::slot(POOL, U256::from(i as u64), U256::from(1u64))) + .collect(); + let updates_b: Vec = (0..n) + .map(|i| StateUpdate::slot(POOL, U256::from(i as u64), U256::from(2u64))) + .collect(); + + group.throughput(Throughput::Elements(n as u64)); + group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, _| { + let mut toggle = false; + b.iter(|| { + let updates = if toggle { &updates_a } else { &updates_b }; + toggle = !toggle; + black_box(cache.apply_updates(black_box(updates))); + }) + }); + } + group.finish(); +} + +/// Per-variant cost of a single `apply_update`: `Slot` vs `Account` vs `Purge`. +fn bench_apply_per_variant(c: &mut Criterion) { + let rt = current_thread_rt(); + + let mut group = c.benchmark_group("apply_per_variant"); + + group.bench_function("slot", |b| { + let mut cache = pool_cache(&rt); + let mut toggle = false; + b.iter(|| { + let value = if toggle { U256::from(1) } else { U256::from(2) }; + toggle = !toggle; + black_box(cache.apply_update(black_box(&StateUpdate::slot( + POOL, + U256::from(0), + value, + )))); + }) + }); + + group.bench_function("account_balance", |b| { + let mut cache = pool_cache(&rt); + let mut toggle = false; + b.iter(|| { + let value = if toggle { U256::from(1) } else { U256::from(2) }; + toggle = !toggle; + black_box(cache.apply_update(black_box(&StateUpdate::Account { + address: POOL, + patch: AccountPatch::default().balance(value), + }))); + }) + }); + + // A relative SlotDelta on a hot slot (seeded once, additive each iter). + group.bench_function("slot_delta_hot", |b| { + let mut cache = pool_cache(&rt); + cache.inject_storage_batch(&[(POOL, U256::from(0), U256::from(1))]); + b.iter(|| { + // Add(0) keeps the value stable so the slot stays hot across iters. + black_box(cache.apply_update(black_box(&StateUpdate::slot_delta( + POOL, + U256::from(0), + SlotDelta::Add(U256::ZERO), + )))); + }) + }); + + // A relative SlotDelta on a cold slot (always skipped, never applied). + group.bench_function("slot_delta_cold", |b| { + let mut cache = pool_cache(&rt); + b.iter(|| { + // POOL is StorageCleared, so an unseeded slot reads ZERO (hot). Use a + // distinct address with no overlay account and no backend slot: cold. + black_box(cache.apply_update(black_box(&StateUpdate::slot_delta( + Address::repeat_byte(0xCD), + U256::from(0), + SlotDelta::Add(U256::from(1)), + )))); + }) + }); + + // The general closure read-modify-write escape hatch. + group.bench_function("modify_slot", |b| { + let mut cache = pool_cache(&rt); + cache.inject_storage_batch(&[(POOL, U256::from(0), U256::from(1))]); + b.iter(|| { + black_box(cache.modify_slot(POOL, U256::from(0), |cur| { + cur.map(|v| v.saturating_add(U256::ZERO)) + })); + }) + }); + + // An `Account` *code* patch: `Bytecode::new_raw` + `hash_slow` (a keccak over + // the code) — likely the most expensive single apply. Toggle two code blobs + // so each apply records a real change. + group.bench_function("account_code", |b| { + let mut cache = pool_cache(&rt); + let code_a = Bytes::from_static(&[0x60, 0x00, 0x60, 0x00, 0xf3]); + let code_b = Bytes::from_static(&[0x60, 0x01, 0x60, 0x01, 0xf3]); + let mut toggle = false; + b.iter(|| { + let code = if toggle { + code_a.clone() + } else { + code_b.clone() + }; + toggle = !toggle; + black_box(cache.apply_update(black_box(&StateUpdate::code(POOL, code)))); + }) + }); + + // Purge mutates the cache, so re-seed each iteration via iter_batched. + group.bench_function("purge_all_storage", |b| { + b.iter_batched( + || { + let mut cache = pool_cache(&rt); + cache.inject_storage_batch(&[ + (POOL, U256::from(0), U256::from(1)), + (POOL, U256::from(1), U256::from(2)), + (POOL, U256::from(2), U256::from(3)), + ]); + cache + }, + |mut cache| { + black_box( + cache + .apply_update(black_box(&StateUpdate::purge(POOL, PurgeScope::AllStorage))), + ); + }, + BatchSize::SmallInput, + ) + }); + + // `PurgeScope::Account` (full account + storage removal). + group.bench_function("purge_account", |b| { + b.iter_batched( + || { + let mut cache = pool_cache(&rt); + cache.inject_storage_batch(&[ + (POOL, U256::from(0), U256::from(1)), + (POOL, U256::from(1), U256::from(2)), + ]); + cache + }, + |mut cache| { + black_box( + cache.apply_update(black_box(&StateUpdate::purge(POOL, PurgeScope::Account))), + ); + }, + BatchSize::SmallInput, + ) + }); + + // `PurgeScope::Slots` (a few specific slots). + group.bench_function("purge_slots", |b| { + b.iter_batched( + || { + let mut cache = pool_cache(&rt); + cache.inject_storage_batch(&[ + (POOL, U256::from(0), U256::from(1)), + (POOL, U256::from(1), U256::from(2)), + (POOL, U256::from(2), U256::from(3)), + ]); + cache + }, + |mut cache| { + black_box(cache.apply_update(black_box(&StateUpdate::purge( + POOL, + PurgeScope::Slots(vec![U256::from(0), U256::from(2)]), + )))); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +/// A *heterogeneous* `apply_updates` batch (Slot + Account + Purge) — exercises +/// the single-lock fast-path drop/re-acquire discipline around the non-slot +/// updates. +fn bench_apply_heterogeneous(c: &mut Criterion) { + let rt = current_thread_rt(); + let mut group = c.benchmark_group("apply_updates_mixed"); + + group.bench_function("slot_account_purge", |b| { + b.iter_batched( + || { + let mut cache = pool_cache(&rt); + cache.inject_storage_batch(&[(POOL, U256::from(9), U256::from(1))]); + cache + }, + |mut cache| { + // The cache is re-seeded each iteration, so a fixed value still + // records real changes (slots start at ZERO, balance/purge act on + // the fresh seed). + let value = U256::from(2); + black_box(cache.apply_updates(black_box(&[ + StateUpdate::slot(POOL, U256::from(0), value), + StateUpdate::slot(POOL, U256::from(1), value), + StateUpdate::balance(POOL, value), + StateUpdate::purge(POOL, PurgeScope::Slots(vec![U256::from(9)])), + StateUpdate::slot(POOL, U256::from(2), value), + ]))); + }, + BatchSize::SmallInput, + ) + }); + + group.finish(); +} + +/// A *distinct-address* `apply_updates` batch — the only fair apples-to-apples +/// comparison against the raw `inject_storage_batch` baseline (each write targets +/// a different address, so no overlay account exists and the fast-path holds the +/// backend storage guard once for the whole run). +fn bench_apply_distinct_addresses(c: &mut Criterion) { + let rt = current_thread_rt(); + let mut group = c.benchmark_group("apply_distinct_addresses"); + + for &n in &[10usize, 100, 1_000] { + let updates_a: Vec = (0..n) + .map(|i| StateUpdate::slot(Address::repeat_byte(i as u8), U256::from(0), U256::from(1))) + .collect(); + let updates_b: Vec = (0..n) + .map(|i| StateUpdate::slot(Address::repeat_byte(i as u8), U256::from(0), U256::from(2))) + .collect(); + + group.throughput(Throughput::Elements(n as u64)); + group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, _| { + let mut cache = pool_cache(&rt); + let mut toggle = false; + b.iter(|| { + let updates = if toggle { &updates_a } else { &updates_b }; + toggle = !toggle; + black_box(cache.apply_updates(black_box(updates))); + }) + }); + } + group.finish(); +} + +criterion_group!( + benches, + bench_apply_slots_batch, + bench_apply_per_variant, + bench_apply_heterogeneous, + bench_apply_distinct_addresses, +); +criterion_main!(benches); diff --git a/docs/KNOWN_ISSUES.md b/docs/KNOWN_ISSUES.md index 0d93a5d..9a72081 100644 --- a/docs/KNOWN_ISSUES.md +++ b/docs/KNOWN_ISSUES.md @@ -1,112 +1,116 @@ # Known issues & limitations A living triage list of bugs, smells, and limitations surfaced during the -publication-readiness review. Items here are **flagged, not fixed** — the test -suite deliberately pins *current* behavior, so changing any of these is a -conscious, reviewable decision (and a `CHANGELOG.md` entry). +publication-readiness review. Items here are either **remaining limitations** or +**recently-fixed issues kept for auditability**; behavior-changing fixes should +carry red/green tests and a `CHANGELOG.md` entry. Confidence legend: **[V]** verified against the source during review; **[R]** reported by the review and worth confirming before acting. -## Correctness / behavior to review - -1. **[V] Silent persistence failures.** `cache::save_binary_state`, - `PrefetchRegistry::save`, and `ImmutableDataCache::save` log a warning on I/O - error but return `()`, so callers cannot detect a failed write (full disk, - permissions, partial flush). Consider returning `Result<()>` (a breaking - change worth taking pre-1.0). Tested today only insofar as the happy-path - round-trip succeeds. - -2. **[V] Access-list L2 profitability uses an approximate gas model.** In - `access_list.rs`, `into_access_list_if_profitable` / `access_list_if_profitable` - estimate L1 calldata cost with hand-rolled RLP-overhead constants - (`4 * 16` per address, `16` per key, `3 * 16` for the list header). This is an - intentional heuristic, not a precise EIP-2930 serialization cost — verify it - against real serialized sizes before relying on the profitability verdict for - anything other than a rough gate. The two functions also duplicate this logic - (a maintenance hazard: a fix to one must be mirrored). - -3. **[V] Profitability swallows provider errors.** The same functions catch all - provider errors and return `Ok(None)`, which is indistinguishable from - "computed: not profitable." A caller cannot tell a skipped check (RPC down) - from a real negative. Consider a result type that distinguishes the two. - -4. **[R] `set_block` with a tag leaves `block.number` stale.** Only - `BlockId::Number(n)` syncs the `NUMBER` opcode value; pinning to a tag (e.g. - `BlockId::latest()`) leaves the previously-set number in the block env. Either - resolve tags to a concrete number at pin time or document the constraint - loudly. - -5. **[R] Duplicate custom-error selectors shadow silently.** `RevertDecoder` +## Recently fixed by `codex/phase-5-known-issues-top5` + +1. **[FIXED] Cold absolute `Account` patches no longer materialize unknown + accounts.** `StateUpdate::Account` is cold-aware: a partial patch against an + address absent from both layers is skipped, does not write a default backend + account, and is surfaced through `StateDiff.skipped_accounts`. Intentional + cold materialization is now explicit via `StateUpdate::AccountUpsert` / + `StateUpdate::account_upsert(...)`. + +2. **[FIXED] Block-context drift after re-pinning.** `set_block` now sets + `block_number` only for concrete numeric pins and clears it for tag/hash/`None` + pins. Block changes, plus non-concrete pin calls that can drift under the same + tag, clear stale `basefee`; callers refresh `NUMBER`/`BASEFEE` together with + `set_block_context` after fetching the new header. + +3. **[FIXED] Synchronous layer-2 escape hatches have an invalidating wrapper.** + Raw handles are now visibly named `unchecked_blockchain_db()` / + `unchecked_backend()`, and `EvmCache::with_blockchain_db_mut(...)` runs a + synchronous `BlockchainDb` mutation and invalidates the COW snapshot base + automatically. + +4. **[FIXED] Explicit persistence failures are observable.** + `cache::save_binary_state`, `PrefetchRegistry::save`, and + `EvmCache::flush` now return `anyhow::Result<()>`. `Drop` remains best-effort + and logs flush errors. + +5. **[FIXED] Access-list profitability uses exact EIP-2930 RLP bytes.** + Arbitrum profitability now centralizes data-gas accounting in + `access_list_rlp_data_gas(...)` and provider/pricing failures propagate as + `Err`, leaving `Ok(None)` for empty/zero-priced/unprofitable lists. + +6. **[FIXED] `simulate_call_with_balance_deltas` now returns the touched access + list.** The pre/post `balanceOf` reads and simulated call share one EVM + journal, and the method now extracts the EIP-2930 access list before + commit/revert, matching the transfer-inspector simulation path. + +7. **[FIXED] On-disk cache files carry magic bytes and a version number.** + `binary_state`, `bytecode`, `ImmutableDataCache`, and V3 tick snapshots now + write a crate-specific magic header plus version `1` before the bincode + payload. Unknown magic/version values and legacy raw-bincode files are treated + as cache misses. + +8. **[FIXED] `call_raw_with_access_list` did not revert its checkpoint on a + transact error.** Both `EvmCache::call_raw_with_access_list` and + `EvmOverlay::call_raw_with_access_list_with` now match on the `transact_one` + result and `checkpoint_revert` on **every** path (success and host error), + matching `call_raw` / `simulate_with_transfer_tracking`. A host-level transact + error no longer leaves the overlay checkpoint un-reverted. + +## Remaining open issues ranked by unexpected-result risk + +1. **[R] Duplicate custom-error selectors shadow silently.** `RevertDecoder` registration replaces an existing entry for the same 4-byte selector with no warning, so an accidental double-registration silently wins. Consider a debug-level log or a `try_register` that reports collisions. -6. **[R] ERC20 `Transfer` decoding assumes the standard layout.** `inspector.rs` +2. **[R] ERC20 `Transfer` decoding assumes the standard layout.** `inspector.rs` reads `from`/`to` from indexed topics and `value` from the first 32 data - bytes. Non-standard or packed `Transfer` encodings parse incorrectly. Also, an - address that appears as both `from` and `to` in one transfer is both - subtracted and added (a semantically-invalid self-transfer is not rejected). + bytes. Non-standard or packed `Transfer` encodings may parse incorrectly or be + skipped. A self-transfer where `from == to` nets to zero for that owner; this + is now documented at the call site. -7. **[R] Panic codes above `u64::MAX` are dropped.** `decode_solidity_panic` +3. **[V] `SystemTime::now().unwrap()` panic risk in EVM construction.** + `build_evm` / `make_local_context` (and the overlay equivalents) call + `SystemTime::now().duration_since(UNIX_EPOCH).unwrap()` when no timestamp + override is set, which panics if the system clock is before the Unix epoch. + Setting an explicit timestamp avoids it; consider a saturating fallback. + +4. **[R] Panic codes above `u64::MAX` are dropped.** `decode_solidity_panic` converts out-of-range panic codes to `None`. Real compiler-emitted panic codes - are single-byte constants, so this is benign in practice; now documented at the + are single-byte constants, so this is benign in practice and documented at the call site. -8. **[V] `simulate_call_with_balance_deltas` returns an empty access list.** It - sets `CallSimulationResult.access_list = AccessList::default()`, unlike - `simulate_with_transfer_tracking` which populates it via `extract_access_list`. - Either the field is meaningless on this path or the population was missed — - the docs now state the field is empty here; reconcile before relying on it. - -9. **[V] `call_raw_with_access_list` does not revert its checkpoint on a transact - error.** It propagates the EVM `transact` error with `?` *before* reverting the - journaled checkpoint, whereas `call_raw` / `simulate_with_transfer_tracking` - revert on every path. A host-level transact error therefore leaves the overlay - checkpoint un-reverted. (Reverts normally on success and on revert/halt.) - -10. **[V] `SystemTime::now().unwrap()` panic risk in EVM construction.** - `build_evm` / `make_local_context` (and the overlay equivalents) call - `SystemTime::now().duration_since(UNIX_EPOCH).unwrap()` when no timestamp - override is set, which panics if the system clock is before the Unix epoch. - Setting an explicit timestamp avoids it; consider a saturating fallback. - ## Code-quality nits -11. **[V] Dead branch in `i128_to_u256`** (`cache/storage_keys.rs`): both the +5. **[V] Dead branch in `i128_to_u256`** (`cache/storage_keys.rs`): both the `value >= 0` and `else` arms evaluate the identical `U256::from(value as u128)`. The two's-complement cast is correct for both signs, so the `if`/`else` can collapse to one line (keep the explanatory comment). -12. **[R] V3 tick-snapshot keys serialize as strings.** `V3PoolTickSnapshot` +6. **[R] V3 tick-snapshot keys serialize as strings.** `V3PoolTickSnapshot` stringifies `i16`/`i32` tick/word keys for bincode, then `parse()`s them back in `to_tick_bitmap`/`to_ticks`, silently dropping any key that fails to parse. A native integer-keyed encoding would be faster and would not fail silently. -13. **[V] On-disk caches have no version header.** `binary_state`, `bytecode`, - `metadata` (`ImmutableDataCache`), and `tick_snapshot` all persist raw bincode - with no magic bytes or version field, so a struct-layout change silently - invalidates every existing cache file (decoded as a miss). A version header - would enable detection/migration. - -14. **[R] Balancer pool id keyed by `Debug` formatting.** `ImmutableDataCache` +7. **[R] Balancer pool id keyed by `Debug` formatting.** `ImmutableDataCache` keys `balancer_pools` by `format!("{:?}", pool_id)`. `Debug` output is not a stable encoding contract; a hex encoding would be safer for a persisted key. ## API ergonomics -15. **[R] `snapshot()` vs `create_snapshot()`.** `snapshot()` returns a low-level +8. **[R] `snapshot()` vs `create_snapshot()`.** `snapshot()` returns a low-level `revm::database::Cache` for in-place `restore()`; `create_snapshot()` returns an `Arc` for cross-thread fan-out. The names don't convey the difference. Docs now cross-reference them (see the rustdoc), but a rename could be considered pre-1.0. -16. **[R] Process-global cache speed mode.** `set_cache_speed_mode` / +9. **[R] Process-global cache speed mode.** `set_cache_speed_mode` / `cache_speed_mode` are a process-wide `static`, so two caches in one process cannot tune concurrency independently. Phase 1 moved configuration toward per-instance (`EvmCacheBuilder::cache_config`); the global setter remains. -17. **[V] `SpeculativeSim` consumption contract.** Both `validate()` and +10. **[V] `SpeculativeSim` consumption contract.** Both `validate()` and `into_optimistic()` take `self` by value, so double-consumption is unreachable under normal ownership. Internally `validate()` uses `.expect("validation handle taken twice")` (defensive) while `into_optimistic()` no-ops if the @@ -115,16 +119,83 @@ Confidence legend: **[V]** verified against the source during review; ## Limitations by design / roadmap -- **No copy-on-write snapshots yet.** `create_snapshot()` deep-clones state - (`O(accounts + slots)`); the COW rewrite is roadmap Pillar A. The `simulation` - benchmarks exist to measure the baseline this will improve on. +- **Copy-on-write snapshots (Phase 5, Pillar A) — done.** `create_snapshot()` is + no longer an O(total state) deep clone. The cold `BlockchainDb` index (layer 2) + is flattened once into an internal, immutable, `Arc`-shared base (per-account + storage shared by `Arc`), memoized across snapshots and rebuilt copy-on-write + only for the addresses that changed; each snapshot folds just the hot CacheDB + delta (layer 1) over a cheap `Arc::clone`. **Residual cost model (honest):** a + snapshot is no longer free. When layer 2 is unchanged since the last snapshot + it still pays an **O(accounts) length-scan** of the layer-2 storage/account + maps (to catch uncontrolled lazy-fetch growth that bypasses the write funnel, + since `foundry-fork-db` cannot be hooked) plus an **O(layer-1) fold** of the hot + delta — so the cost tracks `accounts + changed state`, not total slots. A + full rebuild (first snapshot, or after `set_block`/re-pin) is still O(total + state). `create_snapshot` is now `&mut self` (it memoizes the base, Decision + D5). The retained `create_snapshot_deep_clone()` (the legacy full flatten) is + kept as the A/B benchmark baseline and the read-equivalence reference; the + `create_snapshot` group in `benches/simulation.rs` measures both. Decisions and + the cost model are in [`phase-5-spec.md`](phase-5-spec.md) / `ROADMAP.md`. +- **Layer-2 unchecked accessors remain an explicit contract boundary (Phase 5).** The + snapshot base's growth scan is count/absence-based, which is sufficient for the + supported writers: the crate's own mutators (`apply_update`, `inject_storage_batch`, + the `inject_*` helpers, purges, code overrides) explicitly mark the base dirty, + and the `foundry-fork-db` `SharedBackend` lazy fetch is append-only at a fixed + block (it only inserts on a cache miss, never overwrites in place — a load-bearing + invariant noted in `refresh_base`). Direct out-of-band writes through the + `unchecked_blockchain_db()` / `unchecked_backend()` handles still bypass the + normal write funnel by design. For synchronous `BlockchainDb` map writes, prefer + [`EvmCache::with_blockchain_db_mut`], which invalidates the base automatically + after the closure returns. If using the unchecked handle directly, call + [`EvmCache::invalidate_snapshot_base`] after the write lands and before the next + snapshot (or re-pin via `set_block`). For + `SharedBackend::insert_or_update_storage` / `insert_or_update_address`, the call + only enqueues work on the backend handler; `invalidate_snapshot_base()` does not + wait for that queued update. First synchronize or read back until the expected + value is visible in `BlockchainDb` / through the backend, then invalidate before + creating the snapshot. The rustdoc on both accessors and the hook carries this + warning, and `tests/cow_snapshot.rs` + (`invalidate_snapshot_base_rehonest_after_escape_hatch_write`, + `invalidate_snapshot_base_rehonest_after_existing_account_write`, + `with_blockchain_db_mut_rehonest_after_storage_overwrite`, + `with_blockchain_db_mut_rehonest_after_account_overwrite`) + pins it. - **`protocols` not yet extracted.** The DeFi surface is feature-gated but still - in-crate; `cargo test --no-default-features` is not yet supported because some - unit tests assume the default feature. Extraction into `evm-amm-state` is - planned (roadmap), blocked partly by `ImmutableDataCache` coupling generic - token-decimals with V2/V3/Balancer pool metadata. -- **Event-driven sync (roadmap Pillar B) is not implemented.** Targeted - inject/purge exist; decoding logs into state updates and the WS ingestion loop - with reorg handling are future phases. + in-crate. The generic core builds and tests with `--no-default-features`, but + extraction into `evm-amm-state` is still planned (roadmap), blocked partly by + `ImmutableDataCache` coupling generic token-decimals with V2/V3/Balancer pool + metadata. +- **Event-driven sync (roadmap Pillar B) — reader/writer halves done; live WS + transport is not.** The Phase 3 **writer half** (`StateUpdate` + + `apply_update`/`apply_updates`) and the Phase 4 **reader half** (the `events` + module: `EventDecoder`/`DecoderRegistry`, the ERC-20 + UniswapV3 adapters, and + the `EventPipeline` with `ingest_logs`/`reorg_to`/`reconcile`) are implemented. + What is **not** shipped is a concrete production WS transport: the async + `events::drive`/`LogSource` convenience is generic over a log source and is + exercised only by the offline example feeding an in-memory source; wiring it to + a live `subscribe_logs`/WS provider (and detecting reorgs from block-hash + mismatches) is left to the consumer. +- **[V] V3 event-derived tick maintenance does not reconstruct fee-growth / + oracle state (Phase 4 §6.4).** `UniswapV3Decoder`'s `Mint`/`Burn` handling + maintains `liquidityGross`/`liquidityNet` (tick slot +0), the `initialized` flag + (+3), the `tickBitmap`, and the in-range global `liquidity`, but **not** + `feeGrowthOutside0/1X128` (slots +1/+2), `secondsOutside`, or oracle + observations — these are not derivable from the `Mint`/`Burn`/`Swap` events. + **Swap price/liquidity quoting is unaffected** (the swap-amount math does not + read `feeGrowthOutside`), but fee accounting and `collect`-style reads against + event-maintained ticks are not kept current. Sampled + `EventPipeline::reconcile` (RPC re-read) and reorg `reorg_to` (purge-and-resync) + are the backstop; seed a full tick via `inject_v3_ticks` when fee state matters. +- **`inject_v2/v3_*` layer behavior changed in Phase 3 (Decision 2).** The + `protocols`-gated `inject_v2_pool_metadata` / `inject_v3_tick_bitmap*` / + `inject_v3_ticks*` helpers were refolded onto the write-through + `StateUpdate::Slot` primitive, so they now write **both** cache layers (backend + + overlay-if-present) instead of the previous overlay-only write. This is a + deliberate normalization (one consistent write path), not a bug: signatures and + return values are unchanged and the visible reads are identical; only the slot + *placement* across layers moved. `tests/state_update.rs` + (`inject_v3_tick_bitmap_writes_through_to_backend`) pins the new behavior, and + it is recorded in `CHANGELOG.md` (`### Changed`). The cold-backfill + `inject_storage_batch` deliberately remains layer-2-only. - **Recent toolchain.** MSRV 1.88 and edition 2024 are intentional and CI-enforced; consumers on older toolchains are not supported. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 92d6781..36d2fef 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -72,9 +72,9 @@ RPC node Event-driven sync ← WS logs · new block | **0** | API hygiene + correctness: drop `amms`, fix `set_block` divergence + `block_in_place` panic, commit the tree. | **Done** (`p0-oss-prep`) | | **1** | Engine seam: typed errors, configurable tx/block env, hot-path benches, builder, `protocols` feature. | **Done** (`phase-1-engine-seam`) | | **2** | Freshness core (Pillar C): `Validity` + `FreshnessRegistry`; observation tracker; policies; optimistic verify-and-rerun loop. | **Done** (`phase-2-freshness`) | -| **3** | State-update primitives (Pillar B.1): `StateUpdate` + targeted writers; refold `inject_*`; surface state-diff output. | Planned | -| **4** | Event pipeline + adapters (Pillar B.2): `EventDecoder` trait, V3 adapter, WS ingestion loop, reorg handling. | Planned | -| **5** | COW snapshots (Pillar A): structural sharing; overlay buffer reuse. | Planned | +| **3** | State-update primitives (Pillar B.1): `StateUpdate` + targeted writers; refold `inject_*`; surface state-diff output. | **Done** (`phase-3-state-updates`) | +| **4** | Event pipeline + adapters (Pillar B.2): `EventDecoder` trait, ERC-20 + V3 adapters, ingest/reorg/reconcile pipeline. | **Done** (`phase-4-event-pipeline`) | +| **5** | COW snapshots (Pillar A): structural sharing; overlay buffer reuse. | **Done** (`phase-5-cow-snapshots`) | Cross-cutting (land opportunistically): call tracer Inspector, full offline (`default-features = false`, no provider) build split, CHANGELOG/CONTRIBUTING. @@ -304,6 +304,158 @@ offline `examples/freshness_optimistic.rs`; and `tests/freshness.rs`. --- +## Phase 3 — state-update primitives (detailed, decisions locked) + +Builds **Pillar B.1 — the writer half** of the event → state pipeline: the +generic state-mutation vocabulary and the single apply primitive that writes it +consistently across both cache layers, returning a structured diff. Out of +scope (Phase 4): event decoding (`EventDecoder`, `Log` → `StateUpdate`), the WS +ingestion loop, reorg handling, and overlay-side apply. + +### Locked decisions + +1. **`Account` variant is a partial `AccountPatch`** (`balance`/`nonce`/`code`, + each `Option`), not a full `AccountInfo`: best fit for event-derived writes + (one field at a time) and keeps revm's type out of the public vocabulary. +2. **`inject_v2/v3_*` (`protocols`) normalized to write-through.** Refolded onto + the write-through `StateUpdate::Slot` primitive (backend + overlay-if-present) + instead of the old overlay-only write — a deliberate behavior change recorded + in `CHANGELOG.md` (`### Changed`) and `KNOWN_ISSUES.md`, with a test pinning + the new placement. The cold-backfill `inject_storage_batch` stays layer-2-only. + +### Acceptance — met + +`cargo fmt --check`, `clippy --all-targets -- -D warnings` (default + +`--lib --no-default-features`), `cargo test`, `RUSTDOCFLAGS=-D warnings cargo doc`. + +Landed on `phase-3-state-updates`: `src/state_update.rs` (the generic vocabulary +— `StateUpdate` / `AccountPatch` / `PurgeScope`, the `StateDiff` / `AccountChange` +/ `PurgeRecord` output, reusing `freshness::SlotChange`); `EvmCache::apply_update` +/ `apply_updates` with the dual-layer write-through `Slot`/`Account` and dispatch +`Purge` semantics; the refold of `inject_storage_batch_fresh` / `purge_account` / +`purge_pool_storage` / `purge_pool_slots` / `inject_v2_pool_metadata` / +`inject_v3_*` onto the primitive and the freshness correction-drain routed +through `apply_updates`; the offline `examples/state_update_apply.rs`; +`benches/state_update.rs`; and `tests/state_update.rs`. The §15 addendum adds the +relative / read-modify-write surface — a saturating `SlotDelta`, the +`StateUpdate::SlotDelta` variant, `EvmCache::modify_slot`, and the cold-aware +skip-and-surface contract via the new `StateDiff.skipped` field — to keep +event-derived balances (e.g. ERC-20 `Transfer` deltas) hot without knowing the +resulting absolute value. The §16 post-audit remediation then fixed a +HIGH-severity silent-corruption bug — `cached_storage_value` now mirrors the EVM +`SLOAD` for `StorageCleared`/`NotExisting` overlay accounts instead of returning a +shadowed backend value — and hardened the surface: a no-op `Account` patch no +longer materializes a backend account; the vocabulary and diff gained `serde`; +`StateDiff`/`AccountPatch` became `#[non_exhaustive]`; relative native-balance +tracking landed (`StateUpdate::BalanceDelta`, `EvmCache::modify_account_balance`, +`StateDiff.skipped_balances`, `SkippedBalanceDelta`) with discoverable skip +accessors (`has_skipped`/`skipped_len`/`is_fully_applied`) and the +`StateUpdate::nonce`/`code`/`account` constructors; and `apply_updates` gained a +batched single-lock fast-path (byte-identical to the sequential fold, pinned by an +equivalence test). + +--- + +## Phase 4 — event pipeline + adapters (detailed, decisions locked) + +Builds **Pillar B.2 — the reader half** of the event → state pipeline: decode an +on-chain `Log` into the Phase 3 `StateUpdate` vocabulary, apply it, and run the +reactive maintenance (reconcile, reorg) that keeps event-derived state honest. +Decoders are pure functions of `(log, pre-state)`; the `!Send` cache discipline is +preserved by keeping the tested core synchronous (the async ingestion driver is a +thin convenience). The full build contract is in +[`phase-4-spec.md`](phase-4-spec.md). + +### Locked decisions + +1. **Packed-slot updates → `StateUpdate::SlotMasked`** (a cold-aware RMW masked + write), so a pure decoder can express a partial update to a packed word (V3 + `slot0`) without clobbering the bits it does not own (notably `unlocked`). +2. **V3 adapter coverage → `Swap` **and** `Mint`/`Burn` (full ticks).** `slot0` + + `liquidity` from `Swap`; per-tick `liquidityGross`/`liquidityNet` + + `initialized` + `tickBitmap` + in-range global `liquidity` from `Mint`/`Burn`, + computed against the `StateView`. Fee-growth/oracle state is out of scope (a + documented limitation; reconcile/purge are the backstop). +3. **Reorg → purge-and-resync.** A depth-bounded ring tracks addresses touched per + block; `reorg_to(n)` purges everything touched after `n` so reads re-fetch. + `ValidThrough` is the freshness lever. +4. **Reconciliation → sampled re-read, correct **and** alarm.** `reconcile` samples + event-derived slots and re-reads via `EvmCache::reconcile_slots` (a honest + wrapper over `verify_slots` that errors on a total fetch failure rather than + reporting a false all-clear); the fresh chain value wins and the drift is + surfaced. + +### Acceptance — met + +`cargo fmt --check`, `clippy --all-targets -- -D warnings` (default + +`--lib --no-default-features`), `cargo test` (both feature configs), +`RUSTDOCFLAGS=-D warnings cargo doc`, `cargo bench --no-run`. + +Landed on `phase-4-event-pipeline`: `src/events/` (the generic core — +`EventDecoder`/`StateView`, `DecoderRegistry`, `EventPipeline` with +`ingest_logs`/`reorg_to`/`reconcile` + `BlockDigest`/`ReconcileReport`/ +`ReorgConfig`, and the async `drive`/`LogSource`), the generic +`Erc20TransferDecoder` (`events::erc20`), and the `protocols`-gated +`UniswapV3Decoder`/`UniswapV3Layout` (`events::uniswap_v3`); the cold-aware +`StateUpdate::SlotMasked` vocabulary + `StateDiff.skipped_masks`/`SkippedMask` +(`state_update`) and its dual-layer apply arm; `EvmCache::reconcile_slots` and the +`StateView` impl; the offline `examples/reactive_cache.rs`; +`benches/event_pipeline.rs`; and `tests/event_pipeline.rs` (+ the `SlotMasked` +tests in `tests/state_update.rs`). The §6.4 V3 fee-growth/oracle maintenance gap +is recorded in `KNOWN_ISSUES.md`. + +--- + +## Phase 5 — copy-on-write snapshots (detailed, decisions locked) + +Builds **Pillar A**: replace the O(total state) deep-clone `create_snapshot` with +a two-tier copy-on-write view whose cost tracks *changed* state, not *total* +state. The cold `BlockchainDb` index (layer 2) is flattened once into an +internal, immutable, `Arc`-shared base — both the base as a whole and each +account's storage map are shared by `Arc`, so structural sharing needs no new +dependency (Decision D1) — memoized across snapshots and rebuilt copy-on-write +only for the addresses that changed; each snapshot then folds just the hot +CacheDB delta (layer 1). Reads stay O(1), lock-free, and bit-for-bit identical to +the deep clone. The full build contract is in +[`phase-5-spec.md`](phase-5-spec.md). + +### Locked decisions + +1. **`Arc`-shared maps, not a persistent HAMT** (D1). Reads stay O(1) with no + per-`SLOAD` regression and no external dependency. +2. **Base memoized as immutable; over-invalidation is acceptable, silent + staleness is not** (D2). The write-through funnel marks an address dirty + unconditionally; the differential-equivalence test is the hard backstop. +3. **Keep the deep clone** as `create_snapshot_deep_clone` (D3) — the A/B + benchmark baseline and the read-equivalence reference. +4. **Overlay reuse: buffer reuse *and* `reset()` recycle** (D4) — both in scope. +5. **`create_snapshot` becomes `&mut self`** (D5) — the memoization cost; the + freshness controller and all callers are updated. + +### Acceptance — met + +`cargo fmt --check`, `clippy --all-targets -- -D warnings` (default + +`--lib --no-default-features`), `cargo test` (both feature configs), +`RUSTDOCFLAGS=-D warnings cargo doc`, `cargo bench --no-run`; the +`tests/cow_snapshot.rs` differential-equivalence gate and the existing +snapshot/overlay/freshness tests pass unchanged. + +Landed on `phase-5-cow-snapshots`: the memoized two-tier base (`BaseState` + +the rewritten two-tier `EvmSnapshot` with `account_info`/`storage_value`/`code` +accessors, `src/cache/snapshot.rs`); `EvmCache::refresh_base`/`build_base_full`, +the COW `create_snapshot` (now `&mut self`), the retained +`create_snapshot_deep_clone`, and the `mark_base_dirty`/`invalidate_base` +invalidation wired into `write_slot_through`/`apply_slot_run`/ +`write_account_info_through`/`inject_storage_batch`/the `purge_*` paths and +`set_block` (`src/cache/mod.rs`); `EvmOverlay::reset` plus the reusable +shared-memory buffer recycled across the call methods (`src/cache/overlay.rs`); +the layer-2-seeded A/B + hot-loop + `reset()`-fanout benches +(`benches/simulation.rs`); and the differential-equivalence gate +(`tests/cow_snapshot.rs`). The residual O(accounts) length-scan / O(layer-1) +fold cost model is recorded in `KNOWN_ISSUES.md`. + +--- + ## Key abstractions for later phases (sketches) ```rust diff --git a/docs/phase-2-spec.md b/docs/phase-2-spec.md index c61f19c..baff6b2 100644 --- a/docs/phase-2-spec.md +++ b/docs/phase-2-spec.md @@ -54,7 +54,7 @@ evaluation sims only. `storage_batch_fetcher() -> Option<&StorageBatchFetchFn>`, `inject_storage_batch(&[(Address,U256,U256)])`, `purge_pool_storage`, `purge_pool_slots`, `call_raw_with`/`TxConfig`, `CallSimulationResult`, - `blockchain_db()`, `db_mut()`. + `unchecked_blockchain_db()`, `db_mut()`. - `cache::EvmOverlay` / `cache::EvmSnapshot` (`overlay.rs`/`snapshot.rs`): `EvmOverlay::new(Arc, Option)`, `call_raw`, `simulate_with_transfer_tracking`. `EvmOverlay` is `Send`. diff --git a/docs/phase-3-spec.md b/docs/phase-3-spec.md new file mode 100644 index 0000000..e51886c --- /dev/null +++ b/docs/phase-3-spec.md @@ -0,0 +1,860 @@ +# Phase 3 implementation spec — state-update primitives (Pillar B.1) + +Implementation contract for the **targeted state-mutation vocabulary** and the +single apply primitive that writes it correctly across both cache layers, +returning a structured state diff. Read this **with** +[`ROADMAP.md`](ROADMAP.md) (the "Phase 3" row and the "Pillar B — event → state +pipeline" / "Key abstractions" sections are the design of record). This document +is the precise build contract; where they overlap, prefer this. + +This is **Pillar B.1 — the writer half** of the event → state pipeline. It does +**not** decode events (no `EventDecoder`, no `Log` parsing, no WS loop): that is +Phase 4. Phase 3 builds the vocabulary an event decoder will *emit into* and the +mechanism that *applies* it, with no protocol or event knowledge in the core. + +## 0. Ground rules (non-negotiable) + +- **Branch:** create `phase-3-state-updates` off the current `phase-2-freshness` + HEAD. Commit there in logical steps. Do **not** push, do **not** tag. Commits + must be unsigned: `git -c commit.gpgsign=false commit …` (the 1Password signing + agent is unavailable here). End every commit message with exactly: + `Co-Authored-By: Claude Opus 4.8 (1M context) ` +- **The whole state-update surface is generic core** — it must compile and lint + with `--no-default-features`. `StateUpdate` / `PurgeScope` / `AccountPatch` / + `StateDiff` / `apply_update` / `apply_updates` must NOT depend on the + `protocols` feature. (The *refold* of the `protocols`-gated `inject_v2/v3_*` + helpers stays behind `protocols`, but it consumes the generic primitive.) +- **Green bar at every commit, both feature configs:** + - `cargo fmt --all --check` + - `cargo clippy --all-targets --no-deps -- -D warnings` + - `cargo clippy --lib --no-default-features --no-deps -- -D warnings` + - `cargo test` + - `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps` +- MSRV is 1.88 — no newer-than-1.88 std APIs. Edition 2024. +- **Do not break existing behavior or any existing test.** Existing `inject_*` / + `purge_*` public methods keep their signatures and return values; they become + thin wrappers over the new primitive (the Phase 1 `call_raw` → `call_raw_with` + pattern). The one place a *deliberate* behavior change is on the table is + Decision 2 (§12) — and only with sign-off + a CHANGELOG/KNOWN_ISSUES entry. +- No new dependencies. (`alloy-primitives`, `revm`, `foundry-fork-db` are present.) + +## 1. Objective & scope + +Today the crate writes cached state through a scatter of ad-hoc methods with +**inconsistent layering**: + +| Method | Layer 1 (CacheDB overlay) | Layer 2 (BlockchainDb) | Creates overlay acct? | +| --- | --- | --- | --- | +| `inject_storage_batch` | — | write | no | +| `inject_storage_batch_fresh` | write-through *if present* | write | no | +| `inject_v2_pool_metadata` / `inject_v3_*` | write (via `insert_account_storage`) | — | **yes** | +| `purge_account` | remove acct | remove acct + storage | n/a | +| `purge_pool_storage` | clear storage | remove storage | n/a | +| `purge_pool_slots` | remove slots | remove slots | n/a | +| `override_account_code*` | insert info | insert info | n/a | + +Three different slot-write semantics, no machine-readable record of *what +changed*, and no single vocabulary an event decoder can target. Phase 3 fixes +all three: + +1. **`StateUpdate`** — a small, generic enum: the vocabulary of targeted + mutations (`Slot`, `Account`, `Purge`). This is what a Phase 4 `EventDecoder` + will produce. +2. **`EvmCache::apply_update` / `apply_updates`** — the *single* primitive that + applies a `StateUpdate` (or batch) with **one, documented, consistent** + dual-layer policy, returning a `StateDiff`. +3. **`StateDiff`** — the structured "what actually changed" output (slot/account + diffs + purge records), so callers (and Phase 4's reconciliation) can observe + the effect of an apply. +4. **Refold** the existing `inject_*` / `purge_*` writers onto the primitive so + there is exactly one place the dual-layer write logic lives. + +**In scope:** the generic vocabulary; the apply primitive with write-through +semantics; the state-diff output; refolding the storage-slot and purge writers; +routing the freshness controller's correction drain through the primitive; +offline tests, an example, a benchmark, and docs. + +**Out of scope (document as Phase 4/5 follow-ups, do not build):** +- **Event decoding** — `EventDecoder` trait, V3/V2 adapters, `Log` → `StateUpdate` + (Phase 4). Phase 3 ends at the vocabulary; nothing parses a `Log`. +- **WS ingestion / `on_new_block` apply-and-purge / reorgs / RPC reconciliation** + (Phase 4). +- **Overlay-side apply** — `EvmOverlay::apply` so a live overlay receives updates + mid-fan-out (Phase 4/5). Phase 3 applies to the **`EvmCache`** only. +- **COW snapshots** (Phase 5) — `apply_*` operates on the existing layers. + +## 2. Reuse these existing pieces (do not reinvent) + +- `cache::EvmCache` (`src/cache/mod.rs`): the dual-layer fields + `self.db.cache.accounts` (CacheDB overlay, layer 1) and `self.blockchain_db` + (`accounts()` / `storage()` `RwLock`s, layer 2); the established write-through + pattern in `inject_storage_batch_fresh` (the F1 fix — **the** reference for + correct slot-write layering); `cached_storage_value`; `purge_account` / + `purge_pool_storage` / `purge_pool_slots` (the purge layer logic to fold in); + `self.db.insert_account_info` / `insert_account_storage` (CacheDB writers). +- `freshness::SlotChange { address, slot, old, new }` (`src/freshness.rs`, + re-exported at crate root) — **reuse it** as the slot-diff type; do not define a + parallel one. `StateDiff.slots: Vec`. +- `revm::state::{AccountInfo, Bytecode}` — the account representation in both + layers; `Bytecode::hash_slow()` recomputes a code hash. +- `alloy_primitives::{Address, U256, B256, Bytes}`. +- The offline test/example harness: `tests/common`, `examples/support/mock.rs` + (mocked provider; `from_backend` cache construction with no network). + +## 3. Module layout + +- **`src/state_update.rs`** (new, top-level, generic, **non-`protocols`**): the + pure data types — `StateUpdate`, `PurgeScope`, `AccountPatch`, `StateDiff`, + `AccountChange`, `PurgeRecord` — plus their constructors / small helpers and + in-module unit tests. No `EvmCache` dependency (pure data + logic on itself). +- **`src/cache/mod.rs`**: `EvmCache::apply_update`, `EvmCache::apply_updates`, + and the internal per-variant helpers. Refold `inject_storage_batch_fresh`, + `purge_account`, `purge_pool_storage`, `purge_pool_slots`, + `override_account_code*`, and (Decision 2) `inject_v2/v3_*` onto them. +- **`src/freshness.rs`**: route the `FreshnessController::run` `pending` drain + through `apply_updates` (§9) — behavior-preserving. +- **`src/lib.rs`**: `pub mod state_update;` + re-export + `StateUpdate, PurgeScope, AccountPatch, StateDiff, AccountChange, PurgeRecord`. + +## 4. Types & behavior + +### 4.1 `StateUpdate` — the vocabulary + +```rust +/// A single targeted mutation to cached EVM state. +/// +/// The vocabulary an event decoder (Phase 4) emits and [`EvmCache::apply_update`] +/// consumes. Generic: carries no protocol or event knowledge. +#[derive(Clone, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub enum StateUpdate { + /// Set one storage slot to `value`, authoritative across both cache layers. + Slot { address: Address, slot: U256, value: U256 }, + /// Patch an account's balance/nonce/code (partial — see [`AccountPatch`]). + Account { address: Address, patch: AccountPatch }, + /// Purge cached state for `address` at `scope`; the next read re-fetches. + Purge { address: Address, scope: PurgeScope }, +} +``` + +Constructors for ergonomics: `StateUpdate::slot(addr, slot, value)`, +`StateUpdate::balance(addr, value)`, `StateUpdate::purge(addr, scope)`. The enum +is `#[non_exhaustive]` (new variants — e.g. a code-only convenience — may be +added pre-1.0 without a breaking change). + +### 4.2 `AccountPatch` — partial account mutation + +```rust +/// A partial account mutation: each `Some` field overwrites the cached value, +/// each `None` leaves it unchanged. Setting `code` recomputes the code hash; +/// `Some(empty bytes)` clears code to the empty-code hash. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct AccountPatch { + pub balance: Option, + pub nonce: Option, + pub code: Option, +} +``` +Builders: `AccountPatch::default()`, `.balance(U256)`, `.nonce(u64)`, `.code(Bytes)` +(each returns `Self`). Rationale for **partial** (vs. a full `AccountInfo`): the +Pillar B driver is events, which usually carry *one* field (a `Transfer` changes +a balance, not nonce/code). Partial application avoids forcing a caller to +reconstruct a full `AccountInfo` (and avoids leaking revm's type into the public +vocabulary). **See Decision 1 (§12).** + +### 4.3 `PurgeScope` + +```rust +/// What part of an address's cached state a purge removes. +#[derive(Clone, Debug, PartialEq, Eq)] +#[non_exhaustive] +pub enum PurgeScope { + /// Full account: `AccountInfo` (balance/nonce/code) **and** all storage. + /// Equivalent to today's `purge_account`. + Account, + /// All storage slots; account info preserved. Equivalent to `purge_pool_storage`. + AllStorage, + /// Only the listed storage slots. Equivalent to `purge_pool_slots`. + Slots(Vec), +} +``` + +### 4.4 `StateDiff` — the output + +```rust +/// What an `apply_*` call actually changed. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct StateDiff { + /// Storage slots whose value changed (old != new). + pub slots: Vec, // reused from `freshness` + /// Accounts whose balance/nonce/code-hash changed. + pub accounts: Vec, + /// Purges performed, with what they removed. + pub purged: Vec, +} + +/// An account field delta. Each field is `Some((old, new))` only when it changed. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct AccountChange { + pub address: Address, + pub balance: Option<(U256, U256)>, + pub nonce: Option<(u64, u64)>, + pub code_hash: Option<(B256, B256)>, +} + +/// Record of a purge: how much of each layer it removed. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct PurgeRecord { + pub address: Address, + pub scope: PurgeScope, + /// Storage slots removed from the BlockchainDb backend (layer 2). + pub slots_removed: usize, + /// Whether an `AccountInfo` was removed (only the `Account` scope). + pub account_removed: bool, +} +``` +`StateDiff` helpers: `is_empty()`, `len()` (total changed entries), +`merge(&mut self, other: StateDiff)` (used by `apply_updates` to fold per-update +diffs). **Only actual changes are recorded** — applying a `Slot` whose value +already matches the cache yields an empty diff (idempotence is observable). + +## 5. `EvmCache::apply_update` / `apply_updates` + +```rust +pub fn apply_update(&mut self, update: &StateUpdate) -> StateDiff; +pub fn apply_updates(&mut self, updates: &[StateUpdate]) -> StateDiff; +``` +`apply_updates` folds left, merging each per-update `StateDiff`; later updates +observe the effect of earlier ones (e.g. two `Slot` writes to the same key: the +first records old→a, the second a→b). Both are **synchronous, infallible** +(no RPC — a write primitive, not a fetch). They return the diff; they do not +error. (Account/Slot writes can always succeed against the in-memory layers.) + +### 5.1 `Slot` — write-through (authoritative) + +Identical semantics to `inject_storage_batch_fresh` (the F1-fix reference): +1. `old = self.cached_storage_value(address, slot)` (overlay ▸ backend ▸ `None`). +2. Write `value` into the BlockchainDb backend (layer 2). +3. Write `value` into the CacheDB overlay **iff an overlay account already + exists** for `address` (`self.db.cache.accounts.get_mut`). Do **not** + materialize a new overlay account (preserves the cold-prefetch / layer-2-only + invariant; materializing one could shadow later RPC reads, and a + `StorageCleared` overlay account reads missing slots as ZERO). +4. Record `SlotChange { address, slot, old: old.unwrap_or(ZERO), new: value }` + **only if** `old.unwrap_or(ZERO) != value`. + +> A slot the cache never saw is treated as `old = ZERO` (the value a sim would +> have read), consistent with `verify_slots`. + +### 5.2 `Account` — partial patch, write-through + +1. Load the current `AccountInfo` from the cached layers only (overlay ▸ backend + ▸ `AccountInfo::default()`); remember the `old` field values for the change + record. **No RPC** (apply is a write, not a fetch). +2. Apply each `Some` patch field: `balance`, `nonce`, and for `code` set + `info.code = Some(Bytecode::new_raw(bytes))` (the empty bytecode for empty + input) and `info.code_hash = .hash_slow()`. +3. Write-through, mirroring §5.1: write the patched `AccountInfo` into the + BlockchainDb backend (layer 2) **always**, and into the CacheDB overlay + (`insert_account_info`) **iff an overlay account already exists** (do not + materialize a new overlay account — the read path falls through to the backend + for an absent overlay entry, so a backend-only write is authoritative and we + avoid polluting layer 1). This keeps the winning layer correct without the + cold-backfill hazard. +4. Record an `AccountChange` with `Some((old,new))` only for fields that changed + (compare balance, nonce, code_hash). + +### 5.3 `Purge` — dispatch to existing layer logic + +Dispatch on `scope` to the **existing** purge implementations (now sharing one +home), returning a `PurgeRecord`: +- `Account` → `purge_account` logic: remove from overlay accounts, backend + accounts, backend storage. `account_removed` = removed from any account layer; + `slots_removed` = backend storage slots removed. +- `AllStorage` → `purge_pool_storage` logic (clear overlay storage, remove + backend storage); `slots_removed` = backend slots removed. +- `Slots(slots)` → `purge_pool_slots` logic; `slots_removed` = backend slots + removed. + +## 6. Refold map (existing → primitive) + +Every existing public method **keeps its signature and return value**; it +becomes a wrapper. Existing tests must pass unchanged. + +| Existing | Refold | Public API | +| --- | --- | --- | +| `inject_storage_batch_fresh(&[(a,s,v)])` | `apply_updates` of `Slot`s (discard diff) | unchanged (`-> ()`) | +| `purge_account(a)` | `apply_update(Purge{a, Account})` | unchanged (`-> ()`) | +| `purge_pool_storage(a) -> usize` | `apply_update(Purge{a, AllStorage})`; return `rec.slots_removed` | unchanged | +| `purge_pool_slots(a, slots) -> usize` | `apply_update(Purge{a, Slots(..)})`; return `rec.slots_removed` | unchanged | +| `override_account_code*` | **best-effort**: route its final write through `apply_update(Account{ patch: code })` **only if** behavior-equivalent; it has bespoke target-creation (`MissingTargetBehavior`) + source→target code-copy semantics, so if the refold is not cleanly equivalent, leave the method as-is and only cross-reference the primitive in its doc | unchanged | +| `inject_v2_pool_metadata`, `inject_v3_*` (`protocols`) | build `Vec`, `apply_updates` | **Decision 2 (§12)** | + +**Not refolded (kept distinct, documented):** +- `inject_storage_batch(&[(a,s,v)])` — the **layer-2-only cold-backfill** path + (deliberately no write-through, no overlay touch). This is a *different intent* + from `StateUpdate::Slot` (authoritative write-through). Keep it as the + low-level backfill primitive; add a doc line cross-referencing `apply_update` + for authoritative writes. +- `purge_contracts_storage`, `purge_all_storage` — multi-address / whole-cache + sweeps. Leave as-is (they already share the layer logic); optionally note they + are batch forms of `Purge{AllStorage}`. Not required to refold. + +## 7. Public re-exports + +`src/lib.rs`: `pub mod state_update;` and +```rust +pub use state_update::{ + AccountChange, AccountPatch, PurgeRecord, PurgeScope, StateDiff, StateUpdate, +}; +``` +(`SlotChange` is already re-exported from `freshness`.) + +## 8. `cargo doc` / rustdoc requirements + +- A module-level `//!` doc on `state_update.rs`: the vocabulary, the apply + primitive, the dual-layer write-through policy (one paragraph: backend always, + overlay-if-present, no new overlay account for slots), the `StateDiff` output, + and the **Pillar B.1** framing with an explicit "events are Phase 4" boundary. +- Rustdoc on **every** public item (no `missing_docs` gate, but `-D warnings` + must pass and the surface must be documented thoroughly). +- A short **runnable doctest** on `apply_update` (or the module): build nothing + network-bound — construct `StateUpdate`s and an `AccountPatch`, show the + vocabulary and a `StateDiff` shape. (If a doctest needs an `EvmCache`, gate it + `no_run` and use the example harness pattern; prefer a pure-data doctest.) + +## 9. Freshness integration (behavior-preserving) + +Route `FreshnessController::run`'s `pending` drain (currently +`cache.inject_storage_batch_fresh(&injects)`) through the new primitive: +`cache.apply_updates(&pending.iter().map(|c| StateUpdate::slot(c.address, c.slot, c.new)).collect::>())`. +This is **behavior-identical** (both are write-through), and demonstrates the one +unified write path. Do not change any freshness test expectation. (The validator +itself still flows corrections back as `SlotChange`s; only the main-thread apply +changes its call.) + +## 10. Tests (offline, no network) — authored as the acceptance contract + +These are written **before** implementation and define correctness. Unit tests +in-module (`#[cfg(test)]` in `state_update.rs`); apply/refold integration tests +in a new `tests/state_update.rs` (reuse `tests/common`). + +**`state_update.rs` unit (pure data):** +- `AccountPatch` builders compose; `Default` is all-`None`. +- `StateDiff::merge` concatenates and `is_empty`/`len` count correctly. +- `StateUpdate` constructors produce the expected variants. + +**`tests/state_update.rs` integration (mocked-provider / `from_backend` cache):** +1. **Slot write-through, overlay present:** seed an overlay account + slot; + `apply_update(Slot)`; assert both layers hold the new value and the synchronous + SLOAD path reads it; `StateDiff.slots == [SlotChange{old,new}]`. +2. **Slot write-through, no overlay account:** apply to an address with no overlay + entry; assert the backend holds it, **no overlay account was materialized**, + and a subsequent read sees the value. +3. **Slot no-op:** apply the same value already cached → empty `StateDiff`. +4. **Slot idempotence:** apply twice → first diff non-empty, second empty. +5. **Account balance patch:** patch balance only; assert balance changed, + nonce/code preserved; `AccountChange.balance == Some((old,new))`, + `nonce/code_hash == None`. +6. **Account code patch:** patch code; assert `code_hash` recomputed + (`Bytecode::hash_slow`), `code_hash` delta recorded; balance/nonce preserved. +7. **Cold account patch:** patch an absent account → skipped and surfaced in + `StateDiff.skipped_accounts`; explicit `AccountUpsert` materializes with + patched fields. +8. **Purge Account / AllStorage / Slots:** correct layers cleared; `PurgeRecord` + counts (`slots_removed`, `account_removed`) correct on both layers. +9. **`apply_updates` fold + merge:** a mixed batch (Slot, Account, Purge) → + merged `StateDiff`; later-overrides-earlier ordering for same-key slots. +10. **Refold equivalence:** `purge_pool_storage` wrapper returns the same `usize` + as the pre-refold behavior on a seeded cache; `inject_storage_batch_fresh` + wrapper leaves the cache in the same state as the equivalent `apply_updates`. +11. **(Decision 2, if "normalize"):** `inject_v3_*` now writes through to the + backend (layer 2) — pin the new behavior. + +**Existing suites must stay green** — `tests/freshness.rs`, +`tests/cache_state.rs`, `tests/snapshot_overlay.rs`, the `protocols` cache tests. + +## 11. Docs, example & benchmark + +- **Example** `examples/state_update_apply.rs` (offline, `examples/support`): + build a `from_backend` cache, apply a batch — a `Slot`, an `Account` balance + patch, and a `Purge { Slots }` — then print the returned `StateDiff` (slots + changed, account deltas, purge records). Add a row to the README "Examples" + table (Advanced). +- **Benchmark** `benches/state_update.rs` (offline): `apply_updates` throughput + across batch sizes (1 → 1000), and per-variant cost (Slot vs Account vs Purge), + building the cache once. Register `[[bench]]` in `Cargo.toml` and add a row to + the README "Benchmarks" table. Mirror `benches/freshness.rs` structure. +- **CHANGELOG**: an `### Added` entry for the state-update vocabulary + apply + + diff; if Decision 2 = normalize, a `### Changed` entry for the `inject_v3_*` + layer behavior. +- **ROADMAP**: flip the Phase 3 row to **Done** with the landing branch, mirroring + the Phase 2 "Landed on …" paragraph. +- **KNOWN_ISSUES**: if Decision 2 = normalize, add an entry recording the + `inject_v2/v3_*` layer-behavior change (and that tests now pin it). + +## 12. Decisions (LOCKED) + +> Mirrors the Phase 2 "locked decisions" gate. Both were confirmed with the user +> on 2026-06-15 before the acceptance tests were authored. + +**Decision 1 — `Account` variant shape. → LOCKED: partial `AccountPatch`.** +`Account { address, patch: AccountPatch { balance: Option, nonce: +Option, code: Option } }` (§4.2). Each `Some` overwrites, `None` +leaves as-is. Best fit for event-derived writes (one field at a time); no revm +type leaked into the public vocabulary. (The full-`AccountInfo` alternative from +the ROADMAP sketch is **not** taken.) + +**Decision 2 — `inject_v2/v3_*` refold behavior. → LOCKED: normalize to +write-through.** Refold the `protocols`-gated `inject_v2_pool_metadata` / +`inject_v3_*` helpers onto the write-through `StateUpdate::Slot` primitive +(backend + overlay-if-present) instead of today's layer-1-only write. This is a +deliberate behavior change and **requires**: a CHANGELOG `### Changed` entry, a +KNOWN_ISSUES entry, and test #11 (§10) pinning the new write-through behavior. +The `protocols` pool tests do not pin layer placement, so they stay green. + +## 13. Build order (commit per step, green each time) + +1. `src/state_update.rs`: `StateUpdate`, `PurgeScope`, `AccountPatch`, + `StateDiff`, `AccountChange`, `PurgeRecord` + constructors/helpers + unit + tests; `lib.rs` re-exports. +2. `EvmCache::apply_update` / `apply_updates` (Slot, Account, Purge) + the + `tests/state_update.rs` integration tests. +3. Refold `inject_storage_batch_fresh`, `purge_account`, `purge_pool_storage`, + `purge_pool_slots`, `override_account_code*`, and (per Decision 2) + `inject_v2/v3_*`; route the freshness drain through `apply_updates`. +4. Example + benchmark + README rows. +5. Docs (module `//!`, item rustdoc, doctest), CHANGELOG, ROADMAP → Done, + KNOWN_ISSUES (if normalize). + +## 14. Final acceptance + +Both feature configs green (§0). All new + existing tests pass (`tests/state_update.rs` ++ the in-module unit tests + the untouched existing suites). The example runs +offline and prints a non-trivial `StateDiff`. The benchmark builds and runs. +Report: what landed per file, the public API added, the refold map (with any +behavior change called out), test coverage, and the verification output. + +--- + +## 15. Addendum — relative / read-modify-write updates (decisions LOCKED) + +> Added 2026-06-15 after the §1–§14 surface landed (green, uncommitted). Motivated +> by the event-driven balance-tracking case: a caller indexing ERC-20 `Transfer` +> logs to keep a tracked account's balance hot only learns the **delta** +> (`amount`), not the resulting absolute balance, so the engine must support +> *relative* updates — read the current value, apply a mutation, write back. The +> §4 vocabulary today is **absolute-only**; this addendum adds the relative +> capability. It remains generic core (no protocol knowledge; the slot derivation +> and the ± decision belong to the caller / the Phase-4 decoder). + +### 15.1 The correctness constraint (non-negotiable) + +A relative update is only valid against a value the cache **actually holds**. An +un-fetched ("cold") slot has *no* value — and `cached_storage_value` / `apply_slot` +treat absent as `ZERO`. Applying `delta` to a cold slot would compute +`0 ± amount`, write a wrong value, and (write-through) make it authoritative — +silently corrupting state. Therefore relative application must be **cold-aware**: +apply only when the current value is known; otherwise **skip and surface** it. + +### 15.2 Locked decisions + +**Decision 3 — shape. → LOCKED: vocabulary variant + method.** Add *both*: +(a) a data-level [`StateUpdate::SlotDelta`] variant (so it flows through +`apply_updates` and a Phase-4 `EventDecoder` can emit it as data); (b) a general +`EvmCache::modify_slot` closure escape hatch for arbitrary transforms. + +**Decision 4 — cold-slot handling. → LOCKED: skip & surface.** A `SlotDelta` +targeting a slot absent from **both** layers is **not applied**; it is recorded +in `StateDiff.skipped` so the caller can fetch+seed the true value (the next read +otherwise lazily fetches it). For `modify_slot`, the closure receives +`Option` (`None` when cold) and decides. Overflow is **saturating** +(`Add` clamps at `U256::MAX`, `Sub` at `U256::ZERO`). + +### 15.3 Types (in `src/state_update.rs`) + +```rust +/// A relative storage-slot mutation: read the current value, transform it, write +/// back. Both directions saturate (`Add` at `U256::MAX`, `Sub` at `U256::ZERO`). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum SlotDelta { Add(U256), Sub(U256) } +impl SlotDelta { + /// Apply the (saturating) delta to a current value. + pub fn apply(self, current: U256) -> U256; +} + +// New variant on the existing enum: +pub enum StateUpdate { + Slot { address, slot, value }, + SlotDelta { address: Address, slot: U256, delta: SlotDelta }, // NEW + Account { address, patch }, + Purge { address, scope }, +} +impl StateUpdate { + /// Construct a relative slot update. + pub fn slot_delta(address: Address, slot: U256, delta: SlotDelta) -> Self; +} + +/// A relative update that could not be applied because the slot's current value +/// is unknown (not cached in either layer). Fetch+seed the slot, then retry. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct SkippedDelta { pub address: Address, pub slot: U256, pub delta: SlotDelta } + +// New field on the existing StateDiff (Default = empty): +pub struct StateDiff { + pub slots: Vec, + pub accounts: Vec, + pub purged: Vec, + pub skipped: Vec, // NEW +} +``` +`StateDiff::merge` also extends `skipped`. `is_empty()` / `len()` remain +**changes-only** (slots + accounts + purged) — a skip is *not* a change; document +that `skipped` is separate informational metadata (it does not affect +`is_empty`/`len`, so the §10 no-op/idempotence expectations are unchanged). + +### 15.4 `EvmCache` behavior + +- `apply_update(StateUpdate::SlotDelta { address, slot, delta })`: if + `cached_storage_value(address, slot)` is `Some(current)`, write + `delta.apply(current)` through both layers (reuse the §5.1 write path) and push + a `SlotChange` iff it changed; if `None` (cold), push a `SkippedDelta` to + `diff.skipped` and write nothing. +- `modify_slot(&mut self, address: Address, slot: U256, f: impl FnOnce(Option) -> Option) -> Option`: + call `f` with the current cached value (`None` if cold); if it returns + `Some(new)`, write-through (same path) and return the `SlotChange` iff + `old.unwrap_or(ZERO) != new`; if it returns `None`, write nothing and return + `None`. (The caller owns the cold/overflow policy here; e.g. + `|cur| cur.map(|v| v.saturating_add(amount))` implements skip-on-cold.) +- Refactor the dual-layer slot write out of `apply_slot` into a private + `write_slot_through(address, slot, value)` helper shared by `apply_slot`, + the `SlotDelta` handler, and `modify_slot` (one write path). + +Scope note: account-native-ETH-balance relative updates (an `AccountDelta` / +`modify_account_balance`) are **out of scope** here — the asked case is ERC-20, +whose balances are storage slots. Document that they can be added symmetrically +later if native-ETH tracking is needed. + +### 15.5 Tests (append to `tests/state_update.rs`) + +- `slot_delta_add_applies_to_hot_slot` — seed (backend) 100, `Add(50)` → 150; + `diff.slots == [SlotChange{100,150}]`, `diff.skipped` empty. +- `slot_delta_sub_saturates_at_zero` — seed 30, `Sub(50)` → 0. +- `slot_delta_add_saturates_at_max` — seed `MAX-1`, `Add(10)` → `MAX`. +- `slot_delta_cold_slot_is_skipped_and_surfaced` — fresh (uncached) slot, `Add(50)` + → not applied; `diff.slots` empty; `diff.skipped == [SkippedDelta{..}]`; + `cached_storage_value` still `None`. +- `slot_delta_writes_through_both_layers` — overlay-resident slot (install account + + seed), `Add` updates both overlay and backend. +- `modify_slot_applies_transform` — seed 10, `|c| c.map(|v| v*2)` → 20. +- `modify_slot_closure_skips_cold` — fresh slot, `|c| c.map(|v| v+1)` → returns + `None`, nothing written, slot still cold. +- `modify_slot_can_write_absolute_on_cold` — fresh slot, `|_| Some(7)` → writes 7 + (caller's explicit choice), `SlotChange{0,7}`. +- `state_diff_merge_includes_skipped` — merge concatenates `skipped`. +- `balance_tracking_scenario` — **the motivating end-to-end case**: seed two + holders' balance slots, then apply a `Transfer` as + `[SlotDelta::Sub(amount) on from, SlotDelta::Add(amount) on to]` via + `apply_updates`; assert both balances are correct and `from + to` is conserved. + +### 15.6 Docs / example / changelog + +- Rustdoc on every new item; the module `//!` doc gains a short "relative updates" + paragraph (the cold-aware read-modify-write rule). +- Extend `examples/state_update_apply.rs` (or a focused addition) to show a + `SlotDelta` balance bump **and** a cold-slot skip surfaced via `diff.skipped`. +- Optionally extend `benches/state_update.rs` with a `SlotDelta` apply case + (not required). +- CHANGELOG `### Added`: the relative-update vocabulary (`SlotDelta`, + `StateUpdate::SlotDelta`, `modify_slot`, `StateDiff.skipped`). Note the + `StateDiff` field addition under the pre-1.0 break policy. +- ROADMAP: fold a one-line mention into the Phase 3 "Landed on …" paragraph. + +### 15.7 Acceptance (addendum) + +All of §14 plus: the new tests pass; `diff.skipped` is exercised; the +`balance_tracking_scenario` demonstrates the motivating use case end-to-end; both +feature configs stay green. + +--- + +## 16. Addendum — post-audit remediation (COMPREHENSIVE, decisions LOCKED) + +> Added 2026-06-15 after a 5-lens adversarial audit of the §1–§15 surface (bugs, +> API design, coverage, benchmarks). The user selected the **Comprehensive** +> remediation scope. This section is the precise build contract for that scope. +> Every item below is LOCKED. Where this section conflicts with earlier sections, +> prefer this. Hard rules of §0 still apply (offline tests, both feature configs +> green, MSRV 1.88, edition 2024, no new deps, unsigned commits). + +### 16.0 The correctness bug (P0 — must fix first) + +**Defect (audit HIGH + MED, verified with a reproducer):** the cold-aware safety +guarantee rests on `EvmCache::cached_storage_value` returning what the EVM would +`SLOAD`. That invariant is **false** for an overlay account whose revm +`account_state` is `StorageCleared` or `NotExisting`: for a slot absent from the +overlay storage map, the live `CacheDB::storage`/`storage_ref` returns **ZERO and +never consults the backend**, but `cached_storage_value` (src/cache/mod.rs +~1404-1412) falls through to the BlockchainDb backend and returns +`Some(backend_value)`. Consequences: a `SlotDelta`/`modify_slot` computes +`delta.apply(backend_value)` against a base the EVM never sees (silent +corruption), and `apply_slot` records a wrong `SlotChange.old` and mis-gates the +change predicate. `install_mock_erc20` produces exactly this state +(`replace_account_storage` ⇒ `StorageCleared`), and a backend-only seed via +`inject_storage_batch` is invisible to the EVM — which is why +`balance_tracking_scenario` currently passes while asserting against the buggy +accessor instead of a real `SLOAD`. + +**Fix (LOCKED): make `cached_storage_value` `account_state`-aware**, mirroring +`CacheDB::storage_ref`: +```rust +pub fn cached_storage_value(&self, address: Address, slot: U256) -> Option { + if let Some(db_account) = self.db.cache.accounts.get(&address) { + if let Some(value) = db_account.storage.get(&slot) { + return Some(*value); + } + // Match the EVM SLOAD: a StorageCleared / NotExisting overlay account + // reads a missing slot as ZERO and never consults the backend. + if matches!( + db_account.account_state, + AccountState::StorageCleared | AccountState::NotExisting + ) { + return Some(U256::ZERO); + } + } + let storage = self.blockchain_db.storage().read(); + storage.get(&address).and_then(|s| s.get(&slot).copied()) +} +``` +`AccountState` is revm's enum on `DbAccount` (resolve the exact import path; it is +re-exported from the revm database crate already in use). This single fix repairs +the `SlotDelta`/`modify_slot` base read (HIGH) and `apply_slot`'s `old`/predicate +(MED) at once, and also closes the pre-existing same-root mismatch shared by +`verify_slots` / `inject_storage_batch_fresh`. + +**Tests must validate the EVM SLOAD, not the accessor:** +- **New invariant test** (the red reproducer): with `install_mock_erc20` + + backend-only `inject_storage_batch` seed of slot=100, assert + `cached_storage_value(token, slot) == Some(ZERO)` **and** that it equals what a + real `balance_of`/SLOAD reads (both ZERO). Pre-fix this returns `Some(100)`. +- **Re-point `balance_tracking_scenario`**: seed the holder balance slots in an + **EVM-visible** way (overlay-resident via `db_mut().insert_account_storage`, so + the slots are real to the EVM), apply the `SlotDelta` transfer, and assert the + results via `balance_of` (a real `SLOAD`) in addition to `cached_storage_value`. +- **Present-as-ZERO vs cold**: a slot known to be ZERO (overlay-resident `0`, or a + `StorageCleared` account's absent slot) is **hot** — `SlotDelta::Add(50)` ⇒ 50, + recorded in `diff.slots`, **not** in `diff.skipped`. Cold (no overlay account + **and** no backend value) stays skip-and-surface. Add a test for the hot-zero + case (it is currently the untested seam between Decision-4 skip and apply). + +### 16.1 No-op / cold `Account` patch must not materialize a backend account (audit LOW; tightened in Phase 5) + +`apply_account_patch` (src/cache/mod.rs ~1331-1340) writes the patched +`AccountInfo` into the backend **unconditionally**, so an all-`None` (or +otherwise no-change) patch on an address absent from both layers inserts +`AccountInfo::default()` into the shared backend map while returning an **empty** +diff — breaking no-op parity with the Slot path and (per the cold-account hazard) +masking a future RPC fetch. **Fix (LOCKED):** compute the change first; **only +write-through when at least one field actually changes** (i.e. skip both layer +writes and return `None` when the patched `info` equals the loaded base). +Phase 5 tightened the cold-account contract further: a real field change on an +address absent from both layers now skips and records `SkippedAccountPatch` in +`StateDiff.skipped_accounts`; explicit materialization uses +`StateUpdate::AccountUpsert`. Add no-op and cold-skip idempotence tests (patching +balance to its current value ⇒ empty diff; cold balance patch ⇒ no backend account +materialized). + +### 16.2 Cold absolute-`Account`-patch hazard — fixed in Phase 5 (audit LOW) + +A *partial* absolute `Account` patch on a cold (un-fetched) address used to write +default nonce/code through the shared backend, masking the real on-chain account. +Phase 5 changed this contract: `StateUpdate::Account` is cold-aware and records a +`SkippedAccountPatch` instead; callers that intentionally want a synthetic/default +account use `StateUpdate::AccountUpsert`. + +### 16.3 `serde` on the vocabulary (audit HIGH gap) + +`serde` is a non-optional crate dependency and other public types +(`StorageAccessList`, `PrefetchRegistry`) already derive/serialize. The event +pipeline (the stated motivation) needs to serialize `StateUpdate`s and ship +`StateDiff`s. **Fix (LOCKED):** derive `serde::Serialize, serde::Deserialize` +**unconditionally** on `SlotDelta`, `StateUpdate`, `AccountPatch`, `PurgeScope`, +`StateDiff`, `AccountChange`, `PurgeRecord`, `SkippedDelta`, the new `BalanceDelta` +payload / `SkippedBalanceDelta` (§16.5), **and** `SlotChange` (src/freshness.rs). +Add a JSON round-trip test for a representative `StateUpdate` set and a `StateDiff`. +(All fields are `Address`/`U256`/`B256`/`Bytes`/`u64`/`usize`/`bool` — derives +compile today with the alloy serde features already enabled.) + +### 16.4 `#[non_exhaustive]` on output/record types (audit HIGH) + +`StateDiff` just grew a field as a documented pre-1.0 break, and §16.5 adds +another (`skipped_balances`). **Fix (LOCKED, scoped):** add `#[non_exhaustive]` +to **`StateDiff`** (the aggregate that demonstrably grows) and **`AccountPatch`** +(builder-constructed via `.balance()/.nonce()/.code()` + `Default`). Both are +still constructed by external callers/tests through `Default` + field-assignment +(`StateDiff`) or the builders (`AccountPatch`), so future field additions are +non-breaking at zero ergonomic cost. (`StateUpdate` and `PurgeScope` already are +`#[non_exhaustive]`.) + +**Deliberately NOT `#[non_exhaustive]`** — the leaf record types `SlotChange`, +`AccountChange`, `PurgeRecord`, `SkippedDelta`, and `SkippedBalanceDelta`. These +are routinely **constructed as struct literals in equality assertions** by both +the test suite (`diff.skipped == vec![SkippedDelta { .. }]`, +`diff.slots == vec![SlotChange { .. }]`) and downstream users testing against a +returned diff. `#[non_exhaustive]` would forbid that external construction — a +real, non-zero cost that outweighs the speculative benefit of these stable, +fully-determined shapes gaining a field. (This is a deliberate, documented +departure from the audit finding, which assumed these were read-only; assertion +construction is the counter-case.) + +### 16.5 New capability — account-native-balance delta (audit MED gap) + +Relative-update symmetry: `SlotDelta` covers ERC-20 (storage) balances, but +native-ETH tracking (value transfers, coinbase, selfdestruct) is learned as a +delta too. **Add (LOCKED):** +```rust +// reuse SlotDelta (Add/Sub, saturating) for the relative amount +pub enum StateUpdate { /* … */ BalanceDelta { address: Address, delta: SlotDelta } } // NEW variant +impl StateUpdate { pub fn balance_delta(address: Address, delta: SlotDelta) -> Self; } // NEW ctor + +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[non_exhaustive] +pub struct SkippedBalanceDelta { pub address: Address, pub delta: SlotDelta } // NEW + +pub struct StateDiff { /* … */ pub skipped_balances: Vec } // NEW field + +impl EvmCache { + /// Read-modify-write the native balance. `f` gets the current cached balance + /// (`None` if the account is absent from both layers); `Some(new)` writes it + /// through (preserving nonce/code), `None` writes nothing. + pub fn modify_account_balance( + &mut self, address: Address, f: impl FnOnce(Option) -> Option, + ) -> Option; +} +``` +- **Cold-aware:** "cold" for a balance = the account is absent from **both** layers + (balance unknown). `account_state` does **not** matter here (it governs storage, + not the basic `AccountInfo`). A `BalanceDelta` on a cold account is **not + applied**; it is surfaced in `StateDiff.skipped_balances` (avoids the §16.2 + masking — we never write a default account). On a present account, load the full + `AccountInfo` (overlay ▸ backend), apply the saturating delta to `info.balance`, + preserve nonce/code, write-through (backend always, overlay-if-present), record + an `AccountChange` (balance only) iff it changed. +- `modify_account_balance` is the closure analog (same load/cold rules; `f` decides). +- `StateDiff::merge` extends `skipped_balances`. `is_empty`/`len` stay + **changes-only**. `has_skipped`/`skipped_len`/`is_fully_applied` (§16.6) count + **both** `skipped` and `skipped_balances`. +- Tests: hot apply (Add/Sub/saturation, AccountChange recorded, nonce/code + preserved), cold skip-and-surface (`skipped_balances` populated, no backend + account materialized), `modify_account_balance` hot/cold/`None`. + +### 16.6 Discoverable skip accessors + loud docs (audit MED footgun) + +A cold-skipped relative update is invisible to the natural `is_empty()`/`len()` +success check, so a dropped balance update can break conservation silently. +**Add (LOCKED)** on `StateDiff`: +- `has_skipped(&self) -> bool` — `!skipped.is_empty() || !skipped_balances.is_empty()`. +- `skipped_len(&self) -> usize` — `skipped.len() + skipped_balances.len()`. +- `is_fully_applied(&self) -> bool` — `!self.has_skipped()`. + +Document prominently on `apply_update`/`apply_updates` that after relative +updates the caller **must** check `has_skipped()`/`skipped`/`skipped_balances` — a +cold target is dropped, not applied. Mirror this in the example. + +### 16.7 Constructor symmetry (audit LOW) + +Add convenience constructors for parity with `slot`/`balance`/`purge`/`slot_delta`: +`StateUpdate::nonce(address, u64)`, `StateUpdate::code(address, Bytes)`, +`StateUpdate::account(address, AccountPatch)`. + +### 16.8 Coverage gaps (audit — all listed) + +Add tests (in `tests/state_update.rs` unless noted). Each must assert the +**layer-correct** outcome, not just the accessor: +- **Account-patch backend-write-always:** on an overlay-present account, assert + `backend_balance(...)` updates (not only overlay). +- **Account-patch no-overlay-materialization:** after patching a backend-only / + absent account that *does* change, assert no *new* overlay account is + materialized where the spec says none should be (mirror + `apply_slot_no_overlay_account_is_not_materialized`). +- **Backend-only account patch:** seed an account only in the backend + (`unchecked_blockchain_db().accounts().write().insert`), patch balance, assert + `AccountChange.balance == Some((old,new))`, backend updated, overlay still absent. +- **Nonce-only** and **multi-field (balance+nonce+code)** patches: assert the + respective `AccountChange` fields are `Some`/`None` correctly. +- **Empty-code clear:** patch `code(Bytes::new())` over non-empty code ⇒ + `code_hash` → `KECCAK_EMPTY`; patching empty over already-empty ⇒ `None`. +- **Account-patch idempotence/no-op** (also pins §16.1): balance→current ⇒ empty + diff, no backend materialization. +- **Decision-2 pins:** `inject_v2_pool_metadata_writes_through_to_backend` and + `inject_v3_ticks_writes_through_to_backend` (protocols-gated), mirroring the + existing bitmap test. +- **Purge edges:** purge of an absent address ⇒ `PurgeRecord{account_removed:false, + slots_removed:0}`; `PurgeScope::Slots` with some slots absent ⇒ `slots_removed` + counts only present backend slots; overlay-vs-backend accounting (overlay-only + slots not counted in `slots_removed`). +- **`modify_slot` write-through layers:** overlay-present ⇒ both layers updated; + absent ⇒ backend only, no overlay materialized. +- **Batched == sequential equivalence** (the perf-fast-path safety net, §16.9): + a mixed `apply_updates([...])` batch (distinct addresses, a same-address repeat, + and a `Purge` mid-batch) leaves **byte-identical** layer state **and** an + equivalent merged `StateDiff` to applying each update via `apply_update` in + sequence. This test must pass both before and after the perf work. + +### 16.9 Performance (audit — benchmarks) + +Benchmarks showed `apply_updates` ≈ 4.4× the per-element cost of raw +`inject_storage_batch`, dominated by **per-update `RwLock` churn** (a read lock for +the old value + a separate write lock per slot) and a redundant `SlotDelta` read. +This matters because `inject_storage_batch_fresh` and the `inject_v3_*` writers now +route through `apply_updates` for **bulk** seeding. **Fix (LOCKED):** +1. **Eliminate the `SlotDelta` double read:** the `SlotDelta` arm already reads + `cached_storage_value` for `current`; build the `SlotChange` from that value and + call the shared write path directly instead of routing through `apply_slot` + (which re-reads the same slot). +2. **Batched single-lock fast-path** for `apply_updates`: process consecutive + `Slot`/`SlotDelta` writes holding the backend storage write-guard **once** for + the run (overlay access is lock-free on `self.db.cache.accounts`). Preserve + apply order: when an `Account`/`Purge` update is reached, **drop the guard + first** (those take `accounts()` / `storage()` locks themselves — holding the + storage write-guard across `apply_purge` would deadlock on the non-reentrant + `RwLock`), process it, then lazily re-acquire on the next slot run. Correctness + is pinned by the §16.8 batched==sequential equivalence test and the existing + refold-equivalence tests; **do not** weaken any of them. The old-value read must + stay `account_state`-aware (§16.0) even inside the held guard. +3. Single-update `apply_update`/`apply_slot` may keep the read-then-write split + (correctness first); the batch path is where the lock win is realized. + +### 16.10 Missing benchmarks (audit) + +Extend `benches/state_update.rs` (keep existing cases): `SlotDelta` hot-apply and +cold-skip; `modify_slot`; a **heterogeneous** `apply_updates` batch (Slot + +Account + Purge); `Account` **code** patch (the `Bytecode::new_raw` + `hash_slow` +keccak — likely the most expensive single apply); `PurgeScope::Account` and +`PurgeScope::Slots`; a **distinct-address** `apply_updates` batch (the only fair +apples-to-apples vs the `inject_storage_batch` baseline). All benches stay offline +and must build under `cargo bench --no-run`. + +### 16.11 Docs / CHANGELOG / ROADMAP + +- Rustdoc on every new item; update the `state_update` module `//!` doc to cover + `BalanceDelta`, the skip accessors, and the cold-account warning. +- `examples/state_update_apply.rs`: add a `BalanceDelta` bump + a cold + `BalanceDelta` surfaced via `diff.skipped_balances`, and use `has_skipped()`. +- CHANGELOG: `### Fixed` (the `cached_storage_value` corruption bug; the no-op + Account materialization) and `### Added` (`serde`; `#[non_exhaustive]`; + `BalanceDelta`/`modify_account_balance`/`SkippedBalanceDelta`/ + `StateDiff.skipped_balances`; `has_skipped`/`skipped_len`/`is_fully_applied`; + `StateUpdate::nonce`/`code`/`account`). Note the additive `StateDiff` field and + the `#[non_exhaustive]` additions under the pre-1.0 break policy. +- ROADMAP: extend the Phase 3 "Landed on …" paragraph with the §16 remediation. +- KNOWN_ISSUES: the §16.2 cold-account-patch entry. + +### 16.12 Acceptance (remediation) + +All of §14 plus: every §16 test passes; the corruption reproducer is **red before / +green after** the §16.0 fix; `balance_tracking_scenario` validates via a real +`SLOAD`; the batched==sequential equivalence test passes; `serde` round-trips; +both feature configs green (`cargo test`, `clippy` default + `--no-default-features`, +`fmt`, `RUSTDOCFLAGS=-D warnings doc`); `cargo bench --no-run` builds all benches; +the example runs offline and shows a skipped relative update via `has_skipped()`. diff --git a/docs/phase-4-spec.md b/docs/phase-4-spec.md new file mode 100644 index 0000000..78d2d6d --- /dev/null +++ b/docs/phase-4-spec.md @@ -0,0 +1,728 @@ +# Phase 4 implementation spec — event pipeline + adapters (Pillar B.2) + +Implementation contract for the **reader half** of Pillar B: turn an on-chain +`Log` into the Phase 3 [`StateUpdate`] vocabulary, apply it through +`apply_updates`, and keep the cache **reactively fresh** from the event stream — +with reconciliation, reorg handling, and freshness wiring. Read this **with** +[`ROADMAP.md`](ROADMAP.md) (the "Phase 4" row, the "Pillar B — event → state +pipeline" section, and the "Hard problems to resolve" list) and +[`phase-3-spec.md`](phase-3-spec.md) (the writer half this builds on). This +document is the precise build contract; where they overlap, prefer this. + +Phase 3 built the writer half (`StateUpdate` + `apply_updates` with cold-aware +`SlotDelta`/`BalanceDelta` RMW and `account_state`-correct reads). Phase 4 builds +the decoder, the protocol adapters, and the orchestration that drives them. + +## 0. Ground rules (non-negotiable) + +- **Branch:** create `phase-4-event-pipeline` off the current + `phase-3-state-updates` HEAD. Commit there in logical steps. Do **not** push, + do **not** tag, do **not** open a PR (the overseer does that). Commits must be + **unsigned**: `git -c commit.gpgsign=false commit …` (the 1Password signing + agent is unavailable here). End every commit message with exactly: + `Co-Authored-By: Claude Opus 4.8 (1M context) ` +- **Generic core vs `protocols`.** The pipeline, the `EventDecoder`/`StateView` + traits, the `DecoderRegistry`, the ERC-20 decoder, and the `SlotMasked` + vocabulary addition are **generic core** — they must compile and lint with + `--no-default-features`. Only the UniswapV3 adapter (`uniswap_v3`) is gated + behind the `protocols` feature. +- **Green bar at every commit, both feature configs:** + - `cargo fmt --all --check` + - `cargo clippy --all-targets --no-deps -- -D warnings` + - `cargo clippy --lib --no-default-features --no-deps -- -D warnings` + - `cargo test` + - `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps` + - `cargo bench --no-run` (all benches build offline) +- MSRV is 1.88 — no newer-than-1.88 std APIs. Edition 2024. +- **Do not break existing behavior or any existing test.** The Phase 3 surface + (`StateUpdate`, `apply_updates`, `modify_slot`, `StateDiff`) keeps its shape; + Phase 4 *adds* the `SlotMasked` variant and the `skipped_masks` diff field + under the pre-1.0 break policy (both already-`#[non_exhaustive]` types). +- **No new dependencies.** `alloy-primitives` (with `Log`), `alloy-sol-types` + (`sol!` / `SolEvent`), `alloy-provider`, `futures`, and `tokio` are already + present. Decode logs with `sol!`-generated event types + `SolEvent`, not + hand-rolled byte slicing. + +## 1. Objective & scope + +Today the crate can *write* targeted state but has no way to *derive* those +writes from chain activity: a caller must hand it concrete `StateUpdate`s. Phase +4 closes the loop — decode a `Log` into `StateUpdate`s, apply them, and run the +reactive maintenance (reconcile, reorg) that keeps event-derived state honest. + +**In scope:** + +1. **`StateUpdate::SlotMasked`** — a cold-aware read-modify-write *masked* slot + write (`(old & !mask) | (value & mask)`), so a pure decoder can express a + partial update to a **packed** storage word (e.g. V3 `slot0`) without knowing + or clobbering the bits it does not own. Generic core (§4.1). +2. **`EventDecoder` + `StateView`** — the decoder trait (`Log` + read-only + pre-state view → `Vec`) and the narrow read-only cache view it is + handed. `EvmCache` implements `StateView`. Generic core (§4.2). +3. **`DecoderRegistry`** — dispatches a log to the decoder(s) registered for its + emitting address (and/or topic0) and concatenates their output. Generic core + (§4.3). +4. **`Erc20TransferDecoder`** — generic ERC-20 `Transfer` → relative balance + `SlotDelta`s (the §15 reactive-balance case, now log-driven). Generic core + (§5). +5. **`UniswapV3Decoder`** — `protocols`-gated adapter: `Swap` → `slot0` + (masked sqrtPriceX96 + tick) + `liquidity`; `Mint`/`Burn` → per-tick + `liquidityGross`/`liquidityNet`, the `initialized` flag, `tickBitmap` word + flips, and the global `liquidity` (conditional on the current tick). Computed + against the `StateView` (tick maintenance is inherently RMW). (§6). +6. **`EventPipeline`** — the orchestration: `ingest_logs` (decode+apply + **log-by-log**, in order, recording touched state for reorg tracking), + `reorg_to` (purge-and-resync addresses touched after the new head), and + `reconcile` (sampled RPC re-read via `verify_slots`: correct **and** alarm). + Generic core (§7). +7. **Freshness wiring** — `BlockDigest` surfaces the touched `(address, slot)` + set so a caller can classify event-derived slots (`valid_through` / `pin`) and + call `FreshnessController::on_new_block`. No controller internals change (§8). +8. Offline example, benchmark, docs, CHANGELOG, ROADMAP → Done (§11). + +**Out of scope (document as follow-ups; do not build):** +- **A concrete WS transport / live subscription loop.** The async `drive` + convenience (§7.5) is generic over a log source and is exercised only by the + offline example feeding a vec-backed source; a production WS/`subscribe_logs` + adapter is a follow-up. The *tested* surface is the synchronous core. +- **V3 fee-growth / oracle observation maintenance.** Event-derived tick init + does **not** reconstruct `feeGrowthOutside0/1X128` (slots +1/+2) or oracle + observations — those are not derivable from `Mint`/`Burn`/`Swap`. Swap + *price/liquidity quoting* is unaffected; fee-accounting reads are not + maintained. Document as a KNOWN_ISSUE with reconcile/purge as the backstop + (§6.4). +- **Non-Uniswap-layout V3 (Slipstream slot0).** The adapter assumes the + Uniswap/Pancake `slot0` bit layout (only base slots differ). Slipstream's + different `slot0` packing is a follow-up. +- **COW snapshots** (Phase 5). The pipeline mutates the existing `EvmCache` + layers via `apply_updates`. + +## 2. Reuse these existing pieces (do not reinvent) + +- **`StateUpdate` / `apply_update` / `apply_updates` / `modify_slot`** + (`state_update.rs`, `cache/mod.rs`) — the write half. The pipeline applies + decoded updates through `apply_updates`; the `SlotMasked` handler reuses the + private `write_slot_through` and the `account_state`-aware + `cached_storage_value` (§16.0). +- **`EvmCache::cached_storage_value`** — the `StateView::storage` + implementation (overlay ▸ backend ▸ `None`, `account_state`-correct). +- **`EvmCache::verify_slots`** (`cache/mod.rs`) — the synchronous + fetch-compare-inject reconciliation primitive. `reconcile` is a thin wrapper: + it samples event-derived slots and calls `verify_slots`; the returned + `Vec` is the drift report (verify_slots already injected the fresh + chain values — correct + alarm). +- **`EvmCache::purge_account` / `apply_update(Purge { … })`** — the reorg + purge mechanism. `reorg_to` purges touched addresses through `apply_updates` + of `Purge` updates so the next read re-fetches. +- **`inspector::TransferInspector::parse_transfer`** + the + `TRANSFER_EVENT_SIGNATURE` constant (`src/inspector.rs`) — reuse for the + ERC-20 decoder's signature match and topic decoding (or reuse the same + `sol!` event). Do not redefine the signature constant. +- **`cache::storage_keys`** (`protocols`) — `V3_*`/`PANCAKE_V3_*` slot + constants, `v3_tick_info_storage_keys_with_base`, + `v3_tick_bitmap_storage_key_with_base`, `i256_from_i24`, `i128_to_u256`. The + V3 adapter reuses these for slot derivation and packing. +- **`freshness::{FreshnessController, FreshnessRegistry, Validity, SlotChange}`** + — the freshness wiring target. `SlotChange` is the reconcile-report element. +- **`alloy_sol_types::{sol, SolEvent}`** — generate `Swap`/`Mint`/`Burn` and + ERC-20 `Transfer` event types and decode with `SolEvent::decode_log_data`. +- **The offline harness** — `examples/support/mock.rs` (`offline_cache`, + `install_mock_erc20`, `MockERC20`, `MOCK_ERC20_BALANCE_SLOT`), + `tests/common`. The new tests/example build the cache over the mocked provider + and never touch the network. + +## 3. Module layout + +A new `src/events/` directory module (the crate's only other dir module is +`cache/`): + +- **`src/events/mod.rs`** (generic core): `EventDecoder`, `StateView`, + `DecoderRegistry`, `EventPipeline`, `BlockDigest`, `ReconcileReport`, + `ReorgConfig`, and the async `drive` convenience + its `LogSource` trait. The + module `//!` doc frames Pillar B.2 and the `!Send`-cache discipline. +- **`src/events/erc20.rs`** (generic core): `Erc20TransferDecoder` + config. +- **`src/events/uniswap_v3.rs`** (`#[cfg(feature = "protocols")]`): + `UniswapV3Decoder` + `UniswapV3Layout` config (base slots + tick spacing). +- **`src/state_update.rs`**: add the `SlotMasked` variant, the `slot_masked` + constructor, `SkippedMask`, and the `StateDiff.skipped_masks` field + + `merge`/`has_skipped`/`skipped_len` updates. +- **`src/cache/mod.rs`**: the `SlotMasked` apply arm (reusing `write_slot_through` + + the cold-aware read); `impl events::StateView for EvmCache`. +- **`src/lib.rs`**: `pub mod events;` + re-exports (§9). + +## 4. Core types & behavior + +### 4.1 `StateUpdate::SlotMasked` — cold-aware masked write (generic core) + +```rust +pub enum StateUpdate { + Slot { address, slot, value }, + SlotDelta { address, slot, delta }, + /// Set only the `mask` bits of a storage slot to the corresponding bits of + /// `value`, preserving the rest: `new = (old & !mask) | (value & mask)`. + /// Read-modify-write, **cold-aware** — a masked write to a slot absent from + /// both layers is not applied (the un-masked bits are unknown); it is + /// surfaced in [`StateDiff::skipped_masks`]. + SlotMasked { address: Address, slot: U256, mask: U256, value: U256 }, // NEW + BalanceDelta { address, delta }, + Account { address, patch }, + Purge { address, scope }, +} +impl StateUpdate { + pub fn slot_masked(address: Address, slot: U256, mask: U256, value: U256) -> Self; +} + +/// A masked write ([`StateUpdate::SlotMasked`]) skipped because the target slot +/// was cold (un-masked bits unknown). Fetch+seed the slot, then retry. +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct SkippedMask { pub address: Address, pub slot: U256, pub mask: U256, pub value: U256 } + +pub struct StateDiff { + pub slots: Vec, + pub accounts: Vec, + pub purged: Vec, + pub skipped: Vec, + pub skipped_balances: Vec, + pub skipped_masks: Vec, // NEW +} +``` + +`SkippedMask` is a leaf record constructed as a struct literal in equality +assertions, so — like `SkippedDelta`/`SkippedBalanceDelta` (§16.4) — it is +**not** `#[non_exhaustive]`. `StateUpdate` and `StateDiff` already are. + +Apply behavior (`cache/mod.rs`, mirrors the `SlotDelta` arm): +- Read `old = cached_storage_value(address, slot)` (cold-aware, §16.0). +- If `Some(old)`: `new = (old & !mask) | (value & mask)`; write through both + layers via `write_slot_through`; push a `SlotChange { old, new }` iff + `old != new`. +- If `None` (cold): push `SkippedMask` to `diff.skipped_masks`, write nothing. + +`StateDiff::merge` extends `skipped_masks`. `has_skipped` /`skipped_len` +include `skipped_masks`. `is_empty`/`len` stay **changes-only** (a skip is not a +change). serde derives on `SkippedMask`. Module `//!` doc gains a `SlotMasked` +paragraph (packed-word updates, cold-aware). + +> A masked write with `mask == U256::MAX` equals an absolute `Slot` write but +> **stays cold-skip** (a cold full-mask write is still skipped, unlike `Slot` +> which writes unconditionally). Decoders that want an unconditional absolute +> write use `Slot`; those that must preserve neighbouring bits use `SlotMasked`. + +### 4.2 `EventDecoder` + `StateView` + +```rust +/// Read-only view of current cached state handed to a decoder. +/// +/// Decoders that compute post-state from pre-state (e.g. V3 tick maintenance) +/// read through this; stateless decoders (ERC-20 `Transfer`, V3 `Swap`) ignore +/// it. The view never touches RPC — a slot absent from the cache reads `None`. +pub trait StateView { + /// Current cached value of `(address, slot)` (overlay ▸ backend ▸ `None`), + /// matching what the EVM would `SLOAD` (`account_state`-aware). + fn storage(&self, address: Address, slot: U256) -> Option; +} + +/// Decode one log into zero or more targeted [`StateUpdate`]s. +/// +/// `decode` is a pure function of `(log, pre-state)`: it performs no I/O and +/// emits data (the updates are serializable and replayable against matching +/// pre-state). The pipeline applies the result through `apply_updates`. +pub trait EventDecoder: Send + Sync { + fn decode(&self, log: &Log, view: &dyn StateView) -> Vec; +} +``` + +`EvmCache` implements `StateView` via `cached_storage_value`. Rationale for the +`StateView` parameter (a refinement of the ROADMAP's `fn decode(&self, log) -> +Vec` sketch): event-driven sync is fundamentally *events + +pre-state → post-state*. Most updates are expressible without pre-state +(`SlotDelta` and `SlotMasked` are RMW **at apply time**), but V3 tick +maintenance must read current `liquidityGross`/`liquidityNet`/`tick`/bitmap to +compute the next packed value, and a pure decoder cannot. Handing decoders a +narrow read-only view keeps the output as serializable `StateUpdate` data while +making the stateful adapters expressible and offline-testable (feed a stub +view). + +### 4.3 `DecoderRegistry` + +```rust +#[derive(Default)] +pub struct DecoderRegistry { /* global decoders + per-address decoders */ } + +impl DecoderRegistry { + pub fn new() -> Self; + /// Register a decoder consulted for **every** log. + pub fn register(&mut self, decoder: Arc) -> &mut Self; + /// Register a decoder consulted only for logs emitted by `address`. + pub fn register_for_address(&mut self, address: Address, decoder: Arc) -> &mut Self; + /// Decode `log` through every applicable decoder, concatenating the results + /// (address-scoped decoders first, then global), preserving order. + pub fn decode(&self, log: &Log, view: &dyn StateView) -> Vec; +} +``` + +Dispatch is by emitting address (`log.address`); topic0 filtering is the +decoder's own concern (each decoder returns `vec![]` for a log it does not +recognise). Keep it simple: address-scoped entries + a global list, both +consulted, output concatenated. + +## 5. `Erc20TransferDecoder` (generic core, `events/erc20.rs`) + +```rust +pub struct Erc20TransferDecoder { + /// Balance mapping slot per token (the `balanceOf` mapping's base slot). + balance_slots: HashMap, + /// Fallback balance slot for tokens not in the map. + default_balance_slot: U256, +} +impl Erc20TransferDecoder { + pub fn new(default_balance_slot: U256) -> Self; + pub fn with_token(mut self, token: Address, balance_slot: U256) -> Self; +} +impl EventDecoder for Erc20TransferDecoder { /* … */ } +``` + +Decode rule for a `Transfer(from, to, value)` log (signature match via +`TRANSFER_EVENT_SIGNATURE`; topics/data decoded like `parse_transfer`): +- `slot = balance_slots.get(token).copied().unwrap_or(default_balance_slot)`. +- `balance_key(owner) = U256::from(keccak256(abi_encode((owner, slot))))`. +- Emit, **skipping the zero-address leg** (mint = `from == 0`, burn = `to == 0`): + - if `from != Address::ZERO`: `SlotDelta::Sub(value)` on `balance_key(from)`. + - if `to != Address::ZERO`: `SlotDelta::Add(value)` on `balance_key(to)`. +- A non-`Transfer` log (wrong topic0, < 3 topics, < 32 data bytes) → `vec![]`. + +Cold balances follow the Phase 3 contract: the `SlotDelta` is skipped and +surfaced in `StateDiff.skipped` (the caller seeds the balance, or the next read +lazily fetches it). The decoder ignores the `StateView`. `value == 0` transfers +emit deltas of zero (a no-op at apply — empty diff); that is acceptable. + +## 6. `UniswapV3Decoder` (`protocols`, `events/uniswap_v3.rs`) + +```rust +#[derive(Clone, Debug)] +pub struct UniswapV3Layout { + pub slot0_slot: U256, // V3_SLOT0_SLOT (0) — Uniswap/Pancake + pub liquidity_slot: U256, // V3_LIQUIDITY_SLOT (4) / PANCAKE (5) + pub ticks_base_slot: U256, // V3_TICKS_BASE_SLOT (5) / PANCAKE (6) + pub tick_bitmap_base_slot: U256, // V3_TICK_BITMAP_BASE_SLOT (6) / PANCAKE (7) + pub tick_spacing: i32, // pool tickSpacing (for bitmap word/bit) +} +impl UniswapV3Layout { + pub fn uniswap(tick_spacing: i32) -> Self; // canonical Uniswap V3 slots + pub fn pancake(tick_spacing: i32) -> Self; // PancakeSwap V3 slots +} + +pub struct UniswapV3Decoder { + /// Per-pool layout (slot bases + tick spacing). A log from an unregistered + /// pool decodes to nothing. + pools: HashMap, +} +impl UniswapV3Decoder { + pub fn new() -> Self; + pub fn with_pool(mut self, pool: Address, layout: UniswapV3Layout) -> Self; +} +impl EventDecoder for UniswapV3Decoder { /* … */ } +``` + +`tick_spacing` is required for `Mint`/`Burn` bitmap maintenance: the tickBitmap +is keyed by the **compressed** tick `tick / tick_spacing`. A log from a pool not +in `pools` → `vec![]`. Match events by topic0 (`Swap`/`Mint`/`Burn` signature +hashes from `sol!`); decode with `SolEvent`. + +### 6.1 `Swap` → price + liquidity (stateless) + +`Swap(sender, recipient, amount0, amount1, sqrtPriceX96, liquidity, tick)`: +- **slot0** (`SlotMasked`, preserves observation/feeProtocol/`unlocked` bits): + - `mask = (U256::from(1) << 184) - 1` (low 184 bits = sqrtPriceX96 [0,160) + + tick [160,184)). + - `value = U256::from(sqrtPriceX96) | (tick_24bit << 160)` where `tick_24bit` + is the int24 two's-complement low-24-bits of `tick` + (`U256::from(tick as i32 as u32 & 0x00FF_FFFF)`). + - Emit `StateUpdate::slot_masked(pool, slot0_slot, mask, value)`. +- **liquidity** (absolute — the event carries the post-swap pool liquidity): + - `StateUpdate::slot(pool, liquidity_slot, U256::from(liquidity))`. + +The `unlocked` bit (bit 240) and observation/fee bits are **preserved** by the +mask — clobbering `unlocked` to 0 would make a subsequent quote/swap revert +`LOK`. This is the headline correctness reason for `SlotMasked`. Stateless +(ignores the view); a cold slot0 → `skipped_masks` (the pool must be seeded +first). + +### 6.2 `Mint` → tick + liquidity maintenance (stateful, reads `StateView`) + +`Mint(sender, owner, tickLower, tickUpper, amount, amount0, amount1)` adds +`amount` (uint128 liquidity) over `[tickLower, tickUpper)`. For **each** of +`tickLower` and `tickUpper`, and for the global liquidity, compute the post-state +from the current cached value (read via `view.storage`); emit an **absolute** +`Slot` write of the recomputed word (a packed word recomputed from known +pre-state is an absolute write, not a delta). If a needed word is **cold** +(`view.storage` → `None`), **skip that update and surface it** as a +`SkippedMask`/`SkippedDelta` (choose `SkippedDelta` with a zero-amount marker is +wrong — use a dedicated skip; see §6.5) so the caller knows the pool tick state +is incomplete (re-seed via `inject_v3_ticks`). + +Per tick (`tick` ∈ {`tickLower`, `tickUpper`}): +- **Tick slot +0** (`liquidityGross` [0,128) ‖ `liquidityNet` [128,256) signed): + - base = `v3_tick_info_storage_keys_with_base(tick, ticks_base_slot)[0]`. + - read current word; `gross = low128`, `net = high128 as i128`. + - `gross' = gross + amount` (uint128). + - `net' = net + amount` for `tickLower`, `net' = net - amount` for `tickUpper` + (int128). + - repacked = `U256::from(gross') | (i128_to_u256(net') << 128)`; emit + `Slot(pool, base, repacked)`. +- **Tick slot +3** (`initialized` flag, byte 31 / bit 248 — matching the + existing `inject_v3_ticks` placement): if `gross == 0 && gross' > 0` + (tick newly initialized), set `initialized` by emitting + `SlotMasked(pool, base+3, mask = U256::from(1) << 248, value = U256::from(1) << 248)`. + (No change if it was already initialized.) +- **tickBitmap**: when a tick is newly initialized, flip its bit: + - `compressed = tick / tick_spacing` (floor toward negative infinity — match + Solidity: `tick / tickSpacing` truncates toward zero, and V3 requires + `tick % tickSpacing == 0`, so plain integer division is exact). + - `word_pos = (compressed >> 8) as i16`, `bit_pos = (compressed & 0xFF) as u8`. + - key = `v3_tick_bitmap_storage_key_with_base(word_pos, tick_bitmap_base_slot)`. + - emit `SlotMasked(pool, key, mask = U256::from(1) << bit_pos, value = U256::from(1) << bit_pos)` + (set the bit). On Burn that uninitialises the tick, clear it (value = 0). + +Global **liquidity** (slot `liquidity_slot`): the `Mint` event does **not** +carry the resulting pool liquidity, so it must be derived: read current `slot0` +→ extract `tick` (bits [160,184), sign-extended int24); if +`tickLower <= currentTick < tickUpper`, read current `liquidity` and emit +`Slot(pool, liquidity_slot, current + amount)`. If `slot0` or `liquidity` is +cold, skip+surface. (Safety net: the next `Swap` sets `liquidity` absolutely.) + +### 6.3 `Burn` → the inverse + +`Burn(owner, tickLower, tickUpper, amount, amount0, amount1)`: identical to +`Mint` with the signs inverted: +- `gross' = gross - amount` (uint128, saturating at 0 defensively). +- `net' = net - amount` for `tickLower`, `net' = net + amount` for `tickUpper`. +- If `gross > 0 && gross' == 0` (tick now uninitialised): clear the + `initialized` flag (`SlotMasked` slot+3 value 0) **and** clear the bitmap bit + (`SlotMasked` value 0). +- Global liquidity: `Slot(pool, liquidity_slot, current - amount)` if current + tick in `[tickLower, tickUpper)`. + +> A `Burn` removing all of a tick's liquidity but a same-block re-`Mint` is +> handled by the **log-by-log** apply order (§7.1): the second decode reads the +> first's applied effect through the view. + +### 6.4 Known limitation (document, do not fix) + +Event-derived tick maintenance does **not** set `feeGrowthOutside0/1X128` +(slots +1/+2), `secondsOutside`, or oracle observations — these are not +derivable from `Mint`/`Burn`/`Swap`. **Swap price/liquidity quoting is +unaffected** (the swap-amount math does not depend on `feeGrowthOutside`); fee +accounting and `collect` are not maintained. Record this as a `KNOWN_ISSUES.md` +entry with sampled `reconcile` + reorg `purge` as the backstop, in the project's +honest-freshness spirit. + +### 6.5 Cold-skip surfacing for stateful V3 updates + +When a V3 tick/liquidity update cannot be computed because a needed word is cold, +surface it so the gap is visible (never silently drop it). Reuse +`SkippedMask` for masked sub-word updates (bitmap/initialized) and, for the +absolute tick-word / liquidity writes that were skipped, push a `SkippedMask` +with `mask == U256::MAX` and `value == U256::ZERO` as the "could-not-compute" +marker, **or** (cleaner) add the skipped target to a dedicated field. **Locked +choice:** reuse `SkippedMask` with `mask == U256::MAX, value == 0` as the +cold-tick marker to avoid a fourth skip vector; document this convention on +`SkippedMask`. The pipeline's `BlockDigest.skipped` count (via +`StateDiff::skipped_len`) then includes them, and the caller re-seeds the pool. + +## 7. `EventPipeline` (generic core, `events/mod.rs`) + +```rust +pub struct EventPipeline { + registry: DecoderRegistry, + reorg: ReorgConfig, + touched: VecDeque<(u64, Vec
)>, // ring of per-block touched addrs + derived_slots: HashSet<(Address, U256)>, // event-derived slots (for reconcile sampling) +} + +#[derive(Clone, Debug)] +pub struct ReorgConfig { + /// How many recent blocks of touched-address history to retain for reorg + /// purge (the reorg horizon). Older entries are dropped. + pub depth: usize, + /// Purge scope used on reorg (default `AllStorage` — storage re-fetches but + /// the account header survives; `Account` for a full drop). + pub scope: PurgeScope, +} + +pub struct BlockDigest { + pub block: u64, + /// Merged diff of everything applied for the block (changes-only + skips). + pub applied: StateDiff, + /// Number of logs that decoded to at least one update. + pub decoded_logs: usize, + /// The (address, slot) set written this block (for freshness classification). + pub touched_slots: Vec<(Address, U256)>, +} + +pub struct ReconcileReport { + pub checked: usize, + /// Slots whose event-derived value disagreed with chain truth. Non-empty = + /// drift alarm. `verify_slots` has already injected the fresh values. + pub mismatched: Vec, +} + +impl EventPipeline { + pub fn new(registry: DecoderRegistry) -> Self; // default ReorgConfig + pub fn with_reorg_config(mut self, cfg: ReorgConfig) -> Self; + + /// Decode + apply a block's logs, **log-by-log in order**, recording touched + /// state for reorg tracking. Returns the per-block digest. + pub fn ingest_logs(&mut self, cache: &mut EvmCache, block: u64, logs: &[Log]) -> BlockDigest; + + /// Reorg to `new_head`: purge (per `ReorgConfig.scope`) every address + /// touched in a block **>** `new_head`, drop those ring entries, and return + /// the merged purge diff. The next read re-fetches from RPC. + pub fn reorg_to(&mut self, cache: &mut EvmCache, new_head: u64) -> StateDiff; + + /// Sampled reconciliation: re-read `slots` via `EvmCache::verify_slots` + /// (correct + alarm). Returns the mismatches; an empty `slots` or no fetcher + /// surfaces as appropriate (errors if no fetcher, mirroring `verify_slots`). + pub fn reconcile(&mut self, cache: &mut EvmCache, slots: &[(Address, U256)]) -> Result; + + /// All event-derived slots seen so far (sampling source for `reconcile`). + pub fn derived_slots(&self) -> impl Iterator + '_; +} +``` + +### 7.1 `ingest_logs` — decode + apply **log-by-log** + +For each log, in order: `let updates = registry.decode(log, &*cache); let diff = +cache.apply_updates(&updates); merge into the block diff`. Apply **immediately +per log** (not decode-all-then-apply-all) so a later log's decode sees the +effects of earlier logs in the same block through the `StateView` (e.g. two +overlapping `Mint`s, or a `Burn`+`Mint` pair). Record the touched addresses +(`diff.slots` + `diff.accounts` addresses + `diff.skipped*` targets' addresses) +into the ring under `block`, and the touched `(address, slot)` into +`derived_slots`. Trim the ring to `ReorgConfig.depth`. + +> `&*cache` is used as the `&dyn StateView` while `cache.apply_updates(&mut …)` +> needs `&mut` — sequence them (decode borrow ends before the apply borrow), do +> not hold both. Decode returns owned `Vec`, so there is no +> borrow overlap. + +### 7.2 `reorg_to` — purge-and-resync + +Collect every address in ring entries with `block > new_head`; dedupe; for each, +`apply_update(Purge { address, scope: cfg.scope })`; remove those ring entries +and their `derived_slots`. Merge the purge `StateDiff`s and return. (The caller +then re-ingests the canonical chain's logs for the reorged range, and/or the +next read lazily re-fetches.) + +### 7.3 `reconcile` — correct + alarm + +`let changed = cache.verify_slots(slots)?;` → `ReconcileReport { checked: +slots.len(), mismatched: changed }`. `verify_slots` already injected the fresh +chain values (correct); the returned set is the alarm. Document that a non-empty +`mismatched` means event-derived state had drifted and has now been corrected. + +### 7.4 `!Send` discipline + +All three methods take `&mut EvmCache` and are **synchronous** — they never +`.await`, so the `!Send` cache is never held across a yield. This is what makes +the core deterministically testable offline. + +### 7.5 `drive` — async convenience (thin, example-only) + +A generic `LogSource` (`async fn next_block(&mut self) -> Option<(u64, Vec, ReorgSignal)>`) +and an `async fn drive(pipeline, cache, source, hooks)` that loops: pull a block, +`reorg_to` if signalled, `ingest_logs`, invoke an optional per-block hook (where +the caller wires `FreshnessController::on_new_block` + classification). Runs on +the current task (holds the `!Send` cache across the *source* await only — the +source future is `Send`; the cache is untouched during the await). **Not** +unit-tested beyond a vec-backed `LogSource` smoke test in the example; the +synchronous core (§7.1–7.3) is the contract. + +## 8. Freshness wiring (behavior-preserving) + +No change to `FreshnessController` internals. The integration is demonstrated, +not hard-wired: `BlockDigest.touched_slots` lets a caller mark event-derived +slots `Pinned` or `ValidThrough(block + horizon)` in a `FreshnessRegistry` (so +the optimistic validator does not waste RPC re-verifying state the pipeline keeps +fresh), then call `controller.on_new_block(block)`. The example shows this +end-to-end. Document the recommended pattern (event-driven slots → `Pinned`, +reconciled periodically) on the `EventPipeline` type. + +## 9. Public re-exports (`src/lib.rs`) + +```rust +pub mod events; +pub use events::{ + BlockDigest, DecoderRegistry, EventDecoder, EventPipeline, ReconcileReport, + ReorgConfig, StateView, +}; +pub use events::erc20::Erc20TransferDecoder; +#[cfg(feature = "protocols")] +pub use events::uniswap_v3::{UniswapV3Decoder, UniswapV3Layout}; +// state_update additions: +pub use state_update::{SkippedMask /* + existing */}; +``` + +## 10. Tests (offline, no network) — the acceptance contract + +Authored **before** implementation. In-module unit tests where pure; integration +tests in new `tests/event_pipeline.rs` (reuse `tests/common` + the `mock` +harness pattern). All offline. + +**`state_update.rs` unit (pure):** +- `slot_masked_constructor_produces_variant`. +- `state_diff_merge_extends_skipped_masks_without_counting_it`. +- `slot_masked` serde JSON round-trip; `SkippedMask` round-trip. +- `has_skipped`/`skipped_len`/`is_fully_applied` include `skipped_masks`. + +**`tests/state_update.rs` (masked apply, mocked cache):** +- `slot_masked_sets_only_masked_bits` — seed slot = `0xFFFF…FF00` (overlay), + `SlotMasked{ mask: 0xFF, value: 0x42 }` → `0xFFFF…FF42`; other bits preserved; + `SlotChange{old,new}` recorded. +- `slot_masked_noop_when_masked_bits_already_equal` → empty diff. +- `slot_masked_cold_slot_is_skipped_and_surfaced` → `diff.skipped_masks == + [SkippedMask{..}]`, slot still cold, `has_skipped()`. +- `slot_masked_writes_through_both_layers` — overlay-resident slot → both layers. +- `slot_masked_full_mask_equals_absolute_on_hot_but_skips_cold`. + +**`events` unit / `tests/event_pipeline.rs` (decoders + pipeline):** + +*Decoder purity & registry:* +- `decoder_registry_dispatches_by_address` — a decoder registered for token A + fires only for A's logs; a global decoder fires for all; output concatenated + in order. +- `unknown_log_decodes_to_empty` — non-matching topic0 → `vec![]`. + +*ERC-20 (`Erc20TransferDecoder`):* +- `erc20_transfer_decodes_to_sub_and_add_deltas` — `Transfer(A,B,100)` → + `[SlotDelta::Sub(100) @ balanceSlot(A), SlotDelta::Add(100) @ balanceSlot(B)]` + at the configured mapping slot. +- `erc20_mint_skips_zero_from` / `erc20_burn_skips_zero_to` — only the non-zero + leg emitted. +- `erc20_uses_per_token_slot_override_else_default`. +- `erc20_ingest_updates_balance_and_conserves` — **end-to-end**: build the mock + cache, seed two holders' balance slots (overlay-resident, EVM-visible), + `ingest_logs` a `Transfer` log, assert both balances via `balance_of` + (real `SLOAD`) and that `from + to` is conserved; `digest.applied.slots` has 2 + entries. +- `erc20_cold_balance_transfer_is_skipped_and_surfaced` — unseeded `to` → + `digest.applied.skipped` non-empty; `has_skipped()`. + +*UniswapV3 (`protocols`, gated tests):* +- `v3_swap_sets_price_and_tick_preserving_unlocked` — seed slot0 with a known + packed word incl. `unlocked=1` (bit 240) and a nonzero observation index; + ingest a `Swap` with new sqrtPriceX96/tick; assert slot0's low-184 bits are the + new price/tick **and** bits 184+ (incl. `unlocked`) are unchanged. +- `v3_swap_sets_liquidity_absolute` — `liquidity` slot == event liquidity. +- `v3_swap_cold_slot0_is_skipped` — unseeded slot0 → `skipped_masks`. +- `v3_mint_increments_gross_and_net_signs` — seed tick slot+0 = 0; `Mint(amount)` + at `[lo,hi]` → lo word `gross=amount, net=+amount`; hi word + `gross=amount, net=-amount` (decode the packed words). +- `v3_mint_initializes_tick_and_flips_bitmap` — newly-init tick sets slot+3 + initialized bit and flips the correct bitmap word/bit (using `tick_spacing`). +- `v3_burn_decrements_and_uninitializes` — `Burn` returning gross to 0 clears the + initialized bit and the bitmap bit. +- `v3_mint_updates_global_liquidity_when_in_range` — seed slot0 tick within + `[lo,hi)` and a known `liquidity`; `Mint` → liquidity += amount; out-of-range → + liquidity unchanged. +- `v3_mint_cold_tick_word_is_skipped` — unseeded tick word → surfaced skip, no + write. +- `v3_same_block_burn_then_mint_sees_prior_apply` — two logs in one + `ingest_logs`; the `Mint` decode reads the `Burn`'s applied gross/net. + +*Pipeline (reorg + reconcile):* +- `ingest_records_touched_and_trims_ring_to_depth`. +- `reorg_to_purges_addresses_touched_after_head` — ingest blocks N, N+1, N+2 + touching distinct pools; `reorg_to(N)` purges only N+1/N+2 pools (assert their + storage re-reads cold / re-fetches; N's survives). +- `reorg_to_returns_merged_purge_diff` — `PurgeRecord`s for the purged set. +- `reconcile_reports_mismatch_and_corrects` — stub the batch fetcher so an + event-derived slot disagrees; `reconcile` returns it in `mismatched` and the + cache now holds the fresh value (assert via `cached_storage_value`). +- `reconcile_empty_when_event_state_matches_chain`. +- `reconcile_errs_without_fetcher`. + +**Existing suites stay green** — `tests/state_update.rs`, `tests/freshness.rs`, +`tests/snapshot_overlay.rs`, the `protocols` cache tests. + +## 11. Docs, example & benchmark + +- **Example** `examples/reactive_cache.rs` (offline, `examples/support`): + build a `from_backend`/mock cache; register an `Erc20TransferDecoder` and a + `UniswapV3Decoder` in a `DecoderRegistry`; `ingest_logs` a small vec of logs + (an ERC-20 `Transfer` + a V3 `Swap`) for a block; print the `BlockDigest`; + then demonstrate (a) a `reorg_to` purge, and (b) a `reconcile` drift alarm + against a stub fetcher; wire `FreshnessController::on_new_block` + + `registry.valid_through` on the touched slots. Add a README "Examples" row. +- **Benchmark** `benches/event_pipeline.rs` (offline): decode throughput + (ERC-20 `Transfer`, V3 `Swap`, V3 `Mint`); `ingest_logs` per-block apply across + log-batch sizes (1 → 1000); `reorg_to` purge cost across touched-set sizes. + Register `[[bench]]` in `Cargo.toml`; mirror `benches/state_update.rs`; add a + README "Benchmarks" row. +- **CHANGELOG** `### Added`: the event pipeline (`EventDecoder`/`StateView`/ + `DecoderRegistry`/`EventPipeline`), the ERC-20 + V3 adapters, and the + `SlotMasked` vocabulary + `StateDiff.skipped_masks` (note the additive + `StateDiff` field + new `StateUpdate` variant under the pre-1.0 break policy). +- **ROADMAP**: flip the Phase 4 row to **Done** with the landing branch, a + "Landed on …" paragraph mirroring Phases 2/3. +- **KNOWN_ISSUES**: the §6.4 V3 fee-growth/oracle limitation. +- Rustdoc on **every** public item; module `//!` docs on `events` (Pillar B.2 + framing, `!Send` discipline, the events→Phase-3-vocabulary flow), `events/erc20`, + `events/uniswap_v3`. At least one runnable doctest (a pure decoder on a + hand-built `Log`, or the `SlotMasked` masked-write shape). + +## 12. Decisions (LOCKED) + +Confirmed with the user on 2026-06-16 before the acceptance tests were authored. + +**Decision 1 — packed-slot updates → `StateUpdate::SlotMasked`.** Add the +cold-aware RMW masked-write variant (§4.1) so a pure decoder can express a +partial update to a packed word (V3 `slot0`) without clobbering the bits it does +not own (notably `unlocked`). The "absolute clobber" and "impure decoder" +alternatives were rejected. + +**Decision 2 — V3 adapter coverage → `Swap` **and** `Mint`/`Burn` (full +ticks).** The adapter maintains `slot0`/`liquidity` from `Swap` and per-tick +`liquidityGross`/`liquidityNet`/`initialized` + `tickBitmap` + global +`liquidity` from `Mint`/`Burn` (§6). Fee-growth/oracle state is out of scope +(§6.4). Mint/Burn tick maintenance is computed against the `StateView` +(Decision 1's pure-data model needs the pre-state read). + +**Decision 3 — reorg → purge-and-resync touched addresses.** Track touched +addresses per block in a depth-bounded ring; `reorg_to(n)` purges everything +touched after `n` so reads re-fetch (§7.2). `ValidThrough` is the freshness +lever. Per-slot value rollback rejected. + +**Decision 4 — reconciliation → sampled re-read, correct **and** alarm.** +Opt-in `reconcile` samples event-derived slots and re-reads via `verify_slots`: +the fresh chain value wins (auto-correct) **and** the drift is surfaced (§7.3). +Honest freshness, built in from day one. Alarm-only and defer rejected. + +## 13. Build order (commit per step, green each time) + +1. `state_update.rs` + `cache/mod.rs`: `SlotMasked` variant, `slot_masked`, + `SkippedMask`, `StateDiff.skipped_masks` (+ merge/has_skipped/skipped_len), + serde, the apply arm (reuse `write_slot_through` + cold-aware read), and the + §10 masked-apply tests. Re-exports. +2. `events/mod.rs`: `StateView` (+ `impl … for EvmCache`), `EventDecoder`, + `DecoderRegistry` + dispatch tests. +3. `events/erc20.rs`: `Erc20TransferDecoder` + decoder/ingest tests. +4. `events/uniswap_v3.rs` (`protocols`): `UniswapV3Decoder`/`UniswapV3Layout`, + `Swap`/`Mint`/`Burn` + the §10 V3 tests. +5. `EventPipeline` (`ingest_logs`/`reorg_to`/`reconcile`/`derived_slots`) + + `BlockDigest`/`ReconcileReport`/`ReorgConfig` + the pipeline tests; the async + `drive`/`LogSource` convenience. +6. Example + benchmark + README rows. +7. Docs (module `//!`, item rustdoc, doctest), CHANGELOG, ROADMAP → Done, + KNOWN_ISSUES. + +## 14. Final acceptance + +Both feature configs green (§0). All new + existing tests pass. The example runs +offline and prints a non-trivial `BlockDigest`, a reorg purge, and a reconcile +alarm. The benchmark builds and runs (`cargo bench --no-run`). The V3 adapter +preserves the `slot0` `unlocked`/observation bits under `Swap` and maintains +tick gross/net/initialized/bitmap/global-liquidity under `Mint`/`Burn`, all +cold-aware. Report: what landed per file, the public API added, the decoder/ +adapter behavior (with the §6.4 limitation called out), test coverage, and the +verification output. diff --git a/docs/phase-5-spec.md b/docs/phase-5-spec.md new file mode 100644 index 0000000..384ba14 --- /dev/null +++ b/docs/phase-5-spec.md @@ -0,0 +1,373 @@ +# Phase 5 — copy-on-write snapshots (Pillar A) + +> Status: **build contract**. Authored by the overseer before implementation; the +> red acceptance tests in [`../tests/cow_snapshot.rs`](../tests/cow_snapshot.rs) +> and the extended overlay tests pin this contract and gate the deliverable. +> Decisions below are **locked** (resolved with the user) unless marked OPEN. + +## 0. Ground rules + +1. **No behavior change on reads.** Every read against a snapshot or an overlay + built from it must return *exactly* what today's deep-clone snapshot returns — + bit-for-bit, including the audited `StorageCleared` / `NotExisting` / + two-layer-precedence semantics. This is enforced by a **differential-equivalence + test** (§8.1): the new `create_snapshot()` must be read-indistinguishable from + the retained reference `create_snapshot_deep_clone()` after *every* mutation kind. +2. **`Send + Sync` snapshot, `Send` overlay, lock-free reads.** `EvmSnapshot` + stays `Send + Sync`; `EvmOverlay` stays `Send`. Snapshot/overlay reads must not + take a lock and must not regress to a non-`O(1)` lookup (no persistent/HAMT map + on the read path — see Decision D1). The existing `test_snapshot_is_send_sync` + and `test_overlay_is_send` must keep passing unchanged. +3. **No new external dependency.** Structural sharing is achieved with `Arc` over + the per-account storage maps, not a third-party persistent-map crate (D1). +4. **Keep the deep-clone reachable** for A/B benchmarking and as the equivalence + reference (D3). It is retained as `create_snapshot_deep_clone()`. +5. **Standard bars.** `cargo fmt --check`; `cargo clippy --all-targets -- -D + warnings` (default) **and** `cargo clippy --lib --no-default-features -- -D + warnings`; `cargo test` (both feature configs); `RUSTDOCFLAGS=-D warnings cargo + doc`; `cargo bench --no-run`. + +## 1. Goal + +`EvmCache::create_snapshot()` is today an **O(total state) deep clone** +([`mod.rs`](../src/cache/mod.rs) `create_snapshot`): it copies every account and, +dominantly, **every storage slot** of both cache layers into fresh `HashMap`s on +every call. Pillar A replaces this with a **copy-on-write** scheme whose cost +tracks *changed* state, not *total* state, and whose clones are `Arc` handle +copies rather than deep copies. + +Two pillars (both in scope this phase): + +- **A.1 — structural sharing for `create_snapshot`** (§2–§4). +- **A.2 — overlay buffer / instance reuse** (§5). + +## 2. Model: memoized immutable base + fresh hot-layer fold + +### 2.1 The two layers and the cost asymmetry + +- **Layer 2 — `BlockchainDb` (the cold base).** The lazily-fetched / bulk-seeded + fork index. At a fixed block it is **append-mostly**: a fetched `(addr, slot)` + value is canonical and is not rewritten; only `set_block`/re-pin replaces it, + and the controlled bulk writers (`inject_storage_batch*`) and the write-through + funnel mutate it. This is the *large* state. +- **Layer 1 — `CacheDB` overlay (the hot delta).** revm sim commits, write-through + applies, direct inserts, freshness corrections. This is the *small, changing* + set, and it always **shadows** layer 2 on a read (overlay wins). + +The deep clone re-copies all of layer 2 every call even though it barely changes +between successive snapshots. COW memoizes layer 2 and folds only layer 1 fresh. + +### 2.2 The frozen base + +Add an internal, immutable, `Arc`-shared flatten of **layer 2 only**: + +```rust +// src/cache/snapshot.rs (or a new src/cache/cow.rs, implementer's choice) +pub(crate) struct BaseState { + /// Layer-2 account info, by address. (Layer-2 has no NotExisting concept; + /// that classification is purely a layer-1 property — see §4.) + pub(crate) accounts: HashMap, + /// Layer-2 storage, per account, **shared by `Arc`** so cloning a base is a + /// handle copy, never a per-slot copy. + pub(crate) storage: HashMap>>, + /// Bytecode by hash, derived from `accounts` at build time. + pub(crate) code_by_hash: HashMap, +} +``` + +`EvmCache` memoizes the current base and the bookkeeping needed to keep it honest: + +```rust +// fields on EvmCache +base: Option>, // None until first snapshot / after a reset +base_dirty: HashSet
, // layer-2 addrs changed since `base` was built +base_full_rebuild: bool, // set by set_block / re-pin: rebuild from scratch +base_storage_lens: HashMap, // per-acct layer-2 slot counts at last build +``` + +These fields are **not** part of any public API and **not** serialized. + +### 2.3 `refresh_base(&mut self)` — called at the top of `create_snapshot` + +Produces an up-to-date `Arc` reusing the previous one wherever layer 2 +is unchanged. It must **never mutate an `Arc` that may be shared** with a +live snapshot — on any change it builds a *new* `BaseState` that shares the `Arc`s +of unchanged accounts and rebuilds only changed ones (copy-on-write). + +Algorithm: + +1. **Full rebuild** if `base.is_none() || base_full_rebuild`: + flatten all of layer 2 into a fresh `BaseState` (one `Arc` per account); + record `base_storage_lens`; clear `base_dirty`; clear `base_full_rebuild`. +2. **Else, detect uncontrolled growth** (lazy RPC fetch / prefetch writes layer 2 + from inside `foundry-fork-db`, which we cannot hook): scan + `blockchain_db.storage().read()` and `accounts().read()`; for any address whose + slot count differs from `base_storage_lens`, or any account absent from the base, + add it to `base_dirty`. This is an `O(accounts)` length comparison — **not** an + `O(slots)` value scan. +3. **Else, if `base_dirty` is empty** → reuse the existing `Arc` + unchanged (the common hot-loop case; `create_snapshot` is then `O(1)` for the + base). +4. **Otherwise (some addresses dirty)** → build a new `BaseState`: + - clone the outer maps (an `O(accounts)` clone of `Arc` handles + plain + `AccountInfo`, **no per-slot copy**); + - for each dirty address, rebuild its `Arc>` from the current + layer-2 storage and refresh its `AccountInfo` / `code_by_hash`; + - update `base_storage_lens`; clear `base_dirty`; store as the new `Arc`. + +> Correctness rests on `base_dirty` ∪ the growth scan covering every way layer 2 +> can change such that the change is **not shadowed by layer 1**. §3 enumerates the +> sites. The equivalence test (§8.1) exercises all of them and fails loudly on any +> miss — a missed invalidation is a red test, never a silent stale read. + +### 2.4 `create_snapshot()` — the two-tier snapshot + +```rust +pub fn create_snapshot(&mut self) -> Arc { … } +``` + +Note the signature change to `&mut self` (it now refreshes/memoizes the base). +Steps: + +1. `self.refresh_base()` → `let base = Arc::clone(self.base.as_ref().unwrap());` + (`O(1)` when layer 2 is unchanged). +2. Fold **layer 1** (`self.db.cache.accounts`) into the snapshot's overlay maps and + the cleared/not-existing sets, applying the same classification as today + (§4) — `O(layer-1)`. Per-account overlay storage may be a plain + `HashMap` (the hot set is small); `Arc`-interning it is optional. +3. Construct the two-tier `EvmSnapshot { base, …overlay…, …block ctx… }`. + +Block context (`block_number`, `basefee`, `coinbase`, `prevrandao`, `gas_limit`, +`chain_id`, `timestamp`, `spec_id`) is copied as today. + +### 2.5 New `EvmSnapshot` shape + +```rust +pub struct EvmSnapshot { + pub(crate) base: Arc, + /// Layer-1 accounts that are present to the EVM (NotExisting excluded). + pub(crate) overlay_accounts: HashMap, + /// Layer-1 storage delta. A cleared account ALWAYS has an entry here (possibly + /// empty) so the cleared rule is decided without consulting the base. + pub(crate) overlay_storage: HashMap>, + /// Bytecode introduced by layer 1 (checked before `base.code_by_hash`). + pub(crate) overlay_code_by_hash: HashMap, + pub(crate) storage_cleared: HashSet
, + pub(crate) accounts_not_existing: HashSet
, + pub(crate) block_hashes: HashMap, + // …block context fields unchanged… +} +``` + +All fields stay `pub(crate)` (no public field break). In-crate `#[cfg(test)]` +constructors of `EvmSnapshot` (in `overlay.rs`) must be updated to the new shape. + +## 3. Base-invalidation sites (the correctness checklist) + +Every site below must keep the memoized base honest. Implement as a private +helper (e.g. `self.mark_base_dirty(addr)` / `self.invalidate_base()`). + +| Site | Layer touched | Action | +| --- | --- | --- | +| `write_slot_through(addr, …)` | layer 2 always; layer 1 if present | `mark_base_dirty(addr)` (over-invalidation when also in layer 1 is **safe** — it just re-folds that one account; D2 keeps it simple over clever) | +| `inject_storage_batch` / `inject_storage_batch_fresh` | layer 2 only | `mark_base_dirty(addr)` for each touched addr | +| account info / storage seeded into layer 2 (construction, `inject_v2/v3_*` paths that hit layer 2) | layer 2 | `mark_base_dirty(addr)` | +| `purge_*` removing layer-2 entries | layer 2 | `mark_base_dirty(addr)` (or `invalidate_base()` if simpler for account-level purge) | +| `set_block` / `repin_to_block` | replaces layer 2 | `base_full_rebuild = true` | +| revm commit (`call_raw(commit=true)`, session commit) | **layer 1 only** | **nothing** — folded fresh; never makes the base stale | +| direct `db_mut()` inserts (`insert_account_info`/`insert_account_storage`) | **layer 1** | **nothing** — folded fresh | +| uncontrolled lazy RPC fetch / prefetch | layer 2 | caught by the `O(accounts)` growth scan in `refresh_base` step 2 | + +> The litmus test for "needs invalidation": *can this change a layer-2 value that a +> snapshot read would surface (i.e. that layer 1 does not shadow)?* If yes → dirty. +> Layer-1-only writes are always shadowed → never dirty the base. + +## 4. Read semantics (must equal today's flatten, bit-for-bit) + +`EvmSnapshot` exposes the lookups the overlay needs; `EvmOverlay` calls these +instead of indexing fields directly. + +```rust +impl EvmSnapshot { + /// Account info as the EVM sees it. None for NotExisting (do NOT consult base). + pub(crate) fn account_info(&self, a: Address) -> Option<&AccountInfo> { + if self.accounts_not_existing.contains(&a) { return None; } + self.overlay_accounts.get(&a).or_else(|| self.base.accounts.get(&a)) + } + + /// Storage value, mirroring cached_storage_value / today's flatten. + pub fn storage_value(&self, a: Address, s: U256) -> Option { + if let Some(m) = self.overlay_storage.get(&a) { + if let Some(v) = m.get(&s) { return Some(*v); } + if self.storage_cleared.contains(&a) { return Some(U256::ZERO); } // cleared: base dropped + // not cleared: fall through to base + } + if let Some(v) = self.base.storage.get(&a).and_then(|m| m.get(&s)) { + return Some(v); + } + None + } + + pub(crate) fn code(&self, h: B256) -> Option<&Bytecode> { + self.overlay_code_by_hash.get(&h).or_else(|| self.base.code_by_hash.get(&h)) + } +} +``` + +Invariants the equivalence test pins: +- A **cleared** (`StorageCleared`/`NotExisting`) layer-1 account: snapshot holds + only its overlay slots; an absent slot reads `Some(ZERO)`; base slots are never + surfaced (this is why cleared accounts always get an `overlay_storage` entry). +- A **NotExisting** account: `account_info` → `None`, `storage_value` → `Some(ZERO)` + for any slot; excluded from `overlay_accounts`/`overlay_code_by_hash`. +- A **non-cleared** layer-1 account: overlay slot wins, else base slot, else `None`. +- An address only in layer 2: base slot, else `None`. + +`EvmOverlay::{basic, storage, code_by_hash}` are rewritten to: dirty layer → +`snapshot.account_info/storage_value/code` → (the `NotExisting`/`cleared` +short-circuits already live inside those) → `ext_db` fallback (unchanged) → +default. Behavior must match the current overlay exactly (the existing +`overlay.rs` unit tests must keep passing). + +## 5. Overlay buffer / instance reuse (Pillar A.2) + +### 5.1 `EvmOverlay::reset(&mut self)` — recycle one overlay across many sims + +```rust +/// Clear the per-simulation dirty layer so this overlay can be reused for the +/// next simulation against the same snapshot, without reallocating. +pub fn reset(&mut self) { + self.dirty_accounts.clear(); + self.dirty_storage.clear(); + // keep: snapshot Arc, ext_db, the reusable buffer (§5.2) +} +``` + +A worker doing K sims calls `EvmOverlay::new` once and `reset()` between sims +instead of allocating a fresh overlay (+ dirty maps + `Arc` clone) each time. Must +be exactly equivalent to a fresh overlay: a reset overlay reads the pristine +snapshot base again (regression test §8.2). + +### 5.2 Reusable shared-memory buffer (keep `EvmOverlay: Send`) + +Today each `build_evm` / `build_evm_with_inspector` allocates a fresh +`Rc>>` of 64 KB. Reuse it across calls **without** making the +overlay `!Send`: + +- Store the buffer on the overlay as a **plain `Vec`** (`Send`): + `reusable_buffer: Vec` (pre-allocated to `OVERLAY_SHARED_MEMORY_CAPACITY` in + `new`/`reset` keeps it). +- In the **call methods** (`call_raw`, `simulate_with_transfer_tracking`, + `call_raw_with_access_list_with`) that own the full build→transact→revert cycle: + `let buf = std::mem::take(&mut self.reusable_buffer);` **before** the + `with_db(&mut *self)` borrow, move it into a method-local + `Rc::new(RefCell::new(buf))`, build the EVM with that local context, run, then + after the EVM is dropped reclaim `self.reusable_buffer = Rc::try_unwrap(rc).into_inner(); self.reusable_buffer.clear();` + The `Rc` never lives on the overlay → the overlay stays `Send`. +- Refactor the shared body into e.g. `build_evm_with_local(&mut self, local: LocalContext)`; + the **public `build_evm`** keeps allocating a fresh buffer (it hands out the EVM + and cannot reclaim) — documented. +- A panic between take and reclaim only loses the buffer (re-allocated next call); + no correctness impact. + +`test_overlay_is_send` must still compile/pass. + +## 6. Public API surface + +- `EvmCache::create_snapshot(&mut self) -> Arc` — **signature change** + `&self` → `&mut self` (memoizes the base). Update all call sites (freshness + controller, tests, examples, benches). Record in CHANGELOG `### Changed`. +- `EvmCache::create_snapshot_deep_clone(&self) -> Arc` — **new**, + `#[doc(hidden)] pub`. The retained reference: today's flatten producing a + two-tier snapshot with `base` = the fully-merged flatten and empty overlay maps + (plus the cleared/not-existing sets in place). Used by the equivalence test and + the A/B bench. Stays `&self`. +- `EvmOverlay::reset(&mut self)` — **new** public method. +- `EvmSnapshot::storage_value` — retained (reimplemented over two tiers); used by + the freshness validator. `account_info`/`code` are `pub(crate)`. +- No other public signatures change. + +## 7. Benchmarks (`benches/simulation.rs`) + +The current `populated_cache` seeds everything via `db_mut()` into **layer 1**, +which is *not* how a fork cache holds its cold index. Update/extend: + +1. **Realistic cold index in layer 2.** Add a `populated_cache_layer2` that bulk- + seeds the index via `inject_storage_batch` (the cold-load path) so the cold + state lives in the base. The `create_snapshot` group runs the COW path on it. +2. **A/B group.** For each size, bench both `create_snapshot` (COW) and + `create_snapshot_deep_clone` (legacy) so the win is explicit in one report. +3. **Hot-loop re-snapshot.** New bench: build the cold base, take one snapshot + (warms the base), apply a *small* layer-1 mutation (a handful of slots via + `apply_updates`), then measure `create_snapshot` — this is the memoization win + (should be ≈ flat across cold-index size, vs. the deep clone's slope). +4. **`overlay_fanout`.** Add a `reset()`-recycled variant alongside the + `EvmOverlay::new`-per-iter variant to show the A.2 win. + +Keep all benches offline (mocked provider). Document expected shapes in the module +header (COW `create_snapshot` flat vs. deep-clone sloped; re-snapshot ≈ O(changed)). + +## 8. Tests (red contract — written before implementation) + +### 8.1 `tests/cow_snapshot.rs` — differential equivalence (the gate) + +A helper `assert_equivalent(cache)` builds `cow = cache.create_snapshot()` and +`deep = cache.create_snapshot_deep_clone()` and asserts they are +**read-indistinguishable**, comparing via reads (internal reprs differ by design): +- identical account set (union of probed addresses); `basic`/`account_info` equal + for each (including `None` for NotExisting); +- `storage_value(a, s)` equal for every probed `(a, s)` — including **absent** + slots (expect equal `None`/`Some(ZERO)`), cleared accounts, and not-existing + accounts; +- identical `code` for each probed code hash; identical block context; +- overlays built from each (`EvmOverlay::new(snap, None)`) return identical + `balanceOf` / `call_raw` outputs for a `MockERC20`. + +Drive a single cache through a sequence and assert equivalence **after each step**: +1. empty cache; 2. after `insert_account_info`; 3. after layer-1 storage insert; +4. after `apply_updates` `Slot` write-through (addr in layer 1 → shadowed); +5. after `apply_updates` `Slot` write-through to an addr **absent** from layer 1 + (layer-2-only — the §3 footgun); 6. after `apply_updates` `BalanceDelta`; +7. after a committing `call_raw` (revm commit → layer 1); 8. after + `inject_storage_batch` (layer-2-only, incl. **overwriting** an existing slot at + unchanged length); 9. after a simulated lazy fetch (insert directly into + `blockchain_db` to mimic backend growth — both a new account and a **new slot on + an existing account**); 10. after a `purge_*`; 11. after `set_block`. +Also: take a snapshot, then mutate the cache, and assert the **earlier** snapshot is +unchanged (memoized base is COW, not aliased). + +### 8.2 Overlay reuse (extend `tests/snapshot_overlay.rs` or new module) + +- `reset()` clears dirty state: commit a transfer into an overlay, `reset()`, then + reads observe the pristine snapshot again. +- A reset-recycled overlay across two sims yields identical results to two fresh + overlays. +- `EvmOverlay` stays `Send`; `EvmSnapshot` stays `Send + Sync` (compile asserts). +- Buffer reuse does not change call results (a second `call_raw` on the same + overlay returns the same value as the first). + +The existing `tests/snapshot_overlay.rs` cases (immutability, isolation, cleared, +not-existing) must keep passing **unchanged** — they are part of the contract. + +## 9. Locked decisions + +- **D1 — `Arc`-shared maps, not persistent HAMT.** Reads stay `O(1)` with no + per-`SLOAD` regression; no external dependency. (Rejected: `imbl`/`rpds`.) +- **D2 — base memoized as immutable; over-invalidation is acceptable, silent + staleness is not.** `write_slot_through` marks the address dirty unconditionally + (simpler than reasoning per-call about layer-1 shadowing); the equivalence test + is the hard backstop. +- **D3 — keep the deep clone** as `create_snapshot_deep_clone` for A/B + as the + equivalence reference. +- **D4 — overlay reuse: buffer reuse *and* `reset()` recycle** (both in scope). +- **D5 — `create_snapshot` becomes `&mut self`** (the memoization cost). The + freshness controller and all callers are updated. + +## 10. Acceptance + +All §0.5 bars green in both feature configs; the §8 tests pass; the existing +snapshot/overlay/freshness tests pass unchanged; `benches/simulation.rs` shows the +COW `create_snapshot` and the `reset()` fan-out beating their legacy/`new` +baselines, with the numbers reported. CHANGELOG / ROADMAP (Phase 5 → Done) / +KNOWN_ISSUES updated. Lands on `phase-5-cow-snapshots`, stacked on +`phase-4-event-pipeline`. diff --git a/examples/prefetch_registry.rs b/examples/prefetch_registry.rs index 3c57366..690092b 100644 --- a/examples/prefetch_registry.rs +++ b/examples/prefetch_registry.rs @@ -19,7 +19,7 @@ use alloy_primitives::{Address, U256}; use evm_fork_cache::StorageAccessList; use evm_fork_cache::prefetch_registry::PrefetchRegistry; -fn main() { +fn main() -> anyhow::Result<()> { let pool = Address::repeat_byte(0xAA); let vault_a = Address::repeat_byte(0x01); let vault_b = Address::repeat_byte(0x02); @@ -45,7 +45,7 @@ fn main() { // Persist to disk (bincode) and reload — the shape survives the round trip. let path = std::env::temp_dir().join("evm_fork_cache_example_prefetch.bin"); - registry.save(&path); + registry.save(&path)?; let loaded = PrefetchRegistry::load(&path); let aggregated = loaded.phase_slots("pool_refresh"); @@ -61,4 +61,5 @@ fn main() { ); let _ = std::fs::remove_file(&path); + Ok(()) } diff --git a/examples/reactive_cache.rs b/examples/reactive_cache.rs new file mode 100644 index 0000000..7901f83 --- /dev/null +++ b/examples/reactive_cache.rs @@ -0,0 +1,243 @@ +//! Reactive cache updates from the event stream (Pillar B.2). +//! +//! Decodes on-chain logs into the Phase 3 [`StateUpdate`] vocabulary and applies +//! them to a fork cache — keeping hot state fresh **without** an RPC round-trip +//! per change. It wires up the three pieces Phase 4 adds: +//! +//! 1. A [`DecoderRegistry`] with an [`Erc20TransferDecoder`] (balances) and a +//! [`UniswapV3Decoder`] (a pool's `slot0` price/tick + `liquidity`). +//! 2. An [`EventPipeline`] whose `ingest_logs` decodes + applies a block's logs +//! (log-by-log), surfacing a [`BlockDigest`]. +//! 3. The reactive maintenance: the freshness wiring (pin event-derived slots so +//! the optimistic validator does not re-verify them, then advance the block +//! clock), a sampled **reconcile** drift alarm against a stub fetcher, and a +//! **reorg** purge-and-resync. +//! +//! Runs fully offline against a mocked provider and in-memory logs — no network. +//! Requires the `protocols` feature (the UniswapV3 adapter), which is on by +//! default. +//! +//! Run with: +//! +//! ```sh +//! cargo run --example reactive_cache +//! ``` + +#[cfg(feature = "protocols")] +#[tokio::main(flavor = "multi_thread")] +async fn main() -> anyhow::Result<()> { + imp::run().await +} + +#[cfg(not(feature = "protocols"))] +fn main() { + eprintln!( + "the `reactive_cache` example requires the `protocols` feature (the \ + UniswapV3 adapter). Run it with default features: \ + `cargo run --example reactive_cache`." + ); +} + +#[cfg(feature = "protocols")] +#[path = "support/mock.rs"] +mod mock; + +#[cfg(feature = "protocols")] +mod imp { + use std::collections::HashMap; + use std::sync::Arc; + + use alloy_eips::BlockId; + use alloy_primitives::aliases::{I24, U160}; + use alloy_primitives::{Address, Bytes, I256, Log, U256, keccak256}; + use alloy_sol_types::{SolEvent, SolValue, sol}; + use anyhow::Result; + use evm_fork_cache::cache::{StorageBatchFetchFn, V3_LIQUIDITY_SLOT, V3_SLOT0_SLOT}; + use evm_fork_cache::events::{DecoderRegistry, EventPipeline}; + use evm_fork_cache::freshness::{ + AlwaysVerify, FreshnessController, FreshnessRegistry, Validity, + }; + use evm_fork_cache::{Erc20TransferDecoder, UniswapV3Decoder, UniswapV3Layout}; + + use super::mock; + + sol! { + event Swap(address indexed sender, address indexed recipient, int256 amount0, int256 amount1, uint160 sqrtPriceX96, uint128 liquidity, int24 tick); + } + + /// Hashed `balanceOf[owner]` slot for the MockERC20 fixture (mapping at slot 3). + fn balance_slot(owner: Address) -> U256 { + let key = keccak256((owner, U256::from(mock::MOCK_ERC20_BALANCE_SLOT)).abi_encode()); + U256::from_be_bytes(key.0) + } + + /// Build an ERC-20 `Transfer(from, to, value)` log. + fn transfer_log(token: Address, from: Address, to: Address, value: U256) -> Log { + let sig = keccak256(b"Transfer(address,address,uint256)"); + Log::new_unchecked( + token, + vec![sig, from.into_word(), to.into_word()], + Bytes::copy_from_slice(&value.to_be_bytes::<32>()), + ) + } + + /// Build a UniswapV3 `Swap` log carrying the post-swap price/liquidity/tick. + fn swap_log(pool: Address, sqrt_price: u128, liquidity: u128, tick: i32) -> Log { + let ev = Swap { + sender: Address::repeat_byte(0x5e), + recipient: Address::repeat_byte(0x5f), + amount0: I256::try_from(-1_000i64).unwrap(), + amount1: I256::try_from(1_000i64).unwrap(), + sqrtPriceX96: U160::from(sqrt_price), + liquidity, + tick: I24::try_from(tick).unwrap(), + }; + Log { + address: pool, + data: ev.encode_log_data(), + } + } + + /// Pack a slot0 word: sqrtPriceX96 [0,160), tick [160,184), `unlocked` at bit 240. + fn pack_slot0(sqrt_price: u128, tick: i32) -> U256 { + let tick24 = U256::from((tick as u32) & 0x00FF_FFFF); + let unlocked = U256::from(1) << 240; + U256::from(sqrt_price) | (tick24 << 160) | unlocked + } + + pub async fn run() -> Result<()> { + let mut cache = mock::offline_cache().await?; + + let token = Address::repeat_byte(0x11); + let pool = Address::repeat_byte(0x99); + let alice = Address::repeat_byte(0x22); + let bob = Address::repeat_byte(0x33); + mock::install_default_account(&mut cache, Address::ZERO); + mock::install_default_account(&mut cache, alice); + mock::install_default_account(&mut cache, bob); + mock::install_mock_erc20(&mut cache, token); + mock::install_mock_erc20(&mut cache, pool); // reuse as a storage-cleared pool + + // Seed the holders' balances (EVM-visible) and the pool's slot0 + liquidity. + cache + .db_mut() + .insert_account_storage(token, balance_slot(alice), U256::from(1_000))?; + cache + .db_mut() + .insert_account_storage(token, balance_slot(bob), U256::from(0))?; + cache + .db_mut() + .insert_account_storage(pool, V3_SLOT0_SLOT, pack_slot0(1_000_000, 100))?; + cache + .db_mut() + .insert_account_storage(pool, V3_LIQUIDITY_SLOT, U256::from(5_000))?; + + // 1. Build the decoder registry: ERC-20 balances + the V3 pool. + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(Erc20TransferDecoder::new(U256::from( + mock::MOCK_ERC20_BALANCE_SLOT, + )))); + registry.register(Arc::new( + UniswapV3Decoder::new().with_pool(pool, UniswapV3Layout::uniswap(60)), + )); + let mut pipeline = EventPipeline::new(registry); + + // The freshness side: a controller whose registry we pin event-derived + // slots into so the optimistic validator never re-verifies them by RPC. + let mut controller = FreshnessController::new(FreshnessRegistry::new(), AlwaysVerify); + + // 2. Ingest block 100: Alice sends Bob 250 tokens, and the pool swaps (new + // price/tick + liquidity). Decoded + applied log-by-log. + let block = 100u64; + let digest = pipeline.ingest_logs( + &mut cache, + block, + &[ + transfer_log(token, alice, bob, U256::from(250)), + swap_log(pool, 2_000_000, 7_500, 120), + ], + ); + + println!("=== ingested block {} ===", digest.block); + println!( + " decoded {} log(s) -> {} slot change(s), {} skipped", + digest.decoded_logs, + digest.applied.slots.len(), + digest.applied.skipped_len(), + ); + println!( + " alice balance: {} bob balance: {}", + mock::balance_of(&mut cache, token, alice)?, + mock::balance_of(&mut cache, token, bob)?, + ); + println!( + " pool liquidity slot: {}", + cache.cached_storage_value(pool, V3_LIQUIDITY_SLOT).unwrap() + ); + // slot0: the new price/tick landed, and the `unlocked` bit (240) is + // preserved by the masked write — a clobbered `unlocked` would make a + // quote revert LOK. + let slot0 = cache.cached_storage_value(pool, V3_SLOT0_SLOT).unwrap(); + println!( + " pool slot0 sqrtPriceX96 (low 160b): {}", + slot0 & ((U256::from(1) << 160) - U256::from(1)) + ); + println!( + " pool slot0 unlocked bit preserved: {}", + (slot0 >> 240) & U256::from(1) == U256::from(1) + ); + + // 3a. Freshness wiring: pin the touched slots (kept fresh out-of-band by + // the pipeline) and advance the block clock. + for (addr, slot) in &digest.touched_slots { + controller + .registry_mut() + .set_slot(*addr, *slot, Validity::Pinned); + } + controller.on_new_block(block); + println!( + "\npinned {} event-derived slot(s) into the freshness registry", + digest.touched_slots.len() + ); + + // 3b. Sampled reconcile against chain truth. Stub the fetcher so the + // pool's liquidity reads 7_600 on-chain (a small drift from our + // event-derived 7_500): reconcile corrects the cache AND alarms. + let fresh: HashMap<(Address, U256), U256> = + HashMap::from([((pool, V3_LIQUIDITY_SLOT), U256::from(7_600))]); + let fetcher: StorageBatchFetchFn = Arc::new( + move |requests: Vec<(Address, U256)>, _block: Option| { + requests + .into_iter() + .map(|(a, s)| (a, s, Ok(fresh.get(&(a, s)).copied().unwrap_or(U256::ZERO)))) + .collect() + }, + ); + cache.set_storage_batch_fetcher(fetcher); + + let report = pipeline.reconcile(&mut cache, &[(pool, V3_LIQUIDITY_SLOT)])?; + println!("\n=== reconcile (sampled {} slot) ===", report.checked); + if report.mismatched.is_empty() { + println!(" no drift — event-derived state matches chain"); + } else { + for c in &report.mismatched { + println!( + " DRIFT: {} slot {} : {} -> {} (corrected)", + c.address, c.slot, c.old, c.new + ); + } + } + + // 4. A reorg to block 99 purges everything block 100 touched, so the next + // read re-fetches from RPC (the caller re-ingests the canonical logs). + let purge = pipeline.reorg_to(&mut cache, 99); + println!("\n=== reorg to block 99 ==="); + println!( + " purged {} address(es); pool liquidity now re-reads cold/zero: {:?}", + purge.purged.len(), + cache.cached_storage_value(pool, V3_LIQUIDITY_SLOT), + ); + + Ok(()) + } +} diff --git a/examples/state_update_apply.rs b/examples/state_update_apply.rs new file mode 100644 index 0000000..fa06944 --- /dev/null +++ b/examples/state_update_apply.rs @@ -0,0 +1,183 @@ +//! Apply a batch of targeted [`StateUpdate`]s and inspect the returned +//! [`StateDiff`] (Phase 3, Pillar B.1) — fully offline. +//! +//! Builds a mocked-provider cache, seeds a little state, then applies a mixed +//! batch — a `Slot` write, an `Account` balance patch, and a `Purge { Slots }` — +//! through the single [`EvmCache::apply_update`] / `apply_updates` primitive, and +//! prints what each apply actually changed (slot deltas, account deltas, purge +//! records). It then shows a *relative* `SlotDelta` balance bump on a hot slot and +//! a cold-slot `SlotDelta` surfaced (not applied) via `diff.skipped`. No network +//! is touched. +//! +//! Run with: +//! +//! ```sh +//! cargo run --example state_update_apply +//! ``` + +use alloy_primitives::{Address, U256}; +use anyhow::Result; +use evm_fork_cache::{PurgeScope, SlotDelta, StateUpdate}; + +#[path = "support/mock.rs"] +mod mock; + +use mock::{install_default_account, install_mock_erc20, offline_cache}; + +#[tokio::main(flavor = "multi_thread")] +async fn main() -> Result<()> { + let pool = Address::repeat_byte(0x11); + let holder = Address::repeat_byte(0x22); + + let mut cache = offline_cache().await?; + // A token-like account with overlay storage (so the slot write heals both + // layers) plus an EOA-style account to patch a balance onto. + install_mock_erc20(&mut cache, pool); + install_default_account(&mut cache, holder); + + // Seed some backend storage on the pool so the purge has something to remove + // and the slot write has a recorded `old` value. + cache.inject_storage_batch(&[ + (pool, U256::from(0), U256::from(100)), // e.g. a reserve slot + (pool, U256::from(7), U256::from(1)), // a tick/aux slot we'll purge + (pool, U256::from(8), U256::from(2)), // another slot we'll purge + ]); + + println!("Applying a mixed batch of state updates...\n"); + + let diff = cache.apply_updates(&[ + // 1. Authoritative slot write (e.g. an event-derived reserve update). + StateUpdate::slot(pool, U256::from(0), U256::from(250)), + // 2. Partial account patch: set only the balance, leave nonce/code. + StateUpdate::balance(holder, U256::from(1_000_000)), + // 3. Drop two stale storage slots so the next read re-fetches them. + StateUpdate::purge(pool, PurgeScope::Slots(vec![U256::from(7), U256::from(8)])), + ]); + + println!("StateDiff: {} changed entr(ies)\n", diff.len()); + + println!("Slot changes ({}):", diff.slots.len()); + for change in &diff.slots { + println!( + " {} slot {} : {} -> {}", + change.address, change.slot, change.old, change.new + ); + } + + println!("\nAccount changes ({}):", diff.accounts.len()); + for change in &diff.accounts { + println!(" {}", change.address); + if let Some((old, new)) = change.balance { + println!(" balance: {old} -> {new}"); + } + if let Some((old, new)) = change.nonce { + println!(" nonce: {old} -> {new}"); + } + if let Some((old, new)) = change.code_hash { + println!(" code: {old} -> {new}"); + } + } + + println!("\nPurge records ({}):", diff.purged.len()); + for rec in &diff.purged { + println!( + " {} scope={:?} slots_removed={} account_removed={}", + rec.address, rec.scope, rec.slots_removed, rec.account_removed + ); + } + + // Re-applying the same slot value is a no-op — idempotence is observable. + let again = cache.apply_update(&StateUpdate::slot(pool, U256::from(0), U256::from(250))); + println!( + "\nRe-applying the same slot value -> empty diff: {}", + again.is_empty() + ); + + // --- Relative (read-modify-write) updates ------------------------------- + // + // A caller indexing ERC-20 `Transfer` logs only learns the *delta* + // (`amount`), not the resulting balance. `SlotDelta` reads the current value + // and applies a saturating mutation, write-through. + println!("\n--- Relative SlotDelta updates ---"); + + // A hot (seeded) balance slot: +750 relative to the current value. + let hot_slot = U256::from(0); // we set this to 250 above + let rel = cache.apply_update(&StateUpdate::slot_delta( + pool, + hot_slot, + SlotDelta::Add(U256::from(750)), + )); + for change in &rel.slots { + println!( + " hot : slot {} {} -> {} (Add 750)", + change.slot, change.old, change.new + ); + } + + // A cold slot the cache never fetched: applying `0 ± amount` would corrupt an + // unknown value, so the delta is NOT applied — it is surfaced for the caller + // to fetch+seed the true value and retry. + let cold_slot = U256::from(4_242); + let cold = cache.apply_update(&StateUpdate::slot_delta( + pool, + cold_slot, + SlotDelta::Add(U256::from(100)), + )); + println!( + " cold : applied {} change(s), skipped {} (left for the caller to seed)", + cold.slots.len(), + cold.skipped.len() + ); + for skip in &cold.skipped { + println!( + " skipped: {} slot {} delta={:?}", + skip.address, skip.slot, skip.delta + ); + } + + // --- Relative native-balance updates (BalanceDelta) --------------------- + // + // The same cold-aware read-modify-write rule applies to an account's native + // ETH balance: a `BalanceDelta` on a *present* account bumps its balance; on a + // *cold* account (absent from both layers) it is dropped and surfaced. + println!("\n--- Relative BalanceDelta updates ---"); + + // `holder` was installed above (present), so a +500_000 delta applies. + let bal = cache.apply_update(&StateUpdate::balance_delta( + holder, + SlotDelta::Add(U256::from(500_000)), + )); + for change in &bal.accounts { + if let Some((old, new)) = change.balance { + println!( + " hot : {} balance {} -> {} (Add 500_000)", + change.address, old, new + ); + } + } + + // A cold account the cache never loaded: the balance is unknown, so the delta + // is NOT applied (no default account is materialized to mask the real one) — + // it is surfaced in `diff.skipped_balances`. + let unknown = Address::repeat_byte(0x99); + let cold_bal = cache.apply_update(&StateUpdate::balance_delta( + unknown, + SlotDelta::Add(U256::from(1_000)), + )); + // A cold-skipped relative update produces no change, so it is invisible to the + // changes-only `is_empty()`/`len()` check — callers MUST inspect `has_skipped()`. + println!( + " cold : has_skipped={} skipped_len={} (changes-only len={})", + cold_bal.has_skipped(), + cold_bal.skipped_len(), + cold_bal.len(), + ); + for skip in &cold_bal.skipped_balances { + println!( + " skipped balance: {} delta={:?}", + skip.address, skip.delta + ); + } + + Ok(()) +} diff --git a/fixtures/EventGroundTruthPool.sol b/fixtures/EventGroundTruthPool.sol new file mode 100644 index 0000000..510e9e8 --- /dev/null +++ b/fixtures/EventGroundTruthPool.sol @@ -0,0 +1,126 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 +pragma solidity ^0.8.20; + +/// @title TestV3Pool +/// @notice A faithful **stand-in** for a UniswapV3 pool used by the event → +/// state differential test (`tests/event_ground_truth.rs`). It is NOT +/// verbatim Uniswap bytecode; instead it reproduces the two things our +/// event decoder actually depends on, and lets the Solidity compiler — +/// not the test author — generate them: +/// 1. The real UniswapV3 `slot0` **storage packing**: `slot0` is a +/// struct with the identical field order/widths as +/// `IUniswapV3PoolState.slot0`, so the compiler packs +/// `sqrtPriceX96` (bits [0,160)), `tick` (int24, [160,184)), +/// `observationIndex`/cardinality/`feeProtocol`/`unlocked` ([184,256)) +/// into one word at storage slot 0 exactly as the real pool does. +/// A swap assigns only `.sqrtPriceX96`/`.tick`, so the compiler emits +/// the masked update that preserves the observation/`unlocked` bits — +/// the exact behavior our `StateUpdate::SlotMasked` must reproduce. +/// 2. The canonical `Swap(...)` event signature, emitted with the same +/// `sqrtPriceX96`/`liquidity`/`tick` values written to storage. +/// +/// @dev Storage layout (mirrors UniswapV3Pool so the slots match +/// `UniswapV3Layout::uniswap`): +/// slot 0: slot0 (packed) +/// slot 1: feeGrowthGlobal0X128 (unused, for layout parity) +/// slot 2: feeGrowthGlobal1X128 (unused) +/// slot 3: protocolFees (unused) +/// slot 4: liquidity (uint128) +/// `token0`/`token1` are immutable (baked into code, not stored), so they do +/// not perturb the slot numbering. +/// +/// The swap *outcome* (amounts, new price/tick/liquidity) is supplied by the +/// caller so the test is deterministic; the pool still performs real ERC-20 +/// transfers (emitting canonical `Transfer` logs from the token contracts) and a +/// real compiler-packed `slot0` update. The price math itself is irrelevant to +/// what the event processor reconstructs — it reads `sqrtPriceX96`/`tick` from the +/// emitted event, never from the pool's internals. +interface IERC20 { + function transfer(address to, uint256 amount) external returns (bool); + function transferFrom(address from, address to, uint256 amount) external returns (bool); +} + +contract TestV3Pool { + /// Identical field order/widths to UniswapV3Pool.Slot0 (one packed word). + struct Slot0 { + uint160 sqrtPriceX96; + int24 tick; + uint16 observationIndex; + uint16 observationCardinality; + uint16 observationCardinalityNext; + uint8 feeProtocol; + bool unlocked; + } + + event Swap( + address indexed sender, + address indexed recipient, + int256 amount0, + int256 amount1, + uint160 sqrtPriceX96, + uint128 liquidity, + int24 tick + ); + + Slot0 public slot0; // slot 0 + uint256 private feeGrowthGlobal0X128; // slot 1 + uint256 private feeGrowthGlobal1X128; // slot 2 + uint256 private protocolFees; // slot 3 + uint128 public liquidity; // slot 4 + + address public immutable token0; + address public immutable token1; + + constructor(address _token0, address _token1) { + token0 = _token0; + token1 = _token1; + } + + /// Set the initial packed `slot0` (with `unlocked = true` and a non-zero + /// observation index, so the differential test can prove those bits survive + /// a swap) and the initial `liquidity`. + function initialize(uint160 sqrtPriceX96, int24 tick, uint16 observationIndex, uint128 _liquidity) + external + { + slot0 = Slot0({ + sqrtPriceX96: sqrtPriceX96, + tick: tick, + observationIndex: observationIndex, + observationCardinality: 1, + observationCardinalityNext: 1, + feeProtocol: 0, + unlocked: true + }); + liquidity = _liquidity; + } + + /// Execute a swap with a caller-specified outcome: pull `amountIn` of the + /// input token (real `transferFrom` → `Transfer` log), send `amountOut` of the + /// output token (real `transfer` → `Transfer` log), update the packed `slot0` + /// price/tick (compiler-masked, preserving the observation/`unlocked` bits) + /// and `liquidity`, then emit the canonical `Swap` event with those values. + function swap( + bool zeroForOne, + uint256 amountIn, + uint256 amountOut, + uint160 newSqrtPriceX96, + int24 newTick, + uint128 newLiquidity + ) external { + address tokenIn = zeroForOne ? token0 : token1; + address tokenOut = zeroForOne ? token1 : token0; + IERC20(tokenIn).transferFrom(msg.sender, address(this), amountIn); + IERC20(tokenOut).transfer(msg.sender, amountOut); + + // Real Uniswap assigns the struct fields individually; the compiler emits + // the masked SSTORE that preserves observation/unlocked. This is exactly + // what `StateUpdate::SlotMasked` must reproduce off the event. + slot0.sqrtPriceX96 = newSqrtPriceX96; + slot0.tick = newTick; + liquidity = newLiquidity; + + int256 amount0 = zeroForOne ? int256(amountIn) : -int256(amountOut); + int256 amount1 = zeroForOne ? -int256(amountOut) : int256(amountIn); + emit Swap(msg.sender, msg.sender, amount0, amount1, newSqrtPriceX96, newLiquidity, newTick); + } +} diff --git a/fixtures/README.md b/fixtures/README.md index ce6f067..b52807f 100644 --- a/fixtures/README.md +++ b/fixtures/README.md @@ -43,3 +43,28 @@ jq -r '.deployedBytecode.object' out/MockERC20.sol/MockERC20.json \ jq -r '.bytecode.object' out/MockERC20.sol/MockERC20.json \ | sed 's/^0x//' > fixtures/mock_erc20_creation.hex ``` + +## `TestV3Pool` + +A faithful UniswapV3-pool **stand-in** (see +[`EventGroundTruthPool.sol`](EventGroundTruthPool.sol)) used by the Phase 4 +differential ground-truth test +([`../tests/event_ground_truth.rs`](../tests/event_ground_truth.rs)). It is *not* +verbatim Uniswap bytecode; it reproduces the two things the event decoder depends +on and lets the compiler generate them: the real `slot0` **struct packing** +(`sqrtPriceX96`/`tick`/observation/`unlocked`, matching `UniswapV3Pool.Slot0`) at +storage slot 0, and the canonical `Swap(...)` event. A `swap` performs real ERC-20 +transfers (canonical `Transfer` logs) and a compiler-masked `slot0` update, so the +test can replay only the emitted logs into a twin cache and assert the +event-derived state matches the ground-truth EVM execution bit-for-bit. + +- `test_v3_pool_creation.hex` — creation bytecode, for `deploy_contract`. The + constructor takes `(address token0, address token1)`; storage mirrors Uniswap + (slot 0 = `slot0`, slot 4 = `liquidity`). + +Regenerate with `solc` (the source is `^0.8.20`-compatible): + +```sh +solc --bin --optimize --optimize-runs 200 --overwrite -o out fixtures/EventGroundTruthPool.sol +cp out/TestV3Pool.bin fixtures/test_v3_pool_creation.hex +``` diff --git a/fixtures/test_v3_pool_creation.hex b/fixtures/test_v3_pool_creation.hex new file mode 100644 index 0000000..9f7230e --- /dev/null +++ b/fixtures/test_v3_pool_creation.hex @@ -0,0 +1 @@ +60c060405234801561000f575f80fd5b5060405161075338038061075383398101604081905261002e91610060565b6001600160a01b039182166080521660a052610091565b80516001600160a01b038116811461005b575f80fd5b919050565b5f8060408385031215610071575f80fd5b61007a83610045565b915061008860208401610045565b90509250929050565b60805160a0516106866100cd5f395f818161026d01528181610297015261030d01525f81816069015281816102bd01526102e701526106865ff3fe608060405234801561000f575f80fd5b5060043610610060575f3560e01c80630dfe1681146100645780631a686502146100a85780631ff1a703146100d35780633850c7bd146101b15780635c02d26614610255578063d21220a714610268575b5f80fd5b61008b7f000000000000000000000000000000000000000000000000000000000000000081565b6040516001600160a01b0390911681526020015b60405180910390f35b6004546100bb906001600160801b031681565b6040516001600160801b03909116815260200161009f565b6101af6100e136600461053b565b6040805160e0810182526001600160a01b0395909516808652600285900b602087015261ffff93909316908501819052600160608601819052608086018190525f60a0870181905260c0909601528454600160c81b6001600160b81b0319909116909317600160a01b62ffffff909516949094029390931763ffffffff60b81b1916600160b81b90930261ffff60c81b1916929092171763ffffffff60d81b1916630100000160d81b17909155600480546001600160801b0319166001600160801b03909216919091179055565b005b5f54610204906001600160a01b03811690600160a01b810460020b9061ffff600160b81b8204811691600160c81b8104821691600160d81b8204169060ff600160e81b8204811691600160f01b90041687565b604080516001600160a01b03909816885260029690960b602088015261ffff94851695870195909552918316606086015291909116608084015260ff1660a0830152151560c082015260e00161009f565b6101af6102633660046105a4565b61028f565b61008b7f000000000000000000000000000000000000000000000000000000000000000081565b5f866102bb577f00000000000000000000000000000000000000000000000000000000000000006102dd565b7f00000000000000000000000000000000000000000000000000000000000000005b90505f8761030b577f000000000000000000000000000000000000000000000000000000000000000061032d565b7f00000000000000000000000000000000000000000000000000000000000000005b6040516323b872dd60e01b8152336004820152306024820152604481018990529091506001600160a01b038316906323b872dd906064016020604051808303815f875af1158015610380573d5f803e3d5ffd5b505050506040513d601f19601f820116820180604052508101906103a49190610608565b5060405163a9059cbb60e01b8152336004820152602481018790526001600160a01b0382169063a9059cbb906044016020604051808303815f875af11580156103ef573d5f803e3d5ffd5b505050506040513d601f19601f820116820180604052508101906104139190610608565b505f80546001600160a01b0387166001600160b81b031990911617600160a01b62ffffff871602178155600480546001600160801b0319166001600160801b0386161790558861046b576104668761062a565b61046d565b875b90505f8961047b5788610484565b6104848861062a565b60408051848152602081018390526001600160a01b038a16818301526001600160801b0388166060820152600289900b60808201529051919250339182917fc42079f94a6350d7e6235f29174924f928cc2ac818eb64fed8004e115fbcca67919081900360a00190a350505050505050505050565b80356001600160a01b038116811461050f575f80fd5b919050565b8035600281900b811461050f575f80fd5b80356001600160801b038116811461050f575f80fd5b5f805f806080858703121561054e575f80fd5b610557856104f9565b935061056560208601610514565b9250604085013561ffff8116811461057b575f80fd5b915061058960608601610525565b905092959194509250565b80151581146105a1575f80fd5b50565b5f805f805f8060c087890312156105b9575f80fd5b86356105c481610594565b955060208701359450604087013593506105e0606088016104f9565b92506105ee60808801610514565b91506105fc60a08801610525565b90509295509295509295565b5f60208284031215610618575f80fd5b815161062381610594565b9392505050565b5f600160ff1b820161064a57634e487b7160e01b5f52601160045260245ffd5b505f039056fea2646970667358221220492fd2050a35f65b8045bdcc2057caf77c1aed86d203b17fd05c21e978a89f0d64736f6c63430008170033 \ No newline at end of file diff --git a/src/access_list.rs b/src/access_list.rs index a84c99e..bf9df24 100644 --- a/src/access_list.rs +++ b/src/access_list.rs @@ -15,8 +15,9 @@ use alloy_eips::eip2930::{AccessList, AccessListItem}; use alloy_network::Network; use alloy_primitives::{Address, B256, U256, address}; use alloy_provider::Provider; +use alloy_rlp::Encodable; use alloy_sol_types::{SolCall, sol}; -use anyhow::Result; +use anyhow::{Context as _, Result}; use revm::context::result::ExecutionResult; use tracing::{debug, info}; @@ -141,36 +142,18 @@ impl SmartAccessList { /// - **L2 savings**: `100 gas * entry_count * perArbGas`, where each address /// and each storage key counts as one entry (the EIP-2929 warm-vs-cold /// access discount). - /// - **L1 cost**: `l1_data_gas * l1_base_fee`, where `l1_data_gas` sums the - /// per-byte calldata gas ([`l1_data_gas_for_bytes`]) of every address and - /// key plus a fixed RLP-framing surcharge. - /// - /// # Cost model is approximate - /// - /// The RLP-overhead constants — roughly `4 * 16` gas per address entry, - /// `16` gas per storage key, and `3 * 16` gas for the top-level list headers - /// — are a deliberate **approximation**, not the exact EIP-2930 RLP - /// serialization cost. They assume worst-case non-zero framing bytes and do - /// not account for the real RLP length-prefix sizing, address/key sharing, - /// or rollup-specific compression. Treat this as a rough profitability gate, - /// not a precise gas accounting: a list near the break-even point may be - /// classified either way. + /// - **L1 cost**: `l1_data_gas * l1_base_fee`, where `l1_data_gas` is the + /// exact per-byte calldata gas ([`l1_data_gas_for_bytes`]) of the EIP-2930 + /// RLP-encoded access list. /// /// # Errors /// - /// Returns `Err` only if the call wrapper itself surfaces a non-recoverable - /// error; in practice provider/pricing failures do **not** error. + /// Returns `Err` if the provider/pricing queries fail. /// /// Returns `Ok(None)` when: /// - the list is empty, - /// - the `ArbGasInfo` pricing or L1-base-fee query fails (the error is logged - /// at `debug` and swallowed — see below), /// - either the L2 or L1 gas price reads as zero, or /// - the estimated L1 cost meets or exceeds the L2 savings (not profitable). - /// - /// A `None` returned because a provider query failed is **indistinguishable** - /// from a `None` returned because the list was genuinely unprofitable: both - /// surface as a skipped access list, not as an error. pub async fn into_access_list_if_profitable( self, provider: &P, @@ -182,21 +165,15 @@ impl SmartAccessList { // Query ArbGasInfo for current pricing let arb = ArbGasInfo::new(ARB_GAS_INFO, provider); let prices_call = arb.getPricesInWei(); - let prices = match prices_call.call().await { - Ok(p) => p, - Err(e) => { - debug!(error = %e, "Failed to query ArbGasInfo prices, skipping access list"); - return Ok(None); - } - }; + let prices = prices_call + .call() + .await + .context("failed to query ArbGasInfo prices for access-list profitability")?; let l1_fee_call = arb.getL1BaseFeeEstimate(); - let l1_base_fee = match l1_fee_call.call().await { - Ok(fee) => fee, - Err(e) => { - debug!(error = %e, "Failed to query L1 base fee, skipping access list"); - return Ok(None); - } - }; + let l1_base_fee = l1_fee_call + .call() + .await + .context("failed to query ArbGasInfo L1 base fee for access-list profitability")?; let l2_gas_price = prices.perArbGas; @@ -205,46 +182,14 @@ impl SmartAccessList { return Ok(None); } - // Calculate aggregate L2 savings and L1 cost - let mut total_entries: u64 = 0; - let mut total_l1_data_gas: u64 = 0; - - for item in &self.items { - total_entries += 1; - total_l1_data_gas += l1_data_gas_for_bytes(item.address.as_slice()); - // RLP overhead per address entry (~3-4 bytes, assume non-zero) - total_l1_data_gas += 4 * 16; - - for key in &item.storage_keys { - total_entries += 1; - total_l1_data_gas += l1_data_gas_for_bytes(key.as_slice()); - // RLP length prefix (1 byte, non-zero) - total_l1_data_gas += 16; - } - } - // Top-level RLP list headers (~3 bytes) - total_l1_data_gas += 3 * 16; - - // L2 savings: 100 gas per entry × L2 gas price - let l2_savings_wei = U256::from(total_entries) * U256::from(100) * l2_gas_price; - // L1 cost: serialized data gas × L1 base fee - let l1_cost_wei = U256::from(total_l1_data_gas) * l1_base_fee; - - let profitable = l2_savings_wei > l1_cost_wei; - - info!( - entries = total_entries, - items = self.items.len(), - l2_savings_wei = %l2_savings_wei, - l1_cost_wei = %l1_cost_wei, - l2_gas_price_gwei = %format_gwei(l2_gas_price), - l1_base_fee_gwei = %format_gwei(l1_base_fee), - profitable, - "Access list profitability check" - ); - - if profitable { - Ok(Some(AccessList(self.items))) + let access_list = AccessList(self.items); + if log_access_list_profitability( + &access_list, + l2_gas_price, + l1_base_fee, + "Access list profitability check", + ) { + Ok(Some(access_list)) } else { Ok(None) } @@ -261,31 +206,14 @@ impl SmartAccessList { /// [`SmartAccessList::into_access_list_if_profitable`] for a pre-built /// [`AccessList`]; the two share the same cost model and break-even comparison. /// -/// # Cost model is approximate -/// -/// As with [`SmartAccessList::into_access_list_if_profitable`], the L1 cost is -/// estimated from per-byte calldata gas ([`l1_data_gas_for_bytes`]) plus fixed -/// RLP-framing surcharges (`4 * 16` gas per address, `16` gas per key, `3 * 16` -/// gas for the top-level headers). Those framing constants are an -/// **approximation**, not the exact EIP-2930 RLP serialization cost: they assume -/// worst-case non-zero bytes and ignore real length-prefix sizing and -/// rollup-specific compression. Treat the result as a rough profitability gate. -/// /// # Errors /// -/// Returns `Err` only if the call wrapper itself surfaces a non-recoverable -/// error; in practice provider/pricing failures do **not** error. +/// Returns `Err` if the provider/pricing queries fail. /// /// Returns `Ok(None)` when: /// - the list is empty, -/// - the `ArbGasInfo` pricing or L1-base-fee query fails (the error is logged at -/// `debug` and swallowed), /// - either the L2 or L1 gas price reads as zero, or /// - the estimated L1 cost meets or exceeds the L2 savings (not profitable). -/// -/// A `None` returned because a provider query failed is **indistinguishable** -/// from a `None` returned because the list was genuinely unprofitable: both -/// surface as a skipped access list, not as an error. pub async fn access_list_if_profitable( access_list: AccessList, provider: &P, @@ -296,20 +224,16 @@ pub async fn access_list_if_profitable( // Query ArbGasInfo for current pricing let arb = ArbGasInfo::new(ARB_GAS_INFO, provider); - let prices = match arb.getPricesInWei().call().await { - Ok(p) => p, - Err(e) => { - debug!(error = %e, "Failed to query ArbGasInfo prices, skipping access list"); - return Ok(None); - } - }; - let l1_base_fee = match arb.getL1BaseFeeEstimate().call().await { - Ok(fee) => fee, - Err(e) => { - debug!(error = %e, "Failed to query L1 base fee, skipping access list"); - return Ok(None); - } - }; + let prices = arb + .getPricesInWei() + .call() + .await + .context("failed to query ArbGasInfo prices for access-list profitability")?; + let l1_base_fee = arb + .getL1BaseFeeEstimate() + .call() + .await + .context("failed to query ArbGasInfo L1 base fee for access-list profitability")?; let l2_gas_price = prices.perArbGas; @@ -318,45 +242,12 @@ pub async fn access_list_if_profitable( return Ok(None); } - // Calculate aggregate L2 savings and L1 cost - let mut total_entries: u64 = 0; - let mut total_l1_data_gas: u64 = 0; - - for item in &access_list.0 { - total_entries += 1; - total_l1_data_gas += l1_data_gas_for_bytes(item.address.as_slice()); - // RLP overhead per address entry (~3-4 bytes, assume non-zero) - total_l1_data_gas += 4 * 16; - - for key in &item.storage_keys { - total_entries += 1; - total_l1_data_gas += l1_data_gas_for_bytes(key.as_slice()); - // RLP length prefix (1 byte, non-zero) - total_l1_data_gas += 16; - } - } - // Top-level RLP list headers (~3 bytes) - total_l1_data_gas += 3 * 16; - - // L2 savings: 100 gas per entry × L2 gas price - let l2_savings_wei = U256::from(total_entries) * U256::from(100) * l2_gas_price; - // L1 cost: serialized data gas × L1 base fee - let l1_cost_wei = U256::from(total_l1_data_gas) * l1_base_fee; - - let profitable = l2_savings_wei > l1_cost_wei; - - info!( - entries = total_entries, - items = access_list.0.len(), - l2_savings_wei = %l2_savings_wei, - l1_cost_wei = %l1_cost_wei, - l2_gas_price_gwei = %format_gwei(l2_gas_price), - l1_base_fee_gwei = %format_gwei(l1_base_fee), - profitable, - "Simulation access list profitability check" - ); - - if profitable { + if log_access_list_profitability( + &access_list, + l2_gas_price, + l1_base_fee, + "Simulation access list profitability check", + ) { Ok(Some(access_list)) } else { Ok(None) @@ -483,6 +374,49 @@ pub fn l1_data_gas_for_bytes(data: &[u8]) -> u64 { .sum() } +/// Exact L1 calldata gas for the EIP-2930 RLP encoding of an access list. +pub fn access_list_rlp_data_gas(access_list: &AccessList) -> u64 { + let mut encoded = Vec::with_capacity(access_list.length()); + access_list.encode(&mut encoded); + l1_data_gas_for_bytes(&encoded) +} + +fn access_list_entry_count(access_list: &AccessList) -> u64 { + access_list + .0 + .iter() + .map(|item| 1 + item.storage_keys.len() as u64) + .sum() +} + +fn log_access_list_profitability( + access_list: &AccessList, + l2_gas_price: U256, + l1_base_fee: U256, + message: &'static str, +) -> bool { + let total_entries = access_list_entry_count(access_list); + let total_l1_data_gas = access_list_rlp_data_gas(access_list); + let l2_savings_wei = U256::from(total_entries) * U256::from(100) * l2_gas_price; + let l1_cost_wei = U256::from(total_l1_data_gas) * l1_base_fee; + let profitable = l2_savings_wei > l1_cost_wei; + + info!( + entries = total_entries, + items = access_list.0.len(), + l1_data_gas = total_l1_data_gas, + l2_savings_wei = %l2_savings_wei, + l1_cost_wei = %l1_cost_wei, + l2_gas_price_gwei = %format_gwei(l2_gas_price), + l1_base_fee_gwei = %format_gwei(l1_base_fee), + profitable, + check = message, + "Access list profitability check" + ); + + profitable +} + /// Filter already-warm and excluded addresses from an access list, then apply /// it to the transaction request. /// @@ -571,4 +505,53 @@ mod tests { let addr = Address::repeat_byte(0xFF); assert_eq!(l1_data_gas_for_bytes(addr.as_slice()), 320); } + + #[test] + fn access_list_rlp_data_gas_uses_exact_eip2930_encoding() { + let access_list = AccessList(vec![AccessListItem { + address: Address::ZERO, + storage_keys: Vec::new(), + }]); + + // RLP([[zero_address, []]]) = d7 d6 94 <20 zero bytes> c0. + // Four non-zero framing bytes cost 64 gas; twenty zero address bytes cost + // 80 gas. The old fixed-overhead approximation returned 192. + assert_eq!(access_list_rlp_data_gas(&access_list), 144); + } + + #[tokio::test] + async fn access_list_profitability_provider_error_returns_err() { + use alloy_network::Ethereum; + use alloy_provider::RootProvider; + use alloy_rpc_client::RpcClient; + use alloy_transport::mock::Asserter; + + let provider = RootProvider::::new(RpcClient::mocked(Asserter::new())); + let access_list = AccessList(vec![AccessListItem { + address: Address::repeat_byte(0xAA), + storage_keys: Vec::new(), + }]); + + let err = access_list_if_profitable(access_list, &provider) + .await + .expect_err("provider failures must be distinguishable from unprofitable lists"); + assert!( + err.to_string().contains("ArbGasInfo") || err.to_string().contains("provider"), + "unexpected error: {err:#}" + ); + } + + #[tokio::test] + async fn access_list_profitability_empty_list_still_returns_none() { + use alloy_network::Ethereum; + use alloy_provider::RootProvider; + use alloy_rpc_client::RpcClient; + use alloy_transport::mock::Asserter; + + let provider = RootProvider::::new(RpcClient::mocked(Asserter::new())); + let result = access_list_if_profitable(AccessList::default(), &provider) + .await + .expect("empty list must not query provider"); + assert!(result.is_none()); + } } diff --git a/src/cache/binary_state.rs b/src/cache/binary_state.rs index 1deef61..ccfb3d0 100644 --- a/src/cache/binary_state.rs +++ b/src/cache/binary_state.rs @@ -5,20 +5,25 @@ //! and write a compact binary file. On load, we populate BlockchainDb directly, //! then seed bytecodes from the separate bytecodes.bin cache. //! -//! The file format is raw bincode with no version header or magic bytes, so it -//! is not migratable: a cache written by a build with a different struct layout -//! decodes as a failure (cache miss) rather than being upgraded in place. +//! The file format is a tiny crate-specific envelope (magic bytes + version) +//! followed by bincode payload. Unknown magic/version values are cache misses. use std::path::Path; use std::time::Instant; use alloy_primitives::map::HashMap; use alloy_primitives::{Address, B256, U256}; +use anyhow::{Context as _, Result}; use foundry_fork_db::BlockchainDb; use revm::state::AccountInfo; use serde::{Deserialize, Serialize}; use tracing::{debug, warn}; +use super::versioned; + +const BINARY_STATE_MAGIC: &[u8; 8] = b"EFCSTAT\0"; +const BINARY_STATE_VERSION: u32 = 1; + /// Binary-serializable EVM state. Stores accounts without bytecode (bytecodes /// are loaded separately from bytecodes.bin) and all storage slots. #[derive(Serialize, Deserialize)] @@ -42,16 +47,14 @@ struct BinaryAccountInfo { /// and persisted separately to `bytecodes.bin`; the saved account info keeps /// only the `code_hash`. /// -/// Errors are logged at `warn` level and otherwise swallowed: serialization -/// failures, parent-directory creation failures, and write failures all return -/// without signalling to the caller, so a failed save is indistinguishable from -/// a successful one at the call site. +/// Returns an error if serialization, parent-directory creation, or writing +/// fails, so explicit flush callers can distinguish a successful save from a +/// stale or missing on-disk cache. /// -/// The on-disk format is raw bincode with no version header, so it is not -/// forward/backward compatible: a file written by a build with a different -/// layout will fail to decode on load (treated as a cache miss) rather than +/// The on-disk format carries magic bytes and a version number before the +/// bincode payload. Unknown versions are treated as a cache miss rather than /// being migrated. -pub fn save_binary_state(blockchain_db: &BlockchainDb, path: &Path) { +pub fn save_binary_state(blockchain_db: &BlockchainDb, path: &Path) -> Result<()> { let start = Instant::now(); let accounts: Vec<(Address, BinaryAccountInfo)> = blockchain_db @@ -79,27 +82,28 @@ pub fn save_binary_state(blockchain_db: &BlockchainDb, path: &Path) { let state = BinaryEvmState { accounts, storage }; - match bincode::serialize(&state) { - Ok(data) => { - if let Some(parent) = path.parent() { - let _ = std::fs::create_dir_all(parent); - } - match std::fs::write(path, &data) { - Ok(()) => { - let ms = start.elapsed().as_millis(); - debug!( - accounts = state.accounts.len(), - storage_contracts = state.storage.len(), - bytes = data.len(), - save_ms = ms, - "Saved binary EVM state" - ); - } - Err(e) => warn!(error = %e, "Failed to write binary EVM state"), - } - } - Err(e) => warn!(error = %e, "Failed to serialize binary EVM state"), + let data = versioned::encode( + BINARY_STATE_MAGIC, + BINARY_STATE_VERSION, + &state, + "binary EVM state", + )?; + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent) + .with_context(|| format!("failed to create binary EVM state directory {parent:?}"))?; } + std::fs::write(path, &data) + .with_context(|| format!("failed to write binary EVM state to {path:?}"))?; + + let ms = start.elapsed().as_millis(); + debug!( + accounts = state.accounts.len(), + storage_contracts = state.storage.len(), + bytes = data.len(), + save_ms = ms, + "Saved binary EVM state" + ); + Ok(()) } /// Load binary EVM state and populate the BlockchainDb. @@ -109,9 +113,8 @@ pub fn save_binary_state(blockchain_db: &BlockchainDb, path: &Path) { /// Bytecodes should be seeded separately from bytecodes.bin. /// /// Returns `false` (rather than erroring) when `path` cannot be read or its -/// contents fail to decode as the expected bincode layout; a decode failure is -/// logged at `warn` level. Because the format carries no version header, a file -/// written by an incompatible build is reported as a decode failure here. +/// contents fail the magic/version check or fail to decode as the expected +/// bincode layout; failures are logged at `warn` level. pub fn load_binary_state(blockchain_db: &BlockchainDb, path: &Path) -> bool { let start = Instant::now(); @@ -120,12 +123,14 @@ pub fn load_binary_state(blockchain_db: &BlockchainDb, path: &Path) -> bool { Err(_) => return false, }; - let state: BinaryEvmState = match bincode::deserialize(&data) { - Ok(s) => s, - Err(e) => { - warn!(?e, "Failed to decode binary EVM state, starting fresh"); - return false; - } + let Some(state) = versioned::decode::( + &data, + BINARY_STATE_MAGIC, + BINARY_STATE_VERSION, + "binary EVM state", + ) else { + warn!("Failed to decode binary EVM state, starting fresh"); + return false; }; let account_count = state.accounts.len(); @@ -229,8 +234,18 @@ mod tests { } // Save - save_binary_state(&db, &path); + save_binary_state(&db, &path).expect("save binary state"); assert!(path.exists()); + let bytes = std::fs::read(&path).expect("read saved state"); + assert!( + bytes.starts_with(b"EFCSTAT\0"), + "binary state cache must carry a magic header" + ); + assert_eq!( + &bytes[8..12], + &1u32.to_le_bytes(), + "binary state cache must carry an explicit version" + ); // Load into a fresh db let meta2 = BlockchainDbMeta::default(); @@ -265,6 +280,24 @@ mod tests { let _ = std::fs::remove_dir(&dir); } + #[test] + fn save_binary_state_reports_write_failures() { + let dir = std::env::temp_dir().join("evm_fork_cache_test_binary_state_write_error"); + let _ = std::fs::remove_dir_all(&dir); + let _ = std::fs::remove_file(&dir); + std::fs::write(&dir, b"not a directory").expect("create file path conflict"); + + let db = BlockchainDb::new(BlockchainDbMeta::default(), None); + let path = dir.join("state.bin"); + let err = save_binary_state(&db, &path).expect_err("save must report write failure"); + assert!( + err.to_string().contains("directory") || err.to_string().contains("Not a directory"), + "unexpected error: {err:#}" + ); + + let _ = std::fs::remove_file(&dir); + } + #[test] fn test_load_missing_file_returns_false() { let meta = BlockchainDbMeta::default(); @@ -289,4 +322,25 @@ mod tests { let _ = std::fs::remove_file(&path); let _ = std::fs::remove_dir(&dir); } + + #[test] + fn load_legacy_raw_bincode_returns_false() { + let dir = std::env::temp_dir().join("evm_fork_cache_test_binary_state_legacy"); + let path = dir.join("legacy.bin"); + let _ = std::fs::create_dir_all(&dir); + let legacy = BinaryEvmState { + accounts: Vec::new(), + storage: Vec::new(), + }; + std::fs::write(&path, bincode::serialize(&legacy).unwrap()).unwrap(); + + let db = BlockchainDb::new(BlockchainDbMeta::default(), None); + assert!( + !load_binary_state(&db, &path), + "unversioned legacy bincode must be treated as a cache miss" + ); + + let _ = std::fs::remove_file(&path); + let _ = std::fs::remove_dir(&dir); + } } diff --git a/src/cache/bytecode.rs b/src/cache/bytecode.rs index a894a58..1ef00bc 100644 --- a/src/cache/bytecode.rs +++ b/src/cache/bytecode.rs @@ -6,10 +6,9 @@ //! entries are used to re-seed the `code` of accounts that were restored //! without it. //! -//! Each entry's bytes are hex-encoded for the serde representation, but the -//! file is written as raw bincode with no version header, so a cache written by -//! an incompatible build fails to decode (cache miss) rather than being -//! migrated. +//! Each entry's bytes are hex-encoded for the serde representation. The file is +//! written as a crate-specific versioned envelope followed by bincode payload, so +//! incompatible versions are detected as cache misses. use std::collections::HashMap; use std::path::Path; @@ -18,7 +17,11 @@ use alloy_primitives::Address; use anyhow::Result; use foundry_fork_db::BlockchainDb; use serde::{Deserialize, Serialize}; -use tracing::warn; + +use super::versioned; + +const BYTECODE_CACHE_MAGIC: &[u8; 8] = b"EFCBYTE\0"; +const BYTECODE_CACHE_VERSION: u32 = 1; /// Serializable bytecode cache entry. #[derive(Debug, Clone, Serialize, Deserialize)] @@ -38,15 +41,16 @@ pub(crate) struct BytecodeCache { impl BytecodeCache { /// Load bytecode cache from disk (binary format). /// - /// Returns `None` if `path` cannot be read or its contents fail to decode as - /// bincode for this type; a decode failure is logged at `warn` level. The - /// format carries no version header, so a file from an incompatible build is - /// reported as `None`. + /// Returns `None` if `path` cannot be read, fails the magic/version check, or + /// fails to decode as bincode for this type. pub(crate) fn load(path: &Path) -> Option { let data = std::fs::read(path).ok()?; - bincode::deserialize(&data) - .inspect_err(|e| warn!("Failed to parse bytecode cache (bincode): {}", e)) - .ok() + versioned::decode( + &data, + BYTECODE_CACHE_MAGIC, + BYTECODE_CACHE_VERSION, + "bytecode cache", + ) } /// Save bytecode cache to disk (binary format). @@ -62,7 +66,12 @@ impl BytecodeCache { if let Some(parent) = path.parent() { std::fs::create_dir_all(parent)?; } - let data = bincode::serialize(self)?; + let data = versioned::encode( + BYTECODE_CACHE_MAGIC, + BYTECODE_CACHE_VERSION, + self, + "bytecode cache", + )?; std::fs::write(path, data)?; Ok(()) } @@ -117,6 +126,16 @@ mod tests { }, ); cache.save(&path).expect("save bytecode cache"); + let bytes = std::fs::read(&path).expect("read saved bytecode cache"); + assert!( + bytes.starts_with(b"EFCBYTE\0"), + "bytecode cache must carry a magic header" + ); + assert_eq!( + &bytes[8..12], + &1u32.to_le_bytes(), + "bytecode cache must carry an explicit version" + ); let loaded = BytecodeCache::load(&path).expect("load bytecode cache"); assert_eq!( @@ -128,6 +147,26 @@ mod tests { let _ = std::fs::remove_dir_all(path.parent().unwrap()); } + #[test] + fn load_legacy_raw_bincode_is_none() { + let path = temp_path("legacy"); + let mut cache = BytecodeCache::default(); + cache.contracts.insert( + Address::repeat_byte(0x42), + BytecodeCacheEntry { + bytecode: vec![0x60, 0x00], + }, + ); + std::fs::write(&path, bincode::serialize(&cache).unwrap()).expect("write legacy cache"); + + assert!( + BytecodeCache::load(&path).is_none(), + "unversioned legacy bincode must be treated as a cache miss" + ); + + let _ = std::fs::remove_dir_all(path.parent().unwrap()); + } + #[test] fn load_missing_file_is_none() { assert!(BytecodeCache::load(std::path::Path::new("/nonexistent/bytecodes.bin")).is_none()); diff --git a/src/cache/metadata.rs b/src/cache/metadata.rs index 6ab9472..670de08 100644 --- a/src/cache/metadata.rs +++ b/src/cache/metadata.rs @@ -12,10 +12,14 @@ use std::path::{Path, PathBuf}; use alloy_primitives::{Address, B256, U256}; use anyhow::Result; use serde::{Deserialize, Serialize}; -use tracing::warn; use std::collections::HashSet; +use super::versioned; + +const IMMUTABLE_CACHE_MAGIC: &[u8; 8] = b"EFCMETA\0"; +const IMMUTABLE_CACHE_VERSION: u32 = 1; + /// Configuration for disk-based caching of EVM state. /// /// Enables on-disk persistence of fetched fork state. Cache files are laid out @@ -158,18 +162,17 @@ pub struct ImmutableDataCache { impl ImmutableDataCache { /// Load immutable data cache from disk (binary format). /// - /// Returns `None` if `path` cannot be read or the contents are not valid - /// bincode for this type (a parse failure is logged at `warn` level and - /// swallowed). Callers should treat `None` as "no cache yet" and start fresh. - /// - /// Note: the on-disk format is bincode with no version header, so a cache - /// written by an incompatible build deserializes as a parse failure (`None`) - /// rather than being migrated. + /// Returns `None` if `path` cannot be read, fails the magic/version check, or + /// the payload is not valid bincode for this type. Callers should treat + /// `None` as "no cache yet" and start fresh. pub fn load(path: &Path) -> Option { let data = std::fs::read(path).ok()?; - bincode::deserialize(&data) - .inspect_err(|e| warn!("Failed to parse immutable data cache (bincode): {}", e)) - .ok() + versioned::decode( + &data, + IMMUTABLE_CACHE_MAGIC, + IMMUTABLE_CACHE_VERSION, + "immutable data cache", + ) } /// Save immutable data cache to disk (binary format). @@ -185,7 +188,12 @@ impl ImmutableDataCache { if let Some(parent) = path.parent() { std::fs::create_dir_all(parent)?; } - let data = bincode::serialize(self)?; + let data = versioned::encode( + IMMUTABLE_CACHE_MAGIC, + IMMUTABLE_CACHE_VERSION, + self, + "immutable data cache", + )?; std::fs::write(path, data)?; Ok(()) } diff --git a/src/cache/mod.rs b/src/cache/mod.rs index b83a640..5ae04a2 100644 --- a/src/cache/mod.rs +++ b/src/cache/mod.rs @@ -8,6 +8,7 @@ pub mod snapshot; mod storage_keys; #[cfg(feature = "protocols")] mod tick_snapshot; +mod versioned; pub use binary_state::{load_binary_state, save_binary_state}; pub use metadata::{ @@ -50,13 +51,13 @@ use alloy_primitives::{Address, B256, Bytes, I256, Log, TxKind, U256, keccak256} use alloy_provider::{Provider, network::AnyNetwork}; use alloy_rpc_types_eth::TransactionRequest; use alloy_sol_types::{SolCall, SolValue, sol}; -use anyhow::{Result, anyhow}; +use anyhow::{Context as _, Result, anyhow}; use foundry_fork_db::{BlockchainDb, SharedBackend, cache::BlockchainDbMeta}; use revm::{ Context, ExecuteCommitEvm, ExecuteEvm, InspectEvm, MainBuilder, MainContext, context::{BlockEnv, CfgEnv, Journal, LocalContext, TxEnv, result::ExecutionResult}, context_interface::JournalTr, - database::CacheDB, + database::{AccountState, CacheDB}, primitives::hardfork::SpecId, state::{AccountInfo, Bytecode}, }; @@ -66,6 +67,10 @@ use crate::access_set::StorageAccessList; use crate::errors::{SimError, SimulationError, SimulationResult}; use crate::freshness::SlotChange; use crate::inspector::TransferInspector; +use crate::state_update::{ + AccountChange, AccountPatch, PurgeRecord, PurgeScope, SkippedAccountPatch, SkippedBalanceDelta, + SkippedDelta, SkippedMask, SlotDelta, StateDiff, StateUpdate, +}; use bytecode::BytecodeCache; #[cfg(feature = "protocols")] @@ -128,6 +133,61 @@ fn block_in_place_handle() -> Result { } } +/// Read a storage slot from already-borrowed layers (`account_state`-aware), +/// mirroring [`EvmCache::cached_storage_value`] but operating on a held backend +/// storage guard rather than re-locking. Shared by the batched slot-run fast-path +/// ([`EvmCache::apply_slot_run`]) so the same EVM-SLOAD semantics hold inside the +/// held guard: the overlay slot wins; a `StorageCleared`/`NotExisting` overlay +/// account reads a missing slot as ZERO (the backend is **not** consulted); +/// otherwise it falls through to the backend. +fn read_slot_account_state_aware( + overlay: &std::collections::HashMap, + storage: &std::collections::HashMap, + address: Address, + slot: U256, +) -> Option +where + S1: std::hash::BuildHasher, + S2: std::hash::BuildHasher, +{ + if let Some(db_account) = overlay.get(&address) { + if let Some(value) = db_account.storage.get(&slot) { + return Some(*value); + } + if matches!( + db_account.account_state, + AccountState::StorageCleared | AccountState::NotExisting + ) { + return Some(U256::ZERO); + } + } + storage.get(&address).and_then(|s| s.get(&slot).copied()) +} + +/// Write a storage slot into already-borrowed layers, mirroring +/// [`EvmCache::write_slot_through`] but operating on a held backend storage guard. +/// Backend (layer 2) is always written; the overlay (layer 1) is written only if +/// an overlay account already exists (never materialize a new overlay account). +fn write_slot_into( + overlay: &mut std::collections::HashMap, + storage: &mut std::collections::HashMap, + address: Address, + slot: U256, + value: U256, +) where + S1: std::hash::BuildHasher, + S2: std::hash::BuildHasher + Default, +{ + storage.entry(address).or_default().insert(slot, value); + if let Some(db_account) = overlay.get_mut(&address) { + db_account.storage.insert(slot, value); + } +} + +fn account_patch_is_empty(patch: &AccountPatch) -> bool { + patch.balance.is_none() && patch.nonce.is_none() && patch.code.is_none() +} + static CACHE_SPEED_MODE: AtomicU8 = AtomicU8::new(CacheSpeedMode::Slow as u8); /// Runtime tuning profile for cache-side batch storage fetches. @@ -245,6 +305,7 @@ pub struct EvmCacheBuilder

{ block: Option, cache_config: Option, spec_id: SpecId, + shared_memory_capacity: SharedMemoryCapacity, } impl

EvmCacheBuilder

@@ -258,6 +319,7 @@ where block: None, cache_config: None, spec_id: SpecId::CANCUN, + shared_memory_capacity: SharedMemoryCapacity::default(), } } @@ -299,9 +361,29 @@ where self } + /// Set how much EVM shared memory to pre-allocate per simulation context. + /// + /// Defaults to [`SharedMemoryCapacity::Fixed`] with `64 * 1024` bytes + /// (65,536 bytes). + /// Use `Fixed(n)` to pin a size, or [`SharedMemoryCapacity::Auto`] to size it + /// from the chain state loaded at [`build`](Self::build) time (e.g. a bincode + /// state file supplied via [`cache_config`](Self::cache_config)). See + /// [`SharedMemoryCapacity`] for the trade-offs. + pub fn shared_memory_capacity(mut self, capacity: SharedMemoryCapacity) -> Self { + self.shared_memory_capacity = capacity; + self + } + /// Build the [`EvmCache`], fetching the pinned block's header for context. pub async fn build(self) -> EvmCache { - EvmCache::with_cache(self.provider, self.block, self.cache_config, self.spec_id).await + EvmCache::with_cache_capacity( + self.provider, + self.block, + self.cache_config, + self.spec_id, + self.shared_memory_capacity, + ) + .await } } @@ -313,11 +395,70 @@ type InspectorCacheEvm<'a, INSP> = revm::MainnetEvm< INSP, >; -/// Default initial capacity for shared memory buffer. -/// Set to 64KB based on profiling (16x the REVM default of 4KB). -/// This eliminates reallocation during typical simulations with headroom. +/// Default initial capacity for the EVM shared-memory (working-memory) buffer. +/// 64 KiB (65,536 bytes), chosen from profiling a state-heavy workload (16x the +/// revm default of 4 KiB) so simulations rarely reallocate. Exposed for tuning via +/// [`SharedMemoryCapacity`]. const DEFAULT_SHARED_MEMORY_CAPACITY: usize = 64 * 1024; +/// How much EVM shared memory (per-context working memory) to pre-allocate for +/// simulations. +/// +/// revm grows its shared memory on demand during execution; pre-allocating just +/// avoids repeated reallocations when simulations touch a lot of memory — the +/// original motivation was a state-heavy workload where resizing was hot. The +/// trade-off cuts both ways: a wide parallel fan-out of *small* simulations pays +/// this much memory per overlay, so general users may want a smaller `Fixed` size, +/// while state-heavy users can raise it or let it auto-size from the loaded state. +/// +/// The default is `Fixed(64 * 1024)` (65,536 bytes). Configure it on +/// [`EvmCacheBuilder::shared_memory_capacity`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum SharedMemoryCapacity { + /// Pre-allocate exactly this many bytes. The [`Default`] is + /// `Fixed(64 * 1024)`. + Fixed(usize), + /// Size the buffer from the amount of chain state loaded into the cache at + /// construction (e.g. from a bincode state file via + /// [`CacheConfig`]/[`EvmCacheBuilder::cache_config`]), clamped to a sane + /// floor/ceiling. Falls back to the floor when nothing is loaded. + /// + /// This is a heuristic proxy — persisted state size loosely correlates with the + /// working-set size of simulations over it, not an exact peak-memory model. Use + /// `Fixed` when you have profiled your workload. + Auto, +} + +impl Default for SharedMemoryCapacity { + fn default() -> Self { + Self::Fixed(DEFAULT_SHARED_MEMORY_CAPACITY) + } +} + +impl SharedMemoryCapacity { + /// Floor for [`Auto`](Self::Auto) (and the default fixed size): 64 KiB + /// (65,536 bytes). + pub const MIN_AUTO: usize = DEFAULT_SHARED_MEMORY_CAPACITY; + /// Ceiling for [`Auto`](Self::Auto): 4 MiB. A simulation that needs more than + /// this still works — revm grows the buffer past it on demand. + pub const MAX_AUTO: usize = 4 * 1024 * 1024; + /// Heuristic proxy: bytes of pre-allocated working memory per loaded storage + /// slot. Tune if profiling warrants. + const AUTO_BYTES_PER_SLOT: usize = 16; + + /// Resolve to a concrete byte capacity. `loaded_slots` is the number of layer-2 + /// storage slots present in the cache at construction (0 when nothing is + /// loaded); it is consulted only for [`Auto`](Self::Auto). + pub(crate) fn resolve(self, loaded_slots: usize) -> usize { + match self { + Self::Fixed(bytes) => bytes, + Self::Auto => loaded_slots + .saturating_mul(Self::AUTO_BYTES_PER_SLOT) + .clamp(Self::MIN_AUTO, Self::MAX_AUTO), + } + } +} + /// EVM cache with lazy-loading RPC backend. /// /// Uses `foundry-fork-db` for intelligent caching and request deduplication. @@ -377,6 +518,32 @@ pub struct EvmCache { /// layer hardfork for accurate gas accounting. Configured per-chain via `evm_spec` /// in `chains.toml`. spec_id: SpecId, + /// Memoized, `Arc`-shared flatten of the cold layer-2 index, reused across + /// successive [`create_snapshot`](Self::create_snapshot) calls (Pillar A). + /// `None` until the first snapshot. Rebuilt copy-on-write by + /// [`refresh_base`](Self::refresh_base); never mutated in place once shared. + /// Not part of any public API and not serialized. + base: Option>, + /// Layer-2 addresses changed since `base` was built, folded into the next base + /// rebuild. Populated by the base-invalidation sites (write-through, batch + /// injects, layer-2 seeding, purges). Not serialized. + base_dirty: HashSet

, + /// When set, the next [`refresh_base`](Self::refresh_base) rebuilds the base + /// from scratch. Set by [`set_block`](Self::set_block) / + /// [`repin_to_block`](Self::repin_to_block), which replace layer 2 wholesale. + /// Not serialized. + base_full_rebuild: bool, + /// Per-account layer-2 slot count at the last base build, used by + /// [`refresh_base`](Self::refresh_base)'s `O(accounts)` length-scan to detect + /// uncontrolled lazy-fetch growth that bypasses the write funnel. Not + /// serialized. + base_storage_lens: HashMap, + /// Resolved per-context EVM shared-memory pre-allocation (bytes), from the + /// [`SharedMemoryCapacity`] at construction (resolving `Auto` against the loaded + /// state). Propagated to each [`EvmSnapshot`] so snapshot-backed overlays + /// pre-allocate the same amount. See + /// [`shared_memory_capacity`](Self::shared_memory_capacity). + shared_memory_capacity: usize, } /// Outcome of a balance-delta-tracking simulation. @@ -511,6 +678,31 @@ impl EvmCache { cache_config: Option, spec_id: SpecId, ) -> Self + where + P: Provider + 'static, + { + Self::with_cache_capacity( + provider, + block, + cache_config, + spec_id, + SharedMemoryCapacity::default(), + ) + .await + } + + /// Like [`with_cache`](Self::with_cache) but takes an explicit + /// [`SharedMemoryCapacity`] controlling per-context EVM working-memory + /// pre-allocation. This is what [`EvmCacheBuilder::build`] calls; prefer the + /// builder. With [`SharedMemoryCapacity::Auto`] the buffer is sized from the + /// layer-2 storage loaded at construction (e.g. a bincode state file). + pub async fn with_cache_capacity

( + provider: Arc

, + block: Option, + cache_config: Option, + spec_id: SpecId, + shared_memory_capacity: SharedMemoryCapacity, + ) -> Self where P: Provider + 'static, { @@ -829,6 +1021,20 @@ impl EvmCache { // Extract chain_id from cache config if available, default to Arbitrum let chain_id = cache_config.as_ref().map(|c| c.chain_id).unwrap_or(42161); + // Resolve the shared-memory pre-allocation. For `Auto` we size from the + // amount of layer-2 chain state actually loaded (post-filter), so a large + // bincode state file yields a larger buffer; `Fixed` ignores the count. + let loaded_slots = match shared_memory_capacity { + SharedMemoryCapacity::Auto => blockchain_db + .storage() + .read() + .values() + .map(|s| s.len()) + .sum(), + SharedMemoryCapacity::Fixed(_) => 0, + }; + let shared_memory_capacity = shared_memory_capacity.resolve(loaded_slots); + Self { backend, blockchain_db, @@ -846,14 +1052,17 @@ impl EvmCache { coinbase, prevrandao, block_gas_limit, - shared_memory_buffer: Rc::new(RefCell::new(Vec::with_capacity( - DEFAULT_SHARED_MEMORY_CAPACITY, - ))), + shared_memory_buffer: Rc::new(RefCell::new(Vec::with_capacity(shared_memory_capacity))), rpc_caller: Some(rpc_caller), storage_batch_fetcher: Some(storage_batch_fetcher), batch_block_id, erc20_balance_slots: HashMap::new(), spec_id, + base: None, + base_dirty: HashSet::new(), + base_full_rebuild: false, + base_storage_lens: HashMap::new(), + shared_memory_capacity, } } @@ -928,6 +1137,11 @@ impl EvmCache { batch_block_id: Arc::new(Mutex::new(block.unwrap_or_default())), erc20_balance_slots: HashMap::new(), spec_id, + base: None, + base_dirty: HashSet::new(), + base_full_rebuild: false, + base_storage_lens: HashMap::new(), + shared_memory_capacity: DEFAULT_SHARED_MEMORY_CAPACITY, } } @@ -941,56 +1155,59 @@ impl EvmCache { /// /// Call this after loading AMMs and running simulations to speed up subsequent runs. /// The cache is also automatically flushed when the EvmCache is dropped. - pub fn flush(&self) { + pub fn flush(&self) -> Result<()> { if let Some(cfg) = &self.cache_config { // Save EVM state to binary cache (bincode format) let binary_path = cfg.binary_state_cache_path(); - binary_state::save_binary_state(&self.blockchain_db, &binary_path); + binary_state::save_binary_state(&self.blockchain_db, &binary_path) + .with_context(|| format!("failed to save binary state cache to {binary_path:?}"))?; // Save bytecode cache let bytecode_path = cfg.bytecode_cache_path(); let mut bytecode_cache = BytecodeCache::load(&bytecode_path).unwrap_or_default(); bytecode_cache.merge_from_db(&self.blockchain_db); - if let Err(e) = bytecode_cache.save(&bytecode_path) { - warn!(error = %e, "Failed to save bytecode cache"); - } else { - debug!( - count = bytecode_cache.contracts.len(), - path = ?bytecode_path, - "Updated bytecode cache (binary format)" - ); - } + bytecode_cache + .save(&bytecode_path) + .with_context(|| format!("failed to save bytecode cache to {bytecode_path:?}"))?; + debug!( + count = bytecode_cache.contracts.len(), + path = ?bytecode_path, + "Updated bytecode cache (binary format)" + ); // Save the immutable data cache let immutable_path = cfg.immutable_cache_path(); - if let Err(e) = self.immutable_cache.save(&immutable_path) { - warn!(error = %e, "Failed to save immutable data cache"); - } else { - debug!( - token_decimals = self.immutable_cache.token_decimals.len(), - v2_pools = self.immutable_cache.v2_pools.len(), - v3_pools = self.immutable_cache.v3_pools.len(), - balancer_pools = self.immutable_cache.balancer_pools.len(), - path = ?immutable_path, - "Updated immutable data cache" - ); - } + self.immutable_cache + .save(&immutable_path) + .with_context(|| { + format!("failed to save immutable data cache to {immutable_path:?}") + })?; + debug!( + token_decimals = self.immutable_cache.token_decimals.len(), + v2_pools = self.immutable_cache.v2_pools.len(), + v3_pools = self.immutable_cache.v3_pools.len(), + balancer_pools = self.immutable_cache.balancer_pools.len(), + path = ?immutable_path, + "Updated immutable data cache" + ); // Save the V3 tick snapshot cache (needed for liquidity validation) #[cfg(feature = "protocols")] { let tick_snapshot_path = cfg.tick_snapshot_cache_path(); - if let Err(e) = self.tick_snapshot_cache.save(&tick_snapshot_path) { - warn!(error = %e, "Failed to save V3 tick snapshot cache"); - } else { - debug!( - snapshots = self.tick_snapshot_cache.len(), - path = ?tick_snapshot_path, - "Updated V3 tick snapshot cache" - ); - } + self.tick_snapshot_cache + .save(&tick_snapshot_path) + .with_context(|| { + format!("failed to save V3 tick snapshot cache to {tick_snapshot_path:?}") + })?; + debug!( + snapshots = self.tick_snapshot_cache.len(), + path = ?tick_snapshot_path, + "Updated V3 tick snapshot cache" + ); } } + Ok(()) } /// Get the cache configuration, if any. @@ -1002,24 +1219,73 @@ impl EvmCache { self.cache_config.as_ref() } - /// Get a reference to the underlying [`BlockchainDb`] (the layer-2 backend - /// store of accounts, storage, and bytecodes). + /// Run a synchronous direct mutation against the underlying [`BlockchainDb`] + /// and invalidate the memoized snapshot base afterwards. + /// + /// This is the preferred escape hatch for unavoidable layer-2 map writes such + /// as `accounts().write().insert(...)` or `storage().write().insert(...)`. + /// The closure still bypasses the CacheDB overlay and the normal write funnel, + /// so use higher-level mutators when they can express the change. Unlike + /// [`unchecked_blockchain_db`](Self::unchecked_blockchain_db), this wrapper + /// keeps the copy-on-write snapshot base honest automatically after in-place + /// overwrites whose map cardinality does not change. + pub fn with_blockchain_db_mut(&mut self, f: impl FnOnce(&BlockchainDb) -> R) -> R { + let result = f(&self.blockchain_db); + self.invalidate_base(); + result + } + + /// Get an unchecked reference to the underlying [`BlockchainDb`] (the layer-2 + /// backend store of accounts, storage, and bytecodes). /// /// This exposes an internal store and bypasses the cache's two-layer - /// consistency model: reads here see only the backend layer, not the - /// CacheDB overlay, and any writes performed through it skip the overlay. - /// Prefer the higher-level accessors; use with care. - pub fn blockchain_db(&self) -> &BlockchainDb { + /// consistency model: reads here see only the backend layer, not the CacheDB + /// overlay, and any writes performed through it skip the overlay. Prefer + /// higher-level accessors or [`with_blockchain_db_mut`](Self::with_blockchain_db_mut) + /// for direct synchronous writes. + /// + /// # Snapshot base + /// Writing layer 2 directly through this unchecked handle also bypasses the + /// memoized copy-on-write snapshot base (Pillar A). The next + /// [`create_snapshot`](Self::create_snapshot) only performs a count/absence + /// growth scan over layer 2, which catches lazy RPC-populated accounts/slots + /// because that path only appends at a fixed block. It does **not** catch + /// direct in-place changes where cardinality is unchanged: overwriting an + /// existing storage slot, or changing an existing account's info/code/balance + /// without adding a new account, can leave a stale snapshot base. After such a + /// direct write, call + /// [`invalidate_snapshot_base`](Self::invalidate_snapshot_base) (or re-pin via + /// [`set_block`](Self::set_block)) before the next snapshot. Writes via the + /// crate's own mutators (`inject_storage_batch`, `apply_update`, the `inject_*` + /// helpers, the purges) keep the base honest automatically. + pub fn unchecked_blockchain_db(&self) -> &BlockchainDb { &self.blockchain_db } - /// Get a reference to the underlying [`SharedBackend`] (the lazy RPC-backed - /// fetcher shared across clones). + /// Get an unchecked reference to the underlying [`SharedBackend`] (the lazy + /// RPC-backed fetcher shared across clones). /// - /// This exposes an internal and bypasses the cache's two-layer consistency + /// This exposes an internal handle and bypasses the cache's two-layer consistency /// model: it reads/fetches directly without consulting the CacheDB overlay. /// Prefer the higher-level accessors; use with care. - pub fn backend(&self) -> &SharedBackend { + /// + /// # Snapshot base + /// Lazy RPC fetches through this backend only append missing accounts/slots at + /// the pinned block, so the snapshot growth scan catches them without an + /// explicit invalidation. Direct `SharedBackend::insert_or_update_storage` / + /// `insert_or_update_address` calls are different: they enqueue a background + /// handler request that can rewrite layer-2 entries **in place**, leaving the + /// memoized copy-on-write base stale at an unchanged slot/account count. + /// + /// If you use those helpers directly, first synchronize with the backend + /// handler by reading back the updated account/slot through `SharedBackend` + /// (for example via `basic_ref` / `storage_ref`), then call + /// [`invalidate_snapshot_base`](Self::invalidate_snapshot_base) before the next + /// [`create_snapshot`](Self::create_snapshot). Calling + /// `invalidate_snapshot_base` immediately after `insert_or_update_*` is not, by + /// itself, a guarantee that the queued update has been applied before the next + /// snapshot. + pub fn unchecked_backend(&self) -> &SharedBackend { &self.backend } @@ -1069,15 +1335,26 @@ impl EvmCache { self.storage_batch_fetcher.as_ref() } - /// Inject batch-fetched storage values directly into BlockchainDb. + /// Inject batch-fetched storage values directly into BlockchainDb (layer 2). /// /// This bypasses SharedBackend and makes values available for subsequent /// `storage_ref()` calls and EVM SLOADs. Used after `StorageBatchFetchFn` /// returns results to populate the cache in bulk. - pub fn inject_storage_batch(&self, results: &[(Address, U256, U256)]) { - let mut storage = self.blockchain_db.storage().write(); - for &(addr, slot, value) in results { - storage.entry(addr).or_default().insert(slot, value); + /// + /// Takes `&mut self` (as of Pillar A) so it can mark each touched address dirty + /// for the memoized copy-on-write base; the write itself is still a direct + /// layer-2 backend write. Overwriting an existing slot at an unchanged slot + /// count is invalidated here too, since the `refresh_base` growth scan only + /// catches length changes. + pub fn inject_storage_batch(&mut self, results: &[(Address, U256, U256)]) { + { + let mut storage = self.blockchain_db.storage().write(); + for &(addr, slot, value) in results { + storage.entry(addr).or_default().insert(slot, value); + } + } + for &(addr, _, _) in results { + self.mark_base_dirty(addr); } } @@ -1100,17 +1377,630 @@ impl EvmCache { /// the backend write is already authoritative and materializing an overlay /// entry would pollute layer 1 and could shadow later RPC reads. pub fn inject_storage_batch_fresh(&mut self, results: &[(Address, U256, U256)]) { + // Thin wrapper over the unified write primitive (the F1 fix now lives in + // `apply_slot`). Each tuple becomes a write-through `StateUpdate::Slot`; + // the returned diff is discarded to preserve this method's `-> ()` API. + let updates: Vec = results + .iter() + .map(|&(addr, slot, value)| StateUpdate::slot(addr, slot, value)) + .collect(); + let _ = self.apply_updates(&updates); + } + + /// Apply a single targeted [`StateUpdate`], returning a [`StateDiff`] of what + /// actually changed. + /// + /// This is the single primitive that writes the state-update vocabulary + /// across both cache layers with one consistent, documented policy. It is + /// **synchronous and infallible** — a write, not a fetch, so it never touches + /// RPC and never errors. See the [`state_update`](crate::state_update) module + /// for the dual-layer write-through policy and the diff semantics. + /// + /// - [`StateUpdate::Slot`] — write `value` into the backend (layer 2) always, + /// and into the overlay (layer 1) only if an overlay account already + /// exists. Records a [`SlotChange`] only when the value actually changes + /// (`old.unwrap_or(ZERO) != value`). + /// - [`StateUpdate::SlotDelta`] — *relative*, cold-aware. If the slot has a + /// cached value, write the saturating delta through the same path and record + /// a [`SlotChange`] iff it changed; if the slot is cold (absent from both + /// layers), apply nothing and surface a `SkippedDelta` in `diff.skipped`. + /// - [`StateUpdate::BalanceDelta`] — *relative*, cold-aware native-balance + /// update. If the account is present in either layer, apply the saturating + /// delta to its balance (nonce/code preserved) write-through and record an + /// [`AccountChange`] iff it changed; if the account is cold (absent from both + /// layers), apply nothing and surface a [`SkippedBalanceDelta`] in + /// `diff.skipped_balances` (no default account is materialized). + /// - [`StateUpdate::Account`] — load the current `AccountInfo` from the cached + /// layers (no RPC), apply each `Some` patch field (recomputing the code hash + /// when `code` is set), then write through with the same layer policy. + /// Records an [`AccountChange`] with `Some((old, new))` only for fields + /// that changed. If the account is cold (absent from both layers), apply + /// nothing and surface a [`SkippedAccountPatch`] in + /// `diff.skipped_accounts`. + /// - [`StateUpdate::AccountUpsert`] — same patch semantics, but intentionally + /// materializes a cold/default account when absent from both layers. + /// - [`StateUpdate::Purge`] — dispatch to the matching purge layer logic and + /// record a [`PurgeRecord`]. + /// + /// # Warning — relative updates can be skipped + /// + /// A cold-aware update targeting a **cold** address is *dropped, not applied* + /// unless it is an explicit [`StateUpdate::AccountUpsert`]. Because a skip + /// produces no change, it is invisible to the changes-only + /// [`StateDiff::is_empty`] / [`StateDiff::len`] success check, so after + /// applying cold-aware updates the caller **must** inspect + /// [`StateDiff::has_skipped`] (or the `skipped_*` fields) and fetch+seed the + /// cold target. + /// + /// ```no_run + /// # use alloy_primitives::{Address, U256}; + /// # use evm_fork_cache::StateUpdate; + /// # fn example(cache: &mut evm_fork_cache::cache::EvmCache) { + /// let pool = Address::repeat_byte(0x01); + /// let diff = cache.apply_update(&StateUpdate::slot(pool, U256::from(0), U256::from(42))); + /// assert_eq!(diff.slots.len(), 1); + /// # } + /// ``` + pub fn apply_update(&mut self, update: &StateUpdate) -> StateDiff { + let mut diff = StateDiff::default(); + match update { + StateUpdate::Slot { + address, + slot, + value, + } => { + if let Some(change) = self.apply_slot(*address, *slot, *value) { + diff.slots.push(change); + } + } + StateUpdate::SlotDelta { + address, + slot, + delta, + } => match self.cached_storage_value(*address, *slot) { + // Hot slot: apply the saturating delta write-through. Build the + // change from the value we already read (do not route through + // `apply_slot`, which would re-read the same slot — §16.9.1). + Some(current) => { + let new = delta.apply(current); + self.write_slot_through(*address, *slot, new); + if current != new { + diff.slots.push(SlotChange { + address: *address, + slot: *slot, + old: current, + new, + }); + } + } + // Cold slot: applying `0 ± amount` would corrupt an unknown value, + // so write nothing and surface the skip for the caller to seed. + None => diff.skipped.push(SkippedDelta { + address: *address, + slot: *slot, + delta: *delta, + }), + }, + StateUpdate::SlotMasked { + address, + slot, + mask, + value, + } => match self.cached_storage_value(*address, *slot) { + // Hot slot: overwrite only the masked bits, preserving the rest. + // Build the change from the value we already read (mirroring the + // `SlotDelta` arm; do not re-read through `apply_slot`). + Some(old) => { + let new = (old & !*mask) | (*value & *mask); + self.write_slot_through(*address, *slot, new); + if old != new { + diff.slots.push(SlotChange { + address: *address, + slot: *slot, + old, + new, + }); + } + } + // Cold slot: the un-masked bits are unknown, so the result cannot + // be computed; write nothing and surface the skip for re-seeding. + None => diff.skipped_masks.push(SkippedMask { + address: *address, + slot: *slot, + mask: *mask, + value: *value, + }), + }, + StateUpdate::BalanceDelta { address, delta } => { + match self.apply_balance_delta(*address, *delta) { + // Hot account: the saturating delta was applied. + Ok(Some(change)) => diff.accounts.push(change), + // Hot account but no change (e.g. Sub from 0, Add of 0). + Ok(None) => {} + // Cold account: surface the skip; nothing was materialized. + Err(skipped) => diff.skipped_balances.push(skipped), + } + } + StateUpdate::Account { address, patch } => { + match self.apply_account_patch(*address, patch, false) { + Ok(Some(change)) => diff.accounts.push(change), + Ok(None) => {} + Err(skipped) => diff.skipped_accounts.push(skipped), + } + } + StateUpdate::AccountUpsert { address, patch } => { + if let Some(change) = self + .apply_account_patch(*address, patch, true) + .expect("AccountUpsert never skips cold account patches") + { + diff.accounts.push(change); + } + } + StateUpdate::Purge { address, scope } => { + diff.purged.push(self.apply_purge(*address, scope)); + } + } + diff + } + + /// Apply a batch of [`StateUpdate`]s left-to-right, merging each per-update + /// [`StateDiff`]. + /// + /// Later updates observe the effect of earlier ones: two `Slot` writes to the + /// same key record `old → a` then `a → b`. Like + /// [`apply_update`](Self::apply_update) this is synchronous and infallible. + /// + /// # Performance — batched single-lock fast-path + /// + /// Consecutive `Slot`/`SlotDelta` writes are processed holding the backend + /// storage write-guard **once** for the run (the overlay map is lock-free), so + /// a bulk slot seed pays one lock acquisition instead of one read + one write + /// lock per slot. Apply order is preserved: when an `Account`/`BalanceDelta`/ + /// `Purge` update is reached the guard is dropped first (those take the + /// `accounts()` / `storage()` locks themselves — holding the storage + /// write-guard across them would deadlock the non-reentrant `RwLock`), the + /// update is processed via [`apply_update`](Self::apply_update), then the guard + /// is lazily re-acquired on the next slot run. The result is byte-identical to + /// folding [`apply_update`](Self::apply_update) over the batch. + /// + /// # Warning — relative updates can be skipped + /// + /// See [`apply_update`](Self::apply_update): a cold relative update is dropped, + /// not applied, and is invisible to [`StateDiff::is_empty`] / + /// [`StateDiff::len`]. After a batch with relative updates, check + /// [`StateDiff::has_skipped`]. + pub fn apply_updates(&mut self, updates: &[StateUpdate]) -> StateDiff { + let mut diff = StateDiff::default(); + let mut i = 0; + while i < updates.len() { + match &updates[i] { + // A run of consecutive slot writes: process them under a single + // held storage write-guard, then advance past the run. + StateUpdate::Slot { .. } | StateUpdate::SlotDelta { .. } => { + let run_end = updates[i..] + .iter() + .position(|u| { + !matches!(u, StateUpdate::Slot { .. } | StateUpdate::SlotDelta { .. }) + }) + .map(|off| i + off) + .unwrap_or(updates.len()); + self.apply_slot_run(&updates[i..run_end], &mut diff); + i = run_end; + } + // Account / BalanceDelta / Purge: no held guard (they take their + // own locks), so route through the single-update primitive. + _ => { + diff.merge(self.apply_update(&updates[i])); + i += 1; + } + } + } + diff + } + + /// Apply a run of consecutive `Slot`/`SlotDelta` updates under one held backend + /// storage write-guard (§16.9.2), merging each change into `diff`. + /// + /// The backend storage guard is acquired once for the whole run; overlay access + /// is lock-free (`self.db.cache.accounts`). The old-value read stays + /// `account_state`-aware (matching [`cached_storage_value`](Self::cached_storage_value)): + /// for an overlay account whose slot is absent, a `StorageCleared`/`NotExisting` + /// state reads ZERO and the backend is **not** consulted. Behavior is identical + /// to applying each update via [`apply_update`](Self::apply_update); the + /// `apply_updates_batched_equals_sequential` test pins this. + fn apply_slot_run(&mut self, run: &[StateUpdate], diff: &mut StateDiff) { + // Borrow the two layers as disjoint fields: the backend storage guard + // (layer 2) held for the whole run, and the overlay accounts map (layer 1, + // lock-free). Base invalidation is deferred until after the guard is + // dropped (it needs `&mut self`): collect the layer-2 addresses written + // here and mark them dirty below. + let mut dirtied: Vec

= Vec::new(); + let overlay = &mut self.db.cache.accounts; + let mut storage = self.blockchain_db.storage().write(); + + for update in run { + // Resolve `(address, slot, old, new)` for the write; a cold SlotDelta + // is skipped here (write nothing). `old` is the `account_state`-aware + // read (overlay ▸ cleared-as-ZERO ▸ backend), reused for both the write + // gate and the change record so each slot is read at most once. + let (address, slot, old, new) = match update { + StateUpdate::Slot { + address, + slot, + value, + } => { + let old = read_slot_account_state_aware(overlay, &storage, *address, *slot) + .unwrap_or(U256::ZERO); + (*address, *slot, old, *value) + } + StateUpdate::SlotDelta { + address, + slot, + delta, + } => match read_slot_account_state_aware(overlay, &storage, *address, *slot) { + // Hot: apply the saturating delta to the value already read. + Some(current) => (*address, *slot, current, delta.apply(current)), + // Cold: skip and surface (write nothing). + None => { + diff.skipped.push(SkippedDelta { + address: *address, + slot: *slot, + delta: *delta, + }); + continue; + } + }, + // The caller only ever hands this method slot updates. + _ => unreachable!("apply_slot_run only processes Slot/SlotDelta"), + }; + + write_slot_into(overlay, &mut storage, address, slot, new); + // Layer 2 was written for this address → it must be re-folded into the + // memoized base. Mirrors `write_slot_through`'s `mark_base_dirty`. + dirtied.push(address); + if old != new { + diff.slots.push(SlotChange { + address, + slot, + old, + new, + }); + } + } + + // Drop the storage write-guard before taking `&mut self` for invalidation. + drop(storage); + for address in dirtied { + self.mark_base_dirty(address); + } + } + + /// Write-through a single storage slot (§5.1). Returns a [`SlotChange`] iff + /// the slot's value actually changes. + fn apply_slot(&mut self, address: Address, slot: U256, value: U256) -> Option { + // Old value: overlay ▸ backend ▸ None (treated as ZERO). + let old = self + .cached_storage_value(address, slot) + .unwrap_or(U256::ZERO); + + self.write_slot_through(address, slot, value); + + // Record only an actual change. + (old != value).then_some(SlotChange { + address, + slot, + old, + new: value, + }) + } + + /// The single dual-layer slot write path (§5.1), shared by [`apply_slot`], + /// the [`StateUpdate::SlotDelta`] handler, and [`modify_slot`](Self::modify_slot). + /// + /// Backend (layer 2) is always written; the overlay (layer 1) is written only + /// if an overlay account already exists. A new overlay account is never + /// materialized: that preserves the layer-2-only invariant (a fresh + /// `StorageCleared` overlay account would read missing slots as ZERO and could + /// shadow later RPC reads), and an absent overlay entry falls through to the + /// backend on reads so the backend write is authoritative. + fn write_slot_through(&mut self, address: Address, slot: U256, value: U256) { + // Backend (layer 2): always write. { let mut storage = self.blockchain_db.storage().write(); - for &(addr, slot, value) in results { - storage.entry(addr).or_default().insert(slot, value); + storage.entry(address).or_default().insert(slot, value); + } + + // Overlay (layer 1): write only if an overlay account already exists. + if let Some(db_account) = self.db.cache.accounts.get_mut(&address) { + db_account.storage.insert(slot, value); + } + + // Layer 2 changed → invalidate the memoized base for this address (D2: + // over-invalidation when also shadowed by layer 1 is safe). + self.mark_base_dirty(address); + } + + /// Read-modify-write one storage slot through a caller-supplied transform. + /// + /// The general closure escape hatch behind [`StateUpdate::SlotDelta`] (the + /// data-level form flows through [`apply_update`](Self::apply_update); this is + /// for arbitrary transforms). `f` is called with the current cached value + /// (overlay ▸ backend ▸ `None` when the slot is cold) and decides the new + /// value: + /// + /// - `Some(new)` writes `new` through both layers (the same write path as + /// [`StateUpdate::Slot`]) and returns a [`SlotChange`] iff it changed + /// (`old.unwrap_or(ZERO) != new`); + /// - `None` writes nothing and returns `None`. + /// + /// The caller owns the cold/overflow policy. To skip cold slots (the + /// cold-aware read-modify-write rule), map through the `Option`: + /// `|cur| cur.map(|v| v.saturating_add(amount))` leaves a cold slot untouched. + /// To write an absolute value regardless, ignore the argument: `|_| Some(v)`. + /// + /// ```no_run + /// # use alloy_primitives::{Address, U256}; + /// # fn example(cache: &mut evm_fork_cache::cache::EvmCache) { + /// let token = Address::repeat_byte(0x01); + /// let slot = U256::from(0); + /// // Saturating +100, but only if the slot is already hot. + /// let change = cache.modify_slot(token, slot, |cur| cur.map(|v| v.saturating_add(U256::from(100)))); + /// # let _ = change; + /// # } + /// ``` + pub fn modify_slot( + &mut self, + address: Address, + slot: U256, + f: impl FnOnce(Option) -> Option, + ) -> Option { + let current = self.cached_storage_value(address, slot); + let new = f(current)?; + + self.write_slot_through(address, slot, new); + + let old = current.unwrap_or(U256::ZERO); + (old != new).then_some(SlotChange { + address, + slot, + old, + new, + }) + } + + /// Read-modify-write an account's native balance through a caller-supplied + /// transform. + /// + /// The closure analog of [`StateUpdate::BalanceDelta`] (the data-level form + /// flows through [`apply_update`](Self::apply_update); this is for arbitrary + /// transforms). `f` is called with the account's current native balance + /// (overlay ▸ backend ▸ `None` when the account is absent from **both** + /// layers) and decides the new balance: + /// + /// - `Some(new)` writes `new` through both layers — backend always, overlay + /// only if an overlay account already exists — preserving the account's + /// nonce and code, and returns an [`AccountChange`] (balance only) iff the + /// balance changed; + /// - `None` writes nothing (no account is materialized) and returns `None`. + /// + /// "Cold" for a balance is the account being absent from both layers — or + /// present in the overlay as revm `NotExisting` (absent to the EVM), which the + /// internal account read also treats as cold, mirroring `DbAccount::info()`. + /// To skip cold accounts, map through the `Option`: + /// `|cur| cur.map(|v| v.saturating_add(amount))`. + /// + /// ```no_run + /// # use alloy_primitives::{Address, U256}; + /// # fn example(cache: &mut evm_fork_cache::cache::EvmCache) { + /// let acct = Address::repeat_byte(0x01); + /// // Saturating +100, but only if the account's balance is already known. + /// let change = cache.modify_account_balance(acct, |cur| cur.map(|v| v.saturating_add(U256::from(100)))); + /// # let _ = change; + /// # } + /// ``` + pub fn modify_account_balance( + &mut self, + address: Address, + f: impl FnOnce(Option) -> Option, + ) -> Option { + // Load the full info from the cached layers only (overlay ▸ backend); the + // account is "cold" when absent from both. + let base = self.loaded_account_info(address); + let current_balance = base.as_ref().map(|info| info.balance); + let new_balance = f(current_balance)?; + + // The closure asked to write `new_balance`. Materialize from the loaded + // base (or a default if the caller chose to write a cold account). + let mut info = base.unwrap_or_default(); + let old_balance = info.balance; + info.balance = new_balance; + self.write_account_info_through(address, info); + + (old_balance != new_balance).then_some(AccountChange { + address, + balance: Some((old_balance, new_balance)), + nonce: None, + code_hash: None, + }) + } + + /// Apply a relative (saturating) [`SlotDelta`] to an account's native balance + /// (§16.5). Cold-aware: + /// + /// - `Ok(Some(change))` — present account, balance changed; + /// - `Ok(None)` — present account, balance unchanged (e.g. `Sub` from 0); + /// - `Err(skipped)` — cold account (absent from both layers): nothing applied, + /// nothing materialized. + fn apply_balance_delta( + &mut self, + address: Address, + delta: SlotDelta, + ) -> std::result::Result, SkippedBalanceDelta> { + let Some(mut info) = self.loaded_account_info(address) else { + // Cold: applying a delta against an unknown balance would corrupt it, + // and materializing a default account would mask the real on-chain one. + return Err(SkippedBalanceDelta { address, delta }); + }; + + let old_balance = info.balance; + let new_balance = delta.apply(old_balance); + info.balance = new_balance; + self.write_account_info_through(address, info); + + Ok((old_balance != new_balance).then_some(AccountChange { + address, + balance: Some((old_balance, new_balance)), + nonce: None, + code_hash: None, + })) + } + + /// Load an account's `AccountInfo` from the cached layers only (overlay ▸ + /// backend), without touching RPC. `None` when the account is absent from + /// both layers. + fn loaded_account_info(&self, address: Address) -> Option { + let mut info = if let Some(a) = self.db.cache.accounts.get(&address) { + // Mirror revm `DbAccount::info()` / `basic_ref`: a NotExisting overlay + // account is absent to the EVM (returns None) and does NOT fall through + // to the backend. Without this, a relative balance update / partial + // patch would compute against a stale `info` the EVM never sees. + if matches!(a.account_state, AccountState::NotExisting) { + return None; } + a.info.clone() + } else { + self.blockchain_db + .accounts() + .read() + .get(&address) + .cloned()? + }; + // Normalize like revm `insert_contract`: a ZERO code_hash denotes empty + // code -> KECCAK_EMPTY. Done at load time so a patch's `old_code_hash` + // matches what `write_account_info_through` stores (a self-consistent diff, + // no phantom/under-reported code_hash change). + if info.code_hash == B256::ZERO { + info.code_hash = revm::primitives::KECCAK_EMPTY; + } + Some(info) + } + + /// Write an `AccountInfo` through both layers, mirroring the slot policy: + /// backend (layer 2) always; overlay (layer 1) only if an overlay account + /// already exists (never materialize a new overlay account). + fn write_account_info_through(&mut self, address: Address, mut info: AccountInfo) { + // Normalize the code hash the way revm's `insert_contract` (applied on the + // overlay write below) does, so both layers store an identical hash: a ZERO + // code_hash denotes empty code → KECCAK_EMPTY. Otherwise the overlay would + // hold KECCAK_EMPTY while the backend kept ZERO for the same account. + if info.code_hash == B256::ZERO { + info.code_hash = revm::primitives::KECCAK_EMPTY; } - // Write through to the overlay only for accounts already materialized - // there, so the winning layer reflects the fresh value. - for &(addr, slot, value) in results { - if let Some(db_account) = self.db.cache.accounts.get_mut(&addr) { - db_account.storage.insert(slot, value); + let overlay_present = self.db.cache.accounts.contains_key(&address); + { + let mut accounts = self.blockchain_db.accounts().write(); + accounts.insert(address, info.clone()); + } + if overlay_present { + self.db.insert_account_info(address, info); + } + // Layer-2 account info changed → invalidate the memoized base for this + // address (D2: over-invalidation when also in layer 1 is safe). + self.mark_base_dirty(address); + } + + /// Apply a partial [`AccountPatch`] write-through (§5.2). Returns an + /// [`AccountChange`] iff any field actually changes. + fn apply_account_patch( + &mut self, + address: Address, + patch: &AccountPatch, + allow_cold_upsert: bool, + ) -> std::result::Result, SkippedAccountPatch> { + // 1. Current info from the cached layers only (overlay ▸ backend). No RPC: + // apply is a write, not a fetch. A partial patch on a cold account is + // skipped unless the caller explicitly chose AccountUpsert. + let mut info = match self.loaded_account_info(address) { + Some(info) => info, + None if account_patch_is_empty(patch) => return Ok(None), + None if allow_cold_upsert => AccountInfo::default(), + None => { + return Err(SkippedAccountPatch { + address, + patch: patch.clone(), + }); + } + }; + + let old_balance = info.balance; + let old_nonce = info.nonce; + let old_code_hash = info.code_hash; + + // 2. Apply each `Some` field. + if let Some(balance) = patch.balance { + info.balance = balance; + } + if let Some(nonce) = patch.nonce { + info.nonce = nonce; + } + if let Some(code) = &patch.code { + let bytecode = Bytecode::new_raw(code.clone()); + info.code_hash = bytecode.hash_slow(); + info.code = Some(bytecode); + } + + // 3. Compute the change first. A no-op patch (every field equals the + // loaded base) must NOT write either layer — otherwise an all-`None` + // patch on an absent address would insert `AccountInfo::default()` into + // the shared backend (masking a future RPC fetch) while returning an + // empty diff. Only a real field change materializes anything. + let change = AccountChange { + address, + balance: (old_balance != info.balance).then_some((old_balance, info.balance)), + nonce: (old_nonce != info.nonce).then_some((old_nonce, info.nonce)), + code_hash: (old_code_hash != info.code_hash).then_some((old_code_hash, info.code_hash)), + }; + if change.balance.is_none() && change.nonce.is_none() && change.code_hash.is_none() { + return Ok(None); + } + + // 4. Write-through, mirroring the slot policy: backend always; overlay + // only if an overlay account already exists (do not materialize one). + self.write_account_info_through(address, info); + + Ok(Some(change)) + } + + /// Dispatch a [`PurgeScope`] to the matching layer logic (§5.3), returning a + /// [`PurgeRecord`] of what was removed from each layer. + fn apply_purge(&mut self, address: Address, scope: &PurgeScope) -> PurgeRecord { + match scope { + PurgeScope::Account => { + let (slots_removed, account_removed) = self.purge_account_inner(address); + PurgeRecord { + address, + scope: PurgeScope::Account, + slots_removed, + account_removed, + } + } + PurgeScope::AllStorage => { + let slots_removed = self.purge_pool_storage_inner(address); + PurgeRecord { + address, + scope: PurgeScope::AllStorage, + slots_removed, + account_removed: false, + } + } + PurgeScope::Slots(slots) => { + let slots_removed = self.purge_pool_slots_inner(address, slots); + PurgeRecord { + address, + scope: PurgeScope::Slots(slots.clone()), + slots_removed, + account_removed: false, + } } } } @@ -1126,14 +2016,33 @@ impl EvmCache { /// Return the currently-cached value for a storage slot, if any. /// - /// Checks the CacheDB overlay (layer 1) first, then the BlockchainDb backend - /// (layer 2). Returns `None` when neither layer has seen the slot. Unlike - /// [`read_storage_slot`](Self::read_storage_slot) this never touches RPC. + /// Mirrors what the EVM would `SLOAD` from the cached layers (it never touches + /// RPC, unlike [`read_storage_slot`](Self::read_storage_slot)): + /// + /// 1. The CacheDB overlay (layer 1) wins: if the overlay account holds the + /// slot, return it. + /// 2. Match revm's `CacheDB::storage_ref`: if the overlay account exists but + /// does **not** hold the slot, and its `account_state` is `StorageCleared` + /// or `NotExisting`, the live EVM reads the slot as ZERO and never consults + /// the backend — so return `Some(U256::ZERO)`, **not** the (shadowed) + /// backend value. Returning the backend value here would let a + /// `SlotDelta`/`modify_slot` compute a delta against a base the EVM never + /// sees (silent corruption) and would mis-record `apply_slot`'s `old`. + /// 3. Otherwise fall through to the BlockchainDb backend (layer 2); `None` when + /// neither layer has seen the slot. pub fn cached_storage_value(&self, address: Address, slot: U256) -> Option { - if let Some(db_account) = self.db.cache.accounts.get(&address) - && let Some(value) = db_account.storage.get(&slot) - { - return Some(*value); + if let Some(db_account) = self.db.cache.accounts.get(&address) { + if let Some(value) = db_account.storage.get(&slot) { + return Some(*value); + } + // A StorageCleared / NotExisting overlay account reads a missing slot + // as ZERO and never consults the backend (matching the EVM SLOAD). + if matches!( + db_account.account_state, + AccountState::StorageCleared | AccountState::NotExisting + ) { + return Some(U256::ZERO); + } } let storage = self.blockchain_db.storage().read(); storage.get(&address).and_then(|s| s.get(&slot).copied()) @@ -1153,8 +2062,20 @@ impl EvmCache { /// none is available. This is the synchronous main-thread primitive; the /// background validator performs the equivalent comparison against a snapshot. pub fn verify_slots(&mut self, slots: &[(Address, U256)]) -> Result> { + Ok(self.verify_slots_inner(slots)?.0) + } + + /// Shared implementation for [`verify_slots`](Self::verify_slots) and the + /// pipeline's reconcile path. Returns `(changed, fetched_ok)` where + /// `fetched_ok` is the number of requested slots the fetcher returned a value + /// for (failed per-slot fetches are skipped, not errors). Errors only when no + /// batch fetcher is configured. + fn verify_slots_inner( + &mut self, + slots: &[(Address, U256)], + ) -> Result<(Vec, usize)> { if slots.is_empty() { - return Ok(Vec::new()); + return Ok((Vec::new(), 0)); } let fetcher = self .storage_batch_fetcher @@ -1173,6 +2094,7 @@ impl EvmCache { let mut changed = Vec::new(); let mut to_inject = Vec::new(); + let mut fetched_ok = 0usize; for (addr, slot, fetched) in results { let fresh = match fetched { Ok(value) => value, @@ -1181,6 +2103,7 @@ impl EvmCache { continue; } }; + fetched_ok += 1; // A slot the cache never saw is treated as old = ZERO (the value a // sim would have read), so a non-zero fresh value counts as a change. let old = cached @@ -1202,6 +2125,29 @@ impl EvmCache { if !to_inject.is_empty() { self.inject_storage_batch_fresh(&to_inject); } + Ok((changed, fetched_ok)) + } + + /// Reconciliation re-read used by [`EventPipeline::reconcile`](crate::events::EventPipeline::reconcile). + /// + /// Like [`verify_slots`](Self::verify_slots) it fetches the requested slots, + /// injects the ones that changed, and returns the changed set — but it is + /// **honest about reachability**: it errors not only when no batch fetcher is + /// configured, but also when a non-empty request could not fetch **any** slot + /// (a total fetch failure — e.g. the default RPC fetcher invoked with no usable + /// runtime, or an unreachable provider). Reconciliation that silently "verified + /// nothing" would be a false all-clear, so it surfaces as an error for the + /// caller to retry. A partially-successful fetch returns `Ok` with whatever + /// changed. + pub fn reconcile_slots(&mut self, slots: &[(Address, U256)]) -> Result> { + let (changed, fetched_ok) = self.verify_slots_inner(slots)?; + if !slots.is_empty() && fetched_ok == 0 { + return Err(anyhow!( + "reconcile could not fetch any of the {} requested slot(s) \ + (no usable storage fetcher / provider unreachable)", + slots.len() + )); + } Ok(changed) } @@ -1215,6 +2161,16 @@ impl EvmCache { /// use it when an address is fully volatile (no pinned slots) and even its /// balance/nonce/code can no longer be trusted. pub fn purge_account(&mut self, addr: Address) { + // Thin wrapper over the unified purge primitive; the layer logic lives in + // `purge_account_inner` (shared with `apply_update(Purge { Account })`). + let _ = self.apply_update(&StateUpdate::purge(addr, PurgeScope::Account)); + } + + /// Account-scope purge layer logic. Removes `addr` from the overlay accounts + /// map, the backend accounts map, and the backend storage map. Returns + /// `(backend_slots_removed, account_removed)` where `account_removed` is true + /// if an account entry was removed from either account layer. + fn purge_account_inner(&mut self, addr: Address) -> (usize, bool) { // Layer 1: CacheDB overlay (accounts + their storage live together). let overlay_removed = self.db.cache.accounts.remove(&addr).is_some(); @@ -1225,17 +2181,24 @@ impl EvmCache { .write() .remove(&addr) .is_some(); - let backend_storage_removed = self.blockchain_db.storage().write().remove(&addr).is_some(); + let backend_storage_removed = self.blockchain_db.storage().write().remove(&addr); + let slots_removed = backend_storage_removed + .map(|slots| slots.len()) + .unwrap_or(0); - if overlay_removed || backend_account_removed || backend_storage_removed { + let account_removed = overlay_removed || backend_account_removed; + if account_removed || slots_removed > 0 { debug!( account = %addr, overlay_removed, backend_account_removed, - backend_storage_removed, + backend_storage_slots = slots_removed, "purged account from both cache layers" ); } + // Layer 2 (account + storage) changed for this address → invalidate base. + self.mark_base_dirty(addr); + (slots_removed, account_removed) } /// Get the chain ID used for EVM simulations (the `CHAINID` opcode). @@ -1282,24 +2245,304 @@ impl EvmCache { } } - /// Create an immutable snapshot of the current EVM state for cross-thread - /// fan-out. + /// Create an immutable, `Send + Sync` snapshot of the current EVM state for + /// cross-thread fan-out (the copy-on-write two-tier view, Pillar A). /// - /// Merges both layers (CacheDB overlay + BlockchainDb backend) into a - /// single flat HashMap. The snapshot is `Send + Sync` and can be shared - /// across threads via `Arc`. + /// Rather than deep-copying both layers, this memoizes the cold layer-2 + /// (`BlockchainDb`) index as an `Arc`-shared base — reused as a cheap + /// `Arc::clone` when layer 2 is unchanged, rebuilt copy-on-write only for the + /// addresses that changed — and folds the hot layer-1 (`CacheDB` overlay) + /// delta over it. Layer-1 values shadow the base on reads, reproducing the + /// live cache's layered semantics; the resulting [`EvmSnapshot`] is shared + /// across threads via `Arc`. Its cost tracks *changed* state, not *total* + /// state. (The retained [`create_snapshot_deep_clone`](Self::create_snapshot_deep_clone) + /// is the read-equivalent O(total) reference, kept for benchmarking/testing.) /// - /// CacheDB overlay values take precedence over BlockchainDb values. - /// Use with [`EvmOverlay`] for parallel simulation. - /// - /// For cheap same-thread save/restore of just the overlay, prefer + /// Takes `&mut self` because it refreshes and memoizes the base. For cheap + /// same-thread save/restore of just the overlay, prefer /// [`snapshot`](Self::snapshot) / [`restore`](Self::restore) instead. - pub fn create_snapshot(&self) -> Arc { + pub fn create_snapshot(&mut self) -> Arc { + // 1. Refresh / memoize the cold layer-2 base, then take a cheap Arc handle + // (O(1) when layer 2 is unchanged since the last snapshot). + self.refresh_base(); + let base = Arc::clone(self.base.as_ref().expect("refresh_base sets base")); + + // 2. Fold layer 1 (the hot CacheDB overlay) into the snapshot's overlay + // maps + cleared/not-existing sets, applying the same classification as + // the legacy flatten (O(layer-1)). + let mut overlay_accounts = HashMap::new(); + let mut overlay_storage = HashMap::new(); + let mut overlay_code_by_hash = HashMap::new(); + let mut storage_cleared = std::collections::HashSet::new(); + let mut accounts_not_existing = std::collections::HashSet::new(); + for (addr, db_account) in &self.db.cache.accounts { + let not_existing = matches!(db_account.account_state, AccountState::NotExisting); + let cleared = + not_existing || matches!(db_account.account_state, AccountState::StorageCleared); + + // Account info. Mirror revm `DbAccount::info()` / `loaded_account_info`: + // a NotExisting overlay account is absent to the EVM (`basic` returns + // None), so it must NOT contribute info/code to the overlay — and + // `accounts_not_existing` makes the read short-circuit to None before + // ever consulting the base. + if not_existing { + accounts_not_existing.insert(*addr); + } else { + if let Some(code) = &db_account.info.code { + overlay_code_by_hash.insert(db_account.info.code_hash, code.clone()); + } + overlay_accounts.insert(*addr, db_account.info.clone()); + } + + // Storage. A StorageCleared/NotExisting account's storage is locally + // complete: the overlay holds ONLY its own slots (so a cleared account + // ALWAYS gets an `overlay_storage` entry, possibly empty), an absent + // slot reads ZERO via `storage_cleared`, and the base is never consulted + // for it. A non-cleared overlay account contributes its slots; absent + // slots fall through to the base on a read. + if cleared { + storage_cleared.insert(*addr); + let account_storage: HashMap = + db_account.storage.iter().map(|(k, v)| (*k, *v)).collect(); + overlay_storage.insert(*addr, account_storage); + } else if !db_account.storage.is_empty() { + let account_storage = overlay_storage.entry(*addr).or_default(); + for (slot, value) in &db_account.storage { + account_storage.insert(*slot, *value); + } + } + } + + Arc::new(snapshot::EvmSnapshot { + base, + overlay_accounts, + overlay_storage, + overlay_code_by_hash, + storage_cleared, + accounts_not_existing, + block_hashes: HashMap::new(), + block_number: self.block_number, + basefee: self.basefee, + coinbase: self.coinbase, + prevrandao: self.prevrandao, + gas_limit: self.block_gas_limit, + chain_id: self.chain_id, + timestamp: self.timestamp_override, + spec_id: self.spec_id, + shared_memory_capacity: self.shared_memory_capacity, + }) + } + + /// Force the next [`create_snapshot`](Self::create_snapshot) to rebuild the + /// memoized copy-on-write base from scratch (Pillar A). + /// + /// The crate's own mutators keep the base honest automatically. This is the + /// **escape-hatch re-honest hook**: call it after writing layer 2 directly + /// through [`unchecked_blockchain_db`](Self::unchecked_blockchain_db) or + /// [`unchecked_backend`](Self::unchecked_backend) — those bypass the write + /// funnel, and in-place changes at unchanged cardinality are invisible to the + /// snapshot growth scan. + /// That includes overwriting an existing storage slot and changing an existing + /// account's info/code/balance without adding a new account. Lazy RPC-populated + /// data does not need this call because it only appends accounts/slots, which + /// the growth scan catches. + /// + /// When using `SharedBackend::insert_or_update_*` through + /// [`unchecked_backend`](Self::unchecked_backend), remember those helpers only + /// enqueue a background update. Synchronize/read back the update through + /// `SharedBackend` before the next snapshot; `invalidate_snapshot_base` alone + /// is not a backend-handler synchronization point. Once the direct write is + /// present, calling this before the next snapshot guarantees it reflects that + /// write rather than a stale memoized value. Over-invalidation is always safe + /// (Decision D2); the only cost is one full base rebuild on the next snapshot. + pub fn invalidate_snapshot_base(&mut self) { + self.invalidate_base(); + } + + /// Refresh the memoized cold layer-2 [`BaseState`](snapshot::BaseState), + /// reusing the previous `Arc` wherever layer 2 is unchanged (Pillar A). + /// + /// Called at the top of [`create_snapshot`](Self::create_snapshot). It never + /// mutates an `Arc` that may already be shared with a live + /// snapshot: on any change it builds a *new* `BaseState` that shares the `Arc` + /// handles of unchanged accounts and rebuilds only the changed ones + /// (copy-on-write). + /// + /// Algorithm (see `docs/phase-5-spec.md` §2.3): + /// 1. **Full rebuild** when there is no base yet or `base_full_rebuild` is set + /// (`set_block` / re-pin replaced layer 2): flatten all of layer 2. + /// 2. **Detect uncontrolled growth**: a lazy RPC fetch / prefetch can write + /// layer 2 from inside `foundry-fork-db`, bypassing our write funnel. An + /// `O(accounts)` length-scan over the current layer-2 storage/accounts marks + /// any address whose slot count differs from the recorded length, or any + /// account absent from the base, as dirty. + /// 3. **Nothing dirty** → reuse the existing `Arc` unchanged (the + /// common hot-loop case; the base side of `create_snapshot` is then O(1)). + /// 4. **Some addresses dirty** → build a new `BaseState` sharing the `Arc`s of + /// unchanged accounts and rebuilding only the dirty ones. + fn refresh_base(&mut self) { + // Case 1: full rebuild. + if self.base.is_none() || self.base_full_rebuild { + self.base = Some(Arc::new(self.build_base_full())); + self.base_dirty.clear(); + self.base_full_rebuild = false; + return; + } + + // Case 2: detect uncontrolled layer-2 growth via an O(accounts) length scan + // (NOT an O(slots) value scan). Any address whose slot count changed, or any + // account that newly appeared in layer 2, is folded into `base_dirty`. + // + // LOAD-BEARING INVARIANT: the count/absence scan is sufficient *only* because + // the one uncontrolled layer-2 writer — the foundry-fork-db `SharedBackend` + // lazy fetch — is append-only at a fixed block (its request handler answers an + // already-cached account/slot from the store and only inserts on a miss; it + // never overwrites an existing entry in place). So an uncontrolled fetch can + // only add a new account (caught by the absence check) or a new slot (caught + // by the count check). An in-place value overwrite at unchanged length is + // invisible here; the controlled writers therefore call `mark_base_dirty` + // explicitly, and a direct out-of-band write via `unchecked_blockchain_db()`/`unchecked_backend()` + // must call `invalidate_snapshot_base`. If a future foundry-fork-db bump makes + // the lazy path overwrite-in-place, this scan must gain a value/version check. + { + let db_storage = self.blockchain_db.storage().read(); + for (addr, slots) in db_storage.iter() { + if self.base_storage_lens.get(addr).copied() != Some(slots.len()) { + self.base_dirty.insert(*addr); + } + } + let db_accounts = self.blockchain_db.accounts().read(); + let base = self.base.as_ref().expect("base present in case 2/3/4"); + for addr in db_accounts.keys() { + if !base.accounts.contains_key(addr) { + self.base_dirty.insert(*addr); + } + } + } + + // Case 3: nothing changed → reuse the existing Arc unchanged. + if self.base_dirty.is_empty() { + return; + } + + // Case 4: rebuild copy-on-write — clone the outer maps (Arc handles + + // AccountInfo, no per-slot copy) and rebuild only the dirty addresses. + let prev = self.base.as_ref().expect("base present in case 4"); + let mut accounts = prev.accounts.clone(); + let mut storage = prev.storage.clone(); + + let db_accounts = self.blockchain_db.accounts().read(); + let db_storage = self.blockchain_db.storage().read(); + for addr in self.base_dirty.iter().copied() { + // Account info: refresh from the current layer-2 account, or drop it if + // the account no longer exists in layer 2 (e.g. after a purge). + match db_accounts.get(&addr) { + Some(info) => { + accounts.insert(addr, info.clone()); + } + None => { + accounts.remove(&addr); + } + } + + // Storage: rebuild this account's Arc from the current layer-2 + // storage, or drop it if the account has no layer-2 storage anymore. + match db_storage.get(&addr) { + Some(slots) => { + let rebuilt: HashMap = + slots.iter().map(|(k, v)| (*k, *v)).collect(); + self.base_storage_lens.insert(addr, rebuilt.len()); + storage.insert(addr, Arc::new(rebuilt)); + } + None => { + storage.remove(&addr); + self.base_storage_lens.remove(&addr); + } + } + } + drop(db_accounts); + drop(db_storage); + + // Rebuild the code index from the refreshed accounts (NOT cloned from the + // previous base): a purged or recoded dirty account must not leave a stale + // `code_by_hash` entry, which would diverge from `create_snapshot_deep_clone` + // on a direct `code_by_hash(old_hash)` lookup. Rebuilding from scratch also + // handles shared code hashes correctly (a hash survives iff some present + // account still carries it). + let code_by_hash = Self::code_index(&accounts); + + self.base = Some(Arc::new(snapshot::BaseState { + accounts, + storage, + code_by_hash, + })); + self.base_dirty.clear(); + } + + /// Build the bytecode-by-hash index from a set of (layer-2) accounts, matching + /// the deep-clone reference: a hash is present iff some account carries that + /// code inline. Rebuilt from scratch on every base (re)build so a purged or + /// recoded account never leaves a stale entry — preserving read-equivalence + /// with [`create_snapshot_deep_clone`](Self::create_snapshot_deep_clone). + fn code_index(accounts: &HashMap) -> HashMap { + accounts + .values() + .filter_map(|info| { + info.code + .as_ref() + .map(|code| (info.code_hash, code.clone())) + }) + .collect() + } + + /// Build a fresh [`BaseState`](snapshot::BaseState) by flattening all of layer + /// 2, recording `base_storage_lens`. Shared by `refresh_base`'s full-rebuild + /// path and [`create_snapshot_deep_clone`](Self::create_snapshot_deep_clone). + fn build_base_full(&mut self) -> snapshot::BaseState { let mut accounts = HashMap::new(); + { + let db_accounts = self.blockchain_db.accounts().read(); + for (addr, info) in db_accounts.iter() { + accounts.insert(*addr, info.clone()); + } + } + let code_by_hash = Self::code_index(&accounts); let mut storage = HashMap::new(); + self.base_storage_lens.clear(); + { + let db_storage = self.blockchain_db.storage().read(); + for (addr, slots) in db_storage.iter() { + let converted: HashMap = slots.iter().map(|(k, v)| (*k, *v)).collect(); + self.base_storage_lens.insert(*addr, converted.len()); + storage.insert(*addr, Arc::new(converted)); + } + } + snapshot::BaseState { + accounts, + storage, + code_by_hash, + } + } + + /// The retained deep-clone snapshot — today's full flatten, kept reachable for + /// A/B benchmarking and as the read-equivalence reference (Decision D3). + /// + /// Produces the same two-tier [`EvmSnapshot`](snapshot::EvmSnapshot) shape as + /// [`create_snapshot`](Self::create_snapshot), but with `base` set to the + /// fully-merged flatten of **both** layers and **empty** overlay maps (the + /// cleared / not-existing sets still in place). It is read-indistinguishable + /// from `create_snapshot` by construction (the `tests/cow_snapshot.rs` + /// differential gate pins this), at the cost of an O(total state) deep copy + /// every call — exactly the cost `create_snapshot` now amortizes away. + /// + /// Stays `&self`: it does not touch the memoized base. + #[doc(hidden)] + pub fn create_snapshot_deep_clone(&self) -> Arc { + let mut accounts = HashMap::new(); + let mut storage: HashMap> = HashMap::new(); let mut code_by_hash = HashMap::new(); - // 1. Load from BlockchainDb (persistent cache / Layer 2) + // 1. Load from BlockchainDb (persistent cache / Layer 2). { let db_accounts = self.blockchain_db.accounts().read(); for (addr, info) in db_accounts.iter() { @@ -1312,29 +2555,70 @@ impl EvmCache { { let db_storage = self.blockchain_db.storage().read(); for (addr, slots) in db_storage.iter() { - // Convert from DefaultHashBuilder to RandomState HashMap let converted: HashMap = slots.iter().map(|(k, v)| (*k, *v)).collect(); storage.insert(*addr, converted); } } - // 2. Overlay from CacheDB (Layer 1, takes precedence) + // 2. Overlay from CacheDB (Layer 1, takes precedence). Merge into the same + // flat maps, dropping shadowed entries, exactly as the original + // `create_snapshot` did. A cleared account's storage is routed into + // `overlay_storage` (not the base), because `EvmSnapshot::storage_value` + // only applies the cleared-as-ZERO rule for an address with an + // `overlay_storage` entry — so the cleared semantics must be expressed + // there for both snapshot constructors to read identically. + let mut overlay_storage: HashMap> = HashMap::new(); + let mut storage_cleared = std::collections::HashSet::new(); + let mut accounts_not_existing = std::collections::HashSet::new(); for (addr, db_account) in &self.db.cache.accounts { - if let Some(code) = &db_account.info.code { - code_by_hash.insert(db_account.info.code_hash, code.clone()); + let not_existing = matches!(db_account.account_state, AccountState::NotExisting); + let cleared = + not_existing || matches!(db_account.account_state, AccountState::StorageCleared); + + if not_existing { + accounts_not_existing.insert(*addr); + accounts.remove(addr); + } else { + if let Some(code) = &db_account.info.code { + code_by_hash.insert(db_account.info.code_hash, code.clone()); + } + accounts.insert(*addr, db_account.info.clone()); } - accounts.insert(*addr, db_account.info.clone()); - let account_storage = storage.entry(*addr).or_default(); - for (slot, value) in &db_account.storage { - account_storage.insert(*slot, *value); + + if cleared { + // Cleared: storage is locally complete. Drop any shadowed base + // slots and keep ONLY the overlay slots, in `overlay_storage`. + storage_cleared.insert(*addr); + storage.remove(addr); + let account_storage: HashMap = + db_account.storage.iter().map(|(k, v)| (*k, *v)).collect(); + overlay_storage.insert(*addr, account_storage); + } else { + // Non-cleared: overlay slots win over base; fold them into base. + let account_storage = storage.entry(*addr).or_default(); + for (slot, value) in &db_account.storage { + account_storage.insert(*slot, *value); + } } } - Arc::new(snapshot::EvmSnapshot { + let base = snapshot::BaseState { accounts, - storage, - block_hashes: HashMap::new(), + storage: storage + .into_iter() + .map(|(addr, slots)| (addr, Arc::new(slots))) + .collect(), code_by_hash, + }; + + Arc::new(snapshot::EvmSnapshot { + base: Arc::new(base), + overlay_accounts: HashMap::new(), + overlay_storage, + overlay_code_by_hash: HashMap::new(), + storage_cleared, + accounts_not_existing, + block_hashes: HashMap::new(), block_number: self.block_number, basefee: self.basefee, coinbase: self.coinbase, @@ -1343,9 +2627,32 @@ impl EvmCache { chain_id: self.chain_id, timestamp: self.timestamp_override, spec_id: self.spec_id, + shared_memory_capacity: self.shared_memory_capacity, }) } + /// Mark a layer-2 address dirty so the next [`refresh_base`](Self::refresh_base) + /// re-folds it into the memoized base (Pillar A invalidation; see + /// `docs/phase-5-spec.md` §3). + /// + /// Called from every site that can change a layer-2 value a snapshot read + /// would surface (write-through, batch injects, layer-2 seeding, purges). + /// Over-invalidation is safe (Decision D2): marking an address that is also + /// shadowed by layer 1 just re-folds that one account. + fn mark_base_dirty(&mut self, address: Address) { + self.base_dirty.insert(address); + } + + /// Force a full rebuild of the memoized base on the next + /// [`refresh_base`](Self::refresh_base) (Pillar A invalidation). + /// + /// Used by layer-2 changes too broad to enumerate per-address efficiently + /// (multi-contract / full-storage purges, block re-pins). Coarser than + /// [`mark_base_dirty`](Self::mark_base_dirty) but always correct. + fn invalidate_base(&mut self) { + self.base_full_rebuild = true; + } + /// Update the block that RPC fetches are pinned to. /// /// This re-pins the SharedBackend and the batch storage fetcher to `block`, @@ -1355,30 +2662,41 @@ impl EvmCache { /// To prevent the EVM block context from silently diverging from the pinned /// block, when `block` is a concrete `BlockId::Number(Number(n))` this also /// updates `block_number` (the `NUMBER` opcode) to `n`. For tag-based block - /// ids (`latest`, `pending`, hashes, etc.) the height is not statically known, - /// so `block_number` is left unchanged. - /// - /// `basefee` (the `BASEFEE` opcode) is **not** refreshed here because deriving - /// it requires fetching the block header, which this synchronous method cannot - /// do. Callers that change blocks should refresh it via - /// [`set_block_context`](Self::set_block_context) (e.g. after fetching the new - /// header). Prefer [`repin_to_block`](Self::repin_to_block) when re-pinning to + /// ids (`latest`, `pending`, hashes, etc.) and `None`, the height is not + /// statically known, so `block_number` is cleared. + /// + /// `basefee` (the `BASEFEE` opcode) is **cleared on every block change** and + /// on every non-concrete tag/hash/`None` pin call because deriving it requires + /// fetching the block header, which this synchronous method cannot do. Callers + /// that change blocks should refresh it via + /// [`set_block_context`](Self::set_block_context) after fetching the new + /// header. Prefer [`repin_to_block`](Self::repin_to_block) when re-pinning to /// a concrete height, since it keeps `block_number` and the pinned block in /// lockstep. pub fn set_block(&mut self, block: Option) { - if self.block != block { + let changed = self.block != block; + let concrete_number = match block { + Some(BlockId::Number(BlockNumberOrTag::Number(n))) => Some(n), + _ => None, + }; + if changed { self.block = block; + // Re-pinning replaces layer 2 wholesale (state at a new block): the + // memoized base must be rebuilt from scratch on the next snapshot. + self.invalidate_base(); if let Some(block_id) = block { let _ = self.backend.set_pinned_block(block_id); *self.batch_block_id.lock().unwrap() = block_id; - // Keep the EVM `NUMBER` opcode aligned with the pinned block so the - // two cannot silently diverge. Only a concrete height is meaningful; - // tags (latest/pending/hash) leave `block_number` untouched. - if let BlockId::Number(BlockNumberOrTag::Number(n)) = block_id { - self.block_number = Some(n); - } } } + if changed || concrete_number.is_none() { + self.basefee = None; + } + + // Keep the EVM `NUMBER` opcode aligned with the pin. Only a concrete + // height is meaningful; tags, hashes, and no explicit pin clear it so a + // stale number from an earlier concrete block cannot leak into simulation. + self.block_number = concrete_number; } /// Get the block that RPC fetches are currently pinned to. @@ -1410,9 +2728,10 @@ impl EvmCache { /// Get the block number used for EVM simulations (the `NUMBER` opcode). /// - /// Fetched from the pinned block's header at construction and kept in - /// lockstep with the pin by [`set_block`](Self::set_block) / - /// [`repin_to_block`](Self::repin_to_block). `None` means revm falls back + /// Fetched from the pinned block's header at construction. Concrete-number + /// pins set it via [`set_block`](Self::set_block) / + /// [`repin_to_block`](Self::repin_to_block); tag/hash/`None` pins clear it + /// because their height is not statically known. `None` means revm falls back /// to `0`, which can steer contracts that branch on `block.number` down a /// different code path. Override directly via /// [`set_block_context`](Self::set_block_context). @@ -1423,10 +2742,12 @@ impl EvmCache { /// Get the base fee per gas used for EVM simulations (the `BASEFEE` opcode). /// /// Fetched from the pinned block's header at construction. `None` means - /// revm falls back to `0`. Unlike `block_number` this is **not** refreshed - /// by [`set_block`](Self::set_block); refresh it with - /// [`set_block_context`](Self::set_block_context) after fetching a new - /// header if `BASEFEE` accuracy matters. + /// revm falls back to `0`. This is cleared by [`set_block`](Self::set_block) + /// / [`repin_to_block`](Self::repin_to_block) when the pin changes, and by + /// non-concrete tag/hash/`None` pin calls because those can drift without a + /// concrete number in the API. Refresh it with + /// [`set_block_context`](Self::set_block_context) after fetching a new header + /// if `BASEFEE` accuracy matters. pub fn basefee(&self) -> Option { self.basefee } @@ -1471,16 +2792,12 @@ impl EvmCache { /// /// Updates the SharedBackend pinned block, the batch fetcher block, and the /// EVM block context (`NUMBER` opcode) in lockstep. The current `basefee` is - /// preserved; callers should refresh it via - /// [`set_block_context`](Self::set_block_context) after fetching the new + /// cleared because it cannot be refreshed synchronously; callers should set it + /// via [`set_block_context`](Self::set_block_context) after fetching the new /// block header if `BASEFEE` accuracy matters. pub fn repin_to_block(&mut self, block_number: u64) { let old_block = self.block; - // `set_block` already updates `block_number` for a concrete height; the - // explicit `set_block_context` below preserves `basefee` and keeps the - // re-pin atomic and self-documenting. self.set_block(Some(BlockId::Number(block_number.into()))); - self.set_block_context(Some(block_number), self.basefee); if let Some(BlockId::Number(BlockNumberOrTag::Number(old_num))) = old_block { let drift = block_number.saturating_sub(old_num); @@ -1669,6 +2986,13 @@ impl EvmCache { /// # Arguments /// * `pool_address` - The UniswapV2 pair contract address /// * `metadata` - The cached pool metadata containing token0 and token1 + /// + /// # Layering (Phase 3 change) + /// As of Phase 3 this writes **through** the dual-layer policy via + /// [`apply_updates`](Self::apply_updates) (backend always, overlay-if-present) + /// rather than the old overlay-only write. The slot *placement* is normalized; + /// the visible `token0()` / `token1()` reads are unchanged. The slot writes are + /// now infallible; the `Result` is retained for signature compatibility. #[cfg(feature = "protocols")] #[cfg_attr(docsrs, doc(cfg(feature = "protocols")))] pub fn inject_v2_pool_metadata( @@ -1683,10 +3007,10 @@ impl EvmCache { let token0_value = U256::from_be_slice(metadata.token0.as_slice()); let token1_value = U256::from_be_slice(metadata.token1.as_slice()); - self.db - .insert_account_storage(pool_address, TOKEN0_SLOT, token0_value)?; - self.db - .insert_account_storage(pool_address, TOKEN1_SLOT, token1_value)?; + self.apply_updates(&[ + StateUpdate::slot(pool_address, TOKEN0_SLOT, token0_value), + StateUpdate::slot(pool_address, TOKEN1_SLOT, token1_value), + ]); Ok(()) } @@ -1703,6 +3027,14 @@ impl EvmCache { /// # Arguments /// * `pool_address` - The UniswapV3 pool contract address /// * `tick_bitmap` - Map of word position (int16) to bitmap value (uint256) + /// + /// # Layering (Phase 3 change) + /// As of Phase 3 this writes **through** the dual-layer policy via + /// [`apply_updates`](Self::apply_updates) (backend always, overlay-if-present) + /// rather than the old overlay-only write — so the slots now land in the + /// BlockchainDb backend (layer 2) too. See `CHANGELOG.md` / `KNOWN_ISSUES.md`. + /// The slot writes are now infallible; the `Result` is retained for signature + /// compatibility. #[cfg(feature = "protocols")] #[cfg_attr(docsrs, doc(cfg(feature = "protocols")))] pub fn inject_v3_tick_bitmap( @@ -1724,17 +3056,17 @@ impl EvmCache { tick_bitmap: &std::collections::HashMap, base_slot: U256, ) -> Result { - let mut injected = 0; + let mut updates = Vec::with_capacity(tick_bitmap.len()); for (&word_position, &bitmap_value) in tick_bitmap { let word_position_i256 = i256_from_i16(word_position); let mut slot_preimage = [0u8; 64]; slot_preimage[..32].copy_from_slice(&word_position_i256); slot_preimage[32..64].copy_from_slice(&base_slot.to_be_bytes::<32>()); let storage_slot: U256 = keccak256(slot_preimage).into(); - self.db - .insert_account_storage(pool_address, storage_slot, bitmap_value)?; - injected += 1; + updates.push(StateUpdate::slot(pool_address, storage_slot, bitmap_value)); } + let injected = updates.len(); + self.apply_updates(&updates); Ok(injected) } @@ -1762,6 +3094,13 @@ impl EvmCache { /// # Arguments /// * `pool_address` - The UniswapV3 pool contract address /// * `ticks` - Map of tick index (int24) to tick info + /// + /// # Layering (Phase 3 change) + /// As of Phase 3 this writes **through** the dual-layer policy via + /// [`apply_updates`](Self::apply_updates) (backend always, overlay-if-present) + /// rather than the old overlay-only write. See `CHANGELOG.md` / + /// `KNOWN_ISSUES.md`. The slot writes are now infallible; the `Result` is + /// retained for signature compatibility. #[cfg(feature = "protocols")] #[cfg_attr(docsrs, doc(cfg(feature = "protocols")))] pub fn inject_v3_ticks( @@ -1783,7 +3122,7 @@ impl EvmCache { ticks: &std::collections::HashMap, ticks_slot: U256, ) -> Result { - let mut injected = 0; + let mut updates = Vec::with_capacity(ticks.len() * 2); for (&tick, info) in ticks { let tick_i256 = i256_from_i24(tick); let mut slot_preimage = [0u8; 64]; @@ -1798,8 +3137,7 @@ impl EvmCache { let liquidity_net_u256 = i128_to_u256(info.liquidity_net); let packed_slot0 = liquidity_gross_u256 | (liquidity_net_u256 << 128); - self.db - .insert_account_storage(pool_address, base_slot, packed_slot0)?; + updates.push(StateUpdate::slot(pool_address, base_slot, packed_slot0)); // Also inject slot 3 with the `initialized` flag. // Slot 3 layout: packed (tickCumulativeOutside, secondsPerLiquidityOutsideX128, @@ -1818,12 +3156,11 @@ impl EvmCache { } else { U256::ZERO }; - self.db - .insert_account_storage(pool_address, slot3, initialized_value)?; - - injected += 1; + updates.push(StateUpdate::slot(pool_address, slot3, initialized_value)); } + let injected = ticks.len(); + self.apply_updates(&updates); Ok(injected) } @@ -1898,24 +3235,29 @@ impl EvmCache { let mut evm = self.build_evm(); let checkpoint = evm.journaled_state.checkpoint(); - let result = evm - .transact_one(tx) - .map_err(|e| anyhow!("Failed to transact: {:?}", e))?; - - // Extract access list from journaled state before reverting. - // After transact_one, journaled_state.state contains all touched accounts/slots. - let mut access_list = StorageAccessList::default(); - for (address, account) in evm.journaled_state.state.iter() { - if account.is_touched() { - access_list.accounts.insert(*address); - for (slot_key, _) in account.storage.iter() { - access_list.slots.insert((*address, *slot_key)); + match evm.transact_one(tx) { + Ok(result) => { + // Extract access list from journaled state before reverting. After + // transact_one, journaled_state.state holds all touched accounts/slots. + let mut access_list = StorageAccessList::default(); + for (address, account) in evm.journaled_state.state.iter() { + if account.is_touched() { + access_list.accounts.insert(*address); + for (slot_key, _) in account.storage.iter() { + access_list.slots.insert((*address, *slot_key)); + } + } } + evm.journaled_state.checkpoint_revert(checkpoint); + Ok((result, access_list)) + } + Err(e) => { + // Revert the checkpoint even on a host/transact error so the EVM + // journal is not left dirty (mirrors `call_raw`). + evm.journaled_state.checkpoint_revert(checkpoint); + Err(anyhow!("Failed to transact: {:?}", e)) } } - - evm.journaled_state.checkpoint_revert(checkpoint); - Ok((result, access_list)) } /// Execute a call and return its emitted logs and gas used. @@ -2152,6 +3494,22 @@ impl EvmCache { "Reserved shared memory buffer capacity" ); } + drop(buffer); + // Record the high-water mark so snapshots taken afterwards propagate it to + // their overlays (snapshots copy the capacity at creation time). + self.shared_memory_capacity = self.shared_memory_capacity.max(capacity); + } + + /// The resolved per-context EVM shared-memory pre-allocation, in bytes. + /// + /// This is the [`SharedMemoryCapacity`] configured on the + /// [`EvmCacheBuilder`] resolved to a concrete size (with + /// [`SharedMemoryCapacity::Auto`] resolved against the state loaded at + /// construction), raised by any later [`reserve_shared_memory`](Self::reserve_shared_memory). + /// Each [`create_snapshot`](Self::create_snapshot) copies it onto the snapshot + /// so snapshot-backed [`EvmOverlay`]s pre-allocate the same amount. + pub fn shared_memory_capacity(&self) -> usize { + self.shared_memory_capacity } /// Purge all storage slots for a specific pool from both cache layers. @@ -2167,6 +3525,19 @@ impl EvmCache { /// After purging both layers, the next EVM read for this pool's storage will /// go all the way to the RPC for fresh data. pub fn purge_pool_storage(&mut self, address: Address) -> usize { + // Thin wrapper over the unified purge primitive; returns the backend slot + // count the `AllStorage` scope removed. + self.apply_update(&StateUpdate::purge(address, PurgeScope::AllStorage)) + .purged + .first() + .map(|rec| rec.slots_removed) + .unwrap_or(0) + } + + /// `AllStorage`-scope purge layer logic. Clears the overlay storage for + /// `address` and removes its backend storage map. Returns the number of + /// backend slots removed. + fn purge_pool_storage_inner(&mut self, address: Address) -> usize { // Layer 1: Clear CacheDB overlay let cache_db_cleared = if let Some(db_account) = self.db.cache.accounts.get_mut(&address) { let count = db_account.storage.len(); @@ -2177,11 +3548,13 @@ impl EvmCache { }; // Layer 2: Clear BlockchainDb backend - let mut storage = self.blockchain_db.storage().write(); - let backend_cleared = if let Some(slots) = storage.remove(&address) { - slots.len() - } else { - 0 + let backend_cleared = { + let mut storage = self.blockchain_db.storage().write(); + if let Some(slots) = storage.remove(&address) { + slots.len() + } else { + 0 + } }; if cache_db_cleared > 0 || backend_cleared > 0 { @@ -2193,6 +3566,8 @@ impl EvmCache { ); } + // Layer-2 storage for this address was removed → invalidate base. + self.mark_base_dirty(address); backend_cleared } @@ -2206,6 +3581,21 @@ impl EvmCache { /// /// Returns the number of slots removed from the BlockchainDb backend. pub fn purge_pool_slots(&mut self, address: Address, slots: &[U256]) -> usize { + // Thin wrapper over the unified purge primitive; returns the backend slot + // count the `Slots` scope removed. + self.apply_update(&StateUpdate::purge( + address, + PurgeScope::Slots(slots.to_vec()), + )) + .purged + .first() + .map(|rec| rec.slots_removed) + .unwrap_or(0) + } + + /// `Slots`-scope purge layer logic. Removes the listed slots from the overlay + /// and the backend storage map. Returns the number of backend slots removed. + fn purge_pool_slots_inner(&mut self, address: Address, slots: &[U256]) -> usize { let mut cache_db_removed = 0usize; let mut backend_removed = 0usize; @@ -2219,11 +3609,13 @@ impl EvmCache { } // Layer 2: Remove specific slots from BlockchainDb backend - let mut storage = self.blockchain_db.storage().write(); - if let Some(address_storage) = storage.get_mut(&address) { - for slot in slots { - if address_storage.remove(slot).is_some() { - backend_removed += 1; + { + let mut storage = self.blockchain_db.storage().write(); + if let Some(address_storage) = storage.get_mut(&address) { + for slot in slots { + if address_storage.remove(slot).is_some() { + backend_removed += 1; + } } } } @@ -2238,6 +3630,9 @@ impl EvmCache { ); } + // Layer-2 storage for this address changed (slots dropped) → invalidate + // base. The growth scan only catches length changes; mark explicitly. + self.mark_base_dirty(address); backend_removed } @@ -2277,6 +3672,9 @@ impl EvmCache { "purged contract storage from both cache layers" ); } + // Multiple layer-2 contracts changed → full base rebuild (coarse but + // correct; cheaper than enumerating each touched address here). + self.invalidate_base(); total_purged } @@ -2305,10 +3703,13 @@ impl EvmCache { } // Layer 2: Clear BlockchainDb backend - let mut storage = self.blockchain_db.storage().write(); - let total_slots: usize = storage.values().map(|s| s.len()).sum(); - let contract_count = storage.len(); - storage.clear(); + let (total_slots, contract_count) = { + let mut storage = self.blockchain_db.storage().write(); + let total_slots: usize = storage.values().map(|s| s.len()).sum(); + let contract_count = storage.len(); + storage.clear(); + (total_slots, contract_count) + }; if total_slots > 0 || cache_db_cleared > 0 { warn!( @@ -2318,6 +3719,8 @@ impl EvmCache { "purged ALL storage from both cache layers (full refresh)" ); } + // All layer-2 storage was cleared → full base rebuild. + self.invalidate_base(); total_slots } @@ -2387,8 +3790,9 @@ impl EvmCache { /// reverted. Unlike /// [`simulate_with_transfer_tracking`](Self::simulate_with_transfer_tracking), /// this measures deltas via pre/post balance reads (not transfer-event - /// inspection) and the returned - /// [`access_list`](CallSimulationResult::access_list) is always empty. + /// inspection). The returned [`access_list`](CallSimulationResult::access_list) + /// includes the accounts and slots touched by the pre/post `balanceOf` reads + /// and the simulated call. /// /// # Errors /// Returns an error if building the tx env fails, if a pre/post @@ -2436,11 +3840,13 @@ impl EvmCache { token_deltas.insert(*token, I256::from_raw(post) - I256::from_raw(pre)); } - Ok((gas_used, token_deltas, logs, output)) + let access_list = extract_access_list(&evm.journaled_state.state); + + Ok((gas_used, token_deltas, logs, output, access_list)) })(); match result { - Ok((gas_used, token_deltas, logs, output)) => { + Ok((gas_used, token_deltas, logs, output, access_list)) => { if commit { evm.commit_inner(); } else { @@ -2451,7 +3857,7 @@ impl EvmCache { gas_used, token_deltas, logs, - access_list: AccessList::default(), + access_list, output, }) } @@ -2644,6 +4050,17 @@ impl EvmCache { } /// Override code at `target`, with explicit behavior for missing target accounts. + /// + /// This is intentionally **not** folded onto + /// [`apply_update`](Self::apply_update)'s `Account` code patch: it copies code + /// from a `source` account, preserves the target's existing balance/nonce/ + /// storage, and **unconditionally materializes** the target in the CacheDB + /// overlay (the primary read path for EVM execution, required for the + /// `Create` synthetic-target case). The generic primitive writes the overlay + /// only when an account is already present, so the two are not + /// behavior-equivalent. For a plain code overwrite that follows the + /// dual-layer write-through policy, use + /// `apply_update(StateUpdate::Account { patch: AccountPatch::default().code(..) })`. pub fn override_account_code_with_missing_target( &mut self, source: Address, @@ -2686,6 +4103,12 @@ impl EvmCache { accounts.insert(target, target_info); } + // Layer 2 changed → invalidate the memoized base for `target`. The layer-1 + // `insert_account_info` above currently shadows it on every snapshot read, + // but we dirty unconditionally for uniformity with every other layer-2 write + // site (D2), so base correctness never relies on that shadowing invariant. + self.mark_base_dirty(target); + Ok(()) } @@ -2700,7 +4123,12 @@ impl EvmCache { missing_target: MissingTargetBehavior, ) -> Result { if let Some(account) = self.db.cache.accounts.get(&target) { - return Ok(account.info.clone()); + // A NotExisting overlay account is absent to the EVM (revm + // `DbAccount::info()` returns None); treat it as a missing target + // rather than returning its stale/default info. + if !matches!(account.account_state, AccountState::NotExisting) { + return Ok(account.info.clone()); + } } match missing_target { @@ -2733,6 +4161,16 @@ impl EvmCache { } } +/// Read-only state view for the event pipeline (Pillar B.2): a decoder reads the +/// current cached value of a slot through [`cached_storage_value`](EvmCache::cached_storage_value), +/// which never touches RPC and is `account_state`-aware (a cold slot reads +/// `None`). +impl crate::events::StateView for EvmCache { + fn storage(&self, address: Address, slot: U256) -> Option { + self.cached_storage_value(address, slot) + } +} + impl EvmCache { /// Create a LocalContext that reuses the shared memory buffer. /// @@ -2956,7 +4394,9 @@ impl Drop for EvmCache { fn drop(&mut self) { if self.cache_config.is_some() { debug!("Flushing EVM cache on drop"); - self.flush(); + if let Err(e) = self.flush() { + warn!(error = %e, "Failed to flush EVM cache on drop"); + } } } } @@ -2982,6 +4422,35 @@ fn extract_access_list(state: &revm::state::EvmState) -> AccessList { AccessList(items) } +#[cfg(test)] +mod shared_memory_capacity_tests { + use super::SharedMemoryCapacity as Cap; + + #[test] + fn default_is_fixed_64k() { + assert_eq!(Cap::default(), Cap::Fixed(64 * 1024)); + } + + #[test] + fn fixed_ignores_loaded_slots() { + assert_eq!(Cap::Fixed(8_192).resolve(10_000_000), 8_192); + assert_eq!(Cap::Fixed(0).resolve(123), 0); + } + + #[test] + fn auto_floors_clamps_and_scales() { + // Nothing / little loaded → floor. + assert_eq!(Cap::Auto.resolve(0), Cap::MIN_AUTO); + assert_eq!(Cap::Auto.resolve(1_000), Cap::MIN_AUTO); // 16 KiB < 64 KiB floor + // Linear region (16 bytes/slot). + assert_eq!(Cap::Auto.resolve(10_000), 160_000); + assert_eq!(Cap::Auto.resolve(100_000), 1_600_000); + // Ceiling. + assert_eq!(Cap::Auto.resolve(usize::MAX), Cap::MAX_AUTO); + assert_eq!(Cap::Auto.resolve(262_144), Cap::MAX_AUTO); // 262_144 * 16 == 4 MiB + } +} + #[cfg(all(test, feature = "protocols"))] mod tests { use super::*; @@ -3470,6 +4939,126 @@ mod core_tests { assert_eq!(cache.basefee(), None); } + #[test] + fn set_block_latest_clears_stale_block_context() { + use alloy_provider::RootProvider; + use alloy_rpc_client::RpcClient; + use alloy_transport::mock::Asserter; + + let asserter = Asserter::new(); + let client = RpcClient::mocked(asserter); + let provider = RootProvider::::new(client); + + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + + let mut cache = rt.block_on(EvmCache::new(Arc::new(provider), None)); + cache.set_block_context(Some(148_252_680), Some(50)); + + cache.set_block(Some(BlockId::latest())); + + assert_eq!( + cache.block_number(), + None, + "tag pins must not retain a stale NUMBER context" + ); + assert_eq!( + cache.basefee(), + None, + "set_block cannot refresh BASEFEE synchronously, so it must clear stale values" + ); + } + + #[test] + fn set_block_none_clears_stale_context_even_when_pin_unchanged() { + use alloy_provider::RootProvider; + use alloy_rpc_client::RpcClient; + use alloy_transport::mock::Asserter; + + let asserter = Asserter::new(); + let client = RpcClient::mocked(asserter); + let provider = RootProvider::::new(client); + + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + + let mut cache = rt.block_on(EvmCache::new(Arc::new(provider), None)); + cache.set_block_context(Some(148_252_680), Some(50)); + + cache.set_block(None); + + assert_eq!( + cache.block_number(), + None, + "None pins must not retain a stale NUMBER context" + ); + assert_eq!( + cache.basefee(), + None, + "None pins can drift like tags, so stale BASEFEE must be cleared" + ); + } + + #[test] + fn set_block_number_sets_number_and_clears_stale_basefee() { + use alloy_provider::RootProvider; + use alloy_rpc_client::RpcClient; + use alloy_transport::mock::Asserter; + + let asserter = Asserter::new(); + let client = RpcClient::mocked(asserter); + let provider = RootProvider::::new(client); + + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + + let mut cache = rt.block_on(EvmCache::new(Arc::new(provider), None)); + cache.set_block_context(Some(100), Some(50)); + + cache.set_block(Some(BlockId::Number(BlockNumberOrTag::Number(200)))); + + assert_eq!(cache.block_number(), Some(200)); + assert_eq!( + cache.basefee(), + None, + "set_block cannot refresh BASEFEE synchronously, so it must clear stale values" + ); + } + + #[test] + fn repin_to_block_clears_stale_basefee() { + use alloy_provider::RootProvider; + use alloy_rpc_client::RpcClient; + use alloy_transport::mock::Asserter; + + let asserter = Asserter::new(); + let client = RpcClient::mocked(asserter); + let provider = RootProvider::::new(client); + + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + + let mut cache = rt.block_on(EvmCache::new(Arc::new(provider), None)); + cache.set_block_context(Some(100), Some(50)); + + cache.repin_to_block(200); + + assert_eq!(cache.block_number(), Some(200)); + assert_eq!( + cache.basefee(), + None, + "repin_to_block must not carry stale BASEFEE across blocks" + ); + } + #[test] fn test_build_evm_applies_block_context() { use alloy_provider::RootProvider; @@ -3525,8 +5114,8 @@ mod core_tests { let block_num = Some(148_252_680u64); let basefee_val = Some(50u64); let child = EvmCache::from_backend( - parent.backend().clone(), - parent.blockchain_db().clone(), + parent.unchecked_backend().clone(), + parent.unchecked_blockchain_db().clone(), parent.block(), 42161, block_num, diff --git a/src/cache/overlay.rs b/src/cache/overlay.rs index 4831517..2d84fe2 100644 --- a/src/cache/overlay.rs +++ b/src/cache/overlay.rs @@ -21,9 +21,6 @@ use crate::access_set::StorageAccessList; use crate::errors::{SimError, SimulationError, SimulationResult}; use crate::inspector::TransferInspector; -/// Default initial capacity for shared memory buffer (64KB). -const OVERLAY_SHARED_MEMORY_CAPACITY: usize = 64 * 1024; - type OverlayEvm<'a> = revm::MainnetEvm< Context, ()>, >; @@ -40,6 +37,13 @@ type InspectorOverlayEvm<'a, INSP> = revm::MainnetEvm< /// This type is `Send` (unlike `EvmCache`) because it uses no `Rc`/`RefCell`. /// Each simulation task gets its own `EvmOverlay` with a cheap `Arc::clone` /// of the shared `EvmSnapshot`. +/// +/// # Reuse across simulations (Pillar A.2) +/// +/// A worker doing many sims against the same snapshot can call [`Self::new`] +/// once and [`Self::reset`] between sims instead of allocating a fresh overlay +/// each time. The reusable shared-memory buffer is also recycled across calls — +/// see [`Self::call_raw`] — without making the overlay `!Send`. pub struct EvmOverlay { snapshot: Arc, /// Per-simulation mutations (accounts fetched from ext_db, committed changes). @@ -48,19 +52,56 @@ pub struct EvmOverlay { dirty_storage: HashMap>, /// Optional RPC fallback for data not in snapshot. ext_db: Option, + /// Reusable shared-memory buffer, recycled across the build→transact→revert + /// call methods to avoid reallocating a 64 KB `Vec` per call. + /// + /// Stored as a plain `Vec` (not an `Rc`) so the overlay stays `Send`. A + /// call method `mem::take`s it, wraps it in a method-local `Rc>` + /// for revm's [`LocalContext`], runs, then reclaims and clears it after the + /// EVM is dropped (see [`Self::build_evm_with_local`]). + reusable_buffer: Vec, + /// Target pre-allocation (bytes) for [`Self::reusable_buffer`] and each + /// per-call buffer, taken from the snapshot's configured + /// [`SharedMemoryCapacity`](super::SharedMemoryCapacity) so overlays honor the + /// capacity set on the originating [`EvmCache`]. + buffer_capacity: usize, } impl EvmOverlay { /// Create a new overlay on the given snapshot. + /// + /// The reusable shared-memory buffer is pre-allocated to the snapshot's + /// configured shared-memory capacity (see + /// [`SharedMemoryCapacity`](super::SharedMemoryCapacity)). pub fn new(snapshot: Arc, ext_db: Option) -> Self { + let buffer_capacity = snapshot.shared_memory_capacity; Self { snapshot, dirty_accounts: HashMap::new(), dirty_storage: HashMap::new(), ext_db, + reusable_buffer: Vec::with_capacity(buffer_capacity), + buffer_capacity, } } + /// Clear the per-simulation dirty layer so this overlay can be reused for the + /// next simulation against the same snapshot, without reallocating (Pillar + /// A.2). + /// + /// A worker doing K sims calls [`Self::new`] once and `reset()` between sims + /// instead of allocating a fresh overlay (plus dirty maps plus an `Arc` + /// clone) each time. After `reset()` the overlay reads the pristine snapshot + /// again — it is exactly equivalent to a freshly-built overlay on the same + /// snapshot. The snapshot `Arc`, the optional `ext_db`, and the reusable + /// shared-memory buffer (kept at capacity) are retained. + pub fn reset(&mut self) { + self.dirty_accounts.clear(); + self.dirty_storage.clear(); + // Keep: snapshot Arc, ext_db, and the reusable buffer. The buffer is + // already cleared after each call, so nothing to do for it here. + } + /// Chain ID of the block context captured by the underlying snapshot. /// /// This is the value installed into `cfg.chain_id` by [`Self::build_evm`]. @@ -95,17 +136,29 @@ impl EvmOverlay { self.snapshot.timestamp } - /// Build a revm EVM instance backed by this overlay. + /// A fresh [`LocalContext`] with a newly-allocated 64 KB shared-memory buffer. /// - /// Note: The returned EVM is `!Send` (due to `LocalContext`'s `Rc`), - /// but this is fine because it's created and used within a single task. - pub fn build_evm(&mut self) -> OverlayEvm<'_> { - let local = LocalContext { - shared_memory_buffer: Rc::new(RefCell::new(Vec::with_capacity( - OVERLAY_SHARED_MEMORY_CAPACITY, - ))), + /// Used by the public [`Self::build_evm`], which hands out the EVM and cannot + /// reclaim its buffer afterwards. The internal call methods instead recycle + /// [`Self::reusable_buffer`] via [`Self::build_evm_with_local`]. + fn fresh_local(&self) -> LocalContext { + LocalContext { + shared_memory_buffer: Rc::new(RefCell::new(Vec::with_capacity(self.buffer_capacity))), precompile_error_message: None, - }; + } + } + + /// Build a revm EVM instance backed by this overlay, using a caller-supplied + /// [`LocalContext`]. + /// + /// This is the shared body behind [`Self::build_evm`] and the internal call + /// methods. The call methods pass a `local` wrapping the recycled + /// [`Self::reusable_buffer`] (Pillar A.2) and reclaim it after the EVM is + /// dropped; [`Self::build_evm`] passes a fresh one. + /// + /// Note: the returned EVM is `!Send` (due to `LocalContext`'s `Rc`), + /// but this is fine because it's created and used within a single task. + fn build_evm_with_local(&mut self, local: LocalContext) -> OverlayEvm<'_> { // Read snapshot values before the mutable borrow of self let chain_id = self.snapshot.chain_id; let spec_id = self.snapshot.spec_id; @@ -155,6 +208,20 @@ impl EvmOverlay { evm } + /// Build a revm EVM instance backed by this overlay. + /// + /// This allocates a fresh 64 KB shared-memory buffer each call: it hands the + /// EVM out to the caller and cannot reclaim the buffer afterwards, so it + /// cannot recycle the overlay's reusable buffer. The internal call methods + /// ([`Self::call_raw`], etc.) recycle the buffer instead (Pillar A.2). + /// + /// Note: The returned EVM is `!Send` (due to `LocalContext`'s `Rc`), + /// but this is fine because it's created and used within a single task. + pub fn build_evm(&mut self) -> OverlayEvm<'_> { + let local = self.fresh_local(); + self.build_evm_with_local(local) + } + /// Execute a non-committing call and return the raw [`ExecutionResult`]. /// /// The EVM state is reverted to a checkpoint after execution on *both* @@ -201,24 +268,59 @@ impl EvmOverlay { .build() .map_err(|e| anyhow!("Failed to build tx env: {:?}", e))?; - let mut evm = self.build_evm(); - use revm::context_interface::JournalTr; - let checkpoint = evm.journaled_state.checkpoint(); - let result = evm - .transact_one(tx) - .map_err(|e| anyhow!("Failed to transact: {:?}", e)); - evm.journaled_state.checkpoint_revert(checkpoint); - result - } - - /// Build a revm EVM instance with an inspector, backed by this overlay. - fn build_evm_with_inspector(&mut self, inspector: INSP) -> InspectorOverlayEvm<'_, INSP> { + // Recycle the reusable buffer (Pillar A.2): take it out as a plain Vec + // (keeping the overlay Send), lend it to a method-local Rc for + // revm's LocalContext, then reclaim and clear it after the EVM is dropped. + let buffer = Rc::new(RefCell::new(std::mem::take(&mut self.reusable_buffer))); let local = LocalContext { - shared_memory_buffer: Rc::new(RefCell::new(Vec::with_capacity( - OVERLAY_SHARED_MEMORY_CAPACITY, - ))), + shared_memory_buffer: Rc::clone(&buffer), precompile_error_message: None, }; + + let result = { + let mut evm = self.build_evm_with_local(local); + use revm::context_interface::JournalTr; + let checkpoint = evm.journaled_state.checkpoint(); + let result = evm + .transact_one(tx) + .map_err(|e| anyhow!("Failed to transact: {:?}", e)); + evm.journaled_state.checkpoint_revert(checkpoint); + result + }; + + self.reclaim_buffer(buffer); + result + } + + /// Reclaim the recycled shared-memory buffer after the EVM (and its + /// `LocalContext` clone of the `Rc`) has been dropped, clearing it for the + /// next call. + /// + /// The `Rc` was only ever held by the dropped EVM and this method's local, so + /// `try_unwrap` succeeds in the normal path. If a panic somewhere left an + /// extra strong reference the buffer is simply re-allocated next call — no + /// correctness impact. + fn reclaim_buffer(&mut self, buffer: Rc>>) { + if let Ok(cell) = Rc::try_unwrap(buffer) { + let mut buf = cell.into_inner(); + buf.clear(); + self.reusable_buffer = buf; + } else { + self.reusable_buffer = Vec::with_capacity(self.buffer_capacity); + } + } + + /// Build a revm EVM instance with an inspector, backed by this overlay, using + /// a caller-supplied [`LocalContext`]. + /// + /// Like [`Self::build_evm_with_local`] but attaches `inspector`. The call + /// methods pass a `local` wrapping the recycled [`Self::reusable_buffer`] + /// (Pillar A.2) and reclaim it after the EVM is dropped. + fn build_evm_with_inspector_local( + &mut self, + inspector: INSP, + local: LocalContext, + ) -> InspectorOverlayEvm<'_, INSP> { let chain_id = self.snapshot.chain_id; let spec_id = self.snapshot.spec_id; let timestamp = self.snapshot.timestamp.unwrap_or_else(|| { @@ -326,62 +428,75 @@ impl EvmOverlay { .map_err(|e| SimError::Other(anyhow!("Failed to build tx env: {:?}", e)))?; let inspector = TransferInspector::new(); - let mut evm = self.build_evm_with_inspector(inspector); - - use revm::context_interface::JournalTr; - let checkpoint = evm.journaled_state.checkpoint(); - - let result = evm - .inspect_one_tx(tx) - .map_err(|e| SimError::Other(anyhow!("Failed to transact: {:?}", e))); - - match result { - Ok(ExecutionResult::Success { - logs, - gas_used, - output, - .. - }) => { - let token_deltas = if let Some(token_list) = tokens { - evm.inspector.balance_deltas_for_tokens(owner, token_list) - } else { - evm.inspector.balance_deltas(owner) - }; - - // Extract EIP-2930 access list from journaled state - let access_list = extract_access_list(&evm.journaled_state.state); - - if commit { - evm.commit_inner(); - } else { - evm.journaled_state.checkpoint_revert(checkpoint); - } - Ok(CallSimulationResult { - status: SimStatus::Success, - gas_used, - token_deltas, + // Recycle the reusable buffer (Pillar A.2); reclaimed after the EVM drops. + let buffer = Rc::new(RefCell::new(std::mem::take(&mut self.reusable_buffer))); + let local = LocalContext { + shared_memory_buffer: Rc::clone(&buffer), + precompile_error_message: None, + }; + + let outcome = { + let mut evm = self.build_evm_with_inspector_local(inspector, local); + + use revm::context_interface::JournalTr; + let checkpoint = evm.journaled_state.checkpoint(); + + let result = evm + .inspect_one_tx(tx) + .map_err(|e| SimError::Other(anyhow!("Failed to transact: {:?}", e))); + + match result { + Ok(ExecutionResult::Success { logs, - access_list, - output: output.into_data(), - }) - } - Ok(ExecutionResult::Revert { gas_used, output }) => { - evm.journaled_state.checkpoint_revert(checkpoint); - Err(SimulationError::from_revert(gas_used, output).into()) - } - Ok(ExecutionResult::Halt { reason, gas_used }) => { - evm.journaled_state.checkpoint_revert(checkpoint); - Err(SimError::Halt { - reason: format!("{reason:?}"), gas_used, - }) - } - Err(err) => { - evm.journaled_state.checkpoint_revert(checkpoint); - Err(err) + output, + .. + }) => { + let token_deltas = if let Some(token_list) = tokens { + evm.inspector.balance_deltas_for_tokens(owner, token_list) + } else { + evm.inspector.balance_deltas(owner) + }; + + // Extract EIP-2930 access list from journaled state + let access_list = extract_access_list(&evm.journaled_state.state); + + if commit { + evm.commit_inner(); + } else { + evm.journaled_state.checkpoint_revert(checkpoint); + } + + Ok(CallSimulationResult { + status: SimStatus::Success, + gas_used, + token_deltas, + logs, + access_list, + output: output.into_data(), + }) + } + Ok(ExecutionResult::Revert { gas_used, output }) => { + evm.journaled_state.checkpoint_revert(checkpoint); + Err(SimulationError::from_revert(gas_used, output).into()) + } + Ok(ExecutionResult::Halt { reason, gas_used }) => { + evm.journaled_state.checkpoint_revert(checkpoint); + Err(SimError::Halt { + reason: format!("{reason:?}"), + gas_used, + }) + } + Err(err) => { + evm.journaled_state.checkpoint_revert(checkpoint); + Err(err) + } } - } + }; + + self.reclaim_buffer(buffer); + outcome } /// Execute a non-committing call and return the result plus the touched @@ -464,25 +579,42 @@ impl EvmOverlay { .build() .map_err(|e| anyhow!("Failed to build tx env: {:?}", e))?; - let mut evm = self.build_evm(); - use revm::context_interface::JournalTr; - let checkpoint = evm.journaled_state.checkpoint(); - let result = evm - .transact_one(tx_env) - .map_err(|e| anyhow!("Failed to transact: {:?}", e))?; - - let mut access_list = StorageAccessList::default(); - for (address, account) in evm.journaled_state.state.iter() { - if account.is_touched() { - access_list.accounts.insert(*address); - for (slot_key, _) in account.storage.iter() { - access_list.slots.insert((*address, *slot_key)); + // Recycle the reusable buffer (Pillar A.2); reclaimed after the EVM drops. + let buffer = Rc::new(RefCell::new(std::mem::take(&mut self.reusable_buffer))); + let local = LocalContext { + shared_memory_buffer: Rc::clone(&buffer), + precompile_error_message: None, + }; + + let outcome = { + let mut evm = self.build_evm_with_local(local); + use revm::context_interface::JournalTr; + let checkpoint = evm.journaled_state.checkpoint(); + match evm.transact_one(tx_env) { + Ok(result) => { + let mut access_list = StorageAccessList::default(); + for (address, account) in evm.journaled_state.state.iter() { + if account.is_touched() { + access_list.accounts.insert(*address); + for (slot_key, _) in account.storage.iter() { + access_list.slots.insert((*address, *slot_key)); + } + } + } + evm.journaled_state.checkpoint_revert(checkpoint); + Ok((result, access_list)) + } + Err(e) => { + // Revert the checkpoint even on a host/transact error so the EVM + // journal is not left dirty (mirrors `call_raw`). + evm.journaled_state.checkpoint_revert(checkpoint); + Err(anyhow!("Failed to transact: {:?}", e)) } } - } + }; - evm.journaled_state.checkpoint_revert(checkpoint); - Ok((result, access_list)) + self.reclaim_buffer(buffer); + outcome } /// Write a storage value into this overlay's dirty layer. @@ -544,8 +676,14 @@ impl Database for EvmOverlay { if let Some(info) = self.dirty_accounts.get(&address) { return Ok(Some(info.clone())); } - // 2. Check snapshot (O(1) HashMap lookup, no locks) - if let Some(info) = self.snapshot.accounts.get(&address) { + // 2. Check snapshot (O(1) HashMap lookup, no locks). `account_info` folds + // the two snapshot tiers (overlay ▸ base) and already short-circuits a + // NotExisting account to None — it must NOT fall through to the ext_db, + // mirroring revm `DbAccount::info()` and the live `EvmCache` read. + if self.snapshot.accounts_not_existing.contains(&address) { + return Ok(None); + } + if let Some(info) = self.snapshot.account_info(address) { return Ok(Some(info.clone())); } // 3. RPC fallback @@ -568,8 +706,8 @@ impl Database for EvmOverlay { return Ok(code.clone()); } } - // Check snapshot's code_by_hash index - if let Some(code) = self.snapshot.code_by_hash.get(&code_hash) { + // Check the snapshot's code index (overlay ▸ base). + if let Some(code) = self.snapshot.code(code_hash) { return Ok(code.clone()); } // RPC fallback @@ -586,11 +724,12 @@ impl Database for EvmOverlay { { return Ok(*value); } - // 2. Check snapshot (O(1)) - if let Some(account_storage) = self.snapshot.storage.get(&address) - && let Some(value) = account_storage.get(&index) - { - return Ok(*value); + // 2. Check snapshot (O(1)). `storage_value` folds the two tiers (overlay ▸ + // cleared-as-ZERO ▸ base); a cleared account's absent slot reads ZERO + // and must NOT fall through to the ext_db, mirroring the live EVM SLOAD + // for a StorageCleared/NotExisting account. + if let Some(value) = self.snapshot.storage_value(address, index) { + return Ok(value); } // 3. RPC fallback if let Some(ref ext_db) = self.ext_db { @@ -634,7 +773,47 @@ fn extract_access_list(state: &revm::state::EvmState) -> AccessList { #[cfg(test)] mod tests { use super::*; + use crate::cache::snapshot::BaseState; use revm::primitives::hardfork::SpecId; + use std::collections::HashSet; + + /// Build a two-tier `EvmSnapshot` whose cold base holds the given accounts, + /// storage, and code, with an empty hot overlay — the shape + /// `create_snapshot_deep_clone` produces. The `Arc`-per-account storage of the + /// base is built from the plain per-account maps. + fn snap( + accounts: HashMap, + storage: HashMap>, + code_by_hash: HashMap, + block_hashes: HashMap, + ) -> Arc { + let base = BaseState { + accounts, + storage: storage + .into_iter() + .map(|(addr, slots)| (addr, Arc::new(slots))) + .collect(), + code_by_hash, + }; + Arc::new(EvmSnapshot { + base: Arc::new(base), + overlay_accounts: HashMap::new(), + overlay_storage: HashMap::new(), + overlay_code_by_hash: HashMap::new(), + storage_cleared: HashSet::new(), + accounts_not_existing: HashSet::new(), + block_hashes, + block_number: None, + basefee: None, + coinbase: None, + prevrandao: None, + gas_limit: None, + chain_id: 42161, + timestamp: None, + spec_id: SpecId::CANCUN, + shared_memory_capacity: 64_000, + }) + } #[test] fn test_overlay_is_send() { @@ -655,20 +834,7 @@ mod tests { let addr = Address::repeat_byte(0x01); accounts.insert(addr, info); - let snapshot = Arc::new(EvmSnapshot { - accounts, - storage: HashMap::new(), - block_hashes: HashMap::new(), - code_by_hash: HashMap::new(), - block_number: None, - basefee: None, - coinbase: None, - prevrandao: None, - gas_limit: None, - chain_id: 42161, - timestamp: None, - spec_id: SpecId::CANCUN, - }); + let snapshot = snap(accounts, HashMap::new(), HashMap::new(), HashMap::new()); let mut overlay = EvmOverlay::new(snapshot, None); let result = overlay.basic(addr).unwrap(); @@ -687,20 +853,7 @@ mod tests { account_storage.insert(slot, value); storage.insert(addr, account_storage); - let snapshot = Arc::new(EvmSnapshot { - accounts: HashMap::new(), - storage, - block_hashes: HashMap::new(), - code_by_hash: HashMap::new(), - block_number: None, - basefee: None, - coinbase: None, - prevrandao: None, - gas_limit: None, - chain_id: 42161, - timestamp: None, - spec_id: SpecId::CANCUN, - }); + let snapshot = snap(HashMap::new(), storage, HashMap::new(), HashMap::new()); let mut overlay = EvmOverlay::new(snapshot, None); let result = overlay.storage(addr, slot).unwrap(); @@ -717,20 +870,7 @@ mod tests { account_storage.insert(slot, U256::from(100)); storage.insert(addr, account_storage); - let snapshot = Arc::new(EvmSnapshot { - accounts: HashMap::new(), - storage, - block_hashes: HashMap::new(), - code_by_hash: HashMap::new(), - block_number: None, - basefee: None, - coinbase: None, - prevrandao: None, - gas_limit: None, - chain_id: 42161, - timestamp: None, - spec_id: SpecId::CANCUN, - }); + let snapshot = snap(HashMap::new(), storage, HashMap::new(), HashMap::new()); let mut overlay = EvmOverlay::new(snapshot, None); @@ -748,20 +888,12 @@ mod tests { #[test] fn test_overlay_missing_returns_zero() { - let snapshot = Arc::new(EvmSnapshot { - accounts: HashMap::new(), - storage: HashMap::new(), - block_hashes: HashMap::new(), - code_by_hash: HashMap::new(), - block_number: None, - basefee: None, - coinbase: None, - prevrandao: None, - gas_limit: None, - chain_id: 42161, - timestamp: None, - spec_id: SpecId::CANCUN, - }); + let snapshot = snap( + HashMap::new(), + HashMap::new(), + HashMap::new(), + HashMap::new(), + ); let mut overlay = EvmOverlay::new(snapshot, None); let addr = Address::repeat_byte(0x99); @@ -780,20 +912,7 @@ mod tests { let mut code_by_hash = HashMap::new(); code_by_hash.insert(hash, code.clone()); - let snapshot = Arc::new(EvmSnapshot { - accounts: HashMap::new(), - storage: HashMap::new(), - block_hashes: HashMap::new(), - code_by_hash, - block_number: None, - basefee: None, - coinbase: None, - prevrandao: None, - gas_limit: None, - chain_id: 42161, - timestamp: None, - spec_id: SpecId::CANCUN, - }); + let snapshot = snap(HashMap::new(), HashMap::new(), code_by_hash, HashMap::new()); let mut overlay = EvmOverlay::new(snapshot, None); let result = overlay.code_by_hash(hash).unwrap(); @@ -806,20 +925,7 @@ mod tests { let hash = B256::repeat_byte(0xAB); block_hashes.insert(42u64, hash); - let snapshot = Arc::new(EvmSnapshot { - accounts: HashMap::new(), - storage: HashMap::new(), - block_hashes, - code_by_hash: HashMap::new(), - block_number: None, - basefee: None, - coinbase: None, - prevrandao: None, - gas_limit: None, - chain_id: 42161, - timestamp: None, - spec_id: SpecId::CANCUN, - }); + let snapshot = snap(HashMap::new(), HashMap::new(), HashMap::new(), block_hashes); let mut overlay = EvmOverlay::new(snapshot, None); assert_eq!(overlay.block_hash(42).unwrap(), hash); diff --git a/src/cache/snapshot.rs b/src/cache/snapshot.rs index d36364b..0b6c5ec 100644 --- a/src/cache/snapshot.rs +++ b/src/cache/snapshot.rs @@ -1,19 +1,28 @@ //! Immutable, shareable EVM state snapshots. //! -//! # Flattening model +//! # Two-tier copy-on-write model (Pillar A) //! -//! A snapshot flattens the live cache (CacheDB overlay plus the BlockchainDb -//! backend) into a single immutable, `Send + Sync` view of accounts and -//! storage. The layered lookups of the live cache are collapsed into flat -//! `HashMap`s at creation time, so every read against the snapshot is an O(1) -//! lookup with no locks and no fallback chain. +//! A snapshot is split into two tiers: //! -//! # `Arc` sharing +//! - a **memoized immutable base** (`BaseState`) flattening the *cold* layer-2 +//! `BlockchainDb` index, shared across successive snapshots by `Arc` — both the +//! base as a whole and each account's storage map (`Arc>`) — +//! so taking a snapshot when the cold index is unchanged is an `Arc` handle +//! copy, never a per-slot deep copy; +//! - a small per-snapshot **overlay** folding the *hot* layer-1 CacheDB delta +//! (committed sim changes, write-throughs, freshness corrections), which always +//! shadows the base on a read. //! -//! Because the snapshot is read-only it can be wrapped in an `Arc` and shared -//! across threads, letting many parallel simulations read from one consistent -//! state. Handing a new simulation task its state is a cheap `Arc::clone` -//! rather than a deep copy of the accounts/storage maps. +//! [`super::EvmCache::create_snapshot`] memoizes the base (via the internal +//! `refresh_base`) and folds only layer 1 fresh, so its cost tracks *changed* +//! state, not *total* state. The retained +//! [`super::EvmCache::create_snapshot_deep_clone`] produces the same two-tier +//! shape with everything flattened into the base and empty overlay maps; it is the +//! A/B benchmark baseline and the read-equivalence reference. +//! +//! Reads stay O(1) `HashMap` lookups with no locks (Decision D1: `Arc` sharing, +//! not a persistent/HAMT map), so the snapshot is `Send + Sync` and an +//! [`EvmOverlay`] built from it is `Send`. //! //! # Per-simulation dirty layer //! @@ -28,26 +37,72 @@ //! //! [`EvmOverlay`]: super::EvmOverlay -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; +use std::sync::Arc; use alloy_primitives::{Address, B256, U256}; use revm::primitives::hardfork::SpecId; use revm::state::{AccountInfo, Bytecode}; +/// Memoized, immutable flatten of the **cold layer-2** index (Pillar A). +/// +/// Holds layer-2 (`BlockchainDb`) account info and storage only; the layer-1 +/// `StorageCleared` / `NotExisting` classification is purely a layer-1 property +/// and lives on [`EvmSnapshot`], not here (see the read rules on +/// [`EvmSnapshot::storage_value`]). Each account's storage is wrapped in an `Arc` +/// so that rebuilding the base on a partial change (copy-on-write) shares the +/// `Arc` handles of unchanged accounts instead of deep-copying their slots. +/// +/// Built and memoized by [`EvmCache::refresh_base`](super::EvmCache::refresh_base); +/// shared across snapshots and across threads via `Arc`. +pub(crate) struct BaseState { + /// Layer-2 account info, by address. (Layer 2 has no `NotExisting` concept; + /// that classification is purely a layer-1 property — see [`EvmSnapshot`].) + pub(crate) accounts: HashMap, + /// Layer-2 storage, per account, **shared by `Arc`** so cloning a base — or + /// rebuilding it for an unchanged account — is a handle copy, never a per-slot + /// copy. + pub(crate) storage: HashMap>>, + /// Bytecode by `code_hash`, derived from `accounts` at build time. + pub(crate) code_by_hash: HashMap, +} + /// Immutable EVM state snapshot — `Send + Sync`, shared via `Arc` across threads. /// -/// Contains merged account info + storage from both CacheDB overlay and -/// BlockchainDb backend, providing a single flat `HashMap` view for O(1) lookups. +/// A two-tier copy-on-write view (see the [module docs](self)): an `Arc`-shared, +/// memoized cold base (layer 2) plus a small per-snapshot overlay folding the hot +/// layer-1 CacheDB delta, which shadows the base on reads. Lookups (including the +/// public [`storage_value`](Self::storage_value)) are O(1) and lock-free, and +/// reproduce the live cache's layered semantics bit-for-bit. /// /// Created via [`super::EvmCache::create_snapshot()`]. Each parallel simulation /// task gets its own [`super::EvmOverlay`] backed by a cheap `Arc::clone` of /// the snapshot. pub struct EvmSnapshot { - pub(crate) accounts: HashMap, - pub(crate) storage: HashMap>, + /// Memoized, `Arc`-shared cold layer-2 base. + pub(crate) base: Arc, + /// Layer-1 accounts that are present to the EVM (`NotExisting` excluded). + /// Shadows [`BaseState::accounts`] on a read. + pub(crate) overlay_accounts: HashMap, + /// Layer-1 storage delta, per account. A cleared account (revm + /// `StorageCleared` / `NotExisting`) ALWAYS has an entry here (possibly empty) + /// so the cleared rule is decided without consulting the base. + pub(crate) overlay_storage: HashMap>, + /// Bytecode introduced by layer 1 (checked before [`BaseState::code_by_hash`]). + pub(crate) overlay_code_by_hash: HashMap, + /// Accounts whose storage is locally complete (revm `StorageCleared` / + /// `NotExisting`): a slot absent from `overlay_storage` for such an account + /// reads as ZERO and must NOT fall through to the base or an `ext_db`, + /// mirroring the live EVM SLOAD and + /// [`EvmCache::cached_storage_value`](super::EvmCache::cached_storage_value). + pub(crate) storage_cleared: HashSet
, + /// Accounts that are absent to the EVM (revm `NotExisting`): + /// [`account_info`](Self::account_info) returns `None` for them and must NOT + /// fall through to the base or an `ext_db`, mirroring revm `DbAccount::info()` + /// and [`EvmCache`](super::EvmCache)'s live account read. These addresses are + /// excluded from `overlay_accounts` / `overlay_code_by_hash`. + pub(crate) accounts_not_existing: HashSet
, pub(crate) block_hashes: HashMap, - /// Bytecode lookup by code_hash (derived from accounts at creation time). - pub(crate) code_by_hash: HashMap, // Block context pub(crate) block_number: Option, pub(crate) basefee: Option, @@ -57,25 +112,76 @@ pub struct EvmSnapshot { pub(crate) chain_id: u64, pub(crate) timestamp: Option, pub(crate) spec_id: SpecId, + /// Per-context EVM shared-memory pre-allocation (bytes) copied from the + /// [`EvmCache`](super::EvmCache) at snapshot time, so an [`EvmOverlay`] built + /// from this snapshot pre-allocates the same working-memory size the live cache + /// was configured with (see + /// [`SharedMemoryCapacity`](super::SharedMemoryCapacity)). + pub(crate) shared_memory_capacity: usize, } impl EvmSnapshot { - /// Return the snapshot's value for a storage slot, if present. + /// Account info as the EVM sees it: overlay (layer 1) wins, else the base + /// (layer 2), else `None`. /// - /// Used by the freshness validator to compare a freshly-fetched value - /// against the value the snapshot was built from. A missing entry means the - /// snapshot never captured that slot (it would read as zero in a sim). + /// Returns `None` for a `NotExisting` account without consulting the base, + /// mirroring revm `DbAccount::info()` and the live `EvmCache` account read. + pub(crate) fn account_info(&self, address: Address) -> Option<&AccountInfo> { + if self.accounts_not_existing.contains(&address) { + return None; + } + self.overlay_accounts + .get(&address) + .or_else(|| self.base.accounts.get(&address)) + } + + /// Return the snapshot's value for a storage slot, mirroring the live read. + /// + /// Used by the freshness validator to compare a freshly-fetched value against + /// the value the snapshot was built from. Resolution matches + /// [`EvmCache::cached_storage_value`](super::EvmCache::cached_storage_value) + /// over the two tiers: an overlay (layer-1) slot wins; for a cleared account + /// an absent overlay slot returns `Some(ZERO)` (its storage is locally + /// complete — the base is never consulted); otherwise the base (layer-2) slot + /// is returned, or `None` if neither tier has seen the slot. pub fn storage_value(&self, address: Address, slot: U256) -> Option { - self.storage + if let Some(account_storage) = self.overlay_storage.get(&address) { + if let Some(value) = account_storage.get(&slot) { + return Some(*value); + } + // A StorageCleared / NotExisting account's storage is locally complete: + // an absent slot reads ZERO and never falls through to the base. + if self.storage_cleared.contains(&address) { + return Some(U256::ZERO); + } + // Non-cleared overlay account: fall through to the base below. + } + self.base + .storage .get(&address) .and_then(|s| s.get(&slot).copied()) } + + /// Bytecode by `code_hash`: overlay (layer 1) wins, else the base (layer 2). + pub(crate) fn code(&self, code_hash: B256) -> Option<&Bytecode> { + self.overlay_code_by_hash + .get(&code_hash) + .or_else(|| self.base.code_by_hash.get(&code_hash)) + } } #[cfg(test)] mod tests { use super::*; - use std::sync::Arc; + + /// Build an empty `Arc` for snapshot literals in tests. + fn empty_base() -> Arc { + Arc::new(BaseState { + accounts: HashMap::new(), + storage: HashMap::new(), + code_by_hash: HashMap::new(), + }) + } #[test] fn test_snapshot_is_send_sync() { @@ -87,10 +193,13 @@ mod tests { #[test] fn test_empty_snapshot() { let snap = EvmSnapshot { - accounts: HashMap::new(), - storage: HashMap::new(), + base: empty_base(), + overlay_accounts: HashMap::new(), + overlay_storage: HashMap::new(), + overlay_code_by_hash: HashMap::new(), + storage_cleared: HashSet::new(), + accounts_not_existing: HashSet::new(), block_hashes: HashMap::new(), - code_by_hash: HashMap::new(), block_number: Some(100), basefee: Some(1000), coinbase: None, @@ -99,6 +208,7 @@ mod tests { chain_id: 42161, timestamp: None, spec_id: SpecId::CANCUN, + shared_memory_capacity: 64_000, }; assert_eq!(snap.chain_id, 42161); assert_eq!(snap.block_number, Some(100)); diff --git a/src/cache/tick_snapshot.rs b/src/cache/tick_snapshot.rs index e5f6f5a..145f7b2 100644 --- a/src/cache/tick_snapshot.rs +++ b/src/cache/tick_snapshot.rs @@ -12,7 +12,11 @@ use std::path::Path; use alloy_primitives::{Address, U256}; use anyhow::Result; use serde::{Deserialize, Serialize}; -use tracing::warn; + +use super::versioned; + +const TICK_SNAPSHOT_CACHE_MAGIC: &[u8; 8] = b"EFCTICK\0"; +const TICK_SNAPSHOT_CACHE_VERSION: u32 = 1; /// Per-tick liquidity state for a UniswapV3-style concentrated-liquidity pool. /// @@ -151,15 +155,16 @@ pub struct V3TickSnapshotCache { impl V3TickSnapshotCache { /// Load tick snapshot cache from disk (binary format). /// - /// Returns `None` if `path` cannot be read or its contents fail to decode as - /// bincode for this type; a decode failure is logged at `warn` level and - /// treated as a cache miss. The format has no version header, so a file from - /// an incompatible build also yields `None`. + /// Returns `None` if `path` cannot be read, fails the magic/version check, or + /// fails to decode as bincode for this type. pub fn load(path: &Path) -> Option { let data = std::fs::read(path).ok()?; - bincode::deserialize(&data) - .inspect_err(|e| warn!("Failed to parse V3 tick snapshot cache (bincode): {}", e)) - .ok() + versioned::decode( + &data, + TICK_SNAPSHOT_CACHE_MAGIC, + TICK_SNAPSHOT_CACHE_VERSION, + "V3 tick snapshot cache", + ) } /// Save tick snapshot cache to disk (binary format). @@ -175,7 +180,12 @@ impl V3TickSnapshotCache { if let Some(parent) = path.parent() { std::fs::create_dir_all(parent)?; } - let data = bincode::serialize(self)?; + let data = versioned::encode( + TICK_SNAPSHOT_CACHE_MAGIC, + TICK_SNAPSHOT_CACHE_VERSION, + self, + "V3 tick snapshot cache", + )?; std::fs::write(path, data)?; Ok(()) } diff --git a/src/cache/versioned.rs b/src/cache/versioned.rs new file mode 100644 index 0000000..0ed9681 --- /dev/null +++ b/src/cache/versioned.rs @@ -0,0 +1,66 @@ +use serde::{Serialize, de::DeserializeOwned}; +use tracing::warn; + +use anyhow::{Context as _, Result}; + +const VERSION_BYTES: usize = 4; + +pub(crate) fn encode( + magic: &[u8; 8], + version: u32, + value: &T, + label: &'static str, +) -> Result> { + let payload = + bincode::serialize(value).with_context(|| format!("failed to serialize {label}"))?; + let mut data = Vec::with_capacity(magic.len() + VERSION_BYTES + payload.len()); + data.extend_from_slice(magic); + data.extend_from_slice(&version.to_le_bytes()); + data.extend_from_slice(&payload); + Ok(data) +} + +pub(crate) fn decode( + data: &[u8], + magic: &[u8; 8], + version: u32, + label: &'static str, +) -> Option { + let header_len = magic.len() + VERSION_BYTES; + if data.len() < header_len { + warn!( + cache = label, + bytes = data.len(), + "Cache file is missing version header; treating as cache miss" + ); + return None; + } + + if &data[..magic.len()] != magic { + warn!( + cache = label, + "Cache file has unrecognized magic header; treating as cache miss" + ); + return None; + } + + let version_start = magic.len(); + let found_version = u32::from_le_bytes( + data[version_start..header_len] + .try_into() + .expect("version slice length is fixed"), + ); + if found_version != version { + warn!( + cache = label, + expected = version, + found = found_version, + "Cache file version mismatch; treating as cache miss" + ); + return None; + } + + bincode::deserialize(&data[header_len..]) + .inspect_err(|e| warn!(cache = label, error = %e, "Failed to parse cache payload")) + .ok() +} diff --git a/src/events/erc20.rs b/src/events/erc20.rs new file mode 100644 index 0000000..c9c467f --- /dev/null +++ b/src/events/erc20.rs @@ -0,0 +1,145 @@ +//! Generic ERC-20 `Transfer` decoder (generic core). +//! +//! [`Erc20TransferDecoder`] turns a standard ERC-20 +//! `Transfer(from, to, value)` log into two relative balance updates — a +//! [`SlotDelta::Sub`] on the sender's balance slot and a +//! [`SlotDelta::Add`] on the recipient's — so the cache +//! tracks balances from the event stream without ever reading the resulting +//! absolute balances. It is the log-driven form of the Phase 3 reactive-balance +//! case. +//! +//! # Balance-slot derivation +//! +//! An ERC-20 `balanceOf` is a `mapping(address => uint256)` at some base slot. +//! The decoder hashes the owner into that mapping the canonical Solidity way: +//! `keccak256(abi.encode(owner, balance_slot))`. The base slot is configurable +//! per token ([`with_token`](Erc20TransferDecoder::with_token)) with a default +//! fallback ([`new`](Erc20TransferDecoder::new)), since different tokens place +//! `balanceOf` at different slots. +//! +//! # Mint / burn legs +//! +//! A mint (`from == 0`) or burn (`to == 0`) has no real holder on the +//! zero-address leg, so that leg is **skipped** — only the non-zero side emits a +//! delta. Cold balances follow the Phase 3 contract: the +//! [`SlotDelta`] is skipped at apply time and surfaced in +//! [`StateDiff::skipped`](crate::StateDiff::skipped) (the caller seeds the +//! balance, or the next read lazily fetches it). The decoder ignores the +//! [`StateView`] — it is stateless. + +use std::collections::HashMap; + +use alloy_primitives::{Address, Log, U256, keccak256}; +use alloy_sol_types::SolValue; + +use crate::events::{EventDecoder, StateView}; +use crate::inspector::TransferInspector; +use crate::state_update::{SlotDelta, StateUpdate}; + +/// Decodes ERC-20 `Transfer` logs into relative balance [`SlotDelta`] updates. +/// +/// ``` +/// use alloy_primitives::{Address, Bytes, Log, U256, keccak256}; +/// use alloy_sol_types::SolValue; +/// use evm_fork_cache::events::{EventDecoder, StateView}; +/// use evm_fork_cache::events::erc20::Erc20TransferDecoder; +/// use evm_fork_cache::{SlotDelta, StateUpdate}; +/// +/// // A read-only view that reports every slot cold (decoder is stateless anyway). +/// struct ColdView; +/// impl StateView for ColdView { +/// fn storage(&self, _: Address, _: U256) -> Option { None } +/// } +/// +/// let token = Address::repeat_byte(0x20); +/// let from = Address::repeat_byte(0x21); +/// let to = Address::repeat_byte(0x22); +/// +/// // Transfer(from, to, 100) log: balanceOf mapping at slot 3. +/// let sig = keccak256(b"Transfer(address,address,uint256)"); +/// let log = Log::new_unchecked( +/// token, +/// vec![sig, from.into_word(), to.into_word()], +/// Bytes::copy_from_slice(&U256::from(100).to_be_bytes::<32>()), +/// ); +/// +/// let decoder = Erc20TransferDecoder::new(U256::from(3)); +/// let updates = decoder.decode(&log, &ColdView); +/// +/// let slot = |owner: Address| { +/// U256::from_be_bytes(keccak256((owner, U256::from(3)).abi_encode()).0) +/// }; +/// assert_eq!(updates, vec![ +/// StateUpdate::slot_delta(token, slot(from), SlotDelta::Sub(U256::from(100))), +/// StateUpdate::slot_delta(token, slot(to), SlotDelta::Add(U256::from(100))), +/// ]); +/// ``` +pub struct Erc20TransferDecoder { + /// Balance mapping base slot per token (the `balanceOf` mapping's slot). + balance_slots: HashMap, + /// Fallback balance mapping base slot for tokens not in the map. + default_balance_slot: U256, +} + +impl Erc20TransferDecoder { + /// Create a decoder with `default_balance_slot` as the `balanceOf` mapping + /// base slot for any token without a per-token override. + pub fn new(default_balance_slot: U256) -> Self { + Self { + balance_slots: HashMap::new(), + default_balance_slot, + } + } + + /// Override the `balanceOf` mapping base slot for `token` (builder style). + pub fn with_token(mut self, token: Address, balance_slot: U256) -> Self { + self.balance_slots.insert(token, balance_slot); + self + } + + /// The configured balance mapping base slot for `token` (its override, else + /// the default). + fn balance_slot(&self, token: Address) -> U256 { + self.balance_slots + .get(&token) + .copied() + .unwrap_or(self.default_balance_slot) + } +} + +/// The hashed storage slot of `balanceOf[owner]` for a `mapping(address => +/// uint256)` at `mapping_slot`. +fn balance_key(owner: Address, mapping_slot: U256) -> U256 { + U256::from_be_bytes(keccak256((owner, mapping_slot).abi_encode()).0) +} + +impl EventDecoder for Erc20TransferDecoder { + fn decode(&self, log: &Log, _view: &dyn StateView) -> Vec { + // Reuse the canonical ERC-20 Transfer signature match + topic/data decode. + // Returns None for a non-Transfer log (wrong topic0, <3 topics, <32 data + // bytes). + let Some(transfer) = TransferInspector::parse_transfer(log) else { + return Vec::new(); + }; + + let slot = self.balance_slot(transfer.token); + let mut updates = Vec::with_capacity(2); + + // Skip the zero-address leg (mint = from == 0, burn = to == 0). + if transfer.from != Address::ZERO { + updates.push(StateUpdate::slot_delta( + transfer.token, + balance_key(transfer.from, slot), + SlotDelta::Sub(transfer.value), + )); + } + if transfer.to != Address::ZERO { + updates.push(StateUpdate::slot_delta( + transfer.token, + balance_key(transfer.to, slot), + SlotDelta::Add(transfer.value), + )); + } + updates + } +} diff --git a/src/events/mod.rs b/src/events/mod.rs new file mode 100644 index 0000000..510c558 --- /dev/null +++ b/src/events/mod.rs @@ -0,0 +1,439 @@ +//! Event → state pipeline (Pillar B.2 — the *reader half* of the event pipeline). +//! +//! Phase 3 ([`state_update`](crate::state_update)) built the *writer half*: the +//! generic [`StateUpdate`] vocabulary and the cold-aware +//! [`apply_updates`](crate::cache::EvmCache::apply_updates) that consumes it. This +//! module builds the *reader half*: it turns an on-chain [`Log`] into that same +//! vocabulary and drives it through the cache, keeping event-derived state +//! reactively fresh. +//! +//! # The flow +//! +//! ```text +//! Log ─▶ EventDecoder::decode(log, &StateView) ─▶ Vec +//! │ +//! apply_updates ▼ +//! EvmCache (+ StateDiff) +//! ``` +//! +//! A [`DecoderRegistry`] dispatches a log to the decoders registered for its +//! emitting address (plus any global decoders) and concatenates their output. An +//! [`EventPipeline`] orchestrates a block's logs: [`ingest_logs`] decodes and +//! applies them **log-by-log in order** (so a later log's decode observes the +//! effects of earlier ones through the [`StateView`]), [`reorg_to`] purges the +//! addresses touched after a new head, and [`reconcile`] re-reads sampled +//! event-derived slots against chain truth (correct **and** alarm). +//! +//! [`ingest_logs`]: EventPipeline::ingest_logs +//! [`reorg_to`]: EventPipeline::reorg_to +//! [`reconcile`]: EventPipeline::reconcile +//! +//! # Decoders are pure data functions +//! +//! [`EventDecoder::decode`] is a pure function of `(log, pre-state)`: it performs +//! no I/O and emits serializable, replayable [`StateUpdate`] data. Most updates +//! need no pre-state ([`SlotDelta`](crate::StateUpdate::SlotDelta) and +//! [`SlotMasked`](crate::StateUpdate::SlotMasked) are read-modify-write *at apply +//! time*), but stateful adapters — UniswapV3 tick maintenance must read the +//! current `liquidityGross`/`liquidityNet`/`tick`/bitmap to recompute a packed +//! word — read the narrow read-only [`StateView`]. The view never touches RPC; a +//! slot absent from the cache reads `None` (cold), and a decoder that cannot +//! compute against a cold word surfaces a skip rather than inventing a value. +//! +//! # `!Send` cache discipline +//! +//! [`EvmCache`] is `!Send` (it owns the mutable fork and +//! blocks on RPC internally). All of [`EventPipeline`]'s core methods +//! ([`ingest_logs`](EventPipeline::ingest_logs) / +//! [`reorg_to`](EventPipeline::reorg_to) / +//! [`reconcile`](EventPipeline::reconcile)) take `&mut EvmCache` and are +//! **synchronous** — they never `.await`, so the cache is never held across a +//! yield point. This is what makes the core deterministically testable offline. +//! The async [`drive`] convenience holds the cache only across the *log source* +//! await (the source future is `Send`; the cache is untouched during it). +//! +//! # Freshness wiring +//! +//! [`BlockDigest::touched_slots`] surfaces the `(address, slot)` set written for a +//! block so a caller can classify event-derived slots in a +//! [`FreshnessRegistry`](crate::freshness::FreshnessRegistry) — typically pin them +//! ([`Validity::Pinned`](crate::freshness::Validity::Pinned)) or mark them +//! [`Validity::ValidThrough`](crate::freshness::Validity::ValidThrough) so the +//! optimistic validator does not waste RPC re-verifying state the pipeline keeps +//! fresh — then call +//! [`FreshnessController::on_new_block`](crate::freshness::FreshnessController::on_new_block). +//! No controller internals change. Periodically call +//! [`reconcile`](EventPipeline::reconcile) to sample-check those slots against the +//! chain (honest freshness). + +pub mod erc20; +#[cfg(feature = "protocols")] +#[cfg_attr(docsrs, doc(cfg(feature = "protocols")))] +pub mod uniswap_v3; + +use std::collections::{HashMap, HashSet, VecDeque}; +use std::sync::Arc; + +use alloy_primitives::{Address, Log, U256}; +use anyhow::Result; + +use crate::cache::EvmCache; +use crate::freshness::SlotChange; +use crate::state_update::{PurgeScope, StateDiff, StateUpdate}; + +/// Read-only view of current cached state handed to a decoder. +/// +/// Decoders that compute post-state from pre-state (e.g. UniswapV3 tick +/// maintenance) read through this; stateless decoders (ERC-20 `Transfer`, V3 +/// `Swap`) ignore it. The view never touches RPC — a slot absent from the cache +/// reads `None`. +pub trait StateView { + /// Current cached value of `(address, slot)` (overlay ▸ backend ▸ `None`), + /// matching what the EVM would `SLOAD` (`account_state`-aware). `None` means + /// the slot is **cold** — neither cache layer has seen it. + fn storage(&self, address: Address, slot: U256) -> Option; +} + +/// Decode one log into zero or more targeted [`StateUpdate`]s. +/// +/// `decode` is a pure function of `(log, pre-state)`: it performs no I/O and emits +/// data (the updates are serializable and replayable against matching pre-state). +/// The pipeline applies the result through +/// [`apply_updates`](crate::cache::EvmCache::apply_updates). +/// +/// A decoder returns `vec![]` for any log it does not recognise (wrong topic0, an +/// unregistered emitting address, a malformed payload). The pipeline counts a log +/// as *decoded* only when some decoder produced at least one update for it. +pub trait EventDecoder: Send + Sync { + /// Decode `log` against the read-only pre-state `view` into targeted updates. + fn decode(&self, log: &Log, view: &dyn StateView) -> Vec; +} + +/// Dispatches a log to the decoders registered for its emitting address (and any +/// global decoders) and concatenates their output. +/// +/// Dispatch is by emitting address ([`Log::address`]); topic0 filtering is each +/// decoder's own concern (a decoder returns `vec![]` for a log it does not +/// recognise). Address-scoped decoders are consulted first, then global ones, and +/// the per-decoder outputs are concatenated in that order. +#[derive(Default)] +pub struct DecoderRegistry { + /// Decoders consulted for every log, in registration order. + global: Vec>, + /// Decoders consulted only for logs emitted by a specific address. + per_address: HashMap>>, +} + +impl DecoderRegistry { + /// Create an empty registry with no decoders. + pub fn new() -> Self { + Self::default() + } + + /// Register a decoder consulted for **every** log. + pub fn register(&mut self, decoder: Arc) -> &mut Self { + self.global.push(decoder); + self + } + + /// Register a decoder consulted only for logs emitted by `address`. + pub fn register_for_address( + &mut self, + address: Address, + decoder: Arc, + ) -> &mut Self { + self.per_address.entry(address).or_default().push(decoder); + self + } + + /// Decode `log` through every applicable decoder, concatenating the results + /// (address-scoped decoders first, then global), preserving order. + pub fn decode(&self, log: &Log, view: &dyn StateView) -> Vec { + let mut out = Vec::new(); + if let Some(scoped) = self.per_address.get(&log.address) { + for decoder in scoped { + out.extend(decoder.decode(log, view)); + } + } + for decoder in &self.global { + out.extend(decoder.decode(log, view)); + } + out + } +} + +/// How a reorg purges the addresses touched after the new head. +/// +/// `depth` bounds the per-block touched-address history retained for reorg purge +/// (the reorg horizon); older entries are dropped as new blocks are ingested. +/// `scope` is the [`PurgeScope`] applied to each touched address on +/// [`reorg_to`](EventPipeline::reorg_to). +#[derive(Clone, Debug)] +pub struct ReorgConfig { + /// How many recent blocks of touched-address history to retain for reorg + /// purge (the reorg horizon). Older entries are dropped. + pub depth: usize, + /// Purge scope used on reorg. The default ([`PurgeScope::AllStorage`]) drops + /// storage so it re-fetches but keeps the account header; + /// [`PurgeScope::Account`] drops the whole account. + pub scope: PurgeScope, +} + +impl Default for ReorgConfig { + fn default() -> Self { + Self { + depth: 64, + scope: PurgeScope::AllStorage, + } + } +} + +/// Per-block result of [`EventPipeline::ingest_logs`]. +#[derive(Clone, Debug, Default)] +pub struct BlockDigest { + /// The block whose logs were ingested. + pub block: u64, + /// Merged diff of everything applied for the block (changes-only **and** + /// skips — check [`StateDiff::has_skipped`]). + pub applied: StateDiff, + /// Number of logs that decoded to at least one update. + pub decoded_logs: usize, + /// The `(address, slot)` set written this block (for freshness + /// classification — see the module docs). + pub touched_slots: Vec<(Address, U256)>, +} + +/// Result of [`EventPipeline::reconcile`]. +#[derive(Clone, Debug, Default)] +pub struct ReconcileReport { + /// How many slots were sampled. + pub checked: usize, + /// Slots whose event-derived value disagreed with chain truth. A non-empty + /// list is a **drift alarm**: the cache had drifted and + /// [`verify_slots`](crate::cache::EvmCache::verify_slots) has now injected the + /// fresh chain values (correct + alarm). + pub mismatched: Vec, +} + +/// Orchestrates decoding, applying, reorg handling, and reconciliation of a +/// block's logs against an [`EvmCache`]. +/// +/// Construct one from a [`DecoderRegistry`], then call +/// [`ingest_logs`](Self::ingest_logs) per block. See the [module docs](crate::events) +/// for the freshness-wiring pattern (event-derived slots → +/// [`Pinned`](crate::freshness::Validity::Pinned), reconciled periodically). +pub struct EventPipeline { + registry: DecoderRegistry, + reorg: ReorgConfig, + /// Ring of `(block, touched addresses)` for reorg purge, newest at the back, + /// bounded to `reorg.depth`. + touched: VecDeque<(u64, Vec
)>, + /// Every event-derived `(address, slot)` seen so far (reconcile sampling + /// source). + derived_slots: HashSet<(Address, U256)>, +} + +impl EventPipeline { + /// Create a pipeline over `registry` with the default [`ReorgConfig`]. + pub fn new(registry: DecoderRegistry) -> Self { + Self { + registry, + reorg: ReorgConfig::default(), + touched: VecDeque::new(), + derived_slots: HashSet::new(), + } + } + + /// Override the [`ReorgConfig`] (reorg horizon depth + purge scope). + pub fn with_reorg_config(mut self, cfg: ReorgConfig) -> Self { + self.reorg = cfg; + self + } + + /// Decode + apply a block's logs, **log-by-log in order**, recording touched + /// state for reorg tracking. Returns the per-block [`BlockDigest`]. + /// + /// Each log is decoded against the *current* cache state and applied + /// immediately, so a later log's decode observes the effects of earlier logs + /// in the same block through the [`StateView`] (e.g. a same-block `Burn` after + /// a `Mint`, or two overlapping `Mint`s). The touched addresses are recorded + /// in the depth-bounded reorg ring under `block`, and the touched + /// `(address, slot)` pairs into the reconcile-sampling set. + pub fn ingest_logs(&mut self, cache: &mut EvmCache, block: u64, logs: &[Log]) -> BlockDigest { + let mut digest = BlockDigest { + block, + ..Default::default() + }; + let mut touched_addrs: HashSet
= HashSet::new(); + + for log in logs { + // Decode against the current cache view (immutable borrow), then drop + // that borrow before taking the &mut borrow for apply. Decode returns + // owned data, so the two borrows never overlap. + let updates = self.registry.decode(log, &*cache); + if updates.is_empty() { + continue; + } + let diff = cache.apply_updates(&updates); + + // A log counts as decoded when it produced at least one update. + digest.decoded_logs += 1; + + // Record touched addresses (for reorg) and touched slots (for + // freshness + reconcile) from every category of the diff. + for change in &diff.slots { + touched_addrs.insert(change.address); + self.note_touched_slot(&mut digest, change.address, change.slot); + } + for change in &diff.accounts { + touched_addrs.insert(change.address); + } + for record in &diff.purged { + touched_addrs.insert(record.address); + } + for skip in &diff.skipped { + touched_addrs.insert(skip.address); + self.note_touched_slot(&mut digest, skip.address, skip.slot); + } + for skip in &diff.skipped_balances { + touched_addrs.insert(skip.address); + } + for skip in &diff.skipped_masks { + touched_addrs.insert(skip.address); + self.note_touched_slot(&mut digest, skip.address, skip.slot); + } + + digest.applied.merge(diff); + } + + if !touched_addrs.is_empty() { + self.touched + .push_back((block, touched_addrs.into_iter().collect())); + self.trim_ring(); + } + + digest + } + + /// Reorg to `new_head`: purge (per [`ReorgConfig::scope`]) every address + /// touched in a block **>** `new_head`, drop those ring entries, and return the + /// merged purge [`StateDiff`]. + /// + /// The next read of a purged address re-fetches from RPC. The caller then + /// re-ingests the canonical chain's logs for the reorged range (and/or the + /// next read lazily re-fetches). + pub fn reorg_to(&mut self, cache: &mut EvmCache, new_head: u64) -> StateDiff { + // Collect the addresses touched strictly after the new head, deduped. + let mut to_purge: HashSet
= HashSet::new(); + for (block, addrs) in &self.touched { + if *block > new_head { + to_purge.extend(addrs.iter().copied()); + } + } + + // Drop the rolled-back ring entries and the derived slots they own. + self.touched.retain(|(block, _)| *block <= new_head); + self.derived_slots + .retain(|(addr, _)| !to_purge.contains(addr)); + + let updates: Vec = to_purge + .into_iter() + .map(|addr| StateUpdate::purge(addr, self.reorg.scope.clone())) + .collect(); + cache.apply_updates(&updates) + } + + /// Sampled reconciliation: re-read `slots` via + /// [`EvmCache::verify_slots`](crate::cache::EvmCache::verify_slots) (correct + + /// alarm). Returns the mismatches. + /// + /// It fetches the fresh chain value for each slot, injects the ones that + /// changed (so the cache is **corrected**), and returns the changed set — a + /// non-empty [`ReconcileReport::mismatched`] is the **drift alarm**: + /// event-derived state had drifted and has now been corrected to chain truth. + /// Honest about reachability (via + /// [`EvmCache::reconcile_slots`](crate::cache::EvmCache::reconcile_slots)): it + /// errors when no batch fetcher is configured **or** when a non-empty request + /// could not fetch any slot (a total fetch failure is not a silent all-clear). + /// An empty `slots` is a no-op that returns an empty report. + pub fn reconcile( + &mut self, + cache: &mut EvmCache, + slots: &[(Address, U256)], + ) -> Result { + let mismatched = cache.reconcile_slots(slots)?; + Ok(ReconcileReport { + checked: slots.len(), + mismatched, + }) + } + + /// All event-derived slots seen so far (the sampling source for + /// [`reconcile`](Self::reconcile)). + pub fn derived_slots(&self) -> impl Iterator + '_ { + self.derived_slots.iter().copied() + } + + /// Record a touched slot in both the per-block digest (deduped within the + /// block) and the global all-time reconcile-sampling set. + fn note_touched_slot(&mut self, digest: &mut BlockDigest, address: Address, slot: U256) { + self.derived_slots.insert((address, slot)); + if !digest.touched_slots.contains(&(address, slot)) { + digest.touched_slots.push((address, slot)); + } + } + + /// Trim the reorg ring to the configured depth, dropping the oldest entries. + fn trim_ring(&mut self) { + while self.touched.len() > self.reorg.depth { + self.touched.pop_front(); + } + } +} + +/// A signalled reorg accompanying a block from a [`LogSource`]. +/// +/// `None` means the block extends the current head; `Some(new_head)` asks the +/// driver to [`reorg_to`](EventPipeline::reorg_to) `new_head` before ingesting. +pub type ReorgSignal = Option; + +/// An async source of blocks of logs for [`drive`]. +/// +/// This is the thin async convenience layer (§7.5): a production WS / +/// `subscribe_logs` adapter implements it; the offline example feeds a vec-backed +/// source. The synchronous [`EventPipeline`] core is the tested contract. +pub trait LogSource { + /// Yield the next block: its number, its logs, and an optional reorg signal. + /// `None` ends the stream. + fn next_block( + &mut self, + ) -> impl std::future::Future, ReorgSignal)>> + Send; +} + +/// Drive `pipeline` over `source`, ingesting each block (reorging first when +/// signalled) and invoking `on_block` after each ingest. +/// +/// A thin async convenience over the synchronous core: it pulls a block from the +/// `Send` source (the only `.await`), then synchronously +/// [`reorg_to`](EventPipeline::reorg_to) (if signalled) and +/// [`ingest_logs`](EventPipeline::ingest_logs), holding the `!Send` cache only +/// across the synchronous section. `on_block` is where a caller wires +/// [`FreshnessController::on_new_block`](crate::freshness::FreshnessController::on_new_block) +/// and freshness classification of the digest's touched slots. +pub async fn drive( + pipeline: &mut EventPipeline, + cache: &mut EvmCache, + mut source: S, + mut on_block: F, +) where + S: LogSource, + F: FnMut(&BlockDigest), +{ + while let Some((block, logs, reorg)) = source.next_block().await { + if let Some(new_head) = reorg { + pipeline.reorg_to(cache, new_head); + } + let digest = pipeline.ingest_logs(cache, block, &logs); + on_block(&digest); + } +} diff --git a/src/events/uniswap_v3.rs b/src/events/uniswap_v3.rs new file mode 100644 index 0000000..c93d667 --- /dev/null +++ b/src/events/uniswap_v3.rs @@ -0,0 +1,399 @@ +//! UniswapV3 / PancakeSwap V3 event adapter (`protocols` feature). +//! +//! [`UniswapV3Decoder`] turns a pool's `Swap` / `Mint` / `Burn` logs into the +//! Phase 3 [`StateUpdate`] vocabulary, maintaining the slots a +//! swap simulation reads: +//! +//! - **`Swap`** (stateless) → a masked `slot0` write (new `sqrtPriceX96` + `tick`, +//! **preserving** the observation index and the `unlocked` flag) plus an +//! absolute `liquidity` write (the event carries post-swap liquidity). +//! - **`Mint`/`Burn`** (stateful, reads the [`StateView`]) → per-tick +//! `liquidityGross` / `liquidityNet`, the `initialized` flag, the `tickBitmap` +//! word bit, and the global `liquidity` (conditional on the current tick). +//! +//! The decoder dispatches by emitting address: a log from a pool not registered +//! via [`with_pool`](UniswapV3Decoder::with_pool) decodes to nothing. It matches +//! events by topic0 (`Swap`/`Mint`/`Burn` signature hashes) and decodes with +//! [`SolEvent`]. +//! +//! # `slot0` bit layout (Uniswap / Pancake) +//! +//! `sqrtPriceX96` = bits [0,160), `tick` (int24) = bits [160,184), and the +//! observation index / cardinality / fee-protocol / **`unlocked`** flag occupy +//! bits [184,256). The `Swap` handler masks the low 184 bits, so the high bits — +//! crucially `unlocked` — survive. Clobbering `unlocked` to 0 would make a +//! subsequent quote/swap revert `LOK`; that is the headline reason `Swap` uses a +//! [`SlotMasked`](crate::StateUpdate::SlotMasked) rather than an absolute write. +//! +//! # Tick word packing +//! +//! Tick slot **+0** packs `liquidityGross` (uint128) = bits [0,128) and +//! `liquidityNet` (int128, two's-complement) = bits [128,256). `Mint` adds +//! `amount` to gross at both ticks and to net at the lower / from net at the +//! upper; `Burn` is the inverse. These are recomputed against the **current** +//! cached word read through the [`StateView`] and emitted as absolute `Slot` +//! writes. The `initialized` flag lives at tick slot **+3**, bit 248 (matching +//! `inject_v3_ticks`); the `tickBitmap` is keyed by the compressed tick +//! `tick / tick_spacing`. +//! +//! # Cold-aware +//! +//! When a needed word is cold ([`StateView::storage`] → `None`), the update is +//! **not** computed against an assumed value — it is skipped and surfaced. Masked +//! sub-word updates (bitmap / initialized) surface as their natural +//! [`SkippedMask`](crate::SkippedMask); the absolute tick-word / global-liquidity +//! writes that cannot be computed surface as a `SkippedMask` with +//! `mask == U256::MAX, value == U256::ZERO` (the "could-not-compute" cold marker — +//! see [`SkippedMask`](crate::SkippedMask)). A pool installed with +//! `StorageCleared` storage reads an unseeded slot as `Some(ZERO)` (hot zero), so +//! tick maintenance proceeds from zero; only a pool with no local account reads +//! cold. +//! +//! # Known limitation (§6.4) +//! +//! Event-derived tick maintenance does **not** reconstruct `feeGrowthOutside0/1X128` +//! (tick slots +1/+2), `secondsOutside`, or oracle observations — these are not +//! derivable from `Mint`/`Burn`/`Swap`. **Swap price/liquidity quoting is +//! unaffected** (the swap-amount math does not depend on `feeGrowthOutside`); fee +//! accounting and `collect` are not maintained. Sampled +//! [`reconcile`](crate::events::EventPipeline::reconcile) and reorg +//! [`reorg_to`](crate::events::EventPipeline::reorg_to) are the backstop. See +//! `KNOWN_ISSUES.md`. + +use std::collections::HashMap; + +use alloy_primitives::{Address, Log, U256}; +use alloy_sol_types::{SolEvent, sol}; + +use crate::cache::{ + PANCAKE_V3_LIQUIDITY_SLOT, PANCAKE_V3_TICK_BITMAP_BASE_SLOT, PANCAKE_V3_TICKS_BASE_SLOT, + V3_LIQUIDITY_SLOT, V3_SLOT0_SLOT, V3_TICK_BITMAP_BASE_SLOT, V3_TICKS_BASE_SLOT, + v3_tick_bitmap_storage_key_with_base, v3_tick_info_storage_keys_with_base, +}; +use crate::events::{EventDecoder, StateView}; +use crate::state_update::StateUpdate; + +sol! { + event Swap(address indexed sender, address indexed recipient, int256 amount0, int256 amount1, uint160 sqrtPriceX96, uint128 liquidity, int24 tick); + event Mint(address sender, address indexed owner, int24 indexed tickLower, int24 indexed tickUpper, uint128 amount, uint256 amount0, uint256 amount1); + event Burn(address indexed owner, int24 indexed tickLower, int24 indexed tickUpper, uint128 amount, uint256 amount0, uint256 amount1); +} + +/// Bit position where the `tick` field starts in a packed `slot0` word. +const SLOT0_TICK_SHIFT: usize = 160; +/// Number of low bits of `slot0` owned by `sqrtPriceX96` ‖ `tick` +/// (`[0,160)` + `[160,184)`); the bits above are preserved by the swap mask. +const SLOT0_PRICE_TICK_BITS: usize = 184; +/// Bit position of the `initialized` flag in tick slot +3. +const TICK_INITIALIZED_BIT: usize = 248; + +/// Per-pool V3 storage layout (slot bases + tick spacing). +/// +/// Uniswap V3 and PancakeSwap V3 share the `slot0` bit layout (only the base slot +/// numbers differ — Pancake's `uint32 feeProtocol` shifts subsequent slots by +1). +/// `tick_spacing` is required for `tickBitmap` word/bit math (the bitmap is keyed +/// by the compressed tick `tick / tick_spacing`). +#[derive(Clone, Debug)] +pub struct UniswapV3Layout { + /// Storage slot of `slot0` (packed price / tick / observation / unlocked). + pub slot0_slot: U256, + /// Storage slot of the global `liquidity`. + pub liquidity_slot: U256, + /// Base slot of the `ticks` mapping (`mapping(int24 => Tick.Info)`). + pub ticks_base_slot: U256, + /// Base slot of the `tickBitmap` mapping (`mapping(int16 => uint256)`). + pub tick_bitmap_base_slot: U256, + /// The pool's `tickSpacing` (for compressed-tick bitmap word/bit math). + pub tick_spacing: i32, +} + +impl UniswapV3Layout { + /// The canonical Uniswap V3 layout for a pool with the given `tick_spacing`. + pub fn uniswap(tick_spacing: i32) -> Self { + Self { + slot0_slot: V3_SLOT0_SLOT, + liquidity_slot: V3_LIQUIDITY_SLOT, + ticks_base_slot: V3_TICKS_BASE_SLOT, + tick_bitmap_base_slot: V3_TICK_BITMAP_BASE_SLOT, + tick_spacing, + } + } + + /// The PancakeSwap V3 layout for a pool with the given `tick_spacing` (slots + /// shifted +1 relative to Uniswap; `slot0` stays at slot 0). + pub fn pancake(tick_spacing: i32) -> Self { + Self { + slot0_slot: V3_SLOT0_SLOT, + liquidity_slot: PANCAKE_V3_LIQUIDITY_SLOT, + ticks_base_slot: PANCAKE_V3_TICKS_BASE_SLOT, + tick_bitmap_base_slot: PANCAKE_V3_TICK_BITMAP_BASE_SLOT, + tick_spacing, + } + } +} + +/// Decodes UniswapV3 / PancakeSwap V3 `Swap` / `Mint` / `Burn` logs into targeted +/// [`StateUpdate`]s. +/// +/// Register pools with [`with_pool`](Self::with_pool); a log from an unregistered +/// pool decodes to nothing. +#[derive(Default)] +pub struct UniswapV3Decoder { + /// Per-pool layout. A log from a pool not in this map decodes to nothing. + pools: HashMap, +} + +impl UniswapV3Decoder { + /// Create an empty decoder with no pools registered. + pub fn new() -> Self { + Self::default() + } + + /// Register `pool` with its storage `layout` (builder style). + pub fn with_pool(mut self, pool: Address, layout: UniswapV3Layout) -> Self { + self.pools.insert(pool, layout); + self + } +} + +/// The cold-tick "could-not-compute" marker: a [`StateUpdate::SlotMasked`] with +/// `mask == U256::MAX, value == U256::ZERO`, which `apply_updates` skip-surfaces +/// for a cold slot (see [`SkippedMask`](crate::SkippedMask)). +fn cold_marker(pool: Address, slot: U256) -> StateUpdate { + StateUpdate::slot_masked(pool, slot, U256::MAX, U256::ZERO) +} + +/// Unpack a tick slot +0 word: `(liquidityGross, liquidityNet)`. +fn unpack_tick_word(word: U256) -> (u128, i128) { + let gross = u128::try_from(word & U256::from(u128::MAX)).unwrap_or(0); + let net = u128::try_from((word >> 128) & U256::from(u128::MAX)).unwrap_or(0) as i128; + (gross, net) +} + +/// Pack `(liquidityGross, liquidityNet)` into a tick slot +0 word. +fn pack_tick_word(gross: u128, net: i128) -> U256 { + U256::from(gross) | (U256::from(net as u128) << 128) +} + +/// Convert a `sol!`-decoded int24 tick to `i32` (a tick always fits in i24 ⊂ i32). +fn tick_to_i32(tick: alloy_primitives::aliases::I24) -> i32 { + i128::try_from(tick).unwrap_or(0) as i32 +} + +/// The pre-state context a `Mint`/`Burn` maintenance pass reads against: the pool +/// address, its layout, and the read-only [`StateView`]. +struct LiquidityCtx<'a> { + pool: Address, + layout: &'a UniswapV3Layout, + view: &'a dyn StateView, +} + +impl LiquidityCtx<'_> { + /// Maintenance for one tick endpoint of a `Mint`/`Burn`. `is_burn` selects the + /// sign (mint adds, burn subtracts); `is_lower` selects the `liquidityNet` + /// sign convention (lower += / upper -= on a mint). Appends the recomputed + /// tick-word write plus any `initialized`/bitmap flips (or a cold marker) to + /// `out`. + fn maintain_tick( + &self, + tick: i32, + amount: u128, + is_burn: bool, + is_lower: bool, + out: &mut Vec, + ) { + let keys = v3_tick_info_storage_keys_with_base(tick, self.layout.ticks_base_slot); + let base = keys[0]; + let slot3 = keys[3]; + + // The current packed tick word. Cold → cannot recompute: surface a marker. + let Some(word) = self.view.storage(self.pool, base) else { + out.push(cold_marker(self.pool, base)); + return; + }; + let (gross, net) = unpack_tick_word(word); + + // gross is always +amount on mint, -amount on burn (saturating defensively). + let new_gross = if is_burn { + gross.saturating_sub(amount) + } else { + gross.saturating_add(amount) + }; + // net: lower += amount, upper -= amount on mint; inverse on burn. + let net_delta = amount as i128; + let signed_delta = match (is_burn, is_lower) { + (false, true) => net_delta, // mint lower: + + (false, false) => -net_delta, // mint upper: - + (true, true) => -net_delta, // burn lower: - + (true, false) => net_delta, // burn upper: + + }; + let new_net = net.wrapping_add(signed_delta); + + out.push(StateUpdate::slot( + self.pool, + base, + pack_tick_word(new_gross, new_net), + )); + + // initialized flag (+3, bit 248) + bitmap bit flip on a 0↔positive cross. + let init_mask = U256::from(1) << TICK_INITIALIZED_BIT; + let newly_initialized = gross == 0 && new_gross > 0; + let now_uninitialized = gross > 0 && new_gross == 0; + + if newly_initialized { + out.push(StateUpdate::slot_masked( + self.pool, slot3, init_mask, init_mask, + )); + if let Some(flip) = self.bitmap_flip(tick, true) { + out.push(flip); + } + } else if now_uninitialized { + out.push(StateUpdate::slot_masked( + self.pool, + slot3, + init_mask, + U256::ZERO, + )); + if let Some(flip) = self.bitmap_flip(tick, false) { + out.push(flip); + } + } + } + + /// Build the `tickBitmap` word/bit flip for `tick` (`set` = newly initialized, + /// clear = newly uninitialized). The bitmap is keyed by the compressed tick + /// `tick / tick_spacing` (V3 guarantees `tick % tick_spacing == 0`, so the + /// division is exact). Returns `None` if `tick_spacing` is non-positive + /// (degenerate layout). + fn bitmap_flip(&self, tick: i32, set: bool) -> Option { + if self.layout.tick_spacing <= 0 { + return None; + } + let compressed = tick / self.layout.tick_spacing; + let word_pos = (compressed >> 8) as i16; + let bit_pos = (compressed & 0xFF) as u8; + let key = v3_tick_bitmap_storage_key_with_base(word_pos, self.layout.tick_bitmap_base_slot); + let mask = U256::from(1) << bit_pos; + let value = if set { mask } else { U256::ZERO }; + Some(StateUpdate::slot_masked(self.pool, key, mask, value)) + } + + /// Global-liquidity maintenance for a `Mint`/`Burn`: if the current `slot0` + /// tick is within `[tickLower, tickUpper)`, emit an absolute `liquidity` write + /// of `current ± amount`. Reads `slot0` and `liquidity` through the view; if + /// either is cold, surface a cold marker on the liquidity slot. + fn maintain_global_liquidity( + &self, + tick_lower: i32, + tick_upper: i32, + amount: u128, + is_burn: bool, + out: &mut Vec, + ) { + let liquidity_slot = self.layout.liquidity_slot; + let Some(slot0) = self.view.storage(self.pool, self.layout.slot0_slot) else { + out.push(cold_marker(self.pool, liquidity_slot)); + return; + }; + let current_tick = extract_tick(slot0); + if !(tick_lower <= current_tick && current_tick < tick_upper) { + return; // out of range: global liquidity unchanged. + } + let Some(current_word) = self.view.storage(self.pool, liquidity_slot) else { + out.push(cold_marker(self.pool, liquidity_slot)); + return; + }; + let current = u128::try_from(current_word & U256::from(u128::MAX)).unwrap_or(0); + let new = if is_burn { + current.saturating_sub(amount) + } else { + current.saturating_add(amount) + }; + out.push(StateUpdate::slot( + self.pool, + liquidity_slot, + U256::from(new), + )); + } + + /// Decode a `Mint` or `Burn` (shared tick + liquidity maintenance). + fn decode_liquidity_event( + &self, + tick_lower: i32, + tick_upper: i32, + amount: u128, + is_burn: bool, + ) -> Vec { + let mut out = Vec::new(); + self.maintain_tick(tick_lower, amount, is_burn, true, &mut out); + self.maintain_tick(tick_upper, amount, is_burn, false, &mut out); + self.maintain_global_liquidity(tick_lower, tick_upper, amount, is_burn, &mut out); + out + } +} + +/// Sign-extend the int24 `tick` field (bits [160,184)) out of a packed `slot0`. +fn extract_tick(slot0: U256) -> i32 { + let raw = ((slot0 >> SLOT0_TICK_SHIFT) & U256::from(0x00FF_FFFFu32)).to::(); + // Sign-extend from 24 bits. + if raw & 0x0080_0000 != 0 { + (raw | 0xFF00_0000) as i32 + } else { + raw as i32 + } +} + +impl EventDecoder for UniswapV3Decoder { + fn decode(&self, log: &Log, view: &dyn StateView) -> Vec { + let Some(layout) = self.pools.get(&log.address) else { + return Vec::new(); + }; + let pool = log.address; + let topic0 = match log.topics().first() { + Some(t) => *t, + None => return Vec::new(), + }; + + if topic0 == Swap::SIGNATURE_HASH { + let Ok(swap) = Swap::decode_log_data(&log.data) else { + return Vec::new(); + }; + // slot0: masked write of sqrtPriceX96 [0,160) + tick [160,184), + // preserving observation / feeProtocol / unlocked bits [184,256). + let mask = (U256::from(1) << SLOT0_PRICE_TICK_BITS) - U256::from(1); + let sqrt_price = U256::from_be_slice(swap.sqrtPriceX96.to_be_bytes::<20>().as_slice()); + let tick = tick_to_i32(swap.tick); + let tick24 = U256::from((tick as u32) & 0x00FF_FFFF); + let value = sqrt_price | (tick24 << SLOT0_TICK_SHIFT); + vec![ + StateUpdate::slot_masked(pool, layout.slot0_slot, mask, value), + // liquidity: absolute (the event carries post-swap liquidity). + StateUpdate::slot(pool, layout.liquidity_slot, U256::from(swap.liquidity)), + ] + } else if topic0 == Mint::SIGNATURE_HASH { + let Ok(mint) = Mint::decode_log_data(&log.data) else { + return Vec::new(); + }; + let ctx = LiquidityCtx { pool, layout, view }; + ctx.decode_liquidity_event( + tick_to_i32(mint.tickLower), + tick_to_i32(mint.tickUpper), + mint.amount, + false, + ) + } else if topic0 == Burn::SIGNATURE_HASH { + let Ok(burn) = Burn::decode_log_data(&log.data) else { + return Vec::new(); + }; + let ctx = LiquidityCtx { pool, layout, view }; + ctx.decode_liquidity_event( + tick_to_i32(burn.tickLower), + tick_to_i32(burn.tickUpper), + burn.amount, + true, + ) + } else { + Vec::new() + } + } +} diff --git a/src/freshness.rs b/src/freshness.rs index 302c301..5d5a8a7 100644 --- a/src/freshness.rs +++ b/src/freshness.rs @@ -66,6 +66,7 @@ use crate::cache::{ CallSimulationResult, EvmCache, EvmOverlay, EvmSnapshot, SimStatus, SlotObservationTracker, StorageBatchFetchFn, TxConfig, }; +use crate::state_update::StateUpdate; /// Default minimum observations before the change-frequency data is trusted. pub const DEFAULT_MIN_OBSERVATIONS: u32 = 10; @@ -434,12 +435,17 @@ impl FreshnessPolicy for ObservationDriven { // 4. Results // --------------------------------------------------------------------------- -/// A storage slot whose freshly-fetched value differs from the cached value. +/// A storage slot whose value changed: `old` is the prior cached/snapshot value +/// (`ZERO` if previously uncached), `new` is the resulting value. /// -/// Produced by [`EvmCache::verify_slots`](crate::cache::EvmCache::verify_slots) -/// and by the background validator; `old` is the value the snapshot/cache held, -/// `new` is the value the fetcher returned. -#[derive(Clone, Debug, PartialEq, Eq)] +/// Produced by two paths: the freshness verifier +/// ([`EvmCache::verify_slots`](crate::cache::EvmCache::verify_slots) and the +/// background validator), where `new` is a freshly-fetched value that differed +/// from the cache; and the state-update writer +/// ([`EvmCache::apply_update`](crate::cache::EvmCache::apply_update) / +/// [`apply_updates`](crate::cache::EvmCache::apply_updates)), where `new` is the +/// value just written. +#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] pub struct SlotChange { /// Contract whose storage changed. pub address: Address, @@ -777,12 +783,17 @@ impl FreshnessController { let now = self.clock.now(); // 1. Drain pending corrections into the cache before snapshotting. + // Routed through the unified write primitive (`apply_updates` of + // write-through `Slot`s); behavior-identical to the old + // `inject_storage_batch_fresh`, demonstrating the one write path. { let mut pending = self.pending.lock().unwrap_or_else(|e| e.into_inner()); if !pending.is_empty() { - let injects: Vec<(Address, U256, U256)> = - pending.iter().map(|c| (c.address, c.slot, c.new)).collect(); - cache.inject_storage_batch_fresh(&injects); + let injects: Vec = pending + .iter() + .map(|c| StateUpdate::slot(c.address, c.slot, c.new)) + .collect(); + cache.apply_updates(&injects); pending.clear(); } } @@ -897,9 +908,42 @@ struct ValidatorInput { /// Maximum fixed-point iterations the background validator performs while a /// correction keeps expanding a sim's volatile read set. A backstop against /// pathological contracts that read an unbounded chain of new volatile slots; -/// reaching it yields a best-effort `Corrected` (logged via `tracing::warn!`). +/// reaching it yields [`Validation::Unverified`] (the results have not reached a +/// verified fixed point, so they must not be trusted), logged via `tracing::warn!`. const MAX_VALIDATION_ROUNDS: u32 = 8; +/// Collect batch-fetcher results into a lookup map, requiring **every** requested +/// `(address, slot)` to be present and `Ok`. +/// +/// The validator must never silently trust a gap: a fetch error *or* a slot the +/// fetcher omitted from its response yields `Err(reason)` (mapped to +/// [`Validation::Unverified`] by the caller) rather than defaulting the missing +/// value to zero — a custom fetcher that drops a slot would otherwise produce a +/// false confirmation or correction. +fn collect_fetch_results( + requested: &[(Address, U256)], + results: Vec<(Address, U256, anyhow::Result)>, +) -> Result, String> { + let mut map: HashMap<(Address, U256), U256> = HashMap::new(); + for (addr, slot, value) in results { + match value { + Ok(v) => { + map.insert((addr, slot), v); + } + Err(e) => return Err(format!("fetch failed for {addr}:{slot}: {e}")), + } + } + for &key in requested { + if !map.contains_key(&key) { + return Err(format!( + "fetcher omitted requested slot {}:{}", + key.0, key.1 + )); + } + } + Ok(map) +} + /// The background validation routine. Touches only `Send` data — never the cache. fn run_validator(input: ValidatorInput) -> Validation { let ValidatorInput { @@ -950,21 +994,13 @@ fn run_validator(input: ValidatorInput) -> Validation { return Validation::Confirmed; } - // Fetch fresh values. Any error → Unverified (never trust silently). + // Fetch fresh values. Any error OR any omitted slot → Unverified (never trust + // silently: a missing result must not default to zero). let results = (fetcher)(verify.clone(), validation_block); - let mut fresh: HashMap<(Address, U256), U256> = HashMap::new(); - for (addr, slot, value) in results { - match value { - Ok(v) => { - fresh.insert((addr, slot), v); - } - Err(e) => { - return Validation::Unverified { - reason: format!("fetch failed for {addr}:{slot}: {e}"), - }; - } - } - } + let fresh = match collect_fetch_results(&verify, results) { + Ok(map) => map, + Err(reason) => return Validation::Unverified { reason }, + }; // Checkpoint: cancelled after the fetch returned but before we record any // observations or queue a correction. A cancel seen here discards the @@ -980,7 +1016,8 @@ fn run_validator(input: ValidatorInput) -> Validation { { let mut tracker = tracker.lock().unwrap_or_else(|e| e.into_inner()); for &(addr, slot) in &verify { - let new = fresh.get(&(addr, slot)).copied().unwrap_or(U256::ZERO); + // `collect_fetch_results` guarantees every requested slot is present. + let new = fresh[&(addr, slot)]; let old = snapshot.storage_value(addr, slot).unwrap_or(U256::ZERO); tracker.observe(addr, slot, new, now); if new != old { @@ -1036,26 +1073,35 @@ fn run_validator(input: ValidatorInput) -> Validation { for &(addr, slot, value) in &overrides { overlay.override_slot(addr, slot, value); } - if let Ok((result, access)) = overlay.call_raw_with_access_list_with( + // A host/transact error means the corrected re-run could not execute; + // we must not keep the stale optimistic result and call it "Corrected". + // (A revert/halt is `Ok(..)`, not an `Err`.) → Unverified. + let (result, access) = match overlay.call_raw_with_access_list_with( req.from, req.to, req.calldata.clone(), &req.tx, ) { - results[i] = result_to_sim(result, &access.to_eip2930()); - let new_volatile: Vec<(Address, U256)> = access - .slots - .iter() - .copied() - .filter(|(a, s)| registry.is_volatile(*a, *s, now)) - .collect(); - for &key in &new_volatile { - if !verified.contains(&key) { - new_candidates.insert(key); - } + Ok(v) => v, + Err(e) => { + return Validation::Unverified { + reason: format!("corrected re-run failed for request {i}: {e}"), + }; + } + }; + results[i] = result_to_sim(result, &access.to_eip2930()); + let new_volatile: Vec<(Address, U256)> = access + .slots + .iter() + .copied() + .filter(|(a, s)| registry.is_volatile(*a, *s, now)) + .collect(); + for &key in &new_volatile { + if !verified.contains(&key) { + new_candidates.insert(key); } - sim_reads[i] = new_volatile; } + sim_reads[i] = new_volatile; } // No sim read a changed slot (the change came from the predicted @@ -1064,14 +1110,21 @@ fn run_validator(input: ValidatorInput) -> Validation { if !any_rerun || new_candidates.is_empty() { break; } - // Results already reflect every override applied so far. Stop here rather - // than expanding the verified set further when the cap is reached. + // The fixed point was not reached within the cap: corrections kept opening + // new volatile slots. The results still rest on un-verified state, so we + // must NOT return a trusted `Corrected`. Return `Unverified` without + // queuing any pending corrections (matching the fetch-error paths); the + // still-volatile slots are re-discovered and re-fetched on the next run. if round >= MAX_VALIDATION_ROUNDS { tracing::warn!( rounds = round, - "freshness validator hit fixed-point iteration cap; returning best-effort Corrected" + "freshness validator exceeded fixed-point round cap; returning Unverified" ); - break; + return Validation::Unverified { + reason: format!( + "freshness validation exceeded fixed-point round cap ({MAX_VALIDATION_ROUNDS})" + ), + }; } // Checkpoint: cancelled mid-loop. Results so far reflect the applied @@ -1080,22 +1133,14 @@ fn run_validator(input: ValidatorInput) -> Validation { return Validation::Confirmed; } - // Fetch the newly-discovered candidates; any error → Unverified. + // Fetch the newly-discovered candidates; any error OR omitted slot → + // Unverified (a missing result must not default to zero). let new_vec: Vec<(Address, U256)> = new_candidates.into_iter().collect(); let fetched = (fetcher)(new_vec.clone(), validation_block); - let mut new_fresh: HashMap<(Address, U256), U256> = HashMap::new(); - for (addr, slot, value) in fetched { - match value { - Ok(v) => { - new_fresh.insert((addr, slot), v); - } - Err(e) => { - return Validation::Unverified { - reason: format!("fetch failed for {addr}:{slot}: {e}"), - }; - } - } - } + let new_fresh = match collect_fetch_results(&new_vec, fetched) { + Ok(map) => map, + Err(reason) => return Validation::Unverified { reason }, + }; // Diff + observe the newly fetched slots, growing the changed set. let mut grew = false; @@ -1103,7 +1148,8 @@ fn run_validator(input: ValidatorInput) -> Validation { let mut tracker = tracker.lock().unwrap_or_else(|e| e.into_inner()); for &(addr, slot) in &new_vec { verified.insert((addr, slot)); - let new = new_fresh.get(&(addr, slot)).copied().unwrap_or(U256::ZERO); + // `collect_fetch_results` guarantees every requested slot is present. + let new = new_fresh[&(addr, slot)]; let old = snapshot.storage_value(addr, slot).unwrap_or(U256::ZERO); tracker.observe(addr, slot, new, now); if new != old { diff --git a/src/lib.rs b/src/lib.rs index 434c451..f138977 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -45,6 +45,15 @@ //! - [`freshness`] — the four-layer freshness model (classification, observation, //! policy, mechanism) and the optimistic verify-and-rerun execution loop with //! deferred validation. +//! - [`state_update`] — the generic state-mutation vocabulary (`StateUpdate` / +//! `AccountPatch` / `PurgeScope`, plus relative `SlotDelta` read-modify-write and +//! masked `SlotMasked` writes) applied by `EvmCache::apply_update` / +//! `apply_updates` / `modify_slot`, with a structured `StateDiff` output +//! (Pillar B.1). +//! - [`events`] — the event → state pipeline (Pillar B.2): `EventDecoder` / +//! `StateView` / `DecoderRegistry` decode an on-chain `Log` into `StateUpdate`s, +//! and `EventPipeline` ingests, reorg-purges, and reconciles a block's logs. +//! Ships an ERC-20 `Transfer` decoder and (under `protocols`) a UniswapV3 adapter. //! - [`inspector`] — an [`Inspector`](revm::Inspector) that captures ERC20 //! `Transfer` events to reconstruct balance deltas from a simulation. //! - [`multicall`] — batched read-only calls through Multicall3. @@ -98,14 +107,27 @@ pub mod cache; pub mod create3; pub mod deploy; pub mod errors; +pub mod events; pub mod freshness; pub mod inspector; pub mod multicall; pub mod prefetch_registry; +pub mod state_update; pub use access_set::StorageAccessList; +pub use events::erc20::Erc20TransferDecoder; +#[cfg(feature = "protocols")] +pub use events::uniswap_v3::{UniswapV3Decoder, UniswapV3Layout}; +pub use events::{ + BlockDigest, DecoderRegistry, EventDecoder, EventPipeline, ReconcileReport, ReorgConfig, + StateView, +}; pub use freshness::{ AlwaysVerify, BlockClock, FreshnessClock, FreshnessController, FreshnessParams, FreshnessPolicy, FreshnessRegistry, NeverVerify, ObservationDriven, SimRequest, SlotChange, SpeculativeSim, Validation, Validity, WallClock, }; +pub use state_update::{ + AccountChange, AccountPatch, PurgeRecord, PurgeScope, SkippedAccountPatch, SkippedBalanceDelta, + SkippedDelta, SkippedMask, SlotDelta, StateDiff, StateUpdate, +}; diff --git a/src/prefetch_registry.rs b/src/prefetch_registry.rs index ba5ab86..04fa1ab 100644 --- a/src/prefetch_registry.rs +++ b/src/prefetch_registry.rs @@ -16,6 +16,7 @@ use std::collections::{HashMap, HashSet}; use std::path::Path; use alloy_primitives::{Address, U256}; +use anyhow::{Context as _, Result}; use serde::{Deserialize, Serialize}; use tracing::{debug, info, warn}; @@ -80,36 +81,27 @@ impl PrefetchRegistry { /// Persist the registry to `path` in bincode format, creating parent /// directories as needed. /// - /// This is best-effort: I/O and serialization failures (unwritable parent - /// directory, failed write, or a serialization error) are logged at `warn` - /// and swallowed rather than returned, so a save failure leaves stale or - /// missing on-disk data that [`load`](Self::load) will silently treat as an - /// empty registry on the next cycle. - pub fn save(&self, path: &Path) { - if let Some(parent) = path.parent() - && let Err(e) = std::fs::create_dir_all(parent) - { - warn!(error = %e, "Failed to create prefetch registry directory"); - return; - } - match bincode::serialize(self) { - Ok(data) => { - if let Err(e) = std::fs::write(path, data) { - warn!(error = %e, "Failed to persist prefetch registry"); - } else { - let total_slots: usize = - self.phases.values().map(|al| al.slots.len()).sum::() - + self - .keyed_phases - .values() - .flat_map(|m| m.values()) - .map(|al| al.slots.len()) - .sum::(); - debug!(total_slots, "Saved prefetch registry"); - } - } - Err(e) => warn!(error = %e, "Failed to serialize prefetch registry"), + /// Returns an error if the parent directory cannot be created, serialization + /// fails, or the write fails. + pub fn save(&self, path: &Path) -> Result<()> { + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent).with_context(|| { + format!("failed to create prefetch registry directory {parent:?}") + })?; } + let data = bincode::serialize(self).context("failed to serialize prefetch registry")?; + std::fs::write(path, data) + .with_context(|| format!("failed to persist prefetch registry to {path:?}"))?; + + let total_slots: usize = self.phases.values().map(|al| al.slots.len()).sum::() + + self + .keyed_phases + .values() + .flat_map(|m| m.values()) + .map(|al| al.slots.len()) + .sum::(); + debug!(total_slots, "Saved prefetch registry"); + Ok(()) } /// Record the aggregated access list for `phase`, **overwriting** any access @@ -352,7 +344,7 @@ mod tests { sal.slots.insert((key, U256::from(99))); registry.record_keyed("per_target", key, sal); - registry.save(&path); + registry.save(&path).expect("save registry"); let loaded = PrefetchRegistry::load(&path); assert_eq!(loaded.phases.len(), 1); @@ -374,6 +366,26 @@ mod tests { let _ = std::fs::remove_dir(&dir); } + #[test] + fn save_reports_write_failures() { + let dir = std::env::temp_dir().join("evm_fork_cache_test_prefetch_registry_write_error"); + let _ = std::fs::remove_dir_all(&dir); + let _ = std::fs::remove_file(&dir); + std::fs::write(&dir, b"not a directory").expect("create file path conflict"); + + let registry = PrefetchRegistry::default(); + let path = dir.join("registry.bin"); + let err = registry + .save(&path) + .expect_err("save must report write failure"); + assert!( + err.to_string().contains("directory") || err.to_string().contains("Not a directory"), + "unexpected error: {err:#}" + ); + + let _ = std::fs::remove_file(&dir); + } + #[test] fn test_load_missing_file_returns_default() { let path = std::path::Path::new("/tmp/nonexistent_prefetch_registry.bin"); diff --git a/src/state_update.rs b/src/state_update.rs new file mode 100644 index 0000000..2641023 --- /dev/null +++ b/src/state_update.rs @@ -0,0 +1,910 @@ +//! Targeted state-mutation vocabulary and the structured diff it produces +//! (Pillar B.1 — the *writer half* of the event → state pipeline). +//! +//! This module defines the small, generic vocabulary a future event decoder +//! emits and [`EvmCache::apply_update`](crate::cache::EvmCache::apply_update) +//! consumes, plus the [`StateDiff`] that records what an apply actually changed. +//! It is pure data and logic on itself: it carries **no** protocol or event +//! knowledge and has no dependency on the cache or the `protocols` feature, so +//! it builds under `--no-default-features`. +//! +//! # The vocabulary +//! +//! A [`StateUpdate`] is one targeted mutation: +//! +//! - [`StateUpdate::Slot`] — set a single storage slot, authoritative across +//! both cache layers. +//! - [`StateUpdate::Account`] — apply a partial [`AccountPatch`] +//! (`balance`/`nonce`/`code`, each optional) to an already-known account. +//! - [`StateUpdate::AccountUpsert`] — intentionally materialize a cold account +//! from a partial [`AccountPatch`]. +//! - [`StateUpdate::Purge`] — drop cached state at a [`PurgeScope`] so the next +//! read re-fetches. +//! +//! # The dual-layer write-through policy +//! +//! [`apply_update`](crate::cache::EvmCache::apply_update) applies `Slot` writes +//! through with one consistent rule: the BlockchainDb backend (layer 2) is +//! written **always**; the CacheDB overlay (layer 1) is written **only if an +//! overlay account already exists** for the address. A new overlay account is +//! never materialized for a slot write — the read path falls through to the +//! backend for an absent overlay entry, so a backend-only write is authoritative, +//! and materializing an overlay entry would pollute layer 1 and could shadow +//! later RPC reads. (This mirrors the established +//! [`inject_storage_batch_fresh`](crate::cache::EvmCache::inject_storage_batch_fresh) +//! semantics.) +//! +//! `Account` patches follow the same overlay-if-present write-through policy +//! once the account is already present in either layer. If the account is absent +//! from **both** layers, the patch is skipped and surfaced in +//! [`StateDiff::skipped_accounts`]. Use [`StateUpdate::AccountUpsert`] when the +//! caller intentionally wants to materialize a cold/default account. +//! +//! # The output +//! +//! Every apply returns a [`StateDiff`] of the changes it actually made: the +//! [`SlotChange`]s, [`AccountChange`]s, and [`PurgeRecord`]s. **Only real changes +//! are recorded** — re-applying a value the cache already holds yields an empty +//! diff, so idempotence is observable. +//! +//! # Relative updates / cold-aware read-modify-write +//! +//! Some callers learn only a *delta* (an ERC-20 `Transfer` log carries the +//! transferred `amount`, not the resulting balances), so the vocabulary also +//! supports *relative* updates: [`StateUpdate::SlotDelta`] reads the current slot +//! value, applies a saturating [`SlotDelta`] (`Add` clamps at `U256::MAX`, `Sub` +//! at `U256::ZERO`), and writes the result back through both layers. The general +//! closure form is +//! [`EvmCache::modify_slot`](crate::cache::EvmCache::modify_slot). +//! +//! A relative update is only valid against a value the cache *actually holds*. An +//! un-fetched ("cold") slot has no value, and applying a delta to it would compute +//! `0 ± amount`, write a wrong value, and (write-through) make it authoritative — +//! silently corrupting state. So relative application is **cold-aware**: a +//! `SlotDelta` on a cold slot is **not applied**; it is recorded in +//! [`StateDiff::skipped`] as a [`SkippedDelta`] so the caller can fetch+seed the +//! true value (the next read otherwise lazily fetches it). `modify_slot` hands its +//! closure an `Option` (`None` when cold) and lets the caller decide. +//! +//! # Masked writes to packed words +//! +//! A storage slot often packs several fields (a UniswapV3 `slot0` holds +//! `sqrtPriceX96`, `tick`, an observation index, and the `unlocked` flag in one +//! word). A decoder that learns only some of those fields must update *just* its +//! bits without clobbering the rest. [`StateUpdate::SlotMasked`] is the +//! cold-aware masked read-modify-write for exactly this: it computes +//! `new = (old & !mask) | (value & mask)`, touching only the `mask` bits. Like +//! [`SlotDelta`](StateUpdate::SlotDelta) it is cold-aware — the un-masked bits of +//! a slot absent from both layers are unknown, so a masked write to a cold slot +//! is **not** applied; it is surfaced in [`StateDiff::skipped_masks`] as a +//! [`SkippedMask`]. (A full-mask `SlotMasked` matches an absolute +//! [`Slot`](StateUpdate::Slot) write on a hot slot but still skips on a cold one.) +//! +//! The same relative, cold-aware rule extends to an account's **native balance**: +//! [`StateUpdate::BalanceDelta`] (and the closure form +//! [`EvmCache::modify_account_balance`](crate::cache::EvmCache::modify_account_balance)) +//! read-modify-write `AccountInfo::balance`, preserving nonce/code. "Cold" here +//! means the account is absent from *both* layers (its balance is unknown); a +//! `BalanceDelta` on a cold account is **not applied** — it is surfaced in +//! [`StateDiff::skipped_balances`] as a [`SkippedBalanceDelta`]. This avoids +//! materializing a default account that would mask the real on-chain one. +//! +//! # Checking for skips +//! +//! Because a cold-skipped relative update produces **no** change, it is invisible +//! to the natural [`StateDiff::is_empty`] / [`StateDiff::len`] success check (those +//! are changes-only). A caller applying relative updates **must** therefore check +//! [`StateDiff::has_skipped`] (or inspect [`skipped`](StateDiff::skipped), +//! [`skipped_balances`](StateDiff::skipped_balances), +//! [`skipped_masks`](StateDiff::skipped_masks), or +//! [`skipped_accounts`](StateDiff::skipped_accounts)) — a cold target was +//! dropped, not applied, and a silently-dropped balance/account update can break +//! conservation. [`StateDiff::is_fully_applied`] and +//! [`StateDiff::skipped_len`] are the companions. +//! +//! # Boundary — events are Phase 4 +//! +//! This is the vocabulary a Phase 4 `EventDecoder` will *emit into*; Phase 3 +//! does not decode events. Nothing here parses a `Log` or knows a protocol's +//! storage layout — that is the *reader half* of Pillar B and lands later. + +use alloy_primitives::{Address, B256, Bytes, U256}; + +use crate::freshness::SlotChange; + +/// A relative storage-slot mutation: read the current value, transform it, and +/// write it back. +/// +/// Both directions **saturate** rather than wrap: `Add` clamps at `U256::MAX` +/// and `Sub` clamps at `U256::ZERO`. This is the delta a caller derives from an +/// event (e.g. an ERC-20 `Transfer` amount) without knowing the resulting +/// absolute value. It is applied by [`StateUpdate::SlotDelta`] (cold-aware — see +/// the module docs). +/// +/// ``` +/// use alloy_primitives::U256; +/// use evm_fork_cache::SlotDelta; +/// +/// assert_eq!(SlotDelta::Add(U256::from(50)).apply(U256::from(100)), U256::from(150)); +/// assert_eq!(SlotDelta::Sub(U256::from(50)).apply(U256::from(30)), U256::ZERO); +/// assert_eq!(SlotDelta::Add(U256::from(10)).apply(U256::MAX), U256::MAX); +/// ``` +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub enum SlotDelta { + /// Add to the current value, saturating at `U256::MAX`. + Add(U256), + /// Subtract from the current value, saturating at `U256::ZERO`. + Sub(U256), +} + +impl SlotDelta { + /// Apply the (saturating) delta to a current value. + /// + /// `Add` uses `saturating_add` (clamps at `U256::MAX`); `Sub` uses + /// `saturating_sub` (clamps at `U256::ZERO`). + pub fn apply(self, current: U256) -> U256 { + match self { + SlotDelta::Add(amount) => current.saturating_add(amount), + SlotDelta::Sub(amount) => current.saturating_sub(amount), + } + } +} + +/// A single targeted mutation to cached EVM state. +/// +/// The vocabulary an event decoder (Phase 4) emits and +/// [`EvmCache::apply_update`](crate::cache::EvmCache::apply_update) consumes. +/// Generic: carries no protocol or event knowledge. +/// +/// The enum is `#[non_exhaustive]`: new variants (e.g. a code-only convenience) +/// may be added pre-1.0 without a breaking change. +/// +/// ``` +/// use alloy_primitives::{Address, U256}; +/// use evm_fork_cache::{AccountPatch, PurgeScope, StateUpdate}; +/// +/// let pool = Address::repeat_byte(0x01); +/// +/// // A storage-slot write (authoritative across both cache layers). +/// let slot = StateUpdate::slot(pool, U256::from(0), U256::from(42)); +/// +/// // A balance-only account patch (nonce and code left untouched). +/// let bal = StateUpdate::balance(pool, U256::from(1_000)); +/// assert_eq!( +/// bal, +/// StateUpdate::Account { address: pool, patch: AccountPatch::default().balance(U256::from(1_000)) }, +/// ); +/// +/// // Drop just two storage slots so the next read re-fetches them. +/// let purge = StateUpdate::purge(pool, PurgeScope::Slots(vec![U256::from(0), U256::from(1)])); +/// # let _ = (slot, purge); +/// ``` +#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[non_exhaustive] +pub enum StateUpdate { + /// Set one storage slot to `value`, authoritative across both cache layers. + Slot { + /// Contract whose storage is written. + address: Address, + /// Storage slot key. + slot: U256, + /// New slot value. + value: U256, + }, + /// Apply a *relative* (saturating) mutation to one storage slot. + /// + /// Read-modify-write: the current value is read, the [`SlotDelta`] applied, + /// and the result written back through both layers. **Cold-aware** — a delta + /// on a slot absent from both layers is not applied; it is surfaced in + /// [`StateDiff::skipped`] instead (see the module docs). + SlotDelta { + /// Contract whose storage is written. + address: Address, + /// Storage slot key. + slot: U256, + /// The relative, saturating mutation to apply to the current value. + delta: SlotDelta, + }, + /// Apply a *relative* (saturating) mutation to an account's **native balance**. + /// + /// Read-modify-write: the current `AccountInfo::balance` is read, the + /// [`SlotDelta`] applied, and the result written back through both layers + /// (nonce and code preserved). **Cold-aware** — "cold" here means the account + /// is absent from *both* layers (its balance is unknown). A `BalanceDelta` on a + /// cold account is not applied; it is surfaced in + /// [`StateDiff::skipped_balances`] instead (so no default account is + /// materialized to mask the real on-chain one — see the module docs). + BalanceDelta { + /// Account whose native balance is mutated. + address: Address, + /// The relative, saturating mutation to apply to the current balance. + delta: SlotDelta, + }, + /// Set only the `mask` bits of a storage slot to the corresponding bits of + /// `value`, preserving the rest: `new = (old & !mask) | (value & mask)`. + /// + /// A *masked* read-modify-write: it lets a pure decoder express a partial + /// update to a **packed** storage word (e.g. a UniswapV3 `slot0`, which packs + /// `sqrtPriceX96`, `tick`, the observation index, and the `unlocked` flag into + /// one slot) without knowing or clobbering the bits it does not own. + /// + /// **Cold-aware** — a masked write to a slot absent from *both* cache layers is + /// **not** applied (the un-masked bits are unknown, so the result cannot be + /// computed); it is surfaced in [`StateDiff::skipped_masks`] as a + /// [`SkippedMask`] instead. A masked write with `mask == U256::MAX` equals an + /// absolute [`Slot`](Self::Slot) write on a *hot* slot, but **still skips** on a + /// cold one (unlike [`Slot`](Self::Slot), which writes unconditionally). Use + /// [`Slot`](Self::Slot) for an unconditional absolute write; use `SlotMasked` + /// when neighbouring bits must be preserved. + SlotMasked { + /// Contract whose storage is written. + address: Address, + /// Storage slot key. + slot: U256, + /// Which bits of the slot to overwrite (1 = take from `value`). + mask: U256, + /// The bits to write (only the bits selected by `mask` are applied). + value: U256, + }, + /// Patch an already-known account's balance/nonce/code (partial — see + /// [`AccountPatch`]). + /// + /// Cold-aware: if the account is absent from **both** layers, the patch is not + /// applied and is surfaced in [`StateDiff::skipped_accounts`] as a + /// [`SkippedAccountPatch`]. Use [`StateUpdate::AccountUpsert`] when + /// materializing a cold/default account is intentional. + Account { + /// Account to patch. + address: Address, + /// The partial mutation: each `Some` field overwrites, `None` leaves it. + patch: AccountPatch, + }, + /// Apply an [`AccountPatch`], materializing a cold account when needed. + /// + /// This is the explicit escape hatch for callers that really do want a + /// default account to become authoritative in the backend (for example a + /// synthetic test account). Normal event-derived account patches should use + /// [`StateUpdate::Account`] so a cold account is skipped instead of masking a + /// future RPC fetch. + AccountUpsert { + /// Account to patch or create. + address: Address, + /// The partial mutation: each `Some` field overwrites, `None` leaves it. + patch: AccountPatch, + }, + /// Purge cached state for `address` at `scope`; the next read re-fetches. + Purge { + /// Account whose cached state is purged. + address: Address, + /// What part of the cached state to remove. + scope: PurgeScope, + }, +} + +impl StateUpdate { + /// Construct a [`StateUpdate::Slot`] that sets `(address, slot)` to `value`. + pub fn slot(address: Address, slot: U256, value: U256) -> Self { + Self::Slot { + address, + slot, + value, + } + } + + /// Construct a [`StateUpdate::SlotDelta`] that applies `delta` relative to the + /// current value of `(address, slot)`. + pub fn slot_delta(address: Address, slot: U256, delta: SlotDelta) -> Self { + Self::SlotDelta { + address, + slot, + delta, + } + } + + /// Construct a [`StateUpdate::SlotMasked`] that sets only the `mask` bits of + /// `(address, slot)` to the corresponding bits of `value`. + pub fn slot_masked(address: Address, slot: U256, mask: U256, value: U256) -> Self { + Self::SlotMasked { + address, + slot, + mask, + value, + } + } + + /// Construct a [`StateUpdate::BalanceDelta`] that applies `delta` relative to + /// the account's current native balance. + pub fn balance_delta(address: Address, delta: SlotDelta) -> Self { + Self::BalanceDelta { address, delta } + } + + /// Construct a [`StateUpdate::Account`] that patches only the balance. + pub fn balance(address: Address, value: U256) -> Self { + Self::Account { + address, + patch: AccountPatch::default().balance(value), + } + } + + /// Construct a [`StateUpdate::Account`] that patches only the nonce. + pub fn nonce(address: Address, nonce: u64) -> Self { + Self::Account { + address, + patch: AccountPatch::default().nonce(nonce), + } + } + + /// Construct a [`StateUpdate::Account`] that patches only the runtime code + /// (the code hash is recomputed from `code` when applied). + pub fn code(address: Address, code: Bytes) -> Self { + Self::Account { + address, + patch: AccountPatch::default().code(code), + } + } + + /// Construct a [`StateUpdate::Account`] from a prebuilt [`AccountPatch`]. + pub fn account(address: Address, patch: AccountPatch) -> Self { + Self::Account { address, patch } + } + + /// Construct a [`StateUpdate::AccountUpsert`] from a prebuilt + /// [`AccountPatch`]. + /// + /// Use this only when materializing an account absent from both layers is the + /// desired behavior. For normal patches to known accounts, use + /// [`account`](Self::account). + pub fn account_upsert(address: Address, patch: AccountPatch) -> Self { + Self::AccountUpsert { address, patch } + } + + /// Construct a [`StateUpdate::Purge`] for `address` at `scope`. + pub fn purge(address: Address, scope: PurgeScope) -> Self { + Self::Purge { address, scope } + } +} + +/// A partial account mutation: each `Some` field overwrites the cached value, +/// each `None` leaves it unchanged. Setting `code` recomputes the code hash; +/// `Some(empty bytes)` clears code to the empty-code hash. +/// +/// Partial (rather than a full revm `AccountInfo`) because the Pillar B driver +/// is events, which usually carry *one* field (a `Transfer` changes a balance, +/// not nonce/code). This avoids forcing a caller to reconstruct a full +/// `AccountInfo` and keeps revm's type out of the public vocabulary. +/// +/// The struct is `#[non_exhaustive]`: new fields may be added pre-1.0 without a +/// breaking change. Construct it via [`AccountPatch::default`] + the builders +/// ([`balance`](Self::balance) / [`nonce`](Self::nonce) / [`code`](Self::code)), +/// never a struct literal. +/// +/// # Warning +/// +/// Applying an absolute patch with [`StateUpdate::Account`] on an address absent +/// from **both** cache layers is skipped and surfaced in +/// [`StateDiff::skipped_accounts`]. Use [`StateUpdate::AccountUpsert`] only when +/// default values for un-patched fields (e.g. nonce `0`, empty code) should become +/// authoritative in the backend. +/// +/// ``` +/// use alloy_primitives::{Bytes, U256}; +/// use evm_fork_cache::AccountPatch; +/// +/// let patch = AccountPatch::default() +/// .balance(U256::from(42)) +/// .nonce(7) +/// .code(Bytes::from_static(&[0x60, 0x00])); +/// assert_eq!(patch.balance, Some(U256::from(42))); +/// assert_eq!(patch.nonce, Some(7)); +/// assert_eq!(AccountPatch::default().balance, None); +/// ``` +#[derive(Clone, Debug, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[non_exhaustive] +pub struct AccountPatch { + /// New balance, if set. + pub balance: Option, + /// New nonce, if set. + pub nonce: Option, + /// New runtime code, if set. Setting it recomputes the code hash; empty + /// bytes clear the code to the empty-code hash. + pub code: Option, +} + +impl AccountPatch { + /// Set the balance to overwrite (builder style). + pub fn balance(mut self, balance: U256) -> Self { + self.balance = Some(balance); + self + } + + /// Set the nonce to overwrite (builder style). + pub fn nonce(mut self, nonce: u64) -> Self { + self.nonce = Some(nonce); + self + } + + /// Set the runtime code to overwrite (builder style). The code hash is + /// recomputed from these bytes when the patch is applied. + pub fn code(mut self, code: Bytes) -> Self { + self.code = Some(code); + self + } +} + +/// What part of an address's cached state a purge removes. +/// +/// The enum is `#[non_exhaustive]`: new scopes may be added pre-1.0 without a +/// breaking change. +/// +/// # `StorageCleared`/`NotExisting` accounts +/// Purging *storage* ([`AllStorage`](Self::AllStorage) / [`Slots`](Self::Slots)) +/// removes the slot from the backend so a normal forked account re-fetches it on +/// the next read. For an account revm marks `StorageCleared`/`NotExisting` (a +/// locally-created/cleared account, e.g. after a `CREATE`/selfdestruct), the EVM +/// reads a missing slot as **zero without re-fetching** — its storage is locally +/// complete — so a purged slot reads `0`, not a fresh RPC value. This is correct +/// (such an account has no on-chain storage to refetch), but it means +/// [`Slots`](Self::Slots) does not force a refetch for those accounts. Use +/// [`Account`](Self::Account) (or `purge_account`) to fully drop the account. +#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[non_exhaustive] +pub enum PurgeScope { + /// Full account: `AccountInfo` (balance/nonce/code) **and** all storage. + /// Equivalent to + /// [`EvmCache::purge_account`](crate::cache::EvmCache::purge_account). + Account, + /// All storage slots; account info preserved. Equivalent to + /// [`EvmCache::purge_pool_storage`](crate::cache::EvmCache::purge_pool_storage). + AllStorage, + /// Only the listed storage slots. Equivalent to + /// [`EvmCache::purge_pool_slots`](crate::cache::EvmCache::purge_pool_slots). + Slots(Vec), +} + +/// What an `apply_*` call actually changed. +/// +/// Returned by [`EvmCache::apply_update`](crate::cache::EvmCache::apply_update) +/// and [`apply_updates`](crate::cache::EvmCache::apply_updates). Only real +/// changes are recorded, so a no-op write yields a [`Default`] (empty) diff. +/// +/// The struct is `#[non_exhaustive]`: it has grown fields pre-1.0 +/// ([`skipped`](Self::skipped), [`skipped_balances`](Self::skipped_balances), +/// [`skipped_masks`](Self::skipped_masks), and +/// [`skipped_accounts`](Self::skipped_accounts)) and may grow more. Construct it +/// via [`Default`] + field assignment, never an exhaustive struct literal. +/// +/// # Checking for skips +/// +/// [`is_empty`](Self::is_empty) / [`len`](Self::len) are **changes-only**, so a +/// cold-skipped update ([`SlotDelta`](StateUpdate::SlotDelta) / +/// [`BalanceDelta`](StateUpdate::BalanceDelta) / [`Account`](StateUpdate::Account)) +/// is invisible to them. After applying cold-aware updates, check +/// [`has_skipped`](Self::has_skipped) (or inspect the `skipped_*` fields) — a cold +/// target was dropped, not applied. +#[derive(Clone, Debug, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[non_exhaustive] +pub struct StateDiff { + /// Storage slots whose value changed (`old != new`). + pub slots: Vec, + /// Accounts whose balance/nonce/code-hash changed. + pub accounts: Vec, + /// Purges performed, with what they removed. + pub purged: Vec, + /// Relative slot updates ([`StateUpdate::SlotDelta`]) that were **not** applied + /// because the target slot's current value was unknown (cold). This is + /// informational metadata, not a change: it does **not** affect + /// [`is_empty`](Self::is_empty) / [`len`](Self::len). + pub skipped: Vec, + /// Relative balance updates ([`StateUpdate::BalanceDelta`]) that were **not** + /// applied because the target account was absent from both layers (its balance + /// was unknown). Like [`skipped`](Self::skipped) this is informational + /// metadata, not a change. + pub skipped_balances: Vec, + /// Masked slot updates ([`StateUpdate::SlotMasked`]) that were **not** applied + /// because the target slot's current value was unknown (cold) — the un-masked + /// bits could not be preserved. Like [`skipped`](Self::skipped) this is + /// informational metadata, not a change. + pub skipped_masks: Vec, + /// Account patches ([`StateUpdate::Account`]) that were **not** applied + /// because the account was absent from both layers. Like + /// [`skipped`](Self::skipped) this is informational metadata, not a change. + pub skipped_accounts: Vec, +} + +impl StateDiff { + /// Whether the diff recorded no change at all. + /// + /// Changes-only: counts `slots` + `accounts` + `purged`. A skipped relative + /// update ([`skipped`](Self::skipped) / [`skipped_balances`](Self::skipped_balances)) + /// is informational metadata, not a change, so it does not affect this. + pub fn is_empty(&self) -> bool { + self.slots.is_empty() && self.accounts.is_empty() && self.purged.is_empty() + } + + /// Total number of changed entries (slots + accounts + purges). + /// + /// Changes-only: skipped relative updates are not counted (a skip is not a + /// change). See [`skipped_len`](Self::skipped_len) for the skip count. + pub fn len(&self) -> usize { + self.slots.len() + self.accounts.len() + self.purged.len() + } + + /// Whether any relative update was skipped (slot **or** balance). + /// + /// `true` iff [`skipped`](Self::skipped) or + /// [`skipped_balances`](Self::skipped_balances) is non-empty. A cold-skipped + /// update produces no change, so it is invisible to + /// [`is_empty`](Self::is_empty) — callers applying relative updates should + /// check this to avoid silently dropping a balance update. + pub fn has_skipped(&self) -> bool { + !self.skipped.is_empty() + || !self.skipped_balances.is_empty() + || !self.skipped_masks.is_empty() + || !self.skipped_accounts.is_empty() + } + + /// Total number of skipped relative/masked updates (`skipped` + + /// `skipped_balances` + `skipped_masks` + `skipped_accounts`). + pub fn skipped_len(&self) -> usize { + self.skipped.len() + + self.skipped_balances.len() + + self.skipped_masks.len() + + self.skipped_accounts.len() + } + + /// Whether every relative update in the apply was applied (none skipped). + /// + /// The inverse of [`has_skipped`](Self::has_skipped). + pub fn is_fully_applied(&self) -> bool { + !self.has_skipped() + } + + /// Fold `other` into `self`, concatenating each category. + /// + /// Used by [`apply_updates`](crate::cache::EvmCache::apply_updates) to merge + /// per-update diffs; the concatenation preserves order, so two writes to the + /// same slot record their `old → new` history in sequence. The `skipped`, + /// `skipped_balances`, `skipped_masks`, and `skipped_accounts` metadata are + /// concatenated too. + pub fn merge(&mut self, other: StateDiff) { + self.slots.extend(other.slots); + self.accounts.extend(other.accounts); + self.purged.extend(other.purged); + self.skipped.extend(other.skipped); + self.skipped_balances.extend(other.skipped_balances); + self.skipped_masks.extend(other.skipped_masks); + self.skipped_accounts.extend(other.skipped_accounts); + } +} + +/// An account field delta. Each field is `Some((old, new))` only when it changed. +#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct AccountChange { + /// Account whose fields changed. + pub address: Address, + /// Balance delta `(old, new)`, present only if the balance changed. + pub balance: Option<(U256, U256)>, + /// Nonce delta `(old, new)`, present only if the nonce changed. + pub nonce: Option<(u64, u64)>, + /// Code-hash delta `(old, new)`, present only if the code changed. + pub code_hash: Option<(B256, B256)>, +} + +/// Record of a purge: how much of each layer it removed. +#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct PurgeRecord { + /// Account that was purged. + pub address: Address, + /// The scope that was applied. + pub scope: PurgeScope, + /// Storage slots removed from the BlockchainDb backend (layer 2). + pub slots_removed: usize, + /// Whether an `AccountInfo` was removed (only the [`PurgeScope::Account`] scope). + pub account_removed: bool, +} + +/// A relative update ([`StateUpdate::SlotDelta`]) that could not be applied +/// because the slot's current value is unknown (not cached in either layer). +/// +/// A delta against a cold slot is skipped rather than applied (applying `0 ± +/// amount` would corrupt an unknown value and, write-through, make it +/// authoritative). It is surfaced here so the caller can fetch+seed the true +/// value and retry; otherwise the next read lazily fetches it. +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct SkippedDelta { + /// Contract whose storage slot the delta targeted. + pub address: Address, + /// Storage slot key that was cold. + pub slot: U256, + /// The delta that was not applied. + pub delta: SlotDelta, +} + +/// A relative balance update ([`StateUpdate::BalanceDelta`]) that could not be +/// applied because the account is absent from **both** cache layers (its native +/// balance is unknown). +/// +/// A delta against a cold account is skipped rather than applied (applying it +/// against an assumed-zero balance would corrupt an unknown value, and +/// materializing a default account would mask the real on-chain one). It is +/// surfaced here so the caller can fetch+seed the account and retry. +/// +/// Deliberately **not** `#[non_exhaustive]`: it is a stable, fully-determined leaf +/// record routinely constructed as a struct literal in equality assertions by the +/// test suite and downstream users testing against a returned diff. +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct SkippedBalanceDelta { + /// Account whose native balance the delta targeted. + pub address: Address, + /// The delta that was not applied. + pub delta: SlotDelta, +} + +/// A masked write ([`StateUpdate::SlotMasked`]) that could not be applied because +/// the target slot's current value is unknown (not cached in either layer). +/// +/// A masked write needs the slot's current value to preserve the un-masked bits +/// (`new = (old & !mask) | (value & mask)`); without it the result cannot be +/// computed, so the write is skipped rather than applied against an assumed value. +/// It is surfaced here so the caller can fetch+seed the slot and retry; otherwise +/// the next read lazily fetches the true value. +/// +/// # The `mask == U256::MAX, value == U256::ZERO` cold-tick convention +/// +/// The UniswapV3 adapter (`events::uniswap_v3`) reuses this record as a +/// "could-not-compute" marker for the *absolute* tick-word / global-liquidity +/// writes it must skip when a needed word is cold: it pushes a `SkippedMask` with +/// `mask == U256::MAX` and `value == U256::ZERO`. This avoids adding a fourth skip +/// vector for stateful protocol updates; the count flows through +/// [`StateDiff::skipped_len`] all the same, and the caller re-seeds the pool. +/// +/// Deliberately **not** `#[non_exhaustive]`: it is a stable, fully-determined leaf +/// record routinely constructed as a struct literal in equality assertions by the +/// test suite and downstream users testing against a returned diff. +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct SkippedMask { + /// Contract whose storage slot the masked write targeted. + pub address: Address, + /// Storage slot key that was cold. + pub slot: U256, + /// The mask that was not applied. + pub mask: U256, + /// The value bits that were not applied. + pub value: U256, +} + +/// An account patch ([`StateUpdate::Account`]) that could not be applied because +/// the account is absent from **both** cache layers. +/// +/// A partial patch against a cold account is skipped rather than applied against +/// [`AccountInfo::default`](revm::state::AccountInfo::default), because default +/// nonce/code would become authoritative and mask a later RPC fetch. It is +/// surfaced here so the caller can fetch+seed the account and retry, or opt in to +/// materialization with [`StateUpdate::AccountUpsert`]. +#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +pub struct SkippedAccountPatch { + /// Account whose patch was skipped. + pub address: Address, + /// The patch that was not applied. + pub patch: AccountPatch, +} + +#[cfg(test)] +mod tests { + use super::*; + + fn addr(n: u8) -> Address { + Address::repeat_byte(n) + } + + #[test] + fn account_patch_default_is_all_none() { + let p = AccountPatch::default(); + assert_eq!(p.balance, None); + assert_eq!(p.nonce, None); + assert_eq!(p.code, None); + } + + #[test] + fn account_patch_builders_compose() { + let p = AccountPatch::default() + .balance(U256::from(42)) + .nonce(7) + .code(Bytes::from_static(&[0x60, 0x00])); + assert_eq!(p.balance, Some(U256::from(42))); + assert_eq!(p.nonce, Some(7)); + assert_eq!(p.code, Some(Bytes::from_static(&[0x60, 0x00]))); + } + + #[test] + fn state_update_constructors_produce_expected_variants() { + let a = addr(0xaa); + + assert_eq!( + StateUpdate::slot(a, U256::from(1), U256::from(2)), + StateUpdate::Slot { + address: a, + slot: U256::from(1), + value: U256::from(2), + } + ); + assert_eq!( + StateUpdate::balance(a, U256::from(9)), + StateUpdate::Account { + address: a, + patch: AccountPatch::default().balance(U256::from(9)), + } + ); + assert_eq!( + StateUpdate::account_upsert(a, AccountPatch::default().balance(U256::from(9))), + StateUpdate::AccountUpsert { + address: a, + patch: AccountPatch::default().balance(U256::from(9)), + } + ); + assert_eq!( + StateUpdate::purge(a, PurgeScope::Account), + StateUpdate::Purge { + address: a, + scope: PurgeScope::Account, + } + ); + } + + #[test] + fn state_diff_default_is_empty() { + let d = StateDiff::default(); + assert!(d.is_empty()); + assert_eq!(d.len(), 0); + } + + #[test] + fn state_diff_merge_concatenates_and_counts() { + let a = addr(0xbb); + let mut left = StateDiff::default(); + left.slots.push(SlotChange { + address: a, + slot: U256::from(1), + old: U256::ZERO, + new: U256::from(5), + }); + + let mut right = StateDiff::default(); + right.accounts.push(AccountChange { + address: a, + balance: Some((U256::ZERO, U256::from(3))), + nonce: None, + code_hash: None, + }); + right.purged.push(PurgeRecord { + address: a, + scope: PurgeScope::AllStorage, + slots_removed: 2, + account_removed: false, + }); + + left.merge(right); + assert!(!left.is_empty()); + assert_eq!(left.len(), 3); + assert_eq!(left.slots.len(), 1); + assert_eq!(left.accounts.len(), 1); + assert_eq!(left.purged.len(), 1); + // Concatenation preserves the merged-in slot value. + assert_eq!(left.slots[0].new, U256::from(5)); + } + + #[test] + fn slot_delta_add_applies_saturating() { + assert_eq!( + SlotDelta::Add(U256::from(50)).apply(U256::from(100)), + U256::from(150) + ); + // Saturates at U256::MAX rather than wrapping. + assert_eq!( + SlotDelta::Add(U256::from(10)).apply(U256::MAX - U256::from(1)), + U256::MAX + ); + assert_eq!(SlotDelta::Add(U256::from(5)).apply(U256::MAX), U256::MAX); + } + + #[test] + fn slot_delta_sub_applies_saturating() { + assert_eq!( + SlotDelta::Sub(U256::from(30)).apply(U256::from(100)), + U256::from(70) + ); + // Saturates at zero rather than underflowing. + assert_eq!( + SlotDelta::Sub(U256::from(50)).apply(U256::from(30)), + U256::ZERO + ); + assert_eq!(SlotDelta::Sub(U256::from(1)).apply(U256::ZERO), U256::ZERO); + } + + #[test] + fn state_update_slot_delta_constructor() { + let a = addr(0xcc); + assert_eq!( + StateUpdate::slot_delta(a, U256::from(1), SlotDelta::Add(U256::from(2))), + StateUpdate::SlotDelta { + address: a, + slot: U256::from(1), + delta: SlotDelta::Add(U256::from(2)), + } + ); + } + + #[test] + fn state_diff_merge_extends_skipped_without_counting_it() { + let a = addr(0xdd); + let mut left = StateDiff::default(); + let mut right = StateDiff::default(); + right.skipped.push(SkippedDelta { + address: a, + slot: U256::from(1), + delta: SlotDelta::Sub(U256::from(3)), + }); + + left.merge(right); + assert_eq!(left.skipped.len(), 1); + // A skip is metadata, not a change. + assert!(left.is_empty()); + assert_eq!(left.len(), 0); + } + + #[test] + fn slot_masked_constructor_produces_variant() { + let a = addr(0xee); + assert_eq!( + StateUpdate::slot_masked(a, U256::from(1), U256::from(0xFF), U256::from(0x42)), + StateUpdate::SlotMasked { + address: a, + slot: U256::from(1), + mask: U256::from(0xFF), + value: U256::from(0x42), + } + ); + } + + #[test] + fn state_diff_merge_extends_skipped_masks_without_counting_it() { + let a = addr(0xef); + let mut left = StateDiff::default(); + let mut right = StateDiff::default(); + right.skipped_masks.push(SkippedMask { + address: a, + slot: U256::from(1), + mask: U256::from(0xFF), + value: U256::from(0x42), + }); + + left.merge(right); + assert_eq!(left.skipped_masks.len(), 1); + // A masked skip is metadata, not a change. + assert!(left.is_empty()); + assert_eq!(left.len(), 0); + // But it is discoverable through the skip accessors. + assert!(left.has_skipped()); + assert_eq!(left.skipped_len(), 1); + assert!(!left.is_fully_applied()); + } + + #[test] + fn slot_masked_serde_round_trips() { + let a = addr(0xf0); + let update = StateUpdate::slot_masked(a, U256::from(5), U256::from(0xFF), U256::from(3)); + let json = serde_json::to_string(&update).expect("serialize"); + let back: StateUpdate = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(update, back); + + let mask = SkippedMask { + address: a, + slot: U256::from(1), + mask: U256::from(0xFF), + value: U256::from(2), + }; + let json = serde_json::to_string(&mask).expect("serialize"); + let back: SkippedMask = serde_json::from_str(&json).expect("deserialize"); + assert_eq!(mask, back); + } +} diff --git a/tests/cache_state.rs b/tests/cache_state.rs index cdbb636..8c55b88 100644 --- a/tests/cache_state.rs +++ b/tests/cache_state.rs @@ -8,13 +8,13 @@ mod common; -use alloy_primitives::{Address, Bytes, I256, U256}; -use alloy_sol_types::SolValue; +use alloy_primitives::{Address, B256, Bytes, I256, U256, keccak256}; +use alloy_sol_types::{SolCall, SolValue}; use anyhow::{Context, Result}; use revm::state::{AccountInfo, Bytecode}; use common::{ - MOCK_ERC20_BALANCE_SLOT, balance_of, install_default_account, install_mock_erc20, + MOCK_ERC20_BALANCE_SLOT, MockERC20, balance_of, install_default_account, install_mock_erc20, mock_erc20_creation_code, mock_erc20_runtime, setup_cache, transfer, }; use evm_fork_cache::cache::TxConfig; @@ -123,6 +123,75 @@ async fn simulation_reports_balance_deltas() -> Result<()> { Ok(()) } +#[tokio::test(flavor = "multi_thread")] +async fn balance_delta_simulation_reports_access_list() -> Result<()> { + let mut cache = setup_cache().await?; + let token = Address::repeat_byte(0x44); + let owner = Address::repeat_byte(0x55); + let recipient = Address::repeat_byte(0x66); + + install_default_account(&mut cache, Address::ZERO); + install_default_account(&mut cache, owner); + install_default_account(&mut cache, recipient); + install_mock_erc20(&mut cache, token); + + let balance_slot = U256::from(MOCK_ERC20_BALANCE_SLOT); + cache.insert_mapping_storage_slot(token, balance_slot, owner, U256::from(1_000u64))?; + cache.insert_mapping_storage_slot(token, balance_slot, recipient, U256::ZERO)?; + + let transfer_call = MockERC20::transferCall { + to: recipient, + amount: U256::from(250u64), + }; + let result = cache.simulate_call_with_balance_deltas( + owner, + token, + Bytes::from(transfer_call.abi_encode()), + owner, + [token], + false, + )?; + + assert_eq!( + result.token_deltas.get(&token), + Some(&-I256::from_raw(U256::from(250u64))) + ); + + let owner_balance_slot = B256::from(U256::from_be_bytes( + keccak256((owner, balance_slot).abi_encode()).0, + )); + let recipient_balance_slot = B256::from(U256::from_be_bytes( + keccak256((recipient, balance_slot).abi_encode()).0, + )); + let token_item = result + .access_list + .0 + .iter() + .find(|item| item.address == token) + .expect("access list includes the token account"); + assert!( + token_item.storage_keys.contains(&owner_balance_slot), + "access list includes owner's balance slot" + ); + assert!( + token_item.storage_keys.contains(&recipient_balance_slot), + "access list includes recipient's balance slot" + ); + + assert_eq!( + balance_of(&mut cache, token, owner)?, + U256::from(1_000u64), + "non-committing simulation must not change owner balance" + ); + assert_eq!( + balance_of(&mut cache, token, recipient)?, + U256::ZERO, + "non-committing simulation must not change recipient balance" + ); + + Ok(()) +} + #[tokio::test(flavor = "multi_thread")] async fn set_erc20_balance_with_slot_scan_finds_balance_slot() -> Result<()> { let mut cache = setup_cache().await?; @@ -235,7 +304,7 @@ async fn two_layer_cache_staleness_requires_full_purge() -> Result<()> { // Clearing ONLY the backend leaves the overlay serving stale data. { - let mut storage = cache.blockchain_db().storage().write(); + let mut storage = cache.unchecked_blockchain_db().storage().write(); storage.remove(&token); } assert_eq!( diff --git a/tests/common/mod.rs b/tests/common/mod.rs index 509a1f5..196f3d3 100644 --- a/tests/common/mod.rs +++ b/tests/common/mod.rs @@ -6,7 +6,7 @@ #![allow(dead_code)] use std::collections::HashMap; -use std::sync::Arc; +use std::sync::{Arc, Condvar, Mutex}; use alloy_eips::BlockId; use alloy_primitives::{Address, Bytes, U256, hex}; @@ -128,19 +128,64 @@ pub fn failing_fetcher() -> StorageBatchFetchFn { }) } -/// Build a stub [`StorageBatchFetchFn`] that reports chosen values *and* flips a -/// shared flag the first time it is called. +/// A one-shot synchronous gate: a holder blocks in [`wait`](Gate::wait) until +/// some other thread calls [`release`](Gate::release). Cloning shares the same +/// underlying state, and `release` is sticky — once released, every present and +/// future `wait` returns immediately. /// -/// Used by the Drop-abort test to prove the background validator was cancelled -/// before it ever fetched (so it could not have queued a correction). The -/// returned values otherwise behave exactly like [`stub_fetcher`]. -pub fn tracking_fetcher( +/// Used by the Drop-abort test to make the background validator's fetch +/// deterministically ordered *after* the drop. The fetcher (running on a worker +/// thread) cannot return — and therefore the validator cannot reach its +/// post-fetch checkpoint — until the test has dropped the `SpeculativeSim` and +/// released the gate, eliminating the spawn/poll race regardless of how the +/// multi-thread scheduler interleaves the two threads. +/// +/// Built on a `Mutex` + `Condvar` so the whole thing is `Send + Sync`, +/// which a [`StorageBatchFetchFn`] closure must be. +#[derive(Clone, Default)] +pub struct Gate { + inner: Arc<(Mutex, Condvar)>, +} + +impl Gate { + pub fn new() -> Self { + Self::default() + } + + /// Block until [`release`](Gate::release) has been called (returns + /// immediately if it already has). + pub fn wait(&self) { + let (lock, cv) = &*self.inner; + let mut released = lock.lock().unwrap_or_else(|e| e.into_inner()); + while !*released { + released = cv.wait(released).unwrap_or_else(|e| e.into_inner()); + } + } + + /// Wake any current waiter and let all future waiters pass. + pub fn release(&self) { + let (lock, cv) = &*self.inner; + *lock.lock().unwrap_or_else(|e| e.into_inner()) = true; + cv.notify_all(); + } +} + +/// Build a stub [`StorageBatchFetchFn`] that reports chosen values but blocks on +/// `gate` before returning. +/// +/// Used by the Drop-abort test: by releasing the gate only *after* dropping the +/// `SpeculativeSim`, the test guarantees the validator's fetch completes (and so +/// its post-fetch, correction-queuing checkpoint runs) strictly after the +/// cancel flag is set — so the dropped speculation can never queue a correction, +/// no matter how the scheduler races the two threads. The returned values +/// otherwise behave exactly like [`stub_fetcher`]. +pub fn gated_tracking_fetcher( values: HashMap<(Address, U256), U256>, - called: Arc, + gate: Gate, ) -> StorageBatchFetchFn { Arc::new( move |requests: Vec<(Address, U256)>, _block: Option| { - called.store(true, std::sync::atomic::Ordering::SeqCst); + gate.wait(); requests .into_iter() .map(|(addr, slot)| { diff --git a/tests/cow_snapshot.rs b/tests/cow_snapshot.rs new file mode 100644 index 0000000..892e154 --- /dev/null +++ b/tests/cow_snapshot.rs @@ -0,0 +1,719 @@ +//! Phase 5 (Pillar A) acceptance tests — the **red contract** for copy-on-write +//! snapshots, authored before implementation. +//! +//! The gate is a *differential-equivalence* property: the new, memoized +//! [`EvmCache::create_snapshot`] must be **read-indistinguishable** from the +//! retained reference [`EvmCache::create_snapshot_deep_clone`] after every kind of +//! cache mutation. Because the two use different internal representations (the COW +//! snapshot shares an `Arc`-ed cold base; the reference is a full flatten), they are +//! compared *through reads only* — `storage_value`, overlay `basic`/`storage`, and a +//! `MockERC20` `balanceOf`. Any base-invalidation miss surfaces here as a failed +//! assertion, never as a silent stale read. +//! +//! Also pins the Pillar A.2 overlay-reuse contract: [`EvmOverlay::reset`] recycles +//! an overlay equivalently to a fresh one, and buffer reuse does not change results. +//! +//! All state is injected over a mocked provider — no test touches the network. +//! +//! These reference `create_snapshot_deep_clone` and `EvmOverlay::reset`, which do +//! not exist until the Phase 5 implementation lands; until then this file fails to +//! compile (red), exactly as intended. + +mod common; + +use std::sync::Arc; + +use alloy_primitives::{Address, Bytes, U256, keccak256}; +use alloy_sol_types::{SolCall, SolValue}; +use anyhow::{Result, anyhow}; +use revm::database::AccountState; +use revm::database_interface::Database; +use revm::state::{AccountInfo, Bytecode}; + +use common::{ + MOCK_ERC20_BALANCE_SLOT, MockERC20, install_default_account, install_mock_erc20, setup_cache, + transfer, +}; +use evm_fork_cache::cache::{EvmCache, EvmOverlay, EvmSnapshot}; +use evm_fork_cache::{SlotDelta, StateUpdate}; + +/// `keccak256(abi.encode(owner, slot))` — the hashed mapping slot of +/// `balanceOf[owner]`. +fn mapping_slot(owner: Address, slot: u64) -> U256 { + let key = keccak256((owner, U256::from(slot)).abi_encode()); + U256::from_be_bytes(key.0) +} + +/// Two `AccountInfo`s are equal as the EVM sees them (code identity via code_hash). +fn account_eq(a: &Option, b: &Option) -> bool { + match (a, b) { + (None, None) => true, + (Some(x), Some(y)) => { + x.balance == y.balance + && x.nonce == y.nonce + && x.code_hash == y.code_hash + && x.code.is_some() == y.code.is_some() + } + _ => false, + } +} + +/// Read `balanceOf(owner)` through an overlay (non-committing). +fn overlay_balance_of(overlay: &mut EvmOverlay, token: Address, owner: Address) -> Result { + let call = MockERC20::balanceOfCall { account: owner }; + match overlay.call_raw(owner, token, call.abi_encode().into())? { + revm::context::result::ExecutionResult::Success { output, .. } => Ok( + MockERC20::balanceOfCall::abi_decode_returns(&output.into_data())?, + ), + other => Err(anyhow!("overlay balanceOf failed: {other:?}")), + } +} + +/// Assert the COW snapshot and the deep-clone reference are read-indistinguishable +/// across a probe set of addresses and slots. `label` identifies the mutation step. +fn assert_equivalent(cache: &mut EvmCache, addrs: &[Address], slots: &[U256], label: &str) { + let cow = cache.create_snapshot(); + let deep = cache.create_snapshot_deep_clone(); + + let mut ov_cow = EvmOverlay::new(Arc::clone(&cow), None); + let mut ov_deep = EvmOverlay::new(Arc::clone(&deep), None); + + // Block context must match. + assert_eq!(ov_cow.chain_id(), ov_deep.chain_id(), "{label}: chain_id"); + assert_eq!( + ov_cow.block_number(), + ov_deep.block_number(), + "{label}: block_number" + ); + assert_eq!(ov_cow.basefee(), ov_deep.basefee(), "{label}: basefee"); + assert_eq!( + ov_cow.timestamp(), + ov_deep.timestamp(), + "{label}: timestamp" + ); + + for &a in addrs { + let bc = ov_cow.basic(a).expect("cow basic"); + let bd = ov_deep.basic(a).expect("deep basic"); + assert!( + account_eq(&bc, &bd), + "{label}: basic mismatch at {a}: cow={bc:?} deep={bd:?}" + ); + // Code lookup for the account's code hash must agree (spec §8.1). + if let Some(info) = &bc { + let h = info.code_hash; + assert_eq!( + ov_cow.code_by_hash(h).expect("cow code").original_bytes(), + ov_deep.code_by_hash(h).expect("deep code").original_bytes(), + "{label}: code_by_hash mismatch at {a} (hash {h})" + ); + } + for &s in slots { + assert_eq!( + cow.storage_value(a, s), + deep.storage_value(a, s), + "{label}: snapshot.storage_value mismatch at {a} / {s}" + ); + let scow = ov_cow.storage(a, s).expect("cow storage"); + let sdeep = ov_deep.storage(a, s).expect("deep storage"); + assert_eq!( + scow, sdeep, + "{label}: overlay storage mismatch at {a} / {s}" + ); + } + } +} + +/// The core gate: drive one cache through every mutation kind and assert the COW +/// snapshot stays read-identical to the deep-clone reference after each step. +#[tokio::test(flavor = "multi_thread")] +async fn cow_snapshot_matches_deep_clone_through_mutations() -> Result<()> { + let mut cache = setup_cache().await?; + + let token = Address::repeat_byte(0x11); // cleared layer-1 account (MockERC20) + let owner = Address::repeat_byte(0x22); + let recipient = Address::repeat_byte(0x33); + let pool = Address::repeat_byte(0x77); // layer-2-only, non-cleared + let pool2 = Address::repeat_byte(0x88); // write-through target absent from layer 1 + let pool3 = Address::repeat_byte(0x99); // appears via simulated lazy fetch + let ghost = Address::repeat_byte(0xEE); // becomes NotExisting + + let balance_slot = U256::from(MOCK_ERC20_BALANCE_SLOT); + let owner_bal = mapping_slot(owner, MOCK_ERC20_BALANCE_SLOT); + let recip_bal = mapping_slot(recipient, MOCK_ERC20_BALANCE_SLOT); + + let addrs = [token, owner, recipient, pool, pool2, pool3, ghost]; + // Probe real slots plus an always-absent slot (both must agree on None). + let slots = [ + balance_slot, + owner_bal, + recip_bal, + U256::from(0u64), + U256::from(1u64), + U256::from(7u64), + U256::from(424_242u64), // never set anywhere + ]; + + // 1. Empty cache. + assert_equivalent(&mut cache, &addrs, &slots, "empty"); + + // 2. Layer-1 account inserts. + install_default_account(&mut cache, Address::ZERO); + install_default_account(&mut cache, owner); + install_default_account(&mut cache, recipient); + install_mock_erc20(&mut cache, token); + assert_equivalent(&mut cache, &addrs, &slots, "after account inserts"); + + // 3. Layer-1 storage inserts (mapping balances on the cleared token). + cache.insert_mapping_storage_slot(token, balance_slot, owner, U256::from(1_000u64))?; + cache.insert_mapping_storage_slot(token, balance_slot, recipient, U256::ZERO)?; + assert_equivalent(&mut cache, &addrs, &slots, "after layer-1 storage"); + + // 4. write-through to an address PRESENT in layer 1 (shadowed there). + cache.apply_updates(&[StateUpdate::slot(token, owner_bal, U256::from(2_000u64))]); + assert_equivalent( + &mut cache, + &addrs, + &slots, + "after write-through (in layer 1)", + ); + + // 5. write-through to an address ABSENT from layer 1 (layer-2-only — the §3 + // footgun: the base must capture it). + cache.apply_updates(&[StateUpdate::slot( + pool2, + U256::from(7u64), + U256::from(55u64), + )]); + assert_equivalent( + &mut cache, + &addrs, + &slots, + "after write-through (layer-2-only)", + ); + + // 6. relative native-balance delta. + cache.apply_updates(&[StateUpdate::balance_delta( + owner, + SlotDelta::Add(U256::from(500)), + )]); + assert_equivalent(&mut cache, &addrs, &slots, "after balance delta"); + + // 7. committing revm call (mutates layer 1 only — never stales the base). + transfer(&mut cache, token, owner, recipient, U256::from(250u64))?; + assert_equivalent(&mut cache, &addrs, &slots, "after committed transfer"); + + // 8. layer-2-only cold backfill, including OVERWRITING an existing slot at an + // unchanged length (must still invalidate the base). + cache.inject_storage_batch(&[(pool, U256::from(0u64), U256::from(111u64))]); + assert_equivalent(&mut cache, &addrs, &slots, "after inject (new)"); + cache.inject_storage_batch(&[(pool, U256::from(0u64), U256::from(222u64))]); + assert_equivalent( + &mut cache, + &addrs, + &slots, + "after inject (overwrite, same len)", + ); + + // 9. simulated UNCONTROLLED layer-2 growth (a lazy RPC fetch / prefetch writes + // `BlockchainDb` from inside foundry-fork-db, bypassing our write funnel): + // a brand-new account+slot, and a NEW slot on the existing `pool`. + { + let bdb = cache.unchecked_blockchain_db(); + bdb.storage() + .write() + .entry(pool3) + .or_default() + .insert(U256::from(1u64), U256::from(909u64)); + bdb.accounts().write().insert( + pool3, + AccountInfo { + balance: U256::from(5u64), + ..Default::default() + }, + ); + // New slot on an existing base account (len changes → growth scan must catch). + bdb.storage() + .write() + .entry(pool) + .or_default() + .insert(U256::from(1u64), U256::from(333u64)); + } + assert_equivalent( + &mut cache, + &addrs, + &slots, + "after uncontrolled layer-2 growth", + ); + + // 10. purge. + cache.purge_account(owner); + assert_equivalent(&mut cache, &addrs, &slots, "after purge_account"); + + // 11. NotExisting account (absent to the EVM; storage reads ZERO). + cache.db_mut().insert_account_info( + ghost, + AccountInfo { + balance: U256::from(1u64), + ..Default::default() + }, + ); + cache + .db_mut() + .cache + .accounts + .get_mut(&ghost) + .expect("ghost present") + .account_state = AccountState::NotExisting; + assert_equivalent(&mut cache, &addrs, &slots, "after NotExisting"); + + // 12. set_block (re-pin → full base rebuild path). + cache.set_block(None); + assert_equivalent(&mut cache, &addrs, &slots, "after set_block"); + + Ok(()) +} + +/// Escape-hatch re-honest hook (adversarial-review finding). A direct, out-of-band +/// layer-2 write through `unchecked_blockchain_db()` that overwrites an existing slot at an +/// unchanged slot count is the one mutation the count-based growth scan cannot see, +/// so the memoized base can go stale. `invalidate_snapshot_base()` must restore +/// read-equivalence with the deep-clone reference. +#[tokio::test(flavor = "multi_thread")] +async fn invalidate_snapshot_base_rehonest_after_escape_hatch_write() -> Result<()> { + let mut cache = setup_cache().await?; + let pool = Address::repeat_byte(0x77); // layer-2-only, non-shadowed + let slot = U256::from(0u64); + + cache.inject_storage_batch(&[(pool, slot, U256::from(111u64))]); + let _warm = cache.create_snapshot(); // memoize the base at 111 + + // Out-of-band overwrite at unchanged length (bypasses the write funnel). + { + let bdb = cache.unchecked_blockchain_db(); + bdb.storage() + .write() + .entry(pool) + .or_default() + .insert(slot, U256::from(222u64)); + } + + // The documented re-honest hook must make the next snapshot reflect the write. + cache.invalidate_snapshot_base(); + let cow = cache.create_snapshot(); + let deep = cache.create_snapshot_deep_clone(); + assert_eq!( + cow.storage_value(pool, slot), + deep.storage_value(pool, slot), + "invalidate_snapshot_base must re-honest the base after an out-of-band write" + ); + assert_eq!(cow.storage_value(pool, slot), Some(U256::from(222u64))); + Ok(()) +} + +/// Escape-hatch re-honest hook for account-map overwrites. A direct update of an +/// existing layer-2 account's balance/code at an unchanged account count is also +/// invisible to the count/absence growth scan, so callers must invalidate the +/// memoized base after the direct write lands. +#[tokio::test(flavor = "multi_thread")] +async fn invalidate_snapshot_base_rehonest_after_existing_account_write() -> Result<()> { + let mut cache = setup_cache().await?; + let account = Address::repeat_byte(0xA1); + let code_v1 = Bytecode::new_raw(Bytes::from(vec![0x60u8, 0x01])); + let code_v2 = Bytecode::new_raw(Bytes::from(vec![0x60u8, 0x02, 0x60, 0x03])); + let h1 = code_v1.hash_slow(); + let h2 = code_v2.hash_slow(); + assert_ne!(h1, h2); + + let original = AccountInfo { + balance: U256::from(111u64), + nonce: 1, + code_hash: h1, + code: Some(code_v1.clone()), + account_id: None, + }; + let updated = AccountInfo { + balance: U256::from(222u64), + nonce: 2, + code_hash: h2, + code: Some(code_v2.clone()), + account_id: None, + }; + + { + let bdb = cache.unchecked_blockchain_db(); + bdb.accounts().write().insert(account, original.clone()); + } + let warm = cache.create_snapshot(); // memoize the base with `original`. + let mut ov_warm = EvmOverlay::new(Arc::clone(&warm), None); + let warm_info = ov_warm + .basic(account) + .expect("warm basic") + .expect("warm account"); + assert_eq!(warm_info.balance, original.balance); + assert_eq!(warm_info.nonce, original.nonce); + assert_eq!(warm_info.code_hash, h1); + assert_eq!( + ov_warm + .code_by_hash(h1) + .expect("warm code") + .original_bytes(), + code_v1.original_bytes() + ); + + // Out-of-band account overwrite at unchanged account count (bypasses the + // write funnel and is not detectable by the growth scan). + { + let bdb = cache.unchecked_blockchain_db(); + let mut accounts = bdb.accounts().write(); + assert!( + accounts.contains_key(&account), + "test must update an existing account" + ); + let len_before = accounts.len(); + accounts.insert(account, updated.clone()); + assert_eq!( + accounts.len(), + len_before, + "test must keep the account count unchanged" + ); + } + + cache.invalidate_snapshot_base(); + let cow = cache.create_snapshot(); + let deep = cache.create_snapshot_deep_clone(); + let mut ov_cow = EvmOverlay::new(Arc::clone(&cow), None); + let mut ov_deep = EvmOverlay::new(Arc::clone(&deep), None); + + let cow_basic = ov_cow.basic(account).expect("cow basic"); + let deep_basic = ov_deep.basic(account).expect("deep basic"); + assert!( + account_eq(&cow_basic, &deep_basic), + "invalidate_snapshot_base must re-honest the base after an out-of-band account write: cow={cow_basic:?} deep={deep_basic:?}" + ); + let cow_info = cow_basic.expect("updated cow account"); + assert_eq!(cow_info.balance, updated.balance); + assert_eq!(cow_info.nonce, updated.nonce); + assert_eq!(cow_info.code_hash, h2); + assert_eq!( + ov_cow.code_by_hash(h2).expect("cow h2").original_bytes(), + ov_deep.code_by_hash(h2).expect("deep h2").original_bytes(), + "updated code hash must match the deep clone" + ); + assert_eq!( + ov_cow.code_by_hash(h2).expect("cow h2").original_bytes(), + code_v2.original_bytes() + ); + assert!( + ov_deep.code_by_hash(h1).expect("deep h1").is_empty(), + "sanity: deep clone drops the unreferenced old hash" + ); + assert_eq!( + ov_cow.code_by_hash(h1).expect("cow h1").original_bytes(), + ov_deep.code_by_hash(h1).expect("deep h1").original_bytes(), + "the old code hash must not linger after invalidation" + ); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn with_blockchain_db_mut_rehonest_after_storage_overwrite() -> Result<()> { + let mut cache = setup_cache().await?; + let pool = Address::repeat_byte(0x78); + let slot = U256::from(0u64); + + cache.inject_storage_batch(&[(pool, slot, U256::from(111u64))]); + let _warm = cache.create_snapshot(); + + cache.with_blockchain_db_mut(|bdb| { + bdb.storage() + .write() + .entry(pool) + .or_default() + .insert(slot, U256::from(222u64)); + }); + + let cow = cache.create_snapshot(); + let deep = cache.create_snapshot_deep_clone(); + assert_eq!( + cow.storage_value(pool, slot), + deep.storage_value(pool, slot), + "with_blockchain_db_mut must invalidate the COW base after storage writes" + ); + assert_eq!(cow.storage_value(pool, slot), Some(U256::from(222u64))); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn with_blockchain_db_mut_rehonest_after_account_overwrite() -> Result<()> { + let mut cache = setup_cache().await?; + let account = Address::repeat_byte(0xA2); + let code_v1 = Bytecode::new_raw(Bytes::from(vec![0x60u8, 0x01])); + let code_v2 = Bytecode::new_raw(Bytes::from(vec![0x60u8, 0x02])); + let h1 = code_v1.hash_slow(); + let h2 = code_v2.hash_slow(); + let original = AccountInfo { + balance: U256::from(111u64), + nonce: 1, + code_hash: h1, + code: Some(code_v1), + account_id: None, + }; + let updated = AccountInfo { + balance: U256::from(222u64), + nonce: 2, + code_hash: h2, + code: Some(code_v2.clone()), + account_id: None, + }; + + cache.with_blockchain_db_mut(|bdb| { + bdb.accounts().write().insert(account, original); + }); + let _warm = cache.create_snapshot(); + + cache.with_blockchain_db_mut(|bdb| { + let mut accounts = bdb.accounts().write(); + let len_before = accounts.len(); + accounts.insert(account, updated.clone()); + assert_eq!(accounts.len(), len_before); + }); + + let cow = cache.create_snapshot(); + let deep = cache.create_snapshot_deep_clone(); + let mut ov_cow = EvmOverlay::new(Arc::clone(&cow), None); + let mut ov_deep = EvmOverlay::new(Arc::clone(&deep), None); + let cow_basic = ov_cow.basic(account).expect("cow basic"); + let deep_basic = ov_deep.basic(account).expect("deep basic"); + assert!( + account_eq(&cow_basic, &deep_basic), + "with_blockchain_db_mut must invalidate the COW base after account writes: cow={cow_basic:?} deep={deep_basic:?}" + ); + assert_eq!( + cow_basic.expect("updated cow account").balance, + updated.balance + ); + assert_eq!( + ov_cow.code_by_hash(h2).expect("cow h2").original_bytes(), + code_v2.original_bytes() + ); + Ok(()) +} + +/// Regression (review finding P2): the COW partial rebuild must not leave a stale +/// `code_by_hash` entry when a base account is recoded or purged. Warm the base +/// with a code-bearing account, recode it in layer 2, dirty it via a controlled +/// per-address write (so `refresh_base` takes the Case-4 *partial* rebuild path, +/// not a full rebuild), and assert the old hash no longer resolves — matching the +/// deep-clone reference, which rebuilds its code index from current accounts. +#[tokio::test(flavor = "multi_thread")] +async fn cow_code_index_matches_deep_clone_after_base_account_recoded() -> Result<()> { + let mut cache = setup_cache().await?; + let contract = Address::repeat_byte(0xc0); + let code_v1 = Bytecode::new_raw(Bytes::from(vec![0x60u8, 0x01])); + let code_v2 = Bytecode::new_raw(Bytes::from(vec![0x60u8, 0x02, 0x60, 0x03])); + let h1 = code_v1.hash_slow(); + let h2 = code_v2.hash_slow(); + assert_ne!(h1, h2); + + // Seed a code-bearing account (code_v1) directly into the cold base (layer 2), + // then re-honest the memoized base. + let put_account = |cache: &EvmCache, code: &Bytecode, hash| { + cache.unchecked_blockchain_db().accounts().write().insert( + contract, + AccountInfo { + balance: U256::from(1u64), + nonce: 1, + code_hash: hash, + code: Some(code.clone()), + account_id: None, + }, + ); + }; + put_account(&cache, &code_v1, h1); + cache.invalidate_snapshot_base(); + let warm = cache.create_snapshot(); // base now indexes h1 -> code_v1 + let mut ov_warm = EvmOverlay::new(Arc::clone(&warm), None); + assert_eq!( + ov_warm + .code_by_hash(h1) + .expect("warm code") + .original_bytes(), + code_v1.original_bytes(), + "warm snapshot must resolve the seeded code" + ); + + // Recode the base account to code_v2 (out-of-band), then dirty `contract` via a + // controlled per-address write so the next snapshot takes the Case-4 partial + // rebuild — exactly the path that previously failed to prune the old hash. + put_account(&cache, &code_v2, h2); + cache.apply_updates(&[StateUpdate::slot( + contract, + U256::from(0u64), + U256::from(9u64), + )]); + + let cow = cache.create_snapshot(); + let deep = cache.create_snapshot_deep_clone(); + let mut ov_cow = EvmOverlay::new(Arc::clone(&cow), None); + let mut ov_deep = EvmOverlay::new(Arc::clone(&deep), None); + + // The new hash resolves identically... + assert_eq!( + ov_cow.code_by_hash(h2).expect("cow h2").original_bytes(), + ov_deep.code_by_hash(h2).expect("deep h2").original_bytes(), + "new code hash must match the deep clone" + ); + // ...and the now-unreferenced old hash must NOT linger in the COW base: both + // resolve to empty (the deep clone never had it after the recode). + assert!( + ov_deep.code_by_hash(h1).expect("deep h1").is_empty(), + "sanity: deep clone drops the unreferenced old hash" + ); + assert_eq!( + ov_cow.code_by_hash(h1).expect("cow h1").original_bytes(), + ov_deep.code_by_hash(h1).expect("deep h1").original_bytes(), + "COW must not return stale bytecode for the recoded account's old hash" + ); + Ok(()) +} + +/// COW must not alias: a snapshot taken earlier is unaffected by a later mutation +/// of the same address (the memoized base is rebuilt copy-on-write, not mutated). +#[tokio::test(flavor = "multi_thread")] +async fn earlier_snapshot_unaffected_by_later_base_mutation() -> Result<()> { + let mut cache = setup_cache().await?; + let pool = Address::repeat_byte(0x77); + let slot = U256::from(3u64); + + cache.inject_storage_batch(&[(pool, slot, U256::from(100u64))]); + let early = cache.create_snapshot(); + assert_eq!(early.storage_value(pool, slot), Some(U256::from(100u64))); + + // Mutate the same base slot, then take a second snapshot. + cache.inject_storage_batch(&[(pool, slot, U256::from(200u64))]); + let late = cache.create_snapshot(); + + assert_eq!( + early.storage_value(pool, slot), + Some(U256::from(100u64)), + "the earlier snapshot must still read the pre-mutation value" + ); + assert_eq!( + late.storage_value(pool, slot), + Some(U256::from(200u64)), + "the later snapshot reflects the mutation" + ); + Ok(()) +} + +/// `EvmOverlay::reset` clears the dirty layer so the overlay reads the pristine +/// snapshot again — equivalent to a fresh overlay. +#[tokio::test(flavor = "multi_thread")] +async fn overlay_reset_restores_pristine_snapshot_reads() -> Result<()> { + let mut cache = setup_cache().await?; + let token = Address::repeat_byte(0x44); + let owner = Address::repeat_byte(0x55); + let recipient = Address::repeat_byte(0x66); + + install_default_account(&mut cache, Address::ZERO); + install_default_account(&mut cache, owner); + install_default_account(&mut cache, recipient); + install_mock_erc20(&mut cache, token); + cache.insert_mapping_storage_slot( + token, + U256::from(MOCK_ERC20_BALANCE_SLOT), + owner, + U256::from(10_000u64), + )?; + cache.insert_mapping_storage_slot( + token, + U256::from(MOCK_ERC20_BALANCE_SLOT), + recipient, + U256::ZERO, + )?; + + let snapshot = cache.create_snapshot(); + let mut overlay = EvmOverlay::new(Arc::clone(&snapshot), None); + + // Mutate the dirty layer with a committing transfer through the overlay. + overlay.simulate_with_transfer_tracking( + owner, + token, + MockERC20::transferCall { + to: recipient, + amount: U256::from(4_000u64), + } + .abi_encode() + .into(), + owner, + Some([token]), + true, // commit into the overlay's dirty layer + )?; + assert_eq!( + overlay_balance_of(&mut overlay, token, owner)?, + U256::from(6_000u64), + "post-transfer dirty-layer balance" + ); + + // reset() drops the dirty layer; reads see the pristine snapshot again. + overlay.reset(); + assert_eq!( + overlay_balance_of(&mut overlay, token, owner)?, + U256::from(10_000u64), + "after reset the overlay reads the pristine snapshot" + ); + + // A reset-recycled overlay matches a brand-new overlay across two sims. + let mut fresh = EvmOverlay::new(Arc::clone(&snapshot), None); + assert_eq!( + overlay_balance_of(&mut overlay, token, owner)?, + overlay_balance_of(&mut fresh, token, owner)?, + "recycled overlay == fresh overlay" + ); + Ok(()) +} + +/// Buffer reuse must not change results: repeated calls on one overlay return the +/// same value as the first (the reusable shared-memory buffer is cleared, not +/// corrupted, between builds). +#[tokio::test(flavor = "multi_thread")] +async fn overlay_buffer_reuse_is_result_stable() -> Result<()> { + let mut cache = setup_cache().await?; + let token = Address::repeat_byte(0x44); + let owner = Address::repeat_byte(0x55); + + install_default_account(&mut cache, Address::ZERO); + install_default_account(&mut cache, owner); + install_mock_erc20(&mut cache, token); + cache.insert_mapping_storage_slot( + token, + U256::from(MOCK_ERC20_BALANCE_SLOT), + owner, + U256::from(7_777u64), + )?; + + let snapshot = cache.create_snapshot(); + let mut overlay = EvmOverlay::new(snapshot, None); + + let first = overlay_balance_of(&mut overlay, token, owner)?; + for _ in 0..16 { + assert_eq!( + overlay_balance_of(&mut overlay, token, owner)?, + first, + "repeated calls reusing the buffer must be stable" + ); + } + assert_eq!(first, U256::from(7_777u64)); + Ok(()) +} + +/// Compile-time guards: the COW representation keeps the thread-safety contract. +#[test] +fn snapshot_send_sync_overlay_send() { + fn assert_send_sync() {} + fn assert_send() {} + assert_send_sync::(); + assert_send_sync::>(); + assert_send::(); +} diff --git a/tests/event_ground_truth.rs b/tests/event_ground_truth.rs new file mode 100644 index 0000000..e382d32 --- /dev/null +++ b/tests/event_ground_truth.rs @@ -0,0 +1,317 @@ +//! **Differential ground-truth test** for the event → state pipeline (Phase 4). +//! +//! The decisive correctness check: does feeding the *real emitted logs* of a swap +//! into our event processor reproduce the *exact* state a real EVM execution +//! produced? We run a swap in a ground-truth revm instance and replay only its +//! logs into a twin cache, then assert the token balances and the packed pool +//! `slot0` (price/tick) match bit-for-bit. +//! +//! Setup (an offline stand-in for RPC-fetched state): +//! 1. Deploy two ERC-20 tokens (the `MockERC20` fixture, balances at slot 3) and a +//! `TestV3Pool` (`fixtures/EventGroundTruthPool.sol`) whose `slot0` is a Solidity +//! **struct** with the identical field widths to `UniswapV3Pool.Slot0` — so the +//! *compiler* (not this test) does the real bit-packing, and our +//! `StateUpdate::SlotMasked` is the thing under test. Seed pool liquidity and a +//! swapper balance. +//! 2. Build the identical pre-swap state in a second ("event-driven") cache. The +//! deploy sequence is deterministic, so the token/pool addresses match. +//! 3. Execute a real `swap` against the ground-truth cache (committing) and +//! capture the emitted logs (two ERC-20 `Transfer`s + the canonical `Swap`). +//! 4. Feed only those logs into the event-driven cache via `EventPipeline`, then +//! assert its balances and `slot0` equal the ground-truth cache's. +//! +//! Runs fully offline. Requires the `protocols` feature (the UniswapV3 adapter). +#![cfg(feature = "protocols")] + +mod common; + +use std::sync::Arc; + +use alloy_primitives::aliases::{I24, U160}; +use alloy_primitives::{Address, Bytes, Log, U256, hex, keccak256}; +use alloy_sol_types::{SolCall, SolValue, sol}; +use anyhow::{Result, anyhow}; +use common::{MOCK_ERC20_CREATION_HEX, install_default_account, setup_cache}; +use evm_fork_cache::cache::{EvmCache, V3_LIQUIDITY_SLOT, V3_SLOT0_SLOT}; +use evm_fork_cache::deploy::{build_init_code, encode_constructor_args}; +use evm_fork_cache::events::{DecoderRegistry, EventPipeline}; +use evm_fork_cache::{Erc20TransferDecoder, UniswapV3Decoder, UniswapV3Layout}; +use revm::context::result::ExecutionResult; + +sol! { + interface Token { + function _mint(address to, uint256 amount) external; + function approve(address spender, uint256 amount) external returns (bool); + } + interface Pool { + function initialize(uint160 sqrtPriceX96, int24 tick, uint16 observationIndex, uint128 liquidity) external; + function swap(bool zeroForOne, uint256 amountIn, uint256 amountOut, uint160 newSqrtPriceX96, int24 newTick, uint128 newLiquidity) external; + } +} + +const POOL_CREATION_HEX: &str = include_str!("../fixtures/test_v3_pool_creation.hex"); + +/// The MockERC20 balance mapping slot (`mapping(address => uint256)` at slot 3). +const BALANCE_SLOT: u64 = 3; + +/// Pre-swap parameters, shared by both caches. +const INIT_SQRT_PRICE: u128 = 1u128 << 96; // 2^96 +const INIT_TICK: i32 = 100; +const INIT_OBS_INDEX: u16 = 7; // non-zero, to prove it survives the swap +const INIT_LIQUIDITY: u128 = 1_000_000; +const POOL_RESERVE: u128 = 1_000_000; +const SWAPPER_TOKEN0: u128 = 500_000; + +/// Swap outcome (the test plays the role of the router specifying it). token0 in, +/// token1 out; a *negative* post-swap tick and a full-width `sqrtPriceX96` stress +/// the slot0 packing/sign handling. +const AMOUNT_IN: u128 = 120_000; +const AMOUNT_OUT: u128 = 80_000; +const NEW_TICK: i32 = -50; +const NEW_LIQUIDITY: u128 = 1_050_000; + +fn deployer() -> Address { + Address::repeat_byte(0xd0) +} +fn swapper() -> Address { + Address::repeat_byte(0x5a) +} + +/// Hashed `balanceOf[owner]` storage key. +fn balance_slot(owner: Address) -> U256 { + U256::from_be_bytes(keccak256((owner, U256::from(BALANCE_SLOT)).abi_encode()).0) +} + +fn call(cache: &mut EvmCache, from: Address, to: Address, data: Vec) -> Result<()> { + match cache.call_raw(from, to, Bytes::from(data), true)? { + ExecutionResult::Success { .. } => Ok(()), + other => Err(anyhow!("call to {to} failed: {other:?}")), + } +} + +/// Build the identical pre-swap state in `cache`, returning `(token0, token1, pool)`. +/// +/// The deploy order is fixed, so the deterministic `CREATE` addresses are the same +/// across caches (essential — the captured logs reference these addresses). +fn build_state(cache: &mut EvmCache) -> Result<(Address, Address, Address)> { + install_default_account(cache, Address::ZERO); // coinbase + install_default_account(cache, deployer()); + install_default_account(cache, swapper()); + + // CREATE addresses are deterministic from (deployer, nonce). Pre-install them + // as empty accounts so revm's CREATE collision-check reads the local overlay + // instead of falling through to a (mocked, empty) RPC fetch. + for nonce in 0..3 { + install_default_account(cache, deployer().create(nonce)); + } + + let erc20_creation = hex::decode(MOCK_ERC20_CREATION_HEX.trim())?; + let token0 = cache.deploy_contract( + deployer(), + build_init_code( + &erc20_creation, + // uint8 encodes as a right-aligned 32-byte word, identical to U256. + encode_constructor_args(("Token0".to_string(), "T0".to_string(), U256::from(18))), + ), + )?; + let token1 = cache.deploy_contract( + deployer(), + build_init_code( + &erc20_creation, + encode_constructor_args(("Token1".to_string(), "T1".to_string(), U256::from(18))), + ), + )?; + let pool_creation = hex::decode(POOL_CREATION_HEX.trim())?; + let pool = cache.deploy_contract( + deployer(), + build_init_code(&pool_creation, encode_constructor_args((token0, token1))), + )?; + + // Initialize the pool's packed slot0 + liquidity. + call( + cache, + deployer(), + pool, + Pool::initializeCall { + sqrtPriceX96: U160::from(INIT_SQRT_PRICE), + tick: I24::try_from(INIT_TICK).unwrap(), + observationIndex: INIT_OBS_INDEX, + liquidity: INIT_LIQUIDITY, + } + .abi_encode(), + )?; + + // Seed reserves into the pool and the input balance into the swapper. + for token in [token0, token1] { + call( + cache, + deployer(), + token, + Token::_mintCall { + to: pool, + amount: U256::from(POOL_RESERVE), + } + .abi_encode(), + )?; + } + call( + cache, + deployer(), + token0, + Token::_mintCall { + to: swapper(), + amount: U256::from(SWAPPER_TOKEN0), + } + .abi_encode(), + )?; + // Swapper approves the pool to pull the input. + call( + cache, + swapper(), + token0, + Token::approveCall { + spender: pool, + amount: U256::from(AMOUNT_IN), + } + .abi_encode(), + )?; + + Ok((token0, token1, pool)) +} + +/// Snapshot the four balances + the packed slot0 + liquidity we compare on. +fn observe( + cache: &EvmCache, + t0: Address, + t1: Address, + pool: Address, +) -> Vec<(String, Option)> { + vec![ + ( + "swapper.t0".into(), + cache.cached_storage_value(t0, balance_slot(swapper())), + ), + ( + "swapper.t1".into(), + cache.cached_storage_value(t1, balance_slot(swapper())), + ), + ( + "pool.t0".into(), + cache.cached_storage_value(t0, balance_slot(pool)), + ), + ( + "pool.t1".into(), + cache.cached_storage_value(t1, balance_slot(pool)), + ), + ( + "pool.slot0".into(), + cache.cached_storage_value(pool, V3_SLOT0_SLOT), + ), + ( + "pool.liquidity".into(), + cache.cached_storage_value(pool, V3_LIQUIDITY_SLOT), + ), + ] +} + +#[tokio::test(flavor = "multi_thread")] +async fn event_processor_reproduces_ground_truth_swap() -> Result<()> { + // 1. Ground-truth cache: build state, snapshot the pre-swap state, execute the + // real swap, capture logs + the post-swap state. + let mut truth = setup_cache().await?; + let (token0, token1, pool) = build_state(&mut truth)?; + let pre_swap = observe(&truth, token0, token1, pool); + + let swap_data = Pool::swapCall { + zeroForOne: true, + amountIn: U256::from(AMOUNT_IN), + amountOut: U256::from(AMOUNT_OUT), + newSqrtPriceX96: U160::MAX, // full-width: stresses the [0,160) boundary + newTick: I24::try_from(NEW_TICK).unwrap(), + newLiquidity: NEW_LIQUIDITY, + } + .abi_encode(); + + let logs: Vec = match truth.call_raw(swapper(), pool, Bytes::from(swap_data), true)? { + ExecutionResult::Success { logs, .. } => logs, + other => return Err(anyhow!("ground-truth swap failed: {other:?}")), + }; + // Two ERC-20 Transfers (token0 in, token1 out) + the canonical Swap. + assert_eq!(logs.len(), 3, "expected 2 Transfer logs + 1 Swap log"); + + // 2. Event-driven cache: identical pre-swap state, addresses match. + let mut driven = setup_cache().await?; + let (token0_d, token1_d, pool_d) = build_state(&mut driven)?; + assert_eq!( + (token0, token1, pool), + (token0_d, token1_d, pool_d), + "deterministic deploy addresses must match across caches" + ); + + // Pre-swap, the freshly-built driven cache equals truth's pre-swap snapshot + // (sanity: the deterministic setup really is identical). + assert_eq!(observe(&driven, token0, token1, pool), pre_swap); + + // 3. Feed ONLY the swap's logs into the event pipeline. + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(Erc20TransferDecoder::new(U256::from( + BALANCE_SLOT, + )))); + registry.register(Arc::new( + UniswapV3Decoder::new().with_pool(pool, UniswapV3Layout::uniswap(60)), + )); + let mut pipeline = EventPipeline::new(registry); + let digest = pipeline.ingest_logs(&mut driven, 1, &logs); + + // All three logs decoded to applied changes; nothing skipped (hot state). + assert_eq!(digest.decoded_logs, 3, "all 3 logs should decode"); + assert!( + !digest.applied.has_skipped(), + "no cold skips: {:?}", + digest.applied.skipped_masks + ); + + // 4. The decisive check: event-driven state == ground-truth state, field by field. + let truth_state = observe(&truth, token0, token1, pool); + let driven_state = observe(&driven, token0, token1, pool); + assert_eq!( + driven_state, truth_state, + "event-driven state must match the ground-truth EVM execution" + ); + + // Spell out the headline invariants explicitly (defensive, human-readable). + let slot0_truth = truth.cached_storage_value(pool, V3_SLOT0_SLOT).unwrap(); + let slot0_driven = driven.cached_storage_value(pool, V3_SLOT0_SLOT).unwrap(); + assert_eq!( + slot0_driven, slot0_truth, + "packed slot0 (price/tick) must match bit-for-bit" + ); + // The price actually moved, and the observation/unlocked bits survived. + assert_ne!(slot0_driven, U256::from(INIT_SQRT_PRICE), "slot0 changed"); + assert_eq!( + (slot0_driven >> 240) & U256::from(1), + U256::from(1), + "unlocked bit preserved" + ); + assert_eq!( + (slot0_driven >> 184) & U256::from(0xFFFF), + U256::from(INIT_OBS_INDEX), + "obs index preserved" + ); + + // Balances match the ground truth (token0 in, token1 out). + assert_eq!( + driven + .cached_storage_value(token0, balance_slot(swapper())) + .unwrap(), + U256::from(SWAPPER_TOKEN0 - AMOUNT_IN), + ); + assert_eq!( + driven + .cached_storage_value(token1, balance_slot(swapper())) + .unwrap(), + U256::from(AMOUNT_OUT), + ); + + Ok(()) +} diff --git a/tests/event_pipeline.rs b/tests/event_pipeline.rs new file mode 100644 index 0000000..bc6b2be --- /dev/null +++ b/tests/event_pipeline.rs @@ -0,0 +1,901 @@ +//! Offline acceptance tests for the Phase 4 event pipeline (Pillar B.2). +//! +//! These are the **contract** the implementation must satisfy: the +//! `EventDecoder` / `StateView` traits, the `DecoderRegistry`, the ERC-20 +//! `Transfer` decoder, the UniswapV3 `Swap`/`Mint`/`Burn` adapter, and the +//! `EventPipeline` (`ingest_logs` / `reorg_to` / `reconcile`). Everything runs +//! fully offline (mocked provider, state injected directly, logs built in +//! memory), so no test reaches the network. +//! +//! Layering vocabulary mirrors `tests/state_update.rs`: +//! - **layer 1 / overlay** = the CacheDB overlay (`db_mut().cache.accounts`). +//! - **layer 2 / backend** = the BlockchainDb backend (`unchecked_blockchain_db()`). + +mod common; + +use std::collections::HashMap; +use std::sync::Arc; + +use alloy_primitives::{Address, Bytes, Log, U256, keccak256}; +use anyhow::Result; + +use common::{install_mock_erc20, setup_cache, stub_fetcher}; +use evm_fork_cache::cache::EvmCache; +use evm_fork_cache::events::erc20::Erc20TransferDecoder; +use evm_fork_cache::events::{ + DecoderRegistry, EventDecoder, EventPipeline, ReorgConfig, StateView, +}; +use evm_fork_cache::{PurgeScope, SlotDelta, StateUpdate}; + +// --------------------------------------------------------------------------- +// Shared helpers. +// --------------------------------------------------------------------------- + +/// Hashed storage slot of a `mapping(address => uint256)` at `mapping_slot`. +fn mapping_slot(owner: Address, mapping_slot: u64) -> U256 { + use alloy_sol_types::SolValue; + let key = keccak256((owner, U256::from(mapping_slot)).abi_encode()); + U256::from_be_bytes(key.0) +} + +/// Value of a slot in the BlockchainDb backend (layer 2) only. +fn backend_slot(cache: &EvmCache, addr: Address, slot: U256) -> Option { + cache + .unchecked_blockchain_db() + .storage() + .read() + .get(&addr) + .and_then(|s| s.get(&slot).copied()) +} + +/// A read-only [`StateView`] stub backed by a fixed map (for pure decoder unit +/// tests that do not need a real cache). +struct StubView(HashMap<(Address, U256), U256>); +impl StateView for StubView { + fn storage(&self, address: Address, slot: U256) -> Option { + self.0.get(&(address, slot)).copied() + } +} +fn empty_view() -> StubView { + StubView(HashMap::new()) +} + +/// A test-only decoder that emits a single absolute `Slot` write for every log +/// (used to exercise pipeline mechanics independent of any real protocol). +struct MarkDecoder { + slot: U256, + value: U256, +} +impl EventDecoder for MarkDecoder { + fn decode(&self, log: &Log, _view: &dyn StateView) -> Vec { + vec![StateUpdate::slot(log.address, self.slot, self.value)] + } +} + +/// A test-only decoder that fires only for logs whose first topic equals `topic`. +struct TaggedDecoder { + topic: alloy_primitives::B256, + slot: U256, + value: U256, +} +impl EventDecoder for TaggedDecoder { + fn decode(&self, log: &Log, _view: &dyn StateView) -> Vec { + if log.topics().first() == Some(&self.topic) { + vec![StateUpdate::slot(log.address, self.slot, self.value)] + } else { + vec![] + } + } +} + +/// Build a bare log at `address` with the given topics and empty data. +fn bare_log(address: Address, topics: Vec) -> Log { + Log::new_unchecked(address, topics, Bytes::new()) +} + +// =========================================================================== +// EventDecoder / DecoderRegistry — dispatch. +// =========================================================================== + +#[test] +fn registry_dispatches_address_scoped_then_global() { + let token_a = Address::repeat_byte(0x0a); + let token_b = Address::repeat_byte(0x0b); + + let mut registry = DecoderRegistry::new(); + // Address-scoped: only fires for token_a logs. + registry.register_for_address( + token_a, + Arc::new(MarkDecoder { + slot: U256::from(1), + value: U256::from(11), + }), + ); + // Global: fires for every log. + registry.register(Arc::new(MarkDecoder { + slot: U256::from(9), + value: U256::from(99), + })); + + let view = empty_view(); + + // A token_a log hits both the scoped decoder and the global one. + let updates_a = registry.decode(&bare_log(token_a, vec![]), &view); + assert_eq!(updates_a.len(), 2); + assert!(updates_a.contains(&StateUpdate::slot(token_a, U256::from(1), U256::from(11)))); + assert!(updates_a.contains(&StateUpdate::slot(token_a, U256::from(9), U256::from(99)))); + + // A token_b log hits only the global decoder. + let updates_b = registry.decode(&bare_log(token_b, vec![]), &view); + assert_eq!( + updates_b, + vec![StateUpdate::slot(token_b, U256::from(9), U256::from(99))] + ); +} + +#[test] +fn decoder_returns_empty_for_unrecognized_log() { + let topic = keccak256(b"SomethingElse()"); + let decoder = TaggedDecoder { + topic: keccak256(b"Wanted()"), + slot: U256::from(0), + value: U256::from(1), + }; + let view = empty_view(); + let log = bare_log(Address::repeat_byte(0x01), vec![topic]); + assert!(decoder.decode(&log, &view).is_empty()); +} + +// =========================================================================== +// ERC-20 Transfer decoder. +// =========================================================================== + +/// Build an ERC-20 `Transfer(from, to, value)` log. +fn transfer_log(token: Address, from: Address, to: Address, value: U256) -> Log { + let sig = keccak256(b"Transfer(address,address,uint256)"); + let topics = vec![sig, from.into_word(), to.into_word()]; + Log::new_unchecked( + token, + topics, + Bytes::copy_from_slice(&value.to_be_bytes::<32>()), + ) +} + +#[test] +fn erc20_transfer_decodes_to_sub_and_add_deltas() { + let token = Address::repeat_byte(0x20); + let from = Address::repeat_byte(0x21); + let to = Address::repeat_byte(0x22); + + let decoder = Erc20TransferDecoder::new(U256::from(3)); // default balance slot 3 + let view = empty_view(); + let updates = decoder.decode(&transfer_log(token, from, to, U256::from(100)), &view); + + assert_eq!( + updates, + vec![ + StateUpdate::slot_delta( + token, + mapping_slot(from, 3), + SlotDelta::Sub(U256::from(100)) + ), + StateUpdate::slot_delta(token, mapping_slot(to, 3), SlotDelta::Add(U256::from(100))), + ] + ); +} + +#[test] +fn erc20_mint_skips_zero_from_and_burn_skips_zero_to() { + let token = Address::repeat_byte(0x23); + let holder = Address::repeat_byte(0x24); + let decoder = Erc20TransferDecoder::new(U256::from(3)); + let view = empty_view(); + + // Mint: from == ZERO → only the Add leg. + let mint = decoder.decode( + &transfer_log(token, Address::ZERO, holder, U256::from(7)), + &view, + ); + assert_eq!( + mint, + vec![StateUpdate::slot_delta( + token, + mapping_slot(holder, 3), + SlotDelta::Add(U256::from(7)) + )] + ); + + // Burn: to == ZERO → only the Sub leg. + let burn = decoder.decode( + &transfer_log(token, holder, Address::ZERO, U256::from(7)), + &view, + ); + assert_eq!( + burn, + vec![StateUpdate::slot_delta( + token, + mapping_slot(holder, 3), + SlotDelta::Sub(U256::from(7)) + )] + ); +} + +#[test] +fn erc20_uses_per_token_slot_override_else_default() { + let token_default = Address::repeat_byte(0x25); + let token_custom = Address::repeat_byte(0x26); + let holder = Address::repeat_byte(0x27); + + let decoder = Erc20TransferDecoder::new(U256::from(3)).with_token(token_custom, U256::from(9)); + let view = empty_view(); + + let d = decoder.decode( + &transfer_log(token_default, Address::ZERO, holder, U256::from(1)), + &view, + ); + assert_eq!( + d[0], + StateUpdate::slot_delta( + token_default, + mapping_slot(holder, 3), + SlotDelta::Add(U256::from(1)) + ) + ); + + let c = decoder.decode( + &transfer_log(token_custom, Address::ZERO, holder, U256::from(1)), + &view, + ); + assert_eq!( + c[0], + StateUpdate::slot_delta( + token_custom, + mapping_slot(holder, 9), + SlotDelta::Add(U256::from(1)) + ) + ); +} + +#[test] +fn erc20_non_transfer_log_decodes_to_empty() { + let decoder = Erc20TransferDecoder::new(U256::from(3)); + let view = empty_view(); + // Wrong topic0. + let log = bare_log( + Address::repeat_byte(0x28), + vec![keccak256(b"Approval(address,address,uint256)")], + ); + assert!(decoder.decode(&log, &view).is_empty()); +} + +#[tokio::test] +async fn erc20_ingest_updates_balances_and_conserves() -> Result<()> { + use common::{balance_of, install_default_account}; + + let token = Address::repeat_byte(0x2a); + let alice = Address::repeat_byte(0x2b); + let bob = Address::repeat_byte(0x2c); + + let mut cache = setup_cache().await?; + install_default_account(&mut cache, Address::ZERO); + install_default_account(&mut cache, alice); + install_default_account(&mut cache, bob); + install_mock_erc20(&mut cache, token); + + // Seed both holders' balance slots (overlay-resident, EVM-visible). Balance + // mapping is slot 3 in the MockERC20 fixture. + cache + .db_mut() + .insert_account_storage(token, mapping_slot(alice, 3), U256::from(1000))?; + cache + .db_mut() + .insert_account_storage(token, mapping_slot(bob, 3), U256::from(500))?; + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(Erc20TransferDecoder::new(U256::from(3)))); + let mut pipeline = EventPipeline::new(registry); + + // Alice transfers 200 to Bob. + let digest = pipeline.ingest_logs( + &mut cache, + 100, + &[transfer_log(token, alice, bob, U256::from(200))], + ); + + // Two slot changes applied; nothing skipped. + assert_eq!(digest.block, 100); + assert_eq!(digest.applied.slots.len(), 2); + assert!(!digest.applied.has_skipped()); + assert_eq!(digest.decoded_logs, 1); + + // Balances move by the delta — assert via real SLOAD (balanceOf). + assert_eq!(balance_of(&mut cache, token, alice)?, U256::from(800)); + assert_eq!(balance_of(&mut cache, token, bob)?, U256::from(700)); + // Conservation. + assert_eq!( + balance_of(&mut cache, token, alice)? + balance_of(&mut cache, token, bob)?, + U256::from(1500) + ); + Ok(()) +} + +#[tokio::test] +async fn erc20_cold_balance_transfer_is_skipped_and_surfaced() -> Result<()> { + // A normal forked account (NOT StorageCleared): an unseeded balance slot is + // cold, so the Sub/Add delta is skipped and surfaced. + let token = Address::repeat_byte(0x2d); + let alice = Address::repeat_byte(0x2e); + let bob = Address::repeat_byte(0x2f); + + let mut cache = setup_cache().await?; + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(Erc20TransferDecoder::new(U256::from(3)))); + let mut pipeline = EventPipeline::new(registry); + + let digest = pipeline.ingest_logs( + &mut cache, + 1, + &[transfer_log(token, alice, bob, U256::from(10))], + ); + + // Both legs cold → no slot changes, two surfaced skips. + assert!(digest.applied.slots.is_empty()); + assert!(digest.applied.has_skipped()); + assert_eq!(digest.applied.skipped.len(), 2); + Ok(()) +} + +// =========================================================================== +// EventPipeline — reorg + reconcile (decoder-agnostic mechanics). +// =========================================================================== + +#[tokio::test] +async fn ingest_records_touched_slots() -> Result<()> { + let token = Address::repeat_byte(0x30); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(MarkDecoder { + slot: U256::from(5), + value: U256::from(42), + })); + let mut pipeline = EventPipeline::new(registry); + + let digest = pipeline.ingest_logs(&mut cache, 10, &[bare_log(token, vec![])]); + assert!(digest.touched_slots.contains(&(token, U256::from(5)))); + assert_eq!( + cache.cached_storage_value(token, U256::from(5)), + Some(U256::from(42)) + ); + Ok(()) +} + +#[tokio::test] +async fn reorg_to_purges_addresses_touched_after_head() -> Result<()> { + let token_a = Address::repeat_byte(0x31); // block 10 (survives) + let token_b = Address::repeat_byte(0x32); // block 11 (purged) + let token_c = Address::repeat_byte(0x33); // block 12 (purged) + let slot = U256::from(0); + + let mut cache = setup_cache().await?; + for t in [token_a, token_b, token_c] { + install_mock_erc20(&mut cache, t); + } + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(MarkDecoder { + slot, + value: U256::from(77), + })); + let mut pipeline = EventPipeline::new(registry); + + pipeline.ingest_logs(&mut cache, 10, &[bare_log(token_a, vec![])]); + pipeline.ingest_logs(&mut cache, 11, &[bare_log(token_b, vec![])]); + pipeline.ingest_logs(&mut cache, 12, &[bare_log(token_c, vec![])]); + + // Everything written. + assert_eq!(backend_slot(&cache, token_a, slot), Some(U256::from(77))); + assert_eq!(backend_slot(&cache, token_b, slot), Some(U256::from(77))); + assert_eq!(backend_slot(&cache, token_c, slot), Some(U256::from(77))); + + // Reorg back to block 10: purge B and C (touched after 10), keep A. + let diff = pipeline.reorg_to(&mut cache, 10); + + assert_eq!( + backend_slot(&cache, token_a, slot), + Some(U256::from(77)), + "A untouched" + ); + assert_eq!( + backend_slot(&cache, token_b, slot), + None, + "B storage purged" + ); + assert_eq!( + backend_slot(&cache, token_c, slot), + None, + "C storage purged" + ); + + // The returned diff records the purges (B and C only). + let purged: Vec
= diff.purged.iter().map(|r| r.address).collect(); + assert!(purged.contains(&token_b) && purged.contains(&token_c)); + assert!(!purged.contains(&token_a)); + Ok(()) +} + +#[tokio::test] +async fn reorg_config_account_scope_fully_drops_account() -> Result<()> { + let token = Address::repeat_byte(0x34); + let slot = U256::from(0); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(MarkDecoder { + slot, + value: U256::from(5), + })); + let mut pipeline = EventPipeline::new(registry).with_reorg_config(ReorgConfig { + depth: 64, + scope: PurgeScope::Account, + }); + + pipeline.ingest_logs(&mut cache, 20, &[bare_log(token, vec![])]); + let diff = pipeline.reorg_to(&mut cache, 19); + + assert!( + diff.purged + .iter() + .any(|r| r.address == token && r.account_removed) + ); + Ok(()) +} + +#[tokio::test] +async fn reconcile_reports_mismatch_and_corrects() -> Result<()> { + let token = Address::repeat_byte(0x35); + let slot = U256::from(0); + + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + + // Event pipeline writes an (incorrect) value 50. + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(MarkDecoder { + slot, + value: U256::from(50), + })); + let mut pipeline = EventPipeline::new(registry); + pipeline.ingest_logs(&mut cache, 1, &[bare_log(token, vec![])]); + assert_eq!( + cache.cached_storage_value(token, slot), + Some(U256::from(50)) + ); + + // Chain truth is 100. Reconcile must surface the drift AND correct the cache. + cache.set_storage_batch_fetcher(stub_fetcher(HashMap::from([( + (token, slot), + U256::from(100), + )]))); + let report = pipeline.reconcile(&mut cache, &[(token, slot)])?; + + assert_eq!(report.checked, 1); + assert_eq!(report.mismatched.len(), 1); + assert_eq!(report.mismatched[0].old, U256::from(50)); + assert_eq!(report.mismatched[0].new, U256::from(100)); + // Cache corrected to chain truth. + assert_eq!( + cache.cached_storage_value(token, slot), + Some(U256::from(100)) + ); + Ok(()) +} + +#[tokio::test] +async fn reconcile_empty_when_event_state_matches_chain() -> Result<()> { + let token = Address::repeat_byte(0x36); + let slot = U256::from(0); + + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(MarkDecoder { + slot, + value: U256::from(100), + })); + let mut pipeline = EventPipeline::new(registry); + pipeline.ingest_logs(&mut cache, 1, &[bare_log(token, vec![])]); + + // Chain agrees (100). + cache.set_storage_batch_fetcher(stub_fetcher(HashMap::from([( + (token, slot), + U256::from(100), + )]))); + let report = pipeline.reconcile(&mut cache, &[(token, slot)])?; + assert!(report.mismatched.is_empty()); + Ok(()) +} + +#[tokio::test] +async fn reconcile_errors_without_fetcher() -> Result<()> { + let mut cache = setup_cache().await?; + let registry = DecoderRegistry::new(); + let mut pipeline = EventPipeline::new(registry); + let token = Address::repeat_byte(0x37); + assert!( + pipeline + .reconcile(&mut cache, &[(token, U256::from(0))]) + .is_err() + ); + Ok(()) +} + +// =========================================================================== +// UniswapV3 adapter (protocols-gated). +// =========================================================================== + +#[cfg(feature = "protocols")] +mod uniswap_v3 { + use super::*; + use alloy_primitives::I256; + use alloy_primitives::aliases::{I24, U160}; + use alloy_sol_types::{SolEvent, sol}; + use evm_fork_cache::cache::{ + V3_LIQUIDITY_SLOT, V3_SLOT0_SLOT, V3_TICK_BITMAP_BASE_SLOT, V3_TICKS_BASE_SLOT, + v3_tick_bitmap_storage_key_with_base, v3_tick_info_storage_keys_with_base, + }; + use evm_fork_cache::events::uniswap_v3::{UniswapV3Decoder, UniswapV3Layout}; + + sol! { + event Swap(address indexed sender, address indexed recipient, int256 amount0, int256 amount1, uint160 sqrtPriceX96, uint128 liquidity, int24 tick); + event Mint(address sender, address indexed owner, int24 indexed tickLower, int24 indexed tickUpper, uint128 amount, uint256 amount0, uint256 amount1); + event Burn(address indexed owner, int24 indexed tickLower, int24 indexed tickUpper, uint128 amount, uint256 amount0, uint256 amount1); + } + + fn swap_log(pool: Address, sqrt_price: u128, liquidity: u128, tick: i32) -> Log { + let ev = Swap { + sender: Address::repeat_byte(0x01), + recipient: Address::repeat_byte(0x02), + amount0: I256::try_from(-1i64).unwrap(), + amount1: I256::try_from(1i64).unwrap(), + sqrtPriceX96: U160::from(sqrt_price), + liquidity, + tick: I24::try_from(tick).unwrap(), + }; + Log { + address: pool, + data: ev.encode_log_data(), + } + } + + fn mint_log(pool: Address, tick_lower: i32, tick_upper: i32, amount: u128) -> Log { + let ev = Mint { + sender: Address::repeat_byte(0x03), + owner: Address::repeat_byte(0x04), + tickLower: I24::try_from(tick_lower).unwrap(), + tickUpper: I24::try_from(tick_upper).unwrap(), + amount, + amount0: U256::from(1), + amount1: U256::from(1), + }; + Log { + address: pool, + data: ev.encode_log_data(), + } + } + + fn burn_log(pool: Address, tick_lower: i32, tick_upper: i32, amount: u128) -> Log { + let ev = Burn { + owner: Address::repeat_byte(0x04), + tickLower: I24::try_from(tick_lower).unwrap(), + tickUpper: I24::try_from(tick_upper).unwrap(), + amount, + amount0: U256::from(1), + amount1: U256::from(1), + }; + Log { + address: pool, + data: ev.encode_log_data(), + } + } + + /// Pack a slot0 word: sqrtPriceX96 [0,160), tick [160,184) (int24), and an + /// arbitrary `high` block of preserved bits at [184,256). + fn pack_slot0(sqrt_price: u128, tick: i32, high: U256) -> U256 { + let tick24 = U256::from((tick as u32) & 0x00FF_FFFF); + U256::from(sqrt_price) | (tick24 << 160) | (high << 184) + } + + fn unpack_tick_word(w: U256) -> (u128, i128) { + let gross = u128::try_from(w & U256::from(u128::MAX)).unwrap(); + let net = u128::try_from((w >> 128) & U256::from(u128::MAX)).unwrap() as i128; + (gross, net) + } + + fn pool_with_decoder(tick_spacing: i32) -> (Address, UniswapV3Decoder) { + let pool = Address::repeat_byte(0x40); + let decoder = + UniswapV3Decoder::new().with_pool(pool, UniswapV3Layout::uniswap(tick_spacing)); + (pool, decoder) + } + + #[tokio::test] + async fn v3_swap_sets_price_and_tick_preserving_unlocked() -> Result<()> { + let (pool, decoder) = pool_with_decoder(1); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, pool); + + // Seed slot0 with old price/tick AND the unlocked bit (240) + a nonzero + // observation index (bits 184+). high = unlocked(bit 56 of high) | obs(=7). + let high = (U256::from(1) << 56) | U256::from(7); + let seeded = pack_slot0(1_000_000, 50, high); + cache + .db_mut() + .insert_account_storage(pool, V3_SLOT0_SLOT, seeded)?; + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(decoder)); + let mut pipeline = EventPipeline::new(registry); + + pipeline.ingest_logs(&mut cache, 1, &[swap_log(pool, 2_000_000, 9999, 75)]); + + let result = cache.cached_storage_value(pool, V3_SLOT0_SLOT).unwrap(); + // Low 184 bits are the new price + tick. + let low_mask = (U256::from(1) << 184) - U256::from(1); + let expected_low = U256::from(2_000_000u64) | (U256::from(75u64) << 160); + assert_eq!(result & low_mask, expected_low, "price+tick updated"); + // High bits (incl. unlocked) preserved. + assert_eq!(result >> 184, high, "observation/unlocked bits preserved"); + Ok(()) + } + + #[tokio::test] + async fn v3_swap_sets_liquidity_absolute() -> Result<()> { + let (pool, decoder) = pool_with_decoder(1); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, pool); + cache.db_mut().insert_account_storage( + pool, + V3_SLOT0_SLOT, + pack_slot0(1, 0, U256::from(1) << 56), + )?; + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(decoder)); + let mut pipeline = EventPipeline::new(registry); + + pipeline.ingest_logs(&mut cache, 1, &[swap_log(pool, 1, 123_456, 0)]); + assert_eq!( + cache.cached_storage_value(pool, V3_LIQUIDITY_SLOT), + Some(U256::from(123_456u64)) + ); + Ok(()) + } + + #[tokio::test] + async fn v3_swap_cold_slot0_is_skipped() -> Result<()> { + // Fresh pool with no account → slot0 is cold. + let pool = Address::repeat_byte(0x41); + let decoder = UniswapV3Decoder::new().with_pool(pool, UniswapV3Layout::uniswap(1)); + let mut cache = setup_cache().await?; + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(decoder)); + let mut pipeline = EventPipeline::new(registry); + + let digest = pipeline.ingest_logs(&mut cache, 1, &[swap_log(pool, 5, 5, 5)]); + // slot0 masked write skipped (un-masked bits unknown). + assert!(digest.applied.has_skipped()); + assert_eq!(cache.cached_storage_value(pool, V3_SLOT0_SLOT), None); + Ok(()) + } + + #[tokio::test] + async fn v3_mint_increments_gross_and_net_with_correct_signs() -> Result<()> { + let (pool, decoder) = pool_with_decoder(1); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, pool); // StorageCleared → unseeded ticks read 0 (hot) + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(decoder)); + let mut pipeline = EventPipeline::new(registry); + + let (lo, hi, amount) = (10i32, 20i32, 500u128); + pipeline.ingest_logs(&mut cache, 1, &[mint_log(pool, lo, hi, amount)]); + + let lo_key = v3_tick_info_storage_keys_with_base(lo, V3_TICKS_BASE_SLOT)[0]; + let hi_key = v3_tick_info_storage_keys_with_base(hi, V3_TICKS_BASE_SLOT)[0]; + let (lo_gross, lo_net) = + unpack_tick_word(cache.cached_storage_value(pool, lo_key).unwrap()); + let (hi_gross, hi_net) = + unpack_tick_word(cache.cached_storage_value(pool, hi_key).unwrap()); + + assert_eq!(lo_gross, 500); + assert_eq!(lo_net, 500, "lower tick: net += amount"); + assert_eq!(hi_gross, 500); + assert_eq!(hi_net, -500, "upper tick: net -= amount"); + Ok(()) + } + + #[tokio::test] + async fn v3_mint_initializes_tick_and_flips_bitmap() -> Result<()> { + let (pool, decoder) = pool_with_decoder(1); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, pool); + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(decoder)); + let mut pipeline = EventPipeline::new(registry); + + let (lo, hi) = (10i32, 20i32); + pipeline.ingest_logs(&mut cache, 1, &[mint_log(pool, lo, hi, 500)]); + + // initialized flag (slot +3, bit 248) set for both ticks. + let lo3 = v3_tick_info_storage_keys_with_base(lo, V3_TICKS_BASE_SLOT)[3]; + let hi3 = v3_tick_info_storage_keys_with_base(hi, V3_TICKS_BASE_SLOT)[3]; + let init_bit = U256::from(1) << 248; + assert_eq!( + cache.cached_storage_value(pool, lo3).unwrap() & init_bit, + init_bit + ); + assert_eq!( + cache.cached_storage_value(pool, hi3).unwrap() & init_bit, + init_bit + ); + + // bitmap word 0 (ticks 10 & 20 with tick_spacing 1 → word 0, bits 10 & 20). + let word_key = v3_tick_bitmap_storage_key_with_base(0, V3_TICK_BITMAP_BASE_SLOT); + let bitmap = cache.cached_storage_value(pool, word_key).unwrap(); + assert_eq!(bitmap & (U256::from(1) << 10), U256::from(1) << 10); + assert_eq!(bitmap & (U256::from(1) << 20), U256::from(1) << 20); + Ok(()) + } + + #[tokio::test] + async fn v3_burn_to_zero_uninitializes_and_clears_bitmap() -> Result<()> { + let (pool, decoder) = pool_with_decoder(1); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, pool); + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(decoder)); + let mut pipeline = EventPipeline::new(registry); + + let (lo, hi) = (10i32, 20i32); + // Same-block Mint then Burn of the full amount: the Burn decode sees the + // Mint's applied gross/net via the StateView, returning the tick to 0. + pipeline.ingest_logs( + &mut cache, + 1, + &[mint_log(pool, lo, hi, 500), burn_log(pool, lo, hi, 500)], + ); + + let lo_key = v3_tick_info_storage_keys_with_base(lo, V3_TICKS_BASE_SLOT)[0]; + let (lo_gross, lo_net) = + unpack_tick_word(cache.cached_storage_value(pool, lo_key).unwrap()); + assert_eq!(lo_gross, 0, "gross back to zero"); + assert_eq!(lo_net, 0, "net back to zero"); + + // initialized flag cleared and bitmap bit cleared. + let lo3 = v3_tick_info_storage_keys_with_base(lo, V3_TICKS_BASE_SLOT)[3]; + let init_bit = U256::from(1) << 248; + assert_eq!( + cache.cached_storage_value(pool, lo3).unwrap_or(U256::ZERO) & init_bit, + U256::ZERO + ); + let word_key = v3_tick_bitmap_storage_key_with_base(0, V3_TICK_BITMAP_BASE_SLOT); + assert_eq!( + cache + .cached_storage_value(pool, word_key) + .unwrap_or(U256::ZERO) + & (U256::from(1) << 10), + U256::ZERO + ); + Ok(()) + } + + #[tokio::test] + async fn v3_mint_updates_global_liquidity_when_in_range() -> Result<()> { + let (pool, decoder) = pool_with_decoder(1); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, pool); + + // Current tick 15 is within [10, 20); liquidity seeded to 1000. + cache.db_mut().insert_account_storage( + pool, + V3_SLOT0_SLOT, + pack_slot0(1_000_000, 15, U256::from(1) << 56), + )?; + cache + .db_mut() + .insert_account_storage(pool, V3_LIQUIDITY_SLOT, U256::from(1000))?; + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(decoder)); + let mut pipeline = EventPipeline::new(registry); + + pipeline.ingest_logs(&mut cache, 1, &[mint_log(pool, 10, 20, 500)]); + assert_eq!( + cache.cached_storage_value(pool, V3_LIQUIDITY_SLOT), + Some(U256::from(1500)), + "in-range mint adds to global liquidity" + ); + Ok(()) + } + + #[tokio::test] + async fn v3_mint_leaves_global_liquidity_when_out_of_range() -> Result<()> { + let (pool, decoder) = pool_with_decoder(1); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, pool); + + // Current tick 5 is BELOW [10, 20); liquidity must not change. + cache.db_mut().insert_account_storage( + pool, + V3_SLOT0_SLOT, + pack_slot0(1_000_000, 5, U256::from(1) << 56), + )?; + cache + .db_mut() + .insert_account_storage(pool, V3_LIQUIDITY_SLOT, U256::from(1000))?; + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(decoder)); + let mut pipeline = EventPipeline::new(registry); + + pipeline.ingest_logs(&mut cache, 1, &[mint_log(pool, 10, 20, 500)]); + assert_eq!( + cache.cached_storage_value(pool, V3_LIQUIDITY_SLOT), + Some(U256::from(1000)), + "out-of-range mint does not touch global liquidity" + ); + Ok(()) + } + + #[tokio::test] + async fn v3_mint_cold_tick_word_is_skipped() -> Result<()> { + // Fresh pool, no account → tick words are cold (None), so the tick + // maintenance is skipped and surfaced rather than computed against 0. + let pool = Address::repeat_byte(0x42); + let decoder = UniswapV3Decoder::new().with_pool(pool, UniswapV3Layout::uniswap(1)); + let mut cache = setup_cache().await?; + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(decoder)); + let mut pipeline = EventPipeline::new(registry); + + let digest = pipeline.ingest_logs(&mut cache, 1, &[mint_log(pool, 10, 20, 500)]); + assert!( + digest.applied.has_skipped(), + "cold tick words surfaced as skips" + ); + let lo_key = v3_tick_info_storage_keys_with_base(10, V3_TICKS_BASE_SLOT)[0]; + assert_eq!( + cache.cached_storage_value(pool, lo_key), + None, + "nothing written" + ); + Ok(()) + } + + #[tokio::test] + async fn v3_unregistered_pool_decodes_to_nothing() -> Result<()> { + let known = Address::repeat_byte(0x43); + let unknown = Address::repeat_byte(0x44); + let decoder = UniswapV3Decoder::new().with_pool(known, UniswapV3Layout::uniswap(1)); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, unknown); + + let mut registry = DecoderRegistry::new(); + registry.register(Arc::new(decoder)); + let mut pipeline = EventPipeline::new(registry); + + let digest = pipeline.ingest_logs(&mut cache, 1, &[swap_log(unknown, 5, 5, 5)]); + assert!(digest.applied.is_empty() && !digest.applied.has_skipped()); + assert_eq!(digest.decoded_logs, 0); + Ok(()) + } +} diff --git a/tests/freshness.rs b/tests/freshness.rs index a892883..aa2455a 100644 --- a/tests/freshness.rs +++ b/tests/freshness.rs @@ -16,8 +16,8 @@ use alloy_sol_types::SolCall; use anyhow::Result; use common::{ - MOCK_ERC20_BALANCE_SLOT, MockERC20, failing_fetcher, install_default_account, - install_mock_erc20, panicking_fetcher, setup_cache, stub_fetcher, tracking_fetcher, + Gate, MOCK_ERC20_BALANCE_SLOT, MockERC20, failing_fetcher, gated_tracking_fetcher, + install_default_account, install_mock_erc20, panicking_fetcher, setup_cache, stub_fetcher, }; use evm_fork_cache::cache::{ EvmCache, EvmOverlay, SimStatus, SlotObservationTracker, StorageBatchFetchFn, @@ -73,11 +73,18 @@ async fn verify_slots_detects_and_injects_changes() -> Result<()> { let slot_a = U256::from(10); let slot_b = U256::from(20); - // Cache holds these values. - cache.inject_storage_batch(&[ - (contract, slot_a, U256::from(100)), - (contract, slot_b, U256::from(200)), - ]); + // Cache holds these values, seeded OVERLAY-resident so they are EVM-visible: + // `contract` is a StorageCleared MockERC20, and after the §16.0 fix a + // backend-only `inject_storage_batch` seed on a StorageCleared account is + // shadowed to ZERO by `cached_storage_value` (it mirrors the EVM SLOAD). The + // test's intent is that the cache *holds* these values, so seed the layer that + // actually wins (mirrors `state_update::balance_tracking_scenario`). + cache + .db_mut() + .insert_account_storage(contract, slot_a, U256::from(100))?; + cache + .db_mut() + .insert_account_storage(contract, slot_b, U256::from(200))?; // Stub reports slot_a changed, slot_b unchanged. let values = HashMap::from([ @@ -115,7 +122,13 @@ async fn verify_slots_unchanged_returns_empty() -> Result<()> { install_mock_erc20(&mut cache, contract); let slot = U256::from(7); - cache.inject_storage_batch(&[(contract, slot, U256::from(42))]); + // Overlay-resident seed so the value is EVM-visible (see the note in + // `verify_slots_detects_and_injects_changes`): a backend-only seed on this + // StorageCleared MockERC20 would read as ZERO under the §16.0 fix, so the + // fetcher's matching 42 would (incorrectly) look like a 0 -> 42 change. + cache + .db_mut() + .insert_account_storage(contract, slot, U256::from(42))?; cache.set_storage_batch_fetcher(stub_fetcher(HashMap::from([( (contract, slot), U256::from(42), @@ -235,7 +248,7 @@ async fn purge_account_drops_account_and_storage_from_both_layers() -> Result<() ); // Account gone from the backend accounts map. { - let accounts = cache.blockchain_db().accounts().read(); + let accounts = cache.unchecked_blockchain_db().accounts().read(); assert!(!accounts.contains_key(&token), "backend account removed"); } @@ -322,7 +335,13 @@ async fn cache_with_balance(token: Address, owner: Address, balance: U256) -> Re install_default_account(&mut cache, owner); install_mock_erc20(&mut cache, token); if balance > U256::ZERO { - cache.inject_storage_batch(&[(token, balance_slot_for(owner), balance)]); + // Overlay-resident seed so the balance is EVM-visible: `token` is a + // StorageCleared MockERC20, so a backend-only seed reads as ZERO via the + // account_state-aware read path (invisible to the optimistic sim and the + // snapshot). Mirrors `state_update::balance_tracking_scenario`. + cache + .db_mut() + .insert_account_storage(token, balance_slot_for(owner), balance)?; } Ok(cache) } @@ -377,10 +396,14 @@ async fn run_mismatch_path_corrected_only_affected_rerun() -> Result<()> { install_default_account(&mut cache, owner2); install_mock_erc20(&mut cache, token); install_mock_erc20(&mut cache, token2); - cache.inject_storage_batch(&[ - (token, balance_slot_for(owner), U256::from(1000)), - (token2, balance_slot_for(owner2), U256::from(5000)), - ]); + // Overlay-resident seeds (EVM-visible): both tokens are StorageCleared, so a + // backend-only seed would read ZERO via the account_state-aware read path. + cache + .db_mut() + .insert_account_storage(token, balance_slot_for(owner), U256::from(1000))?; + cache + .db_mut() + .insert_account_storage(token2, balance_slot_for(owner2), U256::from(5000))?; // Fetcher: owner's balance slot DROPPED to 50 (< the 100 transfer, so the // re-run now reverts); owner2's slot unchanged; recipient slots read as zero @@ -547,7 +570,15 @@ async fn run_drains_pending_on_next_run() -> Result<()> { install_default_account(&mut cache, Address::ZERO); install_default_account(&mut cache, owner); install_mock_erc20(&mut cache, token); - cache.inject_storage_batch(&[(token, balance_slot_for(owner), U256::from(1000))]); + // Overlay-resident seed so the balance is EVM-visible on the StorageCleared + // token account (see the note in `verify_slots_detects_and_injects_changes`): + // after the §16.0 fix, a backend-only `inject_storage_batch` seed here reads as + // ZERO via `cached_storage_value` (mirroring the SLOAD), so the live-cache + // assertions below would observe 0 instead of the seeded value. This mirrors + // `state_update::balance_tracking_scenario`. + cache + .db_mut() + .insert_account_storage(token, balance_slot_for(owner), U256::from(1000))?; cache.set_storage_batch_fetcher(stub_fetcher(HashMap::from([( (token, balance_slot_for(owner)), U256::from(2000), @@ -746,7 +777,11 @@ async fn optimistic_result_reports_status_per_outcome() -> Result<()> { install_default_account(&mut cache, owner); install_mock_erc20(&mut cache, token); let slot = balance_slot_for(owner); - cache.inject_storage_batch(&[(token, slot, U256::from(1000))]); + // Overlay-resident (EVM-visible) seed: `token` is a StorageCleared MockERC20, + // so a backend-only seed reads ZERO via the account_state-aware read path. + cache + .db_mut() + .insert_account_storage(token, slot, U256::from(1000))?; cache.set_storage_batch_fetcher(stub_fetcher(HashMap::from([( (token, slot), U256::from(1000), @@ -802,7 +837,11 @@ async fn dropping_after_fetch_started_suppresses_correction() -> Result<()> { install_default_account(&mut cache, owner); install_mock_erc20(&mut cache, token); let slot = balance_slot_for(owner); - cache.inject_storage_batch(&[(token, slot, U256::from(1000))]); + // Overlay-resident (EVM-visible) seed: `token` is a StorageCleared MockERC20, + // so a backend-only seed reads ZERO via the account_state-aware read path. + cache + .db_mut() + .insert_account_storage(token, slot, U256::from(1000))?; // Two rendezvous: R1 = "fetch started", R2 = "released by the test". After R2 // the fetcher reports a CHANGED balance, so absent the cancel the validator @@ -882,7 +921,10 @@ async fn run_corrected_rerun_verifies_newly_read_volatile_slot() -> Result<()> { let slot_a = U256::from(0); let slot_b = U256::from(1); // Snapshot: A = 5 (nonzero) → optimistic takes "return A" and never reads B. - cache.inject_storage_batch(&[(contract, slot_a, U256::from(5))]); + // Overlay-resident (EVM-visible) seed: `contract` is StorageCleared. + cache + .db_mut() + .insert_account_storage(contract, slot_a, U256::from(5))?; // Fresh chain: A dropped to 0 (flips the branch) and B is 777. cache.set_storage_batch_fetcher(stub_fetcher(HashMap::from([ ((contract, slot_a), U256::from(0)), @@ -996,7 +1038,11 @@ async fn run_honors_tx_gas_limit() -> Result<()> { install_default_account(&mut cache, owner); install_mock_erc20(&mut cache, token); let slot = balance_slot_for(owner); - cache.inject_storage_batch(&[(token, slot, U256::from(1000))]); + // Overlay-resident (EVM-visible) seed: `token` is a StorageCleared MockERC20, + // so a backend-only seed reads ZERO via the account_state-aware read path. + cache + .db_mut() + .insert_account_storage(token, slot, U256::from(1000))?; cache.set_storage_batch_fetcher(stub_fetcher(HashMap::from([( (token, slot), U256::from(1000), @@ -1036,11 +1082,17 @@ async fn validator_fetches_at_snapshot_block_despite_repin() -> Result<()> { install_default_account(&mut cache, owner); install_mock_erc20(&mut cache, token); cache.set_block(Some(block_n)); - cache.inject_storage_batch(&[(token, slot, U256::from(1000))]); + // Overlay-resident seed so the balance is EVM-visible on the StorageCleared + // token account (see the note in `verify_slots_detects_and_injects_changes`): + // a backend-only seed would read as ZERO under the §16.0 `cached_storage_value` + // fix, failing the precondition below. + cache + .db_mut() + .insert_account_storage(token, slot, U256::from(1000))?; assert_eq!( cache.cached_storage_value(token, slot), Some(U256::from(1000)), - "PRECONDITION: seeded balance present after set_block + inject" + "PRECONDITION: seeded balance present after set_block + insert" ); // Block-aware fetcher: the snapshot value (1000) at block N, a CHANGED value @@ -1133,6 +1185,12 @@ async fn run_unverified_on_fetcher_error() -> Result<()> { // T3 (part 2): into_optimistic aborts the validation task. The fetcher WOULD // queue a correction (it reports a changed value), so if the abort failed we // would observe a non-zero pending queue. We assert it stays 0. +// +// Determinism mirrors the Drop-abort test below: the validator is allowed to +// reach the synchronous fetch, but the gated fetch cannot return until after +// `into_optimistic()` has set the cancel flag. That makes the product guarantee +// precise: a cancel observed at the post-fetch checkpoint suppresses all +// side-effects, including pending corrections and re-run accounting. #[tokio::test(flavor = "multi_thread")] async fn run_into_optimistic_aborts_validation() -> Result<()> { let token = Address::repeat_byte(0x44); @@ -1140,11 +1198,12 @@ async fn run_into_optimistic_aborts_validation() -> Result<()> { let recipient = Address::repeat_byte(0x66); let mut cache = cache_with_balance(token, owner, U256::from(1000)).await?; + let gate = Gate::new(); // A CHANGED value: if the validator ran, it would queue a correction. - cache.set_storage_batch_fetcher(stub_fetcher(HashMap::from([( - (token, balance_slot_for(owner)), - U256::from(50), - )]))); + cache.set_storage_batch_fetcher(gated_tracking_fetcher( + HashMap::from([((token, balance_slot_for(owner)), U256::from(50))]), + gate.clone(), + )); let mut controller = FreshnessController::new(FreshnessRegistry::new(), AlwaysVerify); let sim = controller.run( @@ -1156,6 +1215,7 @@ async fn run_into_optimistic_aborts_validation() -> Result<()> { )], )?; let results = sim.into_optimistic(); // aborts the background validation + gate.release(); assert_eq!(results.len(), 1); // Give any (incorrectly) surviving task a chance to run, then assert no @@ -1172,9 +1232,20 @@ async fn run_into_optimistic_aborts_validation() -> Result<()> { // T3 (part 1): dropping the SpeculativeSim (no validate/into_optimistic) aborts // the validation task before it can push a correction. The fetcher reports a -// CHANGED value and flips a "called" flag; after the drop + settle we assert the -// pending queue is empty (and, robustly, that the fetcher was never even -// reached) — proving the abort beat the push. +// CHANGED value, so an *uncancelled* validator would queue a correction and bump +// the re-run count; we assert neither happens after the drop. +// +// Determinism: the validator's only correction-queuing path runs *after* its +// fetch returns (the post-fetch cancel checkpoint in `run_validator` gates it). +// We make that ordering race-free with a gate the test controls — the fetcher +// blocks until `gate.release()`, and we release only *after* `drop(sim)` has set +// the cancel flag. So however the multi-thread scheduler interleaves the spawned +// task and this thread, the fetch (and thus the post-fetch checkpoint) can only +// complete once cancellation is already observable, and the correction is +// suppressed. We deliberately do NOT assert the fetcher was never reached: the +// product only guarantees a cancel seen at a checkpoint suppresses side effects, +// not that an in-flight fetch is skipped — asserting the latter was the original +// over-strict, racy condition. #[tokio::test(flavor = "multi_thread")] async fn dropping_speculative_sim_aborts_before_queueing_correction() -> Result<()> { let token = Address::repeat_byte(0x44); @@ -1182,10 +1253,10 @@ async fn dropping_speculative_sim_aborts_before_queueing_correction() -> Result< let recipient = Address::repeat_byte(0x66); let mut cache = cache_with_balance(token, owner, U256::from(1000)).await?; - let called = Arc::new(std::sync::atomic::AtomicBool::new(false)); - cache.set_storage_batch_fetcher(tracking_fetcher( + let gate = Gate::new(); + cache.set_storage_batch_fetcher(gated_tracking_fetcher( HashMap::from([((token, balance_slot_for(owner)), U256::from(50))]), - Arc::clone(&called), + gate.clone(), )); let mut controller = FreshnessController::new(FreshnessRegistry::new(), AlwaysVerify); @@ -1197,9 +1268,12 @@ async fn dropping_speculative_sim_aborts_before_queueing_correction() -> Result< transfer_calldata(recipient, U256::from(100)), )], )?; - // Drop immediately, with NO intervening await, so the abort flag is set - // before the spawned task is ever polled. + // Drop with NO intervening await, then release the gate. Releasing only after + // the drop guarantees the validator's fetch (if it even reaches it) returns + // strictly after the cancel flag is set, so its post-fetch checkpoint bails + // out before queuing anything. drop(sim); + gate.release(); settle().await; @@ -1208,10 +1282,6 @@ async fn dropping_speculative_sim_aborts_before_queueing_correction() -> Result< 0, "dropping the sim must abort validation before it queues a correction" ); - assert!( - !called.load(std::sync::atomic::Ordering::SeqCst), - "the aborted validator should never have reached the fetcher" - ); assert_eq!(controller.rerun_count(), 0, "no re-run after abort"); Ok(()) } @@ -1510,8 +1580,8 @@ async fn run_unverified_without_fetcher() -> Result<()> { // A `from_backend` cache exposes no fetcher (no provider captured). let base = cache_with_balance(token, owner, U256::from(1000)).await?; let mut cache = EvmCache::from_backend( - base.backend().clone(), - base.blockchain_db().clone(), + base.unchecked_backend().clone(), + base.unchecked_blockchain_db().clone(), None, base.chain_id(), None, @@ -1664,3 +1734,122 @@ async fn on_new_block_ages_valid_through() -> Result<()> { ); Ok(()) } + +// =========================================================================== +// Phase 2 review (trust-contract hardening): the validator must NEVER return a +// trusted verdict on incomplete/ambiguous verification. +// =========================================================================== + +/// P2: a custom fetcher that OMITS a requested slot must yield `Unverified`, not +/// a false `Confirmed`/`Corrected` (missing results must not default to zero). +#[tokio::test(flavor = "multi_thread")] +async fn run_unverified_when_fetcher_omits_requested_slot() -> Result<()> { + let token = Address::repeat_byte(0x11); + let owner = Address::repeat_byte(0x22); + let recipient = Address::repeat_byte(0x33); + + let mut cache = cache_with_balance(token, owner, U256::from(1000)).await?; + // A fetcher that returns NOTHING — it omits every requested slot. + cache.set_storage_batch_fetcher(Arc::new( + |_req: Vec<(Address, U256)>, _block: Option| { + Vec::<(Address, U256, Result)>::new() + }, + )); + + let mut controller = FreshnessController::new(FreshnessRegistry::new(), AlwaysVerify); + let req = SimRequest::new(owner, token, transfer_calldata(recipient, U256::from(100))); + let sim = controller.run(&mut cache, vec![req])?; + + let validation = sim.validate().await; + assert!( + matches!(validation, Validation::Unverified { .. }), + "a fetcher that omits a requested slot must yield Unverified, not a false \ + confirmation/correction: {validation:?}" + ); + assert_eq!( + controller.pending_len(), + 0, + "Unverified must not queue any correction" + ); + Ok(()) +} + +/// Build runtime bytecode that reads slots `0..n` in order, returning the first +/// nonzero one (else zero). Reading slot `i+1` is gated on slot `i` being zero, so +/// each correction (slot → 0) opens exactly one new volatile slot — driving the +/// validator's fixed-point loop one round deeper per correction. +fn chained_sload_bytecode(n: u8) -> Bytes { + let ret_dest = 8u16 * (n as u16) + 2; // JUMPDEST offset (after the chain + PUSH1 0) + assert!(ret_dest <= 255, "return dest must fit in PUSH1"); + let ret = ret_dest as u8; + let mut code = Vec::new(); + for i in 0..n { + code.extend_from_slice(&[0x60, i, 0x54, 0x80, 0x60, ret, 0x57, 0x50]); + // PUSH1 i; SLOAD; DUP1; PUSH1 ret; JUMPI (if nonzero -> return it); POP + } + code.extend_from_slice(&[0x60, 0x00]); // all zero: PUSH1 0 + code.extend_from_slice(&[0x5b, 0x60, 0x00, 0x52, 0x60, 0x20, 0x60, 0x00, 0xf3]); + // JUMPDEST; PUSH1 0; MSTORE; PUSH1 0x20; PUSH1 0; RETURN (store TOS, return 32 bytes) + Bytes::from(code) +} + +/// P1: when corrections keep opening new volatile slots past +/// `MAX_VALIDATION_ROUNDS`, the validator must return `Unverified` — NOT a +/// best-effort (trusted) `Corrected` resting on un-verified state — and must +/// queue no corrections. +#[tokio::test(flavor = "multi_thread")] +async fn run_unverified_when_fixed_point_round_cap_exceeded() -> Result<()> { + use revm::state::{AccountInfo, Bytecode}; + + let mut cache = setup_cache().await?; + install_default_account(&mut cache, Address::ZERO); + let caller = Address::repeat_byte(0x66); + install_default_account(&mut cache, caller); + + // A 12-deep chain: each corrected slot opens the next, so the loop needs one + // round per slot — exceeding the 8-round cap well before the chain runs out. + let contract = Address::repeat_byte(0x55); + let code = Bytecode::new_raw(chained_sload_bytecode(12)); + let code_hash = code.hash_slow(); + cache.db_mut().insert_account_info( + contract, + AccountInfo { + balance: U256::ZERO, + nonce: 0, + code: Some(code), + code_hash, + account_id: None, + }, + ); + cache + .db_mut() + .replace_account_storage(contract, Default::default()) + .unwrap(); + // Snapshot: slots 0..12 all nonzero (EVM-visible overlay seed). The optimistic + // run reads only slot 0 (nonzero → returns). + for i in 0..12u64 { + cache + .db_mut() + .insert_account_storage(contract, U256::from(i), U256::from(1))?; + } + // Fresh chain: every slot dropped to 0 (stub returns 0 for all), so each + // correction flips the next branch and the loop never reaches a fixed point. + cache.set_storage_batch_fetcher(stub_fetcher(HashMap::new())); + + let mut controller = FreshnessController::new(FreshnessRegistry::new(), AlwaysVerify); + let req = SimRequest::new(caller, contract, Bytes::new()); + let sim = controller.run(&mut cache, vec![req])?; + + let validation = sim.validate().await; + assert!( + matches!(validation, Validation::Unverified { .. }), + "exceeding the fixed-point round cap must yield Unverified, not a trusted \ + Corrected: {validation:?}" + ); + assert_eq!( + controller.pending_len(), + 0, + "an Unverified (cap-exceeded) validation must queue no corrections" + ); + Ok(()) +} diff --git a/tests/serialization_roundtrip.rs b/tests/serialization_roundtrip.rs index 71850cf..27bdc32 100644 --- a/tests/serialization_roundtrip.rs +++ b/tests/serialization_roundtrip.rs @@ -87,6 +87,16 @@ fn immutable_data_cache_round_trips() { let len_before = cache.len(); cache.save(&path).expect("save immutable cache"); + let bytes = std::fs::read(&path).expect("read immutable cache file"); + assert!( + bytes.starts_with(b"EFCMETA\0"), + "immutable cache must carry a magic header" + ); + assert_eq!( + &bytes[8..12], + &1u32.to_le_bytes(), + "immutable cache must carry an explicit version" + ); let loaded = ImmutableDataCache::load(&path).expect("load immutable cache"); // Counts and scalar values survive the round trip. @@ -118,6 +128,20 @@ fn immutable_data_cache_round_trips() { assert_eq!(bal.last_change_block, U256::from(18_000_000u64)); } +#[test] +fn immutable_data_cache_load_legacy_raw_bincode_is_none() { + let dir = TempDir::new("immutable_legacy"); + let path = dir.path("legacy_immutable_data.bin"); + let mut cache = ImmutableDataCache::default(); + cache.set_token_decimals(Address::repeat_byte(0xA1), 6); + std::fs::write(&path, bincode::serialize(&cache).unwrap()).expect("write legacy cache"); + + assert!( + ImmutableDataCache::load(&path).is_none(), + "unversioned legacy bincode must be treated as a cache miss" + ); +} + #[test] fn immutable_data_cache_load_missing_file_is_none() { let dir = TempDir::new("immutable_missing"); @@ -180,6 +204,16 @@ mod tick_snapshots { assert_eq!(cache.len(), 1); cache.save(&path).expect("save tick cache"); + let bytes = std::fs::read(&path).expect("read tick cache file"); + assert!( + bytes.starts_with(b"EFCTICK\0"), + "tick snapshot cache must carry a magic header" + ); + assert_eq!( + &bytes[8..12], + &1u32.to_le_bytes(), + "tick snapshot cache must carry an explicit version" + ); let loaded = V3TickSnapshotCache::load(&path).expect("load tick cache"); let snap = loaded.get(pool).expect("snapshot present"); @@ -190,6 +224,24 @@ mod tick_snapshots { assert_eq!(snap.to_ticks(), ticks, "ticks survive round trip"); } + #[test] + fn v3_tick_snapshot_cache_load_legacy_raw_bincode_is_none() { + let dir = TempDir::new("v3_ticks_legacy"); + let path = dir.path("legacy_v3_tick_snapshots.bin"); + let pool = Address::repeat_byte(0x77); + let mut cache = V3TickSnapshotCache::default(); + cache.set( + pool, + V3PoolTickSnapshot::from_pool_data(&HashMap::new(), &HashMap::new(), 0, 0), + ); + std::fs::write(&path, bincode::serialize(&cache).unwrap()).expect("write legacy cache"); + + assert!( + V3TickSnapshotCache::load(&path).is_none(), + "unversioned legacy bincode must be treated as a cache miss" + ); + } + #[test] fn v3_tick_snapshot_silently_drops_unparseable_keys() { // Pin the documented behavior (KNOWN_ISSUES): a string key that does not diff --git a/tests/shared_memory_capacity.rs b/tests/shared_memory_capacity.rs new file mode 100644 index 0000000..f3fc029 --- /dev/null +++ b/tests/shared_memory_capacity.rs @@ -0,0 +1,137 @@ +//! Offline tests for the configurable EVM shared-memory pre-allocation +//! ([`SharedMemoryCapacity`]) wired through [`EvmCacheBuilder`]. +//! +//! Covers the three user-facing behaviors: the default, an explicit `Fixed` size, +//! and `Auto` sizing from the chain state loaded at build time (the +//! "intelligently allocate from a bincode state file" path). All offline. + +use std::sync::Arc; +use std::time::{SystemTime, UNIX_EPOCH}; + +use alloy_primitives::{Address, U256}; +use alloy_provider::RootProvider; +use alloy_provider::network::AnyNetwork; +use alloy_rpc_client::RpcClient; +use alloy_transport::mock::Asserter; +use anyhow::Result; +use evm_fork_cache::cache::{CacheConfig, EvmCacheBuilder, SharedMemoryCapacity}; + +fn mock_provider() -> Arc> { + Arc::new(RootProvider::::new(RpcClient::mocked( + Asserter::new(), + ))) +} + +/// A unique temp dir for a disk-backed cache (no two tests collide). +fn unique_cache_dir(tag: &str) -> std::path::PathBuf { + let nanos = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_nanos(); + std::env::temp_dir().join(format!("evm_fork_cache_smc_{tag}_{nanos}")) +} + +#[tokio::test(flavor = "multi_thread")] +async fn default_capacity_is_fixed_64k() -> Result<()> { + let cache = EvmCacheBuilder::new(mock_provider()).build().await; + assert_eq!( + cache.shared_memory_capacity(), + 65_536, + "the default must be Fixed(64 * 1024)" + ); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn fixed_capacity_is_honored() -> Result<()> { + let cache = EvmCacheBuilder::new(mock_provider()) + .shared_memory_capacity(SharedMemoryCapacity::Fixed(8_192)) + .build() + .await; + assert_eq!(cache.shared_memory_capacity(), 8_192); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn auto_capacity_with_no_loaded_state_falls_back_to_floor() -> Result<()> { + // No cache_config → nothing loaded → Auto resolves to the 64 KiB floor. + let cache = EvmCacheBuilder::new(mock_provider()) + .shared_memory_capacity(SharedMemoryCapacity::Auto) + .build() + .await; + assert_eq!( + cache.shared_memory_capacity(), + SharedMemoryCapacity::MIN_AUTO + ); + Ok(()) +} + +/// The headline: `Auto` sizes the buffer from the chain state in a loaded bincode +/// state file. A first cache persists 10 000 storage slots; a second cache built +/// with `Auto` over the same `CacheConfig` loads them and pre-allocates +/// `10_000 * 16 = 160_000` bytes (vs. the 64 KiB default). +#[tokio::test(flavor = "multi_thread")] +async fn auto_capacity_scales_with_loaded_binary_state() -> Result<()> { + let dir = unique_cache_dir("auto"); + let cfg = CacheConfig::new(&dir, 1, Default::default(), Default::default()); + + // First cache: seed 10k slots into layer 2 and persist to the bincode state file. + { + let mut cache = EvmCacheBuilder::new(mock_provider()) + .cache_config(cfg.clone()) + .build() + .await; + let token = Address::repeat_byte(0x11); + let batch: Vec<(Address, U256, U256)> = (0..10_000u64) + .map(|i| (token, U256::from(i), U256::from(i + 1))) + .collect(); + cache.inject_storage_batch(&batch); + cache.flush()?; // writes evm_state.bin + } + + // Second cache: Auto over the same config loads the 10k slots and sizes from them. + let reloaded = EvmCacheBuilder::new(mock_provider()) + .cache_config(cfg.clone()) + .shared_memory_capacity(SharedMemoryCapacity::Auto) + .build() + .await; + assert_eq!( + reloaded.shared_memory_capacity(), + 160_000, + "Auto must size from the 10k loaded slots (10_000 * 16 bytes)" + ); + + // A Fixed override ignores the loaded state. + let fixed = EvmCacheBuilder::new(mock_provider()) + .cache_config(cfg.clone()) + .shared_memory_capacity(SharedMemoryCapacity::Fixed(64 * 1024)) + .build() + .await; + assert_eq!(fixed.shared_memory_capacity(), 65_536); + + let _ = std::fs::remove_dir_all(&dir); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread")] +async fn flush_reports_unwritable_cache_paths() -> Result<()> { + let path_conflict = unique_cache_dir("flush_error"); + std::fs::write(&path_conflict, b"not a directory")?; + let cfg = CacheConfig::new(&path_conflict, 1, Default::default(), Default::default()); + let cache = EvmCacheBuilder::new(mock_provider()) + .cache_config(cfg) + .build() + .await; + + let err = cache + .flush() + .expect_err("flush must report persistence failures"); + let rendered = format!("{err:#}"); + assert!( + rendered.contains("directory") || rendered.contains("Not a directory"), + "unexpected error: {rendered}" + ); + + let _ = std::fs::remove_file(&path_conflict); + Ok(()) +} diff --git a/tests/snapshot_overlay.rs b/tests/snapshot_overlay.rs index 3b18abc..10ae6d4 100644 --- a/tests/snapshot_overlay.rs +++ b/tests/snapshot_overlay.rs @@ -101,7 +101,15 @@ async fn overlays_from_one_snapshot_are_isolated() -> Result<()> { let slot = U256::from(7); let original = U256::from(1u64); - cache.inject_storage_batch(&[(contract, slot, original)]); + // Overlay-resident seed so the value is EVM-visible on the StorageCleared + // MockERC20: after the §16.0 fix, a backend-only `inject_storage_batch` seed on + // a StorageCleared account reads as ZERO via `cached_storage_value` (mirroring + // the EVM SLOAD), so the live-cache assertion below would observe 0. Seeding + // the overlay (the winning layer) is what the test means by "the cache holds + // `original`" and is captured by `create_snapshot`. + cache + .db_mut() + .insert_account_storage(contract, slot, original)?; let snapshot = cache.create_snapshot(); let mut overlay_a = EvmOverlay::new(Arc::clone(&snapshot), None); @@ -169,3 +177,84 @@ async fn overlay_reads_reflect_snapshot_state() -> Result<()> { Ok(()) } + +/// Regression (§16 fix-review HIGH): `create_snapshot` must mirror the live +/// account-state-aware read. A `StorageCleared` account with a backend-only +/// (shadowed) slot reads ZERO live; the snapshot, `storage_value`, and a +/// snapshot-backed overlay must all agree — not the shadowed backend value. Pre- +/// fix the snapshot/overlay read the shadowed 100 while the live cache read 0. +#[tokio::test] +async fn snapshot_mirrors_live_read_for_cleared_account() -> Result<()> { + let token = Address::repeat_byte(0x5c); + let slot = U256::from(MOCK_ERC20_BALANCE_SLOT); // absent from the cleared overlay + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); // sets account_state = StorageCleared + cache.inject_storage_batch(&[(token, slot, U256::from(100))]); // backend-only shadow + + // Live read is ZERO (the §16.0 fix). + assert_eq!(cache.cached_storage_value(token, slot), Some(U256::ZERO)); + + let snapshot: Arc = cache.create_snapshot(); + assert_eq!( + snapshot.storage_value(token, slot), + Some(U256::ZERO), + "snapshot.storage_value must mirror the live cleared read, not the shadowed 100" + ); + + // A snapshot-backed overlay (no ext_db, as the freshness validator uses) must + // also read ZERO for the cleared account's absent slot. + let mut overlay = EvmOverlay::new(Arc::clone(&snapshot), None); + let value = overlay + .storage(token, slot) + .map_err(|e| anyhow!("overlay storage read failed: {e:?}"))?; + assert_eq!( + value, + U256::ZERO, + "snapshot-backed overlay must read ZERO for a cleared account's absent slot" + ); + Ok(()) +} + +/// Regression (round-2 HIGH, account axis): `create_snapshot` / `EvmOverlay::basic` +/// must mirror the live account read for a `NotExisting` account. revm treats such +/// an account as absent (`DbAccount::info()` → None), and `loaded_account_info` +/// already does; the snapshot/parallel path must agree — not surface a phantom +/// existing account with stale info. Pre-fix `EvmOverlay::basic` returned +/// `Some(info)`. +#[tokio::test] +async fn snapshot_basic_returns_none_for_notexisting_account() -> Result<()> { + use revm::database::AccountState; + use revm::database_interface::Database; + use revm::state::AccountInfo; + + let acct = Address::repeat_byte(0x6e); + let mut cache = setup_cache().await?; + // An overlay account revm marks NotExisting (e.g. after a selfdestruct) carries + // (default) info but is absent to the EVM. + cache.db_mut().insert_account_info( + acct, + AccountInfo { + balance: U256::from(1000), + ..Default::default() + }, + ); + cache + .db_mut() + .cache + .accounts + .get_mut(&acct) + .expect("overlay account present") + .account_state = AccountState::NotExisting; + + let snapshot: Arc = cache.create_snapshot(); + let mut overlay = EvmOverlay::new(Arc::clone(&snapshot), None); + let basic = overlay + .basic(acct) + .map_err(|e| anyhow!("overlay basic read failed: {e:?}"))?; + assert!( + basic.is_none(), + "snapshot-backed overlay must read a NotExisting account as absent (None), \ + not a phantom Some(info); got {basic:?}" + ); + Ok(()) +} diff --git a/tests/state_update.rs b/tests/state_update.rs new file mode 100644 index 0000000..815758e --- /dev/null +++ b/tests/state_update.rs @@ -0,0 +1,1839 @@ +//! Offline acceptance tests for the Phase 3 state-update primitives (Pillar B.1). +//! +//! These are the **contract** the implementation must satisfy: the +//! `StateUpdate` vocabulary, `EvmCache::apply_update` / `apply_updates`, the +//! `StateDiff` output, and the refold of the existing writers. Everything runs +//! fully offline (mocked provider, state injected directly), so no test reaches +//! the network. +//! +//! Layering vocabulary used throughout: +//! - **layer 1 / overlay** = the CacheDB overlay (`db_mut().cache.accounts`), +//! which wins on reads. +//! - **layer 2 / backend** = the BlockchainDb backend +//! (`unchecked_blockchain_db().storage()` / `.accounts()`). + +mod common; + +use alloy_primitives::{Address, Bytes, U256}; +use anyhow::Result; + +use common::{ + MOCK_ERC20_BALANCE_SLOT, balance_of, install_default_account, install_mock_erc20, setup_cache, +}; +use evm_fork_cache::cache::EvmCache; +use evm_fork_cache::{ + AccountPatch, PurgeScope, SkippedAccountPatch, SkippedBalanceDelta, SkippedDelta, SkippedMask, + SlotChange, SlotDelta, StateDiff, StateUpdate, +}; +use revm::state::{AccountInfo, Bytecode}; + +// --------------------------------------------------------------------------- +// Layer-inspection helpers (read each cache layer independently). +// --------------------------------------------------------------------------- + +/// Hashed storage slot of `balanceOf[owner]` for the MockERC20 fixture. +fn balance_slot_for(owner: Address) -> U256 { + use alloy_sol_types::SolValue; + let key = + alloy_primitives::keccak256((owner, U256::from(MOCK_ERC20_BALANCE_SLOT)).abi_encode()); + U256::from_be_bytes(key.0) +} + +/// Value of a slot in the CacheDB overlay (layer 1) only. +fn overlay_slot(cache: &mut EvmCache, addr: Address, slot: U256) -> Option { + cache + .db_mut() + .cache + .accounts + .get(&addr) + .and_then(|a| a.storage.get(&slot).copied()) +} + +/// Value of a slot in the BlockchainDb backend (layer 2) only. +fn backend_slot(cache: &EvmCache, addr: Address, slot: U256) -> Option { + cache + .unchecked_blockchain_db() + .storage() + .read() + .get(&addr) + .and_then(|s| s.get(&slot).copied()) +} + +/// Whether the overlay (layer 1) has an account entry for `addr`. +fn overlay_has_account(cache: &mut EvmCache, addr: Address) -> bool { + cache.db_mut().cache.accounts.contains_key(&addr) +} + +/// Overlay (layer 1) balance for `addr`, if an overlay account exists. +fn overlay_balance(cache: &mut EvmCache, addr: Address) -> Option { + cache + .db_mut() + .cache + .accounts + .get(&addr) + .map(|a| a.info.balance) +} + +/// Overlay (layer 1) nonce for `addr`, if an overlay account exists. +fn overlay_nonce(cache: &mut EvmCache, addr: Address) -> Option { + cache + .db_mut() + .cache + .accounts + .get(&addr) + .map(|a| a.info.nonce) +} + +/// Backend (layer 2) balance for `addr`, if a backend account exists. +fn backend_balance(cache: &EvmCache, addr: Address) -> Option { + cache + .unchecked_blockchain_db() + .accounts() + .read() + .get(&addr) + .map(|i| i.balance) +} + +// =========================================================================== +// Pure-data vocabulary (public API, no cache). +// =========================================================================== + +#[test] +fn account_patch_builders_compose() { + let empty = AccountPatch::default(); + assert_eq!(empty.balance, None); + assert_eq!(empty.nonce, None); + assert_eq!(empty.code, None); + + let patch = AccountPatch::default() + .balance(U256::from(42)) + .nonce(7) + .code(Bytes::from_static(&[0x60, 0x00])); + assert_eq!(patch.balance, Some(U256::from(42))); + assert_eq!(patch.nonce, Some(7)); + assert_eq!(patch.code, Some(Bytes::from_static(&[0x60, 0x00]))); +} + +#[test] +fn state_update_constructors_produce_expected_variants() { + let a = Address::repeat_byte(0xaa); + + assert_eq!( + StateUpdate::slot(a, U256::from(1), U256::from(2)), + StateUpdate::Slot { + address: a, + slot: U256::from(1), + value: U256::from(2), + } + ); + assert_eq!( + StateUpdate::balance(a, U256::from(9)), + StateUpdate::Account { + address: a, + patch: AccountPatch::default().balance(U256::from(9)), + } + ); + assert_eq!( + StateUpdate::account_upsert(a, AccountPatch::default().balance(U256::from(9))), + StateUpdate::AccountUpsert { + address: a, + patch: AccountPatch::default().balance(U256::from(9)), + } + ); + assert_eq!( + StateUpdate::purge(a, PurgeScope::Account), + StateUpdate::Purge { + address: a, + scope: PurgeScope::Account, + } + ); +} + +#[test] +fn state_diff_merge_and_is_empty() { + let a = Address::repeat_byte(0xbb); + let mut left = StateDiff::default(); + assert!(left.is_empty()); + assert_eq!(left.len(), 0); + + let mut right = StateDiff::default(); + right.slots.push(SlotChange { + address: a, + slot: U256::from(1), + old: U256::ZERO, + new: U256::from(5), + }); + + left.merge(right); + assert!(!left.is_empty()); + assert_eq!(left.len(), 1); + assert_eq!(left.slots.len(), 1); + assert_eq!(left.slots[0].new, U256::from(5)); +} + +// =========================================================================== +// Slot updates — write-through semantics (mirror inject_storage_batch_fresh). +// =========================================================================== + +#[tokio::test] +async fn apply_slot_writes_through_overlay_resident() -> Result<()> { + // An overlay-resident slot must be healed in BOTH layers, and the change + // must be observable on the synchronous EVM SLOAD path (here a balanceOf + // against a StorageCleared MockERC20 account). + let token = Address::repeat_byte(0x11); + let owner = Address::repeat_byte(0x22); + + let mut cache = setup_cache().await?; + install_default_account(&mut cache, Address::ZERO); // coinbase, for call_raw + install_default_account(&mut cache, owner); + install_mock_erc20(&mut cache, token); + + let slot = balance_slot_for(owner); + cache + .db_mut() + .insert_account_storage(token, slot, U256::from(100))?; + + let diff = cache.apply_update(&StateUpdate::slot(token, slot, U256::from(999))); + + // Both layers reflect the new value. + assert_eq!(overlay_slot(&mut cache, token, slot), Some(U256::from(999))); + assert_eq!(backend_slot(&cache, token, slot), Some(U256::from(999))); + assert_eq!( + cache.cached_storage_value(token, slot), + Some(U256::from(999)) + ); + + // The diff records exactly the one change. + assert_eq!( + diff.slots, + vec![SlotChange { + address: token, + slot, + old: U256::from(100), + new: U256::from(999), + }] + ); + assert!(diff.accounts.is_empty() && diff.purged.is_empty()); + + // The EVM SLOAD path sees the healed value. + assert_eq!(balance_of(&mut cache, token, owner)?, U256::from(999)); + Ok(()) +} + +#[tokio::test] +async fn apply_slot_no_overlay_account_is_not_materialized() -> Result<()> { + // Writing to an address with no overlay entry must populate the backend + // (layer 2) and NOT materialize a layer-1 overlay account — preserving the + // cold-prefetch / layer-2-only invariant. + let addr = Address::repeat_byte(0x33); + let slot = U256::from(7); + + let mut cache = setup_cache().await?; + assert!( + !overlay_has_account(&mut cache, addr), + "precondition: no overlay account" + ); + + let diff = cache.apply_update(&StateUpdate::slot(addr, slot, U256::from(5))); + + assert_eq!(backend_slot(&cache, addr, slot), Some(U256::from(5))); + assert!( + !overlay_has_account(&mut cache, addr), + "no overlay account may be materialized for a layer-2-only slot write" + ); + // The read falls through to the backend. + assert_eq!(cache.cached_storage_value(addr, slot), Some(U256::from(5))); + assert_eq!( + diff.slots, + vec![SlotChange { + address: addr, + slot, + old: U256::ZERO, + new: U256::from(5), + }] + ); + Ok(()) +} + +#[tokio::test] +async fn apply_slot_unchanged_value_yields_empty_diff() -> Result<()> { + let addr = Address::repeat_byte(0x44); + let slot = U256::from(1); + + let mut cache = setup_cache().await?; + // Seed the backend (layer 2) directly — inject_storage_batch does not load + // an account, unlike insert_account_storage, which would fetch a fresh + // address from the (mocked, empty) provider. + cache.inject_storage_batch(&[(addr, slot, U256::from(50))]); + + let diff = cache.apply_update(&StateUpdate::slot(addr, slot, U256::from(50))); + assert!( + diff.is_empty(), + "writing the cached value records no change" + ); + Ok(()) +} + +#[tokio::test] +async fn apply_slot_is_idempotent() -> Result<()> { + let addr = Address::repeat_byte(0x55); + let slot = U256::from(2); + + let mut cache = setup_cache().await?; + // Backend-direct seed (no account load) — see the note in the no-op test. + cache.inject_storage_batch(&[(addr, slot, U256::from(1))]); + + let first = cache.apply_update(&StateUpdate::slot(addr, slot, U256::from(8))); + assert_eq!(first.slots.len(), 1, "first apply records the change"); + + let second = cache.apply_update(&StateUpdate::slot(addr, slot, U256::from(8))); + assert!(second.is_empty(), "re-applying the same value is a no-op"); + Ok(()) +} + +// =========================================================================== +// Account updates — partial patch, write-through. +// =========================================================================== + +#[tokio::test] +async fn apply_account_balance_patch_preserves_other_fields() -> Result<()> { + let token = Address::repeat_byte(0x66); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); // balance 0, nonce 0, code present + + let diff = cache.apply_update(&StateUpdate::Account { + address: token, + patch: AccountPatch::default().balance(U256::from(500)), + }); + + // Balance changed in the overlay (the winning layer); nonce/code preserved. + assert_eq!(overlay_balance(&mut cache, token), Some(U256::from(500))); + assert_eq!(overlay_nonce(&mut cache, token), Some(0)); + + assert_eq!(diff.accounts.len(), 1); + let change = &diff.accounts[0]; + assert_eq!(change.address, token); + assert_eq!(change.balance, Some((U256::ZERO, U256::from(500)))); + assert_eq!(change.nonce, None, "nonce unchanged → no delta"); + assert_eq!(change.code_hash, None, "code unchanged → no delta"); + assert!(diff.slots.is_empty() && diff.purged.is_empty()); + Ok(()) +} + +#[tokio::test] +async fn apply_account_code_patch_recomputes_hash() -> Result<()> { + let addr = Address::repeat_byte(0x77); + let mut cache = setup_cache().await?; + install_default_account(&mut cache, addr); // empty code + + let new_code = Bytes::from_static(&[0x60, 0x00, 0x60, 0x00, 0xf3]); + let expected_hash = Bytecode::new_raw(new_code.clone()).hash_slow(); + + let diff = cache.apply_update(&StateUpdate::Account { + address: addr, + patch: AccountPatch::default().code(new_code.clone()), + }); + + assert_eq!(diff.accounts.len(), 1); + let change = &diff.accounts[0]; + let (old_hash, new_hash) = change.code_hash.expect("code hash changed"); + assert_ne!(old_hash, new_hash); + assert_eq!( + new_hash, expected_hash, + "code hash recomputed from the patched code" + ); + assert_eq!(change.balance, None); + assert_eq!(change.nonce, None); + Ok(()) +} + +#[tokio::test] +async fn apply_account_patch_on_cold_account_is_skipped_and_surfaced() -> Result<()> { + // A partial Account patch against a cold account must not materialize a + // default backend account, because that would mask the real on-chain account. + let addr = Address::repeat_byte(0x88); + let mut cache = setup_cache().await?; + assert!(!overlay_has_account(&mut cache, addr)); + assert_eq!(backend_balance(&cache, addr), None); + + let patch = AccountPatch::default().balance(U256::from(1234)); + let diff = cache.apply_update(&StateUpdate::account(addr, patch.clone())); + + assert_eq!( + backend_balance(&cache, addr), + None, + "cold patch must not materialize a backend account" + ); + assert!(diff.accounts.is_empty()); + assert_eq!( + diff.skipped_accounts, + vec![SkippedAccountPatch { + address: addr, + patch + }] + ); + assert!(diff.has_skipped()); + assert_eq!(diff.skipped_len(), 1); + Ok(()) +} + +#[tokio::test] +async fn account_upsert_intentionally_materializes_absent_account() -> Result<()> { + let addr = Address::repeat_byte(0x88); + let mut cache = setup_cache().await?; + assert!(!overlay_has_account(&mut cache, addr)); + assert_eq!(backend_balance(&cache, addr), None); + + let diff = cache.apply_update(&StateUpdate::account_upsert( + addr, + AccountPatch::default().balance(U256::from(1234)), + )); + + assert_eq!(backend_balance(&cache, addr), Some(U256::from(1234))); + assert_eq!(diff.accounts.len(), 1); + assert_eq!( + diff.accounts[0].balance, + Some((U256::ZERO, U256::from(1234))) + ); + assert!(diff.skipped_accounts.is_empty()); + Ok(()) +} + +// =========================================================================== +// Purge updates — dispatch to the existing layer logic, record what was removed. +// =========================================================================== + +#[tokio::test] +async fn apply_purge_account_clears_both_layers() -> Result<()> { + let token = Address::repeat_byte(0x11); + let owner = Address::repeat_byte(0x22); + + let mut cache = setup_cache().await?; + install_default_account(&mut cache, Address::ZERO); + install_default_account(&mut cache, owner); + install_mock_erc20(&mut cache, token); + + // Populate both layers. + common::transfer(&mut cache, token, owner, owner, U256::from(0)).ok(); + cache + .db_mut() + .insert_account_storage(token, U256::from(1), U256::from(9))?; + cache.inject_storage_batch(&[(token, U256::from(2), U256::from(8))]); + assert!(overlay_has_account(&mut cache, token)); + + let diff = cache.apply_update(&StateUpdate::purge(token, PurgeScope::Account)); + + assert!( + !overlay_has_account(&mut cache, token), + "overlay account removed" + ); + assert_eq!( + cache.pool_storage_slot_count(token), + 0, + "backend storage gone" + ); + { + let accounts = cache.unchecked_blockchain_db().accounts().read(); + assert!(!accounts.contains_key(&token), "backend account removed"); + } + assert_eq!(diff.purged.len(), 1); + assert_eq!(diff.purged[0].address, token); + assert_eq!(diff.purged[0].scope, PurgeScope::Account); + assert!( + diff.purged[0].account_removed, + "an account info was removed" + ); + Ok(()) +} + +#[tokio::test] +async fn apply_purge_all_storage_keeps_account() -> Result<()> { + let token = Address::repeat_byte(0x33); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + cache + .db_mut() + .insert_account_storage(token, U256::from(1), U256::from(9))?; + cache.inject_storage_batch(&[(token, U256::from(2), U256::from(8))]); + + let diff = cache.apply_update(&StateUpdate::purge(token, PurgeScope::AllStorage)); + + assert_eq!( + cache.pool_storage_slot_count(token), + 0, + "backend storage gone" + ); + assert!( + overlay_has_account(&mut cache, token), + "account info preserved" + ); + assert_eq!(diff.purged.len(), 1); + assert_eq!(diff.purged[0].scope, PurgeScope::AllStorage); + assert!(!diff.purged[0].account_removed); + Ok(()) +} + +#[tokio::test] +async fn apply_purge_specific_slots() -> Result<()> { + let token = Address::repeat_byte(0x44); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + cache.inject_storage_batch(&[ + (token, U256::from(1), U256::from(10)), + (token, U256::from(2), U256::from(20)), + (token, U256::from(3), U256::from(30)), + ]); + + let diff = cache.apply_update(&StateUpdate::purge( + token, + PurgeScope::Slots(vec![U256::from(1), U256::from(3)]), + )); + + assert_eq!(backend_slot(&cache, token, U256::from(1)), None); + assert_eq!( + backend_slot(&cache, token, U256::from(2)), + Some(U256::from(20)) + ); + assert_eq!(backend_slot(&cache, token, U256::from(3)), None); + assert_eq!(diff.purged.len(), 1); + assert_eq!(diff.purged[0].slots_removed, 2); + Ok(()) +} + +// =========================================================================== +// apply_updates — fold + merge. +// =========================================================================== + +#[tokio::test] +async fn apply_updates_merges_mixed_batch() -> Result<()> { + let acct = Address::repeat_byte(0x66); + let pool = Address::repeat_byte(0x77); + + let mut cache = setup_cache().await?; + install_default_account(&mut cache, acct); + cache.inject_storage_batch(&[(pool, U256::from(9), U256::from(1))]); + + let diff = cache.apply_updates(&[ + StateUpdate::slot(pool, U256::from(1), U256::from(100)), + StateUpdate::balance(acct, U256::from(500)), + StateUpdate::purge(pool, PurgeScope::Slots(vec![U256::from(9)])), + ]); + + assert!(!diff.slots.is_empty(), "slot write recorded"); + assert!(!diff.accounts.is_empty(), "account patch recorded"); + assert!(!diff.purged.is_empty(), "purge recorded"); + Ok(()) +} + +#[tokio::test] +async fn apply_updates_same_slot_later_overrides() -> Result<()> { + let addr = Address::repeat_byte(0x88); + let slot = U256::from(1); + let mut cache = setup_cache().await?; + + let diff = cache.apply_updates(&[ + StateUpdate::slot(addr, slot, U256::from(10)), + StateUpdate::slot(addr, slot, U256::from(20)), + ]); + + // Each apply contributes its own SlotChange (merge concatenates), so the + // observed history is ZERO->10 then 10->20. + assert_eq!( + diff.slots, + vec![ + SlotChange { + address: addr, + slot, + old: U256::ZERO, + new: U256::from(10) + }, + SlotChange { + address: addr, + slot, + old: U256::from(10), + new: U256::from(20) + }, + ] + ); + assert_eq!(cache.cached_storage_value(addr, slot), Some(U256::from(20))); + Ok(()) +} + +// =========================================================================== +// Refold equivalence — wrappers behave exactly as before. +// =========================================================================== + +#[tokio::test] +async fn refold_purge_pool_storage_returns_same_count() -> Result<()> { + let token = Address::repeat_byte(0x99); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + cache.inject_storage_batch(&[ + (token, U256::from(1), U256::from(10)), + (token, U256::from(2), U256::from(20)), + ]); + + // The wrapper still returns the backend slot count it removed. + let removed = cache.purge_pool_storage(token); + assert_eq!(removed, 2); + assert_eq!(cache.pool_storage_slot_count(token), 0); + Ok(()) +} + +#[tokio::test] +async fn refold_inject_storage_batch_fresh_matches_apply_updates() -> Result<()> { + let token = Address::repeat_byte(0xa1); + let slot = U256::from(4); + + // Path A: the existing wrapper. + let mut a = setup_cache().await?; + install_mock_erc20(&mut a, token); + a.db_mut() + .insert_account_storage(token, slot, U256::from(1))?; + a.inject_storage_batch_fresh(&[(token, slot, U256::from(77))]); + + // Path B: the primitive it now wraps. + let mut b = setup_cache().await?; + install_mock_erc20(&mut b, token); + b.db_mut() + .insert_account_storage(token, slot, U256::from(1))?; + let _ = b.apply_updates(&[StateUpdate::slot(token, slot, U256::from(77))]); + + assert_eq!( + a.cached_storage_value(token, slot), + b.cached_storage_value(token, slot), + "wrapper and primitive leave the cache in the same state" + ); + assert_eq!( + overlay_slot(&mut a, token, slot), + overlay_slot(&mut b, token, slot) + ); + assert_eq!(backend_slot(&a, token, slot), backend_slot(&b, token, slot)); + Ok(()) +} + +// =========================================================================== +// Decision 2 (LOCKED: normalize) — protocols inject_v3_* now writes through to +// the backend (layer 2). Pre-fix this wrote layer 1 only. +// =========================================================================== + +#[cfg(feature = "protocols")] +#[tokio::test] +async fn inject_v3_tick_bitmap_writes_through_to_backend() -> Result<()> { + use std::collections::HashMap; + + let pool = Address::repeat_byte(0xb2); + let mut cache = setup_cache().await?; + + let mut bitmap = HashMap::new(); + bitmap.insert(0i16, U256::from(123)); + bitmap.insert(1i16, U256::from(456)); + + let injected = cache.inject_v3_tick_bitmap(pool, &bitmap)?; + assert_eq!(injected, 2); + + // Normalized to write-through: the backend (layer 2) now holds the slots. + // Before the refold this count was 0 (overlay-only write). + assert!( + cache.pool_storage_slot_count(pool) > 0, + "inject_v3_tick_bitmap must write through to the backend (Decision 2)" + ); + Ok(()) +} + +// =========================================================================== +// §15 addendum — relative / read-modify-write updates. +// +// `SlotDelta` reads the current value, applies a saturating mutation, and writes +// back (write-through). It is cold-aware: a delta on a slot the cache never +// fetched is NOT applied (it would corrupt an unknown balance) — it is skipped +// and surfaced in `StateDiff.skipped`. `modify_slot` is the general closure form. +// =========================================================================== + +#[tokio::test] +async fn slot_delta_add_applies_to_hot_slot() -> Result<()> { + let addr = Address::repeat_byte(0xc1); + let slot = U256::from(1); + let mut cache = setup_cache().await?; + cache.inject_storage_batch(&[(addr, slot, U256::from(100))]); + + let diff = cache.apply_update(&StateUpdate::slot_delta( + addr, + slot, + SlotDelta::Add(U256::from(50)), + )); + + assert_eq!( + cache.cached_storage_value(addr, slot), + Some(U256::from(150)) + ); + assert_eq!( + diff.slots, + vec![SlotChange { + address: addr, + slot, + old: U256::from(100), + new: U256::from(150), + }] + ); + assert!( + diff.skipped.is_empty(), + "a hot slot is applied, not skipped" + ); + Ok(()) +} + +#[tokio::test] +async fn slot_delta_sub_saturates_at_zero() -> Result<()> { + let addr = Address::repeat_byte(0xc2); + let slot = U256::from(1); + let mut cache = setup_cache().await?; + cache.inject_storage_batch(&[(addr, slot, U256::from(30))]); + + let diff = cache.apply_update(&StateUpdate::slot_delta( + addr, + slot, + SlotDelta::Sub(U256::from(50)), + )); + + assert_eq!( + cache.cached_storage_value(addr, slot), + Some(U256::ZERO), + "Sub saturates at zero rather than underflowing" + ); + assert_eq!(diff.slots.len(), 1); + assert_eq!(diff.slots[0].new, U256::ZERO); + Ok(()) +} + +#[tokio::test] +async fn slot_delta_add_saturates_at_max() -> Result<()> { + let addr = Address::repeat_byte(0xc3); + let slot = U256::from(1); + let mut cache = setup_cache().await?; + cache.inject_storage_batch(&[(addr, slot, U256::MAX - U256::from(1))]); + + cache.apply_update(&StateUpdate::slot_delta( + addr, + slot, + SlotDelta::Add(U256::from(10)), + )); + + assert_eq!( + cache.cached_storage_value(addr, slot), + Some(U256::MAX), + "Add saturates at U256::MAX" + ); + Ok(()) +} + +#[tokio::test] +async fn slot_delta_cold_slot_is_skipped_and_surfaced() -> Result<()> { + // The correctness guarantee: a delta against an unknown (cold) value must not + // be applied (it would corrupt the balance) — it is surfaced instead. + let addr = Address::repeat_byte(0xc4); + let slot = U256::from(7); + let mut cache = setup_cache().await?; + assert_eq!( + cache.cached_storage_value(addr, slot), + None, + "precondition: slot is cold" + ); + + let diff = cache.apply_update(&StateUpdate::slot_delta( + addr, + slot, + SlotDelta::Add(U256::from(50)), + )); + + assert!(diff.slots.is_empty(), "nothing applied"); + assert_eq!( + diff.skipped, + vec![SkippedDelta { + address: addr, + slot, + delta: SlotDelta::Add(U256::from(50)), + }] + ); + assert_eq!( + cache.cached_storage_value(addr, slot), + None, + "the cold slot is left untouched so the next read fetches the truth" + ); + Ok(()) +} + +#[tokio::test] +async fn slot_delta_writes_through_both_layers() -> Result<()> { + let token = Address::repeat_byte(0xc5); + let slot = U256::from(2); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + // Overlay-resident seed (account already installed, so no fetch). + cache + .db_mut() + .insert_account_storage(token, slot, U256::from(100))?; + + cache.apply_update(&StateUpdate::slot_delta( + token, + slot, + SlotDelta::Add(U256::from(5)), + )); + + assert_eq!(overlay_slot(&mut cache, token, slot), Some(U256::from(105))); + assert_eq!(backend_slot(&cache, token, slot), Some(U256::from(105))); + Ok(()) +} + +#[tokio::test] +async fn modify_slot_applies_transform() -> Result<()> { + let addr = Address::repeat_byte(0xc6); + let slot = U256::from(1); + let mut cache = setup_cache().await?; + cache.inject_storage_batch(&[(addr, slot, U256::from(10))]); + + let change = cache.modify_slot(addr, slot, |cur| cur.map(|v| v * U256::from(2))); + + assert_eq!( + change, + Some(SlotChange { + address: addr, + slot, + old: U256::from(10), + new: U256::from(20), + }) + ); + assert_eq!(cache.cached_storage_value(addr, slot), Some(U256::from(20))); + Ok(()) +} + +#[tokio::test] +async fn modify_slot_closure_skips_cold() -> Result<()> { + let addr = Address::repeat_byte(0xc7); + let slot = U256::from(1); + let mut cache = setup_cache().await?; + + // The closure returns None for a cold slot, so nothing is written. + let change = cache.modify_slot(addr, slot, |cur| cur.map(|v| v + U256::from(1))); + + assert_eq!(change, None); + assert_eq!(cache.cached_storage_value(addr, slot), None); + Ok(()) +} + +#[tokio::test] +async fn modify_slot_can_write_absolute_on_cold() -> Result<()> { + // The caller may choose to write an absolute value even on a cold slot (it + // had external knowledge). The closure ignores the `None` and returns a value. + let addr = Address::repeat_byte(0xc8); + let slot = U256::from(1); + let mut cache = setup_cache().await?; + + let change = cache.modify_slot(addr, slot, |_| Some(U256::from(7))); + + assert_eq!( + change, + Some(SlotChange { + address: addr, + slot, + old: U256::ZERO, + new: U256::from(7), + }) + ); + assert_eq!(cache.cached_storage_value(addr, slot), Some(U256::from(7))); + Ok(()) +} + +#[test] +fn state_diff_merge_includes_skipped() { + let a = Address::repeat_byte(0xd9); + let mut left = StateDiff::default(); + let mut right = StateDiff::default(); + right.skipped.push(SkippedDelta { + address: a, + slot: U256::from(1), + delta: SlotDelta::Add(U256::from(5)), + }); + + left.merge(right); + assert_eq!(left.skipped.len(), 1); + assert_eq!(left.skipped[0].delta, SlotDelta::Add(U256::from(5))); + // A skip is metadata, not a change: it does not affect is_empty/len. + assert!(left.is_empty(), "a skipped delta is not a recorded change"); + assert_eq!(left.len(), 0); +} + +#[tokio::test] +async fn balance_tracking_scenario() -> Result<()> { + // The motivating use case: index an ERC-20 `Transfer(alice -> bob, amount)` + // as two relative slot updates to keep the tracked balances hot, without ever + // knowing the resulting absolute balances up front. + let token = Address::repeat_byte(0xe0); + let alice = Address::repeat_byte(0x0a); + let bob = Address::repeat_byte(0x0b); + let mut cache = setup_cache().await?; + install_default_account(&mut cache, Address::ZERO); // coinbase, for the SLOAD calls + install_default_account(&mut cache, alice); + install_default_account(&mut cache, bob); + install_mock_erc20(&mut cache, token); + + let alice_slot = balance_slot_for(alice); + let bob_slot = balance_slot_for(bob); + + // Seed the tracked balances once (the "make it hot" step) in an EVM-VISIBLE + // way: overlay-resident, so the StorageCleared token account actually reads + // them on the SLOAD path. (A backend-only inject is invisible here — see + // `cached_storage_value_matches_evm_sload_for_cleared_account`.) + cache + .db_mut() + .insert_account_storage(token, alice_slot, U256::from(1000))?; + cache + .db_mut() + .insert_account_storage(token, bob_slot, U256::ZERO)?; + + // Sanity: the EVM actually sees the seeded balances. + assert_eq!(balance_of(&mut cache, token, alice)?, U256::from(1000)); + + // Transfer(alice -> bob, 300) decodes to two relative updates. + let amount = U256::from(300); + let diff = cache.apply_updates(&[ + StateUpdate::slot_delta(token, alice_slot, SlotDelta::Sub(amount)), + StateUpdate::slot_delta(token, bob_slot, SlotDelta::Add(amount)), + ]); + + // Validate via a real SLOAD (`balanceOf`), not just the cached accessor. + assert_eq!(balance_of(&mut cache, token, alice)?, U256::from(700)); + assert_eq!(balance_of(&mut cache, token, bob)?, U256::from(300)); + assert_eq!( + cache.cached_storage_value(token, alice_slot), + Some(U256::from(700)) + ); + assert_eq!( + cache.cached_storage_value(token, bob_slot), + Some(U256::from(300)) + ); + assert!(diff.skipped.is_empty(), "both slots were seeded (hot)"); + assert_eq!(diff.slots.len(), 2); + + // Conservation: total supply across the two holders is unchanged. + let total = cache.cached_storage_value(token, alice_slot).unwrap() + + cache.cached_storage_value(token, bob_slot).unwrap(); + assert_eq!(total, U256::from(1000)); + Ok(()) +} + +// =========================================================================== +// §16.0 — the audit HIGH correctness bug: cached_storage_value must match the +// EVM SLOAD for a StorageCleared overlay account (else SlotDelta corrupts an +// EVM-invisible base). This test uses only existing symbols so it runs against +// the CURRENT (buggy) code: it is RED before the §16.0 fix, GREEN after. +// =========================================================================== + +#[tokio::test] +async fn cached_storage_value_matches_evm_sload_for_cleared_account() -> Result<()> { + let token = Address::repeat_byte(0x5c); + let owner = Address::repeat_byte(0x5d); + let mut cache = setup_cache().await?; + install_default_account(&mut cache, Address::ZERO); // coinbase, for call_raw + install_default_account(&mut cache, owner); + install_mock_erc20(&mut cache, token); // account_state = StorageCleared + + let slot = balance_slot_for(owner); + // Backend-only seed: invisible to a StorageCleared overlay account's SLOAD. + cache.inject_storage_batch(&[(token, slot, U256::from(100))]); + + // The real EVM SLOAD reads ZERO (StorageCleared, slot absent from overlay, + // backend NOT consulted). + let evm_seen = balance_of(&mut cache, token, owner)?; + assert_eq!( + evm_seen, + U256::ZERO, + "precondition: the EVM cannot see a backend-only seed on a StorageCleared account" + ); + + // cached_storage_value MUST agree with the EVM, not report the shadowed + // backend value (100). Pre-fix it returns Some(100) -> this assert fails. + assert_eq!( + cache.cached_storage_value(token, slot), + Some(U256::ZERO), + "cached_storage_value must mirror the EVM SLOAD (ZERO), not the shadowed backend value" + ); + Ok(()) +} + +// =========================================================================== +// §16.0 — present-as-ZERO is HOT (delta applies to 0), distinct from cold (skip). +// =========================================================================== + +#[tokio::test] +async fn slot_delta_on_present_zero_is_hot_not_skipped() -> Result<()> { + let token = Address::repeat_byte(0x6a); + let slot = U256::from(1); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + // Overlay-resident ZERO: a *known* zero, not an absent (cold) slot. + cache + .db_mut() + .insert_account_storage(token, slot, U256::ZERO)?; + + let diff = cache.apply_update(&StateUpdate::slot_delta( + token, + slot, + SlotDelta::Add(U256::from(50)), + )); + + assert_eq!( + cache.cached_storage_value(token, slot), + Some(U256::from(50)) + ); + assert_eq!( + diff.slots.len(), + 1, + "present-as-zero is hot: the delta applies" + ); + assert!( + diff.skipped.is_empty(), + "present-as-zero must NOT be treated as cold" + ); + Ok(()) +} + +// =========================================================================== +// §16.5 — account-native-balance delta (BalanceDelta + modify_account_balance). +// =========================================================================== + +#[tokio::test] +async fn balance_delta_applies_to_present_account() -> Result<()> { + let acct = Address::repeat_byte(0x71); + let mut cache = setup_cache().await?; + cache.db_mut().insert_account_info( + acct, + AccountInfo { + balance: U256::from(1000), + nonce: 5, + ..Default::default() + }, + ); + + let diff = cache.apply_update(&StateUpdate::balance_delta( + acct, + SlotDelta::Sub(U256::from(300)), + )); + + assert_eq!(overlay_balance(&mut cache, acct), Some(U256::from(700))); + assert_eq!(overlay_nonce(&mut cache, acct), Some(5), "nonce preserved"); + assert_eq!( + backend_balance(&cache, acct), + Some(U256::from(700)), + "write-through to backend" + ); + assert_eq!(diff.accounts.len(), 1); + assert_eq!( + diff.accounts[0].balance, + Some((U256::from(1000), U256::from(700))) + ); + assert!(diff.accounts[0].nonce.is_none(), "nonce unchanged"); + assert!(diff.skipped_balances.is_empty()); + Ok(()) +} + +#[tokio::test] +async fn balance_delta_on_cold_account_is_skipped_and_surfaced() -> Result<()> { + let acct = Address::repeat_byte(0x72); + let mut cache = setup_cache().await?; + assert!(!overlay_has_account(&mut cache, acct)); + + let diff = cache.apply_update(&StateUpdate::balance_delta( + acct, + SlotDelta::Add(U256::from(500)), + )); + + assert!( + diff.accounts.is_empty(), + "nothing applied for an unknown balance" + ); + assert_eq!( + diff.skipped_balances, + vec![SkippedBalanceDelta { + address: acct, + delta: SlotDelta::Add(U256::from(500)), + }] + ); + // Crucially: no account is materialized (avoids masking the real on-chain one). + assert!(!overlay_has_account(&mut cache, acct)); + assert_eq!(backend_balance(&cache, acct), None); + Ok(()) +} + +#[tokio::test] +async fn balance_delta_saturates_at_zero() -> Result<()> { + let acct = Address::repeat_byte(0x73); + let mut cache = setup_cache().await?; + cache.db_mut().insert_account_info( + acct, + AccountInfo { + balance: U256::from(100), + ..Default::default() + }, + ); + + cache.apply_update(&StateUpdate::balance_delta( + acct, + SlotDelta::Sub(U256::from(500)), + )); + + assert_eq!( + overlay_balance(&mut cache, acct), + Some(U256::ZERO), + "Sub saturates at zero" + ); + Ok(()) +} + +#[tokio::test] +async fn modify_account_balance_hot_and_cold() -> Result<()> { + let acct = Address::repeat_byte(0x74); + let mut cache = setup_cache().await?; + cache.db_mut().insert_account_info( + acct, + AccountInfo { + balance: U256::from(10), + ..Default::default() + }, + ); + + let change = cache.modify_account_balance(acct, |cur| cur.map(|v| v * U256::from(3))); + assert_eq!( + change.and_then(|c| c.balance), + Some((U256::from(10), U256::from(30))) + ); + assert_eq!(overlay_balance(&mut cache, acct), Some(U256::from(30))); + + // A cold account: the closure receives None and skips; nothing materialized. + let cold = Address::repeat_byte(0x75); + let none = cache.modify_account_balance(cold, |cur| cur.map(|v| v + U256::from(1))); + assert!(none.is_none()); + assert!(!overlay_has_account(&mut cache, cold)); + assert_eq!(backend_balance(&cache, cold), None); + Ok(()) +} + +// =========================================================================== +// §16.6 — discoverable skip accessors over both skip kinds. +// =========================================================================== + +#[tokio::test] +async fn skip_accessors_reflect_both_skip_kinds() -> Result<()> { + let token = Address::repeat_byte(0x76); + let acct = Address::repeat_byte(0x77); + let mut cache = setup_cache().await?; + + // A cold slot delta and a cold balance delta: both skipped, no change recorded. + let diff = cache.apply_updates(&[ + StateUpdate::slot_delta(token, U256::from(9), SlotDelta::Add(U256::from(1))), + StateUpdate::balance_delta(acct, SlotDelta::Add(U256::from(1))), + ]); + assert!(diff.is_empty(), "changes-only: nothing applied"); + assert!(diff.has_skipped()); + assert_eq!(diff.skipped_len(), 2); + assert!(!diff.is_fully_applied()); + + // A fully-applied update reports no skips. + let hot = Address::repeat_byte(0x78); + cache.db_mut().insert_account_info( + hot, + AccountInfo { + balance: U256::from(5), + ..Default::default() + }, + ); + let diff2 = cache.apply_update(&StateUpdate::balance_delta( + hot, + SlotDelta::Add(U256::from(5)), + )); + assert!(diff2.is_fully_applied()); + assert!(!diff2.has_skipped()); + Ok(()) +} + +// =========================================================================== +// §16.3 — serde round-trip of the vocabulary and the diff. +// =========================================================================== + +#[test] +fn vocabulary_serde_round_trips() { + let a = Address::repeat_byte(0x81); + let updates = vec![ + StateUpdate::slot(a, U256::from(1), U256::from(2)), + StateUpdate::slot_delta(a, U256::from(1), SlotDelta::Sub(U256::from(3))), + StateUpdate::balance_delta(a, SlotDelta::Add(U256::from(4))), + StateUpdate::account(a, AccountPatch::default().balance(U256::from(9)).nonce(2)), + StateUpdate::purge(a, PurgeScope::Slots(vec![U256::from(1)])), + ]; + let json = serde_json::to_string(&updates).expect("serialize updates"); + let back: Vec = serde_json::from_str(&json).expect("deserialize updates"); + assert_eq!(updates, back); + + let mut diff = StateDiff::default(); + diff.slots.push(SlotChange { + address: a, + slot: U256::from(1), + old: U256::ZERO, + new: U256::from(2), + }); + diff.skipped.push(SkippedDelta { + address: a, + slot: U256::from(2), + delta: SlotDelta::Add(U256::from(1)), + }); + diff.skipped_balances.push(SkippedBalanceDelta { + address: a, + delta: SlotDelta::Sub(U256::from(1)), + }); + let djson = serde_json::to_string(&diff).expect("serialize diff"); + let dback: StateDiff = serde_json::from_str(&djson).expect("deserialize diff"); + assert_eq!(diff, dback); +} + +// =========================================================================== +// §16.1 — a no-op Account patch must not materialize a backend account. +// =========================================================================== + +#[tokio::test] +async fn account_patch_noop_does_not_materialize_backend() -> Result<()> { + let acct = Address::repeat_byte(0x95); + let mut cache = setup_cache().await?; + assert!(!overlay_has_account(&mut cache, acct)); + + // All-None patch on an absent account: no change, and crucially no write. + let diff = cache.apply_update(&StateUpdate::account(acct, AccountPatch::default())); + assert!(diff.is_empty()); + assert!(diff.accounts.is_empty()); + assert_eq!( + backend_balance(&cache, acct), + None, + "a no-op patch must not materialize a backend account" + ); + assert!(!overlay_has_account(&mut cache, acct)); + + // balance -> current value on a present account is also a no-op. + let acct2 = Address::repeat_byte(0x96); + cache.db_mut().insert_account_info( + acct2, + AccountInfo { + balance: U256::from(50), + ..Default::default() + }, + ); + let diff2 = cache.apply_update(&StateUpdate::balance(acct2, U256::from(50))); + assert!(diff2.accounts.is_empty(), "balance -> current is a no-op"); + Ok(()) +} + +// =========================================================================== +// §16.8/16.9 — batched apply_updates must equal sequential apply_update (the +// safety net for the single-lock fast-path). Mixed batch: distinct addresses, +// a same-slot repeat, a hot delta, an account patch, and a purge mid-batch. +// =========================================================================== + +#[tokio::test] +async fn apply_updates_batched_equals_sequential() -> Result<()> { + let p = Address::repeat_byte(0xa1); + let q = Address::repeat_byte(0xa2); + let slot1 = U256::from(1); + let slot2 = U256::from(2); + let slot3 = U256::from(3); + + // Build two identically-seeded caches. + async fn seeded(p: Address, q: Address, slot2: U256) -> Result { + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, p); // StorageCleared overlay account + install_default_account(&mut cache, q); + cache.inject_storage_batch(&[(p, slot2, U256::from(99))]); // backend slot for the purge + Ok(cache) + } + let batch = vec![ + StateUpdate::slot(p, slot1, U256::from(500)), + StateUpdate::slot(p, slot1, U256::from(600)), // same slot again (order matters) + StateUpdate::slot_delta(p, slot1, SlotDelta::Add(U256::from(10))), // hot -> 610 + StateUpdate::balance(q, U256::from(1000)), + StateUpdate::purge(p, PurgeScope::Slots(vec![slot2])), // purge mid-batch + StateUpdate::slot(p, slot3, U256::from(7)), // write after the purge + ]; + + let mut batched = seeded(p, q, slot2).await?; + let diff_batched = batched.apply_updates(&batch); + + let mut sequential = seeded(p, q, slot2).await?; + let mut diff_seq = StateDiff::default(); + for u in &batch { + diff_seq.merge(sequential.apply_update(u)); + } + + assert_eq!( + diff_batched, diff_seq, + "batched diff must equal the sequential fold" + ); + for (addr, slot) in [(p, slot1), (p, slot2), (p, slot3)] { + assert_eq!( + batched.cached_storage_value(addr, slot), + sequential.cached_storage_value(addr, slot), + "slot {slot} state diverged between batched and sequential" + ); + } + assert_eq!( + overlay_balance(&mut batched, q), + overlay_balance(&mut sequential, q) + ); + assert_eq!( + backend_balance(&batched, q), + backend_balance(&sequential, q) + ); + // Concrete expected end-state. + assert_eq!( + batched.cached_storage_value(p, slot1), + Some(U256::from(610)) + ); + // Verify the purge via the backend layer directly: `p` is a StorageCleared + // MockERC20, so cached_storage_value reads an absent slot as 0 (mirroring the + // SLOAD) regardless of the purge — the backend map is the meaningful check. + assert_eq!( + backend_slot(&batched, p, slot2), + None, + "slot2 purged from the backend" + ); + assert_eq!(batched.cached_storage_value(p, slot3), Some(U256::from(7))); + Ok(()) +} + +// =========================================================================== +// §16.8 — Account-patch coverage gaps. +// =========================================================================== + +#[tokio::test] +async fn account_patch_writes_through_to_backend_on_overlay_present() -> Result<()> { + let acct = Address::repeat_byte(0x90); + let mut cache = setup_cache().await?; + cache.db_mut().insert_account_info( + acct, + AccountInfo { + balance: U256::from(100), + ..Default::default() + }, + ); + + let diff = cache.apply_update(&StateUpdate::balance(acct, U256::from(500))); + + assert_eq!(overlay_balance(&mut cache, acct), Some(U256::from(500))); + assert_eq!( + backend_balance(&cache, acct), + Some(U256::from(500)), + "backend is always written, even when an overlay account exists" + ); + assert_eq!( + diff.accounts[0].balance, + Some((U256::from(100), U256::from(500))) + ); + Ok(()) +} + +#[tokio::test] +async fn account_patch_on_backend_only_account_does_not_materialize_overlay() -> Result<()> { + let acct = Address::repeat_byte(0x91); + let mut cache = setup_cache().await?; + // Seed only the backend (the cold-prefetched, layer-2-only case). + cache.unchecked_blockchain_db().accounts().write().insert( + acct, + AccountInfo { + balance: U256::from(100), + nonce: 3, + ..Default::default() + }, + ); + + let diff = cache.apply_update(&StateUpdate::balance(acct, U256::from(500))); + + assert_eq!( + diff.accounts[0].balance, + Some((U256::from(100), U256::from(500))), + "old value loaded from the backend" + ); + assert_eq!(backend_balance(&cache, acct), Some(U256::from(500))); + assert!( + !overlay_has_account(&mut cache, acct), + "no overlay account materialized for a backend-only patch" + ); + Ok(()) +} + +#[tokio::test] +async fn account_patch_nonce_only() -> Result<()> { + let acct = Address::repeat_byte(0x92); + let mut cache = setup_cache().await?; + cache.db_mut().insert_account_info( + acct, + AccountInfo { + nonce: 1, + ..Default::default() + }, + ); + + let diff = cache.apply_update(&StateUpdate::nonce(acct, 9)); + + assert_eq!(diff.accounts[0].nonce, Some((1, 9))); + assert!(diff.accounts[0].balance.is_none()); + assert!(diff.accounts[0].code_hash.is_none()); + assert_eq!(overlay_nonce(&mut cache, acct), Some(9)); + Ok(()) +} + +#[tokio::test] +async fn account_patch_multi_field() -> Result<()> { + let acct = Address::repeat_byte(0x93); + let mut cache = setup_cache().await?; + cache + .db_mut() + .insert_account_info(acct, AccountInfo::default()); + + let diff = cache.apply_update(&StateUpdate::account( + acct, + AccountPatch::default() + .balance(U256::from(42)) + .nonce(7) + .code(Bytes::from_static(&[0x60, 0x00])), + )); + + assert!(diff.accounts[0].balance.is_some()); + assert!(diff.accounts[0].nonce.is_some()); + assert!(diff.accounts[0].code_hash.is_some()); + Ok(()) +} + +#[tokio::test] +async fn account_patch_empty_code_clears_to_empty_hash() -> Result<()> { + let token = Address::repeat_byte(0x94); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); // non-empty code + + let diff = cache.apply_update(&StateUpdate::code(token, Bytes::new())); + let empty_hash = Bytecode::new_raw(Bytes::new()).hash_slow(); + assert_eq!( + diff.accounts[0].code_hash.expect("code changed").1, + empty_hash + ); + + // Patching empty over already-empty is a no-op. + let diff2 = cache.apply_update(&StateUpdate::code(token, Bytes::new())); + assert!(diff2.accounts.is_empty(), "empty over empty is a no-op"); + Ok(()) +} + +// =========================================================================== +// §16.8 — modify_slot write-through layer policy. +// =========================================================================== + +#[tokio::test] +async fn modify_slot_writes_through_both_layers() -> Result<()> { + let token = Address::repeat_byte(0x97); + let slot = U256::from(1); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + cache + .db_mut() + .insert_account_storage(token, slot, U256::from(10))?; + + cache.modify_slot(token, slot, |c| c.map(|v| v + U256::from(5))); + + assert_eq!(overlay_slot(&mut cache, token, slot), Some(U256::from(15))); + assert_eq!(backend_slot(&cache, token, slot), Some(U256::from(15))); + Ok(()) +} + +// =========================================================================== +// §16.8 — purge edges. +// =========================================================================== + +#[tokio::test] +async fn purge_absent_account_is_noop_record() -> Result<()> { + let acct = Address::repeat_byte(0x98); + let mut cache = setup_cache().await?; + + let diff = cache.apply_update(&StateUpdate::purge(acct, PurgeScope::Account)); + + assert_eq!(diff.purged.len(), 1); + assert!(!diff.purged[0].account_removed); + assert_eq!(diff.purged[0].slots_removed, 0); + Ok(()) +} + +#[tokio::test] +async fn purge_slots_counts_present_backend_slots_only() -> Result<()> { + let token = Address::repeat_byte(0x99); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + cache.inject_storage_batch(&[(token, U256::from(1), U256::from(10))]); // only slot 1 present + + let diff = cache.apply_update(&StateUpdate::purge( + token, + PurgeScope::Slots(vec![U256::from(1), U256::from(2)]), // slot 2 absent + )); + + assert_eq!( + diff.purged[0].slots_removed, 1, + "only the present backend slot is counted" + ); + Ok(()) +} + +// =========================================================================== +// §16.7 — account-field convenience constructors. +// =========================================================================== + +#[test] +fn state_update_account_field_constructors() { + let a = Address::repeat_byte(0x9a); + assert_eq!( + StateUpdate::nonce(a, 7), + StateUpdate::Account { + address: a, + patch: AccountPatch::default().nonce(7), + } + ); + assert_eq!( + StateUpdate::code(a, Bytes::from_static(&[0x60])), + StateUpdate::Account { + address: a, + patch: AccountPatch::default().code(Bytes::from_static(&[0x60])), + } + ); + let patch = AccountPatch::default().balance(U256::from(1)).nonce(2); + assert_eq!( + StateUpdate::account(a, patch.clone()), + StateUpdate::Account { address: a, patch } + ); +} + +// =========================================================================== +// §16.8 — Decision-2 write-through pins for the remaining protocols injectors. +// =========================================================================== + +#[cfg(feature = "protocols")] +#[tokio::test] +async fn inject_v2_pool_metadata_writes_through_to_backend() -> Result<()> { + use evm_fork_cache::cache::V2PoolMetadata; + + let pool = Address::repeat_byte(0xb3); + let mut cache = setup_cache().await?; + let meta = V2PoolMetadata { + token0: Address::repeat_byte(0x01), + token1: Address::repeat_byte(0x02), + last_block_timestamp: 0, + }; + + cache.inject_v2_pool_metadata(pool, &meta)?; + + assert!( + cache.pool_storage_slot_count(pool) > 0, + "inject_v2_pool_metadata must write through to the backend (Decision 2)" + ); + Ok(()) +} + +#[cfg(feature = "protocols")] +#[tokio::test] +async fn inject_v3_ticks_writes_through_to_backend() -> Result<()> { + use evm_fork_cache::cache::TickInfo; + use std::collections::HashMap; + + let pool = Address::repeat_byte(0xb4); + let mut cache = setup_cache().await?; + let mut ticks = HashMap::new(); + ticks.insert( + 0i32, + TickInfo { + liquidity_gross: 100, + liquidity_net: 50, + initialized: true, + }, + ); + + let injected = cache.inject_v3_ticks(pool, &ticks)?; + assert!(injected > 0); + assert!( + cache.pool_storage_slot_count(pool) > 0, + "inject_v3_ticks must write through to the backend (Decision 2)" + ); + Ok(()) +} + +// =========================================================================== +// §16 fix-review regressions — account_state-awareness on the account axis. +// =========================================================================== + +#[tokio::test] +async fn balance_delta_on_notexisting_overlay_account_is_skipped() -> Result<()> { + // A NotExisting overlay account is absent to the EVM (revm DbAccount::info() + // returns None), even if it carries a stale info.balance. loaded_account_info + // must treat it as cold so a BalanceDelta skips rather than applying to 1000. + use revm::database::AccountState; + let acct = Address::repeat_byte(0x7e); + let mut cache = setup_cache().await?; + cache.db_mut().insert_account_info( + acct, + AccountInfo { + balance: U256::from(1000), + ..Default::default() + }, + ); + cache + .db_mut() + .cache + .accounts + .get_mut(&acct) + .expect("overlay account present") + .account_state = AccountState::NotExisting; + + let diff = cache.apply_update(&StateUpdate::balance_delta( + acct, + SlotDelta::Add(U256::from(500)), + )); + + assert!( + diff.accounts.is_empty(), + "NotExisting account is EVM-absent: the delta must skip, not apply to the stale 1000" + ); + assert_eq!( + diff.skipped_balances, + vec![SkippedBalanceDelta { + address: acct, + delta: SlotDelta::Add(U256::from(500)), + }] + ); + Ok(()) +} + +#[tokio::test] +async fn account_patch_normalizes_zero_code_hash_across_layers() -> Result<()> { + // write_account_info_through normalizes a ZERO code_hash to KECCAK_EMPTY so both + // layers agree (the overlay write does this via insert_contract; the backend + // write must too). Seed the backend (unnormalized) with a ZERO hash, then patch. + use alloy_primitives::B256; + use revm::primitives::KECCAK_EMPTY; + let acct = Address::repeat_byte(0x7f); + let mut cache = setup_cache().await?; + cache.unchecked_blockchain_db().accounts().write().insert( + acct, + AccountInfo { + balance: U256::from(1), + code_hash: B256::ZERO, + ..Default::default() + }, + ); + + cache.apply_update(&StateUpdate::balance(acct, U256::from(2))); + + let backend_hash = cache + .unchecked_blockchain_db() + .accounts() + .read() + .get(&acct) + .map(|i| i.code_hash); + assert_eq!( + backend_hash, + Some(KECCAK_EMPTY), + "backend code_hash must be normalized to KECCAK_EMPTY, not left ZERO" + ); + Ok(()) +} + +// --------------------------------------------------------------------------- +// Phase 4 — SlotMasked: cold-aware read-modify-write masked slot write. +// +// `new = (old & !mask) | (value & mask)`. Only the `mask` bits are touched; the +// rest of the packed word is preserved. Cold (slot absent from both layers) is +// skipped and surfaced in `diff.skipped_masks` (the un-masked bits are unknown). +// --------------------------------------------------------------------------- + +#[tokio::test] +async fn slot_masked_sets_only_masked_bits() -> Result<()> { + let token = Address::repeat_byte(0x11); + let slot = U256::from(0); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + // Seed a packed word with the high bits set and the low byte clear. + let seeded = U256::MAX - U256::from(0xFF); // 0xFF..FF00 + cache.db_mut().insert_account_storage(token, slot, seeded)?; + + // Mask the low byte only, set it to 0x42. + let diff = cache.apply_update(&StateUpdate::slot_masked( + token, + slot, + U256::from(0xFF), + U256::from(0x42), + )); + + let expected = seeded | U256::from(0x42); // high bits preserved, low byte = 0x42 + assert_eq!(cache.cached_storage_value(token, slot), Some(expected)); + assert_eq!( + diff.slots, + vec![SlotChange { + address: token, + slot, + old: seeded, + new: expected, + }] + ); + assert!(diff.skipped_masks.is_empty()); + Ok(()) +} + +#[tokio::test] +async fn slot_masked_noop_when_masked_bits_already_equal() -> Result<()> { + let token = Address::repeat_byte(0x12); + let slot = U256::from(1); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + cache + .db_mut() + .insert_account_storage(token, slot, U256::from(0x42))?; + + // The masked bits already equal the target → no change. + let diff = cache.apply_update(&StateUpdate::slot_masked( + token, + slot, + U256::from(0xFF), + U256::from(0x42), + )); + + assert!( + diff.is_empty(), + "masked write that changes nothing is a no-op" + ); + assert!(diff.skipped_masks.is_empty()); + Ok(()) +} + +#[tokio::test] +async fn slot_masked_cold_slot_is_skipped_and_surfaced() -> Result<()> { + // Fresh address with no overlay account and no backend value: the slot is + // cold. A masked write cannot know the un-masked bits, so it is skipped. + let pool = Address::repeat_byte(0x13); + let slot = U256::from(0); + let mut cache = setup_cache().await?; + + let diff = cache.apply_update(&StateUpdate::slot_masked( + pool, + slot, + U256::from(0xFF), + U256::from(0x42), + )); + + assert!(diff.slots.is_empty()); + assert_eq!( + diff.skipped_masks, + vec![SkippedMask { + address: pool, + slot, + mask: U256::from(0xFF), + value: U256::from(0x42), + }] + ); + assert!(diff.has_skipped()); + assert!(!diff.is_fully_applied()); + assert_eq!(diff.skipped_len(), 1); + // Still cold — nothing was written. + assert_eq!(cache.cached_storage_value(pool, slot), None); + Ok(()) +} + +#[tokio::test] +async fn slot_masked_writes_through_both_layers() -> Result<()> { + let token = Address::repeat_byte(0x14); + let slot = U256::from(2); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + // Overlay-resident (hot) seed so an overlay account exists. + let seeded = U256::from(0xFF00); + cache.db_mut().insert_account_storage(token, slot, seeded)?; + + cache.apply_update(&StateUpdate::slot_masked( + token, + slot, + U256::from(0x00FF), + U256::from(0x0042), + )); + + let expected = U256::from(0xFF42); + assert_eq!( + overlay_slot(&mut cache, token, slot), + Some(expected), + "overlay (layer 1) updated" + ); + assert_eq!( + backend_slot(&cache, token, slot), + Some(expected), + "backend (layer 2) updated" + ); + Ok(()) +} + +#[tokio::test] +async fn slot_masked_full_mask_equals_absolute_on_hot_but_skips_cold() -> Result<()> { + // mask == U256::MAX behaves like an absolute write on a hot slot, but still + // skip-and-surfaces on a cold one (unlike StateUpdate::Slot). + let token = Address::repeat_byte(0x15); + let hot = U256::from(0); + let cold_addr = Address::repeat_byte(0x16); + let cold = U256::from(0); + let mut cache = setup_cache().await?; + install_mock_erc20(&mut cache, token); + cache + .db_mut() + .insert_account_storage(token, hot, U256::from(7))?; + + let hot_diff = cache.apply_update(&StateUpdate::slot_masked( + token, + hot, + U256::MAX, + U256::from(99), + )); + assert_eq!(cache.cached_storage_value(token, hot), Some(U256::from(99))); + assert_eq!(hot_diff.slots.len(), 1); + assert!(hot_diff.skipped_masks.is_empty()); + + let cold_diff = cache.apply_update(&StateUpdate::slot_masked( + cold_addr, + cold, + U256::MAX, + U256::from(99), + )); + assert!(cold_diff.slots.is_empty()); + assert_eq!(cold_diff.skipped_masks.len(), 1); + assert_eq!(cache.cached_storage_value(cold_addr, cold), None); + Ok(()) +} + +#[tokio::test] +async fn slot_masked_serde_round_trips() -> Result<()> { + let update = StateUpdate::slot_masked( + Address::repeat_byte(0x17), + U256::from(5), + U256::from(0xFF), + U256::from(3), + ); + let json = serde_json::to_string(&update)?; + let back: StateUpdate = serde_json::from_str(&json)?; + assert_eq!(update, back); + + let mut diff = StateDiff::default(); + diff.skipped_masks.push(SkippedMask { + address: Address::repeat_byte(0x18), + slot: U256::from(1), + mask: U256::from(0xFF), + value: U256::from(2), + }); + let json = serde_json::to_string(&diff)?; + let back: StateDiff = serde_json::from_str(&json)?; + assert_eq!(diff, back); + Ok(()) +}