Skip to content

perf: make raw trace collection demand-driven - #1541

Closed
nebasuke wants to merge 3 commits into
perf/traces-top-of-stackfrom
perf/traces-on-demand
Closed

nebasuke wants to merge 3 commits into
perf/traces-top-of-stackfrom
perf/traces-on-demand

Conversation

@nebasuke

@nebasuke nebasuke commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #1536, resolving the residual benchmark alert (1.12–1.20x on four scenarios) by only paying for trace collection when something consumes the traces.

Claude summary

Problem

#1536 replaced full-stack clones with top-of-stack capture, but the edr-benchmark runs on it still alerted at 1.12–1.28x against the traces-off baseline. The residual cost of includeCallTraces: All was paid statically on every transaction, even when nothing ever reads traces():

  • per-step single-element Box<[U256]> stack snapshots written into the arena at collect time, plus an ordering DFS per transaction;
  • the per-step top-of-stack capture itself;
  • arenas moved into the napi Response and retained until JS GC, instead of being dropped at filter time like the None path.

The scenario benchmark — like any Hardhat 2 run without solidity-coverage-style plugins — never reads the traces it was paying for. Hardhat 2 already computes exactly the right signal (needsTraces, from its VM event listener counts); it just had no way to act on it.

Approach (one commit each)

  1. Materialize stack tops lazily at marshal time. collect()/take() return the arena bundled with the raw captured tops (edr_solidity::tracing::CallTraces); nothing is written into the arena's steps. Response.traces() — the only consumer of per-step stacks in non-verbose mode — zips the tops with its existing depth-first step walk, materializing JS values directly. The walk's quirks are handled explicitly: the skipped leading STOP step still consumes a captured value, and children skipped by should_skip_call have no steps by definition. This removes the per-step allocations and the DFS from every transaction whose traces are never marshalled. Verified on the benchmark: the All Scenarios aggregate, synthetix, and rocketpool dropped below the alert threshold; seaport 1.28→1.13, openzeppelin 1.26→1.17, uniswap-v3-core 1.24→1.20 (step-dense, dominated by the remaining capture cost).
  2. setIncludeCallTraces(). A runtime toggle for includeCallTraces, mirroring the setVerboseTracing plumbing through SyncProvider/Provider/ProviderData. Traces that are not included are never collected: no arenas retained in responses and no top-of-stack capture for subsequent requests.
  3. Hardhat 2 patch: collect only while the vm trace bridge has listeners. The 2.28.6 patch no longer requests IncludeTraces.All statically at provider creation. Instead the wrapper evaluates the bridge's existing needsTraces listener check before dispatching each request and toggles the setter when the value changes — in practice one napi call per process, since listeners attach at plugin setup. hardhat_reset recreates the inner provider from the creation-time config, so the tracked state resets with it. The same change is mirrored on the Hardhat 2 source branch backing HH2: Port hardhat-network provider to the current EDR API and restore trace events hardhat#8422 (the patch source stays byte-identical to that branch).

With all three, the scenario benchmark exercises the true None path — the same work as the stored baseline — so the gate should go green, while coverage-style consumers pay capture + marshal only while actually listening.

Anticipated questions

  • Doesn't this invert perf: capture only the top of the stack for non-verbose raw traces #1536's plan to adopt StackSnapshotType::Top upstream (feat: add top-of-stack snapshot type paradigmxyz/revm-inspectors#468)? Yes. Upstream Top materializes a boxed single-element snapshot per step during execution — exactly the allocation the lazy path avoids — so migrating to it would now be a regression. Emit events in debug_traceTransaction and debug_traceCall #468 remains useful for other revm-inspectors consumers.
  • Why toggle from the hardhat patch instead of defaulting EDR to demand-driven? EDR can't know whether a consumer will call traces(); Hardhat 2's wrapper is the one place that knows (its VM event listener counts). Hardhat 3 consumers configure includeCallTraces explicitly per provider, unchanged.
  • Mixed configurations within one process? The setter only affects requests handled after it; each transaction's arena carries its own captured tops, so responses collected under different settings marshal independently (empty tops fall back to the arena's recorded stacks, e.g. verbose).

Testing

  • The write_stack_tops unit tests moved to the napi marshal layer as tests of the zip: execution order across nested call frames, the skipped-leading-STOP alignment, and the recorded-stack fallback when no tops were captured.
  • New edr_napi test: setIncludeCallTraces toggles take effect for subsequent requests — traces absent by default, present (with correct per-step tops) after enabling, absent again after disabling.
  • The perf: capture only the top of the stack for non-verbose raw traces #1536 nested-CALL contract test and the feat: add backwards compatibility for Response::traces #1301 trace tests pass unchanged; typings regenerated (setIncludeCallTraces is the only addition).
  • hardhat-tests: the trace-event-bridge spec still emits step/beforeMessage/afterMessage — proving the listeners → toggle → collect → bridge chain end-to-end — and the evm/hardhat modules pass against the updated patch.

@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 6, 2026 09:41 — with GitHub Actions Inactive
@changeset-bot

changeset-bot Bot commented Jul 6, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: b8a0351

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@codecov

codecov Bot commented Jul 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.44444% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.14%. Comparing base (f25666b) to head (8709235).

Files with missing lines Patch % Lines
crates/edr_napi/src/trace.rs 96.06% 3 Missing and 2 partials ⚠️
crates/edr_napi/src/mock.rs 0.00% 2 Missing ⚠️
crates/edr_napi/src/provider/response.rs 50.00% 2 Missing ⚠️
crates/edr_provider/src/provider.rs 66.66% 2 Missing ⚠️
crates/edr_napi/src/provider.rs 91.66% 0 Missing and 1 partial ⚠️
crates/edr_provider/src/data/gas.rs 94.44% 1 Missing ⚠️
Additional details and impacted files
@@                     Coverage Diff                      @@
##           perf/traces-top-of-stack    #1541      +/-   ##
============================================================
+ Coverage                     80.13%   80.14%   +0.01%     
============================================================
  Files                           445      445              
  Lines                         76572    76592      +20     
  Branches                      76572    76592      +20     
============================================================
+ Hits                          61358    61383      +25     
+ Misses                        13091    13087       -4     
+ Partials                       2123     2122       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@nebasuke
nebasuke force-pushed the perf/traces-top-of-stack branch from 16dcddd to f25666b Compare July 6, 2026 11:12
@nebasuke
nebasuke force-pushed the perf/traces-on-demand branch from 5233d06 to 8709235 Compare July 6, 2026 11:13
@nebasuke
nebasuke temporarily deployed to github-action-benchmark July 6, 2026 11:13 — with GitHub Actions Inactive
nebasuke added 3 commits July 6, 2026 11:41
Writing captured tops into the arena's steps at collect time costs one
heap allocation per step (a single-element Box<[U256]> snapshot) plus an
ordering DFS on every transaction, whether or not the traces are ever
read. Consumers that enable includeCallTraces without attaching VM
listeners - the scenario benchmark, and any HH2 run without
solidity-coverage-style plugins - paid it for nothing.

Instead, collect()/take() now return the arena bundled with the raw
captured tops (edr_solidity::tracing::CallTraces), and the bundle
travels to the napi Response as-is. Response.traces() zips the tops
with its existing depth-first step walk, materializing JS values
directly - the only consumer of per-step stacks in non-verbose mode.
Transactions whose traces are never marshalled now pay only the
per-step U256 copy at capture time.

The zip stays aligned with capture order across the walk's quirks: the
skipped leading STOP step still consumes a top, and children skipped by
should_skip_call have no steps by definition. Alignment is
debug_assert-ed at the root; running out of tops degrades to steps
without a stack, as before.

The write_stack_tops unit tests move to the napi marshal layer as
tests of the zip; the edr_napi provider.ts trace tests remain the
end-to-end contract and pass unchanged, as does the JS-visible shape
(typings unchanged).
Mirrors the setVerboseTracing plumbing: a setIncludeCallTraces napi
method through SyncProvider, Provider, and ProviderData, mutating the
observability config used for subsequent requests.

Traces that are not included are never collected (no arenas retained in
responses, no top-of-stack capture), so a consumer that knows whether
anything will read traces() - like Hardhat 2's trace-event bridge,
which already computes exactly that from its VM event listener counts -
can keep collection off while there are no readers instead of paying
for traces statically.
The hardhat patch enabled includeCallTraces: All statically, so every
transaction paid for trace collection (top-of-stack capture, arenas
retained in responses until GC) even when no plugin ever consumes the
vm trace events - the scenario benchmark's configuration, and any run
without solidity-coverage-style plugins.

The wrapper already computes whether traces are needed from its VM
event listener counts; evaluate that before dispatching the request and
toggle the provider's includeCallTraces via the new setter when it
changes. In practice that is one napi call per process, since listeners
attach at plugin setup. hardhat_reset recreates the inner provider from
the creation-time config, so the tracked state resets with it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant