Skip to content

perf: collect Hardhat 2 call traces only while listeners exist - #1809

Draft
Wodann wants to merge 4 commits into
mainfrom
perf/traces-on-demand-v2
Draft

Wodann wants to merge 4 commits into
mainfrom
perf/traces-on-demand-v2

Conversation

@Wodann

@Wodann Wodann commented Oct 3, 2026

Copy link
Copy Markdown
Member

With #1301, Hardhat 2 creates its provider with includeCallTraces: All, so every response records, retains and tears down a call trace arena. Hardhat 2 only reads those traces when a plugin has registered a VM event listener, which the provider benchmark and most projects never do. On the benchmark that static cost is 1.11× main.

This PR collects traces only while a listener exists, and records per-step stack and memory snapshots only while traces are collected.

What changes

  • Provider.setIncludeCallTraces(includeCallTraces) sets which call traces the responses of later requests include, through the same plumbing as setVerboseTracing: SyncProvider, edr_provider::Provider, a SetIncludeCallTraces message applied to ProviderData by the event loop (crates/edr_napi/src/provider.rs, crates/edr_provider/src/event_loop.rs).
  • Hardhat 2 patch: the provider is created without includeCallTraces. Before each request the wrapper evaluates its existing needsTraces expression and calls the setter when the value changed since the last call, so a process with plugins makes one napi call. hardhat_reset recreates the inner provider from the creation-time config and resets the tracked value (patches/hardhat@2.28.6.patch).
  • Snapshots follow includeCallTraces. ObservabilityConfig::recorded_stack() and records_memory() return None/false while call traces are excluded. Only Response::traces() reads step snapshots; the stack-trace path reads pc and op. The observer config is built per EVM run, so the setter switches recording too (crates/edr_provider/src/observability.rs).
  • Benchmark harness: pnpm providerBenchmark --vm-step-listener registers a no-op step listener so the trace-reading path can be measured (js/benchmark).

Compatibility

  • Hardhat 3 never calls the setter and configures includeCallTraces per provider as before; no change.
  • Hardhat 2 projects without tracing plugins now run with the same recording as Hardhat 3's default. Plugins that attach listeners after the first request get traces from the next request on, as needsTraces is evaluated per request.

Benchmark Results

Provider benchmark on edr-benchmark-runner, main baseline ea0b7fc1e. "#1301 alone" is the series with traces statically enabled (run 37063378094); "this PR" is the same series with these commits (run 37088120381, zero alerts). Both runs were taken on top of #1562, which was measured at 0.97× main on this path.

scenario main #1301 alone this PR PR/main
All scenarios 155.4 s 172.1 s 155.9 s 1.00
synthetix 112.4 s 123.3 s 112.9 s 1.00
neptune-mutual 18.1 s 20.0 s 18.0 s 1.00
rocketpool 9.34 s 10.35 s 9.49 s 1.02
openzeppelin 7.57 s 9.31 s 7.51 s 0.99
seaport 4.16 s 4.70 s 4.23 s 1.02
uniswap-v3-core 3.29 s 3.78 s 3.30 s 1.00
safe-contracts 0.52 s 0.60 s 0.52 s 1.01

With a listener attached (--vm-step-listener, local idle machine, medians of 5), this PR against the EDR that Hardhat 2.29.1 ships: seaport 0.82×, uniswap-v3-core 0.84×, openzeppelin 0.87× in wall time; peak RSS 0.83× to 0.88×. Against #1301 alone the listener path is unchanged within 1% on uniswap-v3-core and openzeppelin and 0.66× on seaport.

Hardhat 3 regression benchmark on the series including these commits: run 37093003911, 99 entries across 11 scenarios, zero alerts.

Validation

  • crates/edr_napi pnpm test: 95 passing, 1 pending. New: traces absent by default, present with per-step top-of-stack after setIncludeCallTraces(All), absent after None.
  • cargo test -p edr_provider: two new unit tests on recorded_stack()/records_memory() for excluded and included traces.
  • hardhat-tests: 2811 passing, 167 pending. The trace-event-bridge spec adds: no listener yields no traces; a listener added then removed gets traces only while registered; a listener kept across hardhat_reset still gets steps; a listener removed before a reset and re-added after it. Removing the reset line from the installed patch makes the third case fail.
  • clippy -D warnings, cargo fmt, pnpm lint, pnpm build:typingFile, pnpm install --frozen-lockfile: clean.

Notes for review

  • Ports the two relevant commits of perf: make raw trace collection demand-driven #1541, which was closed unreviewed; its 2026-07-06 benchmark run reached the same parity with main.
  • recordStack: Top allocates a one-element stack copy per step while traces are included, which is why recording is gated on includeCallTraces rather than left to the configured value alone.
  • After a reset the tracked value is set to false rather than "unknown": the new inner provider is created with None, so this is the same state, and a listener that survives the reset triggers one All call on the next request.
  • The Hardhat 2 side needs the same needsTraces toggling in build: upgrade Hardhat 2 to @nomicfoundation/edr v0.25.1 hardhat#7985.

@Wodann
Wodann temporarily deployed to github-action-benchmark October 3, 2026 13:24 — with GitHub Actions Inactive
@changeset-bot

changeset-bot Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: cdde332

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@nomicfoundation/edr Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@codecov

codecov Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.87879% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.51%. Comparing base (034998c) to head (cdde332).

Files with missing lines Patch % Lines
crates/edr_napi/src/mock.rs 0.00% 6 Missing ⚠️
crates/edr_napi/src/provider.rs 90.00% 0 Missing and 1 partial ⚠️
crates/edr_napi_core/src/provider.rs 75.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1809      +/-   ##
==========================================
- Coverage   81.51%   81.51%   -0.01%     
==========================================
  Files         473      473              
  Lines       84041    84105      +64     
  Branches    84041    84105      +64     
==========================================
+ Hits        68505    68554      +49     
- Misses      13245    13254       +9     
- Partials     2291     2297       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Wodann
Wodann temporarily deployed to github-action-benchmark October 3, 2026 13:31 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark October 3, 2026 13:31 — with GitHub Actions Failure
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from 254b7b0 to 1264638 Compare October 4, 2026 00:08
@Wodann
Wodann temporarily deployed to github-action-benchmark October 4, 2026 00:08 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 4, 2026 00:13 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark October 4, 2026 00:13 — with GitHub Actions Failure
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from 1264638 to 12ef4b3 Compare October 5, 2026 20:30
@Wodann
Wodann temporarily deployed to github-action-benchmark October 5, 2026 20:30 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 5, 2026 21:06 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark October 5, 2026 21:06 — with GitHub Actions Failure
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from 12ef4b3 to 508e73b Compare October 6, 2026 22:12
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from 508e73b to ff9e522 Compare October 6, 2026 22:13
@Wodann
Wodann temporarily deployed to github-action-benchmark October 6, 2026 22:14 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 6, 2026 22:22 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 6, 2026 22:22 — with GitHub Actions Inactive
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from ff9e522 to 6f6f717 Compare October 7, 2026 15:47
@Wodann
Wodann temporarily deployed to github-action-benchmark October 7, 2026 15:47 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark October 7, 2026 15:52 — with GitHub Actions Failure
@Wodann
Wodann had a problem deploying to github-action-benchmark October 7, 2026 15:52 — with GitHub Actions Error
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from 6f6f717 to 1bc8f61 Compare October 7, 2026 16:34
@Wodann
Wodann requested a deployment to github-action-benchmark October 7, 2026 16:34 — with GitHub Actions Queued
@Wodann
Wodann temporarily deployed to github-action-benchmark October 7, 2026 17:24 — with GitHub Actions Inactive
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from 0e1814b to 1b08b38 Compare October 19, 2026 15:54
@Wodann
Wodann had a problem deploying to github-action-benchmark October 19, 2026 15:54 — with GitHub Actions Error
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from 1b08b38 to e63dcd4 Compare October 19, 2026 16:17
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from e63dcd4 to 5e7efe4 Compare October 19, 2026 16:18
@Wodann
Wodann temporarily deployed to github-action-benchmark October 19, 2026 16:18 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 19, 2026 16:24 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark October 19, 2026 16:24 — with GitHub Actions Failure
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from 5e7efe4 to a8fe85c Compare October 19, 2026 17:29
@Wodann
Wodann temporarily deployed to github-action-benchmark October 19, 2026 17:29 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 19, 2026 17:33 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark October 19, 2026 17:33 — with GitHub Actions Failure
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from a8fe85c to 6b8cf3b Compare October 21, 2026 21:10
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from 6b8cf3b to daa45d9 Compare October 21, 2026 22:04
@Wodann
Wodann temporarily deployed to github-action-benchmark October 21, 2026 22:04 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark October 21, 2026 22:07 — with GitHub Actions Error
@Wodann
Wodann had a problem deploying to github-action-benchmark October 21, 2026 22:07 — with GitHub Actions Error
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from daa45d9 to 559ed77 Compare October 21, 2026 22:17
@Wodann
Wodann temporarily deployed to github-action-benchmark October 21, 2026 22:18 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark October 21, 2026 22:24 — with GitHub Actions Error
@Wodann
Wodann had a problem deploying to github-action-benchmark October 21, 2026 22:24 — with GitHub Actions Error
@Wodann
Wodann force-pushed the perf/traces-on-demand-v2 branch from 559ed77 to 062845c Compare October 21, 2026 22:49
Hardhat 2 only reads the EVM traces of a response when a VM event
listener exists. Plugins like coverage or tracers register one, but the
harness registered none, so it never measured that path.
`Provider.setIncludeCallTraces` sets which call traces the responses of
later requests include. It follows the `setVerboseTracing` plumbing:
`SyncProvider`, then `Provider`, then a `SetIncludeCallTraces` message
that the event loop applies to `ProviderData`.

Traces that are not included are never collected. A consumer that reads
traces conditionally can keep collection off while nothing reads them.
Hardhat 2 created its provider with `includeCallTraces: All`. Every
response then recorded and retained a call trace arena, even without a
VM event listener to read it.

The provider is now created without call traces. Before each request,
the wrapper evaluates whether a VM event listener exists. When that
differs from the last applied value, it calls `setIncludeCallTraces`.
`hardhat_reset` recreates the provider from the creation-time config,
so the tracked value resets with it.
The tracing inspector recorded step stack and memory snapshots for every
transaction, even while responses excluded call traces. Only
`Response::traces()` reads those snapshots. It only has call traces to
read while they are included. The Solidity stack-trace path reads only
`pc` and `op`.

`recorded_stack()` and `records_memory()` now record nothing while
`include_call_traces` is `None`. The observer config is built from them
for each EVM run, so `setIncludeCallTraces` also switches recording.

This branch was previously deployed

1 inactive deployment
github-action-benchmark — cdde332b Deployed Oct 23, 2026 by Wodann via Run JS Solidity test runner benchmark #5921
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant