Skip to content

feat: add backwards compatibility for Response::traces - #1301

Merged
Wodann merged 9 commits into
mainfrom
feat/tracing-back-compat
Oct 23, 2026
Merged

Wodann merged 9 commits into
mainfrom
feat/tracing-back-compat

Conversation

@Wodann

@Wodann Wodann commented Feb 19, 2026 •

Copy link
Copy Markdown
Member

Resolves #1288. Stacked on #1831, which sits on #1848; the review diff is the five commits of this branch.

#1278 unified tracing on CallTraceArena and dropped Response.traces. Hardhat 2 reads that getter to emit the VM step, beforeMessage and afterMessage events that solidity-coverage, hardhat-gas-reporter and hardhat-tracer consume, so no EDR release since then can ship to Hardhat 2.

This PR reinstates the raw trace format as Response.traces() on top of the arena and makes what a trace records configurable. It restores the old behaviour and fixes what the first version got wrong. The work that goes beyond the old behaviour is in the follow-up PR stacked on this one.

What changes

  • Response.traces() converts each recorded arena into the flat message, step and result sequence the removed edr_tracing collector produced, quirks included: the first STOP step is skipped, a call that reverts before its first step emits no message, DELEGATECALL reports the original caller, and halts only an inspector observes report as a revert instead of panicking (crates/edr_napi/src/trace.rs, result.rs).
  • Configurable recording. ObservabilityConfig gains recordStack: StackSnapshotType (None, Top, Full; default None). setVerboseTracing(true) records the full stack and memory, and disabling it restores the configured value.
  • Hardhat 2 patch 2.28.6 creates the provider with includeCallTraces: All and recordStack: Top, and its bridge reads traces(). It no longer passes defaultTransactionGasLimit, which the provider derives per block since feat: derive the default transaction gas limit from the executing block #1831. transactionGasCap stays for solidity-coverage.
  • StackSnapshotType::Top comes from a fork of revm-inspectors 0.42.2 with feat: add top-of-stack snapshot type paradigmxyz/revm-inspectors#468 applied (Wodann/revm-inspectors, branch feat/top-of-stack-snapshot-0.42).
  • Removed: crates/edr_solidity/src/nested_tracer.rs and the edr_tracing dependency of edr_napi, edr_provider and edr_solidity.

Compatibility

  • traces() is a method, not a getter, so a consumer converts once per response. build: upgrade Hardhat 2 to @nomicfoundation/edr v0.25.1 hardhat#7985 is the Hardhat 2 side.
  • setVerboseTracing(true) records the full stack and memory. On main it used TracingInspectorConfig::all(), which also recorded state diffs, return data and immediates that nothing read.

Behaviour changes

  • Hardhat 2 traces every transaction again, as before refactor: unify provider & soldity test tracing architecture #1278. Collecting only while a plugin listens is the follow-up PR.
  • A failed create reports no contractAddress. The removed collector forwarded revm's CreateOutcome.address, which is set for every create whose frame ran, reverted or halted. geth's callTracer clears to for a CREATE or CREATE2 frame that failed, in callFrame.processOutput of eth/tracers/native/call.go, and reth's call tracer sets it to None for any create with success false, in revm-inspectors' geth_empty_call_frame. The raw trace follows them and a test pins it. Hardhat 2 core only forwards the field to afterMessage listeners, and the validated plugin suites passed with it absent.

Benchmark Results

Provider benchmark on edr-benchmark-runner. pre is @nomicfoundation/edr@0.12.0-next.23, the EDR Hardhat 2.29.1 ships, dispatched on the same runner (run 37049477125). This PR is head c6d484cc4 (run 37368017467) against the stored main baseline of that day. Later heads of the same code measured the same 1.18× against their main. Like pre, this PR traces every transaction, while main traces none, which is the whole gap against main.

scenario pre main this PR PR/main PR/pre
All scenarios 219.6 s 153.8 s 177.0 s 1.15 0.81
synthetix 163.4 s 111.0 s 125.9 s 1.13 0.77
neptune-mutual 21.7 s 18.0 s 20.6 s 1.15 0.95
rocketpool 12.6 s 9.36 s 10.61 s 1.13 0.84
openzeppelin 11.0 s 7.47 s 10.03 s 1.34 0.91
seaport 5.75 s 4.17 s 5.08 s 1.22 0.88
uniswap-v3-core 4.52 s 3.28 s 4.06 s 1.24 0.90
safe-contracts 0.73 s 0.52 s 0.58 s 1.12 0.79

The CI alert against main is expected. With the follow-up PR on top, the series reaches 1.00× main (run 37088120381).

With a VM step listener attached, measured locally against pre on an idle machine, medians of 5 runs:

scenario pre this PR PR/pre time PR/pre RSS
uniswap-v3-core 28.9 s, 4344 MB 24.6 s, 3885 MB 0.85 0.89
openzeppelin 68.9 s, 6826 MB 60.3 s, 5408 MB 0.87 0.79
seaport 38.3 s, 2560 MB 47.8 s, 2595 MB 1.25 1.01

Seaport is the one case slower than pre; the follow-up PR brings it to 0.82×. The Solidity-test benchmark and the Hardhat 3 regression benchmark (99 entries, 11 scenarios) show no alert on this code (run 37539250895).

Validation

  • Third-party runs through a local Verdaccio, repeated on 2026-10-18 from fresh clones one repository at a time: build: upgrade Hardhat 2 to @nomicfoundation/edr v0.25.1 hardhat#7985 at 58316dda8 published as Hardhat 2.29.4 with this EDR at 47f7abfb8, against shipped Hardhat 2.29.1 with EDR 0.12.0-next.23. OpenZeppelin v5.7.0, Uniswap v3-core, Seaport, Rocket Pool, Safe and NexusMutual, plus the solidity-coverage, hardhat-tracer and hardhat-gas-reporter suites, about 25,000 cases per side. Every suite gives the same pass/fail set on both builds except one Safe test that parses debug_traceTransaction output, whose format changed independently of this PR. Listening consumers: Rocket Pool coverage 0.97× the old wall time at 0.63× its peak RSS, the NexusMutual suite with hardhat-tracer loaded 0.88× at 0.58×, OpenZeppelin coverage 0.96× at 1.00×. NexusMutual's old-build coverage run did not fit in memory on the test machine; the first run measured it at 1.01× and 0.79×.
  • Benchmarking with a step listener attached surfaced an abort in the publish build, an unreachable! on StateChangeDuringStaticCall in convert_status. The first commit fixes it and a napi test pins it.
  • The hardhat-tests spec trace-event-bridge.ts covers step and message events for a plain call and a precompile call.

Notes for review

  • Scope is the old behaviour plus fixes. recordStack: Top copies one stack element per step on every transaction while traces are statically on; the follow-up PR removes that cost by collecting only while a listener exists.
  • A memoised traces() that releases the Rust arenas after the first call was measured and dropped: a persistent reference to the JS array made seaport 9% slower and 58% larger in RSS, and a hidden-property variant was within noise.
  • Stack trace generation reads only pc and op from steps, so recordStack does not affect it.
  • The openzeppelin scenario's failure count flakes in listener mode on pre and this PR alike (1617–1675 against the snapshot's 1649); a harness issue, not introduced here.
  • Follow-ups: drop the revm-inspectors fork once EDR is on revm 43 or later, where 0.44 carries Emit events in debug_traceTransaction and debug_traceCall #468; remove crates/tracing, which has no dependents after this PR; build: upgrade Hardhat 2 to @nomicfoundation/edr v0.25.1 hardhat#7985 type-checks once a release contains this PR and should drop its copy of the default gas limit resolver too. A console-address bug in the same conversion is fixed separately in fix: recognise console calls by the Hardhat console address #1821.

@Wodann
Wodann requested a review from Copilot February 19, 2026 17:54
@Wodann Wodann self-assigned this Feb 19, 2026
@Wodann
Wodann had a problem deploying to github-action-benchmark February 19, 2026 17:54 — with GitHub Actions Error
@changeset-bot

changeset-bot Bot commented Feb 19, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: f88951c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@nomicfoundation/edr Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reintroduces backwards compatibility for the Response::traces getter to support Hardhat 2's EthereumJS VM interface. The implementation converts the new CallTraceArena format back to the flat trace format that Hardhat 2 expects.

Changes:

  • Adds Response::traces() method that converts CallTraceArena to flat trace format compatible with Hardhat 2
  • Removes edr_tracing dependency from multiple crates
  • Deletes nested_tracer.rs and related code that are no longer needed
  • Re-enables verbose tracing tests for Hardhat 2 compatibility

Reviewed changes

Copilot reviewed 22 out of 25 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
crates/edr_napi/src/trace.rs Implements conversion from CallTraceArena to flat trace format with Hardhat 2 compatibility quirks
crates/edr_napi/src/provider/response.rs Adds traces() method to Response
crates/edr_provider/src/observability.rs Updates tracing configuration to capture full stack snapshots
patches/hardhat@2.28.4.patch Updates Hardhat 2 integration to use new trace format
crates/edr_napi/test/provider.ts Re-enables verbose tracing tests
Multiple Cargo.toml files Removes edr_tracing dependency
crates/edr_solidity/src/nested_tracer.rs Deleted - no longer needed
Files not reviewed (1)
  • pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread crates/edr_napi/src/trace.rs Resolved
Comment on lines +149 to +154
let tracing_config = if config.verbose_raw_tracing {
TracingInspectorConfig::all()
} else {
TracingInspectorConfig::default_parity().set_steps(true)
TracingInspectorConfig::default_parity()
.set_steps(true)
.set_stack_snapshots(StackSnapshotType::Full)

Copilot AI Feb 19, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The StackSnapshotType::Full configuration is set regardless of verbose mode. This means that the full stack will always be captured in the tracing inspector, even when verbose tracing is disabled.

However, the actual filtering to return only the top of the stack happens in the conversion function raw_trace_from_call_trace_arena based on the verbose parameter. This creates an inconsistency where:

  1. The inspector always captures full stack (performance cost)
  2. The conversion then throws away most of the data when verbose=false

Consider setting StackSnapshotType based on verbose_raw_tracing to avoid unnecessary data collection and improve performance.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's fixed in the stacked PRs rather than here: #1536 drops StackSnapshotType::Full from the non-verbose config and captures only the top of the stack per step.

@Wodann
Wodann force-pushed the feat/tracing-back-compat branch from 87edb00 to 041399f Compare February 19, 2026 18:01
@Wodann
Wodann temporarily deployed to github-action-benchmark February 19, 2026 18:01 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark February 19, 2026 18:08 — with GitHub Actions Error
@Wodann
Wodann had a problem deploying to github-action-benchmark February 19, 2026 18:08 — with GitHub Actions Error
@Wodann
Wodann temporarily deployed to github-action-benchmark February 19, 2026 18:08 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark February 19, 2026 18:16 — with GitHub Actions Failure
@Wodann
Wodann had a problem deploying to github-action-benchmark February 19, 2026 18:16 — with GitHub Actions Failure
@codecov

codecov Bot commented Feb 19, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.57895% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.47%. Comparing base (a05a1c2) to head (f88951c).

Files with missing lines Patch % Lines
crates/edr_provider/src/data.rs 88.11% 6 Missing and 6 partials ⚠️
crates/edr_napi/src/trace.rs 92.59% 7 Missing and 1 partial ⚠️
crates/edr_provider/src/observability.rs 91.66% 2 Missing ⚠️
crates/edr_napi/src/config.rs 90.00% 1 Missing ⚠️
crates/edr_napi/src/context.rs 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1301      +/-   ##
==========================================
+ Coverage   81.02%   81.47%   +0.45%     
==========================================
  Files         474      473       -1     
  Lines       84016    83919      -97     
  Branches    84016    83919      -97     
==========================================
+ Hits        68072    68373     +301     
+ Misses      13661    13253     -408     
- Partials     2283     2293      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Wodann
Wodann requested a review from a team February 20, 2026 19:49
@Wodann
Wodann marked this pull request as ready for review February 20, 2026 22:32

@popescuoctavian popescuoctavian left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, but I believe there is an issue which Copilot pointed out and a small simplification.

Comment thread crates/edr_napi/src/trace.rs Resolved
Comment thread crates/edr_napi/src/trace.rs Resolved
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 14:59 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 15:07 — with GitHub Actions Inactive

@popescuoctavian popescuoctavian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is significant performance regression seen in the benchmarks -- up to 2.45x.

I believe this is caused by the inclusion of the full stack snapshot. However, I don't see an immediate solution here, with the existing revm-inspectors tracing code.

@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 19:23 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 19:40 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 19:40 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 20:53 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 21:29 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian force-pushed the feat/tracing-back-compat branch from d203b12 to 369dc66 Compare February 23, 2026 23:17
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 23:17 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark February 23, 2026 23:19 — with GitHub Actions Inactive
@Wodann

Wodann commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

/bench

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

⏳ EDR CI for this commit hasn't passed yet, so the regression benchmark was not started. Comment /bench again once CI is green.

@Wodann

Wodann commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

/bench

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

🚀 Starting regression benchmark for c6d484cc4da3 against Hardhat main (benchmarks matching test solidity*,test mocha*,test vitest*).

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

❌ Regression benchmark failed for c6d484cc4da3 against Hardhat main. This is either a detected performance regression or an infrastructure failure — see the run for details.

View workflow run

@Wodann

Wodann commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

/bench

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

🚀 Starting regression benchmark for c6d484cc4da3 against Hardhat main (benchmarks matching test solidity*,test mocha*,test vitest*).

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

❌ Regression benchmark failed for c6d484cc4da3 against Hardhat main. This is either a detected performance regression or an infrastructure failure — see the run for details.

View workflow run

@Wodann

Wodann commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

/bench

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

🚀 Starting regression benchmark for c6d484cc4da3 against Hardhat main (benchmarks matching test solidity*,test mocha*,test vitest*).

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

❌ Regression benchmark failed for c6d484cc4da3 against Hardhat main. This is either a detected performance regression or an infrastructure failure — see the run for details.

View workflow run

@Wodann

Wodann commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

/bench

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

🚀 Starting regression benchmark for c08affa5a573 against Hardhat main (benchmarks matching test solidity*,test mocha*,test vitest*).

@Wodann

Wodann commented Oct 6, 2026

Copy link
Copy Markdown
Member Author

/bench

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

🚀 Starting regression benchmark for 635e615e3300 against Hardhat main (benchmarks matching test solidity*,test mocha*,test vitest*).

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

✅ Regression benchmark passed for 635e615e3300 against Hardhat main.

View workflow run

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two trace-conversion edge cases still produce VM events that differ from the legacy behavior.

2 open issues
1 file not reviewed
  • pnpm-lock.yaml: Generated file

Review effort: Balanced

Comment thread crates/edr_napi/src/trace.rs Resolved
Comment thread crates/edr_napi/src/trace.rs Resolved

@popescuoctavian popescuoctavian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a full re-review, looks good overall, thanks! Left a couple of questions and one suggestion, up to you if they should be addressed. Feel free to merge without requesting another review.

Comment thread crates/edr_napi/src/trace.rs Resolved
Comment thread crates/edr_napi/src/trace.rs Resolved
Hardhat 2 consumes raw traces to emit its VM step and message events.
`Response::traces()` converts each recorded call trace arena into the
flat message, step and result sequence the removed `edr_tracing`
collector produced.

The `edr_tracing` dependency and `nested_tracer` are removed from
`edr_napi`, `edr_provider` and `edr_solidity`.
`ObservabilityConfig` gains `recordStack`. Hardhat 2 needs the top of
the stack for its VM step events. Recording the full stack for every
step retains several times as much memory.

`StackSnapshotType::Top` comes from a fork of revm-inspectors 0.41.4
with paradigmxyz/revm-inspectors#468 applied.

Verbose tracing records the full stack and memory while enabled.
Disabling it restores the configured `recordStack`.
The patch for 2.28.6 carries over the 2.28.4 patch and re-enables the
VM event bridge, which reads `Response.traces()`. Hardhat 2 creates its
provider with `includeCallTraces: All`, so responses carry the traces
the bridge reads.

The provider also records the top of the stack for Hardhat 2's VM step
events. Its message events expect precompile calls, which call traces
always include.

A hardhat-tests spec asserts that the bridge emits step and message
events, including for a precompile call.
The Hardhat 2 patch no longer computes `defaultTransactionGasLimit`.
Since the provider made the field optional, EDR derives the same value
per block. On forks it also uses the executing block's hardfork, so
the derived value is better there.

`transactionGasCap` stays, so solidity-coverage keeps lifting the
EIP-7825 transaction gas limit.
The legacy collector dropped any call frame that reverted before its
first step, including the top-level frame. The raw trace conversion now
also applies this rule to a non-create root node and returns no
messages for it.

The legacy collector skipped a leading STOP step only in call frames,
because creation frames carried no code. The conversion now keeps the
leading STOP in creation frames.
The field was added earlier on this branch and nothing ever set it to
anything but its default. Memory snapshots follow setVerboseTracing as
before.
Neither status reaches an arena today. A fatal database error makes revm
return an error, and observe_execution drops the observer on an error
before a response exists. Suspend always sets a new-frame action first,
so it never ends a frame.

A panic would abort the published binary, which is built with panic set
to abort. Both statuses now map to no reason while success stays false.
A test pins that a failed storage read in a nested call fails the
request without collecting a trace. The internal status rests on
revm's interpreter convention and cannot be produced from EDR, so it
has no test.
geth's call tracer and reth report no address for a failed create. The
removed edr_tracing collector reported the address the frame ran at. The
raw trace follows geth and reth. The tests pin that choice.

This branch was previously deployed

1 inactive deployment
github-action-benchmark — f88951cc Deployed Oct 21, 2026 by Wodann via Run JS Solidity test runner benchmark #5876
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-enable traces for Hardhat 2

4 participants