Skip to content

test: pin the debug_trace* JSON-RPC contract - #1907

Merged
Wodann merged 3 commits into
mainfrom
test/debug-trace-contract
Nov 17, 2026
Merged

Wodann merged 3 commits into
mainfrom
test/debug-trace-contract

Conversation

@Wodann

@Wodann Wodann commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

EDR had no test of the JSON that debug_traceTransaction and debug_traceCall return. The only checks lived in hardhat-tests, through the Hardhat 2 package, so a change in revm-inspectors, alloy or EDR's option handling would surface downstream first. Hardhat 2 converts these traces back to its legacy format in the open NomicFoundation/hardhat#7985, and that conversion depends on the exact output.

This PR pins the contract in EDR with 47 unit tests, against the Ethereum execution-apis spec at a pinned commit. It also brings hardhat-tests back to the format Hardhat 2 will ship, so CI guards it from both sides.

What changes

  • crates/edr_provider/src/requests/debug/tests/: each test runs a transaction or call on the in-process fixture, calls the handler and asserts on the serialised JSON. spec.rs holds 34 tests, one per spec requirement, each documented with the clause it pins. edr.rs holds 13 tests of behaviour the spec leaves undefined, with the reason for each in its module doc. The fixture pins Osaka, because Amsterdam changes the gas constants.
  • Spec requirements pinned
  • EDR behaviour pinned
  • hardhat-tests: a revert of feat: add support for Geth built-in tracers #1251's fixture changes restores Hardhat 2's trace format. The Hardhat 2 patch gains #7985's conversion, with its compiled files regenerated. A follow-up PR adds the tracker-heavy scenarios from #7985 and drops the cases the Rust tests now cover.
  • test_utils gains new_local_with_hardfork, transaction_request and TEST_GAS_PRICE.

Validation

  • Mutation checks on the handler, each reverted, fail the test named for them
  • Recompiling the patched Hardhat 2 sources reproduces all ten compiled files in the patch byte for byte. A probe through the patched package returns Hardhat 2's unprefixed format.
  • Every spec requirement was read against opcode-tracer.yaml and trace.yaml at commit 3284ee80c.

Notes for review

  • The first commit is a mechanical revert of about 520,000 lines that reviewers can skip. Review using this link instead.
  • disableMemory: false enables memory when enableMemory is absent since revm-inspectors 0.44, where EDR 0.25.0 ignored it. The option is not in the spec, so the test pins current behaviour.
  • The Some(OutOfGas) error text is revm-inspectors' Debug formatting and still present in 0.44.3, so the fix is upstream. The test is a tripwire.
  • The patch keeps two nits that predate this PR, an unused import and an uncast expression that affects type-checking only. #7985 fixes both, and the patch shrinks once Hardhat 2 releases it.
  • The PR changes tests, the patch and the lockfile's patch hash, none of which ships in a package, so it carries no changeset needed. A pre-Spurious-Dragon test was not possible, because L1Hardfork starts at Byzantium.

@changeset-bot

changeset-bot Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 80d6eae

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.93939% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.54%. Comparing base (cfa3dea) to head (80d6eae).
⚠️ Report is 11 commits behind head on main.

Files with missing lines Patch % Lines
crates/edr_provider/src/test_utils.rs 93.93% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1907      +/-   ##
==========================================
+ Coverage   81.53%   81.54%   +0.01%     
==========================================
  Files         473      473              
  Lines       84080    84138      +58     
  Branches    84080    84138      +58     
==========================================
+ Hits        68553    68614      +61     
+ Misses      13235    13233       -2     
+ Partials     2292     2291       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Wodann
Wodann temporarily deployed to github-action-benchmark October 29, 2026 20:56 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 29, 2026 20:56 — with GitHub Actions Inactive
@Wodann Wodann added the no changeset needed This PR doesn't require a changeset label Oct 29, 2026
@Wodann
Wodann force-pushed the test/debug-trace-contract branch from 03cd180 to b55608a Compare October 30, 2026 02:48
@Wodann
Wodann temporarily deployed to github-action-benchmark October 30, 2026 02:48 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 30, 2026 02:51 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 30, 2026 02:51 — with GitHub Actions Inactive
@Wodann
Wodann force-pushed the test/debug-trace-contract branch from b55608a to b24cc2d Compare October 30, 2026 13:54
@Wodann
Wodann temporarily deployed to github-action-benchmark October 30, 2026 13:54 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 30, 2026 13:57 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 30, 2026 13:57 — with GitHub Actions Inactive
@Wodann
Wodann force-pushed the test/debug-trace-contract branch from b24cc2d to 930280a Compare October 30, 2026 23:45
@Wodann
Wodann temporarily deployed to github-action-benchmark October 30, 2026 23:45 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 30, 2026 23:48 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 30, 2026 23:48 — with GitHub Actions Inactive
@Wodann Wodann self-assigned this Oct 30, 2026
@Wodann
Wodann force-pushed the test/debug-trace-contract branch from 930280a to 3ac49e1 Compare October 31, 2026 01:59
@Wodann
Wodann had a problem deploying to github-action-benchmark October 31, 2026 01:59 — with GitHub Actions Error
@Wodann
Wodann force-pushed the test/debug-trace-contract branch from 3ac49e1 to b5b912e Compare October 31, 2026 02:23
@Wodann
Wodann had a problem deploying to github-action-benchmark October 31, 2026 02:23 — with GitHub Actions Error
@Wodann
Wodann force-pushed the test/debug-trace-contract branch from b5b912e to a2bb74d Compare October 31, 2026 02:44
@Wodann
Wodann had a problem deploying to github-action-benchmark October 31, 2026 02:44 — with GitHub Actions Error
@Wodann
Wodann force-pushed the test/debug-trace-contract branch from a2bb74d to b5707ff Compare October 31, 2026 02:59
@Wodann
Wodann temporarily deployed to github-action-benchmark October 31, 2026 03:00 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark October 31, 2026 04:50 — with GitHub Actions Error
@Wodann
Wodann force-pushed the test/debug-trace-contract branch from 112de34 to 4a4f721 Compare October 31, 2026 04:54
@Wodann
Wodann temporarily deployed to github-action-benchmark October 31, 2026 04:55 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 31, 2026 04:57 — with GitHub Actions Inactive
@Wodann
Wodann temporarily deployed to github-action-benchmark October 31, 2026 04:57 — with GitHub Actions Inactive
@Wodann
Wodann requested a balanced review from Copilot November 6, 2026 08:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Storage reconstruction across reentrant calls and contract creation, plus new RPC lookups, requires native EDR/Hardhat integration validation of legacy-fixture parity and fork behavior that was unavailable during review.

No open issues 🎉

1 file not reviewed
  • pnpm-lock.yaml: Generated file
What changed in this PR

Pins EDR’s debug_traceTransaction and debug_traceCall JSON contract at the provider layer, while preserving coverage of Hardhat 2’s legacy trace format.

Changes:

  • Adds 47 Rust tests covering specification requirements and EDR-specific behavior, using an Osaka-pinned fixture.
  • Adds Hardhat 2 compatibility scenarios and updates existing trace comparisons.
  • Removes replaced fixtures and updates the Hardhat patch hash.
File Description
pnpm-lock.yaml Updates the Hardhat 2 patch hash.
hardhat-tests/​test/​internal/​hardhat-network/​provider/​utils/​assertEqualTraces.ts Adapts comparisons to legacy trace fixtures.
hardhat-tests/​test/​internal/​hardhat-network/​provider/​modules/​debug/​traceTransaction.ts Updates transaction expectations and removes replaced cases.
hardhat-tests/​test/​internal/​hardhat-network/​provider/​modules/​debug/​traceCall.ts Updates call requests and removes replaced cases.
hardhat-tests/​test/​internal/​hardhat-network/​provider/​modules/​debug/​debug-trace.ts Adds legacy-format compatibility and option-handling tests.
hardhat-tests/​test/​fixture-debug-traces/​traceCall/​ethTransfer.ts Removes the replaced transfer fixture.
hardhat-tests/​test/​fixture-debug-traces/​traceCall/​contractAGetMessageDebugConfig.ts Removes the replaced tracing-options fixture.
hardhat-tests/​test/​fixture-debug-traces/​elongatedMemoryRegressionTestTrace.ts Removes the replaced memory regression fixture.
crates/​edr_provider/​src/​test_utils.rs Adds hardfork selection and reusable transaction construction.
crates/​edr_provider/​src/​requests/​debug/​tests/​spec.rs Adds 34 specification contract tests.
crates/​edr_provider/​src/​requests/​debug/​tests/​edr.rs Adds 13 EDR-specific behavior tests.
crates/​edr_provider/​src/​requests/​debug/​tests.rs Adds shared fixtures, bytecode builders, and JSON assertions.
crates/​edr_provider/​src/​requests/​debug.rs Registers the debug test module.

Review effort: Balanced

@Wodann
Wodann force-pushed the test/debug-trace-contract branch from 4a4f721 to 639a308 Compare November 6, 2026 08:37
@Wodann
Wodann temporarily deployed to github-action-benchmark November 6, 2026 08:37 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark November 6, 2026 08:39 — with GitHub Actions Error
@Wodann
Wodann had a problem deploying to github-action-benchmark November 6, 2026 08:39 — with GitHub Actions Error
@Wodann
Wodann force-pushed the test/debug-trace-contract branch from 639a308 to 317983f Compare November 6, 2026 08:46
@Wodann
Wodann temporarily deployed to github-action-benchmark November 6, 2026 08:46 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark November 6, 2026 08:49 — with GitHub Actions Error
@Wodann
Wodann had a problem deploying to github-action-benchmark November 6, 2026 08:49 — with GitHub Actions Error
Commit f4bdc36 (#1251) switched the debug fixtures in hardhat-tests
to geth's format and adjusted the debug tests and assertEqualTraces to
match. A later commit makes the patched Hardhat 2 package return
Hardhat 2's format again, so this reverts the hardhat-tests part of
f4bdc36: the fixture files are the f4bdc36^ versions byte for
byte, and the test and helper hunks are reversed on top of the current
files. The stack-trace and package.json changes of f4bdc36 stay.

The three fixtures that f4bdc36 gave a generation recipe keep it, plus
a note on converting geth's 0x-prefixed output to Hardhat's format.

The debug tests are red from this commit until the patch commit lands.
@Wodann
Wodann force-pushed the test/debug-trace-contract branch from 317983f to 790c55b Compare November 6, 2026 08:58
@Wodann
Wodann temporarily deployed to github-action-benchmark November 6, 2026 08:59 — with GitHub Actions Inactive
@Wodann
Wodann had a problem deploying to github-action-benchmark November 6, 2026 09:01 — with GitHub Actions Error
@Wodann
Wodann had a problem deploying to github-action-benchmark November 6, 2026 09:01 — with GitHub Actions Error
@Wodann
Wodann force-pushed the test/debug-trace-contract branch from 790c55b to ddd22df Compare November 6, 2026 09:19
Add unit tests for the debug_traceTransaction and debug_traceCall
handlers. They assert on the serialised GethTrace, which is the JSON
that the provider returns.

The reference is the Ethereum execution-apis spec at one pinned commit.
The tests follow its opcode logger schema and the method clauses of
trace.yaml.

The tests are split into two modules. The spec module holds tests per
spec requirement, each documented with the clause it pins. The edr
module holds behaviour that the spec leaves undefined, and its module
doc gives the reason EDR pins each one.

The spec tests cover the struct log fields and their formats, and the
meaning of pc, depth, gas and gasCost, including the cold SLOAD charge.
Storage is pinned as a cumulative per-contract snapshot, which a
DELEGATECALL shares with its caller. The refund counter is present on
every step. After an SSTORE restores a slot's original value, the refund
is the EIP-2200 restore refund. The trace's gas deducts the capped
refund. Further tests cover the default stack, the memory, stack,
storage and return data options and an empty tracer. The step limit is
pinned to cap the recorded steps without stopping execution. Other tests
cover the trace results of transfers, reverts, successful calls and a
halting precompile call. debug_traceTransaction is pinned to replay the
preceding transactions of its block.

The edr tests cover the Unknown opcode name and the halt error text.
disableMemory and disableReturnData are not spec options, but alloy
reads them. disableMemory: false enables memory when enableMemory is
absent, and enableMemory takes precedence when both are set.
disableReturnData: false likewise enables return data. The edr tests
also cover the callTracer result and the errors for an unknown tracer or
transaction hash. For debug_traceCall they pin the default block,
contract creation and state isolation.

Hardhat 2 converts these traces back to its own format. That
conversion depends on this contract, so a change in revm-inspectors,
alloy or EDR's option handling now fails here first.

A test_utils helper now builds transaction requests, and another
builds a local fixture on a given hardfork.
Hardhat 2 converts EDR's geth-format debug traces back to its legacy
format in the open NomicFoundation/hardhat#7985. EDR's patched Hardhat
2.28.6 lacked that conversion, so EDR CI tested a format that Hardhat 2
no longer returns.

Port the conversion from #7985. The port adds edrRpcDebugTraceToHardhat2
and its StorageTracker to convertToEdr.ts. It wires the memory default,
the empty tracer handling and the result conversion into the provider's
request method. RpcStructLog gains the returnData field. The debug trace
code matches #7985. Outside it, the port keeps EDR's existing patch and
its formatting.

The compiled files come from TypeScript 5.0.4 with the package's own
2.28.6 tsconfig. That compiler reproduces the patched provider.js byte
for byte. The source maps are now regenerated with the code they map.

The revert commit restored the Hardhat 2 format fixtures and reversed
the test hunks of #1251, so the debug tests are green again from here.
assertEqualTraces loses the refund normalisation that only geth's format
needed and the zeroing of call-opcode gas, which EDR now reports as the
fixtures do. It compares returnValue without its prefix, except for a
failed trace whose fixture recorded none. The unsupported tracer tests
expect EDR's rejection of such a tracer as a JavaScript tracer, and
debug_traceTransaction checks the hash first.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the trace regression coverage and legacy-format restoration.

No open issues 🎉

1 file not reviewed
  • pnpm-lock.yaml: Generated file

Review effort: Balanced

@anaPerezGhiglia

Copy link
Copy Markdown
Contributor

Disclaimer: given the size of the diff, I didn't review this PR line by line. I asked Claude to walk me through it, so sorry in advance if I'm missing something that's in the code.

I see the motivation in the commit message: once NomicFoundation/hardhat#7985 lands, HH2 converts EDR's traces back to its legacy format. But as agreed in the EDR roadmap, we're moving away from hardhat-tests, especially now that HH2 is reaching end of life. So I'm not sure why we're expanding the HH2 patch and moving hardhat-tests back to HH2's legacy format, rather than moving away from hardhat-tests and adding coverage on the Rust side. EDR should test the contract it adheres to, not which tracing options HH2 sends or how it converts the responses, which I think belongs in the Hardhat repo.

The Rust tests in "test: pin the debug_trace* JSON-RPC contract" are the direction I expected from this PR. For golden tests, main already compares EDR's unconverted output against the Geth-format fixtures from #1251. I'd port those to Rust rather than swap them for HH2-format ones.

Could you share why you went with the HH2 patch approach? From my POV it doesn't fit the EDR roadmap, but I might be missing some context.

@Wodann

Wodann commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

Disclaimer: given the size of the diff, I didn't review this PR line by line. I asked Claude to walk me through it, so sorry in advance if I'm missing something that's in the code.

As explained in the notes for reviewer, it's easier to review the link I shared. It excludes the largest commit which is a mechanical revert.

I see the motivation in the commit message: once NomicFoundation/hardhat#7985 lands, HH2 converts EDR's traces back to its legacy format. But as agreed in the EDR roadmap, we're moving away from hardhat-tests, especially now that HH2 is reaching end of life. So I'm not sure why we're expanding the HH2 patch and moving hardhat-tests back to HH2's legacy format, rather than moving away from hardhat-tests and adding coverage on the Rust side. EDR should test the contract it adheres to, not which tracing options HH2 sends or how it converts the responses, which I think belongs in the Hardhat repo.

The Rust tests in "test: pin the debug_trace* JSON-RPC contract" are the direction I expected from this PR. For golden tests, main already compares EDR's unconverted output against the Geth-format fixtures from #1251. I'd port those to Rust rather than swap them for HH2-format ones.

Could you share why you went with the HH2 patch approach? From my POV it doesn't fit the EDR roadmap, but I might be missing some context.

hardhat-tests contains tests that run with Hardhat 2. When we previously upgraded EDR's debug_* support, that release only landed in Hardhat 3 and the local patch for Hardhat 2 (and hardhat-tests). For HH3, it was acceptable as a breaking change.

However, as we are now releasing in Hardhat 2, we were explicitly asked to maintain backwards compatibility with debug_* in Hardhat 2. To do so, I added a backwards compatibility layer in Hardhat 2. That layer does not affect EDR in any way. EDR ships the same API to both HH2 & HH3. The only thing in EDR that is affected by that change is the hardhat-tests, since they use Hardhat 2.

Hardhat 2 itself no longer has any tests of the JSON-RPC layer. All of those live in EDR's hardhat-tests.

There were two options:

  1. revert hardhat-tests back to the old debug_* format to keep the tests (chosen)
  2. delete the debug_* tests from hardhat-tests and copy them to Hardhat 2.

I believe option 1 to be cleaner. The hardhat-tests have historically always tested Hardhat 2 behaviour. This is in line with that.

Moreover, option 2 has at least three downsides:

  • Changes to EDR can silently break the tests that were moved to Hardhat's v2 branch. It would only be noticed after we've already done a release. This is the main reason we moved hardhat-tests to EDR in the first place.
  • When we port hardhat-tests to future tests (which is on the roadmap), we'd have to remember that part of the tests live in Hardhat 2 again.
  • The git history will be lost/convoluted

@anaPerezGhiglia anaPerezGhiglia left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the context. I see the revert as the easiest way to get back the HH2-level coverage we had before #1251, so I won't block on it, though I'm not a fan.

As a follow-up, perhaps we could use the Geth-format tests that this PR replaces (they deep-compared EDR's unconverted output against the fixtures) as the reference for Rust tests of the JSON-RPC response, independent of the HH2 conversion layer. With this PR, a revm-inspectors/alloy change that adds a struct log field would pass both the HH2-format tests and the new Rust tests, so that would close the gap and be a first step in moving away from hardhat-tests. WDYT?
are th

@Wodann

Wodann commented Nov 17, 2026

Copy link
Copy Markdown
Member Author

As a follow-up, perhaps we could use the Geth-format tests that this PR replaces (they deep-compared EDR's unconverted output against the fixtures) as the reference for Rust tests of the JSON-RPC response, independent of the HH2 conversion layer. With this PR, a revm-inspectors/alloy change that adds a struct log field would pass both the HH2-format tests and the new Rust tests, so that would close the gap and be a first step in moving away from hardhat-tests. WDYT? are th

Thanks. I agree the revert is a stopgap and that the goal is to move away from hardhat-tests.

The Rust tests in this PR already test the raw JSON-RPC response. Both the spec and the EDR modules serialise GethTrace to JSON and assert on that, with no Hardhat conversion in the path. They derive expectations from the executed bytecode and the quoted spec clauses rather than from recorded fixtures.

You are right about the gap, though. An added struct log field would pass both suites today. The Hardhat 2 conversion rebuilds each struct log field by field, so an unknown field never reaches assertEqualTraces. On the Rust side, the field tests check that the required keys are present and that three named keys are absent by default, but no test rejects a key outside the spec's StructLog set.

I would rather close that with one Rust test that pins the key set of every step than port the Geth fixtures. I believe that the derived expectations are the stronger oracle.

Happy to add the key-set test in the follow-up.

This branch was previously deployed

1 inactive deployment
github-action-benchmark — 80d6eaec Deployed Nov 6, 2026 by Wodann via Run JS scenario runner benchmark for Hardhat Node style workload #6119
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no changeset needed This PR doesn't require a changeset

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants