Repository navigation
test: pin the debug_trace* JSON-RPC contract - #1907
Conversation
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1907 +/- ##
==========================================
+ Coverage 81.53% 81.54% +0.01%
==========================================
Files 473 473
Lines 84080 84138 +58
Branches 84080 84138 +58
==========================================
+ Hits 68553 68614 +61
+ Misses 13235 13233 -2
+ Partials 2292 2291 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
03cd180 to
b55608a
Compare
b55608a to
b24cc2d
Compare
b24cc2d to
930280a
Compare
930280a to
3ac49e1
Compare
3ac49e1 to
b5b912e
Compare
b5b912e to
a2bb74d
Compare
a2bb74d to
b5707ff
Compare
112de34 to
4a4f721
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Storage reconstruction across reentrant calls and contract creation, plus new RPC lookups, requires native EDR/Hardhat integration validation of legacy-fixture parity and fork behavior that was unavailable during review.
No open issues 🎉
1 file not reviewed
pnpm-lock.yaml: Generated file
What changed in this PR
Pins EDR’s debug_traceTransaction and debug_traceCall JSON contract at the provider layer, while preserving coverage of Hardhat 2’s legacy trace format.
Changes:
- Adds 47 Rust tests covering specification requirements and EDR-specific behavior, using an Osaka-pinned fixture.
- Adds Hardhat 2 compatibility scenarios and updates existing trace comparisons.
- Removes replaced fixtures and updates the Hardhat patch hash.
| File | Description |
|---|---|
pnpm-lock.yaml |
Updates the Hardhat 2 patch hash. |
hardhat-tests/test/internal/hardhat-network/provider/utils/assertEqualTraces.ts |
Adapts comparisons to legacy trace fixtures. |
hardhat-tests/test/internal/hardhat-network/provider/modules/debug/traceTransaction.ts |
Updates transaction expectations and removes replaced cases. |
hardhat-tests/test/internal/hardhat-network/provider/modules/debug/traceCall.ts |
Updates call requests and removes replaced cases. |
hardhat-tests/test/internal/hardhat-network/provider/modules/debug/debug-trace.ts |
Adds legacy-format compatibility and option-handling tests. |
hardhat-tests/test/fixture-debug-traces/traceCall/ethTransfer.ts |
Removes the replaced transfer fixture. |
hardhat-tests/test/fixture-debug-traces/traceCall/contractAGetMessageDebugConfig.ts |
Removes the replaced tracing-options fixture. |
hardhat-tests/test/fixture-debug-traces/elongatedMemoryRegressionTestTrace.ts |
Removes the replaced memory regression fixture. |
crates/edr_provider/src/test_utils.rs |
Adds hardfork selection and reusable transaction construction. |
crates/edr_provider/src/requests/debug/tests/spec.rs |
Adds 34 specification contract tests. |
crates/edr_provider/src/requests/debug/tests/edr.rs |
Adds 13 EDR-specific behavior tests. |
crates/edr_provider/src/requests/debug/tests.rs |
Adds shared fixtures, bytecode builders, and JSON assertions. |
crates/edr_provider/src/requests/debug.rs |
Registers the debug test module. |
Review effort: Balanced
4a4f721 to
639a308
Compare
639a308 to
317983f
Compare
Commit f4bdc36 (#1251) switched the debug fixtures in hardhat-tests to geth's format and adjusted the debug tests and assertEqualTraces to match. A later commit makes the patched Hardhat 2 package return Hardhat 2's format again, so this reverts the hardhat-tests part of f4bdc36: the fixture files are the f4bdc36^ versions byte for byte, and the test and helper hunks are reversed on top of the current files. The stack-trace and package.json changes of f4bdc36 stay. The three fixtures that f4bdc36 gave a generation recipe keep it, plus a note on converting geth's 0x-prefixed output to Hardhat's format. The debug tests are red from this commit until the patch commit lands.
317983f to
790c55b
Compare
790c55b to
ddd22df
Compare
Add unit tests for the debug_traceTransaction and debug_traceCall handlers. They assert on the serialised GethTrace, which is the JSON that the provider returns. The reference is the Ethereum execution-apis spec at one pinned commit. The tests follow its opcode logger schema and the method clauses of trace.yaml. The tests are split into two modules. The spec module holds tests per spec requirement, each documented with the clause it pins. The edr module holds behaviour that the spec leaves undefined, and its module doc gives the reason EDR pins each one. The spec tests cover the struct log fields and their formats, and the meaning of pc, depth, gas and gasCost, including the cold SLOAD charge. Storage is pinned as a cumulative per-contract snapshot, which a DELEGATECALL shares with its caller. The refund counter is present on every step. After an SSTORE restores a slot's original value, the refund is the EIP-2200 restore refund. The trace's gas deducts the capped refund. Further tests cover the default stack, the memory, stack, storage and return data options and an empty tracer. The step limit is pinned to cap the recorded steps without stopping execution. Other tests cover the trace results of transfers, reverts, successful calls and a halting precompile call. debug_traceTransaction is pinned to replay the preceding transactions of its block. The edr tests cover the Unknown opcode name and the halt error text. disableMemory and disableReturnData are not spec options, but alloy reads them. disableMemory: false enables memory when enableMemory is absent, and enableMemory takes precedence when both are set. disableReturnData: false likewise enables return data. The edr tests also cover the callTracer result and the errors for an unknown tracer or transaction hash. For debug_traceCall they pin the default block, contract creation and state isolation. Hardhat 2 converts these traces back to its own format. That conversion depends on this contract, so a change in revm-inspectors, alloy or EDR's option handling now fails here first. A test_utils helper now builds transaction requests, and another builds a local fixture on a given hardfork.
Hardhat 2 converts EDR's geth-format debug traces back to its legacy format in the open NomicFoundation/hardhat#7985. EDR's patched Hardhat 2.28.6 lacked that conversion, so EDR CI tested a format that Hardhat 2 no longer returns. Port the conversion from #7985. The port adds edrRpcDebugTraceToHardhat2 and its StorageTracker to convertToEdr.ts. It wires the memory default, the empty tracer handling and the result conversion into the provider's request method. RpcStructLog gains the returnData field. The debug trace code matches #7985. Outside it, the port keeps EDR's existing patch and its formatting. The compiled files come from TypeScript 5.0.4 with the package's own 2.28.6 tsconfig. That compiler reproduces the patched provider.js byte for byte. The source maps are now regenerated with the code they map. The revert commit restored the Hardhat 2 format fixtures and reversed the test hunks of #1251, so the debug tests are green again from here. assertEqualTraces loses the refund normalisation that only geth's format needed and the zeroing of call-opcode gas, which EDR now reports as the fixtures do. It compares returnValue without its prefix, except for a failed trace whose fixture recorded none. The unsupported tracer tests expect EDR's rejection of such a tracer as a JavaScript tracer, and debug_traceTransaction checks the hash first.
|
Disclaimer: given the size of the diff, I didn't review this PR line by line. I asked Claude to walk me through it, so sorry in advance if I'm missing something that's in the code. I see the motivation in the commit message: once NomicFoundation/hardhat#7985 lands, HH2 converts EDR's traces back to its legacy format. But as agreed in the EDR roadmap, we're moving away from The Rust tests in "test: pin the debug_trace* JSON-RPC contract" are the direction I expected from this PR. For golden tests, Could you share why you went with the HH2 patch approach? From my POV it doesn't fit the EDR roadmap, but I might be missing some context. |
As explained in the notes for reviewer, it's easier to review the link I shared. It excludes the largest commit which is a mechanical revert.
However, as we are now releasing in Hardhat 2, we were explicitly asked to maintain backwards compatibility with Hardhat 2 itself no longer has any tests of the JSON-RPC layer. All of those live in EDR's There were two options:
I believe option 1 to be cleaner. The Moreover, option 2 has at least three downsides:
|
anaPerezGhiglia
left a comment
There was a problem hiding this comment.
Thanks for the context. I see the revert as the easiest way to get back the HH2-level coverage we had before #1251, so I won't block on it, though I'm not a fan.
As a follow-up, perhaps we could use the Geth-format tests that this PR replaces (they deep-compared EDR's unconverted output against the fixtures) as the reference for Rust tests of the JSON-RPC response, independent of the HH2 conversion layer. With this PR, a revm-inspectors/alloy change that adds a struct log field would pass both the HH2-format tests and the new Rust tests, so that would close the gap and be a first step in moving away from hardhat-tests. WDYT?
are th
Thanks. I agree the revert is a stopgap and that the goal is to move away from The Rust tests in this PR already test the raw JSON-RPC response. Both the spec and the EDR modules serialise You are right about the gap, though. An added struct log field would pass both suites today. The Hardhat 2 conversion rebuilds each struct log field by field, so an unknown field never reaches I would rather close that with one Rust test that pins the key set of every step than port the Geth fixtures. I believe that the derived expectations are the stronger oracle. Happy to add the key-set test in the follow-up. |
EDR had no test of the JSON that
debug_traceTransactionanddebug_traceCallreturn. The only checks lived inhardhat-tests, through the Hardhat 2 package, so a change in revm-inspectors, alloy or EDR's option handling would surface downstream first. Hardhat 2 converts these traces back to its legacy format in the open NomicFoundation/hardhat#7985, and that conversion depends on the exact output.This PR pins the contract in EDR with 47 unit tests, against the Ethereum execution-apis spec at a pinned commit. It also brings
hardhat-testsback to the format Hardhat 2 will ship, so CI guards it from both sides.What changes
crates/edr_provider/src/requests/debug/tests/: each test runs a transaction or call on the in-process fixture, calls the handler and asserts on the serialised JSON.spec.rsholds 34 tests, one per spec requirement, each documented with the clause it pins.edr.rsholds 13 tests of behaviour the spec leaves undefined, with the reason for each in its module doc. The fixture pins Osaka, because Amsterdam changes the gas constants.hardhat-tests: a revert of feat: add support for Geth built-in tracers #1251's fixture changes restores Hardhat 2's trace format. The Hardhat 2 patch gains #7985's conversion, with its compiled files regenerated. A follow-up PR adds the tracker-heavy scenarios from #7985 and drops the cases the Rust tests now cover.test_utilsgainsnew_local_with_hardfork,transaction_requestandTEST_GAS_PRICE.Validation
opcode-tracer.yamlandtrace.yamlat commit3284ee80c.Notes for review
disableMemory: falseenables memory whenenableMemoryis absent since revm-inspectors 0.44, where EDR 0.25.0 ignored it. The option is not in the spec, so the test pins current behaviour.Some(OutOfGas)error text is revm-inspectors'Debugformatting and still present in 0.44.3, so the fix is upstream. The test is a tripwire.no changeset needed. A pre-Spurious-Dragon test was not possible, becauseL1Hardforkstarts at Byzantium.