debug_executionWitness: collapse witness root mismatch at chain tip

[canepat]

Running erigon at e2ab414 using this command line:

erigon \
--datadir=/opt/erigon-versions/version-main/datadir \
--http.api=eth,debug,net,web3,txpool,trace,parity,erigon,ots,admin \
--prune.mode=minimal

you query debug_executionWitness at the chain tip:

curl -X POST http://localhost:8545 -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"debug_executionWitness","params":["latest"],"id":1}'

and obtain:

{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"collapse witness root mismatch: calculated=38e64ea87f7e66262db0c0606441cd59b7959d7e4b8374335898274b48868982, expected=d2ec6a89d7ab8e9a6a24999a56476cc8b7aa9e0734674cdaf5c832bdd220bf42"}}

Log fragment:

[INFO] [04-16|12:51:48.807] head validated                           hash=0x25ea61b0707eb7508878ea7bb3ba5ddeedae3dafecc8c932777a012b150a3ff0 number=24892503 age=1s execution=154ms mgas/s=243.56 avg mgas/s=243.56 alloc=14.9GB sys=35.0GB
[INFO] [04-16|12:51:49.017] Timings: Forkchoice                      flush=106ms commit=97ms alloc=14.9GB sys=35.0GB
[INFO] [04-16|12:51:49.021] Timings: Background Prune                prune=2ms initialCycle=false alloc=14.9GB sys=35.0GB
[INFO] [04-16|12:51:56.204] Caplin P2P                               app=caplin peers_count=127
[INFO] [04-16|12:52:02.631] head validated                           hash=0x7ef77673f1d7ab9efa84844ff5d83587e3046a53d19c302b810baabf1383ff40 number=24892504 age=3s execution=72ms mgas/s=330.61 avg mgas/s=249.18 alloc=15.1GB sys=35.0GB
[INFO] [04-16|12:52:02.765] Timings: Forkchoice                      flush=68ms commit=61ms alloc=15.1GB sys=35.0GB
[INFO] [04-16|12:52:02.769] Timings: Background Prune                prune=2ms initialCycle=false alloc=15.1GB sys=35.0GB
[WARN] [04-16|12:52:11.813] [rpc] served                             conn=127.0.0.1:6334 method=debug_executionWitness reqid=1 err="collapse witness root mismatch: calculated=38e64ea87f7e66262db0c0606441cd59b7959d7e4b8374335898274b48868982, expected=d2ec6a89d7ab8e9a6a24999a56476cc8b7aa9e0734674cdaf5c832bdd220bf42"
[INFO] [04-16|12:52:12.991] head validated                           hash=0x77f298b45a4bf7c230e75f590ff0e0a26b54dd9989501b9c367ec73d12fa4c0a number=24892505 age=1s execution=111ms mgas/s=263.48 avg mgas/s=250.10 alloc=15.3GB sys=35.0GB
[INFO] [04-16|12:52:13.173] Timings: Forkchoice                      flush=93ms commit=83ms alloc=15.3GB sys=35.0GB
[INFO] [04-16|12:52:13.177] Timings: Background Prune                prune=1ms initialCycle=false alloc=15.3GB sys=35.0GB
[INFO] [04-16|12:52:25.001] head validated                           hash=0x9c4e3202759549a95758ef7bca0572953df56a0855e6bfb70f844225aa290d19 number=24892506 age=2s execution=97ms mgas/s=304.77 avg mgas/s=253.63 alloc=15.4GB sys=35.0GB

I've tried repeatedly on different head blocks, so it seems to happen consistently.

[VBulikov]

yperbasis added Imp1 as Importance

[lupin012]

I have created on rpc-tests (erigontech/rpc-tests#562):

• 9 rpc-tests on historical block saving Geth Response .
• 1 test (test_07) with latest block (to be execution on latests Qa workflow)

With commitment enabled this tests difference with Geth
https://github.com/erigontech/erigon/actions/runs/25961761055?pr=21224

[awskii]

confirmed yesterday and started refactoring for the implementation first, its too bloated

[lupin012]

@canepat , @awsky: the error is probably due to the commitment history not being enabled—(see the command line). If it is enabled, the response won't match Geth yet (there's a difference in states because Erigon has more MPT nodes), but this specific error does not occur.

[awskii]

i am closing this issue since @lupin012 fixed code reading through the cache in #21224

Follow up issue created #21312