Retesteth usage in a private network

[Infra1515]

Hello,
I am trying to run retesteth against a custom client based on pre-merge geth v1.11.* that is utilising PoW consensus inside a private network.
I am writing because I feel that are some knowledge gaps of how to use it and hoping to get some help. Excuse me in advance if this is not the correct place.
Just for reference, I am running on the code on arm (M1 Pro).

Here are the steps that I am doing:

1. Since I experienced various problem with docker the retest code is compiled locally using the instructions from https://github.com/ethereum/retesteth/blob/develop/circle.yml.
2. Downloaded the test repo - https://github.com/ethereum/tests.
3. I copied the evm binary from myCustomClient/build/bin/evm to $GOPATH/evm and am able to successfully run tests with the following command
4. Started my node on localhost:8545
5. Execute the test with the following command
   ./retesteth/build/retesteth/retesteth -t GeneralStateTests -- --testpath /tests --nodes 0.0.0.0:8545

So far so good, the tests run and 99% of them pass. What I am having trouble understanding is the following:

• Even if I stop my local node that is running on 0.0.0.0:8545 the tests still pass - does this mean that the tests are only executing against the evm binary and not against the client itself or perhaps there is a specific test suite that targets the functionality of the client ? Because of this I am not sure if I am doing the testing process correctly at all
• Configuration and forks: in ~/.retesteth I created a copy of the default config and changed

 "socketType" : "tcp",
 "socketAddress" : ["0.0.0.0:8545"]..

But I see there are also genesis files for specific forks - since I am using a private network starting from block 0 and a custom genesis - how can I specify that and be sure that I am executing the tests correctly?
I tried adding another genesis.json file in the config folder and also another supported fork in the config file under the forks and running the tests with

retesteth/build/retesteth/retesteth -t BlockchainTests -- \
   --testpath $PWD/tests  \
   --nodes 0.0.0.0:8545 --clients myClient --singlenet myClientFork

but here the tests just run and in the end it says 0 had been run.
Full log here:

Retesteth config path: /Users/.retesteth
Active client configurations: 'myClient '
WARNING: ClientConfig (/Users/.retesteth/myClient/config) tmpDir location not found! Switching to default.
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "eips": (1 of 39)
33%...
66%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "example": (2 of 39)
50%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "vm": (3 of 39)
50%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "withdrawals": (4 of 39)
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcBlockGasLimitTest": (5 of 39)
20%...
40%...
60%...
80%...
100%
Skipping bcExploitTest because --all option is not specified.
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcForkStressTest": (7 of 39)
50%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcGasPricerTest": (8 of 39)
33%...
66%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcMultiChainTest": (9 of 39)
14%...
28%...
42%...
57%...
71%...
85%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcRandomBlockhashTest": (10 of 39)
25%...
50%...
75%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcStateTests": (11 of 39)
25%...
50%...
75%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcTotalDifficultyTest": (12 of 39)
18%...
36%...
54%...
72%...
90%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcUncleSpecialTests": (13 of 39)
25%...
50%...
75%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcUncleTest": (14 of 39)
25%...
50%...
75%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcValidBlockTest": (15 of 39)
21%...
42%...
63%...
84%...
100%
Skipping bcWalletTest because --all option is not specified.
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcExample": (17 of 39)
25%...
50%...
75%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcEIP1559": (18 of 39)
23%...
47%...
70%...
94%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcEIP3675": (19 of 39)
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcBlockGasLimitTest": (20 of 39)
100%
Skipping bcForgedTest because bigint exceptions run in progress!
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcInvalidHeaderTest": (22 of 39)
22%...
45%...
68%...
90%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcMultiChainTest": (23 of 39)
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcUncleHeaderValidity": (24 of 39)
24%...
48%...
72%...
96%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcUncleSpecialTests": (25 of 39)
22%...
44%...
66%...
88%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcUncleTest": (26 of 39)
21%...
43%...
65%...
86%...
100%
Filter: ' myClient'
WARNING: /Users/.myClient/end-to-end/tests/src/BlockchainTestsFiller/InvalidBlocks/bcExample does not exist!
WARNING: bcExample no tests detected in folder!
Running tests for config 'Ethereum GO on StateTool' 2
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcEIP1559": (28 of 39)
25%...
50%...
75%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcEIP3675": (29 of 39)
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bc4895-withdrawals": (30 of 39)
22%...
44%...
66%...
88%...
100%
Skipping bcExpectSection because --all option is not specified.
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcByzantiumToConstantinopleFix": (32 of 39)
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcEIP158ToByzantium": (33 of 39)
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcFrontierToHomestead": (34 of 39)
14%...
28%...
42%...
57%...
71%...
85%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcHomesteadToDao": (35 of 39)
25%...
50%...
75%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcHomesteadToEIP150": (36 of 39)
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcBerlinToLondon": (37 of 39)
33%...
66%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcArrowGlacierToMerge": (38 of 39)
33%...
66%...
100%
Filter: ' myClient'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "bcMergeToShanghai": (39 of 39)
100%

*** No errors detected
WARNING: Test folder /Users/.myClient/end-to-end/tests/src/BlockchainTestsFiller/InvalidBlocks/bcForgedTest appears to be unused!
*** Total Tests Run: 0

Not sure If I am using retesteth in the correct manner for testing a private network. Any help or materials/guidance will be appreciated.
Thank you in advance :)

[giskook]

Can retesteth run test through RPC?

[winsvega]

sorry for the late response. message me on @wdimitry telegram I help with the setup.

yes, test RPC are still supported. but if you use fork of geth you must use t8n. if you develop your own client you can develop RPC if this is preferable for you. RPC is not supported by pyspecs so you wont be able to generate .py tests. (but able to run it)

to use nodes the config must be TCP socket style. not t8n.
when you create custom config you must run it with --clients customconfig to use it. otherwise it will use default config
all subfolders in .retesteth/default folder are used by any other configs unless explicitly defined
so to use custom genesis .json you need to define it in .retesteth/customconfig/genesis subfolder.

[zjg555543]

Any

sorry for the late response. message me on @wdimitry telegram I help with the setup.

yes, test RPC are still supported. but if you use fork of geth you must use t8n. if you develop your own client you can develop RPC if this is preferable for you. RPC is not supported by pyspecs so you wont be able to generate .py tests. (but able to run it)

to use nodes the config must be TCP socket style. not t8n. when you create custom config you must run it with --clients customconfig to use it. otherwise it will use default config all subfolders in .retesteth/default folder are used by any other configs unless explicitly defined so to use custom genesis .json you need to define it in .retesteth/customconfig/genesis subfolder.

Hey guy. Thanks for your sharing. I got the similar issue by runing on X Layer private network. Could you help to check it?

/usr/bin/retesteth -t GeneralStateTests/stSystemOperationsTest -- --datadir /tests/config --clients defaultout --testpath /tests 
Running tests using path: /tests
Running 1 test case...
Retesteth config path: /tests/config
Active client configurations: 'defaultout '
Running tests for config 'Ethereum GO on TCP' 2
Test Case "stSystemOperationsTest": (1 of 1)
Instantiated: "result":"cdk-erigon/2.0-upstream/v2.0.0-beta18-bde1b5ecf931c9b367a7ebc0b1d60c0830f83493/linux-amd64/go1.20.10"
WARNING: , Message: The command '/tests/config/defaultout/start.sh 1 2>/dev/null' exited with 512 code., has empty debugInfo! Missing debug Testinfo for test step.
terminate called after throwing an instance of 'test::EthError'
  what():  The command '/tests/config/defaultout/start.sh 1 2>/dev/null' exited with 512 code.
Finishing retesteth run

*** 1 failure is detected in the test module "EthereumTests"
unknown location(0): fatal error: in "GeneralStateTests/stSystemOperationsTest": signal: SIGSEGV, si_code: 128 (memory access violation at address: 0x00000000)
/retesteth/retesteth/testSuites/statetests/StateTestsBoost.cpp(102): last checkpoint: "stSystemOperationsTest" fixture ctor
*** Total Tests Run: 1

*** stack smashing detected ***: terminated