Support Dencun consensus upgrade

[yrong]

Resolves: SNO-406, SNO-775

• Upgrade geth and lodestar to recent release
• Add VersionedExecutionPayloadHeader to support multiple versions of the beacon updates
• Cleanup for unused codes/packages(e.g. package ethashproof based on the deprecated POW consensus which can be removed).
• Test e2e work as expected
• Test the switchover process

[vgeddes]

Nice work Ron! 🙏

[yrong]

@vgeddes I would prefer to split https://linear.app/snowfork/issue/SNO-406 into multiple PRs, in this one we only focus on upgrading packages and add versioned data support in next one.

[vgeddes]

I've just noticed that there are a lot of unrelated changes to the E2E scripts, like making processes run in parallel etc. I vaguely recall previously reviewing such a change and saying that it makes troubleshooting harder if the E2E tests break.

The other reason for not wanting such changes is that roughly 80% of the initialization time is spent in compiling rust code. And the parallelization changes you're introducing here don't improve that. So I think the net gain is minimal.

So I'd like for you to please focus this PR on only the code relevant to the Dencun upgrades.

[yrong]

There are a lot of unrelated changes to the E2E scripts, like making processes run in parallel etc. The other reason for not wanting such changes is that roughly 80% of the initialization time is spent in compiling rust code. And the parallelization changes you're introducing here don't improve that.

Yes, rust compile is the most time consuming and the gain is minimal in this case. But for the second round of start up when there is no rust code change(e.g. cases when there is only change for relayer or solidity code) parallelization will decrease startup time to almost half(from 2 minutes 18 seconds to 1 minture 20 seconds in my laptop) so IMHO it's benefit.

But anyway just remove unrelated changes in 28f23dd to be more focused.

[codecov]

Codecov Report

Attention: 44 lines in your changes are missing coverage. Please review.

Comparison is base (4e24732) 76.25% compared to head (1460d55) 75.08%.

❗ Current head 1460d55 differs from pull request most recent head 8e392c3. Consider uploading reports for the commit 8e392c3 to get more accurate results

Files	Patch %	Lines
parachain/primitives/beacon/src/ssz.rs	0.00%	30 Missing ⚠️
parachain/primitives/beacon/src/types.rs	55.17%	13 Missing ⚠️
parachain/pallets/ethereum-client/src/lib.rs	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #981      +/-   ##
==========================================
- Coverage   76.25%   75.08%   -1.17%     
==========================================
  Files          58       58              
  Lines        2400     2464      +64     
  Branches       72       72              
==========================================
+ Hits         1830     1850      +20     
- Misses        553      597      +44     
  Partials       17       17              

Flag	Coverage Δ
rust	74.00% <33.33%> (-1.37%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[vgeddes]

+1.

[claravanstaden]

Hey @yrong! Great job, this is awesome!

I made a few comments. I would have preferred the relayer cleanup (removing the Ethereum PoW code) in a separate PR, but what's done is done.

I am not convinced adding geth and lodestar as submodules is necessary. Did we make any changes at all that warrants using forked versions? Seems like unnecessary maintenance and I would prefer we use the original projects.

[vgeddes]

I think its simpler to hardcode the source filenames based on the output of:

nix eval --impure --raw --expr 'builtins.currentSystem'

also, we need to support arm/macos too, not just linux/amd64

[yrong]

It seems the download page only provides the amd64 versions, and I tested the darwin-amd64 package did work in my M1.

[vgeddes]

I think you need to look that download page a bit more - there are links to downloads for many architectures:

So depending on the value of builtins.currentSystem, which should be one of the architectures listed in flake.nix, download the appropriate binary.

[yrong]

Yeah, I missed that. Fixed in fb5366f

Btw: nix eval --impure --raw --expr 'builtins.currentSystem' return aarch64-darwin in my laptop which does not match the darwin-arm64* in the download link so switch to uname -m instead.