Very noticable slowdown on block import with --debug-store-bodies and --debug-store-receipts enabled

[kdeme]

Default --debug-store-bodies and --debug-store-receipts are disabled.

If we want a node that can provide all bodies and receipts (without using era/era1/ere files) these flags need to be enabled so that they get stored on import.

When running this import on a node for testing export of ere files, I noticed a significant slow down.

This is most likely due to the fact that each transaction and receipt is stored individually (k:v) in the database. Plus transactionHashToBlockKey additional mapping to support retrieval by hash. This is practically re-encodes the receipts and transactions and does 3N writes per block (tx + receipt + mapping).

It might be worth testing to store the transactions and receipts in two blobs. But this would require some benchmarking / testing.

Of course such change would have a (slight?) impact on the reading side for individual tx/receipt (eth_getTransactionReceipt, eth_getTransactionByHash). Not sure if that would be practically noticable though.

I noticed that withdrawals already went through such change: 40233e5#diff-75f9cc6813e733de906d9a8209ac17d76094e226f1616108803dc9496ec7464e

[kdeme]

I ran it for the first 3M Sepolia blocks (not really the best range to test this due to low amount of tx).

Range: blocks 0–3,000,000 (avg 2.0 tx/block)

Total elapsed

regular: default settings
debug_store: --debug-store-bodies and --debug-store-receipts enabled
chunk_32768: --debug-store-bodies and --debug-store-receipts enabled + --chunk-size set to 32768 (4x default setting)

Run	Elapsed	Overhead
regular	15m 08s	baseline
debug_store	17m 25s	+15.0%
chunk_32768	17m 32s	+16.0%

Increasing chunk size 8192 → 32768 made no meaningful difference.

Little zoomed in on part after 1M blocks:

Block range	avg tx/blk	regular bps	debug_store bps	slowdown
1M – 1.5M	1.5	11 010	10 673	3%
1.5M – 2M	4.0	3 343	2 696	19%
2M – 2.5M	3.1	3 156	2 566	19%
2.5M – 3M	3.3	587	446	24%

[kdeme]

Ran just 200k blocks starting from block 3M.

Range: blocks 3,000,000–3,200,000 (avg 14.4 tx/block)

Total elapsed

patched: Code patch that stores txs and receipts in 1 blob instead.

Run	Elapsed	vs regular	vs debug_store
regular	26m 23s	baseline	—
debug_store	28m 53s	+9.5%	baseline
patched	27m 36s	+4.6%	−4.5%

The blob storage patch recovers roughly half the overhead compared to the per-index layout: 9.5% to 4.6%

However, the actual difference of the debug_store run vs the regular run has gotten smaller while the amount of tx did increase a lot. Which is opposite of what I expected. This is probably because at this point the execution cost of the transactions dominates here making the time overhead appear smaller.

This does make it questionable if this slowdown is still an issue. Or at least should be tested / looked into for different block ranges and perhaps hardware.

I also noticed, when removing the writes for the transactionHashToBlockKey mapping that this did not have the same improvement as the changes of using 1 blob per block transactions / receipts.
This probably means that the biggest chuck of the overhead probably comes from the rlp encoding steps instead of the actual writes?