Ideas for speeding up validator sync

[torkelrogstad]

1. Fetch blocks in JSON-RPC batches
   Preliminary findings are that the network latency isn't actually that big of a deal.
   Serial fetch (debug build): 8.8k blocks per minute
   Batched fetch (debug build): 10.5k blocks per minute
   Initial version of this is implemented in validator: fetch blocks with batched JSON-RPC during initial sync, reduce DB commits #236
2. Implement a pipeline of batch fetch -> deserialize -> process consisting of multiple threads
3. Fetch blocks by reading raw data from disk.
   Network calls does NOT seem to be a limiting factor here. Not sure this would yield anything.
4. Skip committing every block. Sync for example 1k blocks before comitting? Should we then avoid emitting sync events until after the commit? This is implemented in validator: fetch blocks with batched JSON-RPC during initial sync, reduce DB commits #236. Does a massive 10-20x speedup!

Not related to speeding up directly, but should also do:

• Performance measuring through log file analysis. Done in multi: sync analysis tools #235
• Implement/give instructions for how to instrument the binary. Use flamegraph? Done in validator: fetch blocks with batched JSON-RPC during initial sync, reduce DB commits #236

---

Notes

In general, it appears we are bottle-necked by disk. From my analysis it seems like up to 85% of time is spent waiting for disk on my machine (M2 macbook pro). Wins would probably be found in:

1. Batching DB operations where possible
2. Avoiding DB operations where possible (lots of stuff should be storeable in-memory, especially during initial sync)
3. Tuning DB commit behavior
4. Tuning LMDB sync behavior. Summary of these shown below.

Mode	fsync on Commit	Meta Safe	Crash Safe	Speed
Default	✅	✅	✅	🔴 Slow
MDB_NOSYNC	❌	❌	❌	🟢 Fast
MDB_NOMETASYNC	✅	❌	⚠️ Partial	🟡 Medium
MDB_MAPASYNC	❌	❌	❌	🟢 Fastest
MDB_WRITEMAP	–	–	⚠️ Risky	🟡 Fast