Faster store writes, a faster allocator, smaller binaries, and benchmarks - #24
Merged
Merged
Conversation
…release binaries Events kept in a store were written one transaction each, and LMDB syncs on every commit, so storing ran at about 2,300 events a second. They are now held until 512 have passed their checks, a relay has nothing more ready, or the run ends, and written in one transaction with the library's new Store.ingestBatch, then printed: still stored before they are shown. Storing 100,000 events from a local relay now runs at about 16,000 a second. The allocator was page_allocator, which maps pages for every allocation. It is now smp_allocator. Decoding 10,000 npubs went from 135 ms to 17 ms; signing and verifying are about 15 percent faster. Release binaries are built with the new -Dstrip option. The Linux ones carried about 9 MB of debug sections and go from 12 MB to under 3 MB. The nostr pin moves to 0.14.6 for ingestBatch. A test serves 1,200 events through a local relay, two full batches and part of a third, and checks every one is stored and printed; without the final flushes it fails. Closes #23.
BENCHMARKS.md gives what deed costs to run: binary and download sizes, startup, memory, stream throughput, storing and querying 100,000 events, and relays over loopback. Every number comes from bench/run.py, which runs the same measurements against any build, with a local relay (bench/relay.py behind websocat) so nothing leaves the machine. The README gains a short section pointing to it.
Faster store writes, a faster allocator, stripped release binaries, and benchmarks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #23.
What changes
--storeare held until 512 have passed their checks, a relay has nothing more ready, or the run ends, then written in one transaction with the library's newStore.ingestBatch(nostr 0.14.6) and printed: still stored before they are shown. One transaction per event meant one disk sync per event.page_allocator, which mapped pages for every allocation.-Dstripbuild option that the release workflow passes. The Linux binaries carried about 9 MB of debug sections.Measured
Same machine, same 100,000-event corpus, before and after:
The store file for the 100,000 events grew from 86 MB to between 94 and 102 MB across my runs: LMDB keeps more free pages after larger transactions.
Tests
events kept in a store are written in batches, and every one is stored and printedserves 1,200 events through a local relay, two full batches and part of a third, and checks every one reached the store and stdout. Without the final flushes it fails. All 94 tests pass.Release
The last commit bumps the version to 0.3.1.