Ingest a batch of events in one transaction - #120
Merged
Merged
Conversation
Store.ingestBatch gives each event exactly the treatment ingest gives it, in order, inside one write transaction, and each event sees what the ones before it in the batch wrote. ingest opened and committed a transaction per event, and LMDB syncs on every commit, so a client storing events as they arrived paid a sync per event. ingest is now ingestBatch of one event, so there is one copy of the logic. The helpers it used (tombstone check, deletion, replaceable upsert) take the caller's transaction instead of opening their own. Closes #119.
Carries Store.ingestBatch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #119.
Store.ingestBatchisingestfor many events in one write transaction. Every event gets exactly the treatmentingestgives it, in order, and sees what the events before it in the batch wrote: a deletion earlier in a batch tombstones an event later in it, and a newer replaceable event supersedes an older one. One transaction is one commit, and LMDB syncs on every commit, so a batch costs one sync where one-at-a-time ingest cost one per event.ingestis nowingestBatchof one event, so the logic exists once. The tombstone check, deletion and replaceable upsert take the caller's transaction instead of opening their own; the tombstone check now reads inside the write transaction, which is what lets a batch see its own deletions.Tests
ingesttests pass unchanged, which is the check that its outcomes did not move.a batch sees what the events before it in the batch wrote: a deletion before its target, an older then newer then older profile, and a duplicate, in one batch.a batch gives every event the outcome ingest would, one at a time: the same mixed sequence (a foreign deletion, parameterized replaceables out of order, an ephemeral event, a repeat) through both paths, compared result by result and by event count.With the tombstone check removed, three tests fail, including the batch one. All 214 tests pass.
Release
The last commit bumps the version to 0.14.6.