Repository navigation
The feed's store read cannot fall off its index - #380
Merged
Merged
Conversation
Groundwork for #258, and worth having on its own. The store picks its index on the product of authors and kinds. At or under `max_merge_streams`, which is 4096, it opens a cursor per author-kind pair on `idx_author_kind` and every candidate it yields is one the caller asked for. Over that, it drops to the plain author index and pays a decode for every event those authors ever wrote, throwing most of them away. A full follow list is 2048 people plus the reader. One kind is 2049 pairs and fits. Two is 4098 and misses the cap by two. So the moment the feed reads a second kind, which is exactly what reposts require, a heavy follow list turns the feed rebuild into a full scan of everything two thousand people have ever posted. That cliff is invisible from here: nothing in this app knows where the store's threshold is, and the feed would keep working for most readers. Plaza's own routing note measures a real account at 257 follows, so three kinds is 774 pairs and fine. It breaks past about 1365 follows, and it breaks by getting slow rather than by getting wrong, which is the kind of thing nobody reports and nobody finds. So the read is one query per kind now, merged here. The product is always authors times one, whatever the kind list grows to: robust by construction rather than correct up to a follow count nobody is watching. It also holds fewer cursors open at once than the single query did, because the kinds are no longer multiplied together. The merge is a linear pick across `kinds.len` cursors, which is three. That is the same thing the store does internally, with three streams instead of six thousand, and events of one kind come back newest-first already. `eventNewer` orders them the way the store does: newest first, ties broken by id, so the sequence is total and does not depend on which cursor answered first. No behaviour change. The kind list is still `{1}`, so this reads one result and merges nothing, and all 647 tests pass unchanged. One thing kept deliberately: a failed read still abandons the whole rebuild rather than skipping that kind. A feed missing everything of one kind looks like a quiet day rather than a failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork for #258, and worth having on its own. No behaviour change: the kind list is still
{1}, and all 647 tests pass unchanged.The cliff
The store picks its index on the product of authors and kinds (
nostr/src/store.zig:481-487):At or under 4096 it opens a cursor per author-kind pair on
idx_author_kindand every candidate it yields was asked for. Over it, it drops to the plain author index and pays a decode for every event those authors ever wrote.A full follow list is 2048 plus the reader:
{1}idx_author_kind{1,6}It misses by two. So the moment the feed reads a second kind, which is exactly what #258 requires, a heavy follow list turns a feed rebuild into a scan of everything two thousand people have ever posted.
Why it would not have been caught
It breaks by getting slow, not by getting wrong, and only past roughly 1365 follows. Plaza own routing note measures a real account at 257 follows, which is 774 pairs across three kinds and perfectly fine. Nothing in this app knows where the store threshold is, so nothing here would have said a word.
The change
One query per kind, merged here, so the product is always authors times one whatever the kind list grows to. Robust by construction rather than correct up to a follow count nobody is watching. It also holds fewer cursors open at once, since the kinds are no longer multiplied together.
The merge is a linear pick across
kinds.lencursors, which is three, and each kind already comes back newest-first.eventNewerorders exactly as the store does, ties broken by id, so the sequence is total rather than dependent on which cursor answered first.Kept deliberately
A failed read still abandons the whole rebuild rather than skipping that kind. My first pass had it
continue, which would have shown a feed silently missing everything of one kind: that reads as a quiet day rather than as a failure.Next
The repost row itself: resolving the target from the store by its
etag, keying dedup on the reposted note rather than the wrapper, and taking counts from the target. All four reference clients agree on those three points.