Skip to content

The feed's store read cannot fall off its index - #380

Merged
sepehr-safari merged 1 commit into
mainfrom
a-repost-by-a-follow-reaches-the-feed
Sep 22, 2026
Merged

sepehr-safari merged 1 commit into
mainfrom
a-repost-by-a-follow-reaches-the-feed

Conversation

@sepehr-safari

Copy link
Copy Markdown
Contributor

Groundwork for #258, and worth having on its own. No behaviour change: the kind list is still {1}, and all 647 tests pass unchanged.

The cliff

The store picks its index on the product of authors and kinds (nostr/src/store.zig:481-487):

const pairs = std.math.mul(usize, authors.len, kinds.len) catch break :blk null;
if (pairs > max_merge_streams) break :blk null;

At or under 4096 it opens a cursor per author-kind pair on idx_author_kind and every candidate it yields was asked for. Over it, it drops to the plain author index and pays a decode for every event those authors ever wrote.

A full follow list is 2048 plus the reader:

kinds pairs index
{1} 2049 idx_author_kind
{1,6} 4098 falls back, full scan

It misses by two. So the moment the feed reads a second kind, which is exactly what #258 requires, a heavy follow list turns a feed rebuild into a scan of everything two thousand people have ever posted.

Why it would not have been caught

It breaks by getting slow, not by getting wrong, and only past roughly 1365 follows. Plaza own routing note measures a real account at 257 follows, which is 774 pairs across three kinds and perfectly fine. Nothing in this app knows where the store threshold is, so nothing here would have said a word.

The change

One query per kind, merged here, so the product is always authors times one whatever the kind list grows to. Robust by construction rather than correct up to a follow count nobody is watching. It also holds fewer cursors open at once, since the kinds are no longer multiplied together.

The merge is a linear pick across kinds.len cursors, which is three, and each kind already comes back newest-first. eventNewer orders exactly as the store does, ties broken by id, so the sequence is total rather than dependent on which cursor answered first.

Kept deliberately

A failed read still abandons the whole rebuild rather than skipping that kind. My first pass had it continue, which would have shown a feed silently missing everything of one kind: that reads as a quiet day rather than as a failure.

Next

The repost row itself: resolving the target from the store by its e tag, keying dedup on the reposted note rather than the wrapper, and taking counts from the target. All four reference clients agree on those three points.

Groundwork for #258, and worth having on its own.

The store picks its index on the product of authors and kinds. At or under `max_merge_streams`, which is 4096, it opens a cursor per author-kind pair on `idx_author_kind` and every candidate it yields is one the caller asked for. Over that, it drops to the plain author index and pays a decode for every event those authors ever wrote, throwing most of them away.

A full follow list is 2048 people plus the reader. One kind is 2049 pairs and fits. Two is 4098 and misses the cap by two. So the moment the feed reads a second kind, which is exactly what reposts require, a heavy follow list turns the feed rebuild into a full scan of everything two thousand people have ever posted.

That cliff is invisible from here: nothing in this app knows where the store's threshold is, and the feed would keep working for most readers. Plaza's own routing note measures a real account at 257 follows, so three kinds is 774 pairs and fine. It breaks past about 1365 follows, and it breaks by getting slow rather than by getting wrong, which is the kind of thing nobody reports and nobody finds.

So the read is one query per kind now, merged here. The product is always authors times one, whatever the kind list grows to: robust by construction rather than correct up to a follow count nobody is watching. It also holds fewer cursors open at once than the single query did, because the kinds are no longer multiplied together.

The merge is a linear pick across `kinds.len` cursors, which is three. That is the same thing the store does internally, with three streams instead of six thousand, and events of one kind come back newest-first already. `eventNewer` orders them the way the store does: newest first, ties broken by id, so the sequence is total and does not depend on which cursor answered first.

No behaviour change. The kind list is still `{1}`, so this reads one result and merges nothing, and all 647 tests pass unchanged.

One thing kept deliberately: a failed read still abandons the whole rebuild rather than skipping that kind. A feed missing everything of one kind looks like a quiet day rather than a failure.
@sepehr-safari
sepehr-safari merged commit 18422d4 into main Sep 22, 2026
6 checks passed
@sepehr-safari
sepehr-safari deleted the a-repost-by-a-follow-reaches-the-feed branch September 22, 2026 13:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant