Use real measured run; document machine and Atlas variance - #1
Open
usamaamjid wants to merge 1 commit into
Open
Conversation
Numbers now come from an actual run rather than a faster dev box. Drops the claim that a 10-value $in beat a 10,000-value one — that gap was run-to-run noise and reversed on slower hardware. The reproducible finding is that cost per document returned is flat (~8.5 us/doc here) regardless of array size. Also states explicitly that nothing here was run against Atlas. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The published numbers came from a fast dev box. These are from an actual run on the machine the post is written from.
What changed
Numbers are from a real run. Every figure in the README now matches what the benchmark actually printed.
Dropped a claim that didn't reproduce. The README said a 10-value
$inwas slower than a 10,000-value one. That gap was 3ms — run-to-run noise — and it reversed on slower hardware. The honest version: both return 10,000 docs and finish within 5% of each other, at ~8.5 µs/doc. Which one edges ahead is meaningless.Documented the variance that actually exists. 10–30% between runs on one machine, up to 2x between machines.
Said plainly that none of this was run against Atlas. On a hosted cluster every query pays a round trip, large result sets are batched over the wire, shared tiers throttle CPU, and a working set over RAM turns reads into page faults. Each makes documents returned cost more — which strengthens the finding rather than undermining it, but readers shouldn't have to infer that.
Invited falsification. If someone's
explain()output differs, that's a real finding and it should become an issue.What didn't change
The plan is
FETCH ← IXSCANat every size,keysExaminedis exactlyn + 1, and chunking loses. Those held on every machine tried.🤖 Generated with Claude Code