Fix chunk fetch loading the entire index into memory#567
Merged
tobocop2 merged 2 commits intoJul 20, 2026
Conversation
get_chunks_by_source and get_chunks_by_indices rescued a failed filtered query by calling table.to_arrow(), which materializes every chunk of every document -- embedding vectors included -- before filtering in Python. On a large library that is a multi-gigabyte spike, and neighbor expansion hits it once per hit source per query. The fallback guarded a LanceDB version whose FTS-indexed tables rejected .where() on arbitrary columns. That no longer reproduces: a filtered empty-search query returns the right rows on an FTS-indexed table, so the path was dead code held alive by tests that mocked search() into raising. Both functions now let a query failure propagate. An exception names the problem; an OOM kill from the rescue scan leaves nothing to debug. The two mocked-fallback tests are replaced by tests that build the FTS index for real and assert the filtered query still selects rows, so a future LanceDB bump that breaks the predicate fails CI instead of failing in production.
tobocop2
changed the title
The test patched pathlib.Path.expanduser globally; the config validator also expands the data root on every field assignment, so the cfg-restoring fixture raised through the patch at teardown and turned the whole suite red. Patch only the autocomplete module's Path instead. The breakage predates this branch (it came with the data-root expanduser fix on the base) but blocks this PR's CI, so carry the one-test fix here too.
tobocop2
marked this pull request as ready for review
July 20, 2026 15:45
tobocop2
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
get_chunks_by_sourceandget_chunks_by_indicesnormally filter in the database. If that query raised, both fell back totable.to_arrow(), which loads every chunk of every document into memory, embedding vectors included, then filters in Python. Multi-gigabyte spike on a large library, and neighbor expansion hits it once per hit source per query.Closes #565.
Solution
The fallback guarded old LanceDB behavior where FTS-indexed tables rejected
.where(). That no longer reproduces on the pinned version, so it was dead code kept alive by tests that mockedsearch()into raising.Both functions now let a query failure propagate. An exception names the problem; the rescue scan just gets the process OOM-killed. The mocked tests are replaced by ones that build the FTS index for real, so a future LanceDB bump that breaks the query fails CI instead of silently reading everything.