lore: Apply date filters before the FTS candidate cap - #89
Merged
rikvanriel merged 1 commit intoSep 18, 2026
Merged
Conversation
A lore search that combines a broad pattern with since_date or until_date silently returns no results. The multi-field query helpers fetch FTS candidates under a fixed cap and apply the date range in memory afterward. FTS orders candidates by BM25 relevance, not by date, so once a pattern matches more mail than the cap the surviving candidates can fall anywhere in the archive. The date window then keeps none of them. A recipients pattern of "linux-nfs" alone overflows the cap; adding any since_date returns nothing. Build a date_timestamp predicate from the requested range and push it into each field query with only_if(), so the cap applies to date-filtered candidates. The same predicate bounds the empty-pattern table-scan fallback, which currently returns the oldest rows in table order whatever the date range. A stored date_timestamp of 0 means the timestamp is unknown: the indexer stores 0 when a Date header fails RFC 2822 parsing, and the schema migration backfills 0 into every row that predates the column. A plain range bound excludes every such row, dropping all pre-migration mail from a since_date search. Keep timestamp-0 rows in the predicate and settle each one by parsing its RFC 2822 date string. Each of the four lore query paths carries its own copy of the date parsing and predicate building. Move it into a shared LoreDateFilter so all four filter identically, and probe the schema for the date_timestamp column once per process rather than once per search. Signed-off-by: Chuck Lever <cel@kernel.org>
meta-cla
Bot
added
the
CLA Signed
Contributor
|
This should probably bump the database version, but that will happen soon enough with one of the changes in my queue. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A lore search that combines a broad pattern with since_date or until_date returns nothing once the pattern matches more mail than the FTS candidate cap. The candidates are ordered by relevance, not date, so the in-memory date filter applied afterward can discard every one of them. A recipients pattern of "linux-nfs" alone overflows the cap.
The date range now goes into each field query as a date_timestamp predicate, so the cap applies to date-filtered candidates. Rows whose stored timestamp is 0 (unparseable Date header, or backfilled by the schema migration) stay in the predicate and are settled by parsing their RFC 2822 date string, so pre-migration mail is not dropped from a since_date search.
🤖 Generated with Claude Code