Skip to content

fix(store): keep combining marks inside FTS5 tokens - #967

Open
costajohnt wants to merge 1 commit into
tobi:mainfrom
costajohnt:fix/966-fts-separator-combining-marks
Open

costajohnt wants to merge 1 commit into
tobi:mainfrom
costajohnt:fix/966-fts-separator-combining-marks

Conversation

@costajohnt

Copy link
Copy Markdown

Fixes #966

FTS5_SEPARATOR_RUN in src/store.ts splits query terms on anything that is not a letter, number, apostrophe or underscore. Combining marks (\p{M}) fall outside that class, so a decomposed (NFD) word such as impôt (o + U+0302) was split into impo and t, the query became "impo t", and FTS5 returned nothing. This started with #939.

The change adds \p{M} to the character class, so marks stay inside the token; sanitizeFTS5Term then drops the mark as before, and precomposed input is unaffected.

Tests: new cases in test/store-fts-separator-queries.test.ts for NFD impôt and café staying one token and matching; they fail on main. 23 pass under both bun test and vitest, bun run lint clean. CHANGELOG entry under Unreleased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FTS5_SEPARATOR_RUN splits inside a word on NFD input: combining marks are treated as separators (regression from #939, main only)

1 participant