Skip to content

feat(find): make in-book find ignore punctuation - #770

Open
WildHagai wants to merge 1 commit into
kdroidFilter:masterfrom
WildHagai:feat/find-ignore-punctuation
Open

WildHagai wants to merge 1 commit into
kdroidFilter:masterfrom
WildHagai:feat/find-ignore-punctuation

Conversation

@WildHagai

Copy link
Copy Markdown

What

Find-in-book now ignores punctuation. אמר רבי matches אמר, רבי and אמר: רבי, and punctuation typed in the query (אמר, רבי!) no longer blocks a match. The text is scanned through a new stripForFindWithMap, which keeps a map from stripped to original index, so highlights land on the right characters. Niqqud/te'amim insensitivity and final-letter folding are unchanged. normalizeQueryForHebrew now normalizes the query exactly the way the text is normalized.

Edge-case decisions

  • Punctuation counts as a word break, not as nothing: any run of punctuation and/or whitespace reads as a single space. So שלום,עולם (no space after the comma) still matches שלום עולם. Punctuation means ASCII non-alphanumerics plus Unicode category P* (sof pasuq ׃, paseq ׀, dashes, curly quotes, …).
  • Leading and trailing punctuation is dropped: "שלום" and אור׃ search for the bare word, and a highlight never starts or ends on punctuation.
  • A query made only of punctuation normalizes to "" and finds nothing.
  • Maqaf ־ is a word break on both sides, so כל־הארץ and כל הארץ match each other. Before this, a query typed with a maqaf could never match.
  • Geresh/gershayim: a quote mark between two letters (", ', ׳, ״, curly quotes and primes) is dropped rather than turned into a space. So רשי, רש"י, רש״י and רש''י all match each other, and ס'/ס׳ match ס. A quote that opens or closes a quoted word (ואמר "שלום") is still a word break.
  • The highlight now also covers the niqqud of the last matched letter.
  • The TOC search shares this query normalization; it behaves the same and its tests pass.
  • Known trade-off: a quote stuck between two words with no space (אמר"שלום") joins them into one word. This is rare in real texts.

Tests

New cases in HebrewSearchTest (index mapping, commas, punctuation only in the query, acronyms in both directions, geresh, maqaf, sof pasuq, pointed text, highlight bounds) and in BookFindMatchesTest (counting matches and query narrowing regardless of punctuation).

  • HebrewSearchTest 44/44, BookFindMatchesTest 8/8, TocSearchTest 9/9
  • ktlintCheck clean

🤖 Generated with Claude Code

"אמר רבי" now matches "אמר, רבי" / "אמר: רבי", and punctuation typed in
the query no longer blocks a match. The text is scanned through a new
stripForFindWithMap that keeps an index map, so highlights still land on
the original offsets; nikud/te'amim insensitivity is unchanged.

- Any run of punctuation and/or whitespace reads as one space; leading and
  trailing runs are dropped, so a highlight never starts or ends on one.
- Maqaf (U+05BE) is a word break in both text and query (a query typed
  with a maqaf used to never match).
- Quote marks between two letters (" ' ׳ ״ and curly/prime look-alikes)
  are dropped instead, so רשי, רש"י and רש״י all match one another.
  Quotes opening/closing a quoted word stay word breaks.
- A punctuation-only query normalizes to "" and finds nothing.
- The highlight now also covers the points of the last matched letter.

Signed-off-by: Chagai Yisroel Wild <wildhagai@puzzlesoft.co.il>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants