Skip literal prefix index hits inside surrogate pairs (fixes #207) - #209
Closed
jdymitarai wants to merge 1 commit into
Closed
jdymitarai wants to merge 1 commit into
jdymitarai wants to merge 1 commit into
Conversation
When a pattern's literal prefix begins with a lone low surrogate, MachineInput.UTF16Input.index() previously jumped to raw indexOf hits. If the hit is preceded by a high surrogate, it is inside a valid surrogate pair rather than a codepoint boundary. In step(), decoding starting from inside the pair caused it to treat the low surrogate as an isolated rune and erroneously match. Patterns without literal prefixes step codepoint by codepoint and correctly skip such pairs. This commit updates UTF16Input.index() to detect when an indexOf hit is a low surrogate preceded by a high surrogate, skipping it and resuming search at the next character.
Author
|
Closing this PR to avoid review overhead on the team. The context remains in the thread for future reference if helpful. Thanks for your time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #207
Problem
When a pattern contains a literal prefix whose first character is a lone low surrogate (e.g.
\uDC21),MachineInput.UTF16Input.index()previously usedindexOf(str, re2.prefix, pos)to jump directly to matches.When
strcontains a well-formed surrogate pair (e.g.\uD801\uDC21),indexOfmatches the low surrogate\uDC21inside the surrogate pair.When
Machine.javaadvancesposto this hit,step(pos)decodes the low surrogate in isolation as rune0xDC21and matches. In contrast, patterns without a literal prefix (such as\uDC21|\uDC22or[\uD800-\uDFFF]) step codepoint-by-codepoint, decoding the pair as the composite code pointU+10421and stepping past it without matching.Solution
In
MachineInput.UTF16Input.index(), verify whether anindexOfhit is a low surrogate preceded by a high surrogate (i > 0 && Character.isLowSurrogate(str.charAt(i)) && Character.isHighSurrogate(str.charAt(i - 1))). If so, the hit is inside a surrogate pair and is not a valid codepoint boundary; skip it and resume searching ati + 1.Verification
testSurrogatePairInteriorNoMatchinjavatests/com/google/re2j/MatcherTest.javaverifying that lone low surrogate patterns do not match inside surrogate pairs, that search resume finds isolated low surrogates, and thatStringBuilderbehaves identically.