You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
.replace("’", "’") # U+2019 → U+2019, no-op
.replace("‘", "’") # U+2018 → U+2019 (curly-open → curly-close, still not ASCII)
ASCII apostrophe (U+0027) is never produced. Bones-map keys for contraction fragments (n't, 'll, 'd, 're, 'm, 've, 's) are written with ASCII '. Input containing smart apostrophes (U+2019 in "don't") survives unchanged, looks up against ASCII-keyed bone entries, and misses — producing incorrect tokenization for any real-world word-processed or copy-pasted text.
Fix:
.replace("’", "'") # RIGHT SINGLE QUOTATION MARK → ASCII apostrophe
.replace("‘", "'") # LEFT SINGLE QUOTATION MARK → ASCII apostrophe
core/parsing/normalizer.py::normalize_text_for_matchingcontains:ASCII apostrophe (U+0027) is never produced. Bones-map keys for contraction fragments (
n't,'ll,'d,'re,'m,'ve,'s) are written with ASCII'. Input containing smart apostrophes (U+2019 in "don't") survives unchanged, looks up against ASCII-keyed bone entries, and misses — producing incorrect tokenization for any real-world word-processed or copy-pasted text.Fix: