Skip to content
This repository was archived by the owner on Jul 29, 2026. It is now read-only.
This repository was archived by the owner on Jul 29, 2026. It is now read-only.

[S2] normalizer.py no-op replace — smart apostrophes never normalized to ASCII #47

Description

@erinepshovel-code

core/parsing/normalizer.py::normalize_text_for_matching contains:

.replace("’", "’")   # U+2019 → U+2019, no-op
.replace("‘", "’")   # U+2018 → U+2019 (curly-open → curly-close, still not ASCII)

ASCII apostrophe (U+0027) is never produced. Bones-map keys for contraction fragments (n't, 'll, 'd, 're, 'm, 've, 's) are written with ASCII '. Input containing smart apostrophes (U+2019 in "don't") survives unchanged, looks up against ASCII-keyed bone entries, and misses — producing incorrect tokenization for any real-world word-processed or copy-pasted text.

Fix:

.replace("’", "'")   # RIGHT SINGLE QUOTATION MARK → ASCII apostrophe
.replace("‘", "'")   # LEFT SINGLE QUOTATION MARK → ASCII apostrophe

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions