Skip to content
This repository was archived by the owner on Jul 29, 2026. It is now read-only.
This repository was archived by the owner on Jul 29, 2026. It is now read-only.

[S2] Verify parser/turns_rounds.py _WORD_RE handles ASCII apostrophes; add regression test #58

Description

@erinepshovel-code

Tests/test_backend.py::test_tokenize_ascii_apostrophe_contraction verifies that stats.py::_WORD_RE tokenizes "don't can't it's" as whole tokens. The test was added to fix issue #37.

backend/src/edcmbone/parser/turns_rounds.py::_WORD_RE is a visually similar regex but has no equivalent regression test. It is possible that the same apostrophe fix was not applied to the parser regex, or that it was applied but is untested and could regress.

Action required:

  1. Byte-compare stats.py::_WORD_RE and parser/turns_rounds.py::_WORD_RE to confirm whether the character class ['‘’] (or equivalent) is present in both.
  2. If the parser regex is missing the apostrophe character class, apply the same fix that was applied to stats.py for issue stats.py _WORD_RE misses ASCII apostrophe — contractions split on raw input #37.
  3. Add a test to Tests/test_backend.py (or a new file) that feeds "don't can't it's" through the parser tokenization path (not just stats.tokenize) and asserts whole-token output.

Without step 3, even if the regex is currently correct, the fix is unguarded and will not be caught if it regresses.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions