Skip to content

fix(kk): classify wide abbreviations per occurrence - #94

Merged
yisding merged 1 commit into
mainfrom
codex/propose-fix-for-kazakh-abbreviation-issue
Jul 11, 2026
Merged

fix(kk): classify wide abbreviations per occurrence#94
yisding merged 1 commit into
mainfrom
codex/propose-fix-for-kazakh-abbreviation-issue

Conversation

@yisding

@yisding yisding commented Jun 29, 2026

Copy link
Copy Markdown
Owner

Motivation

  • The Kazakh refactor moved formerly-dotted abbreviation protection into the period-classifier global-realize model, but the classifier deduplicates candidates by (abbreviation, follower_char) and can drop later same-key occurrences that need different decisions, producing incorrect splits in cases like обл.x обл.: қала.

Description

  • Enable per-occurrence realization for the Kazakh policy by setting realize_per_occurrence=True on KK_POLICY in sentencesplit/lang/kazakh.py so each candidate is classified from its own original context.
  • Add a regression test test_kk_empty_follower_occurrences_are_classified_independently in tests/lang/test_kazakh.py asserting the обл.x обл.: қала example segments as ['обл.', 'x обл.: қала'].
  • The change preserves the widened follower-class behavior for the frozen set of formerly-dotted stems via the existing regular_follower_overrides field and only alters realization semantics.

Testing

  • Ran the focused Kazakh and classifier tests with uv run pytest tests/lang/test_kazakh.py tests/regression/test_abbr_dot_normalization.py tests/contract/test_period_classifier.py and they passed (all tests in that run succeeded).
  • Ran lint/format checks with uv run ruff check . and uv run ruff format --check . and they succeeded.
  • Ran the full test suite with uv run pytest --cov=sentencesplit tests/ --color yes and the test run completed successfully (the suite passed with the repository's existing skips/xfails).

Codex Task

@charliecreates charliecreates Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the latest changes and I don’t have actionable feedback to address. CI is still in progress, so merge readiness depends on those checks completing.

@yisding
yisding merged commit 1d024a6 into main Jul 11, 2026
9 checks passed
@yisding
yisding deleted the codex/propose-fix-for-kazakh-abbreviation-issue branch July 11, 2026 06:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant