Skip to content

test(tokens): cover encode_text without boundaries, decode_text error policies, and modality edge values - #19

Open
lcampb96 wants to merge 1 commit into
fufuchiu:mainfrom
lcampb96:test-tokens-boundary-and-decode
Open

lcampb96 wants to merge 1 commit into
fufuchiu:mainfrom
lcampb96:test-tokens-boundary-and-decode

Conversation

@lcampb96

Copy link
Copy Markdown

The existing token tests focus heavily on UTF-8 roundtrip with boundaries=True but leave several paths uncovered. This adds eight tests:

  • encode_text with boundaries=False omits BOS/EOS markers
  • encode_text byte offset verification (null byte encodes as ID 4)
  • decode_text with errors=replace substitutes invalid bytes with U+FFFD
  • decode_text with errors=ignore silently drops invalid bytes
  • modality(4) correctly classifies the first text token
  • modality(259) correctly classifies the last text token before the audio range
  • modality(260) correctly classifies the first audio token
  • encode_text(, False) yields an empty list

All 63 tests pass locally (pytest tests/test_tokens.py -q).

@lcampb96
lcampb96 requested a review from fufuchiu as a code owner September 10, 2026 04:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant