Skip to content

Fix incorrect token offsets for multi-byte Unicode characters - #204

Open
naaish-adam wants to merge 2 commits into
humanwhocodes:mainfrom
naaish-adam:fix/multi-byte-unicode-parsing
Open

Fix incorrect token offsets for multi-byte Unicode characters#204
naaish-adam wants to merge 2 commits into
humanwhocodes:mainfrom
naaish-adam:fix/multi-byte-unicode-parsing

Conversation

@naaish-adam

Copy link
Copy Markdown

Currently, slicing strings using token offsets fails for multi-byte Unicode characters like é or ₹ because the tokenizer advances the offset by 1 for all characters regardless of their byte length. This PR fixes it by advancing by byte length (char.len_utf8()) instead of character length for offset values.

Example JSON that produces incorrect offsets:

{
  "Café": "Malé",
  "Owner": "Jérôme",
  "Location": "Île de la Cité, Paris",
  "Items": {
    "Espresso": "€3",
    "Latte": "€4"
  }
}

@nzakas

nzakas commented Mar 10, 2026

Copy link
Copy Markdown
Collaborator

Thanks for putting this together. I'd like to keep the behavior between the Rust and JS implementations the same, otherwise the unified tests will fail. Do you want to take a look at that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants