Repository navigation
fix(engine): allocate tokens in document order - #31
Merged
Merged
Conversation
Regex substitutions are applied right-to-left so that earlier offsets stay valid, but store.add() was called in that same reverse order, so [TAG:N] counters ran backwards: the second IBAN in a text was tokenised [IBAN:1]. Tokens are now allocated in document order first, then substituted right-to-left. Unmasking was always correct; this only fixes what a human or the downstream LLM reads in the masked text.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Regex substitutions are applied right-to-left so that earlier offsets stay valid — but
store.add()was called inside that same reversed loop, so the[TAG:N]counters ran backwards too. In a text with two IBANs, the second one became[IBAN:1].Unmasking was always correct; this only affects what a human (or the downstream LLM) reads in the masked text — and reversed numbering is quietly confusing there.
Fix: allocate tokens in document order first, then substitute right-to-left.
Tests: +3 in
tests/regex-regression.test.ts— appearance order, independent per-type counters, and a full round-trip back to the original.tsc --noEmitclean,eslintclean, full suite 380/380 green. Changeset (patch) included.