Repository navigation
frontend: map server code-point offsets to editor positions (#11) - #13
Merged
Merged
Conversation
ductus reports Python code-point offsets; the editor counts UTF-16 units, so every highlight after an emoji or other astral character was shifted and could split a surrogate pair. track() now gets positionsIn(text), a mapper built from the exact text the report was computed against.
…owercase Review of #11 found the same symptom server-side: rhetoric() matched concede-pivot against s.lower(), and str.lower() can change length ('İ' becomes two code points), so offsets pointed at the wrong characters. Now matched case-insensitively against the original text; the reported value stays lowercase. Also: Span documents that offsets are code points; the editor's module docstring no longer claims offset + 1; the unused toPos/toOffset helpers are removed so they cannot be reused by mistake.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11
ductusreports span offsets in Python code points; the editor (ProseMirror, JavaScript strings) counts UTF-16 units.toPos = offset + 1treated one as the other, so every highlight after an emoji or other astral character was shifted by one per such character and could split a surrogate pair. Paste and file-open were both affected.Change
frontend/src/offsets.ts—positionsIn(text): a code-point-offset → ProseMirror-position function built from the exact text the report was computed against. Fast path (no high surrogate in the text) is the oldoffset + 1; otherwise one pass records each code point's UTF-16 start. Out-of-range offsets clamp to the end.main.ts—track(report, text, positionsIn(text)).textis the text sent to the server, andread()already drops a report whose text is no longer in the editor, so the table always describes the document it maps into. After that instant every position moves through ProseMirror's ownMapping, which is UTF-16 already — nothing else converts.editor.ts—toPos's doc now says it is for UTF-16 offsets and points atpositionsInfor server offsets.Verification
vitest: 26 passed (5 new in
offsets.test.ts: BMP fast path, the Highlights shift after any emoji: server offsets are code points, the editor counts UTF-16 units #11 repro, exhaustive no-surrogate-split over a mixed string checked against Python-style code-point slicing, end-of-text, clamp).Headless Chromium against
ductus-http: a file of 30 × 😀 + the sample's first line now highlights exactlyGreat question,!,delve,tapestry(before:'�😀😀😀😀😀😀�'and every later finding shifted).Server side, same symptom (found in review):
rhetoric()matchedconcede-pivotagainsts.lower(), andstr.lower()can change length ('İ'.lower()is two code points), so its offsets pointed at the wrong characters. Now matched withre.IGNORECASEagainst the original text; the reported value stays lowercase. Pinned bytest_every_offset_indexes_the_text_even_when_lowercasing_changes_its_length(fails onmain). Side effect, negligible:re.IGNORECASEalso lets a leadingİ/ıstand for the pronoun "I" in this one pattern.Spandocstring now states offsets are code points;editor.ts's module docstring no longer claimsoffset + 1; the now-unusedtoPos/toOffsethelpers are removed so they cannot be reused by mistake.Python: 299 passed, 13 skipped (doctests included).
Review
Independent adversarial sub-agent review (browser-driven: emoji, 𝒳, U+2000B, CRLF, em dash — all 9 painted highlights equal their server quotes). It confirmed
track()is the only server-offset→position conversion and that Python emits code points everywhere except theconcede-pivotbug above, which is fixed here along with its doc/cleanup nits.Gate: hosted CI (Python + frontend workflows).
🤖 Generated with Claude Code