Repository navigation
Lexer: support \r and explicit byte escapes in string and bytes literals #249
Description
Problem
Edict's lexer accepts only \", \\, \n, and \t inside string and bytes literals (crates/edict-syntax/src/token.rs, string-literal scanner; main e60411fc). b"\r" fails with Lex / invalid string escape, and there is no explicit byte escape such as \xNN.
The language spec already anticipates this: "Non-ASCII source characters inside bytes-lit are rejected unless escaped as explicit byte escapes", and lists "Bytes literal escapes … have canonical fixtures" as a lexical conformance item to pin before parser freeze (docs/SPEC_edict-language-v1.md, lexical grammar notes). Neither the escape set nor the fixtures exist yet.
Reproduction
Jim's Buffer-decoder probe (flyingrobots/jedit#302, edict/replace-range-probes/buffer-decoder/Decimal.edict) refuses at lex on b"\r" at the public application build boundary, rebuilt today against main e60411fc.
Proposed scope
- Pin the escape set in the spec: at least
\r,\0, and\xHHfor bytes literals; decide whether\u{…}applies to string literals. - Add canonical lexical fixtures (accept and reject) and keep decoding canonical (one spelling per byte value is not required, but the decoded payload must be).
- Keep the change lexical; no new language semantics.
Small, generic, independent of any application. Refs flyingrobots/jedit#296, flyingrobots/jedit#302.
Activity
flyingrobots commentedon Oct 16, 2026
Correction to the scope above, checked against main
e60411fcby building through the public application boundary:Bytes literals are not lexed at all.
b"ab"is read as the identifierbfollowed by a string literal, so bothlen(b"ab")(ExpectedToken: expected RParen, found Str("ab")) andlet w: Bytes<max=2> = b"ab";(expected Semi, found Str("ab")) fail to parse. The Jim Buffer-decoder probe hit the\rescape error first only because lexing runs before parsing.So this issue covers two things: (1) lex
b"…"as a bytes literal per the spec grammar (bytes-lit = "b" , string-lit), and (2) the escape set, including\rand explicit byte escapes. Both remain lexical; no new semantics.