Type, quote and keep YAML scalars exactly as the front matter readers do - #83
Merged
Merged
Conversation
…#80) YamlWriter quoted every value containing `:`, `#`, `"` or `\`, so URLs like `https://xemantic.com/contact` or `C#` came out needlessly double-quoted. A `:` is now only a mapping indicator when followed by a space or at the end of the value, and a `#` only starts a comment after a space; other occurrences, and inner `"` / `\`, stay plain. YAML 1.1 sexagesimals (`12:30`) remain quoted, since Psych and PyYAML type them as numbers. The parser's lenient reading of a mapping indicator inside a plain value (`k: Note: see`) is documented and pinned as a DIVERGENCE. Closes #80 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
YamlWriter already quoted YAML 1.1 sexagesimals (`12:30`), but Psych (Jekyll) and PyYAML also type other shapes YAML 1.2 reads as strings: booleans / nulls in any letter case (`yEs`, `nULL`), integers and floats with `_` / `,` separators, binary, signed hex, leading-zero octal, floats with a leading or trailing dot, looser timestamps (`2024-5-1`, offsets without a colon), and PyYAML's `=` / `<<` tags. The new `isYaml11Typed` covers the union, and both values and keys that match it are now double-quoted. Also guards the `#` comment check against index 0, and documents that a sequence item `- Note: see` is a compact mapping, not a lenient scalar. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
YamlWriter now escapes every character outside YAML's printable set (YAML 1.2 §5.1): C1 controls, U+FFFE / U+FFFF and lone surrogates, which Psych and PyYAML refuse anywhere in a document, even in a block scalar. Surrogate pairs (emoji) stay plain. isYaml11Typed also covers the one-letter booleans `y` / `n` (YAML 1.1, go-yaml v2) and the special floats in any letter case (`.Nan`, `+.InF`), deriving its word set from the parser's reserved literals. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The parser now types the shapes Psych (Jekyll), PyYAML and go-yaml v2 type even though YAML 1.2 reads them as strings: one-letter and mixed-case booleans / nulls, `_` / `,` separated numbers, binary, signed hex, sexagesimals and looser timestamps — including Jekyll's documented `2016-01-01 12:00:00 -0500`, which the writer used to quote and so rewrote on round-trip. Quoting only in the writer is not enough; the typing rules now live in one place, `YamlScalarType.kt`, shared by both sides through `isTypedPlainScalar`. The writer double-quotes a value holding a line break, carriage return or tab, and escapes a byte order mark (YAML 1.2 §5.2). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
YamlScalarType now transcribes the number rules of PyYAML, Psych and go-yaml v2 separately instead of one merged pattern, which typed shapes none of them reads as a number (`1,`, `1_,2`). go-yaml v2 deletes every `_` first, so `1_e5` / `+_1` / `0_x1` are numbers, and it takes an upper-case base prefix (`0X1F`). A timestamp is typed only when it is in range: a date must be in the calendar, a date and time within what Psych's Time accepts. Shapes a reader refuses to load (`=`, `<<`, `0x_`, `.e+4`, `2024-13-45`) stay strings in the parser but are still quoted by the writer, pinned as DIVERGENCE. The colon / comment indicator checks are shared between parser and writer via isMappingColonAt / isCommentStartAt, so the writer now also treats a tab like a space there. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Checked against PyYAML, Psych and go-yaml v2: - quote a date-shaped string only when PyYAML refuses it (2024-2-30 stays plain) - quote only the keys Psych types (yEs, nULL), not y / n, which go-yaml v2 decodes into a string key - stop typing -0b-1 / -0b+1 as int, type 08 / 019 / 0_9 as float - refuse hour 24 beyond 24:00:00 and offset minutes over 59 - share the mapping-colon and comment rules with flow collections - gate the refused-shape patterns on the first character Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- split a timestamp offset as Psych does (at the colon, else two hour digits) and bound the whole offset under a day, so `+05:99` is typed and `+530` / `+2400` stay strings - write base-prefixed digit runs so a long near-miss resolves in linear time instead of backtracking quadratically - end a flow mapping key at a ` #` comment, and start a comment at any `#` after a closed quoted scalar or flow collection - share the mapping-colon check with the key line detector Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- rewrite every YAML number pattern that repeated a group (`(?:_?[0-9])*`, `(?::[0-9])+`) as a character-class run plus a hand check, since the JVM regex engine recurses once per repetition and a long digit run threw StackOverflowError - reject a string holding a character no numeric or timestamp shape can contain before running any pattern - require the offset minutes after a colon, so `+5:` is not a timestamp - refuse a date-only shape with a signed zero year (`-0000-01-01`) - write a lone surrogate as U+FFFD (DIVERGENCE: lossy), as Psych and go-yaml refuse its `\u` escape Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… first
- quote a `y` / `n` key below the top level, since go-yaml v2 decodes a
nested mapping with interface{} keys and reads a plain `y:` as `true`
- type a timestamp-shaped or `:`-holding scalar through its own patterns
only, so the number patterns never run on it
- merge the Psych and PyYAML base-60 int and float into one pattern each,
with Psych's leading `0` limited to two segments
- derive PyYAML's timestamp shape from the shared match instead of a
second pattern
- pin the libyaml comment-after-closed-token rule as a DIVERGENCE test
- make the linear-time test compare two input sizes instead of an
absolute bound, so slower Native / JS regex engines don't trip it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mbers
- keep a flow scalar whose `:` precedes `,` / `[` / `]` / `{` / `}`
verbatim: PyYAML reads a mapping, Psych refuses the line and go-yaml v2
reads the colon as content, so no single reading is right
- keep a block scalar header followed by `#` with no space verbatim, as
PyYAML refuses it
- type a number go-yaml v2 alone reads only within its int64 / uint64 /
float64 range (`1e999`, `0X` and 17 hex digits stay strings)
- match the timestamp pattern once in `isUnsafePlainScalar`
- replace the wall-clock linear-time test with an input large enough that
quadratic backtracking hangs
- name every divergence test `DIVERGENCE - …`
- condense the YAML entries in CLAUDE.md and state the lesson / why rule
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes
markanywhere-yaml's typing and quoting follow what the front matter readers — Psych (Jekyll), PyYAML and go-yaml v2 (Hugo) — actually do, verified by running all three over generated strings rather than from their docs.Parser typing
y/n, numbers with_/,separators, binary / upper-case / signed base prefixes, base 60 (12:30), and looser timestamps — including Jekyll's documented2016-01-01 12:00:00 -0500, which used to come back quoted.1,,-0b-1,2024-2-30,1e999), sotype=int/type=floatkeeps meaning a number.Timebounds, offsets under a day).:before,/[/]/{/}([draft:, x]is a mapping to PyYAML, an error to Psych,"draft:"to go-yaml) and a block scalar header followed by#with no space (PyYAML refuses it).Writer quoting
YamlScalarType.kt).=,<<,0x_,2024-13-45) is quoted too, though the parser keeps it a string.yEs,nULL) are quoted;y/nstay plain at the top level (go-yaml decodes a front matter key as a string) and are quoted below it (a nested key isinterface{}).\uescape).Robustness
StackOverflowError.Housekeeping
DIVERGENCE - …(renamed acrossparseandyaml)../gradlew checkpasses on JVM and JS;markanywhere-yamlalso on macOS Native.🤖 Generated with Claude Code