Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ Both developers and AI agents are expected to add entries as they encounter surp
- **Add an entry** when you encounter something unexpected: a build quirk, a non-obvious constraint, a dependency gotcha, or any behavior that would surprise the next agent or developer.
- **Add an entry** when a developer flags an anti-pattern produced by AI — describe the anti-pattern and the preferred alternative.
- **Do not** add codebase overviews, directory listings, or anything discoverable by reading the source.
- Keep entries concise: one line per lesson, grouped under a heading if a theme emerges.
- Keep entries concise: the lesson and its non-obvious *why*, grouped under a heading if a theme emerges.
Name a symbol or test as a pointer, but don't restate what its source comment already says.

## Conventions

Expand Down Expand Up @@ -411,6 +412,8 @@ If the fix is bounded (one inline construct, one or two lines of lookahead), it'
`renderDumpMarkdown` (Gradle task, `RenderDumpMarkdown.kt` in jvmTest) runs the full `transformHtmlToMarkdown` pipeline over every dump to `build/renderedMarkdown/<name>.md` — the canonical way to regenerate the per-dump golden strings (`dumps/OpenjurTest`, `dumps/HackerNewsTest`) after a pipeline change.
- A `Regex` used with `matches()` in `commonMain` must be **explicitly anchored** (`^(?:a|b)$`) when it contains a top-level alternation: Kotlin/JS resolves `matches` through the leftmost `find`, so the first branch wins on a *prefix* and the whole-input check fails (`0x1F` matched `[-+]?[0-9]+|0x[0-9a-fA-F]+` as `0`; a full timestamp matched the date-only branch) — JVM backtracks across the branches and never shows it.
The YAML scalar typing in `markanywhere-yaml` hit exactly this: green on `jvmTest`, red on `jsBrowserTest`. Dev builds run the JS tests for `yaml`/`parse`/`render`/`html`, so a JVM-only test run is not enough evidence for a regex change.
- A `commonMain` `Regex` applied to untrusted input must not **repeat a group** (`(?:_?[0-9])*`, `(?::[0-9])+`): `java.util.regex` matches each repetition of a group one stack frame deeper, so a few thousand repetitions throw `StackOverflowError` on the JVM (an `Error`, which nothing catches — it takes the whole pipeline down).
Repeat a character class instead (`[0-9_]*`) and check what the group expressed by hand (`YamlScalarType.kt`).
- Facts of the HTML standard needed by more than one of `parse` / `render` / `html` — HTML whitespace (`isHtmlWhitespace` / `isHtmlBlank`), void and raw text elements, `Mark.classList` — live in the leaf module `markanywhere-html-spec` (depends only on `api`), never in `markanywhere-api` (which describes only the event model) and never as private copies.
Only spec facts go there, not project policy (e.g. which tags render as blocks in Markdown); the parser's `isFlankWhitespace` is deliberately a different set (CommonMark "Unicode whitespace", which includes NBSP).
- `MutableMap.putIfAbsent` is **JVM-only** — not in the common stdlib, so it can't be used in `commonMain`.
Expand Down Expand Up @@ -543,11 +546,15 @@ If the fix is bounded (one inline construct, one or two lines of lookahead), it'
The key is an attribute, not the mark name, on purpose: keys collide with tag names (`title` would hit `simplifyHtml`'s head-mode `match("title")`) and `og:title` would look like `<ns:name>` custom markup.
`entry`/`item` are Markdown-native **only inside** a `frontmatter` (the Markdown renderer creates a `YamlWriter` on the mark, routes every following event into `YamlWriter.collect`, and treats the first `unmark` arriving at `depth == 0` as the wrapper's own close; outside one they fall through to the raw-tag path), so they are deliberately NOT in `MARKDOWN_NATIVE_MARK_NAMES`.
Any other mark is **transparent** to the writer (children lay out as if it were absent), which is what lets `renderYaml()` accept a `frontmatter`-wrapped stream unchanged.
The writer's quoting rules reuse the parser's `yamlScalarType` (one copy of the reserved literals / int / float / timestamp rules) and are pinned against it by `YamlRoundTripTest` in `markanywhere-yaml`; `FrontMatterRoundTripTest` in `markanywhere-parse` (whose tests depend on render) covers only the fences and the wrapper — extend the yaml one when you touch either side of the codec.
The writer's quoting rules reuse the parser's `yamlScalarType` (`YamlScalarType.kt`, one copy of the reserved literals / int / float / timestamp rules) and are pinned against it by `YamlRoundTripTest` in `markanywhere-yaml`; `FrontMatterRoundTripTest` in `markanywhere-parse` (whose tests depend on render) covers only the fences and the wrapper — extend the yaml one when you touch either side of the codec.
Typing follows the front matter readers (Psych/Jekyll, PyYAML, go-yaml v2): a shape some reader types must be typed by the **parser** (writer-only quoting rewrote Jekyll's `date: 2016-01-01 12:00:00 -0500` into a quoted string), a shape none types must not be (or `type=int` stops meaning a number), and a shape they read **differently** from one another (`[a:, b]`: a mapping, an error, the string `"a:"`) stays a verbatim line — picking one reading rewrites the document for the others.
Verify a change by running the readers themselves (PyYAML `safe_load`, Ruby `Psych.unsafe_load`, `gopkg.in/yaml.v2` into `map[string]interface{}`) over generated strings, never from their docs or the spec — reasoning from the spec got each of the three rules above wrong at least once.
The one writer-only quoting is a shape a reader **refuses to load** (`=`, `0x_`, `2024-13-45`), pinned as `DIVERGENCE` in `YamlRoundTripTest`.
`renderYaml()` **keeps the trailing newline** (unlike `renderMarkdown()` / `renderHtml()`, which suppress one): a `|+` block scalar at the end of the document needs it, and dropping it broke the round-trip for `z\n\n`.
- Front matter **detection** requires line 2 to be a mapping key (identifier or quoted, then `:` + whitespace/EOL — `isYamlKeyLine`, a public function of `markanywhere-yaml`, NOT a private copy in `FrontMatterFilter`); a `# comment` cannot be line 2 (`---` + `# Heading` is a thematic break + heading), and neither can `http://…` or a `- item`.
Consequence for the YAML writer: `YamlWriter.renderKey` quotes every key that is not identifier-shaped (`"a b"`, `"42"`, `"og:title"`), even where YAML would not require it, so whatever `simplifyHtml` emits first (a `<meta name>` with a space, say) still re-detects.
It also quotes a key that is a reserved literal (`"yes":`) — a YAML 1.1 reader would take a bare `yes:` as a boolean key.
Key typing is not value typing: go-yaml v2 decodes a **top-level** front matter key into a string but a nested one as `interface{}`, so `y:` must stay plain at the top level (it was once rewritten to `"y":`) and be quoted below it (`isTypedPlainKey`).
The writer's plain-key rule (`isIdentifierKey`) and the detector share their character rules in `YamlKeyLine.kt`, and `YamlWriterTest` "should write every key as a line the front matter detector accepts" pins the two — widen one without the other and every document whose *first* entry has such a key silently loses its front matter on re-parse.
**DIVERGENCE (no round-trip)**: a `frontmatter` whose root is a **sequence** (top-level `item`s) renders as `---\n- a\n…`, which line 2 rejects — Markdown reads `---` + `- a` as a thematic break + list, and no front matter consumer (Jekyll, Hugo, `wrapInHtmlDocument`) accepts a non-mapping — so the shape is pinned as such in `FrontMatterRoundTripTest` rather than widened into detection.
- `wrapInHtmlDocument` and `ensureFrontmatterTitle` read the front matter by **depth** (a direct child `entry` is depth 1 / 2 respectively, counted from the `frontmatter` mark): only top-level scalar entries become `<meta>`/`<title>`, only a top-level `entry key=title` counts as an existing title.
Expand All @@ -565,6 +572,7 @@ If the fix is bounded (one inline construct, one or two lines of lookahead), it'
- For semantic event flow testing, use the overloaded `sameAs` infix on `Flow<SemanticEvent>` (defined in `markanywhere-test`) against a `semanticEvents { ... }` builder — this is event-stream comparison, unrelated to HTML.
- For asserting rendered HTML output (e.g. in `markanywhere-render` tests), use `sameAsHtml` (not the generic string `sameAs`) — provides syntax highlighting in the IDE.
- GFM example test naming convention: `example N - <description>` for spec-conformant tests, `example N - DIVERGENCE - <description>` when the parser intentionally diverges from GFM.
Any other test pinning an intentional divergence (from GFM, YAML or a reader's behaviour) starts with `DIVERGENCE - ` (`DIVERGENCE - should keep …`), never `should DIVERGENCE …` or a trailing `… DIVERGENCE`.
Keep `DIVERGENCE` in the name even when updating expectations to match the divergent behavior.
- When asserting on a boolean expression (counts, equality between two nullables, `contains`, etc.), use `assert(expr)` from `com.xemantic.kotlin.test.assert` — it is power-assert-instrumented in the convention plugin, so a failure prints the full expression with subvalues.
Do NOT fall back to `kotlin.test.assertEquals` / `assertTrue` / `assertNotNull` — they discard that information.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ class FrontMatterRoundTripTest {
}

@Test
fun `should DIVERGENCE not re-detect a root sequence front matter`() = runTest {
fun `DIVERGENCE - should not re-detect a root sequence front matter`() = runTest {
// given — YAML allows a root sequence, but front matter detection
// requires a mapping key on line 2 (`---` + `- a` is a thematic
// break followed by a list in Markdown, and no front matter consumer
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ class FrontMatterTest {
}

@Test
fun `should DIVERGENCE auto-close unterminated front matter at EOF`() = runTest {
fun `DIVERGENCE - should auto-close unterminated front matter at EOF`() = runTest {
// given — opener but no closer. DIVERGENCE: spec-correct behavior would
// require buffering the whole document to detect the missing closer
// and fall back to a thematic break + paragraphs. The streaming parser
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ class HtmlBlockInListItemTest {
}

@Test
fun `type 1 pre block inside list item DIVERGENCE`() = runTest {
fun `DIVERGENCE - type 1 pre block inside list item`() = runTest {
// given
val textFlow = /* language=markdown */ """
- intro
Expand Down Expand Up @@ -298,7 +298,7 @@ class HtmlBlockInListItemTest {
}

@Test
fun `blank line inside list-internal html block stays in raw mode DIVERGENCE`() = runTest {
fun `DIVERGENCE - blank line inside list-internal html block stays in raw mode`() = runTest {
// given: a blank line inside a `<div>` block. At top level this would
// transition the block to sub-parse mode and the `- nested` line would
// open a Markdown list. Inside a list item we stay in raw-text mode
Expand Down Expand Up @@ -444,7 +444,7 @@ class HtmlBlockInListItemTest {
}

@Test
fun `type 1 pre block with multi-line opener inside list item DIVERGENCE`() = runTest {
fun `DIVERGENCE - type 1 pre block with multi-line opener inside list item`() = runTest {
// given: `<pre` on the marker line, attribute on the next line, `>`
// closes the opener. Exercises the buffered-opener path in the type-1
// branch of `streamListHtmlBlockLine` — `firstLineBuffer` accumulates
Expand Down
8 changes: 4 additions & 4 deletions markanywhere-parse/src/commonTest/kotlin/HtmlParsingTest.kt
Original file line number Diff line number Diff line change
Expand Up @@ -401,7 +401,7 @@ class HtmlParsingTest {
* over spec-faithful escaping.
*/
@Test
fun `DIVERGENCE inline disallowed script tag and its body are dropped`() = runTest {
fun `DIVERGENCE - inline disallowed script tag and its body are dropped`() = runTest {
// given
val src = "before<script>alert(1)</script>after\n"

Expand All @@ -419,7 +419,7 @@ class HtmlParsingTest {
* including HTML-like substrings, is dropped.
*/
@Test
fun `DIVERGENCE inline disallowed style tag and its body are dropped`() = runTest {
fun `DIVERGENCE - inline disallowed style tag and its body are dropped`() = runTest {
// given
val src = "x<style>a::before{content:\"<x>\"}</style>y\n"

Expand All @@ -437,7 +437,7 @@ class HtmlParsingTest {
* the final `>` (`</script >`), mirroring the block-1 close rule.
*/
@Test
fun `DIVERGENCE inline disallowed close tag tolerates trailing whitespace`() = runTest {
fun `DIVERGENCE - inline disallowed close tag tolerates trailing whitespace`() = runTest {
// given
val src = "a<script>js</script\t >b\n"

Expand Down Expand Up @@ -539,7 +539,7 @@ class HtmlParsingTest {
* always close their `<script>`/`<style>`, so this edge is benign in practice.
*/
@Test
fun `DIVERGENCE unclosed inline disallowed opener drops only the rest of its line`() = runTest {
fun `DIVERGENCE - unclosed inline disallowed opener drops only the rest of its line`() = runTest {
// given
val src = "keep<script>dropped to end of line\nnext line\n"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ import kotlin.test.Test
class Gfm_06_11_Test {

@Test
fun `example 657 - DIVERGENCE disallowed inline tags are dropped not escaped`() = runTest {
fun `example 657 - DIVERGENCE - disallowed inline tags are dropped not escaped`() = runTest {
// given
val textFlow = buildString {
+"<strong> <title> <style> <em>\n"
Expand Down
10 changes: 7 additions & 3 deletions markanywhere-yaml/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,10 @@ The vocabulary is deliberately small and the key lives in an attribute, so a key
- `entry` (attribute `key`) is a mapping entry; `item` is a sequence element.
A root mapping is a run of top-level `entry` marks, a root sequence a run of top-level `item` marks — there is no wrapper mark.
- The kind of value follows from the children: text only is a scalar, nested `entry` marks are a mapping, nested `item` marks are a sequence.
- `type` is present only on a non-string scalar: `bool`, `int`, `float`, `null`, `timestamp` (YAML 1.2 core schema, plus the 1.1 `yes`/`no`/`on`/`off` booleans so a Jekyll / Hugo document round-trips as written).
- `type` is present only on a non-string scalar: `bool`, `int`, `float`, `null`, `timestamp`.
A plain scalar is typed exactly when some reader front matter is written for reads it as something other than a string — the YAML 1.2 core schema, plus the YAML 1.1 shapes Psych (Jekyll), PyYAML and go-yaml v2 (Hugo) still type — so a Jekyll / Hugo document round-trips as written.
Beyond the core schema that means booleans and nulls in any letter case and `y` / `n`, numbers with `_` / `,` separators or a binary / octal / upper-case base prefix (`1_000`, `1,000`, `0b101`, `0X1F`), base-60 numbers (`12:30` is an `int`), and timestamps with a one-digit month or day or a colonless offset (`2016-01-01 12:00:00 -0500`); a date only when it is in the calendar, and a number only go-yaml v2 reads only within its 64-bit range (`1e999` stays a string).
So `type="int"` / `type="float"` says a reader reads a number, not that the text is a Kotlin literal: reading the value yourself takes that reader's rules (drop the separators, resolve the prefix or the base 60).
No text and no type is an empty string; a bare `key:` is `type="null"`; an empty `[]` / `{}` is `type="seq"` / `type="map"` with no children.
- All marks are untagged.

Expand All @@ -50,13 +53,14 @@ Block mappings and sequences (a sequence may sit at its key's own indentation),
Streaming: a `key: value` line commits on its newline.
A bare `key:` (or `-`) is held for one line to decide between a nested block and a null value; a block scalar is buffered until it closes — never past the enclosing construct.

DIVERGENCE (never throws, never loses content): a line outside that subset — a complex `? key`, a directive, a document marker, a multi-line flow collection or quoted scalar, a plain scalar's continuation line, a `- item` inside a mapping — is emitted **verbatim** (with its `\n`) as a text child of the container it sits in, and the writer writes it back as-is.
DIVERGENCE (never throws, never loses content): a line outside that subset — a complex `? key`, a directive, a document marker, a multi-line flow collection or quoted scalar, a plain scalar's continuation line, a `- item` inside a mapping, a shape the front matter readers read differently from one another (`[draft:, x]`, `{a:[1]}`, `|-#note`) — is emitted **verbatim** (with its `\n`) as a text child of the container it sits in, and the writer writes it back as-is.
Anchors, aliases and tags are not resolved: a plain scalar starting with `&`, `*` or `!` is just a string.

## Writing

`renderYaml()` emits block style, one line per scalar, two spaces per nesting level.
A string that would re-parse as another type or shape (`"true"`, `"42"`, `"- dash"`, `"a: b"`, surrounding whitespace, control characters) is double-quoted; a multi-line string becomes a literal block scalar with the chomping indicator its trailing newlines call for; a key is written plain only when identifier-shaped and not a reserved literal.
A string that would re-parse as another type or shape (`"true"`, `"y"`, `"42"`, `"12:30"`, `"- dash"`, `"a: b"`, surrounding whitespace) is double-quoted, with every character YAML does not allow in a document `\u`-escaped; a multi-line string becomes a literal block scalar with the chomping indicator its trailing newlines call for; a key is written plain only when identifier-shaped and not a typed word (`"yes"`, `"y"`, `"null"`).
DIVERGENCE: a string some reader refuses to load when plain — `=`, `<<`, a base prefix with no digit (`0x_`), a date not in the calendar (`2024-13-45`) — is quoted too, although the parser reads it back as a plain string, so the first render quotes such a value in the source.
Any mark other than `entry` / `item` is transparent, so a `frontmatter` wrapper renders the same with or without it.
Every line is terminated — a non-empty document ends with `\n`, which a `|+` block scalar at the end needs.

Expand Down
4 changes: 1 addition & 3 deletions markanywhere-yaml/src/commonMain/kotlin/YamlKeyLine.kt
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,7 @@ public fun isYamlKeyLine(line: String): Boolean {
j
}
while (i < line.length && line[i] == ' ') i++
if (i >= line.length || line[i] != ':') return false
i++
return i == line.length || line[i] == ' ' || line[i] == '\t'
return i < line.length && line.isMappingColonAt(i)
}

// A key [YamlWriter] may write plain: identifier-shaped, so it passes
Expand Down
Loading
Loading