Drop application-state <meta> from front matter; one front matter reader for titles (#82) - #86
Merged
Merged
Conversation
…nk title wins (#82) Single-page apps ship serialised state in <meta content> under framework-private names (LinkedIn's __init, Ember's percent-encoded config/environment, ...), swamping the front matter. simplifyHtml now judges the value: a leading JSON object/array (raw or percent-encoded), a value over 4096 chars, or an HTML-blank value is dropped. Bracketed prefixes like "[Solved] ..." survive. When <head> holds several <title>s, the first non-blank one wins, and it takes precedence over <meta name="title"> in either order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tes at </head> (#82) isApplicationStateMeta moves to ApplicationStateMeta.kt and now parses the value as a JSON object/array (raw or percent-decoded) instead of peeking at its first and last char, so bracketed human text like "[2024] Annual report [PDF]" survives. A blank value is dropped on its own, not as "application state". Title candidates are resolved when <head> closes: the first non-blank <title> wins over <meta name="title"> (matched case-insensitively), and the entry keeps the position of the first candidate. Documented which values don't survive a wrapInHtmlDocument round-trip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#82) isApplicationStateMeta now parses with kotlinx Json.parseToJsonElement instead of a hand-rolled JSON validator. A JSON object is still dropped, but an array survives when it is a flat list of strings or numbers (keywords, article:tag, citation_volume); empty, nested or flag/null-holding arrays, and percent-encoded arrays, are dropped. Meta names are ASCII case-insensitive (HTML §4.2.5): of several names differing only in letter case, the first spelling and value win, and the title candidate logic folds into the same first-wins map. Documented the extra wrapInHtmlDocument round-trip caveat and the kotlinx Json leniency gotcha in CLAUDE.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e meta values (#82) Move HTML's ASCII lowercase into markanywhere-html-spec (asciiLowercase) and use it in simplifyHtml, wrapInHtmlDocument, ensureFrontmatterTitle and AutolinkCollector instead of private copies. wrapInHtmlDocument and ensureFrontmatterTitle now read front matter keys the way HTML reads <meta> names, so a Title key is the title. isApplicationStateMeta judges a percent-encoded value by what it decodes to, keeps flat arrays (an empty one included), and drops an over-long value only when it does not read as text, so a long prose abstract survives. JSON parse failures of any kind are treated as "not JSON". A blank (NBSP-only included) <html lang> is skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#82) - Extract HeadMetadata: first of case-variant duplicate keys wins in both simplifyHtml and wrapInHtmlDocument (wrapInHtmlDocument used last-wins). - ensureFrontmatterTitle judges only the first title entry in any case, so a usable title can no longer hide a blank variant that blanks <title>. - Strip and collapse <title> whitespace as document.title does (stripAndCollapseHtmlWhitespace in markanywhere-html-spec). - Spell a lang meta as `lang`, yielding to <html lang>. - Keep long comma/semicolon-separated keyword lists as text; decode percent escapes with ASCII hex digits only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ment; tidy head metadata (#82) - ensureFrontmatterTitle judges the first scalar title entry, the one wrapInHtmlDocument reads, instead of the first title entry of any type; a replaced entry keeps its key spelling - strip and collapse a title meta like a <title> element; trim lang - detect JSON state serialised into a JSON string - count code points when judging whether a long value reads as text - share HEAD_KEYS / SCALAR_ENTRY_TYPES via HeadMetadata.kt Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… keys (#82) - HeadMetadata.add ignores blank values, so a blank key never shadows a later case variant in simplifyHtml or wrapInHtmlDocument - ensureFrontmatterTitle judges the first non-blank scalar title entry, respells a case-variant key to `title`, and collapses the derived title's whitespace as document.title does - treat double-encoded JSON state inside arrays as application state; count combining marks as letters when judging prose - drop unused kotlinx-serialization-json dependency declaration Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- judge a percent-encoded meta value, including its length, by what it decodes to, and a flat JSON array by its words, so long keyword lists survive the over-long backstop - replace the empty title slot spelled `title` first, inject a title next to an unreadable variant in another spelling, and treat a frontmatter that does not open the stream (or a tagged one) as content, as wrapInHtmlDocument does - share isScalarEntryType / isMetadataValue between wrapInHtmlDocument and ensureFrontmatterTitle; drop the redundant lang blank check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cycle Each step runs as a separate headless `claude -p` process, so every review starts from a fresh context. The loop stops when no critical/major correctness finding remains, when the fixer commits nothing, when a round leaves the build red, or at the round limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oop (#82) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the sentinel - stop when a round commits kotlin-js-store/yarn.lock, which the JS test build narrows, and revert the lock after every fix round - check for uncommitted changes before honouring the sentinel, so a fixer that edited files cannot end the loop with a dirty tree Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tter titles readable (#82) - simplifyHtml keeps the first of duplicate <meta> names (HTML); wrapInHtmlDocument keeps the later front matter entry (Psych/PyYAML), the lowercase spelling beating a case variant - ensureFrontmatterTitle drops blank/null entries spelled `title`, respells the usable variant wrapInHtmlDocument picks, and accepts a frontmatter preceded by blank text, emitting it first - add String.stripHtmlWhitespace() to markanywhere-html-spec and use it in place of private trims - skip JSON array word joining for meta values within the length limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a shadowing one (#82) - wrapInHtmlDocument reads a frontmatter preceded by blank text, moving that text to the start of body, as ensureFrontmatterTitle does - ensureFrontmatterTitle moves a usable `title` entry after a later collection-valued `title` entry, which a later-wins reader would pick - simplifyHtml dedupes meta names on the normalized name (first wins) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne (#82) - normalizeTitle trims non-breaking space at a title's edges (inside it stays content), shared by simplifyHtml and ensureFrontmatterTitle - ensureFrontmatterTitle puts a derived title right after a later collection-valued `title` entry, which a later-wins reader would pick, and picks the usable entry via frontMatterKeySupersedes, the rule HeadMetadata.addFromFrontMatter uses - use getOrPutIfMissing for first-wins HTML metadata - declare kotlinx-serialization-json directly for ApplicationStateMeta Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ensureFrontmatterTitle leaves exactly one title entry, spelled `title`, whenever a title comes out, dropping every other title variant: js-yaml (gray-matter) rejects duplicate keys outright, so reordering duplicates for later-wins readers was not enough - ensureFrontmatterTitle and wrapInHtmlDocument treat only HTML whitespace ahead of the frontmatter as insignificant (an NBSP is content) - isMetadataValue rejects zero-width / format chars as well as whitespace; simplifyHtml judges <title> text with it after normalization - isApplicationStateMeta: a flat JSON array of scalars is metadata whatever its bare words parse as; the length backstop judges a value as written (not percent-decoded) and drops an over-long blob unparsed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- wrapInHtmlDocument and ensureFrontmatterTitle read the front matter through one FrontMatterEntryReader instead of private copies - ensureFrontmatterTitle keeps a nested title entry (localized titles) rather than deriving one from the h1, dropping only empty variants - normalizeTitle trims invisible format chars (zero-width space, BOM) at a title's edges; simplifyHtml judges meta blanks by isMetadataValue - isApplicationStateMeta measures a flat JSON array by its elements joined, and drops a long array opening with a nested element unparsed unless it reads as text - document why asciiLowercase differs from lowercase (Kelvin sign) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…82) - isApplicationStateMeta percent-decodes repeatedly (bounded) and inside JSON strings, and spots a nested element anywhere in an over-long array - ensureFrontmatterTitle drops empty title entries when no title is derived, and derives a title past NBSP text before the h1 (which still keeps a following frontmatter from opening the stream) - normalizeTitle collapses line-breaking whitespace; HeadMetadata judges front matter entries by FrontMatterEntry.isHeadMetadata - condense the front matter title entry in CLAUDE.md Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ensureFrontmatterTitle keeps a usable title variant beside a nested title entry as spelled, and leaves a frontmatter whose root is a sequence unchanged (FrontMatterEntryReader.isSequence) - isApplicationStateMeta shares one layer budget between percent and JSON-string decoding, trims invisible chars (BOM, NBSP) at the edges, and judges an over-long value as written before decoding it - wrapInHtmlDocument strips HTML whitespace around the lang entry - extract trimInvisible; condense the simplifyHtml KDoc Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ensureFrontmatterTitle drops a frontmatter left empty by dropping its empty title entries (it would render as `---` twice, two thematic breaks on re-parse), and judges the title entries of a frontmatter the stream ends inside, an open entry included - isMetadataValue treats control chars (next line, C0) as invisible - clarify the over-long nested-array comment in isApplicationStateMeta Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ensureFrontmatterTitle keeps, beside a nested title entry, only the title variant wrapInHtmlDocument reads (via HeadMetadata, now tracking the source entry), and drops a frontmatter arriving empty even when no title can be derived - share the frontmatter-opens-the-stream rule (opensFrontmatter) between wrapInHtmlDocument and ensureFrontmatterTitle - isApplicationStateMeta judges a flat array past MAX_PARSED_LENGTH as written instead of parsing it; restructure into named helpers - normalizeLang trims invisible chars (BOM, zero-width space) around the lang in simplifyHtml and wrapInHtmlDocument Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ensureFrontmatterTitle closes a frontmatter or h1 the stream ends inside, so its output stays balanced, and treats text showing nothing (BOM, zero-width space) around the h1 as blank, as a title is judged - pick the title entry via FrontMatterEntry.readOver, shared with HeadMetadata, instead of tracking the source entry in MetadataEntry - wrapInHtmlDocument normalizes the title as simplifyHtml reads a <title> - isApplicationStateMeta counts `/` as a word break, so a long list of URLs is kept while base64 stays dropped - drop the now unused stripHtmlWhitespace from markanywhere-html-spec - restructure the ensureFrontmatterTitle and wrapInHtmlDocument KDoc Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ensureFrontmatterTitle no longer drops a leading title entry when that would leave a verbatim line first (front matter detection needs a key on line 2): a kept title variant moves into the first slot instead, or the first slot stays - isMetadataValue treats blank-rendering glyphs (Hangul fillers, blank Braille pattern) as invisible - isApplicationStateMeta never parses a value past MAX_PARSED_LENGTH, dropping one that opens like encoded state, and counts only letters of scripts written without spaces as word breaks - dropHtmlStructuralWhitespace reuses stripAndCollapseHtmlWhitespace - wrapInHtmlDocument checks the frontmatter mark alone (isFrontmatterMark) - html-spec tests use sameAs for string comparisons Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- FrontMatterEntryReader spans each top-level entry over the indented verbatim lines continuing it, and reads a verbatim line defining a key as an entry of unknown value (yamlKeyLineKeyOrNull, new in markanywhere-yaml), never head metadata - ensureFrontmatterTitle derives no title beside a verbatim title line (a multi-line quoted scalar), which made a duplicate key, and drops a title entry together with its continuation lines, which otherwise continued the entry before it; the verbatim-first guard is decided without building the body twice - normalizeTitle keeps bidi controls at the title's edges - isApplicationStateMeta judges only strict JSON as state, so bracketed prose such as [[Wiki]] is kept, and classifies supplementary-plane chars by plane, so a blob of emoji no longer reads as text - HeadMetadata.acceptsFromHtml is the one first-wins and blank rule, shared with simplifyHtml's meta pre-check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) An entry continued on indented lines, or holding indented verbatim lines under a bare `key:`, has a value front matter readers read but the YAML subset keeps verbatim. FrontMatterEntryReader marks it verbatim, so wrapInHtmlDocument emits no truncated or indented <title>/<meta> and ensureFrontmatterTitle keeps it as content. The proper fix, marking such entries in YamlParser, is tracked in #85. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
The PR is a draft, so step 1 says to stop and not proceed with the review. Code review of PR #86 (xemantic/markanywhere): skipped. The PR is open but still a draft, and the review instructions say not to review drafts. I posted no comments on GitHub and no review was run. No Claude comments exist on the PR yet. When the PR is marked ready for review, running Posted by the workflow: the review agent finished without commenting. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #82.
What changes
simplifyHtml: application-state<meta>values are dropped from the front matter.isApplicationStateMeta(ApplicationStateMeta.kt) judges a<meta content>by its value, not its name:serialised state (strict JSON objects and nested arrays, base64-like blobs, long unbroken tokens) is dropped,
while prose, flat word/number arrays and lists of URLs are kept.
On the LinkedIn "People" page this takes the front matter from 782 KB (95% of the dump) back to real page metadata.
Title and head metadata.
<title>wins instead of every<title>in<head>being concatenated.<meta>names are read ASCII case-insensitively, first<meta>wins as in HTML, blank values are skipped (HeadMetadata.kt, shared bysimplifyHtmlandwrapInHtmlDocument).wrapInHtmlDocumentandensureFrontmatterTitleread front matter through one reader (FrontMatterEntryReader),so they agree on which title entry is read and what a usable one is.
ensureFrontmatterTitlenever leaves duplicate title keys (js-yaml rejects them and loses the whole front matter),keeps nested and verbatim titles as content,
and keeps its output balanced when the stream ends inside a frontmatter or
h1.Supporting pieces.
AsciiCaseand HTML-whitespace helpers inmarkanywhere-html-spec;yamlKeyLineKeyOrNullinmarkanywhere-yaml, so a verbatim YAML line can still be matched to the key it defines.Known gaps — tracked in #85
FrontMatterEntryReaderinfers whether an entry's value is known from the shape of the events.The shape is ambiguous, so review rounds kept finding new multi-line YAML shapes.
Rather than keep patching, this PR takes the conservative option:
an entry continued on indented lines, or holding indented verbatim lines under a bare
key:, is read as unknown(no
<title>/<meta>, kept as content byensureFrontmatterTitle) instead of as a truncated or indented value.The proper fix —
YamlParsermarking such entries itself — and the remaining edge cases(merged text events hiding a verbatim
title:line, key-shaped verbatim lines blocking detection, truncated or crossed upstream streams)are in #85.
Testing
./gradlew buildpasses locally (dev targets).SimplifyHtmlTest,WrapInHtmlDocumentTest,EnsureFrontmatterTitleTest,AsciiCaseTest,HtmlWhitespaceTest,YamlKeyLineTest.🤖 Generated with Claude Code