simplifyHtml drops an unrecognised element together with its whole subtree (web components, <picture>, <caption>, …)

[morisil]

Summary

simplifyHtml registers no match("*") catch-all, and the transform framework
drops an unmatched mark together with its whole subtree — by design:

// > 0 while inside a subtree whose enclosing mark chose not to
// descend (no children() call) or had no matcher at all — every
// nested mark and text event is skipped until it closes.
var skipDepth = 0

(Transformation.kt)

So simplifyHtml's tag sets (UNWRAPPED_TAGS, PRESERVE_WITH_ID_TAGS,
INLINE_FORMATTING_TAGS, FORM_ELEMENT_ATTRS, the explicit match("…") rules)
are an implicit allowlist that swallows content, not just a wrapper filter.
An element in none of them loses everything it contains — its text, its
headings, its images, and any actionable control nested under it.

This is the same failure shape as the <option> bug fixed by
SELECT_CONTENT_MODE in ApplyAccessibility.kt: the element disappears
downstream while the ref
DomEventBuilder.mark() already spent on it stays spent, so the rendered
Markdown shows a gap in the otherwise dense ref numbering.

Evidence in the committed dumps

This is not hypothetical — it is losing content in fixtures already in the repo.
Counting only subtrees rooted at an element simplifyHtml does not recognise:

dump	unrecognised roots	refs lost inside them	non-ws chars lost
serp-bing	acf-button-standard ×20, acf-skeleton ×7, acf-icon, acf-dropdown, acf-thumbs-up-down-feedback, data ×3, video, template, iframe ×2	13	475
serp-brave	svelte-css-wrapper ×3	3	0
serp-google	g-snackbar, g-loading-icon, srpx-bugfix	0	31
bbc-news	next-route-announcer, iframe ×4	0	68
serp-duckduckgo	picture, iframe	0	18

(svg/path/… are excluded from the table — those are dropped deliberately,
resolveInlineGraphics handles the named ones earlier in the pipeline.)

The clearest one is Brave. Three <button>s sit inside
<svelte-css-wrapper style="display: contents; …"> — a wrapper that by
definition generates no box and whose children should simply be promoted. In
build/renderedMarkdown/serp-brave.md the refs read
… 15 16 17 _ 19 _ 21 _ 23 24 25 …: 18, 20 and 22 are missing, exactly the
three buttons. Bing loses 13 refs the same way, most of them <button>s inside
<acf-button-standard> (its design-system web components), including a
title="Copy code" control.

<picture> is the one that will bite hardest in the wild: every
<picture><source …><img …></picture> — the standard responsive-image idiom —
drops its <img>. The DuckDuckGo dump happens to carry an alt="" image so
nothing visible is lost there, but on an image-heavy page it would be.

Reproduction

@Test
fun `unrecognised element keeps its content`() = runTest {
    val markdown = semanticEvents(tagged = true) {
        "body" {
            "my-widget" {
                "h2" { +"Heading inside a web component" }
                "p" { +"paragraph inside a web component" }
            }
        }
    }.transformHtmlToMarkdown().renderMarkdown()

    // ACTUAL: "" — the entire subtree is gone
}

Verified dropped-with-subtree today (control: <aside> survives):
picture, caption, dialog, menu, ruby, slot, marquee,
selectedcontent, and any custom element / web component.

<caption> deserves a call-out of its own: it is unwrapped inside a layout
table by applyAccessibility, but on a data table it survives that operator
and then dies here — so a real table silently loses its caption.

Recommended fix

Add a catch-all that unwraps rather than swallows:

match("*") { children() }

placed last, so every existing rule still wins. It is robust to the open set of
custom elements (no allowlist can enumerate acf-* / g-* / svelte-*), and
display: contents wrappers are precisely what unwrapping is for.

The catch is the coupling with SVG, and it should be decided together with
the change: SVG subtrees currently vanish because they are unmatched. With a
catch-all they would spill <path>/<defs>/<symbol> noise into the output
(109 svg marks in the DuckDuckGo dump alone), so the catch-all must land
together with an explicit drop set — the SVG tag family plus template,
iframe, video, audio, canvas, object, embed — added to
DROPPED_TAGS. Every dump golden changes with this, so it wants its own PR.

A narrower alternative, if the allowlist model is worth keeping: add the missing
HTML5 elements explicitly and treat "name contains -" (the custom-element
naming rule) as unwrappable. That fixes the web components and <picture>
without touching the SVG behaviour, at the cost of leaving the next unlisted
standard element to be found the same way this one was.

Note

CLAUDE.md currently states that "raw <script>/<button>/<form> etc.
survive as passthrough since simplifyHtml only drops/unwraps tags it
recognizes". That reads as "unrecognised tags pass through", which is the
opposite of what happens; the entry has been amended.