docs(site): make the llms.txt artifacts discoverable, and fix the landing-card parser

[bloxster]

Makes the generated llms.txt artifacts discoverable, and fixes the landing-page card parser that feeds them.

Discoverability — advertises /llms.txt with rel="describedby" (the relation the llms.txt v2 proposal defines for a file describing a page; alternate/text/markdown is reserved for a page-specific Markdown representation). The descriptor is scoped away from archived doc versions. llms-full.txt stays reachable through llms.txt, the visible docs, and the sitemap. Adds an MCP page describing both artifacts.

Card parser (docs/site/scripts/generate-llms.py) — landing pages are JSX grids, and the generator has to tell code apart from page structure before flattening them. Fixes, each with a regression test:

• A fence opening on a container marker's own line (> ```json, - ```json) is recognized, and its closer is matched at the opener's blockquote depth — a blockquoted line does not close a root fence, and a list-item fence closes on its marker-less closer instead of running to end of file.
• A marker followed by five or more spaces holds an indented code block, not a fence.
• Container columns come from live structure only. A marker inside a fence or a JSX comment no longer leaves a stale column that makes a later root-indented backtick line look like an unterminated fence.
• Comment detection resolves backtick-versus-comment precedence through _inline_regions, so a literal {/* inside a code span opens nothing.
• A code span is bounded by any block boundary, not just a blank line: ATX heading, setext underline, fence, thematic break, blockquote, list item (ordered only at 1), and a line-initial JSX element. A fence four or more columns in does not bound one.
• Tag stripping scans quoted attribute values, including backslash escapes, so a > inside one no longer ends the tag.
• The expected card count comes from the class-keyed signals rather than the wrapper match, so a grid whose wrappers all fail to parse trips the guard instead of falling back to strip_mdx silently. The guard raises rather than logs, since --check compares generated output against committed output and would not catch a systematic regression.

The committed artifacts are unchanged by the parser fixes; they close latent cases, not current output.

[Copilot]

Pull request overview

Improves discovery of Erigon’s LLM documentation artifacts and fixes landing-card extraction.

Changes:

• Advertises artifacts through page metadata, sitemap entries, and documentation links.
• Parses landing cards within individual <Link> boundaries.
• Adds regression tests and regenerates the full corpus.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
llms-full.txt	Updates the repository corpus.
docs/site/static/llms-full.txt	Updates the deployed corpus.
docs/site/scripts/test_generate_llms.py	Adds parser regression tests.
docs/site/scripts/generate-llms.py	Fixes card parsing and adds validation.
docs/site/docusaurus.config.ts	Adds head and sitemap discovery.
docs/site/docs/get-started/why-using-erigon.mdx	Links to LLM artifacts.
docs/site/docs/fundamentals/mcp.mdx	Documents artifact usage.

Suppressed comments (1)

docs/site/docusaurus.config.ts:177

• The MCP page now links both artifacts, so “Nothing else on the web links to them” is inaccurate. The relevant rationale is that static files are omitted from Docusaurus's default sitemap.

          // The llms.txt artifacts live in static/, so Docusaurus never routes
          // them and the default sitemap omits them. Nothing else on the web
          // links to them either, which leaves them unindexable and unreachable
          // by search — append them explicitly.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[yperbasis]

Requesting changes for three correctness issues:

1. docs/site/docusaurus.config.ts: advertise /llms.txt with rel="describedby". The llms.txt v2 proposal reserves rel="alternate" type="text/markdown" for a page-specific Markdown representation and defines rel="describedby" for the llms.txt file covering a page: https://llmstxt.org/#proposal. These site-wide aggregate files are not alternate representations of every page, and v2-aware agents may specifically look for describedby. Keep llms-full.txt discoverable through llms.txt, the visible docs, and the sitemap instead of declaring it a page-wide alternate.

2. docs/site/scripts/generate-llms.py: the mismatch guard is bypassed when every card fails extraction. if not cards: return None runs before expected is computed, so collect_pages silently falls back to strip_mdx. Compute expected before the early return and add an all-malformed-grid regression test. This matches the existing unresolved thread: #23336 (comment).

3. docs/site/docs/fundamentals/mcp.mdx: llms-full.txt is advertised as containing every documentation page in full, but collect_pages replaces the complete body of every card-grid page with the synthesized card list. For example, the generated Why using Erigon entry omits its introduction, benefits prose, and MCP section. Preserve the non-card prose or describe this as a cleaned and synthesized corpus instead of claiming complete page contents.

Reviewed at eee030a9f1c37e951048dc8420c5af6d61fcce63. The 81 documentation-script tests, artifact check, diff check, and GitHub docs build are green.

[bloxster]

@yperbasis all three addressed in 1dd741e, rebased onto your merge of main.

1. rel="describedby" — you're right, and I checked the proposal rather than taking it on trust: it defines describedby for the llms.txt file covering a page and reserves alternate + text/markdown for a per-page Markdown representation. A site-wide index is not an alternate representation of every page. Switched to describedby, and dropped the llms-full.txt head tag entirely — it describes no single page, so it has no business in a per-page relation. It stays reachable via llms.txt, the sitemap, and the MCP page.

2. Guard hole — confirmed before fixing. With every card malformed, cards was empty, return None fired before the count, and collect_pages fell back to strip_mdx: total parser failure was indistinguishable from a prose page, which is precisely what the guard was meant to catch. Now counts first, parses second, and returns None only when the page has no cards. New test verified to fail against the old ordering.

3. "Every documentation page, in full" — confirmed and materially false: why-using-erigon.mdx is 183 source lines and reaches the corpus as its 11-bullet card list alone, losing the introduction, the prose and the MCP section. The table now says "the text of every documentation page", with an explicit sentence that the corpus is cleaned rather than verbatim and that card-grid index pages are reduced to their links.

Verified after the rebase: --check green, 82 tests pass, build clean, describedby on 198/199 pages, zero alternate/text/plain tags and zero llms-full head tags remaining, sitemap still carries both files.

One thing worth flagging: #23335 already merged into release/3.6 with all three defects. I'll open a follow-up there carrying these same fixes.

[bloxster]

Follow-up for release/3.6 opened: #23341 — same three fixes, since #23335 merged before this review landed.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

docs/site/docusaurus.config.ts:100

• The PR description still promises two rel="alternate" type="text/plain" head tags on every page, while this now emits one describedby link and deliberately omits llms-full.txt. Please either restore the advertised tags or update the description and verification so they match the shipped discovery contract.

          rel: 'describedby',
          href: 'https://docs.erigon.tech/llms.txt',

[yperbasis]

Requesting changes for three remaining correctness issues:

1. The landing-card mismatch guard still misses cards whose title marker disappears.
2. The advertised llms.txt index does not directly expose llms-full.txt.
3. The new completeness wording ignores the published archived documentation versions.

Reviewed at 1dd741e8c1. Targeted validation is green: 82 documentation tests and the llms artifact drift check pass.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (2)

docs/site/scripts/generate-llms.py:299

• This block records the old regex's failure sequence and test-level details rather than the lasting invariant, making it easy for the explanation to become stale. Keep only the card-boundary and independent-count rationale; the regression tests preserve the specific history.

# Parsed in two stages, deliberately. A single pattern spanning the whole card
# cannot express "and never cross into the next card": with `[^<]+` for the text
# and `.*?` for the gaps, a description containing inline markup (`<strong>`,
# `<code>`) fails to match locally, and the engine then scans forward and pairs
# the title with the *next* card's description — silently swallowing the card in

docs/site/docusaurus.config.ts:100

• The PR description and verification still promise two rel="alternate" type="text/plain" tags for both artifacts, but this implementation intentionally emits one rel="describedby" link for llms.txt. Please update the PR description and verification claims (including the parser guard's stated count source) to match the final implementation, or restore the advertised behavior.

          rel: 'describedby',
          href: 'https://docs.erigon.tech/llms.txt',

[yperbasis]

Requesting changes for three remaining correctness issues and one repository-guideline issue.

Reviewed at a97ed68440. Targeted validation is green: 84 documentation tests and the llms artifact drift check pass; the GitHub docs-site build is also green.

[bloxster]

Head moved to 0fc5956e3d — main merged in, since #23531 landed the w35 disk-size refresh and conflicted this branch.

The only conflict was positional: main had gained the semantic-version release selection and the currentDocsVersion block, and both sides added declarations directly below archivedVersions. Kept both. nonDocumentRoutes, customFields and ignorePatterns came through intact, and src/releases.test.js from main still passes here.

Artifacts regenerated against the merged docs — 72 pages now rather than 73, which is main's page count. Re-verified on the merged tree: 120 script tests, four-artifact drift check, tsc, and a full build. The descriptor lands on exactly the 72 built pages that llms.txt indexes; the 73rd index entry is llms-full.txt, which is a static file rather than a page, so it correctly has none.

docs-site / build and ci-gate are green on this head.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

[yperbasis]

Requesting changes for four parser-correctness issues and one repository-guideline issue.

1. [P2] Scope the structural card counter to landing pages. _card_shaped_link_count makes expected=1 for an ordinary <Link to="/details"><div className="callout">...</div></Link> on any documentation page. synthesize_landing then raises parsed 0 of 1 instead of returning None, so valid non-landing MDX fails the documentation build.

2. [P2] Mask JSX expressions before structural scans. _code_masked blanks code and JSX comments but leaves expression strings visible. A valid card description such as Use {"</div>"} literally, then keep this text. renders completely in MDX, but _div_span_end treats the string as a real closing tag and emits only Use {". All five counts still agree, so the guard does not detect the truncation. The extraction path must handle the same expression regions.

3. [P2] Recognize fenced blocks nested under Markdown containers. _FENCE_OPEN_RE accepts only zero to three spaces at the raw line start, but the installed MDX compiler accepts a four-space-indented fence under a list. With a blank line after that opener, a fenced lp-card example remains visible to every structural scan and is emitted as a real ## Sections bullet inside a dangling fence. The repository already uses four-space-indented fences.

4. [P2] Preserve the rest of a hero block. _landing_hero emits only the first paragraph while prose removes the complete hero span. A second visible paragraph, link, or call-to-action therefore disappears, contrary to the new document-order and retained-prose contract. Exclude only content that was actually emitted, or flatten the remainder separately.

5. [P3] Remove forensic narratives from test docstrings. For example, test_hero_with_inline_markup_does_not_swallow_the_page records the old regex and its exact failure sequence. The repository guidelines require docstrings to state the invariant and keep this history in the commit or PR description.

The existing tilde-fence thread at discussion_r3851145007 is also valid and unresolved.

Reviewed at 0fc5956e3dad273ad5febbf6ef8120f0a6b40fcf. The 120 documentation-script tests, four-artifact drift check, and git diff --check pass. Focused reproductions confirm the four behavior issues above, and the relevant snippets compile as valid MDX with the repository dependency.

[bloxster]

All five addressed in f594980. Both Copilot threads are fixed and resolved.

1 — shape is not a landing page. The class-keyed signals now decide whether a page is a landing page; the shape-only signal only cross-checks the count once one of them has fired. <Link to="/details"><div className="callout">…</div></Link> on a prose page returns None again. A rename of lp-card under an intact grid still trips the guard; renaming the whole namespace now reads as prose, and --check catches that because the corpus changes. I replaced test_wholesale_namespace_rename_still_trips_the_guard accordingly rather than leave a test asserting the old contract.

2 — expressions masked. The structural mask blanks {…} as well, so a quoted </div> no longer bounds a field early. _inline_regions deliberately does not mask them, because the text an expression renders still belongs in the corpus: MDX renders {"</div>"} as the characters </div>, so _card_text holds that text aside before the tag passes. Your example now emits Use </div> literally, then keep this text. in full.

3 — indent cap removed. This one has an ordering constraint worth recording. Removing the cap had to come before unifying the fence scanners: with the cap in place, unification reclassified this tree's 36 list-nested four-space fences as ordinary text, which stripped <your-datadir> out of a jwt-secret example and emptied a --data '{…}' payload in the corpus. I only found that by diffing the regenerated artifacts, not from the tests. Both changes are in this commit, in that order.

4 — hero. _landing_hero returned only the first paragraph while prose dropped the whole span. It now emits every part of the block, the <h1> aside since the page title carries it, with non-paragraph content flattened separately as you suggested.

5 — docstrings. Narrative removed from test_hero_with_inline_markup_does_not_swallow_the_page and from one I had added myself.

The tilde thread is fixed by the same scanner unification as point 3. The older closing-fence thread was already fixed in 32b1c5e0; I have replied on both and resolved them.

130 script tests, four-artifact drift check, ruff and git diff --check green. All four artifacts are byte-identical and the mask preserves offsets across all 199 md/mdx files in the tree, so nothing shipped moves.

Two things I did not fix, so they are not silently pending. body.count('<div') still counts a <divider prefix, which over-counts and therefore fails loud rather than dropping cards. And repeated unclosed inline constructs are quadratic — a 180 KB page of backticks takes ~21 s and compiles fine in MDX, so it is a plausible way to stall the docs build. Both predate this branch; happy to take either here or separately, whichever you prefer.

[bloxster]

@yperbasis re-requested — current head is f594980, and your review above was against 0fc5956e, three pushes back.

Since my last comment: I ran a full docusaurus build and used it to check the descriptor claim properly rather than assert it. rel="describedby" is on exactly the 72 pages llms.txt indexes — I compared the two sets from the built HTML, and they match with nothing on either side. Archived versions, /search and /404.html carry none. The PR description is rewritten to match what the code actually does now; the old one still described the count guard's previous shape and claimed 73 pages.

All five points and both Copilot threads are done and resolved. Two things I deliberately left, noted at the end of the description rather than left silent: <divider counts as a <div (over-counts, so it fails loud), and repeated unclosed backticks are quadratic. Both predate this branch — happy to take either here or separately.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

docs/site/scripts/generate-llms.py:462

• _jsx_expr_end skips block comments but not valid JavaScript // comments. A } inside a line comment is therefore treated as the expression close, leaving card-shaped JSX later in the same MDX expression unmasked; it can then be parsed as a real card or inflate the count guard. Skip line comments before counting braces and add a regression case.

        if text.startswith('/*', i):
            close = text.find('*/', i + 2)
            if close == -1:
                return -1
            i = close + 2

[yperbasis]

I found two issues that should be fixed before merge:

• [P2] Scope the shape-only count to card grids (docs/site/scripts/generate-llms.py:700)

  _card_shaped_link_count(mask) scans the entire page. Once a real lp-grid makes named nonzero, an unrelated <Link><div>…</div></Link> call-to-action in the hero or surrounding prose increases expected. A one-card grid followed by that normal CTA raises parsed 1 of 2 expected, so the artifacts cannot be regenerated. The preceding logic already identifies this as the CTA idiom, but the gate only protects pages that contain no cards. Please scope this count to the matched grid spans, or otherwise exclude non-grid links, and add a mixed grid-plus-CTA regression test.

• [P2] Preserve CommonMark indentation semantics when detecting fences (docs/site/scripts/generate-llms.py:110)

  ^[ \t]* treats a backtick run at any indentation as a fence. At document root, however, four spaces make it an indented code block, not a fenced block (CommonMark section 4.5). With a literal four-space-indented backtick fence marker followed by a live card grid, this scanner treats the line as an unterminated fence through EOF. The live grid is masked, every count signal disappears, and regeneration quietly emits raw card markup instead of the synthesized list. Please make fence recognition container-aware: allow deeper raw indentation only after accounting for a list or blockquote container, while retaining the three-space limit at document root, and add this case as a regression test.

[bloxster]

Both fixed in b0bcd78 and 02fc390.

Scope the shape-only count. Not scoped to the grid spans: that keys it to lp-grid and undoes the earlier round's point about a rename zeroing every signal at once. I took the other option you left open and excluded non-grid links by shape. A card carries a title and a description, so two boxes; a call to action carries a label, so one. _card_shaped_link_count now counts <Link> tags holding more than one <div>. It reads no class name, so a renamed card is still cross-checked against its siblings, and a call to action stays out of expected whether it stands alone or in a row — arity alone would not do, since buttons come in pairs too and a Get Started / GitHub row beside a grid would still abort at parsed 2 of 4.

The docstring's claim was also wrong and is corrected: named == 0 returns before expected is computed, so a wholesale rename reads as prose and --check is what catches it. There is already a test asserting that.

Container-aware fences. _fence_open_lines tracks a stack of container content columns from list and blockquote markers and accepts a fence at most three columns past the innermost one. The 34 four-space fences in this tree all sit inside list items at content column 4, so they still open; a four-space run at document root no longer does.

Three details, each with a regression test:

• Containers close only at a block boundary. Popping on any dedent broke lazy continuations (CommonMark 5.2): 1. Create the config file wrapping to column 0 closed the item, the fence below stopped being a fence, and strip_mdx read the unfenced {...} as a JSX expression block and dropped the body — silently, and --check is invariant under it once committed. That is this tree's authoring style, so one wrapped sentence would have triggered it.
• A thematic break borrows the list markers but opens nothing. - - - pushed a content column of 2, re-legitimizing a four-space backtick run one line of context away. Same for a blockquote, which ends at a blank line.
• The gap after a marker is measured from the column after it, so a tab lands on the next stop from there and not from the margin, and both patterns tolerate a trailing \r like the rest of the line-anchored ones.

Left alone: _FENCE_CLOSE_RE still accepts any indentation, so a four-space ``` inside a root-level fence ends the region where CommonMark keeps it as content. It predates this PR and neither issue here needs it — happy to fold it in if you want. Related, and the reason the root-level fence test needed a particular fixture: indented code blocks are not masked at all, only fenced code, inline spans and JSX comments, so markup inside a four-space block is read as live structure.

Validation at 02fc390: 91 documentation-script tests, the four-artifact drift check green at 72 pages, git diff --check clean.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

[yperbasis]

Requesting changes for four corpus-correctness issues. The committed artifact check, all 144 documentation-script tests, Ruff, and git diff --check pass, but four focused valid-MDX regression cases fail at this head.

[yperbasis]

[P2] Recognize fences after Markdown container markers. _fence_marker receives the raw line, so > ```json and - ```json are not marked as fenced. strip_mdx then treats the payload as live MDX; in a direct reproduction the blockquoted {"jsonrpc":"2.0"} line becomes empty, and card-shaped JSX may be synthesized. Parse the container prefix before testing the fence and cover both blockquote and same-line list forms.

[bloxster]

Fixed in bf75b48. Reproduced both forms first — the blockquoted {"jsonrpc":"2.0"} line came back empty exactly as you described.

_peel_containers now blanks each container marker out to the column its content starts at, so the fence test sees the line MDX sees while absolute columns stay valid for the indent comparison. Both the blockquote and same-line list forms are covered.

Two things fell out of it that are worth flagging:

• A closer must be matched at the opener’s blockquote depth, not its marker count. Blockquotes continue by repeating >; a list item continues by indentation and its closer carries no marker. Counting markers left a - ```json fence open to end of file, masking every live line below.
• A blockquote with no space after > is still not recognized (_CONTAINER_RE requires whitespace or EOL). That is pre-existing and I left it alone: a bare > at line start is also how a multi-line JSX tag closes, so recognizing it risks false positives on these pages. Happy to take it separately if you want it.

[yperbasis]

[P2] Make tag stripping quote-aware. [^>]* stops at a > inside a quoted JSX attribute. A valid description Before <span title="a > b">inside</span> after. becomes Before b">inside after. while every card count still passes. Scan tags with quoted values correctly rather than terminating at the first greater-than character.

[bloxster]

Fixed in bf75b48. Your example produced Before b">inside after. as written.

Tag matching now scans quoted attribute values, including backslash escapes, so it ends at the first > outside a value. The bare-<-in-prose behaviour is unchanged and still tested.

One measurement worth recording: the new pattern is the same complexity class as the old [^>]* on unclosed-tag input but with a roughly 30× constant. I checked reachability rather than optimising blind — the largest lp-card-desc field in the corpus is 201 bytes and the full generator runs in 0.28s, so it needs tens of KB inside a single card field to matter. I did not rewrite the matcher into a scanner here; say the word if you would rather have that than the regex.

[bloxster]

Proposal: read page text from the built site instead of parsing MDX

I've moved this to draft because I think your four open P2 threads point at something the current approach can't fix, and I'd rather put the alternative in front of you than send another round of patches.

Why

The generator reconstructs what MDX renders by hand — CommonMark block parsing (fences inside containers, code spans vs block boundaries, HTML blocks, setext underlines, indented code blocks, closer depth) plus JSX — to recover the landing-page card grids. Every review round on this PR has found another rule it gets wrong. That isn't bad luck: the surface is a spec being reimplemented, so there is no round at which it is finished.

Docusaurus has already done that work by build time. Components are expanded, links are resolved to absolute paths, build-time variables are substituted, and the page body sits in one <article>.

So the four threads become moot rather than fixed: the code they refer to is gone.

What it measures

Against the previous committed output, 72 pages both sides, none dropped:

	before	after
generate-llms.py	1328 lines	863 lines
{ERIGON_VERSION} left unsubstituted	16	0 (real version)
relative link targets, meaningless in a flat file	31	0
FAQ emitted twice (page prose + its JSON-LD copy)	yes	no
inline <script> and JSX prop objects leaking as text	yes	no
Array<Object>, <CL_host>	eaten as HTML tags	restored
content lost	—	none found in 2518 compared prose lines

llms.txt is byte-identical: the index comes from frontmatter and never used this code. Only llms-full.txt page bodies change.

Chrome subtrees (breadcrumbs, mobile TOC, footer, heading anchors) are dropped by their stable theme-* class tokens. Tables, code fences, lists, emphasis and links render as Markdown.

Costs, stated plainly

• The artifact check now needs a build. generate-llms.py --check reads docs/site/build, so it runs after npm run build. Three workflows were reordered: docs-site-build.yml, docs-version-bump.yml, create-release-branch.yml. Regenerating locally means building the site first.
• New coupling to theme class names. theme-* are the documented-stable ones, and a test fails if any entry in the chrome list stops matching real output. Worth knowing: theme-doc-markdown is the content wrapper, so a prefix match on theme-doc- would delete every page.
• Mermaid is the one thing that only exists in source — Docusaurus renders it client-side, so its fence is absent from built HTML. It is spliced back from the MDX by an ordered fence walk. Position is not preserved; the diagram is appended to the page. One diagram in the corpus today.
• Known and not fixed, all measured at zero occurrences in the current corpus: <pre> inside a table cell, nested <a>, interior multiple spaces inside an inline <code>, a mermaid fence indented under a list item. Underscore-prefixed partials would hard-fail generation (none exist).

On the review history

I put this through several independent reviews before proposing it, and they found real defects in my own rewrite, including two that were briefly in the committed artifact: every code fence double-spaced (Docusaurus wraps each code line in a <div> that also ends in <br>), and chrome stripping silently disabled on all 72 pages because my class list said breadcrumb where the real token is breadcrumbs.

Both slipped through for the same reason, and it is the part I'd flag hardest: the tests were synthetic, and --check compares generated output against output from the same code, so neither could catch that class of defect. The suite now carries fixtures captured verbatim from a real build, and the script tests run after the build so the built-site tests actually execute instead of skipping.

66 tests, ruff clean, --check green before and after a rebuild.

What I'd like from you

Mainly whether you see a reason to keep parsing source that I'm missing — a case where the built HTML is the wrong input. If you'd rather not take the build dependency, that's a fair reason to say no, and I'll go back to fixing the four threads on the current parser instead.

If the direction is fine, the sequencing question is whether this should land on main first or go straight to release/3.6, since DOCS_DEPLOY_BRANCH is release/3.6 and nothing here reaches docs.erigon.tech until it's ported.

---

The implementation is ready but I have not pushed it — this branch still holds the parser fixes you last reviewed, and I didn't want to overwrite those before you've weighed in. Say the word and I'll push it here for you to read as a diff.