Read a paper's own maths: stop the hang, expand its TeX shorthands - #151
Merged
Conversation
…th a comma Importing a paper whose figure caption runs a long sentence through an inline equation hung the tab outright: the splitter walks 280-character windows looking for a comma to break on, backs away from any that falls inside an equation, and reaches for the next one with `lastIndexOf(token, index - 1)`. At index 0 that searches from 0 rather than before it, finds the same comma again, and never stops — a synchronous spin the browser cannot be talked out of. The window lands on an equation's first character whenever a hard cut falls inside that equation, so a caption written as "…an author sees$, which is why…$" reaches the state on the first pass. Once there is nothing left of index 0, there is no break. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A paper published as HTML renders its own maths in the browser, and
hands KaTeX a table of the shorthands its author writes in — \idx for
\phi, \R for \mathbb{R}, \Rot{\theta} for \mathbf{R}_{\theta}. That
table lives in the page's script, which extraction drops, so every
custom command reached the reader as literal \idx: unrendered on the
page and read aloud as raw source.
The table is now read off the page's inline scripts (KaTeX and MathJax
forms alike, as data — never evaluated) and expanded into the maths of
the extracted markdown, before anything downstream sees it. Prose and
fenced code are left alone, where a lone \R belongs to a Windows path
rather than to the page's macros.
Figures with a caption written in TeX also needed rescuing. Defuddle
reads an alt matching \command as a rendered equation image, so folding
such a caption into an empty alt dropped the picture and re-emitted the
caption as one enormous bogus formula — the very paragraph that hung the
sentence splitter. Left inside its figure the caption fares no better:
the figure is rebuilt from plain text, losing the emphasis, the links,
and the delimiters that made the equations equations. Those captions now
move out to a paragraph of their own. Captions with no TeX in them, and
Wikipedia's output byte for byte, are untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Importing https://neovand.github.io/Moire/paper/index.html hung the tab hard enough to need a force quit, and everything that survived the hang read its equations as raw source. Three defects, all reproduced against the live page.
The hang
splitLongSentencewalks a long sentence in 280-character windows looking for a,to break on, backing away from any candidate that falls inside an inline equation. When the last remaining candidate sits at index 0,candidate.lastIndexOf(token, index - 1)clamps its start position to 0, finds the same comma again, and loops forever — a synchronous spin on the main thread, so no error, no timeout, nothing to interrupt it.The window begins on an equation's first character whenever a hard cut lands inside that equation, so a caption written as
…an author sees$, which is why…$reaches the state on its first pass. Delta-debugged down to three lines; the regression test sweeps opener lengths across the window boundary and hangs on the old code.The page's TeX shorthands
A paper published as HTML renders its own maths in the browser and hands KaTeX a macro table —
\idx→\phi,\R→\mathbb{R},\Rot{\theta}→\mathbf{R}_{\theta}. That table lives in the page's script, which extraction drops, so every custom command arrived as literal\idx: unrendered on the page, and read aloud as raw source.tex-macros.tsreads the table off the page's inline scripts (KaTeX and MathJax forms alike — bare keys, single quotes,[body, arity]values) and expands it into the maths of the extracted markdown, before parsing, so the equations render, the deterministic verbaliser reads them, and the language model rewriting them sees ordinary TeX. A macro table is data and is never evaluated; sizes, expansion depth and total substitutions are all bounded. Expansion is confined to maths regions and skips fenced code — in prose a lone\Rbelongs to a Windows path.Figures with maths in the caption
The DOM prepass folds an empty image
altdown from its figcaption so the narration layer has something to describe. Defuddle reads an alt matching\commandas a rendered equation image: it dropped the picture and re-emitted the caption as one enormous bogus formula — the very paragraph that hung the splitter. Left inside its figure such a caption fares no better, since the figure is rebuilt from plain text, losing the emphasis, the links, and the delimiters that made the equations equations.Those captions now move out to a paragraph of their own, and the alt is left empty. Captions with no TeX in them are untouched, and Wikipedia's extraction is byte-for-byte identical.
Verified on the live page
Driven through the real production path in a real browser, not a harness.
Fig. 4's caption now renders its bold, its links and its four equations, with no duplicate. Wikipedia (Quantum mechanics) extracts identically to before.
npm run lint,npm run check, 551 unit tests, 36 e2e and 12 visual snapshots across three viewports all pass;tex-macros.tsis on the coverage include-list at 97% lines.🤖 Generated with Claude Code