Skip to content

whatif: measured speed-ups, and benchmarks on ten documents - #2

Merged
kevinraymond merged 3 commits into
mainfrom
whatif
Nov 10, 2026
Merged

kevinraymond merged 3 commits into
mainfrom
whatif

Conversation

@kevinraymond

Copy link
Copy Markdown
Owner

For 0.2.0, before the tag.

prumo whatif

prumo whatif main.tex [--try preamble,draft,externalize] [--runs 5] [--shell-escape] rebuilds the document with one change at a time and times it against the plain build:

  • preamble precompiles the preamble with mylatexformat. pdfTeX only: XeTeX cannot dump the class's OpenType font, and LuaTeX's dump fails or the build on it does. The row says which, with the error.
  • draft builds without a PDF (-draftmode, -no-pdf on XeTeX), which is what writing the PDF and the images cost.
  • externalize turns on tikz's external library from the command line. Experimental, and refused without --shell-escape. The hook goes to the picture builds as well; without it each picture build typesets the whole document, and that is where 2 pages becoming 3 came from.

The builds run in a temp copy without the probe, one at a time (parallel builds slow each other down), with a warm-up each and then the baseline and the variants in turns. The table compares medians and calls anything inside the spread (2 ms at least) noise. The probe's bound sits beside each saving, and one-time costs (the format build, the first externalized run) are their own column. The output is compared by page count, the words on each page and a 24 dpi render. pdftotext -layout alone flagged an externalized picture whose text moved a column, so it is not enough. --json is schema 1.

Also in here: a mylatexformat row in doctor, doctor --check-log telling an existence check from a read outside the hooks, a whatif fixture with eight claims (all tested), mylatexformat in the TeX image, and make_artifacts.sh taking fixture names.

Benchmarks

Ten documents now. Seven of them are fetched at a pinned commit and never stored here: the ACM sample (BibTeX and biblatex), the metropolis demo, florilegium's manual, the HoTT book, Open Logic and the 7,710-page Stacks Project. bench.py measures the probe's cost (builds with and without it, in turns) and runs whatif on all of them. charts.py draws four charts from the JSON, light and dark. kaobook is left out because thmtools' sibling= fails on the 2026-11-01 kernel.

What came out of it:

  • The probe costs 7 ms on a one-line document and under 6% on everything here.
  • None of the three experiments is a default. The precompiled preamble saves the most where it works (49% on the paper) but works on two of the six pdfLaTeX documents. Draft mode saves 13 to 38% on XeLaTeX and little elsewhere. Externalize saves 15 to 32% on the deck and florilegium for a first run of seconds. On the books that first run takes 31 to 119 minutes for 1 to 13%.
  • On the paper, LuaLaTeX's extra preamble time is biblatex (+151 ms) and fontspec's LuaTeX code (67 ms).

Tested

cargo test, clippy and fmt. The TeX tests in the image, the new fixture's byte-identical PDF included. whatif on the fixture on TeX Live 2021 too.

`prumo whatif` rebuilds the document with one change at a time and times it
against the plain build. The changes are the preamble precompiled with
mylatexformat, draft mode, and tikz's external library (experimental, and only
with --shell-escape). The builds run in a temp copy without the probe, one at
a time: a warm-up each, then the baseline and the variants in turns. The table
compares medians, calls anything inside the spread (2 ms at least) noise, puts
the probe's bound beside each saving, and compares the output by page count,
the words on each page and a 24 dpi render.

The externalize line hands the hook to tikz's picture builds too. Without it
each picture build typesets the whole document, which is where a page count
going from 2 to 3 comes from. XeTeX and LuaTeX cannot precompile a preamble,
and the row says why.

doctor has a mylatexformat row. --check-log now tells a file opened only to
test that it exists from one read outside the hooks (siunitx with tikz opens
color.sty and never reads it). The new fixture has eight claims, all tested.
The TeX image installs mylatexformat, and make_artifacts.sh takes fixture
names.
Seven public documents join the three that live here: the ACM sigconf sample
with BibTeX and with biblatex, the metropolis demo, florilegium's manual, the
HoTT book, Open Logic and the Stacks Project. corpus.toml pins each one to a
commit and fetch.py takes it from its own repository, so nothing under
share-alike or GFDL terms is stored here. kaobook is left out, because
thmtools' sibling= fails on the 2026-11-01 kernel.

bench.py gains --overhead (builds with and without the probe, in turns) and
--whatif. charts.py draws four charts, light and dark, from the JSON: where the
time goes, what the probe costs, package times on pdfLaTeX against LuaLaTeX,
and what whatif saves. Color means phase in all of them, and every pair of
colors is checked for colorblind readers.

The benchmark README has the charts with their tables. The top-level README
links it and says where the precompiled preamble fails. Findings 60 to 64.
Rebuilt alone, the fixture compiled Latin Modern 12 into luaotfload's cache and its
.fls listed that. In the full run CI makes, the biblatex fixture's latexmk passes have
compiled those fonts already, so CI's .fls has no such lines and the shape check failed.
make_artifacts.sh says so now.
@kevinraymond
kevinraymond merged commit 91efe53 into main Nov 10, 2026
4 checks passed
@kevinraymond
kevinraymond deleted the whatif branch November 10, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant