diff --git a/README.md b/README.md index 3bea1a6..3594767 100644 --- a/README.md +++ b/README.md @@ -16,9 +16,8 @@ I had no way to see why. Commenting packages out one at a time is not insight. p is one instrumented build that shows where the time goes, which package loaded which, and what you can drop. -It has been run on my own corpus and on a few public documents (a thesis template, a beamer -deck, a 76-page arXiv paper; see [the benchmarks](docs/benchmarks/README.md)). Please report -what breaks! +It has been run on my own corpus and on public documents from an ACM paper to the 7,710-page +Stacks Project (see [the benchmarks](docs/benchmarks/README.md)). Please report what breaks! This project was made with the assistance of AI (Claude). @@ -63,6 +62,7 @@ prumo prune docs/ --engines pdflatex # corpus-wide keep list and tlmgr remove pl prumo prune docs/ --apply --yes # run the plan; every tlmgr command is logged prumo unused main.tex # which \usepackage lines can go: remove the line, rebuild, compare prumo unused obra.cls --corpus 'docs/**/*.tex' # a class judged against every document that uses it +prumo whatif main.tex --shell-escape # measured speed-ups: precompiled preamble, draft mode, externalized tikz pictures prumo time main.tex --env tblr,minted # which environments to time (default: tikzpicture, axis, tcolorbox, tblr, longtable, tabularx, minted, lstlisting, forest) prumo time output.prumo # records from a build prumo did not run (Overleaf: docs/overleaf.md) prumo doctor --check-log main.tex # files that bypassed the hooks; log vs records @@ -148,6 +148,54 @@ row. `prumo compare main.tex --runs 4` on the same paper: note: xelatex's first run took 4050 ms (font cache build); the reported run is warm. ``` +## What `whatif` measures + +`prumo whatif` rebuilds the document with one change at a time and times it against the +plain build. Here it is on the whatif fixture: + +``` +$ prumo whatif fixtures/whatif/main.tex --shell-escape +== prumo whatif: main (pdflatex, median of 5 runs after a warm-up, 1 page, -shell-escape) + experiment baseline variant saved bound one-time output + preamble 247 ms 90 ms 157 ms 64% 207 ms 319 ms identical + draft 247 ms 244 ms within noise - - n/a (no PDF) + externalize* 247 ms 245 ms within noise 3 ms 503 ms identical +``` + +The three experiments: + +- `preamble` precompiles everything before `\begin{document}` into a format with + mylatexformat and builds on that format. The one-time column is the format build. Only + pdfTeX can do it at all. XeTeX cannot write a format once the class has loaded its + OpenType font, and on LuaTeX either the format build or the build on it fails. Even on + pdfTeX it fails on preambles that open files or play with catcodes, as it did on three of + the six pdfLaTeX documents in the [benchmarks](docs/benchmarks/README.md). The row says + which step failed, with the error. +- `draft` builds without writing a PDF (`-draftmode`, or `-no-pdf` on XeTeX). What it saves + is what writing the PDF and including images cost. There is no PDF to compare. +- `externalize` is experimental. It turns on tikz's external library from the command line, + so each picture is built once into its own PDF and included from then on. It needs + `--shell-escape`, because tikz builds each picture with a shell command. The one-time + column is the first run, which builds every picture. The external library breaks + `remember picture`, overlays and references between pictures, so look at the PDF too. + +Every build runs once to warm up, then five times (`--runs`), one build at a time and without +the probe. The baseline and the variants take turns, so drift hits them all alike. The table +compares medians. A difference no larger than the spread, the slowest run minus the fastest, +is "within noise". The output column compares the page count, the words on each page +(`pdftotext`) and a coarse render of each page with the baseline's. When they differ, the +report says the saving is not a clean one. + +The bound column is a cross-check from one probed build: the preamble phase for `preamble`, +and the time inside `tikzpicture` environments for `externalize`. The fixture's one picture +takes 3 ms, so externalizing it saves nothing. The [benchmarks](docs/benchmarks/README.md) +run all three experiments on ten documents and three engines. Watch the one-time cost. +Externalizing Open Logic's pictures saves 4 s of a 31 s build, after a first run of 93 +minutes. + +Each experiment runs in its own copy of your directory, and the formats and picture files +stay there. Under the table, `whatif` says how to make each change yourself. + ## Running from Docker, nothing installed No TeX Live? No Rust? `docker/Dockerfile.user` builds an image with a full TeX Live, Poppler, @@ -327,6 +375,7 @@ know which of its macros, that is texprof's question. `--out` says), nothing next to your document. `prune --apply` is the one destructive command, it re-prints its plan first and needs `--yes`. - Fetch anything. It reads the local `texlive.tlpdb`; no network, ever. +- Make a speed-up for you. `whatif` measures a change on a copy and says how to make it. - Prove a package unused. "No observable effect" means this document, this build, these comparisons; a package can matter for input the document does not exercise, and the report says so every time. diff --git a/docker/Dockerfile.tex b/docker/Dockerfile.tex index 899dd7a..92f2d72 100644 --- a/docker/Dockerfile.tex +++ b/docker/Dockerfile.tex @@ -22,6 +22,9 @@ RUN tlmgr install cm-super courier fontname siunitx translations csquotes \ && tlmgr path add +# mylatexformat: `whatif` precompiles the whatif fixture's preamble with it. +RUN tlmgr install mylatexformat + # Fixture builds run as the invoking user with HOME under /tmp; keep tlmgr quiet about it. ENV HOME=/tmp/home diff --git a/docs/03-architecture.md b/docs/03-architecture.md index 7f9f44d..86f7a40 100644 --- a/docs/03-architecture.md +++ b/docs/03-architecture.md @@ -24,8 +24,9 @@ `prumo run` produces the model and caches it under `//`, where `` defaults to a per-document directory in the user's cache (`run::default_out`; `prumo where`). Every report command either -uses the cache or runs the build. `prune` merges models across a corpus. `unused` is the only -command that runs more than one build. +uses the cache or runs the build. `prune` merges models across a corpus. `compare` builds once +per engine. `unused` and `whatif` rebuild the document with one change at a time; `whatif`'s +timed builds run without the probe. ## BuildModel @@ -108,6 +109,7 @@ prumo fonts [--json] prumo packages [--tlpdb PATH] [--json] # tlmgr package set, with the file that caused each prumo prune [--tlpdb PATH] [--keep LIST] [--keep-hyphenation all|used] [--apply --yes] prumo unused [--build ] [--jobs N] [--compare text|render] # phase 2 +prumo whatif [--try preamble,draft,externalize] [--runs N] [--shell-escape] [--engine E] [--json] prumo doctor [--check-log] # engines, kernel date, pdffonts, tlpdb, recorder prumo where [] [--out DIR] # the cache root, or a document's artifact directory prumo clean [] [--stale] [--yes] [--json] # list the cache (sizes, documents); delete with --yes @@ -163,6 +165,7 @@ prumo/ report/ text, json, dot renderers prune.rs corpus merge, keep set, plan unused.rs phase 2 + whatif.rs measured speed-ups: precompiled preamble, draft mode, externalized pictures tex/prumo.sty embedded via include_str! scripts/prumo_report.py reference, frozen fixtures/ documents + checked-in build artifacts for no-TeX tests diff --git a/docs/10-verified-findings.md b/docs/10-verified-findings.md index bdbc3d9..149ac65 100644 --- a/docs/10-verified-findings.md +++ b/docs/10-verified-findings.md @@ -254,3 +254,42 @@ Items 1–18 hold, with these corrections and additions: inside page 1's brackets (`[1{.../pdftex.map}]`). On a one-line document the shipout phase is 45 ms of 132 ms on pdfLaTeX, all of it page 1; how much of that is the map was not isolated. +60. **A precompiled preamble works on pdfTeX only.** mylatexformat + (`pdftex -ini -jobname=main-fmt "&pdflatex" mylatexformat.ltx main.tex`, then + `pdflatex -fmt=main-fmt main.tex`) takes the whatif fixture from 244 ms to 91 ms on TeX + Live 2026 and from 137 ms to 58 ms on TeX Live 2021, with the same text. XeTeX stops with + `! Can't \dump a format with native fonts or font-mappings.` even without fontspec: the + class's `\normalsize` loads Latin Modern as an OpenType font for the TU encoding. On TeX + Live 2026, LuaHBTeX writes the format but exits 1 after a Lua error in lua-uni-algos + (`lua-uni-stage-tables.lua:141: attempt to perform arithmetic on a string value`), and a + build on that format dies with `(pdf backend): referenced object has wrong type`. On TeX + Live 2021 the dump succeeds and the build on it dies the same way. +61. **tikz's external library from the command line.** With + `\AddToHook{package/tikz/after}{\usetikzlibrary{external}\tikzexternalize}\input{main}` + on the command line, each picture build runs `\def\tikzexternalrealjob{main}\input{main}`, + which never sees the hook. It typesets the whole document as "the picture", and a full + letter page goes where the picture was: 1 page became 2 on the test document. That is + most likely the 2 → 3 pages in the brief, whose document was not run here. Passing `system call=` with the hook in the picture's + command line fixes it. The text and page count are then identical on all three engines, + on TeX Live 2026 and 2021. tikz's default system call is pdflatex unless the driver says + otherwise, so the system call also names the document's engine. Ten pictures of 150 lines + each went from 944 ms to 138 ms, against the probe's 828 ms in `tikzpicture`. +62. **An existence check is not a read.** siunitx runs `\RequirePackage{color}` at + `begindocument/before`. When tikz has already loaded xcolor, which claims `color.sty`, + the kernel opens `color.sty` and does not read it. pdfTeX's and XeTeX's `.fls` list it, + LuaTeX's does not, and neither the log nor the records show it. `doctor --check-log` + called it "read outside the LaTeX file hooks". With a log it now lists such files as + opened but never read. +63. **`pdftotext -layout` moves an externalized picture's text.** On the benchmark paper, + externalize was reported as changing the output: line 64 of the text differed. The pages + render the same (78 of 240,000 pixels differ at 50 dpi, antialiasing) and every page has + the same words. The picture is an included PDF now, and `-layout` places its text a column + or a line away. `whatif` compares the words on each page and a 24 dpi render instead + (the render check is `unused`'s), and then catches a real one: one page of the metropolis + deck on XeLaTeX and LuaLaTeX has the same words and 0.9% of its pixels changed. +64. **thmtools' `sibling=` fails on the 2026-11-01 kernel.** `\usepackage{amsthm,thmtools}`, + `\declaretheorem[numberwithin=section]{theorem}` and + `\declaretheorem[sibling=theorem]{proposition}` stop with `Command \c@proposition already + defined` (thmtools 0.76, TeX Live 2026). The same file builds on TeX Live 2021 (thmtools + 0.72, kernel 2021-11-15). Every kaobook example fails this way, so kaobook is not in the + benchmarks. diff --git a/docs/STATUS.md b/docs/STATUS.md index 170bfb5..7899de1 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -44,17 +44,26 @@ can check it. ### Benchmarks (`docs/benchmarks/`) -- `run.sh` runs `bench.py` in the full TeX Live 2026 image pinned to one P-core of the - 13900K: four documents (one line; a five-page paper with biber, pgfplots and siunitx; - florilegium's manual; a 3,991-page generated book), three engines, a cold run with fresh - font caches and five warm runs (three for the book), medians per phase. Results and - method in `docs/benchmarks/README.md`; the widest spread is 4%. -- On the one-line document LuaLaTeX spends 235 of 315 ms starting up, 22 ms on the preamble - and 54 ms writing the PDF. On the paper it is 236, 763 and 135 of 1,392 ms (pdfLaTeX 855 - ms in all). On the book LuaTeX's shipout is 2.7 ms a page against pdfTeX's 0.7. A cold - LuaLaTeX run is 4.4 s instead of 0.3 s. -- Scale: the book's 8,669 records are read in 3 ms, and `--html` writes 1.3 MB in 8 ms. - Plain output now lists the slowest `--top` pages instead of all 3,991. +- Ten documents now, three engines. Three live here (one line, the five-page paper, the + 3,991-page generated book). Seven are fetched at a pinned commit and never stored: the + ACM sigconf sample with BibTeX and with biblatex, the metropolis demo deck, florilegium's + manual, the HoTT book, Open Logic and the 7,710-page Stacks Project. `corpus.toml` has + each one's commit, license, engines and prepare step, `fetch.py` lays them out. kaobook + was left out, because thmtools' `sibling=` fails on the 2026-11-01 kernel, in five lines + too. +- `bench.py` has three modes: the phases (as before), `--overhead` (builds with and without + the probe in turns, plus per-package times from the probed records) and `--whatif`. + `charts.py` draws four SVG charts, light and dark, from the JSON. Color means phase in + all of them, the palette checked with the dataviz validator, the gray included after you + could not tell startup from body. +- The probe costs 7 ms on the one-line document and under 6% everywhere. On the paper, + LuaLaTeX's extra preamble time is biblatex (+151 ms) and fontspec's LuaTeX code (67 ms). +- On the corpus, the precompiled preamble works on two of the six pdfLaTeX + documents (paper 49%, deck 25%). Draft pays on XeLaTeX (13 to 38%). Externalize pays on + the deck and florilegium (15 to 32%), but its first run on a long book is a pass per + picture: 93 minutes on Open Logic for 4 s a build. +- The run had seven P-cores busy at once, and its phase numbers came out 0.3 to 5% slower + than the quieter run before it. The README says so. ### Public documents @@ -81,6 +90,36 @@ and a paragraph on texprof. texprof is not a pdfTeX: it is TeX with e-TeX and Pr DVI, and could not build a LaTeX format against the 2026-11-01 kernel. `docs/02-prior-art.md` is corrected; CLAUDE.md still says "pdfTeX only". +### whatif + +- `prumo whatif main.tex [--try preamble,draft,externalize] [--runs 5] [--shell-escape]`, + from the brief. Each experiment rebuilds its own copy of the directory with one change. + The timed builds run without the probe: a warm-up each, then the baseline and the variants + in turns. Medians are compared. A difference no larger than the larger spread (2 ms at + least, the polling resolution) is "within noise". The output is compared by page count, + the words on each page and a 24 dpi render (finding 63). `--json` has the same fields + under schema 1. +- The builds run one at a time, not on `unused`'s parallel pool, because parallel builds + slow each other down. The temp copy, the timeout and the text comparison come from `run` + and `unused`. +- `preamble` works on pdfTeX (the fixture goes from 244 to 91 ms). XeTeX and LuaTeX fail, + and the row gives the error (finding 60). `doctor` has a mylatexformat row and the TeX + image installs it. +- `externalize`: the brief's command line made every picture build typeset the whole + document (finding 61). The system call now carries the hook and names the engine. The text + is identical on all three engines, and ten pictures went from 944 to 138 ms. Without + `--shell-escape` it is refused when asked for and not run by default. The table and the + README mark it experimental. +- New fixture `whatif`, eight claims, all tested. `make_artifacts.sh whatif` regenerates one + fixture now. The fixture turned up finding 62: `doctor --check-log` now tells an + existence check from a read. +- My choices: `--runs` is at least 2. The comparison uses the last run's PDF. Logs go to + `//whatif//`. With `--shell-escape` the probed run is + `--build " -shell-escape"`, so `time` rebuilds afterwards. No `--latexmk` or + `--build`, since their wall time is several passes. +- `cargo test`, clippy and fmt pass, and so do the 19 TeX tests in the image. TeX Live 2021 + runs it too (on a copy of the fixture without siunitx, which that image lacks). + ### Only if cheap - `prumo time output.prumo` and `prumo graph output.prumo` read records from a build prumo @@ -89,9 +128,9 @@ is corrected; CLAUDE.md still says "pdfTeX only". - LuaLaTeX font definitions per page: the `luaotfload.patch_font` hypothesis is wrong for `\addfontfeature`, so nothing shipped (finding 57). -**Blocked:** nothing. **Needs me:** the v0.2.0 tag. **Open questions:** should the -LuaTeX font count go in after all, through `font.nextid()`? Should `compare` default to three -runs, so it reports a median? +**Blocked:** nothing. **Needs me:** review and merge the `whatif` PR, then the v0.2.0 tag. +**Open questions:** should the LuaTeX font count go in after all, through `font.nextid()`? +Should `compare` default to three runs, so it reports a median? ## Earlier milestones, one line each diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index 4e4f6db..ea46cdd 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -1,49 +1,64 @@ # Benchmarks -Where a LaTeX run spends its time, phase by phase, for four documents on three engines. Every -number here comes from `prumo time --json`, and the script that made them is in this -directory, so you can run it on your machine and compare. +Where a LaTeX run spends its time, what prumo's probe costs, and what `prumo whatif`'s three +speed-ups save, on ten documents and three engines. Every number here comes from prumo's +own JSON, and the scripts that made them are in this directory, so you can run them on your +machine and compare. ## The documents -| | what | pages | -|---|---|---| -| `minimal` | `\documentclass{article}`, one word. The startup test. | 1 | -| `standard` | A paper: 11 pt article, fontenc and lmodern on pdfLaTeX, fontspec on XeLaTeX and LuaLaTeX, amsmath, mathtools, siunitx, booktabs, TikZ, pgfplots, microtype, hyperref, cleveref, biblatex with biber, 12 references, figures, a table, cross-references. | 5 | -| `florilegium` | The manual of [florilegium](https://github.com/kevinraymond/florilegium) at commit `833c4fc`. LuaLaTeX only, since the package is written in Lua. | 35 | -| `large` | A book generated by `gen_large.py`: 140 chapters, 2,800 sections of lipsum text, an equation and a cross-reference in each, a table every fifth section, a TikZ picture per chapter. | 3,991 | +| document (`--docs` name) | what | pages | engines | from | +|---|---|---:|---|---| +| minimal (`minimal`) | `\documentclass{article}`, one word. The startup test. | 1 | all three | here | +| paper (`standard`) | A paper: amsmath, siunitx, TikZ, pgfplots, microtype, hyperref, cleveref, biblatex with biber; fontspec on XeLaTeX and LuaLaTeX. | 5 | all three | here | +| ACM paper (`acmart`) | ACM's `sigconf` sample with BibTeX, a tagged PDF (`\DocumentMetadata`). | 7 | all three | [acmart](https://github.com/borisveytsman/acmart/tree/0537b243395b23ad85c5036929f4b1f28429b3a5) `0537b24`, LPPL 1.3c | +| ACM paper, biblatex (`acmart-biblatex`) | The same sample with biblatex and biber. | 8 | all three | the same | +| beamer deck (`metropolis`) | The metropolis beamer theme's demo deck. | 33 | all three | [mtheme](https://github.com/matze/mtheme/tree/2fa6084b9d34fec9d2d5470eb9a17d0bf712b6c8) `2fa6084`, CC BY-SA 4.0 | +| florilegium manual (`florilegium`) | The manual of florilegium, a Lua package. | 35 | LuaLaTeX | [florilegium](https://github.com/kevinraymond/florilegium/tree/833c4fc9ff776c51d0609fd6c96b4a2868806efa) `833c4fc`, MIT | +| HoTT book (`hott`) | The HoTT book. | 492 | pdfLaTeX | [HoTT/book](https://github.com/HoTT/book/tree/578b85cc8d586b1677ec4335148adeb443057d24) `578b85c`, CC BY-SA 3.0 | +| Open Logic (`open-logic`) | The Open Logic Project's complete text. | 1,013 | all three | [OpenLogic](https://github.com/OpenLogicProject/OpenLogic/tree/1e960beff9ed7835bf3e3f1335e21af3439cd107) `1e960be`, CC BY 4.0 | +| generated book (`large`) | A book `gen_large.py` writes: 140 chapters, 2,800 sections of lipsum, an equation and a cross-reference in each, a table every fifth section, a TikZ picture per chapter. | 3,991 | all three | here | +| Stacks Project (`stacks`) | The Stacks Project as one book. | 7,710 | pdfLaTeX | [stacks-project](https://github.com/stacks/stacks-project/tree/a04446e57ec1fbc252a871afcec7752fb2807b14) `a04446e`, GFDL 1.2 | + +Three of them live in this repository. The other seven are taken from their own repositories +at the commit above and are never stored here. Their licenses are their own, and share-alike +and GFDL terms do not mix with this repository's. `corpus.toml` lists every document with its +commit, its license, the engines it builds on and the step that prepares it (acmart builds +its class and samples from `.dtx` files, the Stacks Project writes its book file, the HoTT +book its version file). +`fetch.py` lays them out and keeps the clones in `~/.cache/prumo-bench`. -The first two are in `docs/`. The large one is generated at run time (`gen_large.py` is -deterministic), and the manual is taken from a florilegium clone with `git archive`. +The HoTT book builds on pdfLaTeX only (xypdf needs pdfTeX). The Stacks Project runs on +pdfLaTeX only, the engine its own build uses, because the other two would triple the longest +run. On XeLaTeX and LuaLaTeX, the ACM paper has one page fewer than on pdfLaTeX and Open Logic +two. + +kaobook was tried and left out. All of its examples fail on TeX Live 2026 with `Command +\c@proposition already defined`, and so does a five-line document with amsthm and thmtools' +`sibling=` key. thmtools 0.76 and the 2026-11-01 kernel do not get along. ## How they were run -- **Machine:** Intel Core i9-13900K, Debian 13, Linux 6.12. The container is pinned to one - performance core (`docker run --cpuset-cpus 4`); TeX is single-threaded. +- **Machine:** Intel Core i9-13900K, Debian 13, Linux 6.12. Each run is a container pinned to + one performance core; TeX is single-threaded. This set ran on seven of the eight + performance cores at once: the phases on one, the `whatif` and overhead runs on the + others. The phase numbers came out 0.3 to 5% slower than an earlier run on a quieter + machine. The `whatif` and overhead comparisons are made inside one container, build + against build in turns, so the other cores do not tilt them. - **TeX:** TeX Live 2026 from `texlive/texlive:latest` plus Poppler (the `prumo-dev:tl2026-full` image `make verify-prune` builds): pdfTeX 1.40.29, XeTeX 0.999998, LuaHBTeX 1.24.0, LaTeX 2026-11-01. -- **Each timed run** is `prumo time --rebuild --json` on a copy of the document whose `.aux`, - `.toc` and `.bbl` came from a latexmk build just before. So a run is one engine pass with - the cross-references and the bibliography in place, the way an editor rebuilds. prumo - times that process from spawn to exit. +- **Each timed run** is one engine pass on a copy of the document whose `.aux`, `.toc` and + `.bbl` came from a latexmk build just before, the way an editor rebuilds. The phases come + from `prumo time --rebuild --json`, which times the engine process from spawn to exit. - **Cold** is the first run with a fresh `HOME`: no luaotfload font name database and no fontconfig cache, as on a new machine or after a TeX Live update. The operating system's file cache is not dropped (that needs root on the host), so cold here means cold font caches, not a cold disk. -- **Warm** is five runs after that (three for the large document). Each phase is the median - of its own values over those runs, so the phase medians need not add up to the total's - median exactly. The spread row is the fastest and slowest warm run. - -To run it: - -``` -docs/benchmarks/run.sh # everything, about 30 minutes -docs/benchmarks/run.sh --docs minimal,standard # just these -``` - -`BENCH_IMAGE`, `BENCH_CPUS` and `FLORILEGIUM` (a clone's path) change the defaults. It writes -`results/results.json` (every run) and `results/results.md` (these tables). +- **Warm** is five runs after that, three for the generated book and Open Logic, two for the + Stacks Project. + Each phase is the median of its own values over those runs, so the phase medians need not + add up to the total's median exactly. ## What the phases are @@ -61,97 +76,360 @@ The timer is wall clock on all three engines, with its zero at the start of the | end-document | `\end{document}` starts | the last record, less shipouts | | finalization | the last record | process exit: closing the PDF, writing fonts, xdvipdfmx on XeTeX | -## Results +The charts fold them into four, one color each in every chart: startup (which no change to +the document touches), the preamble, the body (begin-document, typesetting and +end-document) and writing the PDF (shipout and finalization). Each `whatif` experiment +shrinks one of the last three, and its bar has that phase's color. + +## Where the time goes -Medians of the warm runs; the spread row is the fastest and slowest of them. The load average -was 2.61 when the run started, from containers on other cores. The widest spread is 4.0% of -its median (XeLaTeX on the paper); on the two long documents it is under 0.5%. + + + One panel per document, one bar per engine, cut into startup, preamble, body and writing the PDF. Short documents are mostly startup and preamble, long ones almost all body; LuaLaTeX's bar is the longest on every document. + + +
+The numbers, phase by phase ### minimal | phase (ms) | pdflatex | xelatex | lualatex | |---|---:|---:|---:| -| startup | 41.9 | 82.8 | 235.2 | -| preamble | 7.8 | 45.9 | 21.9 | -| begin-document | 1.6 | 1.9 | 2.0 | +| startup | 42.3 | 84.0 | 242.8 | +| preamble | 8.2 | 46.0 | 22.9 | +| begin-document | 1.8 | 2.0 | 2.2 | | typesetting | 0.1 | 0.1 | 0.1 | -| shipout | 45.1 | 0.2 | 35.4 | -| end-document | 0.6 | 0.5 | 0.6 | -| finalization | 38.0 | 135.5 | 19.0 | -| **total (wall)** | **134.9** | **267.4** | **314.9** | -| spread (min–max) | 134–136 | 265–269 | 311–316 | -| cold run (wall) | 134.9 | 3,188.5 | 4,416.3 | +| shipout | 45.3 | 0.2 | 35.9 | +| end-document | 0.7 | 0.6 | 0.7 | +| finalization | 41.1 | 146.8 | 19.1 | +| **total (wall)** | **140.0** | **279.1** | **324.7** | +| spread (min–max) | 137–142 | 275–285 | 322–329 | +| cold run (wall) | 144.2 | 3,273.8 | 4,629.5 | | pages | 1 | 1 | 1 | -### standard +### paper | phase (ms) | pdflatex | xelatex | lualatex | |---|---:|---:|---:| -| startup | 42.3 | 81.7 | 236.1 | -| preamble | 501.9 | 715.2 | 762.8 | -| begin-document | 49.0 | 46.8 | 52.3 | -| typesetting | 148.3 | 266.2 | 202.8 | -| shipout | 59.6 | 19.1 | 63.5 | -| end-document | 2.3 | 3.5 | 3.3 | -| finalization | 51.6 | 56.4 | 71.1 | -| **total (wall)** | **854.7** | **1,191.0** | **1,391.5** | -| spread (min–max) | 852–864 | 1,162–1,209 | 1,374–1,418 | -| cold run (wall) | 859.2 | 4,122.3 | 5,744.9 | +| startup | 43.0 | 83.8 | 247.9 | +| preamble | 499.8 | 719.9 | 780.5 | +| begin-document | 49.7 | 48.2 | 54.4 | +| typesetting | 150.6 | 270.5 | 211.0 | +| shipout | 60.5 | 31.1 | 64.3 | +| end-document | 2.4 | 3.8 | 3.5 | +| finalization | 54.3 | 58.1 | 71.3 | +| **total (wall)** | **859.6** | **1,219.4** | **1,431.5** | +| spread (min–max) | 855–871 | 1,205–1,232 | 1,388–1,439 | +| cold run (wall) | 863.6 | 4,193.7 | 5,977.3 | | pages | 5 | 5 | 5 | -### florilegium +### generated book (3 warm runs) + +| phase (ms) | pdflatex | xelatex | lualatex | +|---|---:|---:|---:| +| startup | 44.3 | 83.4 | 242.5 | +| preamble | 192.3 | 283.8 | 273.7 | +| begin-document | 58.6 | 49.0 | 77.3 | +| typesetting | 14,322.1 | 15,935.5 | 27,305.0 | +| shipout | 2,872.5 | 3,831.2 | 10,949.3 | +| end-document | 25.0 | 22.4 | 24.9 | +| finalization | 119.6 | 230.7 | 157.7 | +| **total (wall)** | **17,643.5** | **20,433.3** | **39,037.4** | +| spread (min–max) | 17,078–17,716 | 20,412–20,688 | 39,004–39,902 | +| cold run (wall) | 17,405.7 | 23,292.1 | 44,183.2 | +| pages | 3991 | 3991 | 3991 | + +pdflatex: 8,669 records, 164 KiB. `prumo time --json` reads the cached run in 3 ms, and `--html` writes 1,279 KiB in 8 ms. + +xelatex: 8,661 records, 165 KiB. `prumo time --json` reads the cached run in 3 ms, and `--html` writes 1,277 KiB in 8 ms. + +lualatex: 8,668 records, 169 KiB. `prumo time --json` reads the cached run in 3 ms, and `--html` writes 1,280 KiB in 8 ms. + +### florilegium manual | phase (ms) | lualatex | |---|---:| -| startup | 238.5 | -| preamble | 274.2 | -| begin-document | 12.1 | -| typesetting | 508.7 | -| shipout | 470.9 | -| end-document | 2.1 | -| finalization | 91.3 | -| **total (wall)** | **1,597.4** | -| spread (min–max) | 1,595–1,602 | -| cold run (wall) | 6,015.2 | +| startup | 245.8 | +| preamble | 294.3 | +| begin-document | 12.7 | +| typesetting | 519.1 | +| shipout | 473.5 | +| end-document | 2.2 | +| finalization | 92.4 | +| **total (wall)** | **1,633.2** | +| spread (min–max) | 1,615–1,663 | +| cold run (wall) | 6,163.4 | | pages | 35 | -### large +### ACM paper + +| phase (ms) | pdflatex | xelatex | lualatex | +|---|---:|---:|---:| +| startup | 41.6 | 82.7 | 242.2 | +| preamble | 749.5 | 1,145.6 | 1,515.8 | +| begin-document | 58.2 | 56.9 | 96.3 | +| typesetting | 995.5 | 1,136.0 | 1,300.8 | +| shipout | 274.0 | 454.7 | 283.9 | +| end-document | 258.3 | 7.5 | 146.7 | +| finalization | 71.4 | 70.0 | 146.2 | +| **total (wall)** | **2,451.0** | **2,951.1** | **3,726.2** | +| spread (min–max) | 2,420–2,518 | 2,943–2,976 | 3,713–3,837 | +| cold run (wall) | 2,469.5 | 5,948.5 | 10,267.5 | +| pages | 7 | 7 | 6 | + +### ACM paper, biblatex + +| phase (ms) | pdflatex | xelatex | lualatex | +|---|---:|---:|---:| +| startup | 42.2 | 82.8 | 238.2 | +| preamble | 936.8 | 1,487.6 | 1,845.2 | +| begin-document | 120.4 | 128.8 | 166.0 | +| typesetting | 1,454.2 | 1,650.9 | 1,765.3 | +| shipout | 302.5 | 559.8 | 299.5 | +| end-document | 289.7 | 16.1 | 164.3 | +| finalization | 72.4 | 67.5 | 142.6 | +| **total (wall)** | **3,217.8** | **3,992.9** | **4,638.7** | +| spread (min–max) | 3,215–3,253 | 3,900–4,002 | 4,615–4,803 | +| cold run (wall) | 3,215.5 | 6,905.8 | 11,349.9 | +| pages | 8 | 8 | 7 | + +### beamer deck + +| phase (ms) | pdflatex | xelatex | lualatex | +|---|---:|---:|---:| +| startup | 42.1 | 82.4 | 240.5 | +| preamble | 355.2 | 499.8 | 565.3 | +| begin-document | 29.7 | 27.8 | 37.5 | +| typesetting | 592.3 | 729.1 | 765.6 | +| shipout | 55.4 | 15.7 | 59.3 | +| end-document | 0.8 | 0.7 | 0.7 | +| finalization | 47.1 | 60.5 | 87.8 | +| **total (wall)** | **1,122.5** | **1,415.5** | **1,753.4** | +| spread (min–max) | 1,120–1,126 | 1,413–1,420 | 1,744–1,764 | +| cold run (wall) | 1,123.7 | 4,317.2 | 10,903.7 | +| pages | 33 | 33 | 33 | + +### HoTT book + +| phase (ms) | pdflatex | +|---|---:| +| startup | 42.9 | +| preamble | 284.1 | +| begin-document | 29.8 | +| typesetting | 2,355.2 | +| shipout | 1,304.0 | +| end-document | 32.4 | +| finalization | 61.4 | +| **total (wall)** | **4,110.1** | +| spread (min–max) | 4,103–4,112 | +| cold run (wall) | 4,112.4 | +| pages | 492 | + +### Open Logic (3 warm runs) | phase (ms) | pdflatex | xelatex | lualatex | |---|---:|---:|---:| -| startup | 42.3 | 82.5 | 236.3 | -| preamble | 183.6 | 279.7 | 276.7 | -| begin-document | 56.7 | 47.0 | 76.5 | -| typesetting | 13,594.2 | 15,694.2 | 27,220.9 | -| shipout | 2,781.5 | 3,778.4 | 10,933.5 | -| end-document | 23.0 | 21.4 | 24.7 | -| finalization | 111.1 | 227.4 | 157.7 | -| **total (wall)** | **16,793.5** | **20,136.1** | **38,914.9** | -| spread (min–max) | 16,757–16,801 | 20,130–20,142 | 38,899–38,944 | -| cold run (wall) | 16,395.2 | 23,068.1 | 43,350.8 | -| pages | 3,991 | 3,991 | 3,991 | +| startup | 42.9 | 83.4 | 241.7 | +| preamble | 479.2 | 512.0 | 534.7 | +| begin-document | 52.2 | 41.7 | 59.0 | +| typesetting | 29,050.9 | 29,592.5 | 34,840.7 | +| shipout | 781.3 | 1,028.1 | 3,577.1 | +| end-document | 39.5 | 37.7 | 42.0 | +| finalization | 82.7 | 159.2 | 162.4 | +| **total (wall)** | **30,530.3** | **31,454.3** | **39,431.6** | +| spread (min–max) | 30,179–30,672 | 31,450–31,493 | 39,057–39,920 | +| cold run (wall) | 30,466.3 | 34,395.1 | 43,067.6 | +| pages | 1013 | 1011 | 1011 | + +### Stacks Project (2 warm runs) + +| phase (ms) | pdflatex | +|---|---:| +| startup | 41.8 | +| preamble | 111.2 | +| begin-document | 180.1 | +| typesetting | 38,218.8 | +| shipout | 5,330.1 | +| end-document | 43.3 | +| finalization | 1,057.0 | +| **total (wall)** | **44,982.3** | +| spread (min–max) | 44,961–45,003 | +| cold run (wall) | 45,053.1 | +| pages | 7710 | +
+ +## What the probe costs + + + + For each document and engine, every build without prumo and every build with it, as a difference from the median build without. + + +A build with prumo pays for the probe's hooks and records, and for `-recorder`'s list of +files. Here a build without prumo and the same build with it ran in turns in one container, +five rounds after a warm-up of each, both timed from spawn to exit. The one-line document pays +7 ms on every engine. The papers and the deck pay 20 to 80 ms, under 6%. The books pay 1.5% or +less, except Open Logic on LuaLaTeX at 3.4%, where the builds without prumo alone spread over +1.2 s. The probe writes 18 records on the one-line document and 15,669 on the Stacks Project. +The phase numbers above are of builds with the probe, so they carry this cost. -pdflatex: 8,669 records (164 KiB); `prumo time --json` on the cached run 3 ms; `--html` 8 ms, 1,279 KiB. +| document | engine | records | plain (ms) | with prumo (ms) | cost | +|---|---|---:|---:|---:|---:| +| minimal | pdflatex | 18 | 133.1 ± 5.1 | 140.3 ± 7.9 | +7.2 ms (+5.4%) | +| minimal | xelatex | 18 | 267.4 ± 13.1 | 275.2 ± 13.9 | +7.8 ms (+2.9%) | +| minimal | lualatex | 18 | 316.2 ± 5.1 | 323.6 ± 8.3 | +7.3 ms (+2.3%) | +| paper | pdflatex | 555 | 838.8 ± 15.4 | 887.2 ± 8.7 | +48.4 ms (+5.8%) | +| paper | xelatex | 547 | 1,144.9 ± 53.7 | 1,195.1 ± 36.4 | +50.2 ms (+4.4%) | +| paper | lualatex | 556 | 1,368.4 ± 53.0 | 1,419.4 ± 44.9 | +50.9 ms (+3.7%) | +| generated book | pdflatex | 8,669 | 16,631.1 ± 261.1 | 16,767.1 ± 221.0 | +135.9 ms (+0.8%) | +| generated book | xelatex | 8,661 | 19,926.3 ± 137.5 | 20,046.6 ± 113.2 | +120.4 ms (+0.6%) | +| generated book | lualatex | 8,668 | 38,694.7 ± 2,050.2 | 38,702.6 ± 2,231.5 | +7.9 ms (+0.0%) | +| florilegium manual | lualatex | 502 | 1,621.4 ± 27.6 | 1,648.2 ± 25.4 | +26.8 ms (+1.7%) | +| ACM paper | pdflatex | 585 | 2,524.2 ± 94.6 | 2,571.0 ± 284.4 | +46.8 ms (+1.9%) | +| ACM paper | xelatex | 562 | 3,017.8 ± 64.3 | 3,083.8 ± 53.1 | +66.1 ms (+2.2%) | +| ACM paper | lualatex | 589 | 3,744.5 ± 108.8 | 3,766.3 ± 126.7 | +21.7 ms (+0.6%) | +| ACM paper, biblatex | pdflatex | 614 | 3,299.9 ± 166.1 | 3,351.9 ± 179.4 | +52.0 ms (+1.6%) | +| ACM paper, biblatex | xelatex | 586 | 3,964.5 ± 49.3 | 4,009.5 ± 62.9 | +45.1 ms (+1.1%) | +| ACM paper, biblatex | lualatex | 613 | 4,604.1 ± 149.0 | 4,682.9 ± 64.5 | +78.8 ms (+1.7%) | +| beamer deck | pdflatex | 795 | 1,098.5 ± 41.3 | 1,144.2 ± 36.6 | +45.7 ms (+4.2%) | +| beamer deck | xelatex | 803 | 1,428.2 ± 24.0 | 1,479.6 ± 38.0 | +51.4 ms (+3.6%) | +| beamer deck | lualatex | 812 | 1,739.5 ± 32.6 | 1,794.3 ± 12.7 | +54.9 ms (+3.2%) | +| HoTT book | pdflatex | 1,678 | 4,121.9 ± 41.8 | 4,185.2 ± 49.0 | +63.3 ms (+1.5%) | +| Open Logic | pdflatex | 10,756 | 29,502.2 ± 813.5 | 29,865.4 ± 511.2 | +363.2 ms (+1.2%) | +| Open Logic | xelatex | 10,737 | 31,370.5 ± 2,661.3 | 31,504.3 ± 910.8 | +133.8 ms (+0.4%) | +| Open Logic | lualatex | 10,746 | 38,005.7 ± 1,223.5 | 39,313.4 ± 1,831.3 | +1,307.7 ms (+3.4%) | +| Stacks Project | pdflatex | 15,669 | 44,712.6 ± 772.3 | 45,019.3 ± 518.8 | +306.8 ms (+0.7%) | -xelatex: 8,661 records (165 KiB); `prumo time --json` on the cached run 3 ms; `--html` 7 ms, 1,277 KiB. +## Package by package: pdfLaTeX against LuaLaTeX -lualatex: 8,668 records (169 KiB); `prumo time --json` on the cached run 3 ms; `--html` 7 ms, 1,279 KiB. + + + The paper's slowest packages on pdfLaTeX and LuaLaTeX. biblatex takes about 150 ms longer on LuaLaTeX and fontspec's LuaTeX code adds about 70 ms; the other packages take the same time on both. + + +On the paper, LuaLaTeX's preamble takes 780 ms and pdfLaTeX's 500. Package by package, most +of that difference is two rows. biblatex takes 289 ms on LuaLaTeX and 138 on pdfLaTeX, and +fontspec's LuaTeX code adds 67 ms that pdfLaTeX never loads. Every other package takes the +same time on both, within a few milliseconds. Each row is the package's own time, without +the packages it loads. biblatex's row includes its style and language +files, which it reads without the file hooks. The times are the medians of the five probed +builds in the overhead set. prumo stops at the package, so why biblatex is slower on LuaTeX +is a question for a macro profiler. + +| package | pdfLaTeX (ms) | LuaLaTeX (ms) | difference | +|---|---:|---:|---:| +| biblatex | 138.4 | 289.1 | +150.7 | +| fontspec-luatex | – | 67.1 | | +| pgfplots | 74.5 | 82.7 | +8.2 | +| siunitx | 33.6 | 35.4 | +1.9 | +| hyperref | 15.2 | 15.8 | +0.6 | +| xcolor | 8.6 | 9.0 | +0.4 | +| cleveref | 9.7 | 9.9 | +0.3 | +| translations | 8.3 | 8.6 | +0.3 | +| amsmath | 4.5 | 4.8 | +0.3 | +| csquotes | 5.7 | 5.8 | +0.1 | +| nameref | 4.8 | 4.9 | +0.1 | +| lipsum | 12.4 | 10.2 | -2.2 | + +## What `whatif` saves + + + + For each document and engine, the share of the build each whatif experiment saved, with the probe's bound and the one-time cost. + + +Each row is `prumo whatif --shell-escape` after the latexmk build, with the same number of +runs as the phases. + +- **The precompiled preamble** saves 49% on the paper and 25% on the beamer deck, both on + pdfLaTeX. Everywhere else it fails or saves nothing. XeTeX and LuaTeX cannot do it at all, + and on pdfTeX mylatexformat could not handle the ACM class, the HoTT book or Open Logic. On + the long books the preamble is under 2% of the build, so there was little to save. +- **Draft mode** saves what writing the PDF costs. That is the most on XeLaTeX, where it skips + xdvipdfmx: 38% on the one-line document, 13% on the paper, 16% on the generated book. On + pdfLaTeX and LuaLaTeX it is 1 to 7% or within noise, except on the HoTT book and the + florilegium manual, where writing the PDF is a third of the build and draft mode saves a + quarter. +- **Externalized pictures** save 15 to 32% on the beamer deck and the florilegium manual, + for a first run of 3 to 13 s. On the long books that first run makes a pass over the whole + book for every picture. It took 31 to 63 minutes on the generated book and 93 to 119 on + Open Logic, for savings of 1 to 13%. Open Logic saves about 4 s a build after that on + pdfLaTeX and LuaLaTeX, so it pays back after 1,400 to 1,800 builds. On XeLaTeX its runs + spread too much to tell. + On XeLaTeX and LuaLaTeX one page of the beamer deck renders a little differently (0.9% of its + pixels, the same words), so those rows say the output changed. + +| document | engine | pages | baseline | preamble | draft | externalize | +|---|---|---:|---:|---|---|---| +| minimal | pdflatex | 1 | 134 ms | noise | noise | – | +| minimal | xelatex | 1 | 271 ms | fails | 104 ms (38%) | – | +| minimal | lualatex | 1 | 317 ms | fails | noise | – | +| paper | pdflatex | 5 | 820 ms | 405 ms (49%) | 12 ms (2%) | 56 ms (7%) | +| paper | xelatex | 5 | 1,177 ms | fails | 150 ms (13%) | 59 ms (5%) | +| paper | lualatex | 5 | 1,333 ms | fails | 52 ms (4%) | 45 ms (3%) | +| generated book | pdflatex | 3,991 | 16,566 ms | noise | 1,112 ms (7%) | 745 ms (4%) | +| generated book | xelatex | 3,991 | 19,914 ms | fails | 3,156 ms (16%) | 286 ms (1%) | +| generated book | lualatex | 3,991 | 38,925 ms | fails | 2,495 ms (6%) | 672 ms (2%) | +| florilegium manual | lualatex | 35 | 1,632 ms | fails | 417 ms (26%) | 518 ms (32%) | +| ACM paper | pdflatex | 7 | 2,461 ms | fails | noise | – | +| ACM paper | xelatex | 7 | 3,012 ms | fails | 186 ms (6%) | – | +| ACM paper | lualatex | 6 | 3,846 ms | fails | noise | – | +| ACM paper, biblatex | pdflatex | 8 | 3,315 ms | fails | noise | – | +| ACM paper, biblatex | xelatex | 8 | 3,956 ms | fails | 183 ms (5%) | – | +| ACM paper, biblatex | lualatex | 7 | 4,641 ms | fails | noise | – | +| beamer deck | pdflatex | 33 | 1,098 ms | 278 ms (25%) | noise | 218 ms (20%) | +| beamer deck | xelatex | 33 | 1,398 ms | fails | 158 ms (11%) | 219 ms (16%), output changed | +| beamer deck | lualatex | 33 | 1,737 ms | fails | 70 ms (4%) | 264 ms (15%), output changed | +| HoTT book | pdflatex | 492 | 4,147 ms | fails | 1,043 ms (25%) | 171 ms (4%) | +| Open Logic | pdflatex | 1,013 | 31,190 ms | fails | 261 ms (1%) | 3,936 ms (13%) | +| Open Logic | xelatex | 1,011 | 32,550 ms | fails | noise | noise | +| Open Logic | lualatex | 1,011 | 41,563 ms | fails | 545 ms (1%) | 4,069 ms (10%) | +| Stacks Project | pdflatex | 7,710 | 45,605 ms | noise | 3,374 ms (7%) | – | + +- beamer deck, xelatex, externalize: output changed, page 19 looks different (96 of 11011 pixels at 24 dpi; same words) +- beamer deck, lualatex, externalize: output changed, page 19 looks different (96 of 11011 pixels at 24 dpi; same words) + +Every row in full, with the spreads, the bounds and the one-time costs, is in +[results/whatif.md](results/whatif.md). ## What they say -- **Where LuaLaTeX's time goes.** On the one-line document, LuaLaTeX spends - 235 ms of its 315 ms starting up, 22 ms on the preamble and 54 ms writing the PDF - (shipout plus finalization). On the five-page paper it is 236, 763 and 135 ms of 1,392 ms. -- **Startup is the engine's fixed cost.** pdfLaTeX starts in 42 ms, XeLaTeX in 83, LuaLaTeX - in 235, the same on every document here. It is most of a short document's time on +- **Startup is the engine's fixed cost.** pdfLaTeX starts in 42 ms, XeLaTeX in 83, LuaLaTeX in + 242, the same on every document here. It is three quarters of the one-line document on LuaLaTeX. -- **The paper's preamble is its biggest phase** on every engine: 502 of 855 ms on pdfLaTeX, - 763 of 1,392 on LuaLaTeX. The sample in the top-level README has biblatex at 39% of LuaLaTeX's. -- **Writing the PDF** (shipout plus finalization) is 111 ms on pdfLaTeX, 76 on XeLaTeX and - 135 on LuaLaTeX for the paper. On the 3,991-page book it is 2.9 s, 4.0 s and - 11.1 s: LuaTeX's shipout costs 2.7 ms a page against pdfTeX's 0.7. -- **Cold font caches cost seconds once.** The first LuaLaTeX run builds luaotfload's font - name database: 4.4 s instead of 0.31 s on the one-line document. XeLaTeX's first run - builds the fontconfig cache: 3.2 s instead of 0.27 s. pdfLaTeX has no cache to build. -- **Scale.** The book's 8,669 records are 164 KiB. prumo reads a cached run in - 3 ms and writes its 1,279 KiB HTML page in 8 ms. +- **Short documents spend their time in the preamble, long ones in the body.** The paper's + preamble is 500 of 860 ms on pdfLaTeX, the ACM paper's 1.5 of 3.7 s on LuaLaTeX. Open + Logic is 95% body, the Stacks Project 85%. +- **LuaLaTeX is the slowest engine on every document here,** 1.3 to 2.3 times pdfLaTeX's time. + On the generated book it takes 13 s longer in the body and 8 s longer writing the PDF: its + shipout is 2.7 ms a page against pdfTeX's 0.7. +- **XeLaTeX is close to pdfLaTeX on long documents** (Open Logic: 31.5 s against 30.5) and 1.2 + to 2 times on short ones, where its startup and xdvipdfmx show. +- **Cold font caches cost seconds once.** The first LuaLaTeX run of the beamer deck takes + 10.9 s instead of 1.75, the ACM paper 10.3 s instead of 3.7, while luaotfload builds its font + name database. XeLaTeX's first run is about 3 s longer on every document, for fontconfig. + pdfLaTeX has no cache to build. +- **The probe costs 7 ms on the one-line document and under 6% everywhere here.** +- **Scale.** The Stacks Project's 7,710 pages build in 45 s on pdfLaTeX and leave 15,669 + records. The generated book's 8,669 records are 164 KiB; prumo reads a cached run of it in + 3 ms and writes its 1,280 KiB HTML page in 8 ms. + +## Running it + +``` +docs/benchmarks/run.sh # the phases, every document +docs/benchmarks/run.sh --docs minimal,standard # just these +docs/benchmarks/run.sh --overhead # builds with and without prumo, in turns +docs/benchmarks/run.sh --whatif --docs hott # prumo whatif --shell-escape +docs/benchmarks/run.sh --report # the markdown again, from the JSON +python3 docs/benchmarks/charts.py # the charts, from the JSON +``` + +`BENCH_IMAGE`, `BENCH_CPUS` and `BENCH_REPOS` change the image, the core and where the clones +are kept. The first run fetches the seven documents that live elsewhere; after that it +needs no network. The results go to `results/`: `results.json`, `overhead/` and `whatif/` +hold every run, and the `.md` files beside them are the tables above. + +Some runs are long. The phases of every document take about an hour on one core. +`--whatif --docs open-logic` takes several hours, most of it in externalize's first run, +which builds every picture with a pass over the whole 1,013-page book. diff --git a/docs/benchmarks/bench.py b/docs/benchmarks/bench.py index f6183a6..f0602dc 100755 --- a/docs/benchmarks/bench.py +++ b/docs/benchmarks/bench.py @@ -1,22 +1,35 @@ #!/usr/bin/env python3 -"""Benchmarks: prumo's phase table for four documents on three engines. +"""Benchmarks: prumo's phase table, and `prumo whatif`, for the documents in corpus.toml. Runs inside a TeX Live image with prumo on PATH and the documents under /bench/docs (run.sh -sets that up). For each document and engine: +lays them out with fetch.py). For each document and engine: - 1. Copy the document to a scratch directory and build it there with latexmk, so the .aux, - .toc and .bbl exist. prumo copies the directory it is given, so every timed run is a - rerun with those files present, the way an editor rebuilds. + 1. Copy the document to a scratch directory, run its `prepare` commands, and build it + there with latexmk, so the .aux, .toc and .bbl exist. prumo copies the directory it is + given, so every timed run is a rerun with those files present, the way an editor + rebuilds. 2. One cold run: a fresh HOME, so no luaotfload font name database and no fontconfig cache. The operating system's file cache is not dropped (that needs root on the host). - 3. --runs warm runs with that HOME. + 3. The document's `runs` warm runs (--runs by default) with that HOME. Every timed run is `prumo time --rebuild --json`. The phases come from prumo, the wall time is prumo's own measure of the engine process. Writes results.json (every run, plus the medians) and results.md (the tables) to --out. -Usage: bench.py [--runs 5] [--docs minimal,standard,florilegium,large] - [--engines pdflatex,xelatex,lualatex] [--out results] +With --overhead, step 1 is followed by --runs rounds of two builds in turns, after a +warm-up of each: the plain engine command, and the same with what prumo adds (-recorder, +-jobname and the probe, from /probe). Both are timed here the same way, spawn to exit, so the +difference is the probe's cost. Each probed run's records are read with `prumo time --json +.prumo` for the per-package times. Results go to overhead/.json. + +With --whatif, step 1 is followed by `prumo whatif --json --shell-escape` with the same +number of runs, in the HOME latexmk warmed. Each document's reports go to +whatif/.json under --out, so several containers can run at once on different +documents, and whatif.md is written from every file there. --report rewrites the two +markdown files from the JSON without building anything. + +Usage: bench.py [--runs 5] [--docs minimal,standard,...] [--engines pdflatex,xelatex,lualatex] + [--whatif | --overhead | --report] [--out results] """ import argparse import json @@ -26,21 +39,23 @@ import statistics import subprocess import time +import tomllib from pathlib import Path -DOCS = { - # name: (directory under /bench/docs, main file, engines it supports, warm runs or None) - "minimal": ("minimal", "main.tex", None, None), - "standard": ("standard", "main.tex", None, None), - "florilegium": ("florilegium", "florilegium-doc.tex", ["lualatex"], None), - "large": ("large", "main.tex", None, 3), -} +DOCS = tomllib.loads((Path(__file__).resolve().parent / "corpus.toml").read_text()) +ENGINES = ["pdflatex", "xelatex", "lualatex"] + + +def title(doc): + """What the tables call a document: its title in corpus.toml.""" + return DOCS[doc].get("title", doc) PHASES = ["startup", "preamble", "begin_document", "typesetting", "shipout", "end_document", "finalization"] LATEXMK_FLAG = {"pdflatex": "-pdf", "xelatex": "-xelatex", "lualatex": "-lualatex"} def sh(cmd, **kw): - return subprocess.run(cmd, capture_output=True, text=True, **kw) + # TeX output is not always UTF-8 (a log quotes the bytes a file had). + return subprocess.run(cmd, capture_output=True, text=True, errors="replace", **kw) def version_line(cmd): @@ -63,21 +78,33 @@ def machine(): "xetex": version_line(["xetex", "--version"]), "luahbtex": version_line(["luahbtex", "--version"]), "prumo": version_line(["prumo", "--version"]), - "florilegium_commit": os.environ.get("FLORILEGIUM_COMMIT", "?"), + "commits": {n: d["commit"] for n, d in DOCS.items() if "commit" in d}, } +def doc_dir(doc, scratch): + """The document's own directory inside its scratch copy.""" + return scratch / DOCS[doc].get("dir", ".") + + def prepare(doc, engine, scratch): - """Copy the document and build it once with latexmk so the aux files exist.""" - src = Path("/bench/docs") / DOCS[doc][0] + """Copy the document, run its prepare commands, build it once with latexmk so the aux + files exist. Returns (ok, log tail, HOME used).""" + src = Path("/bench/docs") / doc if scratch.exists(): shutil.rmtree(scratch) shutil.copytree(src, scratch) - env = dict(os.environ, HOME=str(scratch.parent / f"home-prep-{doc}-{engine}")) - main = DOCS[doc][1] - r = sh(["latexmk", LATEXMK_FLAG[engine], "-interaction=nonstopmode", main], cwd=scratch, env=env) - pdf = scratch / main.replace(".tex", ".pdf") - return r.returncode == 0 and pdf.is_file(), r.stdout[-2000:] + r.stderr[-2000:] + home = scratch.parent / f"home-prep-{doc}-{engine}" + env = dict(os.environ, HOME=str(home)) + if DOCS[doc].get("prepare"): + r = sh(["bash", "-ec", DOCS[doc]["prepare"]], cwd=scratch, env=env) + if r.returncode != 0: + return False, "prepare: " + r.stdout[-1000:] + r.stderr[-1000:], home + main = DOCS[doc]["main"] + d = doc_dir(doc, scratch) + r = sh(["latexmk", LATEXMK_FLAG[engine], "-interaction=nonstopmode", main], cwd=d, env=env) + pdf = d / main.replace(".tex", ".pdf") + return r.returncode == 0 and pdf.is_file(), r.stdout[-2000:] + r.stderr[-2000:], home def timed_run(scratch, main, engine, home, out): @@ -164,69 +191,282 @@ def table(results, doc, engines): return "\n".join(lines) + "\n" -def main(): - ap = argparse.ArgumentParser() - ap.add_argument("--runs", type=int, default=5) - ap.add_argument("--docs", default="minimal,standard,florilegium,large") - ap.add_argument("--engines", default="pdflatex,xelatex,lualatex") - ap.add_argument("--out", default="results") - a = ap.parse_args() - out = Path(a.out) - out.mkdir(parents=True, exist_ok=True) - engines = a.engines.split(",") - work = Path("/tmp/bench") +def phases_md(results): + m = results["machine"] + md = [f"Machine: {m['cpu']}, pinned to {m['cpus_allowed']} CPU; {m['pdftex']}; image {m['image']}. " + f"Warm medians of {results['runs']} runs unless the heading says otherwise; the cold run has a fresh HOME.\n"] + for doc in (d for d in DOCS if d in results): + engines = [e for e in ENGINES if e in results[doc]] + runs = DOCS[doc].get("runs") + md.append(f"\n### {title(doc)}{f' ({runs} warm runs)' if runs else ''}\n\n" + table(results, doc, engines)) + for e in engines: + entry = results[doc][e] + if "failed" in entry: + md.append(f"\n{e}: the latexmk build failed, so nothing was timed.\n") + sc = entry.get("scale") + if sc: + md.append( + f"\n{e}: {sc['record_lines']:,} records, {sc['record_bytes'] / 1024:,.0f} KiB. " + f"`prumo time --json` reads the cached run in {sc['analyze_cached_ms']:,.0f} ms, and " + f"`--html` writes {sc['html_bytes'] / 1024:,.0f} KiB in {sc['html_ms']:,.0f} ms.\n" + ) + return "".join(md) + + +def run_phases(a, out, work): results = {"machine": machine(), "runs": a.runs} for doc in a.docs.split(","): - _, main_file, supported, runs_override = DOCS[doc] - runs = runs_override or a.runs + runs = DOCS[doc].get("runs") or a.runs results[doc] = {} - for engine in engines: - if supported and engine not in supported: - results[doc][engine] = {"skipped": f"the document needs {', '.join(supported)}"} + for engine in a.engines.split(","): + if engine not in DOCS[doc].get("engines", ENGINES): continue print(f"== {doc} on {engine}", flush=True) scratch = work / f"{doc}-{engine}" - ok, log = prepare(doc, engine, scratch) + ok, log, _ = prepare(doc, engine, scratch) if not ok: results[doc][engine] = {"failed": "latexmk", "log": log} print(f" latexmk failed:\n{log[-800:]}", flush=True) continue + d = doc_dir(doc, scratch) + main_file = DOCS[doc]["main"] home = work / f"home-{doc}-{engine}" cache = work / f"cache-{doc}-{engine}" - for d in (home, cache): - shutil.rmtree(d, ignore_errors=True) - d.mkdir(parents=True) - cold = summary(*timed_run(scratch, main_file, engine, home, cache)) + for p in (home, cache): + shutil.rmtree(p, ignore_errors=True) + p.mkdir(parents=True) + cold = summary(*timed_run(d, main_file, engine, home, cache)) print(f" cold {cold['wall_ms']:9.1f} ms wall, {cold['pages']} pages", flush=True) warm = [] for i in range(runs): - s = summary(*timed_run(scratch, main_file, engine, home, cache)) + s = summary(*timed_run(d, main_file, engine, home, cache)) warm.append(s) print(f" warm {i + 1} {s['wall_ms']:9.1f} ms wall", flush=True) entry = {"cold": cold, "warm": warm, "median": median_of(warm)} if doc == "large": - entry["scale"] = scale(scratch, main_file, engine, home, cache) + entry["scale"] = scale(d, main_file, engine, home, cache) if (cache / f"{engine}.html").is_file(): shutil.move(str(cache / f"{engine}.html"), str(out / f"large-{engine}.html")) results[doc][engine] = entry (out / "results.json").write_text(json.dumps(results, indent=1)) - md = [f"Machine: {results['machine']['cpu']}, pinned to {results['machine']['cpus_allowed']} CPU; " - f"{results['machine']['pdftex']}; image {results['machine']['image']}. " - f"Warm medians of {a.runs} runs (3 for the large document); the cold run has a fresh HOME.\n"] + (out / "results.json").write_text(json.dumps(results, indent=1)) + (out / "results.md").write_text(phases_md(results)) + print(phases_md(results)) + + +def run_whatif(a, out, work): + """`prumo whatif` on each document and engine; one JSON file per document.""" + (out / "whatif").mkdir(exist_ok=True) + for doc in a.docs.split(","): + runs = DOCS[doc].get("runs") or a.runs + entry = {"machine": machine(), "runs": runs} + for engine in a.engines.split(","): + if engine not in DOCS[doc].get("engines", ENGINES): + continue + print(f"== whatif {doc} on {engine}", flush=True) + scratch = work / f"{doc}-{engine}" + ok, log, home = prepare(doc, engine, scratch) + if not ok: + entry[engine] = {"failed": "latexmk", "log": log} + print(f" latexmk failed:\n{log[-800:]}", flush=True) + continue + cache = work / f"cache-whatif-{doc}-{engine}" + shutil.rmtree(cache, ignore_errors=True) + env = dict(os.environ, HOME=str(home), PRUMO_CACHE=str(cache)) + main = doc_dir(doc, scratch) / DOCS[doc]["main"] + cmd = ["prumo", "whatif", "--json", "--shell-escape", "--engine", engine, "--runs", str(runs), str(main)] + t0 = time.monotonic() + r = sh(cmd, env=env) + elapsed = round(time.monotonic() - t0, 1) + if r.returncode != 0: + entry[engine] = {"failed": "prumo whatif", "log": r.stderr[-2000:]} + print(f" prumo whatif failed:\n{r.stderr[-800:]}", flush=True) + else: + entry[engine] = {"report": json.loads(r.stdout), "elapsed_s": elapsed} + for row in entry[engine]["report"]["experiments"]: + print(f" {row['experiment']:<12} {cell(row)}", flush=True) + (out / "whatif" / f"{doc}.json").write_text(json.dumps(entry, indent=1)) + (out / "whatif.md").write_text(whatif_md(out)) + + +# The environments prumo watches by default (run::DEFAULT_ENVIRONMENTS), as it loads the probe. +PROBE_ENVS = "tikzpicture,axis,tcolorbox,tblr,longtable,tabularx,minted,lstlisting,forest" + + +def run_overhead(a, out, work): + """Plain and probed builds in turns, timed alike: what prumo's probe costs.""" + (out / "overhead").mkdir(exist_ok=True) for doc in a.docs.split(","): - md.append(f"\n### {doc}\n\n" + table(results, doc, engines)) - if doc == "large": - for e in engines: - sc = results[doc].get(e, {}).get("scale") - if sc: - md.append( - f"\n{e}: {sc['record_lines']:,} records ({sc['record_bytes'] / 1024:,.0f} KiB); " - f"`prumo time --json` on the cached run {sc['analyze_cached_ms']:,.0f} ms; " - f"`--html` {sc['html_ms']:,.0f} ms, {sc['html_bytes'] / 1024:,.0f} KiB.\n" + # --runs for every document: the cost is a few percent, and a long document's two or + # three runs cannot tell that from one slow run. + runs = a.runs + entry = {"machine": machine(), "runs": runs} + for engine in a.engines.split(","): + if engine not in DOCS[doc].get("engines", ENGINES): + continue + print(f"== overhead {doc} on {engine}", flush=True) + scratch = work / f"{doc}-{engine}" + ok, log, home = prepare(doc, engine, scratch) + if not ok: + entry[engine] = {"failed": "latexmk", "log": log} + print(f" latexmk failed:\n{log[-800:]}", flush=True) + continue + d = doc_dir(doc, scratch) + main = DOCS[doc]["main"] + job = main.removesuffix(".tex") + plain = [engine, "-interaction=nonstopmode", main] + probed = [engine, "-interaction=nonstopmode", "-recorder", f"-jobname={job}", + f"\\RequirePackage{{prumo}}\\PrumoEnvironments{{{PROBE_ENVS}}}\\input{{{job}}}"] + env = dict(os.environ, HOME=str(home)) + env_probed = dict(env, TEXINPUTS="/probe:") + + def timed(cmd, e): + t0 = time.monotonic() + r = subprocess.run(cmd, cwd=d, env=e, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + return round((time.monotonic() - t0) * 1000, 3), r.returncode + + timed(plain, env) + timed(probed, env_probed) + res = {"plain_ms": [], "probed_ms": [], "units": {}, "exit_codes": []} + for i in range(runs): + p, pc = timed(plain, env) + q, qc = timed(probed, env_probed) + res["plain_ms"].append(p) + res["probed_ms"].append(q) + res["exit_codes"].append([pc, qc]) + r = sh(["prumo", "time", "--json", str(d / f"{job}.prumo")], env=env) + if r.returncode == 0: + j = json.loads(r.stdout) + res["pages"] = len(j["pages"]) + for u in j["units"]: + res["units"].setdefault(f"{u['kind']} {u['name']}", []).append(u["exclusive_ms"]) + print(f" run {i + 1}: plain {p:9.1f} ms, probed {q:9.1f} ms", flush=True) + res["records"] = sum(1 for _ in open(d / f"{job}.prumo")) + entry[engine] = res + (out / "overhead" / f"{doc}.json").write_text(json.dumps(entry, indent=1)) + (out / "overhead.md").write_text(overhead_md(out)) + + +def overhead_md(out): + """The probe's cost per document and engine.""" + md = ["| document | engine | records | plain (ms) | with prumo (ms) | cost |", "|---|---|---:|---:|---:|---:|"] + for doc in DOCS: + f = out / "overhead" / f"{doc}.json" + if not f.is_file(): + continue + entry = json.loads(f.read_text()) + for e in ENGINES: + r = entry.get(e) + if not r or "plain_ms" not in r: + continue + p, q = statistics.median(r["plain_ms"]), statistics.median(r["probed_ms"]) + sp = max(r["plain_ms"]) - min(r["plain_ms"]) + sq = max(r["probed_ms"]) - min(r["probed_ms"]) + md.append(f"| {title(doc)} | {e} | {r['records']:,} | {p:,.1f} ± {sp:,.1f} | {q:,.1f} ± {sq:,.1f} | " + f"{q - p:+,.1f} ms ({(q - p) / p * 100:+.1f}%) |") + return "\n".join(md) + "\n" + + +def cell(row): + """One experiment in the summary table.""" + st = row["status"] + if st == "not-run": + return "–" + if st == "failed": + return "fails" + if row["within_noise"]: + text = "noise" + elif row["saved_ms"] >= 0: + text = f"{row['saved_ms']:,.0f} ms ({row['saved_pct']:.0f}%)" + else: + text = f"{-row['saved_ms']:,.0f} ms slower" + if row["output"]["status"] == "changed": + text += ", output changed" + return text + + +def whatif_md(out): + """The summary table, the notes it needs, and every row in full.""" + reports = {} + for doc in DOCS: + f = out / "whatif" / f"{doc}.json" + if f.is_file(): + reports[doc] = json.loads(f.read_text()) + if not reports: + return "" + exps = ["preamble", "draft", "externalize"] + md = ["| document | engine | pages | baseline | " + " | ".join(exps) + " |", + "|---|---|---:|---:|" + "---|" * len(exps)] + notes, full = [], [] + for doc, entry in reports.items(): + for e in ENGINES: + if e not in entry: + continue + if "report" not in entry[e]: + md.append(f"| {title(doc)} | {e} | | | {entry[e]['failed']} failed | | |") + continue + r = entry[e]["report"] + rows = {x["experiment"]: x for x in r["experiments"]} + md.append( + f"| {title(doc)} | {e} | {r['baseline_pages']:,} | {r['baseline']['median_ms']:,.0f} ms | " + + " | ".join(cell(rows[x]) if x in rows else "" for x in exps) + " |" + ) + for x in exps: + row = rows.get(x) + if not row: + continue + if row["status"] == "failed": + notes.append(f"- {title(doc)}, {e}, {x}: {row['error']}") + elif row["status"] == "measured" and row["output"]["status"] == "changed": + notes.append(f"- {title(doc)}, {e}, {x}: output changed, {row['output']['evidence']}") + bound = "" if row.get("bound_ms") is None else f"{row['bound_ms']:,.0f}" + if row["status"] == "measured": + v = row["variant"] + one = "" if row["one_time_ms"] is None else f"{row['one_time_ms']:,.0f}" + full.append( + f"| {title(doc)} | {e} | {x} | {r['baseline']['median_ms']:,.1f} ± {r['baseline']['spread_ms']:,.1f} | " + f"{v['median_ms']:,.1f} ± {v['spread_ms']:,.1f} | {row['saved_ms']:,.1f} | {bound} | {one} | " + f"{row['output']['status']} |" ) - (out / "results.md").write_text("".join(md)) - (out / "results.json").write_text(json.dumps(results, indent=1)) - print("".join(md)) + else: + why = row.get("reason") or row.get("error", "") + full.append(f"| {title(doc)} | {e} | {x} | | | {row['status']}: {why.replace('|', '/')} | {bound} | | |") + md.append("") + if notes: + md += ["Failures and changed output:", ""] + notes + [""] + md += ["Every row (ms; ± is the spread, slowest run minus fastest):", "", + "| document | engine | experiment | baseline | variant | saved | bound | one-time | output |", + "|---|---|---|---:|---:|---:|---:|---:|---|"] + full + return "\n".join(md) + "\n" + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--runs", type=int, default=5) + ap.add_argument("--docs", default=",".join(DOCS)) + ap.add_argument("--engines", default=",".join(ENGINES)) + ap.add_argument("--whatif", action="store_true") + ap.add_argument("--overhead", action="store_true") + ap.add_argument("--report", action="store_true") + ap.add_argument("--out", default="results") + a = ap.parse_args() + unknown = [d for d in a.docs.split(",") if d not in DOCS] + if unknown: + raise SystemExit(f"not in corpus.toml: {', '.join(unknown)}") + out = Path(a.out) + out.mkdir(parents=True, exist_ok=True) + work = Path("/tmp/bench") + if a.report: + if (out / "results.json").is_file(): + (out / "results.md").write_text(phases_md(json.loads((out / "results.json").read_text()))) + (out / "whatif.md").write_text(whatif_md(out)) + (out / "overhead.md").write_text(overhead_md(out)) + elif a.overhead: + run_overhead(a, out, work) + elif a.whatif: + run_whatif(a, out, work) + else: + run_phases(a, out, work) if __name__ == "__main__": diff --git a/docs/benchmarks/charts.py b/docs/benchmarks/charts.py new file mode 100644 index 0000000..8af07bb --- /dev/null +++ b/docs/benchmarks/charts.py @@ -0,0 +1,522 @@ +#!/usr/bin/env python3 +"""Charts for docs/benchmarks/README.md, as SVG, from the JSON under results/. Standard +library only. Each chart is written twice, for a light and a dark page; the README picks one +with , and the exact numbers are in the tables under each chart. + + phases each document in its own panel and on its own axis: one bar per engine, cut + into startup, preamble, body and writing the PDF, with every run's wall time + overhead every plain and every probed run, against the plain runs' median + packages the paper's package load times on pdfLaTeX and LuaLaTeX, by the difference + whatif what each experiment saved, as a share of the build, with the probe's bound + +Color means phase, in every chart: startup is a neutral gray (no experiment changes it), +dark on a light page and light on a dark one, the preamble violet, the body magenta, writing +the PDF yellow. The same hue marks the experiment that shrinks that phase. Engines are told +apart by position and label, and in the package chart by a hollow or filled dot. The colors +are the dataviz reference palette's. Every pair of the four, the gray included, is at least +OKLab ΔE 13 apart under simulated protan and deutan vision and 19 apart under normal vision, +in both modes (measured with its validate_palette.js; the mid gray it had before sat 1.1 +from the body's magenta for deutan readers). + +Usage: charts.py [--results DIR] [--out DIR] +""" +import argparse +import json +import statistics +import tomllib +from html import escape +from pathlib import Path + +HERE = Path(__file__).resolve().parent +CORPUS = tomllib.loads((HERE / "corpus.toml").read_text()) + +THEMES = { + "light": { + "surface": "#fcfcfb", "ink": "#0b0b0b", "ink2": "#52514e", "muted": "#898781", + "grid": "#e1e0d9", "axis": "#c3c2b7", + "startup": "#52514e", "preamble": "#6250d6", "body": "#e87ba4", "pdf": "#eda100", + "inside": {"pdf": "#0b0b0b"}, + }, + "dark": { + "surface": "#1a1a19", "ink": "#f0efec", "ink2": "#c3c2b7", "muted": "#898781", + "grid": "#2c2c2a", "axis": "#383835", + "startup": "#c3c2b7", "preamble": "#9085e9", "body": "#d55181", "pdf": "#c98500", + "inside": {"pdf": "#0b0b0b", "startup": "#0b0b0b"}, + }, +} +ENGINES = ["pdflatex", "xelatex", "lualatex"] +NAME = {"pdflatex": "pdfLaTeX", "xelatex": "XeLaTeX", "lualatex": "LuaLaTeX"} +GROUPS = [ + ("startup", "startup", ["startup"]), + ("preamble", "preamble", ["preamble"]), + ("body", "body", ["begin_document", "typesetting", "end_document"]), + ("pdf", "writing the PDF", ["shipout", "finalization"]), +] +EXPERIMENTS = [("preamble", "preamble", "precompiled preamble"), ("draft", "pdf", "draft mode"), + ("externalize", "body", "externalized pictures")] +FONT = "system-ui, -apple-system, 'Segoe UI', sans-serif" +W = 860 + + +def title(doc): + """What a reader calls the document: the title in corpus.toml, else its key.""" + return CORPUS.get(doc, {}).get("title", doc) + + +def text_width(s, size=12): + """Rough width of system-ui text; there are no font metrics here, so per-class widths.""" + w = 0.0 + for ch in s: + if ch in "il.,:;|!'·": + w += 0.28 + elif ch in "fjrt() -": + w += 0.36 + elif ch.isdigit() or ch in "%×+−": + w += 0.58 + elif ch.isupper() or ch in "mw": + w += 0.74 + else: + w += 0.55 + return w * size + + +def fmt_time(ms): + if ms < 1000: + return f"{ms:,.0f} ms" + if ms >= 120_000: + return f"{ms / 60_000:,.0f} min" + return f"{ms / 1000:,.2f} s" if ms < 10_000 else f"{ms / 1000:,.1f} s" + + +def nice_ticks(hi, n=4): + """Round tick values from 0 to at least hi.""" + raw = hi / n + mag = 10 ** len(str(int(raw))) / 10 if raw >= 1 else 1 + step = next(s * mag for s in (1, 2, 2.5, 5, 10) if s * mag >= raw) + ticks, v = [], 0.0 + while v < hi + step * 0.999: + ticks.append(v) + v += step + return ticks + + +class Svg: + def __init__(self, height, t, title, desc): + self.t = t + self.parts = [ + f'', + f"{escape(title)}{escape(desc)}", + f'', + ] + + def add(self, s): + self.parts.append(s) + + def text(self, x, y, s, size=12, fill="ink", anchor="start", weight=None): + w = f' font-weight="{weight}"' if weight else "" + nums = ' font-variant-numeric="tabular-nums"' if any(c.isdigit() for c in s) else "" + self.add(f'{escape(s)}') + + def line(self, x1, y1, x2, y2, stroke, width=1, tip=None): + t = f"{escape(tip)}" if tip else "" + self.add(f'{t}') + + def bar(self, x, y, w, h, fill, tip, round_end=True): + """A bar from a baseline at x: square there, 4 px rounded at the data end.""" + if w <= 0: + return + r = min(4, w / 2, h / 2) if round_end else 0 + d = (f"M{x:.1f},{y:.1f} h{w - r:.1f} a{r},{r} 0 0 1 {r},{r} v{h - 2 * r:.1f} " + f"a{r},{r} 0 0 1 -{r},{r} h-{w - r:.1f} z") + self.add(f'{escape(tip)}') + + def dot(self, cx, cy, fill, tip, hollow=False, r=4): + t = self.t + if hollow: + self.add(f'{escape(tip)}') + else: + self.add(f'{escape(tip)}') + + def swatch(self, x, y, key, name): + self.add(f'') + self.text(x + 15, y, name, fill="ink2") + return x + 15 + text_width(name) + 18 + + def header(self, title, subtitle): + self.text(24, 34, title, size=17, weight="600") + self.text(24, 56, subtitle, fill="ink2") + + def done(self): + return "\n".join(self.parts + [""]) + "\n" + + +def load(results_dir): + def per_doc(sub): + out = {} + for doc in CORPUS: + f = results_dir / sub / f"{doc}.json" + if f.is_file(): + out[doc] = json.loads(f.read_text()) + return out + + f = results_dir / "results.json" + return { + "phases": json.loads(f.read_text()) if f.is_file() else {}, + "whatif": per_doc("whatif"), + "overhead": per_doc("overhead"), + } + + +def caption(data): + """Machine, TeX Live and runs, for the foot of every chart.""" + p = data["phases"] + m = p.get("machine") or next(iter(data["whatif"].values()), {}).get("machine", {}) + cpu = m.get("cpu", "?").replace("13th Gen Intel(R) Core(TM) ", "") + fmt = next((p[d][e]["cold"]["format_date"] for d in CORPUS if d in p for e in ENGINES + if "cold" in p[d].get(e, {})), None) + runs = sorted({(CORPUS[d].get("runs") or p.get("runs", 5)) for d in CORPUS}) + by_runs = {} + for d in CORPUS: + if CORPUS[d].get("runs"): + by_runs.setdefault(CORPUS[d]["runs"], []).append(title(d)) + fewer = "; ".join(f"{' and '.join(names)} {n}" for n, names in sorted(by_runs.items(), reverse=True)) + return (f"{cpu}, one P-core per container · TeX Live 2026{f', LaTeX {fmt}' if fmt else ''} · " + f"warm runs: {max(runs)}{f'; {fewer}' if fewer else ''}") + + +def walls(entry): + return [r["wall_ms"] for r in entry["warm"] if r["wall_ms"] is not None] + + +def doc_order(data): + """Documents from the quickest build to the slowest (the first engine's median).""" + p = data["phases"] + + def key(doc): + for e in ENGINES: + if "median" in p.get(doc, {}).get(e, {}): + return p[doc][e]["median"]["wall"] + return float("inf") + + return sorted((d for d in CORPUS if d in p or d in data["whatif"] or d in data["overhead"]), key=key) + + +def phases_chart(data, t): + p = data["phases"] + docs = [d for d in doc_order(data) if any("median" in p.get(d, {}).get(e, {}) for e in ENGINES)] + pw = (W - 48 - 24) / 2 + head, row_h, axis_h, gap = 30, 22, 26, 22 + + def panel_h(doc): + n = sum("median" in p[doc].get(e, {}) for e in ENGINES) + return head + n * row_h + axis_h + + top = 100 + heights = [] + for i in range(0, len(docs), 2): + heights.append(max(panel_h(d) for d in docs[i:i + 2])) + height = top + sum(heights) + gap * (len(heights) - 1) + 40 + svg = Svg(height, t, "Where the time goes", + "One panel per benchmark document, each on its own time axis. For each engine a bar cut into " + "startup, preamble, body and writing the PDF (medians of the warm runs), with every run's wall " + "time as a dot.") + svg.header("Where the time goes", "Median of the warm runs, cut into phases. The dots are each run's wall " + "time. Each panel has its own axis.") + x = 24 + for key, name, _ in GROUPS: + x = svg.swatch(x, 84, key, name) + svg.dot(x + 4, 80, "ink", "one run's wall time") + svg.text(x + 14, 84, "one run", fill="ink2") + y = top + for i in range(0, len(docs), 2): + for j, doc in enumerate(docs[i:i + 2]): + px = 24 + j * (pw + 24) + engines = [e for e in ENGINES if "median" in p[doc].get(e, {})] + pages = p[doc][engines[0]]["cold"]["pages"] + svg.text(px, y + 14, title(doc), size=13, weight="600") + svg.text(px + text_width(title(doc), 13) * 1.2 + 10, y + 14, f"{pages:,} page{'s' if pages != 1 else ''}", + size=11, fill="muted") + label_w, val_w = 70, 62 + bx, bw = px + label_w, pw - label_w - val_w + hi = max(max(walls(p[doc][e])) for e in engines) + ticks = nice_ticks(hi, 3) + scale = bw / ticks[-1] + y0, y1 = y + head - 4, y + head + len(engines) * row_h + for tk in ticks: + gx = bx + tk * scale + svg.line(gx, y0, gx, y1, "axis" if tk == 0 else "grid") + svg.text(gx, y1 + 14, fmt_time(tk) if tk else "0", size=10, fill="muted", + anchor="start" if tk == 0 else "middle") + for k, e in enumerate(engines): + m = p[doc][e]["median"] + ry = y + head + k * row_h + cy = ry + row_h / 2 + svg.text(bx - 8, cy + 4, NAME[e], fill="ink2", anchor="end") + parts = [(key, name, sum(m[ph] or 0 for ph in phs)) for key, name, phs in GROUPS] + total = sum(v for *_, v in parts) + cx = bx + shown = [(key, name, v) for key, name, v in parts if v * scale >= 1] + for n, (key, name, v) in enumerate(shown): + w = v * scale - (2 if n < len(shown) - 1 else 0) + svg.bar(cx, ry + 4, max(w, 1), row_h - 8, key, + f"{doc}, {NAME[e]}: {name} {v:,.1f} ms ({v / total * 100:.0f}%)", + round_end=n == len(shown) - 1) + pct = f"{v / total * 100:.0f}%" + # The last segment ends where the run dots are: leave them room. + room = 8 if n < len(shown) - 1 else 22 + if w >= text_width(pct, 10) + room: + svg.text(cx + 4, cy + 3.5, pct, size=10, fill=t["inside"].get(key, "#ffffff"), weight="600") + cx += w + 2 + for wv in walls(p[doc][e]): + svg.dot(bx + wv * scale, cy, "ink", f"{doc}, {NAME[e]}: a run of {fmt_time(wv)}", r=3) + svg.text(px + pw, cy + 4, fmt_time(m["wall"]), fill="ink2", anchor="end") + y += heights[i // 2] + gap + svg.text(24, height - 16, caption(data), size=11, fill="muted") + return svg.done() + + +def grouped_rows(data, source): + """(document or "", engine, entry) rows, the document named on its first row only.""" + rows = [] + for doc in doc_order(data): + entry = data[source].get(doc) + if not entry: + continue + engines = [e for e in ENGINES if e in entry and "failed" not in entry[e]] + for i, e in enumerate(engines): + rows.append((title(doc) if i == 0 else "", e, entry[e], i == 0)) + return rows + + +def overhead_chart(data, t): + rows = grouped_rows(data, "overhead") + if not rows: + return None + label_w, row_h, gap, top = 250, 22, 10, 100 + rel = [] + for *_, r, _ in rows: + base = statistics.median(r["plain_ms"]) + rel += [v / base * 100 - 100 for v in r["plain_ms"] + r["probed_ms"]] + lo, hi = min(min(rel), -1), max(max(rel), 1) + step = next(s for s in (1, 2, 5, 10, 20, 50) if (hi - lo) / s <= 8) + lo, hi = step * (lo // step), step * -(-hi // step) + plot_x, plot_w = label_w, W - label_w - 150 + xs = lambda v: plot_x + (v - lo) / (hi - lo) * plot_w # noqa: E731 + height = top + sum(row_h + (gap if first else 0) for *_, first in rows) + 64 + svg = Svg(height, t, "What the probe costs", + "For each benchmark document and engine, every build without prumo and every build with it " + "(the probe and -recorder), as a difference from the median build without it.") + svg.header("What the probe costs", "Builds with the probe and -recorder against builds without, in turns. " + "0 is the median without, the tick the median with.") + x = 24 + svg.dot(x + 4, 80, "ink2", "a build without prumo", hollow=True) + svg.text(x + 14, 84, "without prumo", fill="ink2") + x += 14 + text_width("without prumo") + 22 + svg.dot(x + 4, 80, "ink", "a build with prumo") + svg.text(x + 14, 84, "with prumo", fill="ink2") + y0 = top + y = top + for doc, e, r, first in rows: + if first: + y += gap + y += row_h + y1 = y + v = lo + while v <= hi + 1e-9: + svg.line(xs(v), y0 + gap - 4, xs(v), y1, "axis" if v == 0 else "grid") + svg.text(xs(v), y1 + 16, f"{v:+.0f}%" if v else "0", size=10, fill="muted", anchor="middle") + v += step + y = top + for doc, e, r, first in rows: + if first: + y += gap + if y > top + gap: + svg.line(24, y - gap / 2, label_w - 8, y - gap / 2, "grid") + cy = y + row_h / 2 + if doc: + svg.text(24, cy + 4, doc, weight="600") + svg.text(label_w - 12, cy + 4, NAME[e], fill="ink2", anchor="end") + base = statistics.median(r["plain_ms"]) + for val in r["plain_ms"]: + svg.dot(xs(val / base * 100 - 100), cy, "ink2", f"{doc}, {NAME[e]}: {fmt_time(val)} without prumo", + hollow=True) + for val in r["probed_ms"]: + svg.dot(xs(val / base * 100 - 100), cy, "ink", f"{doc}, {NAME[e]}: {fmt_time(val)} with prumo") + med = statistics.median(r["probed_ms"]) + d = med - base + svg.line(xs(d / base * 100), y + 3, xs(d / base * 100), y + row_h - 3, "ink", 2, + f"{doc}, {NAME[e]}: median with prumo {fmt_time(med)}, without {fmt_time(base)}") + svg.text(plot_x + plot_w + 14, cy + 4, f"{d:+,.0f} ms ({d / base * 100:+.1f}%)", fill="ink2") + y += row_h + svg.text(plot_x + plot_w + 14, top + 2, "median cost", size=11, fill="muted") + svg.text(24, height - 16, caption(data), size=11, fill="muted") + return svg.done() + + +def packages_chart(data, t, doc="standard", a="pdflatex", b="lualatex", n=12): + entry = data["overhead"].get(doc, {}) + if a not in entry or b not in entry or "units" not in entry[a]: + return None, None + med = {e: {k: statistics.median(v) for k, v in entry[e]["units"].items() if k.split()[0] in ("pkg", "cls")} + for e in (a, b)} + keys = set(med[a]) | set(med[b]) + rows = sorted(keys, key=lambda k: -max(med[a].get(k, 0), med[b].get(k, 0)))[:n] + rows.sort(key=lambda k: -(med[b].get(k, 0) - med[a].get(k, 0))) + label_w, row_h, top = 150, 24, 100 + hi = max(max(med[a].get(k, 0), med[b].get(k, 0)) for k in rows) + ticks = nice_ticks(hi, 5) + plot_w = W - label_w - 140 + xs = lambda v: label_w + v / ticks[-1] * plot_w # noqa: E731 + pre = data["phases"].get(doc, {}) + pa = pre.get(a, {}).get("median", {}).get("preamble") + pb = pre.get(b, {}).get("median", {}).get("preamble") + height = top + len(rows) * row_h + 66 + svg = Svg(height, t, f"Package load times on {NAME[a]} and {NAME[b]}", + f"The {doc} document's slowest packages and classes, exclusive time on {NAME[a]} (hollow) and " + f"{NAME[b]} (filled), sorted by how much longer {NAME[b]} takes.") + sub = f"The paper ({doc}): each package's own time, without the packages it loads, median of the runs." + svg.header(f"Package load times, {NAME[a]} against {NAME[b]}", sub) + x = 24 + svg.dot(x + 4, 80, "preamble", NAME[a], hollow=True) + svg.text(x + 14, 84, NAME[a], fill="ink2") + x += 14 + text_width(NAME[a]) + 22 + svg.dot(x + 4, 80, "preamble", NAME[b]) + svg.text(x + 14, 84, NAME[b], fill="ink2") + if pa and pb: + svg.text(x + 14 + text_width(NAME[b]) + 22, 84, + f"whole preamble: {fmt_time(pa)} and {fmt_time(pb)}", fill="muted") + y1 = top + len(rows) * row_h + for tk in ticks: + svg.line(xs(tk), top, xs(tk), y1, "axis" if tk == 0 else "grid") + svg.text(xs(tk), y1 + 16, f"{tk:g} ms", size=10, fill="muted", anchor="middle") + svg.text(label_w + plot_w + 18, top - 6, "difference", size=11, fill="muted") + table = [f"| package | {NAME[a]} (ms) | {NAME[b]} (ms) | difference |", "|---|---:|---:|---:|"] + for i, k in enumerate(rows): + kind, name = k.split(" ", 1) + cy = top + i * row_h + row_h / 2 + svg.text(label_w - 12, cy + 4, name + (" (class)" if kind == "cls" else ""), anchor="end") + va, vb = med[a].get(k), med[b].get(k) + if va is not None and vb is not None: + svg.line(xs(min(va, vb)), cy, xs(max(va, vb)), cy, "axis", 2) + svg.dot(xs(va), cy, "preamble", f"{name}: {va:,.1f} ms on {NAME[a]}", hollow=True) + svg.dot(xs(vb), cy, "preamble", f"{name}: {vb:,.1f} ms on {NAME[b]}") + svg.text(label_w + plot_w + 18, cy + 4, f"{vb - va:+,.0f} ms", fill="ink2") + elif vb is not None: + svg.dot(xs(vb), cy, "preamble", f"{name}: {vb:,.1f} ms, {NAME[b]} only") + svg.text(label_w + plot_w + 18, cy + 4, f"{NAME[b]} only", fill="muted") + else: + svg.dot(xs(va), cy, "preamble", f"{name}: {va:,.1f} ms, {NAME[a]} only", hollow=True) + svg.text(label_w + plot_w + 18, cy + 4, f"{NAME[a]} only", fill="muted") + fa = "–" if va is None else f"{va:,.1f}" + fb = "–" if vb is None else f"{vb:,.1f}" + diff = "" if va is None or vb is None else f"{vb - va:+,.1f}" + table.append(f"| {name}{' (class)' if kind == 'cls' else ''} | {fa} | {fb} | {diff} |") + svg.text(24, height - 16, caption(data), size=11, fill="muted") + return svg.done(), "\n".join(table) + "\n" + + +def whatif_chart(data, t): + rows = [(d, e, r["report"], f) for d, e, r, f in grouped_rows(data, "whatif") if "report" in r] + if not rows: + return None + label_w, gap, row_h, top = 250, 10, 22, 104 + panel_gap = 16 + panel_w = (W - label_w - 24 - 2 * panel_gap) / 3 + plot_w = panel_w - 70 + height = top + sum(row_h + (gap if first else 0) for *_, first in rows) + 88 + svg = Svg(height, t, "What prumo whatif saves", + "For each benchmark document and engine, the share of the build each whatif experiment saved, " + "with the probe's bound for it and the one-time cost.") + svg.header("What prumo whatif saves", "Share of the build saved (medians). The tick is the probe's bound: the " + "most the change can save.") + y_end = top + sum(row_h + (gap if first else 0) for *_, first in rows) + for j, (name, key, title) in enumerate(EXPERIMENTS): + px = label_w + j * (panel_w + panel_gap) + svg.swatch(px, 84, key, title) + for pct in (0, 50, 100): + gx = px + pct / 100 * plot_w + svg.line(gx, top + gap - 4, gx, y_end, "axis" if pct == 0 else "grid") + svg.text(gx, y_end + 16, f"{pct}%", size=10, fill="muted", anchor="start" if pct == 0 else "middle") + y = top + for doc, e, r, first in rows: + if first: + y += gap + if y > top + gap: + svg.line(24, y - gap / 2, label_w - 8, y - gap / 2, "grid") + cy = y + row_h / 2 + if doc: + svg.text(24, cy + 4, doc, weight="600") + svg.text(label_w - 12, cy + 4, NAME[e], fill="ink2", anchor="end") + base = r["baseline"]["median_ms"] + by = {x["experiment"]: x for x in r["experiments"]} + for j, (name, key, title) in enumerate(EXPERIMENTS): + px = label_w + j * (panel_w + panel_gap) + row = by.get(name) + if not row: + continue + label = f"{doc or ''} {NAME[e]}, {title}".strip() + right = px + if row.get("bound_ms") is not None and row["status"] == "measured" and base > 0: + bx = px + min(row["bound_ms"] / base, 1) * plot_w + svg.line(bx, y + 4, bx, y + row_h - 4, "ink2", 2, f"{label}: bound {fmt_time(row['bound_ms'])}") + right = bx + if row["status"] != "measured": + note = "–" if row["status"] == "not-run" else "fails" + svg.text(max(px + 4, right + 5), cy + 4, note, size=11, fill="muted") + continue + noise = row["within_noise"] + if row["saved_ms"] > 0: + w = min(row["saved_pct"], 100) / 100 * plot_w + changed = row["output"]["status"] == "changed" + tip = (f"{label}: saved {fmt_time(row['saved_ms'])} of {fmt_time(base)} ({row['saved_pct']:.0f}%)" + + (", within noise" if noise else "") + + (f"; output changed: {row['output']['evidence']}" if changed else "")) + svg.bar(px, y + 6, max(w, 2), row_h - 12, "muted" if noise else key, tip) + right = max(right, px + max(w, 2)) + if noise: + svg.text(right + 5, cy + 4, "noise", size=11, fill="muted") + continue + if row["saved_ms"] < 0: + svg.text(right + 5, cy + 4, f"{fmt_time(-row['saved_ms'])} slower", size=11, fill="ink2") + continue + lab = f"{row['saved_pct']:.0f}%" + (" ≠" if row["output"]["status"] == "changed" else "") + once = f" · once {fmt_time(row['one_time_ms'])}" if row.get("one_time_ms") else "" + room = px + panel_w - (right + 5) + if once and text_width(lab + once, 11) <= room: + lab += once + svg.text(right + 5, cy + 4, lab, size=11, fill="ink2") + y += row_h + svg.text(24, height - 54, "gray: within noise, no larger than the runs' spread. fails: the variant did not " + "build. –: not tried here. ≠: the output changed.", size=11, fill="muted") + svg.text(24, height - 38, "once: paid on the first build only, building the format or every picture.", + size=11, fill="muted") + svg.text(24, height - 16, caption(data), size=11, fill="muted") + return svg.done() + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--results", type=Path, default=HERE / "results") + ap.add_argument("--out", type=Path, default=HERE / "img") + a = ap.parse_args() + a.out.mkdir(parents=True, exist_ok=True) + data = load(a.results) + for mode, t in THEMES.items(): + charts = {"phases": phases_chart(data, t) if data["phases"] else None, + "overhead": overhead_chart(data, t), "whatif": whatif_chart(data, t)} + charts["packages"], table = packages_chart(data, t) + for name, svg in charts.items(): + if svg: + (a.out / f"{name}-{mode}.svg").write_text(svg) + if table: + (a.results / "packages.md").write_text(table) + print(f"charts in {a.out}") + + +if __name__ == "__main__": + main() diff --git a/docs/benchmarks/corpus.toml b/docs/benchmarks/corpus.toml new file mode 100644 index 0000000..3bc130e --- /dev/null +++ b/docs/benchmarks/corpus.toml @@ -0,0 +1,116 @@ +# The benchmark documents. fetch.py lays them out under one directory and bench.py builds +# them. Each document comes from one of three places: +# +# source = "docs/" this repository +# generate = "