Skip to content

fix(quarto-chromium): CI-safe headless Chromium for typst/mermaid renders - #21

Merged
emmanuelbruno merged 5 commits into
feat/post-install-checksfrom
fix/chromium-ci-headless-render
Sep 10, 2026
Merged

fix(quarto-chromium): CI-safe headless Chromium for typst/mermaid renders#21
emmanuelbruno merged 5 commits into
feat/post-install-checksfrom
fix/chromium-ci-headless-render

Conversation

@emmanuelbruno

@emmanuelbruno emmanuelbruno commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Why

Render CI on ghcr.io/ebpro/jupyter-base:quarto-full-feat-post-install-checks crashes on the mermaid step of the notes profile (typst). Evidence:

  • ebpro/notebook-ci run 34048342510, job 101527336817 (step "Render with Quarto (Docker)", exit 1), and run 33537153211 (identical, on the quarto-full-develop-b511eae tag):
    ERROR: AssertionError: 
    Error occurred during cleanup: TypeError: Child process has already terminated
    
  • Affects all current quarto-full tags. The old solen image failed earlier with a different, visible error (ERROR: Chrome not found) — this one is silent.
  • Note: this repo has no git tagsquarto-full-feat-post-install-checks is a GHCR-only image tag, built from branch feat/post-install-checks (HEAD 45e4ad6). The published image also carries a stale build-cache layer (quarto 1.8.24 vs the versions SoT 1.8.26) — a post-merge rebuild refreshes that too.

Root cause — CONFIRMED (verified locally, emulated amd64, indicative)

21 shared libraries are missing from the image (verified via ldd on the bundled Chromium build): libnss3, libnssutil3, libsmime3, libnspr4, libatk-1.0, libatk-bridge-2.0, libcups, libdrm, libxkbcommon, libXcomposite, libXdamage, libXfixes, libXrandr, libgbm, libpango-1.0, libcairo, libgtk-3, libgdk-3, libasound, libatspi, libxshmfence.

Chromium therefore dies at startup with exit 127 (error while loading shared libraries: libnss3.so: cannot open shared object file) — before any flag matters. Quarto's criClient error path (an assert(readerResult.done) after reading a single stderr chunk) then masks the real failure as the bare ERROR: AssertionError:, and the cleanup handler reports Child process has already terminated.

Refinements from the code + local runs:

  • Quarto already passes --no-sandbox --disable-gpu to the browser spawn, so the sandbox was not the gap.
  • /dev/shm is 64 MB in CI (faithful locally); A/B runs with/without --disable-dev-shm-usage failed identically, so the shim's --disable-dev-shm-usage is defense-in-depth, not the primary fix.
  • Local evidence: the exact CI signature was reproduced with a quarto render of a minimal typst+mermaid doc on the unmodified image; ldd is clean after installing the 18 verified noble/t64 packages (package set installed live in-container, not guessed). QEMU user-mode emulation cannot execute this Chrome build (zygote fatal / SEGV), so a full-render PASS could not be demonstrated locally — compensated by ldd-clean + exact repro + in-container logic tests of the new check.

Changes

  1. features/quarto-chromium/install.sh — apt block installing the 18 verified runtime libraries (with the ldd evidence as a comment), plus an idempotent in-place shim of the discovered Chromium binary: original moved to <bin>.real, POSIX sh wrapper at the same path adding --no-sandbox --disable-gpu --disable-dev-shm-usage (marker-guarded, chmod 755, chowned to the notebook user). In-place wrapping is layout-agnostic and covers both of Quarto's discovery routes (the /usr/local/bin + ~/.local/bin symlinks and the ~/.local/share/quarto/chromium install dir all resolve to this file).
  2. features/quarto-chromium/feature.json — the existence-grep post-install check is replaced with a functional headless smoke: timeout 60 chromium --headless --disable-gpu --dump-dom about:blank, asserting exit 0 and non-empty output (real chrome stderr is surfaced on failure). A broken Chromium now fails the build, not courseware CI.
  3. features/quarto-cli/install.sh — build-time discovery of the bundled tools dir ($(dirname "$QUARTO_EXE")/tools/*/typst, no hardcoded arch) appended to PATH in both the /usr/local/bin/quarto wrapper and /etc/profile.d/quarto.sh. Quarto itself resolves bundled typst via QUARTO_BIN_PATH; this makes direct typst invocations from user notebook cells resolve.

Validation

  • PR CI runs ci-validate only (static: feature validation, profile/devcontainer generation, versions sync, pytest, ruff) — ci-build is gated to develop/workflow_dispatch and does not run for this PR.
  • The functional post-install check gates real builds: per-feature at build time (RUN_POSTINSTALL_CHECKS=true) and post-build via the run-postinstall-checks workflow input (default false).
  • Local (emulated amd64, indicative): repro + ldd + apt verification transcript as described above; shim wrapper tested for 5 idempotency/repair scenarios; check command logic tested in-container under the exact runner semantics (bash, set -uo pipefail, eval) for success / non-zero-exit / empty-output paths.

After merge

  1. Rebuild/publish the tag from this branch (ci-publish) — also refreshes the stale 1.8.24 → 1.8.26 layer.
  2. Re-run the ebpro/notebook-ci canary (its ci-render.yml already points at quarto-full-feat-post-install-checks).
  3. Roll out to the 17 remaining courseware render wrappers.

If still failing

  • Missing fonts (affects render quality, not startup).
  • --shm-size on the docker run in the render workflow as a workflow-side fallback for the 64 MB /dev/shm.

Addition (2026-09-09): Quarto 1.8.24 → 1.10.18. The canary on the PR image got past the Chromium boot failure but exposed a second, internal Quarto SVG→PNG bug (Couldn't find an svg element in svg string); the upgrade addresses it. New image build to follow.

Copilot AI lite review requested due to automatic review settings September 7, 2026 19:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The Chromium shim currently derives the .real path from dirname "$0", which breaks when Chromium is invoked via the symlinks created by this feature (common PATH resolution path).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR hardens the quarto-full image’s headless Chromium setup so Quarto typst/mermaid renders don’t crash in CI, by ensuring required shared libraries are present, adding a CI-safe Chromium invocation path, and making Quarto’s bundled tools discoverable on PATH for direct use.

Changes:

  • Install the verified Ubuntu runtime libraries needed for Quarto’s downloaded Chromium to start successfully.
  • Add an in-place Chromium shim (and a functional post-install smoke test) so broken headless Chromium fails the image build rather than failing downstream render CI.
  • Extend Quarto CLI wrappers to add Quarto’s bundled tools directory (e.g., typst) to PATH for direct user invocations.
File summaries
File Description
features/quarto-cli/install.sh Adds Quarto bundled tools directory discovery and appends it to PATH in system and wrapper scripts.
features/quarto-chromium/install.sh Installs Chromium runtime libs and introduces an in-place shim wrapper for CI-safe Chromium flags.
features/quarto-chromium/feature.json Replaces existence checks with a functional headless Chromium smoke test.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +179 to +183
#!/bin/sh
# solen-chromium-shim: CI-safe flags for headless Chromium in restricted
# containers (baked by features/quarto-chromium; original: ${bin_name}.real)
exec "\$(dirname "\$0")/${bin_name}.real" --no-sandbox --disable-gpu --disable-dev-shm-usage "\$@"
SHIM

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed — the readlink guard I added only ran at wrap time (resolving where the binary to wrap lives); the generated wrapper itself still used dirname "$0" at runtime, so invocation via the /usr/local/bin/chromium or ~/.local/bin/chromium symlinks looked for <bin>.real in the symlink directory and died with exit 127. Fixed in b0bb6f7: the wrapper now resolves its own canonical path first (self="$(readlink -f "$0" 2>/dev/null || echo "$0")") before deriving the .real target. Verified in-container with a fake binary reproducing the exact symlink layout: old logic failed via symlink (exec: /usr/local/bin/chrome.real: not found, rc=127); new logic passes via both symlinks and direct invocation, with re-wrap idempotency and original-restoration still passing.

The generated wrapper derived the .real target from dirname("$0"),
which breaks when the shim is invoked via the /usr/local/bin or
~/.local/bin symlinks (the common PATH resolution route): $0 points at
the symlink, so the wrapper looked for <bin>.real in the symlink
directory and failed with exit 127. The wrapper now resolves its own
canonical path via readlink -f first.

Verified in-container with a fake binary: old logic failed via symlink
(exec: /usr/local/bin/chrome.real: not found, rc=127); new logic passes
via both symlinks and direct invocation, and re-wrap idempotency /
original-restoration scenarios still pass.
@emmanuelbruno
emmanuelbruno merged commit ec798c7 into feat/post-install-checks Sep 10, 2026
4 checks passed
@emmanuelbruno
emmanuelbruno deleted the fix/chromium-ci-headless-render branch September 10, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants