CY2024 Mesa software rendering fails from an ASWF/distro LLVM 17 collision

[nicolaspopravka]

Summary

CY2024 is the one remaining failure in the full-range run-demo canary
(CY2023/2025/2026/2027 all render). The CY2024 prebuilt registers only the GL
renderer, and its GL render crashes during Qt's offscreen GL setup under the
same headless harness that works on every other year. No image is produced.

Reproduction (headless GHA runner, free)

run-demo mounts demo/run1 into ghcr.io/nicolaspopravka/usd-render-benchmark:2024
and runs render_script.sh under Xvfb + Mesa llvmpipe software rendering
(dnf install mesa-dri-drivers mesa-libEGL libepoxy, LIBGL_ALWAYS_SOFTWARE=1 xvfb-run).
CY2024 prebuilt's usdrecord only accepts --renderer GL:

Running: rez env usd -- usdrecord --camera main_cam --renderer "GL" --purposes render   assets/full_assets/Teapot/Teapot.usd "renderers/GL/Teapot.jpg"
QXcbIntegration: Cannot create platform offscreen surface, neither GLX nor EGL are enabled
QXcbIntegration: Cannot create platform OpenGL context, neither GLX nor EGL are enabled
/tmp/rez_context_dl8ojeev/rez-shell.sh: line 8:    84 Segmentation fault      (core dumped) usdrecord --camera main_cam --renderer GL --purposes render assets/full_assets/Teapot/Teapot.usd renderers/GL/Teapot.jpg
Command exited with non-zero status 139
Time: 0:02.15
Memory: 241100 KB
ERROR: usdrecord returned non-zero exit status

• Canary runs: 34265884133 (2024) and 34244219356 (GL-token round) show the same failure.
• Runnable image: ghcr.io/nicolaspopravka/usd-render-benchmark:2024 (built from
  aswf/ci-vfxall:2024 via the stack's work/cy-runnable overlay branch).
• The renderers/GL/Teapot.jpg present in the branch is the mounted
  checkout's own committed example output, not a new frame.

Root cause confirmed locally (2026-09-09)

The Qt GLX plugin is present and loads successfully. Mesa's software driver fails to load because the image selects an incompatible LLVM library:

/usr/lib64/dri/swrast_dri.so: undefined symbol: LLVMInitializeAMDGPUTargetInfo, version LLVM_17

The image puts /usr/local/lib first in LD_LIBRARY_PATH. Mesa therefore resolves libLLVM-17.so to ASWF's LLVM 17.0.6, built with only NVPTX;X86 targets. The distro driver requires the AMDGPU initializer even for this CPU-rendering invocation. The compatible distro LLVM already exists at /usr/lib64/llvm17/lib/libLLVM-17.so; no LLVM installation is needed.

This explains the sequence: Mesa driver loading fails, GLX is unavailable, Qt context creation fails, and the unchecked framebuffer construction used by stock usdrecord segfaults.

Tested runnable image:
ghcr.io/nicolaspopravka/usd-render-benchmark@sha256:fd23d615cf10bfcacb185b9ca7f40e41db7131d221dd67b4a95bae77a795f12b
(ASWF 2024-clang17.9, OpenUSD 24.08, Qt/PySide6 6.5.6, distro Mesa 23.1.4).

A matched original → distro → original LLVM selection in one disposable container gave:

Check	Original	Distro LLVM first	Original restored
Load swrast_dri.so	Missing LLVM symbol	Loads	Missing LLVM symbol
Qt context/current binding	Fails	Succeeds	Fails
Qt framebuffer construction	Exit 139	Valid/bound, exit 0	Exit 139

The successful case reports llvmpipe LLVM 17.0.6 and OpenGL 4.5. The Qt test mirrors stock framebuffer setup without importing OpenUSD. The installed distro LLVM passed RPM verification before the comparison. Note that glxinfo returns 0 even when it prints the no-visual error.

Workaround for the Teapot rerun

Inside the container, after the existing Mesa installation and before starting Xvfb:

export LD_LIBRARY_PATH=/usr/lib64/llvm17/lib:$LD_LIBRARY_PATH
LIBGL_ALWAYS_SOFTWARE=1 xvfb-run -a bash render_script.sh

Apply the environment to both Xvfb and the client. QT_QPA_PLATFORM=xcb is unnecessary for this correction: XCB was already active.

The underlying driver error can be checked without Qt or a scene:

python3 -c 'import ctypes; ctypes.CDLL("/usr/lib64/dri/swrast_dri.so")'
LD_LIBRARY_PATH=/usr/lib64/llvm17/lib:$LD_LIBRARY_PATH \
  python3 -c 'import ctypes; ctypes.CDLL("/usr/lib64/dri/swrast_dri.so")'

Upstream candidate and remaining validation

ASWF's Clang recipe defaults to host;NVPTX; ci-common deploys the package under /usr/local and prioritizes that library directory. The tested image's llvm-config --targets-built confirms NVPTX X86.

The upstream candidate is the collision between this reduced-target LLVM build and distro Mesa's LLVM dependency. A durable fix needs to keep Mesa on a compatible LLVM or make the exposed LLVM satisfy its requirements; simply adding one missing target has not been built or validated. Separately, stock usdrecord could detect failed context creation before constructing the framebuffer.

The graphics mechanism is confirmed locally; the corrected full Teapot render remains pending. On the local Docker host, even usdrecord --help exits with SIGILL (132), independently of this Qt failure. I will rerun Teapot on the native runner with the workaround and check that it produces a new image. No ASWF report or OpenUSD report has been filed yet.

[nicolaspopravka]

Validated on the GHA runner (2026-09-09).

The workaround is now in the run-demo workflow (usd-render-benchmark-stack main, 19f79f1): when /usr/lib64/llvm17/lib is present, it is prepended to LD_LIBRARY_PATH before Xvfb. Full-range dispatch:

• CY2024 canary run 34328189691 → demo/run1 render Success 0:02.15, image written (renderers/GL/Teapot.jpg sha 67e7217e…), log clean — no QXcbIntegration lines, Camera: + HdStormRendererPlugin + Recording time code then exit 0.
• The CY2024 frame is byte-identical to the CY2023 GL frame (67e7217e…), both real-GL renders.
• CY2023/2025/2026/2027 unchanged (2023 67e7217e…; 2025–27 5b1a1943…) — the env line is a no-op on the other years.

Result: the full-range run-demo canary is now 5/5 green.

Harness-level resolution only. The upstream candidate in the body (ASWF Clang recipe host;NVPTX + ci-common prioritizing /usr/local, so distro Mesa cannot load its LLVM-dependent swrast_dri.so) remains documented, not filed — an ASWF report or a distro-LLVM/hermes preference decision is Nicolas'/Codex's call.