fix(overlay): fall back to zink GL after a failed start, instead of guessing

[srsholmes]

Problem

The overlay crash-loops and never opens on some AMD machines. Reported by a user on CachyOS desktop (OneXPlayer Super X, Radeon 8060S) and reproduced locally on CachyOS desktop (Radeon 890M).

The bundled CEF 147 segfaults inside startEventLoop() during GL init on radeonsi — before any Loadout code runs. Bun reports panic(main thread): Segmentation fault at address 0x0; the delivered signal is SIGILL from Bun's panic handler. systemd then restarts it every 5s forever.

Fix

Don't predict who's affected — detect the failure and react:

• ExecStopPost leaves a marker at %t/loadout-overlay-gl-fallback when the service exits for any reason other than a clean stop.
• Restart=on-failure brings the unit back, sees the marker, and retries with MESA_LOADER_DRIVER_OVERRIDE=zink.

zink routes GL through Vulkan/RADV and keeps hardware acceleration (glxinfo: zink Vulkan 1.4 (AMD Radeon 890M, RADV STRIX1)), unlike LIBGL_ALWAYS_SOFTWARE=1 which drops to llvmpipe and costs CPU/battery on a handheld. vulkan-radeon already ships alongside mesa, so no new dependency.

The marker lives under %t (/run/user/<uid>), which the boot clears — so the normal driver is retried once per boot. When the underlying bug is fixed upstream this stops engaging on its own. Self-expiring; nothing to remember to revert.

Why there is no version gate — I measured, and the obvious one is wrong

An earlier revision of this PR gated on AMD && mesa >= 26.1, from a timeline that appeared to implicate a mesa upgrade. Review challenged it, I tested it, and it is false.

Holding everything else constant on kernel 7.1.3, radeonsi still crashes on:

Downgraded to	radeonsi result
mesa 26.0.5 (from 26.1.4)	still crashed
llvm-libs 22.1.3 (from 22.1.8)	still crashed
libdrm 2.4.131 (from 2.4.134)	still crashed

And on mesa 26.0.5 the driver matrix is identical to 26.1.4 — radeonsi crashes, zink works, software works. The mesa version is not the trigger.

The original timeline was void: the last successful start (11:27:57) ran continuously until the 11:41:57 reboot, so it was already running when mesa upgraded at 11:36 — the overlay was never once started in the window that would have implicated mesa. The update was also 605 packages, not the four I named.

A mesa gate would therefore have been:

• over-inclusive — firing on SteamOS/Bazzite machines that may be fine;
• provably under-inclusive — Bazzite 43 ships mesa 26.0.5, which I've now proven is affected, and would never have been fixed.

The true trigger is still unidentified. The last known-good run predates a kernel bump 7.0.1 → 7.1.3, which is the prime remaining suspect (untested — needs a reboot). Detecting the failure sidesteps this entirely: it cannot be wrong about who is affected, and it covers Debian/Ubuntu/NixOS which have neither pacman nor rpm.

It also deletes two real bugs that review found in the version-gate code: rpm -q fails open (its "package X is not installed" message goes to stdout, proven from rpm's source query.cc:470 → RPMLOG_NOTICE → stdout, so it was consumed as a version string), and sort -V fired on any non-numeric garbage.

Verification

End-to-end on CachyOS desktop, from a cleared marker (simulating a fresh boot):

13:04:24  Started Loadout Overlay
13:04:24  panic(main thread): Segmentation fault at address 0x0
          [ExecStopPost arms marker]
13:04:37  Started Loadout Overlay
13:04:37  [loadout-overlay] previous start failed - retrying GL via zink
13:04:37  GlobalShortcut: Event loop ready

NRestarts=1; live process environment confirms MESA_LOADER_DRIVER_OVERRIDE=zink. Also verified:

• a deliberate systemctl stop does not arm the fallback ($SERVICE_RESULT=success) — checked explicitly, since otherwise every stop would silently pin zink;
• a real failure does arm it;
• %t resolves to /run/user/1000 and $SERVICE_RESULT is readable in ExecStopPost (probed directly, not assumed);
• unit parses clean under systemd-analyze verify.

Cost

One crash per boot on affected machines, then recovery (~5s via RestartSec). The overlay is opened on demand, so this is invisible in practice. It also produces one coredump per boot on affected machines. That's the price of not guessing — and given the predicate would have been wrong, it's the right trade.

Distro impact

• SteamOS / Bazzite / Debian / anything: no version sniffing, no package-manager assumptions. Machines that work keep radeonsi and never see the fallback; machines that crash get fixed regardless of distro or mesa version.
• Respects a user-set MESA_LOADER_DRIVER_OVERRIDE and never overrides it.

Applied to both loadout-overlay.service and the scripts/install.sh heredoc, kept byte-identical (per c0f36c4, these drift).

Notes

Branch name is legacy (fix/overlay-mesa-26-radeonsi-zink) — the mesa-26 framing is exactly what this revision abandons.

Separate from #212 (gamescope detection self-match) so each reverts independently. Both touch the same ExecStart region, so whichever lands second needs a rebase.

🤖 Generated with Claude Code

[srsholmes]

Reviewed + tested on SteamOS hardware — design confirmed, one fix pushed

Tested on a SteamOS box with the same GPU as the reporter: AMD Radeon 8060S Graphics (radeonsi, strix_halo) — i.e. the same silicon as wouter's OneXPlayer Super X, different distro. Environment: kernel 6.18.38-valve1-1-neptune, mesa 26.1.2 (git-18b3910ac0), systemd 257, gamescope-wl live in Gaming Mode.

The "detect, don't predict" call is right — here's independent evidence

The overlay is completely healthy here on radeonsi:

NRestarts=0
ActiveState=active
SubState=running
OpenGL renderer string: AMD Radeon 8060S Graphics (radeonsi, strix_halo, ACO, DRM 3.64, 6.18.38-valve1)

Same GPU, mesa 26.1.2 — which sits between the 26.0.5 and 26.1.4 this PR already proved both crash — and radeonsi is fine. So the trigger is provably neither the GPU nor the mesa version, and any hardware-or-version gate would have shipped wrong. This corroborates the PR's own downgrade matrix from a second distro.

It also fits the kernel suspicion left open in the description: SteamOS is on 6.18.38-valve1, CachyOS on 7.1.3.

zink works here too, including under gamescope, on both displays:

OpenGL renderer string: zink Vulkan 1.4(AMD Radeon 8060S Graphics (RADV STRIX_HALO) (MESA_RADV))

/usr/lib/dri/zink_dri.so and radeon_icd.x86_64.json are both present, so if the fallback ever engages on a Deck it won't break.

The systemd comment-stripping assumption holds — confirmed against the live ExecStart property dump on systemd 257. The # comment block added inside the sh -c is stripped before exec, so it's safe.

The problem: it armed on the wrong signal

ExecStopPost=-/bin/sh -c '[ "$SERVICE_RESULT" = "success" ] || touch "%t/loadout-overlay-gl-fallback"'

This fires on any unclean exit, which is a far wider net than the bug it targets. The radeonsi crash lands ~118ms in, inside startEventLoop(), before a line of our code runs. A crash two hours later — OOM, a later CEF fault, the move/resize crash vendor/README.md documents, a SIGKILL — armed the marker identically and silently moved the machine onto zink for the rest of the boot.

On a device where radeonsi works fine (measured above: NRestarts=0) that swaps a working driver for no reason and buries the real cause. Every subsequent bug report then arrives on a driver nobody chose.

It also undercuts the "if zink fails too … no worse than before" claim: that only holds if the marker never false-arms. On a box where radeonsi works but zink is absent (no Vulkan ICD — VM, nouveau-only), one unrelated crash pins a broken driver for the whole boot, turning a transient crash into a permanent loop.

Fix pushed (4aef101)

Gate on "died before it was ever ready", which is precisely the GL-init signature. The app writes a readiness flag once it has a live event loop (registering a global shortcut proves GL came up); the unit clears it on every start; the fallback arms only if it's still absent at exit.

Truth table, exercised against the real ExecStopPost predicate:

scenario	result
clean stop, was ready	not armed
clean stop, never ready	not armed
crash before ready (GL init bug)	ARMED
crash AFTER ready (unrelated bug)	not armed
core-dump before ready	ARMED
SIGKILL after ready	not armed

CachyOS behaviour is unchanged — the crash lands before the flag is ever written, so the fallback still engages and still fixes the overlay. systemd-analyze --user verify passes; typecheck clean.

⚠️ Gotcha worth recording

If anyone is tempted to do this with a timer instead: %s is a systemd specifier (user shell), so date +%s in a unit silently becomes date +/bin/bash. It needs %%s. The readiness flag sidesteps that entirely.

What to verify on CachyOS

The readiness flag is the new load-bearing piece:

ls -la "$XDG_RUNTIME_DIR/loadout-overlay-ready"        # should exist once the overlay is up
journalctl --user -u loadout-overlay -b | grep -i zink # 'retrying GL via zink' on the 2nd start

Expected: first start crashes (no flag) → marker armed → second start logs retrying GL via zink → overlay opens → flag appears.

The regression to watch for: if the flag never appears on a working overlay, the gate would stop the fallback from ever arming — which would show up immediately as the overlay not recovering.

Note on the root cause

The PR description is right and the "was due to a mesa update" theory is the one thing the evidence contradicts — radeonsi still crashed on mesa 26.0.5 / llvm 22.1.3 / libdrm 2.4.131, and SteamOS runs 26.1.2 on radeonsi with zero restarts. Kernel remains the prime untested suspect. None of which matters to the fix: detection can't be wrong about who's affected.

_{🤖 Reviewed and tested with Claude Code on SteamOS / Radeon 8060S}

[srsholmes]

Verified 4aef101 on CachyOS — the machine that actually triggers the bug

The review asked for CachyOS verification of the readiness gate, since it was tested on SteamOS where radeonsi never crashes — i.e. the gate had never run on hardware that triggers it. Ran it here.

Environment: CachyOS, kernel 7.1.3-2-cachyos-deckify, systemd 261, Radeon 880M/890M (Strix), gamescope. Built 4aef101 from source and installed it.

All three cases pass

Scenario	Expected	Result
Crash before ready (the real radeonsi GL bug)	arms → zink recovery	✅ pass
SIGKILL after a healthy start	not armed	✅ pass
Clean systemctl stop	not armed	✅ pass (result=success)

Crash-before-ready — the genuine bug, end to end:

16:57:41  Starting → panic(main thread): Segmentation fault at address 0x0
16:57:49  Main process exited, code=exited, status=132     ← no ready flag → marker armed
16:57:54  Scheduled restart → [loadout-overlay] previous start failed - retrying GL via zink
16:57:54  === CEF EVENT LOOP STARTED ===  → GlobalShortcut: Event loop ready

Post-ready kill — run deterministically with a Restart=no drop-in so the auto-restart couldn't confound the result (my first attempt raced: the restart hit radeonsi and armed the marker legitimately ~5s later, which is easy to misread as a false-arm):

service state: failed / result=signal
ready flag still present: YES
MARKER ARMED BY POST-READY KILL: NO

Under d2f3059 that identical kill would have armed and pinned zink for the boot. That's the regression this commit fixes, reproduced on real hardware.

The stated regression risk didn't materialise

if the flag never appears on a working overlay, the gate would stop the fallback from ever arming

The flag appears reliably. XDG_RUNTIME_DIR and %t both resolve to /run/user/1000, so app and unit agree on the path — worth noting since that's an implicit cross-file contract with no shared constant, and it fails silently (degrades to arm-on-any-failure) if a session ever sets them differently.

Comment-stripping in the continued sh -c also holds on systemd 261, not just 257 — proven by the unit running at all; had it not stripped, exec launcher would be commented out.

Review points (not blockers)

1. The gate narrows false-arming without closing it. It separates "died before ready" from "died after ready" — not "GL init killed us" from "something else killed us early". Any early failure (bad DISPLAY, missing lib, a transient race) still arms zink on a machine where radeonsi is fine. The worst case raised in review — radeonsi works but zink is absent (VM, nouveau-only) — is narrowed, not eliminated: a transient early crash there still pins a broken driver for the whole boot. Worth stating plainly rather than implying it's solved.
2. Correctness is coupled to code placement. The flag means "we reached index.ts:787", a proxy for "GL init survived". Sound today, but anyone moving that block earlier or adding an await before it silently breaks the gate, with no test to catch it.
3. Minor: the flag-clearing ExecStartPre sits after the ≤30s backend-wait ExecStartPre. Not a bug — clearing state just reads better before a long wait.

Unrelated finding worth its own issue

CEF re-parents into app-org.chromium.Chromium-<pid>.scope (ppid → user manager), outside loadout-overlay.service. A systemctl kill -s KILL logged "Killed unit cgroup … with SIGKILL" yet the tree survived it and kept running. So systemctl stop doesn't reap the overlay, and a leaked instance holds the CEF profile lock + devtools port — a later start then comes up with Cannot create profile / bind() failed: Address already in use, registers its shortcut, reports input intercept ready, and looks healthy in systemctl status but has no window and never opens. I hit exactly that during this testing. Not caused by this PR, but it makes crash/restart cycles more expensive than they look.

_{🤖 Tested with Claude Code on CachyOS / Radeon 890M / systemd 261}

[srsholmes]

Root cause found — it was never the kernel, mesa, or CEF

Follow-up to my verification comment above: dug past Bun's panic handler with gdb and traced the crash to its origin. Full analysis and fix in #214; the short version for this thread:

Arch's webkit2gtk 2.52.5-1 (in the 2026-07-14 update batch that started all this) regressed to exporting libstdc++'s private std::call_once internals from JSC. Electrobun's wrapper links webkit2gtk even in CEF mode, JSC wins interposition over libstdc++, and std::call_once's TLS state splits in two: mesa's libLLVM writes the callback into one copy, __once_proxy reads NULL out of the other → call *0x0 → the exact Segmentation fault at address 0x0 during radeonsi's LLVM shader-compiler init.

This resolves the open questions in the PR description:

• "The trigger is still unidentified / kernel is the prime suspect" — answered: webkit2gtk 2.52.3→2.52.5. The kernel was a bystander; no reboot bisect needed.
• Why the mesa/llvm downgrade matrix all still crashed — mesa and LLVM were never the broken parts; webkit2gtk stayed at 2.52.5 through those tests.
• Why zink works — ACO never calls LLVM, so the poisoned call_once path is never taken.
• Why SteamOS is fine on the same silicon — its webkit2gtk build doesn't ship the Arch 2.52.5 regression. (Verifiable on the SteamOS box with one command: nm -D /usr/lib/libjavascriptcoregtk-4.1.so.0 | grep once_proxy — T = affected, U or nothing = fine.)

Implication for this PR: the zink fallback is no longer the fix for this bug (#214 removes the trigger and keeps machines on native radeonsi), but it just spent a month proving exactly why a detect-and-react net is worth carrying — it kept affected machines usable through an upstream regression nobody predicted. I'd merge both: #214 as the cure, this as the safety net, with the description's root-cause section updated to point at the webkit2gtk finding.

_{🤖 Analysis with Claude Code — gdb + LD_DEBUG + package-diff forensics on CachyOS / Radeon 890M}