Skip to content

Wait for a slow-exiting shell before restarting it - #13141

Closed
Nejcc wants to merge 3 commits into
omacom:quattrofrom
Nejcc:fix/restart-shell-slow-exit
Closed

Nejcc wants to merge 3 commits into
omacom:quattrofrom
Nejcc:fix/restart-shell-slow-exit

Conversation

@Nejcc

@Nejcc Nejcc commented Sep 24, 2026 •

Copy link
Copy Markdown

Problem

omarchy restart shell can report success and leave no shell running (no bar).

The kill loop runs timeout 5 quickshell kill …. When the shell is busy enough to take longer than 5 s to exit, timeout ends the command, the loop reads that as "no instances left", and the replacement is launched while the old shell is still alive. The new quickshell -n exits at once as a duplicate ("An instance of this configuration is already running."), which omarchy-launch-shell treats as a deliberate stop. The readiness ping can then be answered by the still-dying old shell, so restart exits 0 just before that shell goes away.

I hit this during a plugin stress test; the journal showed the new instance refusing as a duplicate 5 s before the old one logged "Exiting due to IPC request".

Reproduce

pid=$(quickshell list -p "$OMARCHY_PATH/shell" --any-display -j | jq -r '.[0].pid')
kill -STOP "$pid"; ( sleep 7; kill -CONT "$pid" ) &
omarchy restart shell; echo "exit=$?"    # exit=0
sleep 4; quickshell list -p "$OMARCHY_PATH/shell" --any-display   # no running instances, no bar

Fix

After the kill loop, wait until quickshell list reports no instance for the config. Anything still alive after a 5 s grace period gets SIGKILL; if an instance still won't go after 10 s, restart refuses to launch a duplicate and says so instead of claiming success.

Testing

  • test/shell.d/restart-shell-test.sh: the fake quickshell learns list and a stubborn mode where kill times out with the instance alive. The new case fails on quattro ("restart forces a shell that will not exit") and passes with this change; the existing cases still pass.
  • The reproduction above now ends with exactly one fresh shell and the bar back.
  • ./test/all: the only failures are 5 that fail identically on unchanged quattro on my machine (missing omarchy-pkgs checkout, VS Code theme, kernel headers), unrelated to this change.

Fixes #14085

Fixes #14013

A shell busy enough to outlast the 5s kill timeout ended the kill loop
while still alive. The replacement then exited at once as a duplicate,
and the readiness ping could be answered by the dying shell, so
omarchy restart shell reported success and left no shell running.

Wait until quickshell lists no instance, force any that outlast a 5s
grace period, and refuse to launch if one still won't exit.
@llstrk

llstrk commented Sep 24, 2026

Copy link
Copy Markdown

Automated AI review

Community review: Independent automated community review, unaffiliated with the Omarchy team, intended to help prepare PRs for their review.

Verified: the new wait loop fixes the reported case, a shell that exits cleanly but takes longer than the 5 s kill timeout. Two issues remain on the new SIGKILL path: the old shell's supervisor relaunches it, and the PID passed to kill -KILL is not validated.

Old shell exits cleanly about 7 s after the kill request (synthetic Quickshell, production scripts) restart exit live shells afterwards
base 0 0 (fresh -n refused as a duplicate)
head 0 1

Verified:

  • jq -r '.[]?.pid' matches the Quickshell v0.3.1 list -j output: a JSON array of objects with a numeric pid, or plain text when no instance is running.
  • The loop waits on the same instance-lock liveness that quickshell -n uses for its duplicate check.
  • The PR's test passes all 8 cases on head. On base the new case fails at restart forces a shell that will not exit, as the description states.

A force-killed shell is relaunched by its old supervisor

In a session the shell runs under omarchy-launch-shell. That launcher treats exit status 137 as a crash and runs quickshell -n again after a 1 s backoff. Nothing tells it that the SIGKILL came from a restart. Meanwhile the restart has already dispatched a fresh launcher. Quickshell's -n duplicate check is not atomic with lock creation: launch.cpp creates the instance lock only after GUI and plugin initialization. So the two launchers race.

Timeline from a synthetic Quickshell, driven by the production head omarchy-restart-shell and omarchy-launch-shell. Times are relative to the old supervisor's "status 137; relaunching" log line. The startup delays are chosen parameters, not measurements.

                                    0.2 s startup       1.2 s startup
restart dispatches new launcher     +0.11 s             +0.11 s
new instance passes -n check        +0.12 s             +0.12 s
new instance creates its lock       +0.33 s             +1.33 s
old supervisor's -n relaunch        +1.01 s  refused    +1.01 s  passes (no lock yet)
live shells afterwards              1                   2
restart exit                        0                   0

Impact: this affects only the escalation the PR adds, a shell still alive about 9-10 s into the restart. With a fast startup the only effect is a misleading crash-style journal entry. If startup between the check and the lock takes longer than about 0.9 s, two shells run the same config, and the restart still reports success: its readiness ping only checks that some shell answers. How long real Quickshell takes at that point was not measured. A third ordering is possible by source: the old supervisor wins, the fresh launcher is refused, and the shell keeps running under the old supervisor.

Suggested change: stop the old supervisor before forcing its shell. Its stop trap handles SIGTERM, forwards it to the shell and exits without relaunching. So SIGTERM the verified omarchy-launch-shell parent of a lingering PID first, and keep SIGKILL for whatever remains. A restart marker that the launcher checks before relaunching would also work. Keep relaunching after unrelated crashes. A diagnostic copy of the restart script that adds this SIGTERM step removed the relaunch at both startup delays and still passes the PR's test. Trade-off: in the 10 s give-up path, the old supervisor no longer brings a shell back on its own.

Optional test improvement: the new test's stubborn shell is a bare sleep, not a child of omarchy-launch-shell, and its assertions run before the launcher's 1 s backoff ends. In the synthetic runs, running the real launcher as its supervisor and counting launches after the backoff exposed the extra relaunch.

An unreadable instance record turns into kill -KILL 0

quickshell list decides an instance is live from its lock, then prints the pid stored inside the lock file. It never checks whether reading that data succeeded (paths.cpp checkLock/collectInstances, command.cpp listInstances). With Qt 6.11.2, a short read leaves the value at 0. Two conditions can produce such a record for a live shell:

  • A lock that was taken but whose data was never fully written. createLock takes the lock before writing and does not check the write.
  • A shell started by Quickshell 0.2.1 that is still running after an upgrade to 0.3.x. The 0.3.x reader expects two extra fields, and the pid read runs past the end of the old record.
list -j  ->  [{"pid": 0, ...}]  ->  lingering=(0)  ->  after grace: kill -KILL 0
                                                        = SIGKILL to the restart's own process group

Impact: kill -KILL 0 signals every process in the caller's process group. The restart dies with status 137 and no message, and the old shell is left running with no replacement. When the restart runs from omarchy-update through omarchy-update-restart, that group normally includes the update job itself. The upgrade case also needs the old shell to still be alive after the grace period. Base never passed a PID from list to kill, so this exposure is new in this PR. It was established from Quickshell and Qt source and was not executed.

Suggested change: validate each PID before signalling it, while still counting every listed instance so the loop keeps waiting and the 10 s give-up message still applies:

if (( SECONDS >= grace_deadline )); then
  for pid in "${lingering[@]}"; do
    [[ $pid =~ ^[1-9][0-9]*$ ]] && (( pid != $$ )) && kill -KILL "$pid" 2>/dev/null
  done
fi

Optionally, also confirm the target is a Quickshell process before forcing it.

Note: the comment kept above the kill loop still says the loop "can't race a dying shell". The new comment directly below it explains that it can.


Review information

Test scope: source reading of Omarchy at head and base, Quickshell v0.3.1 (plus the v0.2.1 lock layout), Qt 6.11.2 QDataStream and GNU timeout. Isolated command-only runs used the production restart and launch scripts with a synthetic Quickshell modeled on the upstream source, plus the PR's own test on head and base. Real Quickshell, Hyprland and the PR's live SIGSTOP reproduction were not run. Timings above come from the synthetic model. The pid: 0 path is source-derived only.

AI process: Opus 5.5 Medium coordination and synthesis, independent Opus 5.5 Xhigh and GPT 6 Sol Xhigh technical assessments with targeted follow-up questions, Opus 5.5 Medium editorial check.

Opt out: To stop receiving these reviews, reply to this comment saying so.

omarchybot and others added 2 commits October 3, 2026 03:41
omarchy-launch-shell treats a SIGKILLed shell as a crash and relaunches it after a one-second backoff, racing the replacement restart has just dispatched: two launchers, and with a slow enough startup two shells. TERM the supervisor first; its stop trap forwards the signal and exits without relaunching.

quickshell list reports pid 0 for an instance whose lock data it cannot read, and kill -KILL 0 signals the restart's own process group, which under omarchy-update includes the update. Only positive pids are forced; an unforceable instance still counts as lingering, so restart gives up with its message rather than launching a duplicate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@omarchybot omarchybot added the verified Omarchy Triage has verified that this issue is ready for final review label Oct 3, 2026
@greptile-apps

greptile-apps Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Medium risk] Adds graceful shutdown logic to the shell restart script.

The PR appears safe to merge, with a non-blocking improvement needed for failed instance queries.

Findings

  1. P2 Query errors skip the wait ▶

Summary

This PR waits for the old shell to disappear before launching its replacement. After five seconds, it stops the supervisor and force-kills the remaining shell. After ten seconds, it reports failure rather than launching a duplicate.

  • Adds tests for a stubborn shell, its supervisor, and an unreadable instance PID.
  • The instance query should distinguish errors from a confirmed empty list.
  • Tests were inspected but not run.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Request shell shutdown] --> B[List remaining instances]
  B --> C{Instances remain?}
  C -->|No| D[Launch replacement]
  C -->|Yes, under five seconds| B
  C -->|Yes, five to ten seconds| E[Stop supervisor and force shell exit]
  E --> B
  C -->|Yes, ten seconds elapsed| F[Report failure without launching]
Loading

Reviews (1) · Last reviewed commit: "Stop a forced shell's supervisor and nev..."

Comment thread bin/omarchy-restart-shell
# restart reports success and leaves no shell at all. So wait until no
# instance is listed, and force any that outlast a grace period.
shell_instance_pids() {
quickshell list -p "$CONFIG_DIR" --any-display -j 2>/dev/null | jq -r '.[]?.pid' 2>/dev/null

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Query errors skip the wait

If the instance query fails or returns unreadable JSON, shell_instance_pids produces no PIDs. The caller treats that as proof that the old shell has exited. After the kill command times out, this can launch the replacement too early, bringing back the duplicate-launch race and leaving the user without a bar.

Distinguish a confirmed empty list from a failed query, and report the latter instead of launching.

@omarchybot omarchybot added the ready Good to merge label Oct 3, 2026
@omarchybot

Copy link
Copy Markdown
Collaborator

Reviewed and tested on a disposable Omarchy VM. The bug is real and this fixes it: when the old shell takes more than 5 s to exit, timeout ends the kill loop with it still alive, and the replacement is refused as a duplicate. Your slow-exit test fails on quattro and passes here. This is also what #14085 and #14013 report after omarchy update (the old shell logs "Exiting due to IPC request", then 5–6 s later the new one says "already running"), so the description now closes both.

I pushed two commits to your branch:

  • c4c0f03 merges quattro, which had added a notification-recovery case at the end of test/shell.d/restart-shell-test.sh. Both cases are kept.
  • 13c705c fixes the two problems @llstrk found in the force-kill path. Before the SIGKILL, restart now sends TERM to the lingering shell's omarchy-launch-shell parent. Otherwise that supervisor treats exit 137 as a crash and relaunches after its 1 s backoff, racing the replacement restart has just started. Its stop trap exits without relaunching. The commit also forces only positive pids, because quickshell list reports pid 0 for an instance lock it cannot read, and kill -KILL 0 would kill restart's own process group (under omarchy update, that includes the update). It also corrects the comment above the kill loop, which still claimed the loop could not race a dying shell. Each fix has a regression test, and each test fails with its fix removed: with the supervisor left alive the test sees a third launch, and with the pid guard removed restart dies with status 137.

On the worker, restart-shell-test.sh passes 12 of 12 at 13c705c. Across the full ./test/cli and ./test/shell runs, the only failures are 7 that also fail on unchanged quattro (VS Code theme, kernel headers, logo animation and others unrelated to this change).

Greptile's point that a failed quickshell list reads as "no instances" doesn't hold up as a defect. A failed list falls back to exactly what quattro does today. With no instance running, list -j prints plain text rather than JSON, so telling an error from an empty list would mean parsing that text. And refusing to launch when listing fails would leave no shell at all in the cases where listing breaks with nothing running, such as a full runtime tmpfs (#13287).

Other open pull requests fix the same bug: #11015 waits up to 30 s but then launches anyway, and also carries a separate Bar.qml performance change. #11014 retries the launch, but stops retrying once the dying old shell answers the readiness ping. #7546 stops the supervisors, but gives up after 5 s and launches nothing. This one is the smallest of them and the only one that both waits for the old shell to exit and refuses to launch a duplicate when it won't. #12983 overlaps on the signal fallback, but it addresses a wedged IPC socket (#12641), which is a different bug.

Second opinion: Codex at medium effort compared all of these and reviewed the final head. It found no defect, and agreed that this pull request is the one to keep. Its independence from this review isn't guaranteed. This now waits on the maintainer.

@omarchybot omarchybot removed verified Omarchy Triage has verified that this issue is ready for final review ready Good to merge labels Oct 4, 2026
@omarchybot

Copy link
Copy Markdown
Collaborator

Closing this alternative after #11015 merged into quattro. The merged restart gives the old shell 30 seconds to exit and then checks for its disappearance before replacing it; the worker reproduction covered the seven-second slow-exit race. This settles the reported five-second duplicate-refusal case. Force-killing permanently wedged instances and retrying other launch failures are not claimed fixed.

Scope checked by GPT-6 in Codex and Codex Medium against the discussion and landed change; review independence is not guaranteed. No tests were rerun for this cleanup.

Omabot on behalf of DHH

@omarchybot omarchybot closed this Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

3 participants