Skip to content

fix(embed-lock): evict a live holder that has outlived its session cap - #944

Open
carronch wants to merge 1 commit into
tobi:mainfrom
carronch:fix/embed-lock-evict-overdue-holder
Open

carronch wants to merge 1 commit into
tobi:mainfrom
carronch:fix/embed-lock-evict-overdue-holder

Conversation

@carronch

@carronch carronch commented Sep 7, 2026

Copy link
Copy Markdown

Problem

A PID that still answers is not proof of a live embed.

The embed session cap (--timeout, default 30 min, DEFAULT_EMBED_MAX_DURATION_MS) is a setTimeout that aborts an AbortController (LLMSession in src/llm.ts). It cannot fire while the event loop is blocked inside a synchronous native node-llama-cpp call. So a process wedged in Metal/CUDA keeps its PID, keeps .qmd-embed.lock, and every later qmd embed prints Another embed process is already running. Skipping. for as long as the zombie lives.

Observed on an Apple M5 Pro (macOS 26.5.2, qmd 2.8.3, node-llama-cpp 3.20.0, Bun 1.3.11): a nightly qmd embed faulted inside AddonContext::GetEmbedding, the runtime turned the fault into a signal-handler spin, and the process sat at 100% CPU for 39 hours. The two following nightly embeds skipped themselves. qmd status gave no hint that the lock was the reason. Same hardware class and error signature as #735.

Fix

The lockfile now records {"pid", "startedAt", "maxDurationMs"} instead of a bare PID.

On EEXIST, a live holder is still respected, unless it has held the lock longer than twice the cap it declared (24 h if it declared --timeout 0). Then it is evicted, the new handle carries reclaimedFrom, and the CLI prints:

Reclaimed the embed lock from process 11128: it has held it for 2340 min, past its 60 min limit. If that process is still running it is stuck in a native call; kill it.

Twice the cap, so a holder that is merely slow to notice its own abort is not evicted while winding down.

Bare-PID lockfiles written by earlier versions still parse: age comes from the file's mtime, cap from the caller's own --timeout. Malformed content is treated as stale, as before.

Changes

  • src/cli/embed-lock.ts: JSON lock record, readEmbedLockRecord, isOverdueEmbedLockHolder, embedLockMaxAgeMs, embedLockReclaimedMessage; tryAcquireEmbedLock(lockPath, { maxDurationMs, now }). release() still only unlinks its own lock.
  • src/cli/qmd.ts: passes the run's cap into the lock; warns on eviction.
  • test/embed-lock.test.ts: 7 new cases (fresh live holder still blocks; overdue live holder evicted and reported; uncapped ceiling; legacy bare-PID lock aged by mtime; malformed content). Existing cases adapted to the JSON record. The cross-process holder test is unchanged and still passes.
  • CHANGELOG.md: entry under [Unreleased].

Verified: tsc --noEmit clean, oxlint clean, vitest run test/ 1150 passed, bun test test/ 1237 passed (Bun 1.4.2, macOS 26.5.2).

Notes

🤖 Generated with Claude Code

https://claude.ai/code/session_01UbPmKhuD7KxgFBH93Us8xN

A PID that still answers is not proof of a live embed. The embed session
cap (--timeout, default 30 min) is a JS timer: it cannot fire while the
event loop is blocked inside a native node-llama-cpp call, so a process
wedged in Metal/CUDA keeps its PID, keeps .qmd-embed.lock, and every later
`qmd embed` prints "Another embed process is already running. Skipping."
for as long as the zombie lives. Observed on an Apple M5 Pro: a run that
faulted in GetEmbedding spun for 39 hours and blocked two nightly embeds.

The lockfile now records {pid, startedAt, maxDurationMs}. On EEXIST a live
holder is still respected, unless it has held the lock longer than twice
the cap it declared (24 h if it declared none); then it is evicted, the
new holder's handle carries `reclaimedFrom`, and the CLI warns with the
holder's PID and age so the user can kill it. Bare-PID lockfiles from
earlier versions still parse, aged by the file's mtime and capped by the
caller's own --timeout.

Refs tobi#735 (same M5 deadlock, lock held by a live process). Complementary
to tobi#924, which hardens holder identity but not holder age.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UbPmKhuD7KxgFBH93Us8xN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant