test(queue): verify worker first-heartbeat < STUCK_CUTOFF + live run-queue smoke

[rdfitted]

Follow-up from #126 (PR #134). The durable run queue uses STUCK_CUTOFF_MS = 90s: a running row whose heartbeat_at is older than the cutoff (or NULL) becomes reclaimable by reclaim_stuck / try_claim.

• Verify a freshly-spawned worker emits its first heartbeat within 90s of spawn, so reclaim_stuck never requeues a healthy-but-quiet worker. Heartbeats are designed to be independent of git output, but project DNA notes some CLIs index 8–15 min before producing a diff — confirm the heartbeat path really is independent and lands well inside 90s. Tune STUCK_CUTOFF_MS / heartbeat cadence if not.
• Run the live queue E2E smoke (needs the Tauri shell, so it couldn't run headlessly): launch a real Hive session → spawn a worker → confirm GET /api/sessions/{id}/queue shows running → (heartbeat) → finalized accurately → kill the app mid-run → resume → confirm the row is reclaimable and the dashboard reflects it.

Acceptance: measured first-heartbeat latency is comfortably under the cutoff (or the cutoff is adjusted); the live smoke passes end-to-end.

[rdfitted]

Partially resolved by PR #169. Leaving this open — the remaining work is a decision, not an implementation.

Shipped

Cadence now reaches every CLI behavior. heartbeat_line was interpolated only in the ExplicitPolling arm, so ActionProne, InstructionFollowing and Interactive — covering 6 of the 8 allowlisted CLIs — received no heartbeat instruction at all. Those workers were permanently one sweep from being reclaimed while healthy.

Real margin against the cutoff. Templates instructed "every 60-90s" against STUCK_CUTOFF_MS = 90_000, so a worker obeying the upper bound of its own instruction sat exactly at the cutoff. Cadence is now derived from the cutoff via HEARTBEAT_SAFETY_FACTOR, yielding 20-40s — a worker can miss a full beat and stay healthy. Derived in one place, substituted from the constant rather than context.variables, so a caller who forgets it cannot ship a cadence-free prompt.

A critical collision, found in review. The tighter beat collided with MAX_NO_PROGRESS_CONTINUATIONS = 5: record_heartbeat increments no_progress_count when the incoming status matches the last, and prompts emit a fixed status per phase, so six beats at the 20s floor finalized a healthy worker in 100 seconds — terminally, since record_heartbeat then matches zero rows and reclaim_stuck can never recover it. The fix for this issue would have disabled the recovery path this issue exists to protect. Now derived from a wall-clock window instead of a raw beat count.

#155 needed no change — the completed-heartbeat instruction sits outside the CliBehavior match, so PR #156 genuinely covered all behaviors. A guard assertion was added so gating it later fails.

Still open — bullet 1's real question

The issue assumes first-heartbeat latency is independent of agent startup. It isn't: a cold codex or agy index runs 8-15 minutes before producing output, and no prompt cadence can beat that. Three options, and the third is the one worth deciding:

1. Raise STUCK_CUTOFF_MS — delays genuine stall detection to accommodate a prompt. Fixing the wrong side.
2. Tighten the instructed cadence further — burns agent turns on curl and still cannot beat a cold index.
3. Derive the heartbeat from PTY output activity in the backend — makes liveness independent of the agent self-reporting, which is what this issue's author assumed it already was. This is a design change, and it would obsolete most of the prompt surgery above.

Option 3 changes the reliability model, so it needs an explicit call rather than a default.

Still open — bullet 2 is unshippable as written

"The dashboard reflects it" cannot pass: queueStore (agents.ts:114) has zero Svelte consumers — the only two references in src/ are its own definition and its own listener. There is no dashboard for it to reflect in. Either drop the bullet or scope a QueuePanel component as separate work.

Still open — the live E2E

Launch session → spawn worker → watch running/heartbeat/finalized → kill app → resume → confirm reclaim. Needs the Tauri shell and real CLI spend, and there is no e2e harness in this repo (package.json has dev/build/check/test only, no playwright). This is a human checklist item, and it is the only way to confirm real CLI startup fits inside the first-heartbeat window.

[rdfitted]

Measured on v0.45.0 during a live smoke run — the answer to the first checkbox is no, and there is a worse defect behind it. Filed as #251.

A codex worker was reclaimed 105s after spawn (cutoff is 90s) while its process was alive with 142s of CPU. The reclaim then rewrote worker_id to the pending: sentinel, which severs the finalize path permanently — the worker finished its work and its completed heartbeat became a silent no-op. Row ended at queued/finalized: 0.

Leaving this issue open for the per-CLI first-heartbeat measurement and the live E2E smoke; #251 carries the defect itself.