K8s run zombie after stdin_hold_timeout kill: agent dead 40+ min, run s…

[warren-run-bot]

Summary

seeds: sync 2026-08-27

Agent notes

The in-pod agent-entrypoint idle watchdog kills a stdin-held harness on
stdin_hold_timeout, but the pod (entrypoint finalize poller) keeps reading
Running, so the run row stayed 'running' with false liveness for the full
40-min finalize ceiling and the emptyDir work was lost with no salvage.

• New canonical wire failure reason 'agent_died' (src/core/wire.ts),
  surfaced in the UI label map, OpenAPI (regenerated), and the judge
  extension's hand-maintained wire mirror.
• inferFailureReason classifies the stdin_hold_timeout system witness as
  agent_died instead of the anonymous 'crashed'.
• The watchdog terminal-reconcile net reaps a still-live pod whose event
  log carries the witness (stream=system only, via a stream-narrowed
  hasKind probe): the reap parks the finalize intent the in-pod finalize
  loop is polling for, so workspace collection + salvage fire while the
  emptyDir still exists.
• General infra-death salvage stays warren-6c94's scope.

Tests: state.agent-died.test.ts, watchdog-reconcile.test.ts. check:all 12/12.

Run

• Warren run: run_qa0e9xax8c8n
• Agent: pi
• Cost: $2.11 (82.2k in / 27.5k out / 7.2M cache-r)

Seeds

• warren-7f0b — K8s run zombie after stdin_hold_timeout kill: agent dead 40+ min, run stayed running, no salvage

Commits (3)

• c0b9a55 seeds: sync 2026-08-27
• e557d85 mulch: update expertise
• f89f839 fix(k8s): reap watchdog-killed zombie runs as failed(agent_died) and drive salvage (warren-7f0b)

Files changed

docs/openapi.yaml                      |  2 +
 extensions/judge/src/warren-wire.ts    |  4 +-
 src/core/wire.ts                       | 62 ++++++++++++++---------------
 src/db/repos/events.ts                 | 18 +++++++--
 src/runs/reap/state.agent-died.test.ts | 65 +++++++++++++++++++++++++++++++
 src/runs/reap/state.ts                 | 33 ++++++++++++++++
 src/runs/watchdog-reconcile.test.ts    | 71 ++++++++++++++++++++++++++++++++++
 src/runs/watchdog-reconcile.ts         | 27 ++++++++++++-
 src/ui/src/lib/labels.ts               |  1 +
 9 files changed, 247 insertions(+), 36 deletions(-)

Prompt

Show prompt

Work seeds issue warren-7f0b. First run sd show warren-7f0b --json from the repo root — the issue body is the spec; this summary is only a digest.
Context of current main: PRs #1070-#1092 merged yesterday (Direction C operator console UI revamp, plan pl-7e38). Earlier today the tracker closed stale P1s warren-06b8, warren-4fe1, warren-9628, warren-df2c. Three other P1 runs are being dispatched in parallel with yours (warren-968d, warren-7f0b, warren-5a5c, warren-df5c minus your own), so expect .seeds/issues.jsonl churn on main while you work — do not touch seeds rows other than your own issue's.
Scope: on the K8s runtime, when the agent-entrypoint watchdog kills the harness (e.g. stdin_hold_timeout), the run can stay 'running' with false liveness for 40+ minutes and the work in the pod's emptyDir is lost. Make the watchdog kill drive the run to a terminal failure state (agent_died/stalled style failureReason from the canonical wire vocabulary in src/core/wire.ts) and trigger the salvage path before the emptyDir disappears. Touches src/runtime/k8s/ lifecycle and the reap/salvage path. Related open P2 warren-6c94 covers general infra-death salvage — keep your scope to the watchdog-kill zombie case, and reference warren-6c94 in comments/issue notes rather than absorbing it. Respect the runtime-id guard (scripts/check-runtime-ids.ts): per-runtime behavior belongs behind the adapter/provider seam.
Quality gates are terminal: bun run check:all must be green before you commit and report done. Run it from the repo root.
Close the issue with sd close warren-7f0b --reason "<one-line evidence-citing reason>", then commit everything (including the .seeds change).

---

🤖 Opened by warren run run_qa0e9xax8c8n