feat: isolate browser code execution, add mockable runtime tests, harden idempotency and prod deploy

[rgxdev]

Summary

Production-hardening pass across five areas:

• Skill/plugin isolation: browser_run_playwright_code (apps/server/src/tools-builtin/browser.ts) previously ran arbitrary user code via a raw new Function("page", ...) in-process — full process.env/require/Bun access, same threat model as pre-ADR-0007 custom_code. It now runs through a new runVmSandboxedCode primitive in plugin-sandbox.ts: a same-process node:vm context whose only global is page, with no require/process/Bun/ambient fetch reachable, a timeout, and a capped/structured error — the same guarantees runIsolatedCustomCode's subprocess gives custom_code, minus the process boundary (a live Playwright Page handle can't cross a process boundary the way custom_code's serializable ctx does; see the doc comment on runVmSandboxedCode for the exact trade-off).
• Agent runtime reliability tests: agent-runtime.ts now calls every model request through a single swappable streamChatImpl reference (defaults to the real streamChat), with test-only setters. agent-runtime.test.ts adds a scripted-model harness proving: planning → direct execution → completion, a real tool call persisted to the task-event timeline, approval pause → approve/reject (using a real file_delete tool, not a simulated approval), an autonomy-budget pause mid-run, transient retry → success, retry-exhausted → dead_letter, a non-transient failure with no retry, cancellation mid-model-call, and delegation depth-limit/cycle-prevention. None of these hit a real local/cloud model.
• Durable run state / idempotency: closed a TOCTOU race in resolveApprovalDecision with an atomic claimApprovalRequest CAS (a single conditional UPDATE ... WHERE status = 'pending', not check-then-write), so two concurrent approve/reject calls for the same approval can't both execute the underlying action. Added packages/core-agent-engine/src/run-transitions.ts (assertAgentRunTransitionAllowed/isTerminalAgentRunStatus) and wired it into approvals.ts and agent-runtime.ts's completion write so a stale/racing write can't revive an already-terminal run.
• Runs/Approvals UX: added tasks.retry (backed by an atomic claimTaskForRetry CAS, same pattern as the approval CAS) plus a "Retry" button and a dedicated failed-state callout on the task detail page. Fixed a real gap: the frontend's AgentRunStatus type was missing dead_letter entirely, so it had no badge color anywhere it's rendered (task detail, agent detail, the cross-cutting Runs board) — now added everywhere. (The pending-approval nav badge already existed; no change needed there.)
• Production hardening: non-root USER bun in both Dockerfiles (confirmed against the upstream oven/bun image source that this user exists) with the SQLite volume mount pre-chowned so it stays writable; HSTS + Permissions-Policy added to Caddyfile; and the secret-scan workflow was actually broken — gitleaks/gitleaks-action@v2 failed on every single run (12/12) with a missing-license error unrelated to any real finding, so it never completed a scan while continue-on-error: true silently kept it green. Replaced with a direct OSS gitleaks CLI download (no license gate), kept non-blocking until this fixed version's first real run is triaged.

Test plan

• bun install --frozen-lockfile
• bunx biome check . (clean on every file touched/added this session; pre-existing repo-wide formatting drift on main is unrelated and unchanged)
• bun run typecheck — 0 errors across all 9 packages
• bun run build — server + web build clean
• bun test — 342/342 pass (30 new: agent-runtime reliability, plugin-sandbox/browser isolation, run-transitions, task-retry CAS)
• docker compose -f docker-compose.pc.yml config / docker-compose.server.yml config — both parse cleanly
• Manually verified the Retry button end-to-end: started the real dev stack, seeded a failed/dead_letter task, drove the task detail page with a real browser session, clicked Retry, and confirmed via direct DB inspection that a new run was created and the atomic claim/re-execution flow behaved correctly (screenshots taken, not attached)

🤖 Generated with Claude Code

https://claude.ai/code/session_01CxqiVNBG8nbxwBQpWAJcZD

---

Generated by Claude Code