Bare-pod backend: pause_by_id deletes pod, resume_by_id is a no-op — suspended sessions are unrecoverable

[j-s]

Problem

SandboxBackend.pause_by_id() defaults to calling stop_by_id(), which deletes the pod entirely. resume_by_id() is a no-op (returns None). The KubernetesExecutorBackend inherits both defaults without overriding either.

Meanwhile, get_or_spawn() in agent.py:846-873 treats state='suspended' as resumable: it calls resume_by_id() (no-op), checks status (pod is gone), and raises RuntimeError("failed to resume suspended sandbox: ...") — surfaced to the end user as "Failed to start the agent runtime."

Only the CRD-based KubernetesAgentSandboxBackend actually supports pause/resume (replicas 0→1), but there is no guard preventing the suspend codepath from running on backends that don't support it.

Impact

Every deployment using the default bare-pod backend accumulates suspended sessions in the DB (we had 2,206) whose pods are already deleted. Any user message to one of those threads permanently errors until the DB row is manually cleaned up.

Suggested fix

Either:

1. Have get_or_spawn() fall through to spawn a new sandbox when resume fails (instead of raising), clearing agent_thread_id since the session state died with the pod
2. Or have KubernetesExecutorBackend override pause_by_id to set DB state to gone instead of suspended, so resume is never attempted

Option 1 is more defensive and handles edge cases in the CRD backend too. We're running it as a hotfix now.

Relevant code

• services/api/api/sandbox/base.py:135-141 — base class defaults
• services/api/api/agent.py:846-873 — resume logic that assumes pod exists
• services/api/api/sandbox/kubernetes.py — bare-pod backend, no pause/resume override

[zac-idiliq]

Third independent production confirmation (plain kubernetes backend, chart v0.1.49, codex harness). We hit this 2026-06-09 with 11 stuck suspended sessions — every previously-working Slack thread errored failed to resume suspended sandbox on its next mention while new threads worked fine. Diagnosis matched this issue exactly (base.py:135-141 defaults inherited un-overridden, agent.py ~846-873 re-raise).

Two additions to what's written above:

1. The trigger isn't only idle TTL. Sessions also get suspended at any age by idle_capacity_eviction when active sessions exceed MAX_ACTIVE_SANDBOX_SESSIONS — so on the bare-pod backend this bites in completely regular use under mild load, not just after a 24h lull. (Footnote: SUSPENDED_RETENTION_S = 7d means each trapped thread "self-heals" after a week when the row expires, which can make the bug look intermittent.)

2. For cleaning up stuck rows, there's a remedy that preserves agent context. Rather than deleting the rows (loses agent_thread_id), re-label them:

UPDATE sandbox_sessions SET state = 'error', updated_at = NOW() WHERE state = 'suspended';

'error' is in the reusable-states set, so the next message routes the session down the existing "container is gone → preserve agent_thread_id → respawn" path a few lines below the raise — the thread comes back with its context intact. We run this as a 5-minute CronJob as a stopgap and it has fully neutralized the issue in production; happy to share the manifest if useful.

On the real fix: option 1 above is the right shape, with one refinement — preserve agent_thread_id on the fall-through rather than clearing it (the suspend path destroyed the pod, not the harness thread state, which lives with the session row). #470 (closed here, lives in the Tavus fork) implements exactly that and independently reports the same bug from a third deployment.