Sandbox agent delivery fixes: web_fetch empty-body bug + ephemeral key refresh for long runs

[sweetmantech]

Tracking issue for two sandbox-agent reliability fixes surfaced by the 2026-07-02/03 scheduled-task email audits (follow-ups to chat#1829 delivery and chat#1833 hallucination, both resolved). Both defects silently kill customer deliveries: the agent does the work, the customer gets nothing. Interim mitigations (prompt patches steering agents to bash curl, WMG credit top-up) are live in scheduled_actions; this issue tracks the real fixes in api.

Goal

1. web_fetch (the sandbox agent's HTTP tool, api/lib/agent/tools/webFetchTool.ts) returns the actual response body inside the Vercel sandbox, and a zero-byte write failure surfaces as success: false — never as an empty-body success. Agents stop concluding "every data source is blocked" when the tool is what's broken.
2. Agent runs longer than the 15-minute ephemeral key TTL (DEFAULT_EPHEMERAL_KEY_TTL_MS, api/lib/keys/mintEphemeralAccountKey.ts L7) keep a valid $RECOUP_API_KEY for their entire lifetime, so end-of-run sends to POST /api/emails stop dying with 401s logged as rejected in email_send_log.

PRs (updated 2026-07-02)

PR	Item	Base	State
api#TBD	web_fetch: replace process-substitution body capture with temp file; treat exit-23 + empty body as failure; unit tests	test	⏳ not opened yet
api#TBD	ephemeral key refresh for long runs: re-mint + re-inject $RECOUP_API_KEY before TTL expiry inside runAgentWorkflow	test	⏳ not opened yet

Merge sequencing: the two PRs are independent — no ordering constraint. Both target test per api repo rules. No docs/database changes required (no contract or schema change).

Evidence (context)

• web_fetch, 2026-07-02 prod traces (chat_messages, chats 8064ee8f-a8fb-4ed5-8d99-7ff7c0f6c94f and 1bf6b172-1067-4390-82fe-8f5252b57045): every web_fetch call returned {"body": "", "status": 200, "success": true, "truncated": true} — 12/12 URLs in the WMG run including sanity probes to example.com (twice) and a raw GitHub README. In the same run, bash curl to docs.recoupable.dev returned real content, so sandbox networking is fine — the tool's write path is what fails. The identical command construction works locally (full body + status, exit 0), so the failure is sandbox-runtime-specific.
• Mechanism: webFetchTool.ts L66 writes the body via bash process substitution (-o >(head -c 10000 >&3) with fd-3 juggling). curl exiting 23 ("write error") with zero bytes matches a failed open/write of the /dev/fd/N pipe inside the sandbox's runCommand context. L100/L117 then map exit 23 to success: true, truncated: true without checking the body is non-empty — a hard failure becomes an empty 200.
• Key expiry, 2026-07-02 09:06 run (chat 9078748a-9309-491c-a98e-0107a9fee0e8): agent researched and built a full 19.5kb report for a real customer, then every POST /api/emails attempt 401'd (key past expires_at, enforced in api/lib/auth/getAccountIdByApiKey.ts L29-30) and was logged rejected. The report was lost. One long run can emit dozens of rejected rows this way, polluting the guard-block metric in the task-email audit.
• Cost of the pair on 2026-07-02: 3 of ~20 real customer email tasks delivered nothing for tool-gap reasons; 1 more lost a finished report to key expiry.

Open — fixes (independent, pick either first)

• web_fetch: capture the body via temp file and fail loudly on zero-byte writes. (api#TBD)

  • Why: process substitution -o >(head -c 10000 >&3) (webFetchTool.ts L58-91) fails inside the Vercel sandbox — curl exit 23 with 0 bytes — and L100/L117 misreport that as success: true, truncated: true. Agents get empty bodies with 200s for every URL and conclude the internet is blocked.
  • Fix: (1) rewrite the command as curl -sS -o "$tmp" -w '%{http_code}' <url>; head -c 10000 "$tmp"; rm -f "$tmp" — no process substitution, no fd juggling; (2) truncated = body file size > 10,000 bytes, not exit code; (3) exit 23 (or any nonzero) with an empty body returns success: false with the stderr text; (4) unit tests: normal body, >10KB truncation, zero-byte write error; (5) verify inside a real sandbox run (the local repro passes today, so local green is not sufficient proof).
  • Done when: a preview-deployed agent run calls web_fetch on example.com and receives the actual HTML body; a forced write-failure case returns success: false; the WMG/Fat Beats-style trend research works through web_fetch in a sandbox without falling back to curl.

• Ephemeral key refresh for runs that outlive the 15-minute TTL. (api#TBD)

  • Why: provisionRunSession mints one recoup_sk_ key with DEFAULT_EPHEMERAL_KEY_TTL_MS = 15 min (mintEphemeralAccountKey.ts L7) and injects it as $RECOUP_API_KEY; nothing refreshes it. Runs >15 min lose auth mid-flight — typically at the final send step, the most expensive place to fail.
  • Fix (proposed): in runAgentWorkflow (api/app/lib/workflows/runAgentWorkflow.ts), track the key's minted-at time; before any step that would execute with <2 min of TTL left, re-mint via the existing mintEphemeralAccountKey(accountId), update the sandbox env (buildRecoupExecEnv path), and revoke the old key. Alternative considered: raising the TTL — rejected as default (weakens the chat#1813 defense-in-depth); acceptable fallback is minting with ttlMs scaled to the run's configured max duration.
  • Done when: a deliberately long (>20 min) preview run sends its email successfully at the end; email_send_log shows sent (not rejected) for that run; expired-key rejected rows with null account_id stop appearing for runs that completed normally.

Architecture decisions

• Fix the tool and the workflow, not the prompts. The 2026-07-02 prompt patches (curl fallbacks in scheduled_actions for the Fat Beats and WMG tasks) are mitigations that prove the workaround, not the fix — every other task in the fleet still depends on web_fetch. The platform fix restores the whole fleet without per-task prompt surgery.
• Keep the 15-minute TTL as the default; refresh instead of extending. The short TTL is deliberate (chat#1813: the long-lived service key never enters the sandbox; ephemeral keys limit blast radius). Refreshing preserves that property for long runs.

Source references

• Audit skill that surfaced both: recoupable/skills → skills/recoup-internal-task-email-audit (round-2 learnings, skills#73, document the 401-rejected signature).
• Key files: api/lib/agent/tools/webFetchTool.ts, api/lib/keys/mintEphemeralAccountKey.ts, api/lib/auth/getAccountIdByApiKey.ts, api/lib/chat/runs/provisionRunSession.ts, api/app/lib/workflows/runAgentWorkflow.ts.
• Related, out of scope here: Perplexity quota exhaustion currently 401s POST /api/research/web / /api/research/deep (billing action, not a code change); deleteTask orphan-schedule minting (api/lib/tasks/deleteTask.ts L30-35, parallel Promise.all deletion) — separate cleanup, six existing orphans already deleted 2026-07-02.