feat(stop-failure): auto-heal poisoned-transcript crash loops - #613
feat(stop-failure): auto-heal poisoned-transcript crash loops#613olegbrok wants to merge 1 commit into
Conversation
When extended thinking is on, a cancelled sibling in a parallel tool batch
makes Claude Code reconstruct the latest assistant message, breaking the
crypto signatures on its thinking blocks. Anthropic then rejects every
resend with a 400 ("thinking blocks ... cannot be modified"). The poisoned
message is baked into the transcript, so `claude --continue` resumes it and
the agent crash-loops forever until a human force-restarts it.
Detect and self-heal at the StopFailure endpoint:
- classify_stop_failure() upgrades the coarse unknown/invalid_request type
to `poisoned_transcript` from the error-message signature.
- PoisonedTranscriptTracker fires on either 2 signature hits on one
session_id, or 4 rapid any-class failures on one session (tight-loop
fallback for future variants; the time window keeps sporadic rate_limits
on a healthy long session from false-triggering). Rate-limited to one
heal per agent per 10 min.
- On trigger, set force_fresh_context_once + force_restart(bypass_guard) so
the agent respawns onto a fresh transcript, abandoning the poison. Two-
phase commit: a failed respawn doesn't burn the cooldown. Operator gets
an informational DM (self-heal is never silent).
Follow-up to the tmux --continue liveness fix: that protects the fresh
boot this heal triggers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Heartbeat maintenance (2026-06-01): rebased onto current Rebased cleanly — no conflicts in Tests: 🤖 Heartbeat maintenance Generated by Claude Code |
|
Heartbeat maintenance (2026-06-04): Tests: Force-push from this environment is blocked (proxy 403), so no rebase commit here. 🤖 Heartbeat maintenance Generated by Claude Code |
Summary
Detects and self-heals the crash loop that wedged Barsik today (force-restarted by hand at the time). When extended thinking is on, a cancelled sibling in a parallel tool batch makes Claude Code reconstruct the latest assistant message, breaking the crypto signatures on its
thinkingblocks. Anthropic then rejects every resend with a 400 (thinking blocks ... cannot be modified). The poisoned message is baked into the transcript, soclaude --continueresumes it and the agent crash-loops forever until a human intervenes.This is upstream Claude Code behavior — we can't prevent the corruption, but we can detect the wedge and escape it.
How it works (all at the StopFailure endpoint)
classify_stop_failure()upgrades the coarseunknown/invalid_requesttype CC reports topoisoned_transcriptby matching the error-message signature.PoisonedTranscriptTracker, belt + suspenders):session_id.session_id(catches future poison variants the string match misses). A time window keeps sporadicrate_limits on a healthy long-lived session from false-triggering.force_fresh_context_once+force_restart(bypass_guard=True)so the agent respawns onto a fresh transcript, abandoning the poison. Runs after the feat: use structured stop_details in message responses for richer session state #108 turn-resolve so the wedged caller is unblocked first.Reuses the existing StopFailure → classify → tracker infra (#104/#599).
poisoned_transcriptis intentionally not inTRANSPORT_FAILURE_POLICIES— those classes need a human; this one is auto-fixed.Relationship to #612
Follow-up to the tmux
--continueliveness fix (#612): that protects the fresh boot this heal triggers.Test plan
classify_stop_failure— real production 400,must remainvariant, pass-through for auth/rate_limit, "thinking" word alone doesn't trip, empty defaultsPoisonedTranscriptTracker— signature threshold, session-change streak reset, loop threshold, loop-window break, no-session_id, cooldown, commit-reset clears streak+anchor, per-agent isolationforce_restart(bypass_guard=True)+force_fresh_context_once), non-poison never heals, non-tmux session duck-outs, failed restart doesn't burn cooldownpytest tests/test_auth_alerts.py tests/test_stop_failure_hook.py tests/test_api.py→ 386 passedruff checkclean🤖 Generated with Claude Code