Skip to content

feat(stop-failure): auto-heal poisoned-transcript crash loops - #613

Open
olegbrok wants to merge 1 commit into
mainfrom
fix/poisoned-transcript-auto-heal
Open

feat(stop-failure): auto-heal poisoned-transcript crash loops#613
olegbrok wants to merge 1 commit into
mainfrom
fix/poisoned-transcript-auto-heal

Conversation

@olegbrok

Copy link
Copy Markdown
Collaborator

Summary

Detects and self-heals the crash loop that wedged Barsik today (force-restarted by hand at the time). When extended thinking is on, a cancelled sibling in a parallel tool batch makes Claude Code reconstruct the latest assistant message, breaking the crypto signatures on its thinking blocks. Anthropic then rejects every resend with a 400 (thinking blocks ... cannot be modified). The poisoned message is baked into the transcript, so claude --continue resumes it and the agent crash-loops forever until a human intervenes.

This is upstream Claude Code behavior — we can't prevent the corruption, but we can detect the wedge and escape it.

How it works (all at the StopFailure endpoint)

  • Classifyclassify_stop_failure() upgrades the coarse unknown/invalid_request type CC reports to poisoned_transcript by matching the error-message signature.
  • Detect (PoisonedTranscriptTracker, belt + suspenders):
    1. Signature — 2 thinking-block hits on the same session_id.
    2. Tight-loop fallback — 4 rapid any-class failures on one session_id (catches future poison variants the string match misses). A time window keeps sporadic rate_limits on a healthy long-lived session from false-triggering.
    • Rate-limited to ≤1 heal per agent per 10 min (no reset storms).
  • Heal — set force_fresh_context_once + force_restart(bypass_guard=True) so the agent respawns onto a fresh transcript, abandoning the poison. Runs after the feat: use structured stop_details in message responses for richer session state #108 turn-resolve so the wedged caller is unblocked first.
  • Two-phase commit — a failed respawn doesn't burn the cooldown, so the next poison retries instead of being silenced.
  • Operator DM — self-heal is never silent; Brad gets an informational alert (reuses the feat: Heartbeat pulse — send_heartbeat() MCP tool + animated pulse UI #104/feat(#104): alert operator on billing/sustained-rate-limit StopFailures #599 operator-chat resolution).

Reuses the existing StopFailure → classify → tracker infra (#104/#599). poisoned_transcript is intentionally not in TRANSPORT_FAILURE_POLICIES — those classes need a human; this one is auto-fixed.

Relationship to #612

Follow-up to the tmux --continue liveness fix (#612): that protects the fresh boot this heal triggers.

Test plan

  • classify_stop_failure — real production 400, must remain variant, pass-through for auth/rate_limit, "thinking" word alone doesn't trip, empty defaults
  • PoisonedTranscriptTracker — signature threshold, session-change streak reset, loop threshold, loop-window break, no-session_id, cooldown, commit-reset clears streak+anchor, per-agent isolation
  • Endpoint — classifies poison, second hit triggers heal (asserts force_restart(bypass_guard=True) + force_fresh_context_once), non-poison never heals, non-tmux session duck-outs, failed restart doesn't burn cooldown
  • pytest tests/test_auth_alerts.py tests/test_stop_failure_hook.py tests/test_api.py → 386 passed
  • ruff check clean

🤖 Generated with Claude Code

When extended thinking is on, a cancelled sibling in a parallel tool batch
makes Claude Code reconstruct the latest assistant message, breaking the
crypto signatures on its thinking blocks. Anthropic then rejects every
resend with a 400 ("thinking blocks ... cannot be modified"). The poisoned
message is baked into the transcript, so `claude --continue` resumes it and
the agent crash-loops forever until a human force-restarts it.

Detect and self-heal at the StopFailure endpoint:
- classify_stop_failure() upgrades the coarse unknown/invalid_request type
  to `poisoned_transcript` from the error-message signature.
- PoisonedTranscriptTracker fires on either 2 signature hits on one
  session_id, or 4 rapid any-class failures on one session (tight-loop
  fallback for future variants; the time window keeps sporadic rate_limits
  on a healthy long session from false-triggering). Rate-limited to one
  heal per agent per 10 min.
- On trigger, set force_fresh_context_once + force_restart(bypass_guard) so
  the agent respawns onto a fresh transcript, abandoning the poison. Two-
  phase commit: a failed respawn doesn't burn the cooldown. Operator gets
  an informational DM (self-heal is never silent).

Follow-up to the tmux --continue liveness fix: that protects the fresh
boot this heal triggers.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

bradbrok commented Jun 1, 2026

Copy link
Copy Markdown
Owner

Heartbeat maintenance (2026-06-01): rebased onto current main (ec39201 — 20 commits since PR was opened on 52b00ac, including isolation_mode/provisioning seam, per-agent signing keys, shared-SSE key binding, CommandRunner/UnixUserProvisioner, live token analytics, and ultracode effort tier).

Rebased cleanly — no conflicts in api.py, auth_alerts.py, or either test file. Force-push to fix/poisoned-transcript-auto-heal not possible from this web session (proxy returns 403 on force-push). Rebase is ready locally.

Tests: test_stop_failure_hook.py — 27 passed, test_auth_alerts.py — 46 passed. Branch is CI-green and mergeable_state=clean so GitHub can auto-merge without a rebase if approved now.

🤖 Heartbeat maintenance


Generated by Claude Code

bradbrok commented Jun 4, 2026

Copy link
Copy Markdown
Owner

Heartbeat maintenance (2026-06-04): main has moved 4 commits since last heartbeat on ec39201 — PRs #671#674 (security: API auth hardening, path-traversal containment, SQLite owner-only perms, self-update target verification). None touch api.py's StopFailure/poison-transcript paths or the auth-alerts/stop-failure test files, so mergeable_state remains clean.

Tests: test_stop_failure_hook.py — 22 passed, test_auth_alerts.py — 15 passed (local suite on current main).

Force-push from this environment is blocked (proxy 403), so no rebase commit here.

🤖 Heartbeat maintenance


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants