Skip to content

Preserve tool evidence and safely recover interrupted task completion - #1518

Merged
obsessixnv merged 2 commits into
mainfrom
fix/scheduled-evidence-recovery
Sep 16, 2026
Merged

obsessixnv merged 2 commits into
mainfrom
fix/scheduled-evidence-recovery

Conversation

@obsessixnv

@obsessixnv obsessixnv commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

What changed, and why

Scheduled runs could lose completion evidence because their durable tool transcript contained the 4,000-byte UI preview. They could also report success after ignoring verifier findings, or dead-letter after a provider failure even though a preceding tool step had completed successfully.

Record complete redacted tool results before display truncation and project enclosing status/version fields before deep metric profiles. Recheck verifier repairs, with at most three reviews; unresolved findings and malformed or unavailable verification prevent success.

Resume retryable provider failures from a completed step's input, retaining tool results and usage instead of restarting the original tool sequence. Recovery remains suppressed if a tool started in the failed step or an earlier tool result indicated failure. Preserve iteration limits, remove provider-specific reasoning from replay, recover structured SSE status codes, and retain bounded redacted diagnostics. Authentication and validation errors stay terminal.

How you verified it

  • Regression tests pass for complete evidence through the core/scheduled/verifier boundary, bounded repair reviews, malformed verdicts, completed-write fallback without duplicate execution, reasoning removal, preserved iteration limits, provider status classification, and redaction.
  • Release compilation, tagged vet, lint, the full Go suite, and the full race suite pass against isolated local PostgreSQL databases. A verbose database smoke test confirms PASS rather than SKIP. make govulncheck reports no reachable vulnerabilities.
  • make ci-web passes under Node 24: dependency audit, lint, typecheck, 1,639 tests across 153 files, and production build. Built-in guides and web copies are synchronized.
  • The live-browser CI failure was traced to the fake model replaying its worker script for the independent verifier. A new regression first reproduced invalid verdicts for all five scheduled/A2A completion fixtures; it now passes after adding opt-in non-streaming fixture responses. Both fake-model packages pass with -race; release compilation and the full lint gate pass on the fixture patch.
  • All CI checks pass on 51c930ba, including the live scheduled-task browser journey, mocked browser suite, full Go race lane, CodeQL, Semgrep, and container vulnerability scan.
  • Existing contained-panic coverage continues to require suppression of uncertain tool replay.

Scope and deviations

The runtime remains general purpose. Source-scope and historical-correction rules live in the companion Pages and bundle changes: ElcanoTek/pages#96 and ElcanoTek/elcano-config#145. No customer schedules, source data, page schema, connector permissions, or network policy are changed.

ADR-0065 narrows ADR-0035 only at a completed-step checkpoint. Verification now fails closed when configured; this can make an unavailable verifier fail a run that previously appeared successful.


  • Title and behavior description are written for release notes
  • Design note and operator guides updated
  • ADR added for the completed-step recovery invariant
  • Diff is scoped to reliable task completion and recovery

@obsessixnv
obsessixnv marked this pull request as ready for review September 16, 2026 12:12
@obsessixnv
obsessixnv merged commit 02c9dbd into main Sep 16, 2026
21 checks passed
@obsessixnv
obsessixnv deleted the fix/scheduled-evidence-recovery branch September 16, 2026 12:39
obsessixnv added a commit that referenced this pull request Sep 17, 2026
…exceeds a token budget (#1561)

Closes #1534.

## Problem

Compaction fired only on context-**window** pressure (90%), and
scheduled runs only warned unless `FLEET_SCHEDULED_AUTO_COMPACT=1`. A
1M-window model never reaches 90% in a 100-turn run, yet every turn
resends the whole transcript and pays for it: Reklaim `6bd0c212` resent
~115K tokens per call for 99 calls at an 11% cache-hit rate — 11.35M
prompt tokens, $8.29 — and no compaction ever ran.

## Change

- **Resend-budget trigger** (`checkContextPressure`, scheduled engines
only — `requireCompactionOptIn` is the driver's scheduled marker): once
`LastStepPromptTokens` (the anti-spiral signal the window path already
uses) exceeds `FLEET_CONTEXT_RESEND_BUDGET_TOKENS` (default **80 000**;
`0` disables), the same oldest-half `proactiveCompact` runs through the
same governed summarizer. It is a cost control, not a safety valve, so
it is deliberately **not** behind `FLEET_SCHEDULED_AUTO_COMPACT`, which
keeps guarding the window-pressure path. Interactive chats are
untouched.
- **Observability**: `fleet.context_compacted` now carries `trigger`
(`resend_budget` | `window`) and, for the budget path, `used_tokens` +
`resend_budget_tokens`; the session log gets a `[context_compacted]
trigger=resend_budget used=… budget=…` breadcrumb.
- **Knob**: registered in `knobs.go` (scopeExternal) and the env-file
allowlist; `AGENT-RUNTIME.md` compaction table + a paragraph. The
per-result lever the issue also asked for already exists
(`max_tool_output_bytes`, default 64 KiB); the docs point at it.

## Contract change, on purpose

`TestRun_ContextPressure_ScheduledWarnsOnlyWithoutFlag` ("a scheduled
run must not silently rewrite its transcript without the opt-in") now
disables the budget trigger explicitly: that test pins the
window-pressure contract, while the cost trigger is default-on for
scheduled runs — with #1518's completed-step checkpoints making
re-drives safe, and the breadcrumb + event making it not silent.
Expected effect on the Reklaim job: ~50K resent tokens per turn instead
of ~115K.

Tests: `TestRun_ResendBudget_ScheduledCompactsWithoutOptIn` (run-loop,
event payload + breadcrumb),
`TestCheckContextPressure_ResendBudgetIsScheduledOnly` (interactive
ignores; scheduled compacts; 0 disables),
`TestCheckContextPressure_WindowCompactionNamesTrigger`,
`TestContextResendBudgetTokensKnob`; existing context/compaction suites
green; vet + golangci-lint clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant