Skip to content

chore(release): release 20261012d (restart recovery, upstream sync through acd873d41c) - #257

Merged
dreampiler merged 30 commits into
mainfrom
kd/release-20261012d
Oct 12, 2026
Merged

dreampiler merged 30 commits into
mainfrom
kd/release-20261012d

Conversation

@dreampiler

Copy link
Copy Markdown
Owner

Release 20261012d: release c (0539e71517) plus the restart-recovery fixes below, PRs #245, #251, #252, #254 and #255, and fork main through 79b3967a98, which carries the upstream sync through acd873d41c (#256). upstream/main is an ancestor of this branch. Not for merge until the operator swaps production in a maintenance window.

Root causes fixed here

1. Dynamic sessions stuck after a restart (every recovery path refused) — 27f0f52831, 410c924eb2
A restart left a dynamic session with its route at action_required (agent died first, policy stop), an open recovery block for the interrupted turn, and an open turn row no runtime would settle (every refused message.add opens one through CreateMessage → getOrStartTurn, task/service/service_messages.go:294-301). On release b/c:

  • session.recover / automatic continue_from_history refused every non-active route as dynamic_route_pending (orchestrator/task_operations.go:3851-3854; the only exception was a stranded starting route, interrupted_turn_recovery.go:207-218).
  • session.route_action refused any open turn (orchestrator/service.go:1781-1799, "route actions require a settled turn").
  • After agent.cancel, the route action re-sent the last prompt (dynamic_launch.go manual relaunch → promptTask), which the open recovery block refused (task_operations.go:6959-6963) → route_action_launch_failed.
  • Retry on a FAILED session whose durable route stayed active was refused as a pending recovery by the engine (ResumePendingNow, dynamic/engine.go).
    Fix:
  • An explicit recovery action reclaims an action_required route on its persisted candidate when no runtime is live (new Engine.ReclaimActionRequired, action_required → starting, generation/status fenced); a failed recovery returns it to action_required.
  • A route action settles turn rows orphaned on a parked route (no live runtime, no resume/queued dispatch/steer in flight); a live runtime still refuses.
  • A manual route action on a session a restart-interrupted turn blocks hands its successor to continue_from_history (try_next: on the newly selected candidate; retry: on the current candidate) instead of re-sending the prompt whose outcome is unknown. Stale generation, live runtime and Office tasks keep the old answer.

2. Queued messages piling up on idle primaries — 5839b219ac, f5553f5107
Every drain trigger belongs to a live runtime or a new message; nothing drains a primary that was idle at shutdown, and a drain refused by a recovery block never started the continuation that releases it (event_handlers_workflow.go drain refusal points, task_operations.go identity drain). message.queue.drain reported that refusal as INTERNAL_ERROR.

  • The startup lifecycle sweep ends with one pass over sessions with queue rows or parked dispatches: a WAITING primary (or a FAILED primary carrying an interrupted-turn block) with Auto-run on continues from history, drains directly, or resumes through the queue-drain ensure (ceiling refusals are recorded for replay). Paused queues, Office, archived, non-primary, coordinator and parked-route sessions are skipped.
  • A drain refused by an interrupted-turn block starts the automatic continue_from_history (primary sessions with queued work only).
  • message.queue.drain returns session_recovery_required.

3. Message-woken sessions failing with "agent delivery checkpoint missing" — 10acddd52f
#241 added the continue_from_history fallback only to session.recover resume. The lazy resume a message or queued dispatch performs starts the agent asynchronously and only saw session failed: … (task_operations.go waitForSessionReady). The start-failure callback now records a refusal only continue_from_history settles (checkpoint missing, durable adoption blocked); the lazy-resume wait (and a synchronous refusal) report it as a recovery-required error, which the message and queue paths already answer with automatic continue_from_history on a new N+1 stream.

4. Office overview resurfacing old failures — ca5b76310d, d18a9da33f
failedSessionEvents used updated_at as the failure time and window; a step move rewrote updated_at on every session of the task because SetSessionPrimary cleared is_primary on all rows (task/repository/sqlite/session.go:5026). The overview now uses completed_at (loaded by the overview query); the primary clear touches only the current primary. The overview part also applies to upstream PR kdlbs#4045.

Ports of unreleased branch fixes (hand-ported onto release code)

  • 35b1a90b62 ← kd/upstream-restart-recovery 4b14826: SCHEDULING-orphan step entry runs detached (no longer holds the startup sweep).
  • 927bccc482 ← fix/idle-focus-resume-automatic-origin b82abcd (fix(orchestrator): admit focus-driven idle resume as an automatic launch kdlbs/kandev#4406 review follow-up): focus resume keeps its permissions through ceiling replay. Superseded by upstream's acd873d41c in the sync merge: the port's files now match upstream and its claim shortcut is replaced by upstream's admitClaimedIdleFocusResume.
  • 78921b969c ← fix/executor-retry-after-workspace-preparing 8fa1da7 (fix(executor): retry a session binding after the workspace election kdlbs/kandev#4415 review follow-up): the election loser no longer steals the primary flag.
    Everything else in kd/upstream-restart-recovery, kd/upstream-ceiling-replay, kd/upstream-environment-failure-recovery, kd/upstream-workflow-stage-moves, kd/agentctl-instance-create-deadline is a re-port of fork fixes already in release b.

Merged PRs

#245 office cross-workspace access (default off) · #251 delivery row retirement · #248 · #249 · #250 · #252 MCP task priority · #254 automation restart settlement · #255 review writer/dynamic reviewer · #256 upstream sync through acd873d41c.

Upstream sync (#256)

13 upstream commits: kdlbs#4433, kdlbs#4432, kdlbs#4420, desktop window size, kdlbs#4435, kdlbs#4434, kdlbs#4436, kdlbs#4408, kdlbs#4414, kdlbs#3898, kdlbs#4438, kdlbs#4437, acd873d41c. The fork originals of kdlbs#4406, kdlbs#4414, kdlbs#4420 and the desktop fix were already here; the merged files match upstream except for fork-only additions, so no duplicate fork copy remains. Release-side conflicts (a6f82570e9): manager_lock_order_test.go takes upstream's kdlbs#4414 test (a superset of release c's), ceiling_dispatch_admission.go/task_operations.go take upstream's focus admission, docs/public/operations.md keeps upstream's retention cadence with the release's session-state compaction notes, office/AGENTS.md keeps both rows. The fork-main conflicts and decisions are listed in #256.
Deferred to a later release by the operator: kdlbs#4415 shared-group wait, kdlbs#4417 two race fixes, kdlbs#4421 false warning, kdlbs#4409 upstream version.

Upstream candidates

27f0f52831 + 410c924eb2 (dynamic route/recovery mutual block), 5839b219ac + f5553f5107 (queued work after restart), 10acddd52f (lazy-resume continuation fallback, pairs with the #241 parts), d18a9da33f (primary clear). ca5b76310d belongs to kdlbs#4045.

Validation (Windows)

  • Pre-fix reproduction: TestFailedSessionEventsUseTheFailureTime and TestLazyResumeCheckpointMissingRequiresContinuation fail on the pre-fix files (go test -overlay); TestLazyResumeOrdinaryStartFailureStaysAFailure is a guard and passes on both. The five TestRestartStrandedDynamicRoute_* tests use the new reclaim API and cannot compile on the pre-fix code; the mutual block itself was reproduced there with a temporary test (route action "route actions require a settled turn", recovery "dynamic_route_pending", successor launch failed).
  • Package runs on 0d52c304d9:
    • pass: agent/runtime, agent/runtime/dynamic, orchestrator/messagequeue, orchestrator/handlers, review, office/dashboard, persistence/requiredstores, task/deliveryretention.
    • orchestrator: the two passthrough attachment tests (Windows path separators, known baseline) and upstream's focus lease-loss test, whose assertion chore(sync): merge upstream main through acd873d41c #256 fixes (93ec2ba7d9, passes after the merge).
    • agent/runtime/lifecycle (Cursor MCP retry tests skipped; they hang on Windows on fork main too): 78 failures, all in fork main's Windows failure set.
    • automation: upstream's run-page test only (Windows timestamp tie, 3 of 5 runs).
    • persistence: 4 baseline failures plus upstream's 2 Windows-only snapshot tests.
    • backendapp: 3 failures, identical on e5a2f4030c.
    • task/repository/sqlite: 3 timestamp-tie flakes that also fail on e5a2f4030c when repeated.
    • worktree: 56 failures before the 40-minute timeout; 55 also fail on fork main or e5a2f4030c, the other is upstream's new cross-drive relative-path test.
  • Earlier isolated runs on e5a2f4030c (mock agent, fresh homes, never 38430/~/.kandev): A–E plain and dynamic, agents-first and backend-first restarts; dynamic sessions left action_required + open block recovered by message and by session.recover; route try_next and retry on blocked sessions continued from history and the next message was answered; a message queued during a restart was delivered with no action (106 s); a session contaminated like fix(lifecycle): recover streams that never reached a delivery checkpoint #241 failed native resume with failed to initialize ACP: agent delivery checkpoint missing and was continued automatically; Office cross-workspace toggle defaults off; web UI renders.
  • Isolated run on 0d52c304d9 (fresh home, dynamic routing on): health reports v0.97.0-593-g0d52c304d9; the embedded UI renders; three dynamic sessions started and answered; after a mid-turn agents-first restart the queued message was delivered with no action and a route retry on the blocked session continued from history with a follow-up turn. One mid-turn session was left FAILED by the old backend's automatic continuation (Windows cannot prove the retired runtime's termination; the same happened on e5a2f4030c at 01:56) and refuses messages until an explicit session.recover continue_from_history, which worked. Poison scenarios were not rerun on this build.

🤖 Generated with Claude Code

dreampiler and others added 30 commits October 12, 2026 14:39
Production (10 sessions, ~1,500 inbox rows/min) grew agent_delivery_inbox
from 74k to 162k rows in an hour while one five-second pass per minute
pruned at most ~1,100-1,500 payloads. On this base the pass budget, not
the proof, is the ceiling: 50-row leases, a 100 ms yield and a 60 s gap.

- Worker: a pass that runs out of its budget while removals still
  outnumber protected candidates, or that commits cleanup despite a
  transient batch deadline, resets backoff and starts the next pass after
  5 s instead of the interval. Quiet, discovery-only and
  protected-dominated passes keep the interval; passes without committed
  cleanup keep the jittered backoff, so a transient deadline inside a
  productive pass no longer escalates toward five minutes.
- Discovery: payload discovery selects only event types the locked proof
  can compact. Session-state events (capabilities, modes, MCP
  attachments, context windows, permission requests) were never pruned or
  retired yet filled bounded pages on every frontier wrap.

Batch, byte, deadline and yield bounds and every proof, including the
fork's open recovery block guard, are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…20261012

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resumes on release 20261012 failed with "agent delivery checkpoint
missing". Retention did not cause it: the failing streams have no SQL
cursor and no inbox rows at all, while the session journal still retains
them from sequence 1 with nothing acknowledged.

Two producers leave such a stream behind:
- The pre-#235 build dropped every event of affected executions, so their
  generation was committed without a single SQL receipt.
- On the current build a context continuation waits for its N+1 commit
  for at most 30s, but the commit happens only after the whole launch
  returns. A launch slower than 30s (production: startup workspace
  refresh) stopped the processor, the attempt was rolled back and the
  unacknowledged successor prefix stayed in the journal; every retry of
  the same successor stream was then refused.

- Give the continuation successor wait a 3 minute bound and end it as
  soon as its updates stream detaches.
- Replay an uncommitted continuation successor's abandoned prefix from
  zero, like the fresh bootstrap case; receipts still wait for the commit.
  Committed generations keep the upstream refusal.
- When native resume is refused for a missing checkpoint, continue from
  history, which starts a new successor stream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Free-model failover kept returning to models that had just failed:
unavailable/overloaded/at-capacity always blocked for one minute, the
strike count reset on the first streamed output, and the block scope
came from the error code only, although the same notice means different
things on different gateways.

- One cause table (dynamic/failure_scope.go) maps classifier rule, code
  and path (model provider prefix) to a cause, and each cause to a scope
  and duration: account allowances block the account until the stated
  reset or renewal, OpenCode Go usage blocks its paid tier 5h/24h/7d,
  free daily allowances wait for 00:00 UTC, withdrawn models 24h,
  intermittent endpoints 1-15m, model hosts 5-30m and the model on every
  path when a second path fails within 15 minutes, unrecognized
  unavailable notices 1m..1h. Provider reset hints still win.
- Strikes reset when a turn completes without a provider error, not on
  first output.
- try_next, policy skip and the interrupted successor exclude the same
  model's other paths only for causes that can reach every path.
- Credential-less profiles get an agent+provider account key (and a
  free/paid tier key) so account-wide causes reach sibling profiles.
- routingerr: rules for the catalogue signatures that were unclassified
  or mis-scoped (Zen/OpenRouter daily limits, upstream host pools,
  withdrawn models, Go account settings, Endpoint is unavailable,
  header timeouts, streamed 5xx, Internal Server Error, NVIDIA hosts,
  Cannot connect); limit reason codes survive sanitization.
- The automatic successor of a session's own failed attempt is handed
  its unit at the session ceiling instead of being refused once the
  attempt left the counted population.
- The successor search reads circuits without claiming probe leases and
  claims only the winner; a launch failure is recorded before its probe
  is released so its strike climbs.
- A dynamic attempt's recovery card reports dynamic_route_attempt
  instead of missing_evidence.
- mock-agent: /provider-failure:<model>:<kind> fails one model only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0261012b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…012b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A long shell tool emits many in-progress tool_update events and each one
carries the whole (bounded) output so far. Every event is a durable inbox
row, so one tool call kept O(n^2) payload bytes until the 1h payload
grace: production stored 399 updates of ~277 KB (110 MB) for a single
Codex shell call in 34 s and the DB grew 3.4 GB in one hour.

After a nonterminal tool_update is projected, lifecycle remembers each
projected progress-only update (in_progress/running/started/pending) as a
hint of sequence plus receive-time versioned hash, keyed by stream and
tool call. When a later update of the same tool call is projected, the
hints are queued to one tracked background drain, so projection and ACK
never wait. The repository re-proves every hint under the stream lock:
the later update must be the stored, projected receipt with the same
hash; each candidate goes through the worker's locked can_prune proof
with the grace replaced by supersession, must be an earlier nonterminal
tool_update of the same stream identity, and its persisted hash and byte
count must match the hint. The existing compaction statement clears the
body and keeps envelope, hash, byte count and generic receipt, exactly as
worker compaction does. Rows are never deleted; cursors, unprojected
rows, terminal rows, final tool statuses and the newest update are not
touched. Live websocket updates and canonical tool messages are
unchanged.

The mock agent gains e2e:shell_stream to reproduce cumulative shell
output in isolated instances.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and recovered prompt

A recoverable agent failure started a terminal stop (stopExecutionAgentctl
waits for agentctlLifecycleMu write side) while a queued prompt for the same
session resumed it through RecoverAgentPromptStream. restoreRecoveredFailedExecution
held a client read lease across markBootReadyFromFailed, whose persistence
path re-took the same lease via AgentctlURL() inside ExecutionStore.WithRLock.
With the writer pending, the recursive RLock blocked forever while holding the
store read lock, so ExecutionStore.Add and every store reader stalled and the
whole backend stopped launching or streaming (production 2026-10-12, twice).

- AgentctlURL no longer takes the client lease: the base URL is immutable per
  client, so it reads the override atomically or tries the source lock and
  never blocks under the store lock.
- restoreRecoveredFailedExecution releases the lease right after GetStatus,
  before marking the execution ready.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tore lock

Cancel escalation and a late prompt completion for the same execution can
deadlock the whole backend. Since kdlbs#2900, newAgentEventPayload snapshots
promptTurnID through execution.promptLifecycleMu. Three paths build that
payload inside ExecutionStore.WithLock, so they lock store.mu and then
promptLifecycleMu:

- markReadyEventForExecution (reached from escalateStuckCancel)
- PreparePassthroughRunning
- prepareRecoveredAvailability

The completion side locks in the opposite order. claimPromptCompletion and
handlePromptHandoffEvent hold promptLifecycleMu and then call
ExecutionStore.WithLock, which is also the order BeginPrompt uses.

When an agent does not acknowledge a cancel within the timeout,
escalateStuckCancel marks the execution ready while the agent's cancelled
turn completes. If the escalation holds or queues for the store lock while
the completion claim holds promptLifecycleMu, neither can proceed.
ExecutionStore.mu is global, so every GetBySessionID caller then blocks too:
permission resolution, session stop, agent cancel, stall publication and new
launches stop for all sessions until the backend restarts. Goroutine
profiles from a production instance showed this pair repeatedly, with the
completion parked in ExecutionStore.WithLock and markReadyEventForExecution
parked in promptTurnIDSnapshot under the store lock.

Add ExecutionStore.WithPromptLifecycleLock, which follows BeginPrompt: look
up the execution, lock promptLifecycleMu, lock the store, and confirm the ID
still maps to the same execution. The three payload paths use it and read
promptTurnID directly while the lock is held.

TestMarkReadyDoesNotDeadlockWithPromptCompletion holds one store reader so
each payload path queues for the store lock before the completion claim
takes promptLifecycleMu. Before this change all three cases deadlock; with
it they complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 0e5d235)
…ordinator-mediated control

Adds a default-off Office workspace setting, cross_workspace_access
(enabled, allowed_workspace_ids, coordinator_task_ids). When on, the
Office workspace's agents can read other workspaces (tasks, sessions,
conversations, plans) and message their tasks through new runtime
routes, CLI commands and capability-gated MCP tools. Reach is decided
per call by internal/office/crossworkspace: owner workspace.read
(including the org boundary when organizations are on) intersected with
the optional allow-list. Control actions stay with each workspace's
coordinator.

Messages from other tasks to an Office task, coordinator reports via
report_to_handoff_source_kandev, and handed-off tasks reaching a
terminal state become comments that wake the Office assignee.

Spec: docs/specs/office/requirements/office-cross-workspace-access.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…QLite writer

The required-persistence probe pinged the single SQLite writer connection
and ran every required-table check through that same writer, all inside
one 5s timeout. Under sustained agent write load the probe queued behind
real writes, missed the deadline, and after three such probes the
middleware answered 503 persistence_unavailable for every listed store
while the database kept committing normally.

- Table existence checks now read the schema through the reader pool,
  with their own budget that starts after the writer ping.
- The writer driver registers a commit hook that records the last commit
  time. A writer ping that only hits its deadline while the writer has
  committed within the last probe window is contention, not an outage,
  and no longer counts as a probe failure. Without recent commits the
  ping gets a longer budget (3x the probe timeout).
- Real failures still fail: a closed pool, driver errors, missing tables,
  and a held writer that stops committing. The maintenance TryAcquire
  deferral and the consecutive-failure threshold are unchanged.

Tests cover a busy committing writer (stays healthy, then fails once it
stops committing while held), a missing table while the writer commits,
and a closed writer with a recent commit. The two existing fixtures that
used separate in-memory writer/reader databases now create the required
tables on the reader too, since schema reads go through the reader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A restart can leave a dynamic-profile session with its route parked at
action_required, an open recovery block for the interrupted turn, and a
turn row no runtime will settle. Every recovery path then refused the
others' state: session.recover and the automatic continuation refused the
parked route as dynamic_route_pending, session.route_action refused the
orphaned turn ("route actions require a settled turn"), and after a cancel
the route action re-sent the last prompt, which the recovery block refused
(route_action_launch_failed).

- An explicit recovery action (resume, continue_from_history, fresh_start,
  resume_new_branch) reclaims an action_required route on its persisted
  candidate when no runtime is live: the durable route and the session
  projection move to starting on the same generation, which the existing
  stranded-starting admission accepts. A failed recovery returns the route
  to action_required.
- A route action settles turn rows that are orphaned on a parked route
  (no live runtime, no resume, queued dispatch or steer in flight). A turn
  with a live runtime still refuses the action.
- A manual route action on a session blocked by a restart-interrupted turn
  parks its claimed generation and runs continue_from_history on it after
  releasing the route-action lock, instead of re-sending the prompt whose
  outcome is unknown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… drains

Every queue drain trigger belongs to a live runtime or a new message, so a
primary session that was idle with queue rows at shutdown kept them, and a
drain refused by a restart-interrupted turn's recovery block never started
the continuation that releases it. message.queue.drain reported that
refusal as INTERNAL_ERROR.

- The startup lifecycle sweep ends with one pass over sessions holding queue
  rows or parked dispatches. A primary WAITING_FOR_INPUT session with
  Auto-run on continues a restart-interrupted turn from history, drains
  directly when its runtime survived, or resumes through the queue-drain
  ensure (ceiling refusals are recorded for replay). Paused queues, Office,
  archived, non-primary, coordinator and parked-route sessions are skipped.
- A drain refused by an interrupted-turn recovery block starts the same
  automatic continue_from_history a refused prompt gets, when queue rows or
  a parked dispatch wait.
- message.queue.drain maps a recovery block to session_recovery_required.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ound

The startup re-entry of a stranded SCHEDULING task waited for the agent to
resume inline, holding the lifecycle-token pass, dependency launches and
queued-session delivery for up to one launch timeout per orphan.

Ported from kd/upstream-restart-recovery (4b14826).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eplay

A focus resume of an idle-parked completed session refused by a full
ceiling was deferred without its completed-session permission, and the
sweep dropped the completed destination as superseded, so the session never
resumed once capacity freed. The deferred resume record now stores the
completed-session permission and the idle-suspension check, the replay
restores both, and the session's own queued launch no longer blocks the
replay that owns it.

Ported from fix/idle-focus-resume-automatic-origin (b82abcd, kdlbs#4406).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both starts saw no session and planned to become the task's first, primary
session. The winner creates it while the loser waits, so the loser's retry
kept a stale primary decision and SetSessionPrimary took the flag from the
winner. Re-read the task's sessions before the retry: drop the first-session
origin (and the initial runtime overrides this start added) when a session
exists, and do not claim primary when one is already primary.

Ported from fix/executor-retry-after-workspace-preparing (8fa1da7, kdlbs#4415).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng delivery rows

Delivery row retirement never advanced: retirement walks each stream from
retired_sequence+1, and every stream starts with agent_capabilities, a
session-state event outside the retention classifier. retired_sequence stayed
0 on every cursor, so agent_delivery_inbox and its generic
agent_delivery.event receipts grew at the full inflow rate; only payloads
were pruned.

- Classify the known session-state events (capabilities, modes, models, MCP
  attachment, context window, permission notices, turn markers, usage,
  background work) like projected output. After projection with a completed
  generic receipt no resume or replay path reads the row: resume and
  continue_from_history read only unprojected rows, duplicates skip the
  callback and expired replays reconcile from the authenticated projected
  cursor. Unknown types still stay intact and stop retirement.
- Retirement discovery visits each stream's retirement head and pages
  forward only where that head is compacted and outside the replay window,
  instead of walking every compacted receipt. Receipts inside the window are
  no longer rescanned and an expired head is reached in one bounded batch.
- An open recovery block protects delivery rows only while its session is
  live and current. Blocks of completed/failed/cancelled sessions or of a
  session superseded by a newer primary stay open as the recovery fence but
  no longer pin payload cleanup or retirement.

Generic receipts retire with their inbox rows; terminal, workflow,
submission and cursor evidence is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… resume is refused

Sessions whose durable delivery stream never reached a checkpoint still
failed with "failed to initialize ACP: agent delivery checkpoint missing"
when a message or a queued dispatch woke them. #241 added the
continue_from_history fallback only to session.recover resume; the lazy
resume a prompt performs starts the agent asynchronously and only saw the
session turn FAILED, so the message failed and the session stayed broken.

The asynchronous start-failure callback now records a start refused for a
delivery reason only continue_from_history settles (checkpoint missing or
durable adoption blocked), and the lazy resume wait reports that FAILED
session, like a synchronous refusal, as a recovery-required error with the
unresolved_durable_work reason. The message and queue paths already answer
that error with automatic continue_from_history on a new successor stream,
which carries the recorded message. Other start failures are unchanged.

Automatic continuation for queued work now only wakes a primary session.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 24-hour feed used a session's updated_at as both the failure time and
the window filter, so any later write to an old FAILED row brought it back as
a new failure. A step move on one task rewrote all 14 of its sessions, and
five sessions that failed on 10-10 and 10-11 appeared as failures at the
time of the move.

The overview session query now loads completed_at, and a finished session's
failure time is its completion time (updated_at only for a row without one).
The failure feed and the status checks that compare a failure with later
progress or a later step entry use that time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Promoting a primary session cleared is_primary on every session of the task
and so rewrote updated_at on all of them, including long-finished rows. A
workflow step move promotes its destination session, so each move made old
sessions look freshly updated. The clear now touches only the current
primary, and only when it is not the session being promoted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…se 20261012d

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ttlement partial index

The fork-only idx_agent_delivery_effects_pending indexed every effect row by
(effect_type, created_at, stream_id, sequence) for a single reader,
ListPendingAgentDeliverySettlements (172 MB on a 14 GB production copy).
Only unsettled terminal settlements match that scan.

Replace it with idx_agent_delivery_effects_pending_settlement, a partial
index over effect_type = 'agent_delivery.terminal_settlement' AND
state <> 'completed', and spell that predicate as literals in the query so
both engines can prove the partial index. Startup drops the legacy index with
an idempotent DROP INDEX IF EXISTS; an older binary recreates its own index on
rollback. The plan stays an ordered index seek without a temp sort.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ft FAILED

When the agents die before the backend, a session can end FAILED with an
unproven runtime stop and an open recovery block for its interrupted turn,
while its queue still holds work. The startup queue pass skipped it because it
only took WAITING_FOR_INPUT sessions. A FAILED primary now qualifies when, and
only when, it carries a restart-interrupted turn's recovery block; the pass
then continues it from history like a waiting session. A FAILED session
without such a block is left for its owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…61012d

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… release 20261012d

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…etry

Retry on a dynamic session that a restart-interrupted turn still blocks was
answered by the route engine, which refused it as a pending recovery when the
durable route stayed active while the session itself ended FAILED (the agents
died before the backend and the automatic recovery could not prove the old
runtime stopped). For such a session Retry now runs continue_from_history on
the current candidate after the route-action lock is released, which settles
the block without re-sending the interrupted prompt. A stale generation, a
live runtime, or an Office task keep the route handler's answer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…61012d

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1012d

Takes the fork's upstream sync (kd/upstream-sync-20261013) so the release
contains upstream main through acd873d.

Conflicts:
- lifecycle/manager_lock_order_test.go: upstream's kdlbs#4414 test, a superset
  of the release's lock order test for the same fix.
- ceiling_dispatch_admission.go, task_operations.go: upstream's
  acd873d idle focus admission replaces the release's port of the same
  review follow-up (b82abcd); the remaining port files now match upstream.
- docs/public/operations.md: upstream's retention cadence with the
  release's session-state compaction and retirement discovery notes.
- office/AGENTS.md: both rows.
- event_handlers_automation.go: the sync branch's lint annotation on
  upstream's exact-turn settlement (same change as the sync follow-up).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Takes the sync branch's retry-run restart settlement fix (2eb7036) and
its lint annotation, which the release merge already carried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fork main now carries the upstream sync through acd873d (#256), so
upstream/main is an ancestor of this release. The only new content is the
sync's focus lease test assertion fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dreampiler
dreampiler merged commit e96e10f into main Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant