Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 95 additions & 18 deletions docs/plan-durability.md
Original file line number Diff line number Diff line change
Expand Up @@ -434,16 +434,81 @@ task's PREPARED record. If the crash happens after close ledger events append
but before plan file writes complete, the next `loadPlan()` call detects the
hash mismatch and rebuilds plan files from the authoritative ledger.

### Restart reconciliation and authority boundaries (issue #2668)

Restart reconciliation must keep durable workflow policy separate from
process-local execution authority. The following distinction is intentional:

| Durable across restart | Ephemeral and never resurrected as authority |
| --- | --- |
| `.swarm/plan-ledger.jsonl` and its replayed plan identity | Session QA overrides and the session `auto-proceed` override |
| The plan execution profile and persisted QA-gate profile for that plan identity | In-memory session context, child handles, timers, and retry/circuit state |
| Evidence/WAL records, reservation/lease records, and their owner-visible recovery classifications | Live ownership/lease authority from a prior process; it must be re-proven after restart |

The plan ledger remains authoritative. `plan.json` and `plan.md` are derived
projections, and `get_qa_gate_profile` reports the persisted profile for the
exact plan identity; it does not turn a session override into durable policy.
Similarly, a session-only override may affect the current process, but a new
process must not infer execution authority from that override, an old lease
record, or an in-memory child handle.

The restart sequence is deliberately ordered:

1. Snapshot coordination rehydrates the session with a generation/scope fence.
Transient authority is reset; a late result from an older generation cannot
clear or settle newer work.
2. After the plugin manifest is available, the post-resolution coordinator
calls authoritative `loadPlan()` before building the rehydration cache or
exposing a projection-dependent status. This keeps repair and replay off the
bounded plugin-manifest path.
3. The coordinator exposes readiness (`running`, `succeeded`, `superseded`,
`failed`, or `timed_out`) through the status/inspect surfaces. A superseded
attempt is settled but not successful: a fresh current-generation attempt
must perform recovery instead of reusing it. A failed or timed-out attempt
is retried only after the previous attempt has settled; an unsettled attempt
remains unknown rather than being guessed healthy.
4. Durable evidence, WAL settlement, and reservation records are classified
for the owner. Recovery may release an expired reservation only with
corroborated owner absence; expiration by itself is not proof.

#### Missing or corrupt projections

Never repair `plan.json` or `plan.md` by hand. A missing, stale, or malformed
projection is a derived-state problem:

- `loadPlan()` replays `.swarm/plan-ledger.jsonl`, compares the projection
identity/hash, and rewrites the derived projection when the ledger is valid.
- A corrupt ledger suffix is quarantined and replay resumes from the last
valid event. Restart-owned replay rechecks its exact hydration authority
after the integrity-read await and immediately before publishing the unique
quarantine side file; a superseded restart leaves no stale quarantine
artifact. If the remaining history is not sufficient to prove the plan
identity, the plan stays unknown and operator recovery is required.
- The persisted QA profile is read by exact plan identity. A missing or
mismatched profile is not silently replaced by a session override.
- Inspect with `/swarm status`, `/swarm diagnose`, `get_approved_plan`, and
`get_qa_gate_profile`; use `/swarm recover --coordination` for coordination
readiness and `/swarm recover <task_id>` for an owner-visible recovery
classification. See the [recovery runbook](troubleshooting/recovery-runbook.md)
for the decision table.

Replay and accepted recovery are idempotent: repeating the same restart or
recovery observation must not mint a new identity, append a duplicate ledger
decision, revive a prior process's authority, or erase an uncertain external
effect. Unknown, ambiguous, and corrupt states remain visible until evidence
supports a bounded transition.

## Corruption Handling

If a ledger entry fails validation:

1. The bad suffix is **quarantined** to `.swarm/plan-ledger.quarantine`
1. The bad suffix is **quarantined** to a unique
`.swarm/plan-ledger.quarantine.<timestamp>.<content-hash>` side file
2. Replay continues from the last valid event

```
.swarm/plan-ledger.jsonl ← continues with clean events
.swarm/plan-ledger.quarantine ← bad entries isolated (never replayed)
.swarm/plan-ledger.quarantine.* ← bad entries isolated (never replayed)
```

## Migration from v6.41.x
Expand Down Expand Up @@ -678,16 +743,23 @@ Three layers, with distinct authority:
- **Process-local** — the four live maps themselves and `pendingRehydrations`
(bounded by their pre-existing lifecycle mechanisms: the 2-hour idle-TTL
sweep for sessions, `resetSwarmState` for the rest), plus the per-project
registries in `src/session/hydration-ownership.ts` (hydration generation
counters, per-project rehydration caches, per-project hydrated-aggregate
key sets — FIFO-capped at 32 entries each, with the directory→key memo
FIFO-capped at 64). All are cleared by `resetSwarmState`.
registries in `src/session/hydration-ownership.ts` (hydration authority
records, per-project rehydration caches, per-project hydrated-aggregate key
sets — FIFO-capped at 32 entries each, with the directory→key memo
FIFO-capped at 64). `resetSwarmState` clears the bounded registries, but not
the process-monotonic authority epoch: reusing an epoch after reset could
revive a stale pre-reset token.
- **Project-local** — ownership stamps on each session:
`owningProjectKey` (the canonical project root that created or restored the
session; never serialized — the hydrating directory defines it, snapshot
bytes never do) and `hydrationStamp` (the per-project hydration generation
the session was created/restored at). Sessions created without a directory
are unowned and survive every hydration (fail-open toward preservation).
bytes never do), `hydrationStamp` (the per-project hydration generation), and
its paired `hydrationAuthorityEpoch` (the process-local authority
incarnation). The generation and epoch together identify the authority that
created/restored the session; an older epoch is stale even when its numeric
generation is larger after bounded-record eviction or reset. All three
fields are process-local and never serialized. Sessions created without a
directory are unowned and survive every hydration (fail-open toward
preservation).
- **Authoritative** — the durable ledger and SQLite snapshot store. Hydration
only READS them and never writes them (invariant 5); a re-hydration replaces
the project's own snapshot-derived sessions from the durable read, nothing
Expand All @@ -696,16 +768,21 @@ Three layers, with distinct authority:
Rules a hydration for project K follows (`rehydrateState`,
`src/session/snapshot-reader.ts`):

1. **Fence (generation):** each initiation (`loadSnapshot` entry,
1. **Fence (authority):** each initiation (`loadSnapshot` entry,
`startSnapshotCoordinationInitialization`, retry) captures a scope
`{projectKey, generation}` from a monotonic per-project counter. An apply
whose generation is older than the project's current counter is refused
with zero mutation — a timed-out initializer settling late cannot publish
over the state of any newer hydration.
2. **Stamp (recency):** an accepted apply at generation `g` evicts only
sessions with `owningProjectKey === K` AND `hydrationStamp <= g`. A live
session created after `g` began carries stamp `g+1` and survives its own
project's in-flight hydration.
`{projectKey, generation, authorityEpoch}`. The per-project generation
orders retained records; the process-monotonic epoch makes the scope
non-reusable after FIFO eviction, reset, and reinsertion. An apply whose
exact authority is no longer current is refused with zero mutation — a
timed-out initializer settling late cannot publish over the state of any
newer hydration, even when its numeric generation is reused.
2. **Stamp (recency):** an accepted apply at authority
`{ generation: g, authorityEpoch: e }` evicts sessions with
`owningProjectKey === K` when their epoch is not `e`, or when their paired
`hydrationStamp <= g`. A live session created after `g` began carries the
current epoch and stamp `g+1`, so it survives its own project's in-flight
hydration. Comparing the epoch before the numeric stamp closes the ABA
window where FIFO eviction or reset reintroduces K at generation 1.
3. **Scope of mutation:** another project's sessions, unowned sessions, and
`toolAggregates` keys the project never published are untouched.
`toolAggregates` replacement is limited to the keys K's previous hydration
Expand Down
47 changes: 47 additions & 0 deletions docs/releases/pending/2668-restart-policy-reconciliation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Restart policy reconciliation

## What changed

- Restart hydration now has an explicit authority boundary: the plan ledger,
durable execution profile, persisted QA-gate profile, evidence, and recovery
WALs, and lease records are durable inputs; session overrides, active
ownership, live lease authority, child handles, timers, and retry/circuit
state are ephemeral and are not revived as execution authority.
- The post-resolution restart coordinator uses authoritative `loadPlan()` before
projection/cache inspection. Missing or stale `plan.json`/`plan.md` files are
regenerated from `.swarm/plan-ledger.jsonl`; invalid ledger suffixes remain
quarantined instead of being silently discarded. An initializer that loses
its exact hydration authority reports `superseded`, never reusable success,
so a current-generation retry still performs recovery. Authority includes a
process-monotonic epoch, preventing FIFO eviction or reset from reviving an
old callback when a numeric per-project generation is reused.
- Session recency, workflow-cache entries, and hydrated aggregate ownership
pair their numeric state with that process-local authority epoch, so an older
session/cache/aggregate owner cannot survive a hydration merely because its
numeric stamp or generation is larger after a generation is reused (ABA).
- Corrupt-ledger replay carries the same authority fence through every recovery
replay call site (including schema-invalid/missing projections, `savePlan`,
and `rebuildPlan`) and its integrity/quarantine path. Spec-staleness output
and save/rebuild marker publications use the same post-await fence, so an
obsolete restart cannot publish misleading recovery artifacts after
supersession.
- Interrupted, cancelled, stale, ambiguous, corrupt, live-wedge, and
old-generation results remain owner-visible. Recovery releases or repairs
local state only when the durable evidence proves that transition; uncertain
provider or worktree effects stay uncertain.
- The recovery runbook documents the operator restart/inspect/recover sequence;
the registered host journey covers policy/identity and task inspection, with
settlement categories exercised directly by its deterministic classifier
cases.

## Operator guidance

After a restart, inspect `/swarm status`, `/swarm diagnose`,
`get_approved_plan`, and `get_qa_gate_profile`. Use `/swarm recover
--coordination` only after a failed or timed-out coordination attempt has
settled, and use `/swarm recover <task_id>` for receipt-backed stale or
live-wedge repair. Do not hand-edit derived plan projections or force a live
foreign owner. See [`docs/troubleshooting/recovery-runbook.md`](../../troubleshooting/recovery-runbook.md).

No new plan identity or ledger migration is required. Repeated replay and
accepted recovery are idempotent.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Hydration is project-owned and generation-fenced
# Hydration is project-owned and authority-fenced

## What

Expand All @@ -14,12 +14,13 @@ no longer publish over newer state:
its own snapshot state. Calling it without a directory keeps the legacy
clear-all (direct-test path only).
- Each hydration initiation (`loadSnapshot` entry, SQLite snapshot
coordination initialization and retries) captures a generation scope from a
monotonic per-project counter (`src/session/hydration-ownership.ts`). An
apply whose generation has been superseded is refused with zero mutation —
a timed-out initializer that settles late can no longer wipe state written
by a newer generation. Live sessions created while a hydration is in flight
carry a stamp above it and survive that hydration's own apply.
coordination initialization and retries) captures an exact authority scope
from `src/session/hydration-ownership.ts`: a per-project generation plus a
process-monotonic epoch that cannot be reused after bounded-registry eviction
or reset. An apply whose authority has been superseded is refused with zero
mutation — a timed-out initializer that settles late can no longer wipe
state written by a newer generation. Live sessions created while a hydration
is in flight carry a stamp above it and survive that hydration's own apply.
- The plan/evidence rehydration cache is per-project instead of a
process-global singleton: whichever project built the cache last no longer
feeds foreign workflow states to another project's new sessions.
Expand All @@ -45,8 +46,10 @@ another project's sessions (issue #2667).
- New session fields `owningProjectKey` and `hydrationStamp` are never
serialized; the snapshot field-parity guard covers them.
- The new per-project registries are bounded (FIFO, 32 projects) with an
explicit `resetSwarmState` reset path; canonical project keys resolve
through a bounded memo (one realpath per directory spelling per process —
no new init-path filesystem cost).
explicit `resetSwarmState` reset path. The small authority-epoch scalar is
deliberately process-monotonic across resets so an old token can never
become current again. Canonical project keys resolve through a bounded memo
(one realpath per directory spelling per process — no new init-path
filesystem cost).
- Ownership and generation rules are documented in
`docs/plan-durability.md` § Live-State Ownership and Hydration Fencing.
56 changes: 51 additions & 5 deletions docs/testing/execute-journey.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,57 @@ A plan whose identity (swarm/title) was mutated after approval is refused by
- Deterministic transport: a constructor-injected `ScriptedHostClient`
(real SDK response shapes) records every host call; native-task child
outputs are scripted at the hook boundary. No live model, no network.
- Tests: `tests/unit/execute-journey/j01…j07` (per-file CI on all three
OSes). j04's restart uses `resetSwarmStatePreservingSingletons()` between
in-process boots — a real second OS process would run the same hydration
path; this in-process limitation is the disclosed boundary of the restart
claim.
- Tests: `tests/unit/execute-journey/j01…j08` (per-file CI on all three
OSes). j04's restart and j08's policy-reconciliation restart use
`resetSwarmStatePreservingSingletons()` between in-process boots — a real
second OS process would run the same hydration path; this in-process
limitation is the disclosed boundary of the restart claim.

## Restart policy reconciliation journey (#2668)

The registered host fixture
`tests/unit/execute-journey/j08-restart-policy-reconciliation.test.ts`
qualifies the boundary between durable policy and ephemeral execution
authority. It uses the same `execute-journey-driver.ts`, real
`OpenCodeSwarmPlugin.server()`/`bootSwarmPluginHost`, disposable git project,
XDG-hermetic environment, and constructor-injected `ScriptedHostClient` as the
other journey fixtures. It does not use a live model or network.

The deterministic sequence is:

1. Boot the plugin, create and approve a plan, persist its execution profile,
and persist the QA-gate profile for the exact plan identity.
2. Set a session-only QA or auto-proceed override, then start a scoped coder
dispatch and leave its durable evidence as the restart input.
3. Restart the host. The post-resolution coordinator must replay the
authoritative plan ledger before projection/cache inspection, while the
snapshot path applies its generation fence and clears ephemeral authority.
4. Inspect `get_approved_plan`, `get_qa_gate_profile`, and the registered task
inspection path. The durable execution/QA policy and plan identity must
remain; the prior session override, ownership, live lease authority, child
handle, and timer must not be treated as permission to execute. Any durable
lease record is recovery evidence only.
5. Exercise the settlement classifier directly with deterministic owner states:
provably dead work is `stale`, a live or foreign owner is `ambiguous`, and
unreadable evidence is `corrupt`. The registered journey separately proves
that an old-generation late result is refused without clearing newer work.
Expired leases are released only when owner absence is corroborated.

Separate frozen acceptance cases remove or corrupt a derived projection and
run post-resolution coordination again. A valid ledger must rebuild the
projection without a new plan identity or duplicate recovery decision;
insufficient or corrupt authoritative history remains visibly unknown.

Run the registered fixture in isolation with:

```sh
bun test tests/unit/execute-journey/j08-restart-policy-reconciliation.test.ts
```

The report must retain evidence for both boots, the exact plan binding, the
persisted QA profile, registered task-inspection output, and the direct typed
interrupted/cancelled/uncertain classifier cases. A passing fixture does not
convert an uncertain provider or worktree effect into a local success claim.

## Executed host/runtime cells

Expand Down
Loading
Loading