chore(release): reconcile native fix with dev

Author: LeXwDeX

.github/releases/v1.0.40.md

@@ -9,28 +9,44 @@
 - **Tools survive automatic compaction, #539**: a high-usage `step-finish` could abort a slow local tool before its result reached the session processor. Native LLM now delivers all local tool results before terminal events. Parallel tools settle completely; explicit user cancellation still interrupts execution.
 - **Summary diffs retain later small entries, #526**: skip an oversized diff individually instead of dropping every following entry. Remove the unused legacy `session.summary_diffs` column through a tested database migration.
 - **Identical durable events no longer consume storage or sequence numbers, #527**: suppress byte-identical fresh appends within the same aggregate/type while preserving explicit-sequence replay. Batch results retain input alignment; legacy rows require no hash backfill.
+- **Config startup preserves npm lock files, #542**: keep an existing lock unchanged when the plugin SDK resolves entirely from local or bundled packages. Mixed registry requests and genuine package changes still regenerate the lock.
+
+---
+
+### 🏗️ Architecture / Refactor
+
+- **Deleted-session storage reclamation, #537**: remove durable event residue for deleted aggregates, wire cleanup into session deletion, and add tested SQLite reclamation support. This release does not run the deferred #531 maintenance operation on the user's existing database.
 
 ---
 
 ### ⚙️ CI / Engineering
 
 - **Delivery tracking, #520 and #532 through #535**: close linked issues after dev merges, preserve repository-specific SpecGit harness files, restore failed bootstrap state, reject unsupported branch types before remote writes, and verify that delivery PRs target dev.
 - **macOS installation verification, #536**: verify release archive checksums before extraction and validate the installed binary's signature after quarantine clearing and re-signing. Added a negative checksum control and a real macOS installation acceptance test.
+- **Local npm fixture isolation, #540**: keep real package-installation regressions independent of online vulnerability-audit latency while retaining their assertions and deadlines.
 
 ---
 
 ### 🧪 Test Summary
 
 ```
-LLM / native runtime / processor targeted suites: 66 pass, 1 existing skip, 0 fail
-opencode typecheck: passed
+Integration CI baseline (dev 8060765fcc):
+core: 1225 pass, 6 skip, 0 fail
+opencode: 4426 pass, 23 skip, 1 todo, 0 fail
+HttpAPI coverage / auth / effect: 230 pass each, no failures or missing routes
+Generated client and SDK freshness: passed
+Typecheck, DAG core gate, Linux and Windows E2E: passed
+
+Merged native/session/TUI regressions: 36 pass, 0 fail
+Merged npm regressions: 8 pass, 0 fail
+Merged opencode package typecheck: passed
 ```
 
 ---
 
 ### 🔍 Verification
 
-The slow-tool regression was observed failing before the fix and passing afterward through the real session processor and a local HTTP model endpoint. Additional cases cover parallel local tools and explicit cancellation. Formal publication requires independent review, SpecGit acceptance, and the main-branch Typecheck, Linux unit, and Linux/Windows E2E gates. Reported model usage in the regression is deterministic test input; no live model context limit is inferred from it.
+The slow-tool regression was observed failing before the fix and passing afterward through the real session processor and a local HTTP model endpoint. Additional cases cover parallel local tools and explicit cancellation. Independent Standards and Spec reviews found no code blockers. The integration statistics above come from [dev CI](https://github.com/LeXwDeX/OpenCode-GraphAgent/actions/runs/33868550950); they identify the tested baseline and do not substitute for the final release PR's Typecheck, Linux unit, Linux/Windows E2E and SpecGit acceptance gates. Reported model usage in the regression is deterministic test input; no live model context limit is inferred from it.
 
 ---
 

docs/adr/0001-event-residue-scrub-and-sqlite-auto-vacuum-reclamation.md

@@ -0,0 +1,118 @@
+# ADR 0001: Event residue scrub and SQLite auto-vacuum reclamation
+
+- **Status:** Accepted
+- **Date:** 2026-09-04
+- **Issue:** #524 (delivery: PR #537 → `dev`)
+- **Supersedes:** none
+
+## Context
+
+The durable event store had no reclamation semantics. `Event.remove(aggregateID)` ran only on
+explicit session removal and covered the session aggregate alone, and SQLite ran without
+`auto_vacuum`, so deleted rows never returned pages to the filesystem. Two failure shapes
+motivated the decision:
+
+- A crash between the session-row delete and any cleanup stranded durable event aggregates
+  whose `SessionTable` and `WorkflowTable` read models were both gone. Replaying such an
+  aggregate is impossible: the `WorkflowCreated` projector INSERT dies on the
+  `workflow.session_id` foreign key once the session row is gone (pinned by
+  `packages/opencode/test/dag/dag-replay-idempotency.test.ts`), so residue can neither be
+  replayed nor re-materialized — it can only be deleted.
+- `database.ts` initialized WAL and pragmas after driver open, so an application-level
+  `PRAGMA auto_vacuum` could not take effect: after WAL initialization the pragma silently
+  yields NONE even on an empty database.
+
+The decision checkpoint was approved on 2026-09-04 with the scope locked below.
+
+## Decisions
+
+1. **Explicit session + dag scrub.** `Session.remove` captures every related dag aggregate ID
+   before the `Deleted` publish (the projector's session-row delete FK-cascades the workflow
+   rows inside the publish transaction, so a post-publish lookup would see nothing) and removes
+   each dag event aggregate after the session aggregate — terminal workflows included. The
+   per-dag scrub is soft-degrading (a failure is logged and the aggregate is left for the
+   startup sweep) but preserves interruption (`Cause.hasInterrupts` re-raise, the
+   `EventResidueSweep` sibling discipline).
+2. **Guarded default-on startup sweep.** `EventResidueSweep` runs one pass per process start,
+   forked into the layer scope so it can neither block nor fail startup. Eligibility is the
+   zero-live-read-model rule: an aggregate in `event_sequence` with neither a `session` nor a
+   `workflow` row. Removal is a single atomic guarded `DELETE` that re-evaluates both
+   NOT EXISTS guards inside the statement (no select-then-delete TOCTOU window), so an
+   aggregate recreated concurrently survives. Wired into `AppLayer` and the HttpApiApp node
+   graph, so every serving process sweeps; the pass is idempotent.
+3. **New databases: FULL before WAL.** Both SQLite drivers (`sqlite.bun.ts`, `sqlite.node.ts`)
+   set `auto_vacuum=FULL` at the driver layer, before `journal_mode=WAL`, and only on a
+   genuinely empty (0-page) file. An immediate SQLITE_BUSY from a second opener racing the
+   first is tolerated: the pragma is a persistent header property and runs before any
+   WAL/migration write, so the first-write winner sets FULL for the database.
+4. **Existing databases: explicit conversion only.** Legacy `auto_vacuum=NONE` databases are
+   never converted at startup — startup is detect-only (a warning pointing at the command). The
+   only conversion path is `opencode db vacuum --db <path>`: the target must be named
+   explicitly and must already exist as a regular file (vacuum never creates a database), runs
+   FULL → VACUUM → `wal_checkpoint(TRUNCATE)` outside any startup path, and fails nonzero
+   unless the `PRAGMA auto_vacuum` readback is exactly FULL. Exclusive access is a hard
+   requirement (a concurrent writer fails VACUUM with SQLITE_BUSY).
+5. **Archived-session retention: off and deferred.** No retention policy for archived sessions
+   ships in this decision (Phase 3).
+6. **Active truncation: rejected.** Truncating active/retained session event history and event
+   snapshot folding are rejected; incremental replay (`seq > after`, ascending) and sync
+   cursors must keep observing unbroken per-aggregate histories.
+7. **`incremental_vacuum` is forbidden.** A disposable bun:sqlite prototype reproduced an
+   exit-139 crash under the incremental mode; no code path may enable it.
+
+## Consequences and risks
+
+- Deleting events on legacy NONE databases still does not shrink the file until an operator
+  runs the explicit conversion; disk usage grows until then.
+- `auto_vacuum=FULL` pays its known SQLite overhead (pointer-map pages, per-update mapping) on
+  every new database in exchange for automatic page reclamation.
+- The sweep runs once per process start: residue created and abandoned within a single process
+  lifetime waits for the next start. This is accepted because the shapes it targets are
+  crash/in-flight zombies.
+- The conversion command requires exclusive access; the error guidance says to close running
+  opencode processes and retry.
+- Replay and sync contracts are preserved by construction: only whole aggregates with no live
+  read model are ever removed, and such aggregates are unreplayable anyway (FK death), so no
+  consumer can observe the removal as a gap in a replayable history.
+
+## Alternatives considered
+
+- **Rely on replay instead of scrubbing** — rejected: a wiped dag aggregate whose session row
+  is gone dies on the workflow foreign key during re-materialization, so replay cannot replace
+  deletion.
+- **Silent startup conversion of legacy databases** — rejected: converting requires a blocking
+  full VACUUM; startup stays non-blocking and detect-only.
+- **`PRAGMA incremental_vacuum`** — rejected (decision 7).
+- **A recurring background reaper** — rejected in favor of one idempotent guarded pass per
+  process start; residue is crash-shaped, not steady-state throughput.
+- **Truncate or fold active event histories** — rejected (decision 6).
+
+## Rollout and rollback
+
+Rollout lands as ordinary PRs through `dev` per the release train; no operator action is
+required — new databases get FULL automatically, legacy databases keep working unchanged (with
+a detect-only warning), and the sweep is default-on. Rollback is removing the sweep from the
+app graphs and reverting the driver pragma: the sweep is additive and idempotent, and legacy
+databases were never written by any of this. A database created with FULL keeps its header
+mode; reverting one is itself an explicit operator VACUUM and is not automated.
+
+## Acceptance
+
+- Active/retained session replay is unchanged; only zero-live-read-model aggregates are
+  removed (guarded delete re-checked inside the statement).
+- Cleanup failures never block the application path; interruption is preserved, not logged as
+  failure.
+- Disposable-file tests demonstrate page reclamation and the new/existing database behavior;
+  no startup-time full VACUUM exists.
+- `bun run test:dag-core`, focused event/session tests, package typecheck, and migration
+  freshness checks pass in CI.
+
+## Non-goals
+
+- **No global bounded-retention claim.** Live and retained sessions keep their full event
+  history indefinitely; this decision bounds nothing by age, size, or count.
+- **No tombstones, unarchive, or sync changes.** Offline deletion tombstones, unarchive
+  semantics, and sync cursor/protocol changes stay out of scope.
+- **No authorization for #531 or live-database work.** This decision does not authorize running
+  VACUUM or any cleanup against a live local database; the destructive operator procedure
+  remains the human-only issue #531.

packages/core/src/database/database.ts

@@ -2,7 +2,8 @@ export * as Database from "./database"
 
 import { EffectDrizzleSqlite } from "@opencode-ai/effect-drizzle-sqlite"
 import { layer as sqliteLayer } from "#sqlite"
-import { Context, Effect, Layer } from "effect"
+import { Cause, Context, Effect, Layer } from "effect"
+import { sql } from "drizzle-orm"
 import { Global } from "../global"
 import { Flag } from "../flag/flag"
 import { isAbsolute, join } from "path"
@@ -29,6 +30,27 @@ export const layer = Layer.effect(
     yield* db.run("PRAGMA busy_timeout = 5000")
     yield* db.run("PRAGMA cache_size = -64000")
     yield* db.run("PRAGMA foreign_keys = ON")
+    // #524: genuinely new databases were switched to auto_vacuum=FULL by the
+    // sqlite driver BEFORE WAL init. A legacy database keeps its NONE mode —
+    // converting one silently at startup would need a blocking full VACUUM —
+    // so it is only detected and surfaced softly here; conversion is the
+    // explicit user-triggered `opencode db vacuum --db <path>` command.
+    // Detect-only means detect-only: a failed readback degrades to a warning
+    // (the layer body is orDie'd, so an unhandled failure would kill startup),
+    // while an interruption is always re-raised.
+    const autoVacuum = yield* db.get<{ auto_vacuum: number }>(sql`PRAGMA auto_vacuum`).pipe(
+      Effect.catchCause((cause) =>
+        Cause.hasInterrupts(cause)
+          ? Effect.interrupt
+          : Effect.logWarning("database auto_vacuum readback failed — skipping the detect-only check", { cause }).pipe(
+              Effect.as(undefined),
+            ),
+      ),
+    )
+    if (autoVacuum?.auto_vacuum === 0)
+      yield* Effect.logWarning(
+        "database auto_vacuum is NONE — deleted pages stay allocated until converted; run `opencode db vacuum --db <path>` (prints its path with `opencode db path`)",
+      )
     yield* db.run("PRAGMA wal_checkpoint(PASSIVE)")
     yield* DatabaseMigration.apply(db)
 

packages/core/src/database/sqlite.bun.ts

@@ -161,11 +161,40 @@ const nativeLayer = (config: Config) =>
         create: config.create ?? true,
       })
       yield* Effect.addFinalizer(() => Effect.sync(() => native.close()))
+      // #524: auto_vacuum must be set BEFORE any WAL initialization — after
+      // WAL init the pragma silently yields NONE even on an empty database.
+      // Only a genuinely empty (0-page) file is eligible: on any existing
+      // database the pragma would be a no-op at best, so legacy NONE
+      // databases are never written here (startup stays detect-only; the
+      // explicit conversion lives in ./vacuum).
+      native.run("PRAGMA busy_timeout = 5000;")
+      if (config.readonly !== true) setAutoVacuumFull(native)
       if (config.disableWAL !== true) native.run("PRAGMA journal_mode = WAL;")
       return native
     }),
   )
 
+/**
+ * Setting auto_vacuum needs the write lock of a read-header-then-write
+ * upgrade, which SQLite fails with an immediate SQLITE_BUSY the busy handler
+ * cannot retry — a second opener racing the first one's initialization on the
+ * same new file hits it. Skipping on BUSY is safe: auto_vacuum is a
+ * persistent header property and every opener runs this pragma BEFORE any
+ * WAL/migration write, so whichever connection wins the first-write race sets
+ * FULL for the database.
+ */
+function setAutoVacuumFull(native: Database) {
+  const page = native.query<{ page_count: number }, []>("PRAGMA page_count").get()
+  if (!page || page.page_count !== 0) return
+  try {
+    native.run("PRAGMA auto_vacuum = FULL;")
+  } catch (cause) {
+    if (!isSqliteBusy(cause)) throw cause
+  }
+}
+
+const isSqliteBusy = (cause: unknown) => cause instanceof Error && /SQLITE_BUSY|database is locked/i.test(cause.message)
+
 const sqliteLayer = (config: Config) => Layer.effect(Client.SqlClient, make(config))
 
 const drizzleLayer = Layer.effect(

packages/core/src/database/sqlite.node.ts

@@ -156,11 +156,40 @@ const nativeLayer = (config: Config) =>
         open: true,
       })
       yield* Effect.addFinalizer(() => Effect.sync(() => native.close()))
+      // #524: auto_vacuum must be set BEFORE any WAL initialization — after
+      // WAL init the pragma silently yields NONE even on an empty database.
+      // On an existing non-empty database the pragma is a SQLite no-op, so
+      // legacy NONE databases are never converted here (startup stays
+      // detect-only; the explicit conversion lives in ./vacuum).
+      native.exec("PRAGMA busy_timeout = 5000;")
+      if (config.readonly !== true) setAutoVacuumFull(native)
       if (config.disableWAL !== true && config.readonly !== true) native.exec("PRAGMA journal_mode = WAL;")
       return native
     }),
   )
 
+/**
+ * Setting auto_vacuum needs the write lock of a read-header-then-write
+ * upgrade, which SQLite fails with an immediate SQLITE_BUSY the busy handler
+ * cannot retry — a second opener racing the first one's initialization on the
+ * same new file hits it. Skipping on BUSY is safe: auto_vacuum is a
+ * persistent header property and every opener runs this pragma BEFORE any
+ * WAL/migration write, so whichever connection wins the first-write race sets
+ * FULL for the database.
+ */
+function setAutoVacuumFull(native: DatabaseSync) {
+  const page: unknown = native.prepare("PRAGMA page_count").get()
+  const pageCount = typeof page === "object" && page !== null && "page_count" in page ? page.page_count : undefined
+  if (typeof pageCount !== "number" || pageCount !== 0) return
+  try {
+    native.exec("PRAGMA auto_vacuum = FULL;")
+  } catch (cause) {
+    if (!isSqliteBusy(cause)) throw cause
+  }
+}
+
+const isSqliteBusy = (cause: unknown) => cause instanceof Error && /SQLITE_BUSY|database is locked/i.test(cause.message)
+
 const sqliteLayer = (config: Config) => Layer.effect(Client.SqlClient, make(config))
 
 const drizzleLayer = Layer.effect(