Skip to content

feat(opencode): add db stats, prune, and vacuum CLI commands - #43456

Open
AndyS77 wants to merge 4 commits into
anomalyco:devfrom
AndyS77:db-cleanup
Open

feat(opencode): add db stats, prune, and vacuum CLI commands#43456
AndyS77 wants to merge 4 commits into
anomalyco:devfrom
AndyS77:db-cleanup

Conversation

@AndyS77

@AndyS77 AndyS77 commented Aug 19, 2026 •
edited
Loading

Copy link
Copy Markdown

Issue for this PR

Partially addresses #33356

Type of change

  • Bug fix
  • Refactor / code improvement

What does this PR do?

VACUUM INTO + file swap (replaces plain VACUUM):

Plain VACUUM in WAL mode can write a file comparable to the database size into the WAL before completing. On a 104 GB database, VACUUM wrote ~56 GB into the WAL within 45 minutes without completing (see #33356 comment).

The db vacuum command now uses a two-phase approach:

  1. PRAGMA wal_checkpoint(TRUNCATE) — flush WAL into main database
  2. VACUUM INTO '<path>.vacuum' — create compacted copy to separate file
  3. Close connection, atomic file swap, clean up WAL/SHM files
  4. Reopen and verify with page count comparison

Retention-based pruning (--max-age <days>):

Orphan-only pruning reclaims almost nothing on real-world databases where bloat is dominated by events for sessions users still have in their list. After deleting all sessions older than 30 days on the 104 GB database, 85 GB of events remained for live sessions.

The db prune command now supports --max-age <days> to delete sessions older than N days (based on time_updated), then cascades through pruneOrphanedEvents to clean up their events.

Removed --vacuum flag from prune: Running VACUUM inside the prune handler was unsafe (same WAL blow-up risk). Users should run opencode db vacuum as a separate step after pruning.

How did you verify your code works?

  • dbStats test: verifies table row counts, page count, page size, freelist count
  • pruneOrphanedEvents tests: verifies orphaned events are deleted, live session events are preserved
  • pruneOldSessions tests: verifies sessions older than max-age are deleted with their events, recent sessions are preserved
  • bun turbo typecheck --concurrency=1 --filter=opencode: passes clean
  • Code reviewed by @code-reviewer (previous commits)

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

AndyS77 and others added 3 commits August 19, 2026 16:36
@AndyS77 zai-glm-52
test(opencode): db maintenance CLI tests - stats, prune, vacuum
a091662 
Refs BUG-33356

Co-Authored-By: zai-glm-52 <noreply@ai.local>
Agent: @Feature-dev
Scope: BUG-33356
@AndyS77
feat(opencode): add db stats, prune, and vacuum CLI commands
319f6c8 
- db stats: shows table row counts, database size, freelist pages
- db prune: removes orphaned events for deleted sessions (--dry-run, --vacuum)
- db vacuum: reclaims free space with VACUUM and reports reclaimed MB
- auto_vacuum = INCREMENTAL enabled for new databases
- Exports dbStats() and pruneOrphanedEvents() for programmatic use

Refs BUG-33356
@AndyS77
fix(opencode): address code review findings for db maintenance
90b029e 
- Fix pruneOrphanedEvents: use same orphan criteria for event_sequence
  deletion (LEFT JOIN session) instead of NOT IN (SELECT FROM event)
  which could delete valid sequence entries for live sessions
- Fix VacuumCommand: use PRAGMA page_size instead of hardcoded 4096
- Remove unnecessary @opentelemetry/sdk-trace-web dev dependency

Refs BUG-33356
@github-actions github-actions Bot added the needs:compliance This means the issue will auto-close after 2 hours. label Aug 19, 2026
@AndyS77 AndyS77 mentioned this pull request Aug 19, 2026
Open
@github-actions github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Aug 19, 2026
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Thanks for updating your PR! It now meets our contributing guidelines. 👍

renekris added a commit to renekris/opencode-lowmem that referenced this pull request Aug 26, 2026
@renekris
feat(opencode): db stats and vacuum subcommands (port of upstream ano…
ea44d23 
…malyco#43456, partial)

`opencode db stats` reports page metadata, freelist, db/WAL sizes and
per-table row counts with approximate payload bytes over a read-only
connection (no AppRuntime — its shared layer opens the db writable).
`opencode db vacuum` is an offline explicit VACUUM behind a liveness
guard: on Linux a best-effort /proc scan for open db/WAL/SHM handles
plus an exclusive lock; elsewhere the exclusive lock only, with the
reduced guarantee printed. locking_mode=EXCLUSIVE is taken before the
guard transaction so the lock persists through COMMIT into VACUUM.
Not ported from anomalyco#43456: orphan prune, auto_vacuum.

Upstream anomalyco#43456 by AndyS77; adapted (low-level wrapper, lock
sequencing, proofs) for the fork.

Tests: stats/vacuum on real SQLite fixtures including WAL lock-interval
regression, live-process refusal, and reclaim-on-bloat.

README: ported row for anomalyco#43456 (partial); customs rows for durable
paging and the background-job terminal ring.
renekris added a commit to renekris/opencode-lowmem that referenced this pull request Aug 26, 2026
@renekris
chore(build): hard-fail on shim mismatch; overhaul fork README
1a54eb5 
fork-build.sh now exits 1 when the live shim does not point at the fresh
binary unless OPENCODE_FORK_BUILD_ALLOW_SHIM_MISMATCH=1 is set, and asserts
the shim reports the stamped version after install. README gains the
anomalyco#43456 partial-port row (AndyS77), durable-paging and terminal-ring
customs rows, the complete 17-knob inventory, the v1.18.22 base, corrected
author credits, and an honest deferred-ports section (anomalyco#40698 is deferred
for scope, not a missing seam).
@romanilyin romanilyin mentioned this pull request Aug 28, 2026
Open
6 tasks
@heisenberg600 heisenberg600 mentioned this pull request Sep 6, 2026
Open
6 tasks
@Yuepixel Yuepixel mentioned this pull request Sep 10, 2026
Open
@AndyS77 opencode
fix(opencode): use VACUUM INTO + file swap and add --max-age pruning
37b0d71 
Plain VACUUM in WAL mode can write a file comparable to the database size
into the WAL before completing. On large databases (100+ GB) this causes
the WAL to blow up and the operation to never finish.

Replace plain VACUUM with VACUUM INTO + atomic file swap:
  1. PRAGMA wal_checkpoint(TRUNCATE) — flush WAL into main DB
  2. VACUUM INTO '<path>.vacuum' — create compacted copy
  3. Close connection, swap files, clean up WAL/SHM

Add --max-age <days> option to 'db prune' for retention-based pruning of
live sessions. Orphan-only pruning reclaims almost nothing on real-world
databases where bloat is dominated by events for sessions users still have
in their list (anomalyco#33356).

Remove --vacuum flag from prune (separate 'db vacuum' command is safer).

Refs anomalyco#33356

Co-Authored-By: opencode <opencode@anomaly.co>
Agent: @bug-fix
Model: msp-crow/zai-glm-52
Scope: anomalyco#33356
@AndyS77

AndyS77 commented Sep 16, 2026

Copy link
Copy Markdown
Author

Updated based on community data from #33356 (#33356 (comment)):

1. VACUUM INTO + file swap — plain VACUUM in WAL mode causes WAL blow-up on large databases. Now uses VACUUM INTO to a separate file, then atomic swap after closing the connection.

2. --max-age pruning — orphan-only pruning reclaims almost nothing on real-world databases where bloat is dominated by events for live sessions. Added --max-age option to prune sessions older than N days.

3. Removed --vacuum from prune — VACUUM inside the prune handler was unsafe. Users should run 'opencode db vacuum' separately.

Also related: #41711 (feat(storage): add safe database maintenance controls) is a more comprehensive approach that explicitly references this PR as complementary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@AndyS77