Fabro SlateDB WAL compaction never triggered after run deletion — stale WAL files leak indefinitely

[burhanyurur]

---

type: bug-report
id: TASK003005
title: Fabro SlateDB WAL compaction never triggered after run deletion — stale WAL files leak indefinitely
description: >
Fabro v0.260.0-nightly (build a84bbfb) stores workflow run state in an embedded SlateDB
database at ~/.fabro/storage/objects/slatedb/. When fabro rm deletes a run, it writes
a tombstone record to the SlateDB WAL and removes the run from the in-memory index, but
NEVER triggers a SlateDB compaction. The original WAL segments containing the deleted run
data persist forever.

Simultaneously, fabro system prune only scans the active run index to find deletion
candidates. Since deleted runs are already removed from that index, prune reports
count=0 bytes=0 every time — it has no path to discover orphaned WAL segments.

Observed impact on this machine: 1,983 stale WAL files consuming 8.4 MB accumulated from
deleted runs. A manual SlateDB reset (stop server, rm -rf slatedb/, restart server) was
applied as a workaround on 2026-06-23, reclaiming the space.

Additionally, fabro system df and fabro system info disagree on run counts (df shows
runs the info index does not), confirming the index/storage inconsistency.
code_locations:

• path: ~/.fabro/storage/objects/slatedb/wal/
  language: n/a (SlateDB WAL segments)
• path: ~/.fabro/storage/objects/slatedb/manifest/
  language: n/a (SlateDB manifest)
• path: ~/.fabro/storage/logs/server.log
  language: n/a (server log — no errors logged during rm or prune)
• path: ~/.fabro/logs/cli.2026-06-23.log
  language: n/a (CLI log — shows pruning runs count=0 bytes=0 dry_run=true)
  impact: >
• Every fabro rm invocation leaks WAL files — disk usage grows without bound
• fabro system prune is a no-op — reports count=0 even with thousands of stale segments
• fabro system info reports 0 runs while fabro system df shows 1 run — index/storage mismatch
• fabro system repair runs reports "No run repair issues found" — false negative
• Operators have no built-in way to reclaim leaked WAL storage
• Manual workaround (stop server, nuke SlateDB, restart) is destructive and loses all run history
  remediation_guidance:
• Root cause is in the Fabro server binary
  Options in priority order:
  1. File an upstream bug report against Fabro with the SlateDB WAL leak reproduction steps
  2. If Fabro source is accessible: fix rm to trigger slatedb.compact() after tombstone write;
     fix prune to scan SlateDB directly for orphaned segments, not just the active run index;
     fix repair runs to detect the index/storage count mismatch
  3. As a local mitigation, add a cron/scheduled task that periodically runs the SlateDB reset
     workaround (stop server → rm slatedb/ → start server) during maintenance windows
  4. Add a CI health check that alerts when SlateDB WAL file count exceeds a threshold (e.g. >100)
• Temporary workaround (already applied once):

  fabro server stop
  rm -rf ~/.fabro/storage/objects/slatedb
  fabro server start

status: pending
created_at: 2026-06-23

[swerner]

Thanks for digging into this @burhanyurur. SlateDB doesn't reclaim WAL files immediately after a run is deleted. By default, it flushes accumulated data after roughly 64MB or 4096 WAL flushes, after that obsolete WALs can be garbage collected. At 1,983 / 8.4 MB, you'd be below those thresholds.

Have you seen it exceed 4,096 WAL files or continue growing after crossing the size threshold and waiting at least five minutes? If so, could you share just the command sequence needed to reproduce it?

Did the system df / system info mismatch appear after manually removing SlateDB? That could explain it. I wouldn't recommend automating that reset because it also deletes durable run history.