[WIP] Add custom compaction scheduler with space amplification trigger#189
Draft
[WIP] Add custom compaction scheduler with space amplification trigger#189
Conversation
SlateDB's default Size-Tiered Compaction doesn't clean up tombstones fast enough for Silo's high-churn workload where task keys are created and deleted within minutes, causing ~96% CPU from scanning accumulated tombstones. This adds a custom CompactionScheduler that extends STC with a space amplification safety valve (inspired by RocksDB's Universal Compaction): when non-bottom-run data exceeds 25% of the bottom run, a full compaction is forced to drop tombstones. Key changes: - Custom scheduler with min_compaction_sources=2 (vs STC's 4) and configurable space amplification threshold (default 25%) - Three new Prometheus gauges per shard: l0_sst_count, sorted_run_count, space_amplification_percent (updated every ~5s in the reaper loop) - WebUI shard page shows space amp % with color-coded health indicators (green/amber/red thresholds) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
airhorns
force-pushed
the
custom-compaction
branch
from
d704326 to
7af3dae
Compare
airhorns
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CompactionSchedulerthat extends size-tiered compaction with a space amplification trigger: when non-bottom-run data exceeds 25% of the bottom run, forces a full compaction to drop tombstonesmin_compaction_sources=2(vs default 4)silo_shard_l0_sst_count,silo_shard_sorted_run_count,silo_shard_space_amplification_percentTest plan
cargo test --test compaction_scheduler_tests— 8 new integration testscargo test --test compact_shard_tests— existing compaction tests passcargo test --test shard_cleanup_tests— cleanup tests passcargo clippy— no new warnings🤖 Generated with Claude Code
Note
Medium Risk
Changes core SlateDB compaction scheduling and wiring, which can materially affect write/read amplification and storage usage under load. Also adds periodic manifest reads for metrics/UI, which could introduce overhead or new failure modes if misused.
Overview
Adds a new custom SlateDB
CompactionScheduler(SiloCompactionScheduler) that keeps size-tiered compaction behavior but forces a full compaction when space amplification exceeds a configurable threshold (default 25%), and makes L0 compaction more aggressive (defaultmin_compaction_sources=2). The scheduler is wired into shard DB creation and includesvalidate/generatehandling plus unit tests for option parsing and space-amp calculation.Exposes compaction health via a new
LsmState.space_amplification_percent, three new per-shard Prometheus gauges, and a server-side periodic poll (~5s) to update them. The WebUI shard detail page now displays space amplification with color-coded health and updated warning thresholds.Written by Cursor Bugbot for commit 7af3dae. This will update automatically on new commits. Configure here.