feat(pebbleds): opt-in value separation

docs/changelogs/TBD.md

@@ -0,0 +1,48 @@
+# Kubo changelog vTBD
+
+<a href="https://ipshipyard.com/"><img align="right" src="https://github.com/user-attachments/assets/39ed3504-bb71-47f6-9bf8-cb9a1698f272" /></a>
+
+This release was brought to you by the [Shipyard](https://ipshipyard.com/) team.
+
+- [vTBD](#vtbd)
+
+## vTBD
+
+- [Overview](#overview)
+- [🔦 Highlights](#-highlights)
+  - [🧪 Opt-in value separation for pebbleds](#-opt-in-value-separation-for-pebbleds)
+- [📝 Changelog](#-changelog)
+- [👨‍👩‍👧‍👦 Contributors](#-contributors)
+
+### Overview
+
+### 🔦 Highlights
+
+#### 🧪 Opt-in value separation for pebbleds
+
+The experimental `pebbleds` datastore can now keep large values in separate
+append-only blob files instead of inside pebble's sstables: set
+`"valueSeparationEnabled": true` in the datastore spec. An IPFS block is a
+large value behind a small key, and the inline layout forces background
+compactions to rewrite unchanged block bytes and key-only scans (the
+listing phase of `ipfs repo gc` and `ipfs refs local`) to read through all
+block data. On a repo that keeps blocks in pebble (the `pebbleds` profile
+default), those scans ran 19-58x faster in synthetic benchmarks, random
+block reads 10-30% faster, and ordinary (non-fsync) writes 15-35% slower.
+Storage reclamation after deletes is also slower: pebble rewrites a blob
+file only once its share of dead bytes exceeds
+`valueSeparationTargetGarbageRatio` (default 0.20), so after `ipfs repo gc`
+disk usage falls gradually, and up to a fifth of blob-file bytes may stay
+unreclaimed.
+
+Value separation is experimental in pebble upstream and requires pebble
+database format `24` or newer; a repo that pinned an older
+`"formatMajorVersion"` must raise it first, which is irreversible. Read the
+precautions in
+[datastores.md](https://github.com/ipfs/kubo/blob/master/docs/datastores.md#value-separation-experimental)
+before enabling. Nothing changes for nodes that do not opt in; `flatfs`
+remains the default and the right choice for most users.
+
+### 📝 Changelog
+
+### 👨‍👩‍👧‍👦 Contributors

docs/datastores.md

@@ -114,15 +114,20 @@ If they are not configured (or assigned their zero-valued), then default values
 * `bytesPerSync`: int, Sync sstables periodically in order to smooth out writes to disk. (default: 512KB)
 * `disableWAL`: true|false, Disable the write-ahead log (WAL) at expense of prohibiting crash recovery. (default: false)
 * `cacheSize`: Size of pebble's shared block cache. (default: 8MB)
-* `formatVersionMajor`: int, Sets the format of pebble on-disk files. If 0 or unset, automatically convert to latest format.
+* `formatMajorVersion`: int, Sets the format of pebble on-disk files. If 0 or unset, automatically convert to latest format.
 * `l0CompactionThreshold`: int, Count of L0 files necessary to trigger an L0 compaction.
 * `l0StopWritesThreshold`: int, Limit on L0 read-amplification, computed as the number of L0 sublevels.
 * `lBaseMaxBytes`: int, Maximum number of bytes for LBase. The base level is the level which L0 is compacted into.
 * `maxConcurrentCompactions`: int, Maximum number of concurrent compactions. (default: 1)
 * `memTableSize`: int, Size of a MemTable in steady state. The actual MemTable size starts at min(256KB, MemTableSize) and doubles for each subsequent MemTable up to MemTableSize (default: 4MB)
 * `memTableStopWritesThreshold`: int, Limit on the number of queued of MemTables. (default: 2)
 * `walBytesPerSync`: int: Sets the number of bytes to write to a WAL before calling Sync on it in the background. (default: 0, no background syncing)
-* `walMinSyncSeconds`: int: Sets the minimum duration between syncs of the WAL. (default: 0)
+* `walMinSyncIntervalSeconds`: int: Sets the minimum duration between syncs of the WAL. (default: 0)
+* `valueSeparationEnabled`: true|false, Store large values in separate blob files instead of inside sstables. Experimental, see [Value separation](#value-separation-experimental) below before enabling. (default: false)
+* `valueSeparationMinimumSize`: int, Values at or above this size (in bytes) are stored in blob files. (default: 1024)
+* `valueSeparationMaxBlobReferenceDepth`: int, Limit on the number of overlapping blob files a single sstable may reference; lower values favor read locality over write amplification. (default: 10)
+* `valueSeparationRewriteMinimumAgeSeconds`: int, Minimum age of a blob file before it is eligible for a space-reclaiming rewrite. (default: 900)
+* `valueSeparationTargetGarbageRatio`: float in [0.0, 1.0], Fraction of dead bytes tolerated in blob files before pebble rewrites them to reclaim space; lower values reclaim space sooner at the cost of extra background writes. (default: 0.20)
 
 > [!TIP]
 > Start using pebble with only default values and configure tuning items are needed for your needs. For a more complete description of these values, see: `https://pkg.go.dev/github.com/cockroachdb/pebble@vA.B.C#Options` (where `A.B.C` is pebble version from Kubo's `go.mod`).
@@ -141,6 +146,59 @@ Without the `"formatMajorVersion"` in the pebble datastore config, the database
 
 When installing a new version of kubo when `"formatMajorVersion"` is configured, migration does not upgrade this to the latest available version. This is done because a user may have reasons not to upgrade the pebble database format, and may want to be able to downgrade kubo if something else is not working in the new version. If the configured pebble database format in the old kubo is not supported in the new kubo, then the configured version must be updated and the old kubo run, before installing the new kubo.
 
+#### Value separation (experimental)
+
+Set `"valueSeparationEnabled": true` and pebble writes values of at least
+`valueSeparationMinimumSize` to separate append-only blob files, keeping only
+keys and small references in the sstables. This matters when blocks live in
+pebble, as they do under the stock `pebbleds` profile: an IPFS block is a
+large value (typically 256KiB) behind a roughly 60-byte key, so with the
+inline layout every background compaction rewrites block bytes that never
+changed, and key-only scans read through all block data just to list the
+keys.
+
+With blob files, key-only scans skip block data and compactions move small
+references instead of block bytes. Listing all keys (the enumeration phase
+of `ipfs repo gc`, `ipfs refs local`, and `ipfs repo stat`) ran 19-58x
+faster in single-machine synthetic benchmarks (NVMe, ext4, 4KiB-256KiB
+blocks), and random 256KiB block reads 10-30% faster on default pebble
+settings (on a heavily tuned store, reads ranged from 10% slower to 26%
+faster). Ordinary (non-fsync) writes measured 15-35% lower throughput,
+because blocks still travel through the WAL and memtables; workloads that
+fsync every write showed no penalty in these runs. Gains grow with value
+size, so a repo of mostly small records has little to gain.
+
+Deleted values free disk space in the background: when the fraction of dead
+bytes in a blob file exceeds `valueSeparationTargetGarbageRatio` and the
+file is older than `valueSeparationRewriteMinimumAgeSeconds`, pebble
+rewrites it. After `ipfs repo gc`, expect disk usage to fall gradually
+rather than immediately, and expect blob files to keep up to that fraction
+of dead bytes indefinitely. Lower the ratio to reclaim more space at the
+cost of extra background writes.
+
+> [!WARNING]
+> Read these precautions before enabling:
+>
+> * Pebble marks this feature experimental
+>   (`Options.Experimental.ValueSeparationPolicy`). Only enable it on repos
+>   you can rebuild or restore from a backup.
+> * Blob files require pebble database format `24`
+>   (`pebble.FormatValueSeparation`) or newer. If your config pins an older
+>   `"formatMajorVersion"`, kubo refuses to start until you raise it, and
+>   raising it is **irreversible**: older versions of pebble (and therefore
+>   older versions of kubo) cannot open the database afterwards. See
+>   [Use of `formatMajorVersion`](#use-of-formatmajorversion) above.
+> * Enabling affects only new writes. Existing values move to blob files as
+>   regular compactions rewrite them, so benefits on old data appear over
+>   time, not at restart.
+> * Disabling later is safe: existing blob files stay readable and fold back
+>   into sstables through future compactions. The database format stays at
+>   `24` or newer either way, so disabling does not restore downgrade
+>   compatibility.
+> * A `valueSeparation*` option set to `0` (or omitted) uses the default
+>   listed above. For the most aggressive space reclamation, use a small
+>   non-zero `valueSeparationTargetGarbageRatio` such as `0.05`, not `0`.
+
 ## badgerds
 
 Uses [badger](https://github.com/dgraph-io/badger) as a key-value store.

plugin/plugins/pebbleds/pebbleds.go

@@ -18,6 +18,19 @@ var Plugins = []plugin.Plugin{
 	&pebbledsPlugin{},
 }
 
+// Defaults for pebble's experimental value separation, applied when
+// "valueSeparationEnabled" is true and the corresponding option is unset.
+// Pebble has no defaults of its own here (a nil policy means disabled), so
+// these are Kubo's choices: separate everything above the size of a typical
+// small DAG node, and follow the reference values used in pebble's own
+// ValueSeparationPolicy documentation for the rest.
+const (
+	DefaultValueSeparationMinimumSize           = 1024
+	DefaultValueSeparationMaxBlobReferenceDepth = 10
+	DefaultValueSeparationRewriteMinimumAge     = 15 * time.Minute
+	DefaultValueSeparationTargetGarbageRatio    = 0.20
+)
+
 type pebbledsPlugin struct{}
 
 var _ plugin.PluginDatastore = (*pebbledsPlugin)(nil)
@@ -109,6 +122,29 @@ func (*pebbledsPlugin) DatastoreConfigParser() fsrepo.ConfigFromMap {
 		if err != nil {
 			return nil, err
 		}
+		valueSepEnabled, err := getConfigBool("valueSeparationEnabled", params)
+		if err != nil {
+			return nil, err
+		}
+		valueSepMinSize, err := getConfigInt("valueSeparationMinimumSize", params)
+		if err != nil {
+			return nil, err
+		}
+		valueSepMaxRefDepth, err := getConfigInt("valueSeparationMaxBlobReferenceDepth", params)
+		if err != nil {
+			return nil, err
+		}
+		valueSepRewriteAgeSec, err := getConfigInt("valueSeparationRewriteMinimumAgeSeconds", params)
+		if err != nil {
+			return nil, err
+		}
+		valueSepGarbageRatio, err := getConfigFloat("valueSeparationTargetGarbageRatio", params)
+		if err != nil {
+			return nil, err
+		}
+		if !valueSepEnabled && (valueSepMinSize != 0 || valueSepMaxRefDepth != 0 || valueSepRewriteAgeSec != 0 || valueSepGarbageRatio != 0) {
+			return nil, fmt.Errorf("valueSeparation* options require \"valueSeparationEnabled\": true")
+		}
 
 		if formatMajorVersion == 0 {
 			// Pebble DB format not configured. Automatically ratchet the
@@ -141,6 +177,50 @@ func (*pebbledsPlugin) DatastoreConfigParser() fsrepo.ConfigFromMap {
 			}
 		}
 
+		if valueSepEnabled {
+			// Blob files are only readable at FormatValueSeparation or
+			// newer. Refuse to silently ratchet a repo that pinned an
+			// older format: raising formatMajorVersion is irreversible
+			// and must stay an explicit user decision.
+			if formatMajorVersion < pebble.FormatValueSeparation {
+				return nil, fmt.Errorf(
+					"valueSeparationEnabled requires \"formatMajorVersion\" >= %d (currently %d); raising it is IRREVERSIBLE, see docs/datastores.md#pebbleds",
+					int(pebble.FormatValueSeparation), int(formatMajorVersion))
+			}
+			if valueSepMinSize < 0 || valueSepMaxRefDepth < 0 || valueSepRewriteAgeSec < 0 {
+				return nil, fmt.Errorf("valueSeparation* options must not be negative")
+			}
+			if valueSepGarbageRatio < 0 || valueSepGarbageRatio > 1 {
+				return nil, fmt.Errorf("\"valueSeparationTargetGarbageRatio\" must be within [0.0, 1.0]")
+			}
+			if valueSepMinSize == 0 {
+				valueSepMinSize = DefaultValueSeparationMinimumSize
+			}
+			if valueSepMaxRefDepth == 0 {
+				valueSepMaxRefDepth = DefaultValueSeparationMaxBlobReferenceDepth
+			}
+			rewriteAge := DefaultValueSeparationRewriteMinimumAge
+			if valueSepRewriteAgeSec != 0 {
+				rewriteAge = time.Duration(valueSepRewriteAgeSec) * time.Second
+			}
+			if valueSepGarbageRatio == 0 {
+				valueSepGarbageRatio = DefaultValueSeparationTargetGarbageRatio
+			}
+			if c.pebbleOpts == nil {
+				c.pebbleOpts = &pebble.Options{FormatMajorVersion: formatMajorVersion}
+			}
+			policy := pebble.ValueSeparationPolicy{
+				Enabled:               true,
+				MinimumSize:           valueSepMinSize,
+				MaxBlobReferenceDepth: valueSepMaxRefDepth,
+				RewriteMinimumAge:     rewriteAge,
+				TargetGarbageRatio:    valueSepGarbageRatio,
+			}
+			// The policy is ignored unless Experimental.EnableColumnarBlocks
+			// is true; go-ds-pebble calls EnsureDefaults, which sets it.
+			c.pebbleOpts.Experimental.ValueSeparationPolicy = func() pebble.ValueSeparationPolicy { return policy }
+		}
+
 		return &c, nil
 	}
 }
@@ -174,6 +254,22 @@ func getConfigInt(name string, params map[string]any) (int, error) {
 	return 0, nil
 }
 
+func getConfigFloat(name string, params map[string]any) (float64, error) {
+	val, ok := params[name]
+	if ok {
+		fval, ok := val.(float64)
+		if !ok {
+			ival, ok := val.(int)
+			if !ok {
+				return 0, fmt.Errorf("%q field was not a float64 or an integer", name)
+			}
+			return float64(ival), nil
+		}
+		return fval, nil
+	}
+	return 0, nil
+}
+
 func (c *datastoreConfig) DiskSpec() fsrepo.DiskSpec {
 	return map[string]any{
 		"type": "pebbleds",

plugin/plugins/pebbleds/pebbleds_test.go

@@ -0,0 +1,127 @@
+package pebbleds
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/cockroachdb/pebble/v2"
+)
+
+func parseConfig(t *testing.T, params map[string]any) (*datastoreConfig, error) {
+	t.Helper()
+	p := &pebbledsPlugin{}
+	cfg, err := p.DatastoreConfigParser()(params)
+	if err != nil {
+		return nil, err
+	}
+	return cfg.(*datastoreConfig), nil
+}
+
+func TestValueSeparationDisabledByDefault(t *testing.T) {
+	c, err := parseConfig(t, map[string]any{
+		"type": "pebbleds",
+		"path": "pebbleds",
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if c.pebbleOpts != nil && c.pebbleOpts.Experimental.ValueSeparationPolicy != nil {
+		if c.pebbleOpts.Experimental.ValueSeparationPolicy().Enabled {
+			t.Fatal("value separation must be disabled unless requested")
+		}
+	}
+}
+
+func TestValueSeparationEnabledDefaults(t *testing.T) {
+	c, err := parseConfig(t, map[string]any{
+		"type":                   "pebbleds",
+		"path":                   "pebbleds",
+		"valueSeparationEnabled": true,
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if c.pebbleOpts == nil || c.pebbleOpts.Experimental.ValueSeparationPolicy == nil {
+		t.Fatal("expected a value separation policy to be configured")
+	}
+	got := c.pebbleOpts.Experimental.ValueSeparationPolicy()
+	want := pebble.ValueSeparationPolicy{
+		Enabled:               true,
+		MinimumSize:           DefaultValueSeparationMinimumSize,
+		MaxBlobReferenceDepth: DefaultValueSeparationMaxBlobReferenceDepth,
+		RewriteMinimumAge:     DefaultValueSeparationRewriteMinimumAge,
+		TargetGarbageRatio:    DefaultValueSeparationTargetGarbageRatio,
+	}
+	if got != want {
+		t.Fatalf("policy = %+v, want %+v", got, want)
+	}
+	if c.pebbleOpts.FormatMajorVersion < pebble.FormatValueSeparation {
+		t.Fatalf("format %d does not support value separation", c.pebbleOpts.FormatMajorVersion)
+	}
+}
+
+func TestValueSeparationEnabledCustomValues(t *testing.T) {
+	// JSON numbers arrive as float64, mimic that.
+	c, err := parseConfig(t, map[string]any{
+		"type":                                 "pebbleds",
+		"path":                                 "pebbleds",
+		"valueSeparationEnabled":               true,
+		"valueSeparationMinimumSize":           float64(4096),
+		"valueSeparationMaxBlobReferenceDepth": float64(5),
+		"valueSeparationRewriteMinimumAgeSeconds": float64(60),
+		"valueSeparationTargetGarbageRatio":       0.5,
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	got := c.pebbleOpts.Experimental.ValueSeparationPolicy()
+	want := pebble.ValueSeparationPolicy{
+		Enabled:               true,
+		MinimumSize:           4096,
+		MaxBlobReferenceDepth: 5,
+		RewriteMinimumAge:     60 * time.Second,
+		TargetGarbageRatio:    0.5,
+	}
+	if got != want {
+		t.Fatalf("policy = %+v, want %+v", got, want)
+	}
+}
+
+func TestValueSeparationRequiresSupportedFormat(t *testing.T) {
+	_, err := parseConfig(t, map[string]any{
+		"type":                   "pebbleds",
+		"path":                   "pebbleds",
+		"valueSeparationEnabled": true,
+		"formatMajorVersion":     float64(pebble.FormatValueSeparation - 1),
+	})
+	if err == nil {
+		t.Fatal("expected error for formatMajorVersion below FormatValueSeparation")
+	}
+	if !strings.Contains(err.Error(), "formatMajorVersion") {
+		t.Fatalf("error should mention formatMajorVersion, got: %v", err)
+	}
+}
+
+func TestValueSeparationOptionsRequireEnabled(t *testing.T) {
+	_, err := parseConfig(t, map[string]any{
+		"type":                       "pebbleds",
+		"path":                       "pebbleds",
+		"valueSeparationMinimumSize": float64(4096),
+	})
+	if err == nil {
+		t.Fatal("expected error for valueSeparation* option without valueSeparationEnabled")
+	}
+}
+
+func TestValueSeparationGarbageRatioValidation(t *testing.T) {
+	_, err := parseConfig(t, map[string]any{
+		"type":                              "pebbleds",
+		"path":                              "pebbleds",
+		"valueSeparationEnabled":            true,
+		"valueSeparationTargetGarbageRatio": 1.5,
+	})
+	if err == nil {
+		t.Fatal("expected error for garbage ratio > 1.0")
+	}
+}