feat: private-content caching opt-in with fleet-wide delete purge

USER-GUIDE.md

@@ -557,7 +557,8 @@ Runtime settings are stored in the database and take effect immediately without
 | `download_cache_max_object_bytes` | `67108864` (64 MiB) | Per-object ceiling for the download cache: larger files are never cached and always stream through the temp path. Bounds: **1–2^40** |
 | `download_cache_min_uses` | `1` | How many times an object must be requested (per instance) before its bytes are cached. `1` caches on first download; raising it keeps one-hit-wonders from displacing hot content. Bounds: **1–100** |
 | `download_cache_inactive_secs` | `0` (off) | Inactivity expiry for cached objects: entries not **accessed** within this window are deleted regardless of the size budget. Independent pruning axis — also bounds how long cached plaintext can linger on an instance's disk. Example: `604800` = 7 days. Bounds: **0–315360000** |
-| `download_cache_seed_on_upload` | `true` | Seed the download cache from a **public** upload's staged bytes the moment its network store succeeds, so publish-then-read is served from local disk with no network fetch (a rename — zero extra I/O). Applies where uploads are processed — the writer (or the whole instance in an all-in-one deployment); on a reader fleet each reader still warms by read-through. Seeds respect the size ceiling and byte budget but deliberately skip `download_cache_min_uses`; the sweeper evicts wrong bets. Disable for archive-shaped workloads (upload-heavy, rarely re-read) |
+| `download_cache_seed_on_upload` | `true` | Seed the download cache from an upload's staged bytes (public; private too when `download_cache_private` is on) the moment its network store succeeds, so publish-then-read is served from local disk with no network fetch (a rename — zero extra I/O). Applies where uploads are processed — the writer (or the whole instance in an all-in-one deployment); on a reader fleet each reader still warms by read-through. Seeds respect the size ceiling and byte budget but deliberately skip `download_cache_min_uses`; the sweeper evicts wrong bets. Disable for archive-shaped workloads (upload-heavy, rarely re-read) |
+| `download_cache_private` | `false` | Allow **private** uploads into the download cache (read-through and seeding). Off by default because it keeps decrypted private content on every caching instance's disk — a new at-rest surface. With it on, deleting an upload purges the handling instance synchronously and every other instance within about one sweep tick (~60s) via the purge log. Turning it back off removes already-cached private files fleet-wide (next tick online, boot reconciliation for instances that were down); see the [download-cache deployment guide](docs/guides/download-cache.md) for the exact contract |
 
 For the `download_cache_*` family, the [download-cache deployment guide](docs/guides/download-cache.md) covers sizing, the stats telemetry, per-setting fleet scope, and the privacy posture.
 

cmd/indelible/main.go

@@ -188,6 +188,15 @@ func main() {
 	// upload-queue dequeue, hash-chain writes). Defers here are function-scoped,
 	// so they still fire on shutdown.
 	if cfg.WorkersEnabled {
+		// One-time Go-side backfill of uploads.cache_key (V2-873 migration
+		// 013): the key is a Go-side digest, so SQL couldn't fill it. Writer
+		// singleton, idempotent, cheap once caught up.
+		if n, err := services.NewUploadService(db).BackfillCacheKeys(); err != nil {
+			slog.Warn("cache_key backfill incomplete; boot reconciliation may re-warm some entries", "stamped", n, "error", err)
+		} else if n > 0 {
+			slog.Info("backfilled uploads.cache_key", "rows", n)
+		}
+
 		uploadWorker := worker.NewUploadWorker(db, cfg, dlCache)
 		uploadWorker.Start()
 		defer uploadWorker.Stop()

docs/guides/download-cache.md

@@ -67,7 +67,8 @@ Every instance owns its cache — there is no shared cache tier, by design (shar
 
 Cached entries are **plaintext bytes on the instance's disk**, held outside the audit surface of the upload store. That is why:
 
-- **Only public uploads are cached.** Private content would put plaintext where a disk snapshot, misconfigured backup, or shared volume could expose it, and a deleted-and-shredded upload must not survive in a cache copy. (Whether a private-content opt-in should ever exist — with the fleet purge propagation it would need — is tracked separately as V2-873.)
+- **Public uploads are cached unconditionally; private uploads only behind `download_cache_private`** (default off). Private caching puts decrypted plaintext where a disk snapshot, misconfigured backup, or shared volume could expose it — enabling it is an explicit operator decision, and cache volumes then deserve the same care as the database and temp-upload disks. Authorization is unaffected: the cache is only consulted after the row/owner/visibility checks, so a cache hit never bypasses the token gate. **Turning the opt-in off removes already-cached private plaintext fleet-wide**, not just future admissions: online instances purge on their next sweep tick, and an instance that was down applies the policy during its boot reconciliation.
 - Cache files are owner-only (`0600`) under `<data_dir>/cache/objects`, named by content digest — the name never reveals the DataMap (which is the retrieval capability), and the digest is domain-separated from the plaintext hash.
 - `download_cache_inactive_secs` doubles as the bound on how long unread cached plaintext can linger; budget `0` (or the env override `0`) drains an instance completely.
-- **Deleting an upload purges its cached copy synchronously on the instance that handles the delete** (V2-824): the purge runs before the record is removed, and a purge that cannot unlink fails the delete rather than reporting a deletion while the plaintext remains readable. A download already in flight when the delete lands finishes streaming (its descriptor outlives the unlink) and any promotion it makes is taken back out. That take-back is best-effort: in the doubly-degraded case (a promotion raced the delete AND the final unlink failed) the leftover bytes are not servable — the deleted record 404s before the cache is consulted — and fall to eviction/inactivity; durable purge retry arrives with the fleet purge log (V2-873). On a reader fleet, other instances' cached copies of a deleted *public* upload are unreachable immediately (the record is gone) and age out by eviction/inactivity — acceptable for public bytes, and exactly why private content is not cached.
+- **Deleting an upload purges its cached copy synchronously on the instance that handles the delete** (V2-824): the purge runs before the record is removed, and a purge that cannot unlink fails the delete rather than reporting a deletion while the plaintext remains readable. A download already in flight when the delete lands finishes streaming (its descriptor outlives the unlink) and any promotion it makes is taken back out. That take-back is best-effort in the instant, but the purge log below retries it durably.
+- **Deletes propagate to the whole fleet through the purge log.** The delete appends the upload's cache keys to `cache_purge_log` **in the same transaction** as the row removal (no interleaving can observe the row live with the log entry missing, or vice versa); every instance's sweep worker consumes the log tail each tick and unlinks local copies. A key whose unlink fails is retried from the log next tick without delaying later purges (they are applied the same tick; only the high-water mark waits). Each instance also runs a **full liveness reconciliation** of its cache directory against live upload rows (`uploads.cache_key`) at boot and every 12 hours — the guaranteed backstop that covers deletes that happened while it was down, purge-log rows pruned past the 7-day retention, and any unlink that stayed stuck across ticks. **The purge-window contract:** the instance handling the delete purges synchronously (the API fails rather than report a deletion with the plaintext still readable); every other instance purges within about one sweep tick (~60s). If database read-replicas are ever introduced, replica lag adds to that window. On a reader fleet, other instances' cached copies of a deleted *public* upload are unreachable immediately (the record is gone) and age out by eviction/inactivity — acceptable for public bytes, and exactly why private content is not cached.

docs/guides/scaling.md

@@ -103,6 +103,7 @@ Fleet notes:
 - The cache is **per instance** — every reader warms and evicts its own copy from its own traffic. With no load-balancer affinity, popular objects end up replicated on every reader; at typical working-set sizes (small hot objects) that is the right trade, and it is what sizing the budget per instance assumes.
 - The `download_cache_*` settings are **fleet-global values applied per instance** (readers share the writer's database). For a fleet with **different disk sizes per reader**, set the env override `INDELIBLE_DOWNLOAD_CACHE_MAX_BYTES` on the instances that differ — env beats the DB setting on that instance only, and `0` disables its cache outright.
 - The cache can never be the reason uploads pause: a background sweeper evicts it — aggressively, toward empty if necessary — as soon as the data volume approaches the disk-alert worker's critical threshold, and otherwise keeps it under budget by LRU with an optional inactivity window (`download_cache_inactive_secs`).
+- **Deletes reach every reader's cache within about a minute.** The delete appends purge keys to a shared `cache_purge_log`; each instance's sweep worker applies them on its 1-minute tick, and boot reconciliation covers instances that were down. Relevant mainly with `download_cache_private` on — a deleted *public* upload's cached bytes are unreachable immediately either way (the record is gone).
 - **Upload seeding is writer-local.** `download_cache_seed_on_upload` (default on) promotes a public upload's staged bytes into the cache when the store succeeds — but uploads only pass through the writer, so on a role-split fleet this warms the **writer's** cache, not the readers'. Readers warm by read-through: the first download per reader is still a cold fetch. The full publish-then-read win applies to all-in-one deployments, where writer and reader are the same process.
 
 ## Load balancer notes

internal/database/migrations/postgres/013_download_cache_purge.sql

@@ -0,0 +1,33 @@
+-- +goose Up
+
+-- V2-873: fleet-wide download-cache purge propagation.
+--
+-- uploads.cache_key is the serve-path cache key (KeyForIdentifier over
+-- data_map when present, else datamap_address — the same preference order
+-- downloadETag uses), set when an upload reaches a terminal stored status and
+-- backfilled at writer boot for pre-existing rows. Cache keys are one-way
+-- digests, so without this column no instance can map its cached files back
+-- to upload rows (needed by boot reconciliation).
+ALTER TABLE uploads ADD COLUMN cache_key TEXT;
+CREATE INDEX idx_uploads_cache_key ON uploads(cache_key);
+
+-- cache_purge_log is the delete fan-out: DeleteUpload's service half appends
+-- the deleted upload's cache keys here (insert-first, so a failed delete can
+-- at worst cause a spurious purge that re-warms), and every instance's cache
+-- sweep worker consumes the tail each tick, dropping local entries. Rows are
+-- pruned after a retention window by writer-role instances only (reader
+-- discipline, V2-514); instances offline longer than the retention are
+-- covered by boot reconciliation against uploads.cache_key.
+CREATE TABLE cache_purge_log (
+    id BIGSERIAL PRIMARY KEY,
+    cache_key TEXT NOT NULL,
+    deleted_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX idx_cache_purge_log_deleted_at ON cache_purge_log(deleted_at);
+
+-- +goose Down
+
+DROP TABLE cache_purge_log;
+DROP INDEX idx_uploads_cache_key;
+ALTER TABLE uploads DROP COLUMN cache_key;

internal/database/migrations/sqlite/013_download_cache_purge.sql

@@ -0,0 +1,33 @@
+-- +goose Up
+
+-- V2-873: fleet-wide download-cache purge propagation.
+--
+-- uploads.cache_key is the serve-path cache key (KeyForIdentifier over
+-- data_map when present, else datamap_address — the same preference order
+-- downloadETag uses), set when an upload reaches a terminal stored status and
+-- backfilled at writer boot for pre-existing rows. Cache keys are one-way
+-- digests, so without this column no instance can map its cached files back
+-- to upload rows (needed by boot reconciliation).
+ALTER TABLE uploads ADD COLUMN cache_key TEXT;
+CREATE INDEX idx_uploads_cache_key ON uploads(cache_key);
+
+-- cache_purge_log is the delete fan-out: DeleteUpload's service half appends
+-- the deleted upload's cache keys here (insert-first, so a failed delete can
+-- at worst cause a spurious purge that re-warms), and every instance's cache
+-- sweep worker consumes the tail each tick, dropping local entries. Rows are
+-- pruned after a retention window by writer-role instances only (reader
+-- discipline, V2-514); instances offline longer than the retention are
+-- covered by boot reconciliation against uploads.cache_key.
+CREATE TABLE cache_purge_log (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    cache_key TEXT NOT NULL,
+    deleted_at DATETIME NOT NULL DEFAULT (datetime('now'))
+);
+
+CREATE INDEX idx_cache_purge_log_deleted_at ON cache_purge_log(deleted_at);
+
+-- +goose Down
+
+DROP TABLE cache_purge_log;
+DROP INDEX idx_uploads_cache_key;
+ALTER TABLE uploads DROP COLUMN cache_key;

internal/handlers/uploads.go

@@ -2,7 +2,6 @@ package handlers
 
 import (
 	"context"
-	"database/sql"
 	"encoding/json"
 	"errors"
 	"fmt"
@@ -818,9 +817,16 @@ func DownloadUpload(db *database.DB, cfg *config.Config, cache *downloadcache.St
 		queueWait := time.Duration(settingsSvc.GetIntWithBounds(
 			"download_queue_wait_secs", 30, 0, 600,
 		)) * time.Second
+		// Private content is cacheable only behind the explicit opt-in
+		// (V2-873): it puts decrypted plaintext on this instance's disk, so
+		// the operator owns that call. Public stays cacheable uncondition-
+		// ally. Auth is unaffected either way — the cache is consulted only
+		// after the row/owner/visibility checks above.
 		var cacheKey string
-		if cacheBudget > 0 && etag != "" && upload.Visibility == "public" {
-			cacheKey = strings.Trim(etag, `"`)
+		if cacheBudget > 0 && etag != "" {
+			if upload.Visibility == "public" || settingsSvc.GetBool("download_cache_private", false) {
+				cacheKey = strings.Trim(etag, `"`)
+			}
 		}
 		// The coalesce loop clears cacheKey on wait-timeout; remember the
 		// original eligibility so the hit/miss accounting (V2-825) still
@@ -1237,7 +1243,7 @@ func DeleteUpload(db *database.DB, cache *downloadcache.Store) http.HandlerFunc
 		// while a cached copy of the content remains locally readable. For a
 		// private upload the row delete destroys the DataMap, so this
 		// ordering is what makes the shred honest.
-		keys := cachePurgeKeys(upload)
+		keys := upload.CacheKeys()
 		if cache != nil {
 			for _, k := range keys {
 				if err := cache.Drop(k); err != nil {
@@ -1280,22 +1286,6 @@ func DeleteUpload(db *database.DB, cache *downloadcache.Store) http.HandlerFunc
 	}
 }
 
-// cachePurgeKeys returns every cache key an upload's bytes could live under:
-// the serve path keys on the local DataMap when one exists while seeding keys
-// on the network address, so a purge must cover both derivations.
-func cachePurgeKeys(u *services.Upload) []string {
-	var keys []string
-	for _, id := range []sql.NullString{u.DataMap, u.DatamapAddress} {
-		if !id.Valid || id.String == "" {
-			continue
-		}
-		k := downloadcache.KeyForIdentifier(id.String)
-		if len(keys) == 0 || keys[0] != k {
-			keys = append(keys, k)
-		}
-	}
-	return keys
-}
 
 // effectiveAllowlist resolves the content-type allowlist for an upload using
 // the override chain: token > user > system setting > built-in default.

internal/handlers/uploads_delete_purge_test.go

@@ -212,3 +212,45 @@ func TestDeleteResidual_OrphanedCacheBytesNotServable(t *testing.T) {
 		t.Fatalf("orphaned cache bytes were served: %d", w.Code)
 	}
 }
+
+// V2-873: private uploads enter the cache only behind download_cache_private,
+// and their deletion purges the plaintext like any other entry.
+
+func TestDownloadUpload_PrivateCachedWhenOptedIn(t *testing.T) {
+	const content = "private but cached by choice"
+	fake := newCacheFakeAntd(t, content)
+	router, token, cfg, db, _ := newCacheTestEnvWithStore(t, fake.srv.URL, map[string]string{
+		"download_cache_max_bytes": "1048576",
+		"download_cache_private":   "true",
+	})
+	uuid := makeUpload(t, router, db, token, "opted-in.txt", "private", "dm-opted-in")
+
+	warmCache(t, router, token, uuid, cfg.DataDir, content)
+	if w := doDownload(router, token, uuid, ""); w.Code != http.StatusOK || w.Body.String() != content {
+		t.Fatalf("repeat download: %d %q", w.Code, w.Body.String())
+	}
+	if got := fake.privateHits.Load(); got != 1 {
+		t.Fatalf("antd private fetches = %d, want 1 (repeat must be a cache hit)", got)
+	}
+}
+
+func TestDeleteUpload_PurgesPrivateCachedCopy(t *testing.T) {
+	const content = "private plaintext must die with the DataMap"
+	fake := newCacheFakeAntd(t, content)
+	router, token, cfg, db, store := newCacheTestEnvWithStore(t, fake.srv.URL, map[string]string{
+		"download_cache_max_bytes": "1048576",
+		"download_cache_private":   "true",
+	})
+	uuid := makeUpload(t, router, db, token, "shred-me.txt", "private", "dm-shred-me")
+	cached := warmCache(t, router, token, uuid, cfg.DataDir, content)
+
+	if w := doDelete(router, token, uuid); w.Code != http.StatusOK {
+		t.Fatalf("delete: %d %s", w.Code, w.Body.String())
+	}
+	if _, err := os.Stat(cached); !os.IsNotExist(err) {
+		t.Fatalf("private plaintext outlived the delete: %v", err)
+	}
+	if count, _ := store.Stats(); count != 0 {
+		t.Fatalf("cache still indexes %d entries after private delete", count)
+	}
+}

internal/services/settings_cache.go

@@ -85,3 +85,14 @@ func (c *CachedSettingsService) GetIntWithBounds(key string, fallback, min, max
 	}
 	return n
 }
+
+// GetBool returns the setting as a bool ("true"/"false"), or fallback when
+// missing or empty. Validation constrains stored values to that pair, so any
+// other value is treated as fallback.
+func (c *CachedSettingsService) GetBool(key string, fallback bool) bool {
+	v, err := c.Get(key)
+	if err != nil || v == "" {
+		return fallback
+	}
+	return v == "true"
+}

internal/services/settings_validation.go

@@ -36,6 +36,7 @@ var typedValidators = map[string]func(string) error{
 	"download_cache_min_uses":              optionalIntInRange(1, 100),
 	"download_cache_inactive_secs":         optionalIntInRange(0, 315360000), // 0 = off; max 10 years
 	"download_cache_seed_on_upload":        oneOf("true", "false"),
+	"download_cache_private":               oneOf("true", "false"),
 }
 
 // oneOf builds a validator that requires the value to be in the allowed set.