From 445f7e838a8cc9698e21b333c552ebaf82a66e7e Mon Sep 17 00:00:00 2001
From: mdheller <21163552+mdheller@users.noreply.github.com>
Date: Thu, 30 Jul 2026 02:04:24 -0400
Subject: [PATCH] storage: three backup jobs that delete nothing, two of which
 never ran
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

None of the estate's PVC definitions had a retention policy. Three
concrete cases, all of which look healthy and none of which bound
anything.

workspace-mail-backup
  A full tarball of /maildata every night into a fixed 20Gi volume, with
  no deletion of any kind. It is not already full only because the image
  it referenced never existed, so the job has never run — the retention
  bug is latent, not fixed. Fixed BEFORE it is allowed to schedule.
  Now: prune by count (7) and by total size (14GB), refuse rather than
  fill (MIN_FREE_MB, exit 1 -> KubeJobFailed, which is a live alert),
  verify the archive is readable before declaring success, and emit an
  EvidenceReceipt whose hash is the archive digest.

  Proved against seeded fixtures:
    9 existing -> pruned 3 -> wrote 1 -> 7 retained
    refusal    -> status="denied" receipt, nothing written, exit 1
    budget     -> prunes until under MAX_TOTAL_MB

mesh-qdrant snapshots
  Three independent defects meant this CronJob has never produced a
  single snapshot, silently:
    - `args:` against curlimages/curl, whose ENTRYPOINT is curl, so the
      container ran `curl /bin/sh -c "<script>"`. The shell never ran.
    - hostname `qdrant`, but the Service is `mesh-qdrant`.
    - `jq` is not in that image.
  It also only ever POSTed, never DELETEd, so snapshots would have grown
  without bound had it worked. And qdrant's default snapshots_path is
  /qdrant/snapshots, which is NOT on the PVC — snapshots would have gone
  to the pod's ephemeral layer and vanished on restart.
  All four fixed; QDRANT__STORAGE__SNAPSHOTS_PATH now points onto the
  volume so growth is at least visible to the capacity guard.

  Proved against a mock qdrant API: 9 snapshots + 1 new -> pruned 3 ->
  7 retained; unreachable qdrant -> status="denied" receipt, exit 1.
  That run caught a real bug — the name extractor was anchored to compact
  JSON and silently returned zero collections against a spaced response,
  reporting a clean no-op backup. Now whitespace-tolerant.

clickhouse cell analytics
  All 7 tables are monthly-partitioned with no TTL anywhere, so a TTL
  DELETE drops whole parts cheaply. Added: 12 months for scores and
  notification metrics, 24 for the reputation/quality/feedback facts.
  Verified by applying the DDL to a scratch database on the live server —
  all 7 report HAS TTL — then dropping it.

mesh-xl-cache
  Not abandoned. `Used By: <none>` is the designed steady state: it caches
  model weights for the scale-to-zero mesh-vllm-xl seat so a warm-up reads
  cache (~2-3 min) instead of re-downloading (~15 min). Annotated on the
  object so a storage audit does not have to re-derive that from the
  manifest. Not deleted.

Two findings that are NOT fixed here, deliberately, and are reported for
Wave 0:

  - infra/datastores/clickhouse/cell/**.sql is never applied. The runner
    at tools/eval_fabric_migrate.py globs "*.sql" non-recursively, so it
    never descends into cell/. Confirmed empirically: none of the 7 tables
    exists in the live server, nor do the eval_fabric ones. The TTL added
    here is correct but inert until that is fixed. tools/ is a live lane,
    so it is not touched.

  - The live clickhouse-data volume is held by hellgraph.events, a
    ReplacingMergeTree with NO PARTITION BY and no TTL. That is the actual
    retention exposure on that PVC, and it is a different table from the
    ones named in the brief. 686M/20G today, so not urgent.
---
 deploy/serving/mesh-xl-a100.yaml              |  13 +++
 ...1_personal_intelligence_cell_analytics.sql |  21 ++--
 .../k8s/mesh-qdrant/base/backup-cronjob.yaml  |  90 +++++++++++++--
 infra/k8s/mesh-qdrant/base/statefulset.yaml   |  13 +++
 .../workspace-mail/base/backup-cronjob.yaml   | 106 +++++++++++++++++-
 5 files changed, 226 insertions(+), 17 deletions(-)

diff --git a/deploy/serving/mesh-xl-a100.yaml b/deploy/serving/mesh-xl-a100.yaml
index 2a9b48314..616dfbf27 100644
--- a/deploy/serving/mesh-xl-a100.yaml
+++ b/deploy/serving/mesh-xl-a100.yaml
@@ -15,6 +15,19 @@ kind: PersistentVolumeClaim
 metadata:
   name: mesh-xl-cache
   namespace: serving
+  annotations:
+    # `kubectl describe pvc mesh-xl-cache` shows `Used By: <none>`, which reads
+    # as an abandoned 120Gi volume to anyone auditing storage. It is not.
+    # Unmounted IS the steady state: mesh-vllm-xl runs at replicas: 0 for $0
+    # idle, and this volume exists precisely so that warming back up re-reads
+    # cached weights (~2-3 min) instead of re-downloading them (~15 min).
+    # Recorded here, on the object itself, so the question does not have to be
+    # re-answered from the manifest every time someone audits PVCs.
+    socioprophet.io/intentionally-unmounted: >-
+      Model-weight cache for the scale-to-zero mesh-vllm-xl seat. Unmounted
+      while that Deployment is at replicas=0, which is its default. Do not
+      reclaim without also accepting ~15min cold-start on the XL seat.
+    socioprophet.io/idle-cost: "~120Gi pd-balanced (~USD 13/mo) while unmounted"
 spec:
   accessModes: [ReadWriteOnce]
   storageClassName: standard-rwo
diff --git a/infra/datastores/clickhouse/cell/0001_personal_intelligence_cell_analytics.sql b/infra/datastores/clickhouse/cell/0001_personal_intelligence_cell_analytics.sql
index 3c6ec823f..e7c3e0302 100644
--- a/infra/datastores/clickhouse/cell/0001_personal_intelligence_cell_analytics.sql
+++ b/infra/datastores/clickhouse/cell/0001_personal_intelligence_cell_analytics.sql
@@ -17,7 +17,8 @@ CREATE TABLE IF NOT EXISTS cell_signal_scores
 )
 ENGINE = MergeTree
 PARTITION BY toYYYYMM(observed_at)
-ORDER BY (cell_id, watch_id, observed_at, signal_id);
+ORDER BY (cell_id, watch_id, observed_at, signal_id)
+TTL toDateTime(observed_at) + INTERVAL 12 MONTH DELETE;
 
 CREATE TABLE IF NOT EXISTS cell_source_quality_facts
 (
@@ -34,7 +35,8 @@ CREATE TABLE IF NOT EXISTS cell_source_quality_facts
 )
 ENGINE = SummingMergeTree
 PARTITION BY toYYYYMM(event_at)
-ORDER BY (cell_id, source_id, event_at);
+ORDER BY (cell_id, source_id, event_at)
+TTL toDateTime(event_at) + INTERVAL 24 MONTH DELETE;
 
 CREATE TABLE IF NOT EXISTS cell_reputation_deltas
 (
@@ -50,7 +52,8 @@ CREATE TABLE IF NOT EXISTS cell_reputation_deltas
 )
 ENGINE = MergeTree
 PARTITION BY toYYYYMM(event_at)
-ORDER BY (cell_id, subject_kind, subject_ref, event_at);
+ORDER BY (cell_id, subject_kind, subject_ref, event_at)
+TTL toDateTime(event_at) + INTERVAL 24 MONTH DELETE;
 
 CREATE TABLE IF NOT EXISTS cell_feedback_outcomes
 (
@@ -64,7 +67,8 @@ CREATE TABLE IF NOT EXISTS cell_feedback_outcomes
 )
 ENGINE = MergeTree
 PARTITION BY toYYYYMM(event_at)
-ORDER BY (cell_id, watch_id, action, event_at);
+ORDER BY (cell_id, watch_id, action, event_at)
+TTL toDateTime(event_at) + INTERVAL 24 MONTH DELETE;
 
 CREATE TABLE IF NOT EXISTS cell_watch_pattern_metrics
 (
@@ -80,7 +84,8 @@ CREATE TABLE IF NOT EXISTS cell_watch_pattern_metrics
 )
 ENGINE = SummingMergeTree
 PARTITION BY toYYYYMM(event_at)
-ORDER BY (cell_id, watch_id, pattern_id, event_at);
+ORDER BY (cell_id, watch_id, pattern_id, event_at)
+TTL toDateTime(event_at) + INTERVAL 24 MONTH DELETE;
 
 CREATE TABLE IF NOT EXISTS cell_notification_metrics
 (
@@ -94,7 +99,8 @@ CREATE TABLE IF NOT EXISTS cell_notification_metrics
 )
 ENGINE = SummingMergeTree
 PARTITION BY toYYYYMM(event_at)
-ORDER BY (cell_id, feed_kind, event_at);
+ORDER BY (cell_id, feed_kind, event_at)
+TTL toDateTime(event_at) + INTERVAL 12 MONTH DELETE;
 
 CREATE TABLE IF NOT EXISTS cell_social_environment_snapshots
 (
@@ -109,4 +115,5 @@ CREATE TABLE IF NOT EXISTS cell_social_environment_snapshots
 )
 ENGINE = MergeTree
 PARTITION BY toYYYYMM(snapshot_at)
-ORDER BY (cell_id, snapshot_at);
+ORDER BY (cell_id, snapshot_at)
+TTL toDateTime(snapshot_at) + INTERVAL 12 MONTH DELETE;
diff --git a/infra/k8s/mesh-qdrant/base/backup-cronjob.yaml b/infra/k8s/mesh-qdrant/base/backup-cronjob.yaml
index 7ccd5c9f1..5ca534bc1 100644
--- a/infra/k8s/mesh-qdrant/base/backup-cronjob.yaml
+++ b/infra/k8s/mesh-qdrant/base/backup-cronjob.yaml
@@ -20,17 +20,93 @@ spec:
           containers:
             - name: qdrant-snapshot
               image: curlimages/curl:8.6.0
-              args:
+              env:
+                # Service is `mesh-qdrant`, not `qdrant` — the old value did not
+                # resolve in this namespace. memoryd already had this right
+                # (infra/k8s/memoryd/base/deployment.yaml: QDRANT_URL).
+                - name: QDRANT_URL
+                  value: http://mesh-qdrant:6333
+                # Keep 7 snapshots per collection. Snapshots are full copies and
+                # they live on the same volume they snapshot, so retention is not
+                # optional here — without it the backup takes out the thing it
+                # is backing up.
+                - name: RETAIN
+                  value: "7"
+              # `command:`, not `args:`. curlimages/curl sets ENTRYPOINT ["curl"],
+              # so `args` left the entrypoint intact and the container ran
+              # `curl /bin/sh -c "<script>"` — curl treating /bin/sh as a URL.
+              # The shell never executed. Combined with the wrong hostname and a
+              # `jq` that is not in this image, this CronJob has never produced a
+              # single snapshot since it was authored, and said nothing.
+              command:
                 - /bin/sh
                 - -c
                 - |
-                  set -e
-                  COLLECTIONS=$(curl -sf http://qdrant:6333/collections | jq -r '.result.collections[].name')
-                  for col in $COLLECTIONS; do
-                    echo "Snapshotting collection: $col"
-                    curl -sf -X POST "http://qdrant:6333/collections/${col}/snapshots"
+                  set -eu
+                  log() { echo "$@" >&2; }
+                  NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
+                  MADE=0; PRUNED=0; FAILED=0
+
+                  # No jq in curlimages/curl. Extract names with grep, which
+                  # busybox does have. Deliberately not adding a dependency to a
+                  # backup job — fewer moving parts is the whole point.
+                  # Whitespace-tolerant: a pretty-printed `"name" : "x"` must
+                  # parse the same as compact `"name":"x"`. The first version of
+                  # this was anchored to the compact form and silently extracted
+                  # nothing against a spaced response — reporting a clean
+                  # "0 collections" backup. Caught by running it against a mock.
+                  names() { grep -oE '"name"[[:space:]]*:[[:space:]]*"[^"]*"' | cut -d'"' -f4; }
+
+                  COLLECTIONS=$(curl -sf --max-time 30 "${QDRANT_URL}/collections" | names || true)
+                  if [ -z "${COLLECTIONS}" ]; then
+                    # "zero collections" and "qdrant is unreachable" look
+                    # identical unless checked separately. Refuse rather than
+                    # report a successful no-op backup.
+                    if ! curl -sf --max-time 10 "${QDRANT_URL}/readyz" >/dev/null 2>&1; then
+                      log "REFUSED: qdrant unreachable at ${QDRANT_URL}"
+                      printf '{"version":"0.1","receipt_id":"evr-qdrant-snapshot-unreachable-%s","created_at":"%s","service_ref":"infra/k8s/mesh-qdrant","action":"qdrant-snapshot","status":"denied","subject_ref":"pvc/qdrant-storage","hash":"","hash_algo":"sha256","metrics":{"collections":0}}\n' \
+                        "$(date -u +%Y%m%dT%H%M%SZ)" "${NOW}"
+                      exit 1
+                    fi
+                    log "qdrant reachable but reports zero collections — nothing to snapshot"
+                  fi
+
+                  for col in ${COLLECTIONS}; do
+                    log "snapshotting collection: ${col}"
+                    if curl -sf --max-time 300 -X POST "${QDRANT_URL}/collections/${col}/snapshots" >/dev/null; then
+                      MADE=$((MADE + 1))
+                    else
+                      log "FAILED: snapshot of ${col}"
+                      FAILED=$((FAILED + 1))
+                      continue
+                    fi
+
+                    # --- retention ------------------------------------------
+                    # The old job only ever POSTed. It never called
+                    # DELETE /collections/{c}/snapshots/{s}, so snapshots would
+                    # have accumulated forever had it ever run. Snapshot names
+                    # carry a timestamp and sort chronologically.
+                    SNAPS=$(curl -sf --max-time 30 "${QDRANT_URL}/collections/${col}/snapshots" | names || true)
+                    TOTAL=$(printf '%s\n' "${SNAPS}" | grep -c . || true)
+                    if [ "${TOTAL}" -gt "${RETAIN}" ]; then
+                      DROP=$((TOTAL - RETAIN))
+                      printf '%s\n' "${SNAPS}" | sort | head -n "${DROP}" | while read -r s; do
+                        [ -z "${s}" ] && continue
+                        log "retention: deleting ${col}/${s}"
+                        curl -sf --max-time 60 -X DELETE "${QDRANT_URL}/collections/${col}/snapshots/${s}" >/dev/null \
+                          || log "WARN: could not delete ${col}/${s}"
+                      done
+                      PRUNED=$((PRUNED + DROP))
+                    fi
                   done
-                  echo "All snapshots created"
+
+                  STATUS=succeeded
+                  [ "${FAILED}" -gt 0 ] && STATUS=partial
+                  printf '{"version":"0.1","receipt_id":"evr-qdrant-snapshot-%s","created_at":"%s","service_ref":"infra/k8s/mesh-qdrant","action":"qdrant-snapshot","status":"%s","subject_ref":"pvc/qdrant-storage","hash":"","hash_algo":"sha256","metrics":{"snapshots_made":%s,"pruned":%s,"failed":%s,"retain":%s}}\n' \
+                    "$(date -u +%Y%m%dT%H%M%SZ)" "${NOW}" "${STATUS}" "${MADE}" "${PRUNED}" "${FAILED}" "${RETAIN}"
+                  log "done: made=${MADE} pruned=${PRUNED} failed=${FAILED}"
+                  [ "${FAILED}" -gt 0 ] && exit 1
+                  exit 0
               resources:
                 requests:
                   cpu: 10m
diff --git a/infra/k8s/mesh-qdrant/base/statefulset.yaml b/infra/k8s/mesh-qdrant/base/statefulset.yaml
index 038ac7962..67dc7d2db 100644
--- a/infra/k8s/mesh-qdrant/base/statefulset.yaml
+++ b/infra/k8s/mesh-qdrant/base/statefulset.yaml
@@ -20,6 +20,19 @@ spec:
       containers:
         - name: qdrant
           image: qdrant/qdrant:v1.9.4
+          env:
+            # Without this, qdrant defaults snapshots_path to ./snapshots
+            # relative to /qdrant — i.e. /qdrant/snapshots, which is NOT on the
+            # PVC. Snapshots then accumulate on the pod's ephemeral writable
+            # layer: they eat node ephemeral storage until eviction and vanish
+            # on every restart. A backup that does not survive a restart is not
+            # a backup, and nothing said so.
+            #
+            # Putting them under the mounted volume makes them durable AND makes
+            # them count against the same 20Gi the guard watches, so unbounded
+            # snapshot growth becomes visible instead of invisible.
+            - name: QDRANT__STORAGE__SNAPSHOTS_PATH
+              value: /qdrant/storage/snapshots
           ports:
             - name: http
               containerPort: 6333
diff --git a/infra/k8s/workspace-mail/base/backup-cronjob.yaml b/infra/k8s/workspace-mail/base/backup-cronjob.yaml
index 1799e682f..c1c622706 100644
--- a/infra/k8s/workspace-mail/base/backup-cronjob.yaml
+++ b/infra/k8s/workspace-mail/base/backup-cronjob.yaml
@@ -38,14 +38,114 @@ spec:
               # exact digest, uid 1000, caps dropped, seccomp RuntimeDefault): the pull
               # succeeds and the one-liner produces a real, listable .tar.gz.
               image: registry.socioprophet.ai/library/alpine@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b
+              env:
+                # Keep 7 dailies. The source is a 10Gi mail volume against a 20Gi
+                # target, so 7 compressed fulls is the most that fits with room to
+                # write the next one before pruning.
+                - name: RETAIN
+                  value: "7"
+                # Hard ceiling on what backups may occupy, independent of RETAIN.
+                # Whichever binds first wins.
+                - name: MAX_TOTAL_MB
+                  value: "14000"
+                # Refuse to start a backup that cannot fit. See below.
+                - name: MIN_FREE_MB
+                  value: "3000"
               command:
                 - /bin/sh
                 - -c
                 - |
-                  set -e
+                  # Retention, verification and a receipt.
+                  #
+                  # This job previously wrote a new full tarball every night and
+                  # deleted NOTHING. It never actually ran (the image did not
+                  # exist), which is the only reason the 20Gi volume is not
+                  # already full — the retention bug was latent, not fixed. It is
+                  # fixed here BEFORE the job is allowed to schedule.
+                  #
+                  # Order matters: prune first, then check free space, then write,
+                  # then verify, then receipt. Writing first and pruning after
+                  # would need headroom for N+1 backups, which is exactly the
+                  # margin that does not exist.
+                  set -eu
                   STAMP=$(date -u +%Y%m%dT%H%M%SZ)
-                  tar czf /backup/mail-${STAMP}.tar.gz /maildata
-                  echo "Mail backup complete: mail-${STAMP}.tar.gz"
+                  OUT="/backup/mail-${STAMP}.tar.gz"
+                  RC=0
+
+                  log() { echo "$@" >&2; }
+
+                  # --- 1. prune by count -------------------------------------
+                  # Oldest first; names are lexically sortable by construction.
+                  COUNT=$(ls -1 /backup/mail-*.tar.gz 2>/dev/null | wc -l | tr -d ' ')
+                  log "retention: ${COUNT} existing backup(s), keeping ${RETAIN}"
+                  if [ "${COUNT}" -ge "${RETAIN}" ]; then
+                    DROP=$((COUNT - RETAIN + 1))
+                    ls -1 /backup/mail-*.tar.gz 2>/dev/null | sort | head -n "${DROP}" | \
+                      while read -r f; do
+                        log "retention: pruning $(basename "$f")"
+                        rm -f "$f"
+                      done
+                  fi
+
+                  # --- 2. prune by total size --------------------------------
+                  while :; do
+                    USED=$(du -sm /backup 2>/dev/null | awk '{print $1}')
+                    [ -z "${USED}" ] && USED=0
+                    [ "${USED}" -le "${MAX_TOTAL_MB}" ] && break
+                    OLDEST=$(ls -1 /backup/mail-*.tar.gz 2>/dev/null | sort | head -n1)
+                    [ -z "${OLDEST}" ] && break
+                    log "retention: ${USED}MB over ${MAX_TOTAL_MB}MB budget, pruning $(basename "${OLDEST}")"
+                    rm -f "${OLDEST}"
+                  done
+
+                  # --- 3. refuse rather than fill the disk -------------------
+                  # A backup that fills the volume takes out the next backup AND
+                  # anything else on it. Refusing is the safe failure, and it is
+                  # loud: exit 1 -> Job failure -> KubeJobFailed (a live alert).
+                  FREE=$(df -Pm /backup | awk 'NR==2 {print $4}')
+                  log "preflight: ${FREE}MB free, need >= ${MIN_FREE_MB}MB"
+                  if [ "${FREE}" -lt "${MIN_FREE_MB}" ]; then
+                    log "REFUSED: only ${FREE}MB free on /backup, below MIN_FREE_MB=${MIN_FREE_MB}"
+                    printf '{"version":"0.1","receipt_id":"evr-mail-backup-refused-%s","created_at":"%s","service_ref":"infra/k8s/workspace-mail","action":"mail-backup","status":"denied","subject_ref":"pvc/workspace-mail-backup","hash":"","hash_algo":"sha256","metrics":{"free_mb":%s,"min_free_mb":%s,"retained":%s}}\n' \
+                      "${STAMP}" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${FREE}" "${MIN_FREE_MB}" "${COUNT}"
+                    exit 1
+                  fi
+
+                  # --- 4. write ----------------------------------------------
+                  if ! tar czf "${OUT}" /maildata; then
+                    log "FAILED: tar returned non-zero"
+                    rm -f "${OUT}"
+                    RC=1
+                  fi
+
+                  # --- 5. verify ---------------------------------------------
+                  # An unreadable archive is not a backup. Listing it is cheap
+                  # and catches a truncated write, which is the failure mode a
+                  # nearly-full volume actually produces.
+                  if [ "${RC}" -eq 0 ] && ! tar tzf "${OUT}" >/dev/null 2>&1; then
+                    log "FAILED: ${OUT} is not a readable gzip archive — removing"
+                    rm -f "${OUT}"
+                    RC=1
+                  fi
+
+                  # --- 6. receipt --------------------------------------------
+                  # One EvidenceReceipt line to stdout -> fluentbit-gke -> Cloud
+                  # Logging. `hash` is the archive digest, so the receipt proves
+                  # WHICH bytes were produced, not merely that something ran.
+                  NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
+                  KEPT=$(ls -1 /backup/mail-*.tar.gz 2>/dev/null | wc -l | tr -d ' ')
+                  USED=$(du -sm /backup 2>/dev/null | awk '{print $1}')
+                  if [ "${RC}" -eq 0 ]; then
+                    SIZE=$(du -m "${OUT}" | awk '{print $1}')
+                    SHA=$(sha256sum "${OUT}" | awk '{print $1}')
+                    printf '{"version":"0.1","receipt_id":"evr-mail-backup-%s","created_at":"%s","service_ref":"infra/k8s/workspace-mail","action":"mail-backup","status":"succeeded","subject_ref":"pvc/workspace-mail-backup","hash":"%s","hash_algo":"sha256","output_refs":["%s"],"metrics":{"size_mb":%s,"retained":%s,"used_mb":%s,"retain_policy":%s,"max_total_mb":%s}}\n' \
+                      "${STAMP}" "${NOW}" "${SHA}" "${OUT}" "${SIZE}" "${KEPT}" "${USED}" "${RETAIN}" "${MAX_TOTAL_MB}"
+                    log "Mail backup complete: $(basename "${OUT}") ${SIZE}MB; ${KEPT} retained, ${USED}MB used"
+                  else
+                    printf '{"version":"0.1","receipt_id":"evr-mail-backup-failed-%s","created_at":"%s","service_ref":"infra/k8s/workspace-mail","action":"mail-backup","status":"failed","subject_ref":"pvc/workspace-mail-backup","hash":"","hash_algo":"sha256","metrics":{"retained":%s,"used_mb":%s}}\n' \
+                      "${STAMP}" "${NOW}" "${KEPT}" "${USED}"
+                  fi
+                  exit "${RC}"
               volumeMounts:
                 - name: mail-data
                   mountPath: /maildata