release: promote staging to prod

[suisuss]

Promotes staging to prod (35 commits since the last sync).

KEEP-683 (wave 1 of 2) - the piece needing operator follow-up

Includes the gas-denormalization write side only: migration 0096 (adds nullable workflow_executions.gas_used_wei), the finalize writer, and the backfill script. The read-switch (#1434) is intentionally NOT in this promotion - it must not go live on prod until the prod backfill has run and the equivalence check passes, or the dashboard would read ~0 gas for historical rows.

Migration 0096 is the only schema change in this batch and is safe to run inline: nullable ADD COLUMN, no default, catalog-only in PG11+ (no rewrite, no @requires-db-prep).

Required operator steps AFTER this deploys to prod

1. Run the backfill against prod (large - millions of rows; resumable via --after-id):

   # via the prod migrator image, DATABASE_URL from the prod db-url secret
   pnpm tsx scripts/backfill-workflow-gas.ts --dry-run
   pnpm tsx scripts/backfill-workflow-gas.ts --yes
   

2. Run the equivalence gate on prod (per-org + grand-total SUM(gas_used_wei) == old JSON sum). Validated on staging: 0 mismatches.
3. Only then promote perf: switch analytics gas reads to the denormalized column #1434 (the read-switch) in a later wave.

During the window between this deploy and the backfill completing, prod reads are unchanged (still JSON), so there is no user-visible regression - the column is write-only until #1434.

Also included (accumulated staging changes)

KEEP-493 (workflow editor UX), KEEP-612 (detection follow-ups), KEEP-679 (metrics collector serialization + timeouts), TECH-6480 (metrics scrape cache), and a batch of security hardening fixes. These are independent of KEEP-683 and were already merged + deployed on staging.

Verification done on staging

• app-7b8ffcd rolled out, migration 0096 applied via Helm hook.
• Backfill ran clean (43 gas-bearing rows, cursor exhausted).
• Equivalence gate: 0 per-org mismatches, exact grand-total match.

[suisuss]

Post-deploy operator runbook (prod gas backfill + verification)

Run these AFTER this promotion deploys to prod. This is the write side only; the read-switch (#1434) stays out until the gate below passes. Until then prod reads are unchanged (still JSON), so the column being partially populated is not user-visible.

Setup

cd <TechOps workspace>
export KUBECONFIG=<path to the prod kubeconfig>
REG=020732533267.dkr.ecr.us-east-2.amazonaws.com/keeperhub-prod   # note: us-east-2, not us-east-1
# Derive the deployed SHA from the running prod image (no guessing):
SHA=$(kubectl get deploy keeperhub-common -n keeperhub \
  -o jsonpath='{.spec.template.spec.containers[0].image}' | sed 's/.*:app-//')
IMG=$REG:migrator-$SHA

DB secret: keeperhub-common-db-url (key keeperhub-common-db-url) -> DATABASE_URL. Namespace: keeperhub.

1. Dry-run (read-only; confirms column exists + total scope)

kubectl run kh-backfill-dryrun -n keeperhub --image="$IMG" --restart=Never \
  --overrides='{"spec":{"restartPolicy":"Never","containers":[{"name":"backfill","image":"'"$IMG"'","command":["sh","-c","cd /app && pnpm tsx scripts/backfill-workflow-gas.ts --dry-run"],"env":[{"name":"DATABASE_URL","valueFrom":{"secretKeyRef":{"name":"keeperhub-common-db-url","key":"keeperhub-common-db-url"}}}]}]}}'
kubectl logs kh-backfill-dryrun -n keeperhub | tail
kubectl delete pod kh-backfill-dryrun -n keeperhub

2. Controlled probe, then full backfill (writes; --yes required on non-local DB)

# 10-batch probe first to confirm writes + measure per-batch time
... scripts/backfill-workflow-gas.ts --yes --max-batches 10
# full run
... scripts/backfill-workflow-gas.ts --yes
# resume after interruption (cursor is logged each batch):
... scripts/backfill-workflow-gas.ts --yes --after-id <last_cursor>

Same pod pattern as step 1 with the command swapped. Resume is idempotent: the gas_used_wei IS NULL guard means a re-run never re-touches populated rows and never fights the live finalize writer.

3. Equivalence gate (read-only) - run only after the backfill completes

kubectl run kh-psql-verify -n keeperhub --image=postgres:16-alpine --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"psql","image":"postgres:16-alpine","command":["sleep","900"],"env":[{"name":"DATABASE_URL","valueFrom":{"secretKeyRef":{"name":"keeperhub-common-db-url","key":"keeperhub-common-db-url"}}}]}]}}'
kubectl wait --for=condition=Ready pod/kh-psql-verify -n keeperhub --timeout=60s

WITH col AS (
  SELECT w.organization_id AS org, COALESCE(SUM(we.gas_used_wei),0) AS col_total
  FROM workflow_executions we JOIN workflows w ON we.workflow_id = w.id
  GROUP BY w.organization_id
), js AS (
  SELECT w.organization_id AS org,
    COALESCE(SUM(CAST(CASE WHEN jsonb_typeof(wel.output)='string'
      THEN (wel.output #>> '{}')::jsonb->>'gasUsed' ELSE wel.output->>'gasUsed' END AS NUMERIC)),0) AS json_total
  FROM workflow_execution_logs wel
  JOIN workflow_executions we ON wel.execution_id = we.id
  JOIN workflows w ON we.workflow_id = w.id
  WHERE (CASE WHEN jsonb_typeof(wel.output)='string'
      THEN (wel.output #>> '{}')::jsonb->>'gasUsed' ELSE wel.output->>'gasUsed' END) IS NOT NULL
  GROUP BY w.organization_id
)
SELECT COALESCE(col.org,js.org) AS org, COALESCE(col.col_total,0) col_total,
       COALESCE(js.json_total,0) json_total,
       COALESCE(col.col_total,0)-COALESCE(js.json_total,0) AS diff
FROM col FULL OUTER JOIN js ON col.org=js.org
WHERE COALESCE(col.col_total,0) <> COALESCE(js.json_total,0)
ORDER BY abs(COALESCE(col.col_total,0)-COALESCE(js.json_total,0)) DESC;

Expect 0 rows. Validated on staging: 0 per-org mismatches, exact grand-total match.

---

Two notes that matter

1. The verification query reproduces the heavy scan this ticket exists to kill. The js (JSON) side is the same unindexed per-row JSONB aggregation that touched ~33 GB on Sky and drove the 2026-05-29 incident. Running the gate on prod briefly reproduces that load, so run it off-peak. It is a one-time verification and the last time we ever pay that cost - once perf: switch analytics gas reads to the denormalized column #1434 ships, nothing reads gas that way again.

2. Run the gate only after the backfill has fully completed (cursor exhausted). If run mid-backfill it will report false mismatches for rows not yet filled (column < JSON), which looks like a correctness failure but is just incomplete progress. Confirm the full run logged done ... last_cursor=<end> with the final batch smaller than the batch size before trusting the gate. Do not promote perf: switch analytics gas reads to the denormalized column #1434 until this passes.