docs: bare-repo invariant in deploy, runbook, local setup

- docs/operations/deploy.md — boot sequence describes git clone --bare,
  reconcile state machine now references update-ref / merge-tree /
  commit-tree plumbing. Env table note for CFP_DATA_REPO_PATH calls
  out emptyDir + re-clone-on-boot.
- docs/operations/runbook.md — recovery for "API won't boot" drops the
  delete-PVC step (no PVC for data). The "Drop into the pod" snippet
  inspects the bare repo via git --git-dir / `git show HEAD:.gitsheets`.
- .claude/CLAUDE.md — Local setup section: contributors clone --bare;
  optional second clone for a working-tree browser; Pod boot bullet
  reflects bare clone + emptyDir.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: themightychris

.claude/CLAUDE.md

@@ -100,10 +100,11 @@ Two background concerns keep that store in sync with the world:
 ## Local setup
 
 1. `asdf install` — picks up Node from `.tool-versions`
-2. Clone the data repo as a sibling: `git clone git@github.com:CodeForPhilly/codeforphilly-data.git ../codeforphilly-data` (checkout `fixture` for a small seed, or `published` for the full laddr import)
-3. `cp .env.example .env` and edit — point `CFP_DATA_REPO_PATH` at your sibling clone (absolute path recommended; relative paths resolve from `apps/api/`, not repo root)
-4. `npm install`
-5. `npm run dev` — api + web concurrently
+2. **Bare-clone** the data repo as a sibling: `git clone --bare git@github.com:CodeForPhilly/codeforphilly-data.git ../codeforphilly-data` (use `--branch fixture` for a small seed, or `--branch published` for the full laddr import). The app reads via gitsheets' tree-object interface and *requires* bare — see [specs/behaviors/storage.md](../specs/behaviors/storage.md) → "The data clone is bare."
+3. *(optional)* If you want a working tree to browse/edit records by hand, clone *from* your bare into a second directory: `git clone ../codeforphilly-data ../codeforphilly-data-wt`. Push/pull between them; the app doesn't care.
+4. `cp .env.example .env` and edit — point `CFP_DATA_REPO_PATH` at your bare sibling clone (absolute path recommended; relative paths resolve from `apps/api/`, not repo root)
+5. `npm install`
+6. `npm run dev` — api + web concurrently
 
 ```bash
 npm install                 # install all workspaces
@@ -124,7 +125,7 @@ Typical change flow:
 1. **Merge to `main`** — CI builds + tests; nothing deploys yet.
 2. **Publish image** (currently manual) — `docker build --platform=linux/amd64 -t ghcr.io/codeforphilly/codeforphilly-ng:sandbox . && docker push …`. Apple-silicon dev machines must set the platform flag — cluster nodes are amd64.
 3. **GitOps pickup** — `cfp-sandbox-cluster` projects from our `deploy/kustomize/`; on its own merge, applies via `kubectl apply -k`.
-4. **Pod boot** — single replica, `Recreate` strategy. Container entrypoint clones the data repo on first boot (PVC persists across pods). Node boots: env → store load → **reconcile** (ff/rebase/escape-hatch against `origin/<CFP_DATA_BRANCH>`) → **push daemon** → routes → SPA. `/api/health/ready` returns 200 once stores are loaded *and* reconciled.
+4. **Pod boot** — single replica, `Recreate` strategy. Container entrypoint bare-clones the data repo on every pod start (the data volume is `emptyDir`, so first boot = every fresh pod). Node boots: env → store load → **reconcile** (ff/replay/escape-hatch against `origin/<CFP_DATA_BRANCH>`) → **push daemon** → routes → SPA. `/api/health/ready` returns 200 once stores are loaded *and* reconciled.
 5. **Live data updates** — independent of app deploy. Pushes to `published` trigger the [hot-reload webhook](../docs/operations/runbook.md#hot-reload-webhook); the pod rebuilds in-memory state in place, no restart.
 
 Constraints worth knowing before touching anything deploy-shaped:

docs/operations/deploy.md

@@ -104,43 +104,49 @@ curl http://localhost:3001/                  # SPA index.html
 The container entrypoint (`deploy/docker/entrypoint.sh`) only handles the
 bits that *must* run before the Node process exists:
 
-- Trusts the PVC mount via `git config --global safe.directory`.
+- Trusts the data path via `git config --global safe.directory`.
 - Sets a pseudonymous git identity (`CodeForPhilly API
-  <api@users.noreply.codeforphilly.org>`) for any committer line a future
-  rebase might write.
-- On first pod boot — and only then — does a full-history `git clone` of
-  `CFP_DATA_REMOTE` into `CFP_DATA_REPO_PATH` when no `.git` directory
-  exists. On subsequent boots the PVC already holds a clone; no clone is
-  performed.
-- Refreshes `origin`'s URL to whatever `CFP_DATA_REMOTE` is set to (lets
-  operators rotate the remote without re-cloning the PVC).
-- `exec`s the API. That's all — about a dozen lines of shell now.
+  <api@users.noreply.codeforphilly.org>`) for any committer line the
+  reconcile's commit-replay path might write.
+- On first pod boot — `data` is an `emptyDir`, so this is every fresh
+  pod — does a `git clone --bare --branch $CFP_DATA_BRANCH` of
+  `CFP_DATA_REMOTE` into `CFP_DATA_REPO_PATH`. Bare means no working
+  tree; gitsheets operates on the git object DB directly. See
+  [`specs/behaviors/storage.md`](../../specs/behaviors/storage.md) →
+  "The data clone is bare."
+- Refreshes `origin`'s URL to whatever `CFP_DATA_REMOTE` is set to
+  (operators can rotate the remote with a pod restart; the new
+  `emptyDir` re-clones from the new URL).
+- `exec`s the API. That's all.
 
 Then `exec node apps/api/dist/index.js`. Inside node, `buildApp()` registers
 plugins ([apps/api/src/app.ts](../../apps/api/src/app.ts)) in order: env →
-CORS → cookies → trace IDs → error mapper → **store** (loads public +
-private into memory) → **reconcile** (fetch + ff/rebase/escape-hatch against
+CORS → cookies → trace IDs → error mapper → **store** (opens the bare
+public clone via `openRepo({ gitDir })`, loads public + private into
+memory) → **reconcile** (fetch + ff/rebase-replay/escape-hatch against
 origin — see below) → **push daemon** (starts pushing transact'd commits to
 `CFP_DATA_REMOTE`) → services (FTS) → rate limit → idempotency → session
 middleware → swagger → routes → static SPA. Fastify's `listen()` doesn't
 fire until all of those resolve, so once `/api/health/ready` returns 200
-both stores have loaded **and** the working tree has been reconciled with
-origin.
+both stores have loaded **and** local refs have been reconciled with origin.
 
 ### Reconciliation state machine
 
 Lives in [`apps/api/src/store/reconcile.ts`](../../apps/api/src/store/reconcile.ts)
-and is invoked at boot by the reconcile plugin. Same state machine the
-shell used to run, just structured Node so exit codes propagate naturally
-and the same code is reusable from the future hot-reload webhook (#65):
+and is invoked at boot by the reconcile plugin. Operates entirely on the
+object DB via plumbing (`update-ref`, `merge-tree --write-tree`,
+`commit-tree`) so it works against the bare clone with no working tree:
 
 - in sync → no-op (`'in-sync'`)
-- behind → fast-forward (`'fast-forwarded'`)
+- behind → fast-forward via `git update-ref refs/heads/<branch>` (CAS
+  against old commit) (`'fast-forwarded'`)
 - ahead → push (`'pushed-ahead'`; push daemon retries on push failure)
-- diverged + clean rebase → rebase + push (`'rebased'`)
-- diverged + conflicts → abort rebase, create + push a
-  `conflicts/<UTC-timestamp>` branch from the pre-rebase HEAD, hard-reset
-  local to origin (`'conflict-escaped'`; logged at ERROR level so operators
+- diverged + clean replay → `merge-tree --write-tree` + `commit-tree`
+  per local commit on top of remote tip, then `update-ref` + push
+  (`'rebased'`)
+- diverged + replay conflict → preserve pre-replay HEAD on
+  `conflicts/<UTC-timestamp>`, push it, fast-forward local refs to
+  remote tip (`'conflict-escaped'`; logged at ERROR level so operators
   see it in production logs)
 - fetch itself fails (network blip) → log warn, continue with local state
   (`'fetch-failed'`)
@@ -158,24 +164,26 @@ skips reconciliation entirely.
 
 ## Data repo on disk
 
-The API operates on a working tree at `/app/data` backed by a PVC. The
-entrypoint ensures the working tree exists (cloning on first boot); the
-API-side reconcile plugin then synchronizes that tree with `CFP_DATA_REMOTE`
-on every boot, and the push daemon pushes commits made during the pod's
-lifetime back to the remote.
+The API operates on a **bare** clone at `/app/data` backed by an
+`emptyDir` volume. The entrypoint clones (`git clone --bare`) on every
+pod start since `emptyDir` doesn't survive restarts. Within a pod's
+lifetime, the API-side reconcile plugin synchronizes local refs with
+`CFP_DATA_REMOTE` (boot reconcile + hot-reload webhook), and the push
+daemon pushes commits made during the pod's lifetime back to the
+remote.
 
 Implications:
 
-- **PVC contents are durable enough to outlive a single pod**, which lets the
-  push daemon finish pushing any commits made just before pod terminate.
-  But the source of truth is the git remote, not the PVC — wiping the PVC
-  is safe (the next boot re-clones).
+- **No PVC for data.** The git remote is the source of truth; the
+  pod's bare clone is recoverable from there. Pod restart is the
+  recovery primitive — there's nothing to delete first, and no
+  Multi-Attach errors during node failover.
 - **The deploy key matters.** When `CFP_DATA_REMOTE` is SSH (the
   default), the entrypoint relies on `GIT_SSH_COMMAND` (set in the
-  ConfigMap) pointing at the mounted private key. Rotation: replace the
-  SealedSecret, restart the pod. See
-  [secrets.md](secrets.md#data-repo-deploy-key) and the rotation procedure
-  in [sandbox-deploy.md](sandbox-deploy.md#rotating-the-deploy-key).
+  ConfigMap) pointing at the mounted private key. Rotation: replace
+  the SealedSecret, restart the pod. See
+  [secrets.md](secrets.md#data-repo-deploy-key) and the rotation
+  procedure in [sandbox-deploy.md](sandbox-deploy.md#rotating-the-deploy-key).
 
 ## Bucket provisioning (production)
 
@@ -212,7 +220,7 @@ comments. Production pod gets these mounted:
 | `NODE_ENV` | ConfigMap | `production` |
 | `PORT` | ConfigMap | `3001` |
 | `HOST` | ConfigMap | `0.0.0.0` |
-| `CFP_DATA_REPO_PATH` | ConfigMap | `/app/data` (PVC mount) |
+| `CFP_DATA_REPO_PATH` | ConfigMap | `/app/data` — bare gitdir, backed by an `emptyDir`; re-cloned on every pod boot |
 | `CFP_DATA_REMOTE` | Secret | git URL (ssh in prod) |
 | `CFP_DATA_BRANCH` | ConfigMap | e.g. `fixture` / `main` |
 | `CFP_DATA_RELOAD_SECRET` | **Secret** | Shared bearer-token for the hot-reload webhook; when unset the `/api/_internal/reload-data` endpoint returns 503. See [runbook.md](runbook.md#hot-reload-webhook). |

docs/operations/runbook.md

@@ -23,7 +23,7 @@ Look for one of the four common boot failures:
 |------------------|-------|-----|
 | `[entrypoint] ERROR: CFP_DATA_REMOTE is unset` | The Secret containing `CFP_DATA_REMOTE` isn't reaching the pod. | Check `kubectl get secret codeforphilly-secrets -o yaml`; verify the SealedSecret in the GitOps repo decrypted successfully (look at the sealed-secrets controller logs). |
 | `fatal: could not read Username for 'https://...'` or `Permission denied (publickey)` | Bad/missing data-repo credentials. | Verify the `codeforphilly-data-deploy-key` Secret holds a valid `id_ed25519` whose public key has push access to the data repo. See [secrets.md](secrets.md#data-repo-deploy-key). |
-| `Failed to open public gitsheets store` | Working tree corrupt or missing `.gitsheets/` configs. | Exec into the pod, inspect `/app/data/.gitsheets/`. Recovery: `kubectl delete pvc codeforphilly-data -n <ns>`, then trigger a rollout — the entrypoint re-clones from `CFP_DATA_REMOTE`. |
+| `Failed to open public gitsheets store` | Bare clone corrupt or missing `.gitsheets/` configs. | Exec into the pod, inspect `/app/data/refs/`, `/app/data/objects/`, and verify `.gitsheets/` exists in HEAD via `git --git-dir=/app/data show HEAD:.gitsheets`. Recovery: restart the pod — `data` is an `emptyDir`, so a fresh pod re-clones from `CFP_DATA_REMOTE` automatically. |
 | `Failed to load private store (s3)` | Bucket creds wrong, bucket gone, or network ACL blocks egress. | Confirm `S3_*` env in the ConfigMap + Secret. From the pod, `curl $S3_ENDPOINT` to confirm reachability. |
 | `environment variable ... is required` | A required env (`CFP_DATA_REPO_PATH`, `STORAGE_BACKEND`, `CFP_JWT_SIGNING_KEY`) is missing. | Manifest regression. Compare against `deploy/kustomize/base/configmap.yaml` + the GitOps repo's SealedSecret. |
 
@@ -37,8 +37,10 @@ kubectl -n codeforphilly debug -it deploy/codeforphilly \
 From inside:
 
 ```bash
-# Is the data repo really there?
-ls -la /app/data /app/data/.gitsheets
+# Is the bare data repo really there? Bare gitdir lives at the path root —
+# no .git subdir; expect HEAD, config, objects/, refs/ at the top.
+ls -la /app/data
+git --git-dir=/app/data show HEAD:.gitsheets
 
 # Are env vars present?
 env | grep -E '^(CFP_|S3_|STORAGE_|GITHUB_)' | sort
@@ -71,8 +73,10 @@ kubectl -n codeforphilly-rewrite-sandbox set image \
   deploy/codeforphilly codeforphilly=ghcr.io/codeforphilly/codeforphilly-ng:<known-good-tag>
 ```
 
-Data is **not** in the PVC long-term; it's in the git remote. Deleting the
-PVC and letting the entrypoint re-clone is safe.
+The bare data clone lives in an `emptyDir` — re-cloned from the git remote on
+every pod boot. Pod restart is the recovery primitive; there's no PVC to
+delete. (A `codeforphilly-private` PVC still exists for the S3-fallback
+private store; only `codeforphilly-data` was retired.)
 
 ## "Readiness flapping / 503 spikes"