Warm-lease: remaining work after Medium tier (repo+commit snapshot, Full pool, prod enablement)

[gbasin]

Context

The Medium content-keyed warm-cache tier is operationally complete on master as of 2026-06-27:

• Phase 0 (feat(sessions): forward repo/branch to Centaur at spawn #118) repo→Centaur wiring · Phase 1 (Centaur docs(voice): canonical remaining-work backlog + refresh status #27) node depcache + sccache · Phase 2.1 (feat(warmcache): Atrium CAS store for dep-cache hydration (warm-lease Phase 2.1) #124) Atrium warmcache_blobs CAS store + session-scoped cache routes.
• Phase 2.2 — runtime wiring (feat(warmcache): wire warm-cache hydrate + daemon capture into the runtime (warm-lease 2.2d) #139): gated warmcache-hydrate init container (read path) + daemon receipt-driven capture with a store-stability gate (write path) + event=warmcache_{hydrate,capture} logs. Validated by a native binary e2e and a real-pod kind e2e (centaur/services/api-rs/crates/centaur-node-sync/ci/warmcache-pod-e2e.sh, informational CI step).
• Phase 2.3 — eviction (feat(warmcache): warm-cache eviction — Atrium TTL/size-cap + node depcache LRU (warm-lease 2.3) #140): Atrium TTL + per-workspace size-cap eviction of warmcache_blobs (mig 052 last_hydrated_at; whole-group deletes inside the existing GC worker → existing CAS GC reclaims; default-safe behind artifactGcEnabled); Centaur node depcache LRU size-cap sweep (src/depcache.rs, node-global, default-OFF).

Full plan + design: notes/warm-lease-build-plan.md · centaur/services/api-rs/crates/centaur-node-sync/docs/warmcache-tier-design.md. Status memory: warm-lease-warmcache-tier.

This issue tracks work deliberately deferred until Medium proves out in production, plus the narrowed follow-up scope from the 2026-06-30 prepared-workspace POCs.

2026-06-30 POC findings: prepared workspace lowers

We tested the remaining “repo+commit working-set snapshot” idea in a VM with overlayfs, pnpm, and a same-filesystem depcache/prepared lower setup.

Synthetic branch-churn POC

• Exact-commit prepared lower worked: session writes landed in the upperdir and did not mutate the prepared lower.
• pnpm hardlink dedupe was preserved when the prepared lower and depcache were on the same filesystem (nlink=2).
• Nearest-ancestor / branch-delta reuse was unsafe: a commit-B workspace mounted over a commit-A prepared lower still saw files deleted by commit B. Generated files only became correct if install/postinstall overwrote them. This means nearest-ancestor reuse needs real deletion/whiteout/clean semantics, not just “lower=A, upper=B”.

Real Atrium surface/ POC

Ran against a clean copy of Atrium surface/ with host node_modules excluded, Node 24, pnpm 10.33.0, 1,382 packages, and a real validation command (pnpm --filter @atrium/surface-client typecheck) from both views.

Final validated run:

online store warm install:          12.26s
current clean offline install:       6.92s
current quick typecheck:             1.57s

prepare exact-commit lower:          3.63s
prepared no-op offline install:      1.67s
prepared quick typecheck:            1.57s

isolation: ok

Storage signal from that run:

clean surface source:        1.90 GB
pnpm depcache:               953 MB
prepared surface lower:      2.88 GB apparent
current install upper:       1.02 GB
prepared session upper:      2.8 MB
depcache + prepared lower:   2.95 GB hardlink-aware total
sample prepared package nlink: 2

Interpretation:

• There is real startup value even after the Medium depcache exists: Atrium surface/ still spent ~6.9s relinking node_modules from a warm pnpm store, while an exact-commit prepared lower reduced no-op install/verification to ~1.7s.
• The value is repo/workload dependent and likely only worth it for hot repos/commits with repeated sessions.
• The safe implementation boundary is exact commit only. Lockfile-only or nearest-ancestor reuse is not safe without more complex cleanup semantics.

1. Operational enablement (do first — gates the rest)

The tier is built but default-safe / partly default-off. To prove Medium out in prod:

• Enable Atrium warm-cache eviction conservatively: set ARTIFACT_GC_ENABLED=1 (the eviction sweep runs inside the existing artifact GC worker). Start with defaults unless prod data says otherwise: WARMCACHE_EVICT_TTL_DAYS=30, WARMCACHE_WORKSPACE_SIZE_CAP_BYTES=50GiB.
• Enable node depcache LRU conservatively (centaur): NODE_SYNC_DEPCACHE_EVICT_ENABLED=true in the DaemonSet. Start with default NODE_SYNC_DEPCACHE_MAX_BYTES=10GiB / NODE_SYNC_DEPCACHE_EVICT_EVERY_N=30, then tune from disk pressure and hit-rate.
• Confirm Atrium populates centaur_session_repos for the sessions you want warmed, and that the agent pod carries CENTAUR_THREAD_KEY (the value the cache routes resolve to a workspace).
• Watch event=warmcache_hydrate hit=… / event=warmcache_capture … / event=depcache_evict … in VictoriaLogs to confirm real hit-rate before tuning.

2. Recommended remaining feature scope

• Instrument first: identify hot repo+commit working sets. Before building more cache machinery, add/review metrics or logs that answer: repo, commit/ref, lockfile hash, toolchain/image key, warmcache hit/miss, install/setup duration, session count per key, and prepared-lower candidate reuse count.
• Build exact-commit prepared workspace lowers only if hot-key reuse justifies it. Key by repo identity + exact commit SHA + lockfile hash(es) + toolchain/image/OS/arch + install/setup command/env/config. Prepare by materializing repo at that exact commit and running install/setup ahead of time; mount it read-only as a lowerdir with an empty per-session upperdir. This attacks the remaining relink/setup cost that Medium depcache does not eliminate.
• Keep prepared lowers same-filesystem with depcache where possible. The POCs showed pnpm hardlinks can avoid duplicating package contents (nlink=2, hardlink-aware depcache+prepared total ~2.95GB for Atrium surface). Cross-filesystem/object-store restoration may lose that property and should be separately measured before rollout.
• Do not build nearest-ancestor or lockfile-only prepared-lower reuse for now. The POC contradicted the safety assumption: deleted files from the lower can leak into the session view. Revisit only with explicit deletion/whiteout semantics, git clean-style correction, and adversarial tests for deleted files, generated files, branch switches, lockfile drift, and setup scripts.

3. Deferred / likely-not-now features

• Phase “Full” — per-repo warm pool (plan §7). Keep deferred. Extending WarmPoolManager workload key (harness, persona) → (harness, persona, repo, branch) and prebuild-on-push is only worthwhile after Medium hit-rate and exact-commit prepared-lower reuse data show enough demand. Cost: idle pods, key explosion, freshness logic, and operational tuning. Benefit: fastest starts for a small set of very hot repo/branch keys.
• Nearest-ancestor Gitpod-Classic-style snapshots. Do not include in first build. Benefit could be broader reuse across branch churn, but cost/correctness risk is high: deletion leakage, generated artifact drift, setup side effects, and tricky invalidation. Current evidence says exact-commit is the safe subset.

4. Hardening / known limitations

• Capture stability window: the daemon captures a store once its (file_count,size,mtime) is stable across 2 ticks. A >1-tick stall mid-install could rarely capture a partial store. Bounded-safe today (hydrate never forces --offline, so a partial cache just degrades to a network fetch), but a quiesce/teardown-driven capture trigger would be tighter. Defer until logs show this matters.
• cargo git/ deps not captured — only cargo/registry .crate files. Add cargo/git to the captured set if git-source deps matter in real workloads.
• Promote the warmcache kind e2e from informational → required once it is proven non-flaky (it currently shares the node-sync pod-native overlay e2e job, which flaked once on a crates-io network reset).

5. Orthogonal freebie (plan §8, any time)

• Node image pre-pull / lazy-load (kube-fledged / SOCI / GKE Image Streaming) for the multi-GB agent image — the research's highest-ROI single win, ~zero idle cost, independent of the cache work.

---

Filed as a checkpoint after the Medium tier landed (#139, #140). Updated 2026-06-30 after synthetic + real Atrium surface/ prepared-workspace POCs. Nothing here blocks the shipped Medium tier; the next buildable cache feature should be exact-commit prepared lowers only after production hot-key instrumentation shows enough reuse.