perf(runtime): integrate EnvironmentRevision-keyed Cloudflare WarmPool

[Yevanchen]

Area

Runtime

Parent roadmap and priority

This is the first P0-1 implementation slice for the warm-pool direction in #387, specifically Direction 3 → c. Pay in advance. Target scope: 1–2 weeks.

This issue deliberately references, but does not close, #387; that issue remains the roadmap and evidence ledger.

Problem

A newly assigned Task/cattle Sandbox still puts container wake on the user's first-response critical path. #387 measured the first container-starting RPC at roughly 2.4–4.8s when cold, while a warm Sandbox brought prepare_run down to roughly 1.0–1.3s.

The shipped session prewarm and idle-grace work helps a known Session or a follow-up turn, but it cannot hide the first cold start for a new Session. Users still see avoidable first-turn latency and variance even when the control plane and provider are otherwise healthy.

User impact

On a pool hit, a new Task/cattle Session should receive a clean, compatible, already-running Sandbox so container boot is no longer in the first-turn path. A miss or pool failure must use the official pool's cold-allocation path without failing the user request or waiting for refill.

Decision: reuse Cloudflare's official WarmPool

Use the official WarmPool shipped in Cloudflare's Sandbox SDK; do not build a parallel D1 inventory, claim-lease state machine, lock service, or refill controller.

Primary references:

• WarmPool implementation
• Burst scaling, eager refill, capacity accounting, and stats improvements
• Container-incarnation fencing and teardown/capacity race fixes

At implementation time, verify that the stable @cloudflare/sandbox version used by Mosoo exposes the required behavior. Prefer upgrading to a stable upstream release or contributing the missing hook/fix upstream. Do not copy or fork the complete pool controller into Mosoo. If a safety-critical upstream capability is unavailable, keep target = 0 until it is available.

Mosoo integration scope

Mosoo should add only the platform-boundary adaptation that the upstream generic pool cannot know:

1. Route each logical runtimeSubjectId through the official pool to obtain its physical container UUID; retain the upstream idempotent logical-to-physical assignment behavior.
2. Partition eligibility by a deterministic, versioned compatibility key containing at least (appId, EnvironmentRevision hash, runtime image/ABI, network-policy ABI). A different runtime-affecting input must never share a pool.
3. Revalidate the exact current compatibility key at assignment time. This avoids adopting stale warm capacity after an EnvironmentRevision changes.
4. Apply the immutable Mosoo network boundary before any operation that can start the prewarmed container. Add only the narrowest upstream-supported adapter/hook required for this ordering.
5. Prewarm only safe immutable platform state and Environment package artifacts. Do not start an Agent Driver, inject run/session credentials, mount Session files, or run user code with side effects before assignment.
6. After assignment, inject request-specific setup/secrets and continue through the existing Sandbox/Driver/session lifecycle. Normal runtime lifecycle then owns the assigned resource; pool cleanup must not reclaim it.
7. Fence work to the exact physical container incarnation. Do not retry side effects against a replacement incarnation, and do not count a capacity slot as free until teardown has completed.

P0-1 should use explicitly configured eligible compatibility keys and fixed targets. Adaptive demand prediction and automatic hot-revision ranking can wait for evidence.

Minimal state model

Do not persist a second Mosoo pool state machine in D1. The official WarmPool remains the source of truth for physical allocation and maintains the relevant lifecycle internally:

Observed phase	Meaning	Mosoo behavior
warming	Upstream is starting capacity	Observe only; never route user state to it
ready	Healthy, unassigned capacity for the exact key	Eligible for upstream atomic assignment
assigned	Bound idempotently to one logical runtime subject	Existing runtime lifecycle owns it
tearing_down	Stop/discard is in progress	Keep the slot counted until teardown completes

A failure is an event/result, not a durable custom Mosoo state. Mosoo may project pool statistics for telemetry, but that projection must not become a second allocation authority.

Assignment, concurrency, expiry, and refill

• Use the upstream Durable Object's serialized pop/assignment path. Do not add D1 conditional claiming for pool entries.
• Repeated allocation for the same logical runtimeSubjectId must return the same live physical assignment; different subjects must never receive the same physical incarnation.
• Concurrent losers and pool misses immediately continue through the upstream cold-allocation behavior. They do not queue behind refill.
• If the target subject already has a runtime-subject record, use the existing lifecycle rather than creating another pool assignment.
• Require the upstream eager-refill behavior after consumption, a single in-flight refill guard, bounded parallel scale-up, and capacity re-checks between batches.
• Health sweep/discard and refill failures must stay off the user request path. Repeated failures alert and leave the pool below target rather than spinning or exceeding the cap.
• An unhealthy, stale-key, stopped, or ambiguous-incarnation resource is destroyed; it is never returned to ready for another Session.
• target = 0 disables allocation and drains only unassigned warm capacity. Existing assigned Sessions remain under normal lifecycle ownership.

Observability

Reuse upstream pool stats and add only the Mosoo dimensions needed to evaluate user impact and cost:

• Counts/gauges: configured target, requested/allocated/available capacity, assignment hit/miss/cold overflow, warm/start/discard/refill success and failure, teardown pending, and incarnation mismatch.
• Timings: assignment latency, ready age at assignment, refill duration, warm-hit prepare_run, and end-to-end TTFT.
• Structured logs/traces may carry a non-secret compatibility-key prefix, logical runtime-subject id, physical container UUID, incarnation, outcome, and fallback reason.
• Aggregate metrics must avoid unbounded raw-hash labels.
• Alerts: configured/global cap exceeded, repeated refill failure, teardown stuck, orphan assignment, or network policy not applied before start.
• Provide a read-only operational snapshot sufficient to answer: target, available count, recent hit rate, cold overflow, recent failures, teardown backlog, and estimated standby container-hours.

Capacity and cost guardrails

• target = 0 is the kill switch.
• Start staging at 1 entry for one eligible key. Initial production ceiling is 5 total standby entries; a separate explicit config change is required to exceed 5. P0-1 hard cap: 20 standby entries across all keys.
• Do not create a pool for every observed revision. Only explicitly configured eligible keys receive refill.
• Configure and seed the upstream global maximum explicitly; refuse refill when it would be exceeded. Prefer cold overflow over uncontrolled spend.

Under Cloudflare's Active CPU model, an idle pool is mainly a provisioned-memory cost. At the currently published memory rate, a standard-1 (4 GiB) has a memory-only standby baseline of about $0.036/hour: 20 ≈ $0.72/hour, roughly $500/month, while 5 ≈ $130/month (730-hour month, rounded). Provisioned disk, Durable Objects, logs/requests, network, and active CPU are additional.

Pricing is a planning estimate, not a launch promise. Re-check Cloudflare Containers pricing and the actual billing dashboard immediately before production enablement.

Non-goals

• No custom D1 pool inventory, claim lease, pool reconciler, or competing allocation source of truth.
• No local fork/copy of the complete upstream WarmPool implementation.
• No Pet/Assistant state pooling or sharing of live conversation/Driver state.
• No cross-App pool reuse in P0-1.
• No predictive autoscaler, global scheduler, multiple SLA pools, or user-facing pool configuration.
• No user-facing latency guarantee or automatic retry/replay of a user request after runtime failure.
• No instance-type upgrade, image-diet project, or provider prompt-cache work in this issue.
• No execution of unassigned user setup code or creation of provider sessions before assignment.

Compatibility and migration

• No public API, GraphQL, or Web UI change is required.
• Existing cold allocation remains the compatibility and failure path.
• Environment edits continue to affect only future Sessions; a changed compatibility key cannot use older warm capacity.
• No D1 schema is expected for physical pool allocation. If an audit projection later proves necessary, it must be append-only and must not participate in assignment correctness.

Acceptance criteria

• Implementation uses the official Cloudflare WarmPool assignment/refill controller; review confirms there is no parallel D1 pool table, claim lease, refill controller, or copied upstream implementation.
• The pinned stable SDK version is documented and verified to include the required eager-refill/capacity behavior and container-incarnation/teardown safety; otherwise rollout remains disabled.
• Deterministic tests prove identical canonical revisions share a compatibility key and every runtime-affecting change moves to a different key; no secret appears in identifiers or telemetry.
• A focused integration test with concurrent logical subjects and one ready entry proves one warm assignment, no double assignment, and safe cold overflow. Repeating the same logical subject is idempotent.
• Wrong-key, stale-revision, unhealthy, already-assigned, stopped-incarnation, and already-existing-subject cases take the safe cold or existing-lifecycle path.
• Network constraints are applied before the first container-starting operation; claim-specific credentials, files, setup, Driver, and user code are applied only after assignment.
• Refill failure, upstream cap enforcement, kill switch, teardown completion, and incarnation mismatch have focused runnable coverage at the Mosoo integration boundary.
• A claimed Sandbox is clean and preserves current Task/cattle Session isolation, network policy, package artifact, setup, file-mount, credential, and Driver ordering semantics.
• Deployed staging A/B uses the same workload shape and at least 32 valid pairs. It records hit/miss/overflow identity, shows at least a 2s median reduction in prepare_run on hits, reports TTFT/p95 and correctness, and shows no reliability regression.
• A 24-hour staging soak stays within the configured upstream cap and leaves no orphan unassigned resource after target = 0 plus one cleanup interval.
• Production enablement is blocked on pricing re-verification and an operator-confirmed cost ceiling.

Rollout

1. Dependency gate: verify the pinned stable SDK behavior and Mosoo's pre-start network hook; leave target = 0 if either safety requirement is missing.
2. Dark: wire assignment and telemetry with target = 0; verify cold behavior is unchanged.
3. Staging canary: one eligible key, target 1; run concurrency, stale-key, refill-failure, incarnation, and cleanup fault cases.
4. Staging gate: target up to 5; run the 32-pair A/B and 24-hour leak/cost soak.
5. Production canary: after price review, one eligible key at target 1, then at most 5 if hit rate, latency, reliability, and standby-hours are acceptable.
6. Later, out of P0-1: consider more keys or a cap up to 20 only with measured demand and an explicit cost decision.

Rollback is one configuration change: set target = 0, keep cold overflow active, drain only unassigned warm capacity, and leave assigned Sessions under normal lifecycle ownership.

Alternatives considered

• Custom D1 pool state machine: rejected because it duplicates Cloudflare's official serialized assignment, refill, health, and capacity controller and creates two possible sources of truth.
• Existing Session prewarm / idle grace: valuable for known Sessions and follow-up turns, but cannot remove a new Session's first container wake.
• Cold allocation only: zero standby spend, but preserves the measured first-turn latency and variance.
• Larger instance or image-only work: perf: TTFT roadmap after control-plane work — placement, prompt caching, container wake #387 found that instance upgrades did not remove the boot floor; image work does not move container wake fully off the user path.
• Predictive autoscaling now: adds policy and failure modes before the fixed-size pool has proven hit rate or economics.

Contribution

Not right now; implementation ownership can be assigned separately.

Checklist

• I searched existing issues before opening this request; perf: TTFT roadmap after control-plane work — placement, prompt caching, container wake #387 is the parent roadmap and no standalone warm-pool issue was found.
• I described the problem before the proposed solution.
• I kept this request focused on one improvement.