fix(apple-runner): evict stale runner cache keys after a build - #3247
janicduplessis wants to merge 3 commits into
Conversation
|
Automated review (Claude): no data-loss bug on the default path; two narrow races/gaps worth fixing, the rest checked and fine. Confirmed (low to medium)
Checked and fine
|
|
The eviction logic in 0542d32 looks correct, but the build-then-sweep route has no live proof yet. The sweep runs in the daemon right after a real build, and only unit fixtures with a stubbed xcodebuild cover it. Two things are unproven: a real start that builds still answers while the sweep deletes 150-230 MB trees, and the stale_cache_evicted decision fires on the real route. Please run this on a booted iOS simulator from this head. Seed Not blocking, take or leave: the comment at runner-cache-retention.ts:100 says the next build cleans a half-deleted key, but the build deliberately skips cleanup on cache_metadata_missing and builds over it, so "never a hit; the next build for this key overwrites it" is accurate. No test reaches the under-lock mtime re-check at line 92 or the metadata-first unlink, because the snapshot filter already excludes fresh keys. A test that touches a key's metadata after the listing and asserts it survives would cover both. listActiveRunnerLeaseArtifacts at runner-lease.ts:479 re-derives the live/unknown owner rule that classifyRunnerLease already owns. The empty catches at runner-artifact.ts:212 and retention.ts:50 swallow every eviction failure, so a debug diagnostic there would help while the start path stays non-failing. I looked for a smaller design and found none materially smaller, since the PR already reuses the cache lock, the lease store and the decision emitter. Could the liveness filter go through classifyRunnerLease and drop the inline rule? The Smoke failure is likely unrelated. It fails in the iOS XCTest regression step, which runs raw |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Lease-state and filesystem-read failures can allow eviction of cache keys still used by live runners.
Review effort: Balanced
Findings: 2
Open (3)
What changed in this PR
Adds bounded, lease-aware retention for stale Apple runner cache keys after successful builds.
Changes:
- Adds configurable cache eviction with lock, age, and lease safeguards.
- Runs eviction asynchronously after builds.
- Adds tests and user documentation for retention behavior.
| File | Description |
|---|---|
website/docs/docs/configuration.md |
Documents the retention setting. |
website/docs/docs/commands.md |
Explains automatic cache eviction. |
packages/platform-apple/src/runner/runner-lease.ts |
Exposes active lease artifacts. |
packages/platform-apple/src/runner/runner-cache.ts |
Supports non-waiting locks and eviction diagnostics. |
packages/platform-apple/src/runner/runner-cache-retention.ts |
Implements stale-key eviction. |
packages/platform-apple/src/runner/runner-artifact.ts |
Triggers eviction after builds. |
packages/platform-apple/src/runner/__tests__/runner-cache-retention.test.ts |
Covers retention scenarios and build integration. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| export function listActiveRunnerLeaseArtifacts(): { xctestrunPath: string; cacheKey?: string }[] { | ||
| return listRunnerLeases() |
| | Metro and install helpers | `AGENT_DEVICE_METRO_BEARER_TOKEN`, `AGENT_DEVICE_BUNDLETOOL_JAR` | Public | | ||
| | App hooks and logs | `AGENT_DEVICE_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_IOS_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_MACOS_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_ANDROID_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_APP_LOG_MAX_BYTES`, `AGENT_DEVICE_APP_LOG_MAX_FILES`, `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS`, `AGENT_DEVICE_EVENT_LOG_MAX_BYTES` | Public. Byte caps take whole integers (`5242880`), not `5MB`. | | ||
| | Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. | | ||
| | Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED`, `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` is how many runner cache keys per platform survive a new build (default 3, `0` keeps all). | |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
9 issues found across 7 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/platform-apple/src/runner/__tests__/runner-cache-retention.test.ts">
<violation number="1" location="packages/platform-apple/src/runner/__tests__/runner-cache-retention.test.ts:257">
P3: The stale key's mtime is anchored to the file's fixed `NOW_MS` (2026-10-05), but the sweep triggered inside `ensureXctestrunArtifact` runs with the real `Date.now()`. Until the ambient clock passes ~2026-09-05, `nowMs - lastUsedMs` is negative and key(7) is never evicted, so `vi.waitFor` times out and the test fails purely based on wall-clock date. Seed the stale key with a mtime relative to `Date.now()` (e.g. pass a 30-day-old timestamp computed at test setup) so the assertion does not depend on the machine clock being past a fixed date.</violation>
</file>
<file name="website/docs/docs/commands.md">
<violation number="1" location="website/docs/docs/commands.md:284">
P3: This sweep is best-effort: builds do not await it, and cleanup failures are swallowed, so stale keys can remain. Qualify the deletion as best-effort.</violation>
<violation number="2" location="website/docs/docs/commands.md:284">
P3: `derived/<platform>/` is not the real layout: keyed cache directories are named per platform *and* device kind. `resolveRunnerDerivedBasePath` (runner-cache-metadata.ts) joins `RUNNER_DERIVED_ROOT/derived` with `resolveRunnerDerivedBaseName`, whose values are `ios-simulator`, `ios-device`, `tvos-simulator`, `tvos-device`, `macos`, `visionos-simulator`, `visionos-device` (apple-runner-platform.ts `derivedBaseName`). A reader following the docs will look for `~/.agent-device/apple-runner/derived/ios/`, which never exists. The same looseness works into the configuration.md wording: the keep count applies per platform-kind folder (e.g. `ios-simulator` and `ios-device` are swept independently), not per platform.</violation>
</file>
<file name="packages/platform-apple/src/runner/runner-lease.ts">
<violation number="1" location="packages/platform-apple/src/runner/runner-lease.ts:454">
P2: A lease-directory read failure is treated as no leases, so eviction can delete a cache still used by another runner. Distinguish `ENOENT` from scan errors and abort eviction on other failures.</violation>
<violation number="2" location="packages/platform-apple/src/runner/runner-lease.ts:483">
P1: Treat `owner-state-dir-gone` as active here; otherwise a live lease owner can lose its cache when runner identity probing is unavailable.</violation>
<violation number="3" location="packages/platform-apple/src/runner/runner-lease.ts:484">
P1: A transient failure reading the recorded runner PID's start time can make this function report an active runner lease as inactive, allowing the cache sweep to delete products and the `.xctestrun` file while that runner is still using them. Treat a live but unverified PID as retained here; identity uncertainty is safe to resolve as extra retention, not eviction.</violation>
</file>
<file name="website/docs/docs/configuration.md">
<violation number="1" location="website/docs/docs/configuration.md:133">
P3: This wording conflicts with the new managed-cache sweep: managed keys are evicted after builds, while `.tmp/` only constrains cleanup of custom derived-path overrides. Clarify the two cleanup paths so operators do not infer that the managed cache is never cleaned.</violation>
<violation number="2" location="website/docs/docs/configuration.md:133">
P3: Describe this as the number of most-recent keys guaranteed to survive; keys used within the last day, locked by a build, or referenced by an active lease can also survive.</violation>
</file>
<file name="packages/platform-apple/src/runner/runner-cache-retention.ts">
<violation number="1" location="packages/platform-apple/src/runner/runner-cache-retention.ts:50">
P3: `evictIfUnused` failures are swallowed by the empty `catch {}` in the eviction loop, and `listCacheKeyDirectories`/`lastUsedMs` silently return `[]`/`0` on any fs error. A sweep that can't read the base directory or fails to remove a key reports an empty result with no diagnostic, so an operator cannot tell that the cleanup is not running — the feature's whole point is reclaiming disk. Emit a warn diagnostic (like `process_lock_release_unverified` does in host-kit) when a candidate cannot be evicted.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Turn on auto-fix | Re-trigger cubic
| ...(lease.ownerStateDir ? { stateDir: lease.ownerStateDir } : {}), | ||
| }); | ||
| if (liveness === 'live' || liveness === 'unknown') return true; | ||
| return lease.runnerPid !== null && isLeaseRunnerProcessIntact(lease, lease.runnerPid); |
There was a problem hiding this comment.
P1: A transient failure reading the recorded runner PID's start time can make this function report an active runner lease as inactive, allowing the cache sweep to delete products and the .xctestrun file while that runner is still using them. Treat a live but unverified PID as retained here; identity uncertainty is safe to resolve as extra retention, not eviction.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/platform-apple/src/runner/runner-lease.ts, line 484:
<comment>A transient failure reading the recorded runner PID's start time can make this function report an active runner lease as inactive, allowing the cache sweep to delete products and the `.xctestrun` file while that runner is still using them. Treat a live but unverified PID as retained here; identity uncertainty is safe to resolve as extra retention, not eviction.</comment>
<file context>
@@ -455,14 +463,29 @@ function listRunnerLeasesForOwner(owner: {
+ ...(lease.ownerStateDir ? { stateDir: lease.ownerStateDir } : {}),
+ });
+ if (liveness === 'live' || liveness === 'unknown') return true;
+ return lease.runnerPid !== null && isLeaseRunnerProcessIntact(lease, lease.runnerPid);
+ })
+ .map(({ xctestrunPath, cacheKey }) => ({ xctestrunPath, cacheKey }));
</file context>
| return lease.runnerPid !== null && isLeaseRunnerProcessIntact(lease, lease.runnerPid); | |
| return ( | |
| lease.runnerPid !== null && | |
| (isLeaseRunnerProcessIntact(lease, lease.runnerPid) || isProcessAlive(lease.runnerPid)) | |
| ); |
| owner: { pid: lease.ownerPid, startTime: lease.ownerStartTime }, | ||
| ...(lease.ownerStateDir ? { stateDir: lease.ownerStateDir } : {}), | ||
| }); | ||
| if (liveness === 'live' || liveness === 'unknown') return true; |
There was a problem hiding this comment.
P1: Treat owner-state-dir-gone as active here; otherwise a live lease owner can lose its cache when runner identity probing is unavailable.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/platform-apple/src/runner/runner-lease.ts, line 483:
<comment>Treat `owner-state-dir-gone` as active here; otherwise a live lease owner can lose its cache when runner identity probing is unavailable.</comment>
<file context>
@@ -455,14 +463,29 @@ function listRunnerLeasesForOwner(owner: {
+ owner: { pid: lease.ownerPid, startTime: lease.ownerStartTime },
+ ...(lease.ownerStateDir ? { stateDir: lease.ownerStateDir } : {}),
+ });
+ if (liveness === 'live' || liveness === 'unknown') return true;
+ return lease.runnerPid !== null && isLeaseRunnerProcessIntact(lease, lease.runnerPid);
+ })
</file context>
| if (liveness === 'live' || liveness === 'unknown') return true; | |
| if (liveness === 'live' || liveness === 'unknown' || liveness === 'owner-state-dir-gone') return true; |
| ); | ||
| } | ||
|
|
||
| function listRunnerLeases(): RunnerLease[] { |
There was a problem hiding this comment.
P2: A lease-directory read failure is treated as no leases, so eviction can delete a cache still used by another runner. Distinguish ENOENT from scan errors and abort eviction on other failures.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/platform-apple/src/runner/runner-lease.ts, line 454:
<comment>A lease-directory read failure is treated as no leases, so eviction can delete a cache still used by another runner. Distinguish `ENOENT` from scan errors and abort eviction on other failures.</comment>
<file context>
@@ -444,6 +444,14 @@ function listRunnerLeasesForOwner(owner: {
+ );
+}
+
+function listRunnerLeases(): RunnerLease[] {
let entries: fs.Dirent[];
const root = resolveRunnerLeaseRoot();
</file context>
| 'ios-simulator', | ||
| ); | ||
| base = simulatorBase; | ||
| const stale = seedKey(key(7), 30); |
There was a problem hiding this comment.
P3: The stale key's mtime is anchored to the file's fixed NOW_MS (2026-10-05), but the sweep triggered inside ensureXctestrunArtifact runs with the real Date.now(). Until the ambient clock passes ~2026-09-05, nowMs - lastUsedMs is negative and key(7) is never evicted, so vi.waitFor times out and the test fails purely based on wall-clock date. Seed the stale key with a mtime relative to Date.now() (e.g. pass a 30-day-old timestamp computed at test setup) so the assertion does not depend on the machine clock being past a fixed date.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/platform-apple/src/runner/__tests__/runner-cache-retention.test.ts, line 257:
<comment>The stale key's mtime is anchored to the file's fixed `NOW_MS` (2026-10-05), but the sweep triggered inside `ensureXctestrunArtifact` runs with the real `Date.now()`. Until the ambient clock passes ~2026-09-05, `nowMs - lastUsedMs` is negative and key(7) is never evicted, so `vi.waitFor` times out and the test fails purely based on wall-clock date. Seed the stale key with a mtime relative to `Date.now()` (e.g. pass a 30-day-old timestamp computed at test setup) so the assertion does not depend on the machine clock being past a fixed date.</comment>
<file context>
@@ -0,0 +1,267 @@
+ 'ios-simulator',
+ );
+ base = simulatorBase;
+ const stale = seedKey(key(7), 30);
+ const unrelatedPlatform = path.join(path.dirname(simulatorBase), 'macos', key(8));
+ fs.mkdirSync(unrelatedPlatform, { recursive: true });
</file context>
| - If a fresh runner launch gets stuck before accepting connections, Agent Device invalidates that runner session and launches it once more without forcing a rebuild. | ||
| - CI may cache `~/.agent-device/apple-runner/derived` when the cache key includes the exact Agent Device package contents and selected Xcode version. | ||
| - Runner reuse is authorized only by the cache metadata's content manifest: a restored tree whose files no longer match the recorded digests, modes, or symlink targets is discarded and rebuilt. A cache key must stay exact — the runtime never falls back to a broader cache. | ||
| - Every runner source or Xcode change creates a new cache key under `~/.agent-device/apple-runner/derived/<platform>/`. After a build, Agent Device deletes keys beside the new one that are neither among the 3 most recently used (the new key included) nor used in the last day, and never one a live runner lease points at or a build is holding. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP=<n>` changes that count and `0` keeps every key. A set `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` is never swept. |
There was a problem hiding this comment.
P3: This sweep is best-effort: builds do not await it, and cleanup failures are swallowed, so stale keys can remain. Qualify the deletion as best-effort.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At website/docs/docs/commands.md, line 284:
<comment>This sweep is best-effort: builds do not await it, and cleanup failures are swallowed, so stale keys can remain. Qualify the deletion as best-effort.</comment>
<file context>
@@ -281,6 +281,7 @@ agent-device prepare ios-runner --platform ios --timeout 240000
- If a fresh runner launch gets stuck before accepting connections, Agent Device invalidates that runner session and launches it once more without forcing a rebuild.
- CI may cache `~/.agent-device/apple-runner/derived` when the cache key includes the exact Agent Device package contents and selected Xcode version.
- Runner reuse is authorized only by the cache metadata's content manifest: a restored tree whose files no longer match the recorded digests, modes, or symlink targets is discarded and rebuilt. A cache key must stay exact — the runtime never falls back to a broader cache.
+- Every runner source or Xcode change creates a new cache key under `~/.agent-device/apple-runner/derived/<platform>/`. After a build, Agent Device deletes keys beside the new one that are neither among the 3 most recently used (the new key included) nor used in the last day, and never one a live runner lease points at or a build is holding. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP=<n>` changes that count and `0` keeps every key. A set `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` is never swept.
- Certification is fail-closed: when a product tree cannot be certified at all — a product escaping the derived-data root, an unreadable subtree, a file over 128 MB, or a non-regular entry such as a socket — the build fails with `runner_cache_uncertifiable` naming the path instead of launching uncertified bytes. Point `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` at a plain directory the current user owns; replacing the tree (the error's hint says how) clears a refusal.
- Runner build/start output is written to the session's `runner.log`. The top-level `daemon.log` is reserved for daemon lifecycle/startup issues.
</file context>
| - Every runner source or Xcode change creates a new cache key under `~/.agent-device/apple-runner/derived/<platform>/`. After a build, Agent Device deletes keys beside the new one that are neither among the 3 most recently used (the new key included) nor used in the last day, and never one a live runner lease points at or a build is holding. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP=<n>` changes that count and `0` keeps every key. A set `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` is never swept. | |
| Every runner source or Xcode change creates a new cache key under `~/.agent-device/apple-runner/derived/<platform>/`. After a build, Agent Device best-effort deletes keys beside the new one that are neither among the 3 most recently used (the new key included) nor used in the last day, and never one a live runner lease points at or a build is holding. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP=<n>` changes that count and `0` keeps every key. A set `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` is never swept. |
| | Metro and install helpers | `AGENT_DEVICE_METRO_BEARER_TOKEN`, `AGENT_DEVICE_BUNDLETOOL_JAR` | Public | | ||
| | App hooks and logs | `AGENT_DEVICE_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_IOS_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_MACOS_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_ANDROID_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_APP_LOG_MAX_BYTES`, `AGENT_DEVICE_APP_LOG_MAX_FILES`, `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS`, `AGENT_DEVICE_EVENT_LOG_MAX_BYTES` | Public. Byte caps take whole integers (`5242880`), not `5MB`. | | ||
| | Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. | | ||
| | Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED`, `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` is how many runner cache keys per platform survive a new build (default 3, `0` keeps all). | |
There was a problem hiding this comment.
P3: This wording conflicts with the new managed-cache sweep: managed keys are evicted after builds, while .tmp/ only constrains cleanup of custom derived-path overrides. Clarify the two cleanup paths so operators do not infer that the managed cache is never cleaned.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At website/docs/docs/configuration.md, line 133:
<comment>This wording conflicts with the new managed-cache sweep: managed keys are evicted after builds, while `.tmp/` only constrains cleanup of custom derived-path overrides. Clarify the two cleanup paths so operators do not infer that the managed cache is never cleaned.</comment>
<file context>
@@ -130,7 +130,7 @@ These env vars are the supported user-facing configuration surface. Other `AGENT
| Metro and install helpers | `AGENT_DEVICE_METRO_BEARER_TOKEN`, `AGENT_DEVICE_BUNDLETOOL_JAR` | Public |
| App hooks and logs | `AGENT_DEVICE_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_IOS_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_MACOS_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_ANDROID_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_APP_LOG_MAX_BYTES`, `AGENT_DEVICE_APP_LOG_MAX_FILES`, `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS`, `AGENT_DEVICE_EVENT_LOG_MAX_BYTES` | Public. Byte caps take whole integers (`5242880`), not `5MB`. |
-| Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. |
+| Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED`, `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` is how many runner cache keys per platform survive a new build (default 3, `0` keeps all). |
| Install/update and platform helpers | `AGENT_DEVICE_NO_UPDATE_NOTIFIER`, `AGENT_DEVICE_MACOS_HELPER_BIN`, `AGENT_DEVICE_ANDROID_SNAPSHOT_HELPER_SESSION` | Public operator controls |
| macOS app backend | `AGENT_DEVICE_MACOS_APP_BACKEND`, `AGENT_DEVICE_MACOS_GHOST_CURSOR` | Public operator controls, read by the daemon. `native` drives macOS app sessions through the macOS helper instead of XCTest; see [Commands](/docs/commands). Unset or `xctest` keeps the runner. The drawn agent pointer adds about 0.3 s to each native click, fill, type, and scroll; `AGENT_DEVICE_MACOS_GHOST_CURSOR=0` turns it off. Restart the daemon after changing either value. |
</file context>
| | Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED`, `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` is how many runner cache keys per platform survive a new build (default 3, `0` keeps all). | | |
| | Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED`, `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` | Public operator controls. Forced cleanup of override paths is allowed only under project `.tmp/`; keyed-cache eviction applies to managed cache paths, not `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` overrides. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` sets how many most-recent cache keys per platform are retained (default 3, `0` disables eviction). | |
| - If a fresh runner launch gets stuck before accepting connections, Agent Device invalidates that runner session and launches it once more without forcing a rebuild. | ||
| - CI may cache `~/.agent-device/apple-runner/derived` when the cache key includes the exact Agent Device package contents and selected Xcode version. | ||
| - Runner reuse is authorized only by the cache metadata's content manifest: a restored tree whose files no longer match the recorded digests, modes, or symlink targets is discarded and rebuilt. A cache key must stay exact — the runtime never falls back to a broader cache. | ||
| - Every runner source or Xcode change creates a new cache key under `~/.agent-device/apple-runner/derived/<platform>/`. After a build, Agent Device deletes keys beside the new one that are neither among the 3 most recently used (the new key included) nor used in the last day, and never one a live runner lease points at or a build is holding. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP=<n>` changes that count and `0` keeps every key. A set `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` is never swept. |
There was a problem hiding this comment.
P3: derived/<platform>/ is not the real layout: keyed cache directories are named per platform and device kind. resolveRunnerDerivedBasePath (runner-cache-metadata.ts) joins RUNNER_DERIVED_ROOT/derived with resolveRunnerDerivedBaseName, whose values are ios-simulator, ios-device, tvos-simulator, tvos-device, macos, visionos-simulator, visionos-device (apple-runner-platform.ts derivedBaseName). A reader following the docs will look for ~/.agent-device/apple-runner/derived/ios/, which never exists. The same looseness works into the configuration.md wording: the keep count applies per platform-kind folder (e.g. ios-simulator and ios-device are swept independently), not per platform.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At website/docs/docs/commands.md, line 284:
<comment>`derived/<platform>/` is not the real layout: keyed cache directories are named per platform *and* device kind. `resolveRunnerDerivedBasePath` (runner-cache-metadata.ts) joins `RUNNER_DERIVED_ROOT/derived` with `resolveRunnerDerivedBaseName`, whose values are `ios-simulator`, `ios-device`, `tvos-simulator`, `tvos-device`, `macos`, `visionos-simulator`, `visionos-device` (apple-runner-platform.ts `derivedBaseName`). A reader following the docs will look for `~/.agent-device/apple-runner/derived/ios/`, which never exists. The same looseness works into the configuration.md wording: the keep count applies per platform-kind folder (e.g. `ios-simulator` and `ios-device` are swept independently), not per platform.</comment>
<file context>
@@ -281,6 +281,7 @@ agent-device prepare ios-runner --platform ios --timeout 240000
- If a fresh runner launch gets stuck before accepting connections, Agent Device invalidates that runner session and launches it once more without forcing a rebuild.
- CI may cache `~/.agent-device/apple-runner/derived` when the cache key includes the exact Agent Device package contents and selected Xcode version.
- Runner reuse is authorized only by the cache metadata's content manifest: a restored tree whose files no longer match the recorded digests, modes, or symlink targets is discarded and rebuilt. A cache key must stay exact — the runtime never falls back to a broader cache.
+- Every runner source or Xcode change creates a new cache key under `~/.agent-device/apple-runner/derived/<platform>/`. After a build, Agent Device deletes keys beside the new one that are neither among the 3 most recently used (the new key included) nor used in the last day, and never one a live runner lease points at or a build is holding. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP=<n>` changes that count and `0` keeps every key. A set `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` is never swept.
- Certification is fail-closed: when a product tree cannot be certified at all — a product escaping the derived-data root, an unreadable subtree, a file over 128 MB, or a non-regular entry such as a socket — the build fails with `runner_cache_uncertifiable` naming the path instead of launching uncertified bytes. Point `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` at a plain directory the current user owns; replacing the tree (the error's hint says how) clears a refusal.
- Runner build/start output is written to the session's `runner.log`. The top-level `daemon.log` is reserved for daemon lifecycle/startup issues.
</file context>
| - Every runner source or Xcode change creates a new cache key under `~/.agent-device/apple-runner/derived/<platform>/`. After a build, Agent Device deletes keys beside the new one that are neither among the 3 most recently used (the new key included) nor used in the last day, and never one a live runner lease points at or a build is holding. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP=<n>` changes that count and `0` keeps every key. A set `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` is never swept. | |
| + - Every runner source or Xcode change creates a new cache key under `~/.agent-device/apple-runner/derived/<platform>-<simulator|device>/` (for example `ios-simulator`, `ios-device`, `tvos-device`). After a build, Agent Device deletes keys beside the new one that are neither among the 3 most recently used (the new key included) nor used in the last day, and never one a live runner lease points at or a build is holding. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP=<n>` changes that count and `0` keeps every key. A set `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH` is never swept. |
| for (const { derived } of candidates) { | ||
| try { | ||
| if (await evictIfUnused(derived, nowMs)) evicted.push(derived); | ||
| } catch {} |
There was a problem hiding this comment.
P3: evictIfUnused failures are swallowed by the empty catch {} in the eviction loop, and listCacheKeyDirectories/lastUsedMs silently return []/0 on any fs error. A sweep that can't read the base directory or fails to remove a key reports an empty result with no diagnostic, so an operator cannot tell that the cleanup is not running — the feature's whole point is reclaiming disk. Emit a warn diagnostic (like process_lock_release_unverified does in host-kit) when a candidate cannot be evicted.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/platform-apple/src/runner/runner-cache-retention.ts, line 50:
<comment>`evictIfUnused` failures are swallowed by the empty `catch {}` in the eviction loop, and `listCacheKeyDirectories`/`lastUsedMs` silently return `[]`/`0` on any fs error. A sweep that can't read the base directory or fails to remove a key reports an empty result with no diagnostic, so an operator cannot tell that the cleanup is not running — the feature's whole point is reclaiming disk. Emit a warn diagnostic (like `process_lock_release_unverified` does in host-kit) when a candidate cannot be evicted.</comment>
<file context>
@@ -0,0 +1,108 @@
+ for (const { derived } of candidates) {
+ try {
+ if (await evictIfUnused(derived, nowMs)) evicted.push(derived);
+ } catch {}
+ }
+ return evicted;
</file context>
| | Metro and install helpers | `AGENT_DEVICE_METRO_BEARER_TOKEN`, `AGENT_DEVICE_BUNDLETOOL_JAR` | Public | | ||
| | App hooks and logs | `AGENT_DEVICE_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_IOS_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_MACOS_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_ANDROID_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_APP_LOG_MAX_BYTES`, `AGENT_DEVICE_APP_LOG_MAX_FILES`, `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS`, `AGENT_DEVICE_EVENT_LOG_MAX_BYTES` | Public. Byte caps take whole integers (`5242880`), not `5MB`. | | ||
| | Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. | | ||
| | Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED`, `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` is how many runner cache keys per platform survive a new build (default 3, `0` keeps all). | |
There was a problem hiding this comment.
P3: Describe this as the number of most-recent keys guaranteed to survive; keys used within the last day, locked by a build, or referenced by an active lease can also survive.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At website/docs/docs/configuration.md, line 133:
<comment>Describe this as the number of most-recent keys guaranteed to survive; keys used within the last day, locked by a build, or referenced by an active lease can also survive.</comment>
<file context>
@@ -130,7 +130,7 @@ These env vars are the supported user-facing configuration surface. Other `AGENT
| Metro and install helpers | `AGENT_DEVICE_METRO_BEARER_TOKEN`, `AGENT_DEVICE_BUNDLETOOL_JAR` | Public |
| App hooks and logs | `AGENT_DEVICE_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_IOS_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_MACOS_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_ANDROID_APP_EVENT_URL_TEMPLATE`, `AGENT_DEVICE_APP_LOG_MAX_BYTES`, `AGENT_DEVICE_APP_LOG_MAX_FILES`, `AGENT_DEVICE_APP_LOG_REDACT_PATTERNS`, `AGENT_DEVICE_EVENT_LOG_MAX_BYTES` | Public. Byte caps take whole integers (`5242880`), not `5MB`. |
-| Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. |
+| Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED`, `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` is how many runner cache keys per platform survive a new build (default 3, `0` keeps all). |
| Install/update and platform helpers | `AGENT_DEVICE_NO_UPDATE_NOTIFIER`, `AGENT_DEVICE_MACOS_HELPER_BIN`, `AGENT_DEVICE_ANDROID_SNAPSHOT_HELPER_SESSION` | Public operator controls |
| macOS app backend | `AGENT_DEVICE_MACOS_APP_BACKEND`, `AGENT_DEVICE_MACOS_GHOST_CURSOR` | Public operator controls, read by the daemon. `native` drives macOS app sessions through the macOS helper instead of XCTest; see [Commands](/docs/commands). Unset or `xctest` keeps the runner. The drawn agent pointer adds about 0.3 s to each native click, fill, type, and scroll; `AGENT_DEVICE_MACOS_GHOST_CURSOR=0` turns it off. Restart the daemon after changing either value. |
</file context>
| | Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED`, `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` is how many runner cache keys per platform survive a new build (default 3, `0` keeps all). | | |
| | Apple runner setup | `AGENT_DEVICE_IOS_TEAM_ID`, `AGENT_DEVICE_IOS_SIGNING_IDENTITY`, `AGENT_DEVICE_IOS_PROVISIONING_PROFILE`, `AGENT_DEVICE_IOS_BUNDLE_ID`, `AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH`, `AGENT_DEVICE_IOS_CLEAN_DERIVED`, `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` | Public operator controls. Cleanup is only automatic for override paths under project `.tmp/`. `AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP` is the number of most-recent runner cache keys per platform guaranteed to survive a new build (default 3, `0` disables cleanup); keys used within the last day, locked by a build, or referenced by an active lease may also survive. | |
|
Pushed 503e977. Live proof below, then the code changes. Live proof (this head, booted iOS 27.0 simulator)Isolation: the shared
Before: (1) Diagnostics from the daemon request log (paths shortened to
(2) After: Stub gone, both other real keys and the current key intact. (3) Snapshot, run while the build and sweep were in flight and again after they finished ( The daemon, runner and simulator were mine and were stopped and deleted afterwards. Code changes
|
|
The earlier review at 0542d32 asked for live proof, and that is now in place. The quoted daemon log and directory listing from this head match what that review asked for, though I did not rerun the simulator proof myself. The code still needs two changes before merge, and Compatibility & Provenance is red because of it. There are no conflicts.
This also needs a fix before merge, as the open P1 threads below say: the lease filter in The inline threads on the readdir fallback (Copilot, cubic), the stale-owner lease check (Copilot, cubic) and the null start-time probe (cubic P1) still apply. Six docs-wording threads on The fallow failure at 503e977 has two parts. The unused export above comes from this PR's changes since 0542d32. The complexity findings in |
Summary
Runner cache keys under
~/.agent-device/apple-runner/derived/<platform>/cache-<hash>are never removed. Any runner source or Xcode change mints a new 150-230 MB key and the old one can never match again; one Mac reached 3.45 GB across 20 keys. Details and measurements are in #3246.After a build (the only event that adds a key), the sweep removes sibling keys in the same platform folder unless they are:
.agent-device-runner-cache.jsonmtime, which every reuse rewrites;cache-<hash>.lock, acquired without waiting);AGENT_DEVICE_IOS_RUNNER_CACHE_KEEP=<n>sets the count,0disables. A setAGENT_DEVICE_IOS_RUNNER_DERIVED_PATHis never swept. The sweep runs off the start path, is best-effort, and loads lazily so the eager closure budgets are unchanged.Not in this PR: trimming build scratch inside a kept key (about 97% of each key), and the un-keyed
pnpm build:packageoutput. Both are listed in #3246.Closes #3246
Validation
Tested at 0542d32.
pnpm check:affected --runpassed, pluspnpm check:fallow --base origin/main, the eager-closure and package-closure tests, and all ofpackages/platform-apple/src/runner. New tests cover keep count, the one-day floor, stubs, non-key entries, a held lock, live, dead and handed-off leases,0, the path override, and the sweep after a realensureXctestrunArtifactbuild with a stubbedxcodebuild.I ran the sweep against a metadata-only copy of a real 20-key cache and its 44 leases. It kept 3 keys per platform and would free about 2.1 GB of 3.45 GB (1.5 GB iOS simulator, 0.6 GB macOS).
Not run: a live simulator build through the hook (it would delete other agents' keys in the shared cache home), and a physical device.