Skip to content

refactor(ios): point xcodebuild at the scoped simulator set and delete the XCTestDevices redirect - #2963

Merged
thymikee merged 7 commits into
mainfrom
refactor/2935-xcodebuild-simulator-set
Sep 25, 2026
Merged

thymikee merged 7 commits into
mainfrom
refactor/2935-xcodebuild-simulator-set

Conversation

@thymikee

@thymikee thymikee commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

For a simulator in a scoped set (--ios-simulator-device-set), both runner xcodebuild phases now get -DVTSimulatorSetLocation=<set>. With that, xcodebuild resolves and runs the test inside the user's set, so nothing has to swap ~/Library/Developer/XCTestDevices any more. The XCTestDevices redirect is deleted: the symlink and backup, the host-wide lock, the session redirect handle, the simulator_set_redirect handoff and adoption refusals, and the close-retain exclusion for scoped sets.

This removes the data-loss path from #2935 by construction. On Macs whose Xcode shims are armed, the shims run xcodebuild -runFirstLaunch, which cleans XCTestDevices. XCTestDevices now never points at user data, so those cleanups can't delete it. Scoped-set runners also work on those Macs now, where #2947 would have refused them.

Evidence for the key: static presence in every Xcode from 14.0 through 27.1, and live runs on 26.2 and 27.1 (see the #2935 redesign comment). If a future Xcode drops the key, the runner fails with a typed simulator_set_destination_not_found error that names the set path and the Xcode version, and never falls back to a redirect. A one-time cleanup at daemon startup undoes a redirect that an older version left behind. It only touches state that agent-device wrote.

Closes #2935. Supersedes #2947. 31 files. Production +171/−499, net −328. Tests +288/−879.

Validation

Tested head 9e623baff2:

  • check:affected --run passes (3758 tests, 497 files, command-docs 12/12). Also passing: typecheck, lint, format:check, layering, and fallow with no issues in changed files (one pre-existing inherited finding, unchanged). apple-runner project: 622/622 tests.
  • Live, Xcode 26.2, armed shims: on a scoped set, a cold open, snapshot, a runner-backed press, snapshot and close all pass.
    • Both xcodebuild phases carry the argument.
    • A watcher sampled XCTestDevices 116 times, and it was a real directory every time. The device survived 6 -runFirstLaunch cleanups, including simctl --set XCTestDevices delete all during the build.
    • A fresh daemon adopted the scoped runner, which main refused.
    • The default set works unchanged, with no argument added.
  • Live, daemon-startup legacy-redirect restore (v0.21.14 leftovers): three runs — a symlink with its backup, a symlink with no backup, and a read-only Library/Developer (EACCES) — the daemon stays up in all three, and each outcome is recorded in daemon.log. A scoped runner start with no per-start restore call still resolves and interacts correctly, with the host XCTestDevices directory untouched throughout.
  • The simulator_set_destination_not_found message text is now captured from real Xcode 26.2 output (three xcodebuild invocations, all exit 70), not invented.
  • The pipeline ran three adversarial reviews, a final ship review, and a maintainer review round.
  • Xcode coverage: there is live evidence on 26.2 and 27.1. Older Xcodes are out of scope, because App Store submission now requires building with Xcode 26 or 27.

Review in cubic

…on, not a redirect

xcodebuild resolves a simulator in a custom CoreSimulator set through the
DVTSimulatorSetLocation Xcode user default, passed as -DVTSimulatorSetLocation=<set>.
Every runner xcodebuild phase (build-for-testing and test-without-building) now
names the scoped set beside its -destination, so the runner never needs
~/Library/Developer/XCTestDevices. Xcode's first-launch cleanup deletes every
device in XCTestDevices, and while it was a symlink into the user's set that
cleanup deleted the user's simulators (#2935).

Deleted: the XCTestDevices rename/symlink/reconcile machinery, the host-global
xctest-device-set.lock, the session redirect handle and its releases, the
simulator_set_redirect handoff refusal and the scoped-set adoption refusal.

A simulator runner start puts back a symlink or XCTestDevices.agent-device-backup
an older version left behind. A destination xcodebuild cannot find for a scoped
set fails with details.reason simulator_set_destination_not_found, naming
details.simulatorSetPath and details.xcodeVersion, in both xcodebuild phases.

Closes #2935
close stopped the runner of a simulator in a scoped set only because that
runner held the XCTestDevices redirect and its host-global lock for its
lifetime (#1013). The redirect is gone, so a scoped-set runner is retained
under idle-stop like a default-set one.
…t behind

The legacy cleanup unlinked any XCTestDevices symlink on every simulator
startup, so a host that links XCTestDevices to another volume lost the
link on a default-set open. A symlink is now removed only when the
agent-device backup sits beside it, or when it points at this runner's
own scoped simulator set.
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-25 13:09 UTC

@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.83 MB 4.83 MB -3.7 kB
Package (unpacked) 4.83 MB 4.83 MB -3.7 kB
Package (download) 1.45 MB 1.45 MB -948 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 26.9 ms 26.6 ms -0.3 ms
CLI --help 76.6 ms 76.0 ms -0.6 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 31 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/platform-apple/src/runner/runner-device-set.ts">

<violation number="1" location="packages/platform-apple/src/runner/runner-device-set.ts:37">
P1: This migration misses legacy backup names that the previous redirect code explicitly supported. A host left with `.agent-device-xctestdevices-backup-*` can remain redirected, leaving the old `XCTestDevices` data-loss risk in place; scan and restore those legacy backups too.</violation>
</file>

<file name="packages/platform-apple/src/runner/runner-session.ts">

<violation number="1" location="packages/platform-apple/src/runner/runner-session.ts:257">
P2: Adopted legacy simulator runners skip this cleanup because adoption returns before this line. Move the cleanup before adoption, or make adoption perform it, so stale redirect state is removed during handoff.</violation>
</file>

<file name="src/platform-runtime-resource-cleanup.ts">

<violation number="1" location="src/platform-runtime-resource-cleanup.ts:88">
P2: Re-enabling warm retention for scoped-set simulators lets a later `open` reuse a runner whose build is bound to the previous simulator set. `ensureRunnerSession`/`resolveReusableRunnerSession` (packages/platform-apple/src/runner/runner-session.ts:150-158, 391+) reuse the retained session keyed only by `device.id` and never compare the set, and warm reuse skips the xcodebuild phase that would pass `-DVTSimulatorSetLocation`. If the same UDID is later resolved in a different set (a set cloned from the default set preserving UDIDs, or a copied device), the retained runner silently services the device with an artifact built for the old set — no `simulator_set_destination_not_found` error, no rebuild. Validate the set before reuse (e.g., compare `existing.device.simulatorSetPath` against the requested set and stop+rebuild on mismatch).</violation>
</file>

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

device: DeviceInfo,
xctestDeviceSetPath: string = path.join(os.homedir(), 'Library', 'Developer', 'XCTestDevices'),
): void {
const backupPath = `${xctestDeviceSetPath}.agent-device-backup`;

@cubic-dev-ai cubic-dev-ai Bot Sep 25, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: This migration misses legacy backup names that the previous redirect code explicitly supported. A host left with .agent-device-xctestdevices-backup-* can remain redirected, leaving the old XCTestDevices data-loss risk in place; scan and restore those legacy backups too.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/platform-apple/src/runner/runner-device-set.ts, line 37:

<comment>This migration misses legacy backup names that the previous redirect code explicitly supported. A host left with `.agent-device-xctestdevices-backup-*` can remain redirected, leaving the old `XCTestDevices` data-loss risk in place; scan and restore those legacy backups too.</comment>

<file context>
@@ -1,392 +1,66 @@
+  device: DeviceInfo,
+  xctestDeviceSetPath: string = path.join(os.homedir(), 'Library', 'Developer', 'XCTestDevices'),
+): void {
+  const backupPath = `${xctestDeviceSetPath}.agent-device-backup`;
+  const backupExists = fs.existsSync(backupPath);
+  const isSymlink =
</file context>
Fix with cubic

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declined, with evidence. No commit on main ever wrote a .agent-device-xctestdevices-backup-* name: git log --oneline -S "agent-device-xctestdevices-backup" origin/main -- packages src returns only #354 (the original redirect), #1062 (a module split) and #2598 (a lock change). In #354 itself the only writer used ${XCTestDevices}.agent-device-backup (no -* suffix); the -backup-* form was only ever read, referring to a pre-squash commit on the #354 branch that never landed on main. No released agent-device version can leave a host in that state, so there's nothing for the daemon-startup scan to find.

Comment thread packages/platform-apple/src/runner/runner-error-classification.ts Outdated
Comment thread packages/platform-apple/src/runner/runner-device-set.ts Outdated
Comment thread packages/platform-apple/src/runner/runner-startup-transport.ts Outdated
Comment thread packages/platform-apple/src/runner/runner-session.ts Outdated
// an external xctestrun that never launches are different steps, and a caller told "developer disk
// image" should not have to know which one this run happened to take.
try {
if (device.kind === 'simulator') restoreLegacyXctestDeviceSetRedirect(device);

@cubic-dev-ai cubic-dev-ai Bot Sep 25, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Adopted legacy simulator runners skip this cleanup because adoption returns before this line. Move the cleanup before adoption, or make adoption perform it, so stale redirect state is removed during handoff.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/platform-apple/src/runner/runner-session.ts, line 257:

<comment>Adopted legacy simulator runners skip this cleanup because adoption returns before this line. Move the cleanup before adoption, or make adoption perform it, so stale redirect state is removed during handoff.</comment>

<file context>
@@ -247,16 +247,14 @@ async function startRunnerSessionWithLease(
   // an external xctestrun that never launches are different steps, and a caller told "developer disk
   // image" should not have to know which one this run happened to take.
   try {
+    if (device.kind === 'simulator') restoreLegacyXctestDeviceSetRedirect(device);
     xctestrunArtifact = await measureRunnerStartupStep(
       startupTimings,
</file context>
Fix with cubic

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Originally fixed at 3dd240c by moving the restore ahead of tryAdoptRunnerSessionFromLease in startRunnerSessionWithLease, so one call covered both routes.

Superseded at 9e623ba: the restore left the runner-start path entirely and moved to a one-time daemon-startup step (see the top-level reply, design question). Neither adoption nor a build can skip a step that no longer runs on that path, so this is moot rather than fixed-in-place. The commit that added the adoption-ordering call and its test deleted both.

!params.hasScreenRecording &&
!params.hasLease &&
!params.device.simulatorSetPath
!params.hasLease

@cubic-dev-ai cubic-dev-ai Bot Sep 25, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Re-enabling warm retention for scoped-set simulators lets a later open reuse a runner whose build is bound to the previous simulator set. ensureRunnerSession/resolveReusableRunnerSession (packages/platform-apple/src/runner/runner-session.ts:150-158, 391+) reuse the retained session keyed only by device.id and never compare the set, and warm reuse skips the xcodebuild phase that would pass -DVTSimulatorSetLocation. If the same UDID is later resolved in a different set (a set cloned from the default set preserving UDIDs, or a copied device), the retained runner silently services the device with an artifact built for the old set — no simulator_set_destination_not_found error, no rebuild. Validate the set before reuse (e.g., compare existing.device.simulatorSetPath against the requested set and stop+rebuild on mismatch).

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/platform-runtime-resource-cleanup.ts, line 88:

<comment>Re-enabling warm retention for scoped-set simulators lets a later `open` reuse a runner whose build is bound to the previous simulator set. `ensureRunnerSession`/`resolveReusableRunnerSession` (packages/platform-apple/src/runner/runner-session.ts:150-158, 391+) reuse the retained session keyed only by `device.id` and never compare the set, and warm reuse skips the xcodebuild phase that would pass `-DVTSimulatorSetLocation`. If the same UDID is later resolved in a different set (a set cloned from the default set preserving UDIDs, or a copied device), the retained runner silently services the device with an artifact built for the old set — no `simulator_set_destination_not_found` error, no rebuild. Validate the set before reuse (e.g., compare `existing.device.simulatorSetPath` against the requested set and stop+rebuild on mismatch).</comment>

<file context>
@@ -85,8 +85,7 @@ export const platformResourceCleanup: PlatformResourceCleanup = Object.freeze({
       !params.hasScreenRecording &&
-      !params.hasLease &&
-      !params.device.simulatorSetPath
+      !params.hasLease
     );
   },
</file context>
Fix with cubic

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at 3dd240c. Identity is now simulatorAddressFor (core/simctl.ts), the existing owner of "a udid with the set that holds it" — no new keying scheme. It now stores the resolved set path, so two addresses are equal exactly when they name the same simulator; every existing consumer re-resolves the path, so this doesn't change their behavior (new test in core/tests/simctl.test.ts).

Reuse: resolveReusableRunnerSession stops a registered session whose device isn't isSameRunnerSimulator and starts fresh, before the external-xctestrun early return, so external runners are covered too.
Lease/adoption: RunnerLease.simulatorSetPath is written by buildRunnerLease(device) (no longer optional-by-omission), and adoption refuses with typed reason simulator_set_mismatch on a set difference.
The slot stays keyed by udid (map key, lease file/lock) because xcodebuild cleanup and lease paths are per-udid — two concurrent runners for one udid in two sets would kill each other's xcodebuild. It's matched, not re-keyed.

Tests: runner-session-lifecycle.test.ts "the same udid in another simulator set starts its own runner instead of reusing this one"; runner-adoption.test.ts "a runner leased for the same udid in another simulator set is never adopted" (A->B, A->default, default->A); runner-device-set.test.ts "one udid in two simulator sets names two simulators". Mutation check by hand: disabling either check fails its test.

Comment thread packages/platform-apple/src/runner/__tests__/runner-early-exit-diagnosis.test.ts Outdated
Comment thread packages/platform-apple/src/runner/__tests__/runner-early-exit-diagnosis.test.ts Outdated
Comment thread packages/platform-apple/src/runner/__tests__/runner-device-set.test.ts Outdated
… set in destination errors

- Reuse and lease adoption now require the same simulator: one udid in one
  resolved simulator set. The lease records the scoped set; a runner leased
  or registered for another set is stopped and restarted, never reused.
- The runner derives a simulator's set through simulatorAddressFor, which
  now resolves the set path, instead of restating the simulator predicate.
- simulator_set_destination_not_found names the set and the selected Xcode
  in its message. The Xcode version comes from the toolchain fingerprint
  owner, so external and adopted sessions report it too; the artifact no
  longer carries its own copy. The hint points at the error instead of
  printing field names.
- The legacy XCTestDevices restore runs before adoption, so both startup
  routes pass through it, and a step another daemon already took counts
  as done instead of failing the startup.
@thymikee

Copy link
Copy Markdown
Member Author

At 18078a3 this replaces the XCTestDevices lock, symlink and handle with one argument helper used at both xcodebuild spawns, for net −328 production lines. I found no blocking defect. CI is green, and there are no conflicts.

One design question first: would the legacy-redirect cleanup be smaller as a single step when the daemon or the Apple backend starts, instead of a call on every runner start that goes through a re-export in runner-xctestrun.ts? With the lock and symlink gone, this looks like the only seam left. If there is a reason to keep it on the runner start path, please say what it is.

Related to that: could a symlink left by an older agent-device, with no backup next to it, survive a runner start and a daemon kill (runner-device-set.ts#L41)? And should the restore at runner-session.ts#L257 ignore ENOENT and EACCES, so that a cleanup error cannot fail a runner start?

The live Xcode 26.2 and 27.1 runs are taken from the PR body. The simulator_set_destination_not_found test uses an invented output shape, not output captured from an Xcode that ignores the key.

Not blocking: the session tests could import runner-device-set.ts directly and drop the re-export. The hint in runner-error-classification.ts (L347) may also need a fallback when an external xctestrun has no xcodeVersion.

…de 26.2 output

The simulator_set_destination_not_found fixtures were invented text. They
are now the output Xcode 26.2 (17C52) printed for the runner's own
build-for-testing and test-without-building argv, with the destination
naming a scoped-set udid and -DVTSimulatorSetLocation naming a set that
does not hold it, and for the same build without the key. Destination
lines that identify the host's hardware and simulators are left out.
…at daemon startup

The restore ran on every simulator runner start and reached the session
through a runner-xctestrun re-export. It is now one daemon-startup step on
the platform owner lifecycle, after daemon.log publication. Without a
device to compare against, it removes any symlink at XCTestDevices, as
released versions did on every scoped runner start, and restores the
agent-device backup. Each repair and any failure is recorded in daemon.log;
a failure stops neither the daemon nor a runner start, because the runner
no longer reads XCTestDevices.
@thymikee

Copy link
Copy Markdown
Member Author

Addressed the review at 18078a3. New head 9e623ba.

Design question (single startup step): Moved the restore off the runner-start path entirely. It now runs once, at daemon startup, through a typed PlatformOwnerLifecycle.restoreLegacyXctestDeviceSetRedirect() step (same seam as the Web orphan cleanup). The per-start call, the runner-xctestrun.ts re-export, and the mocks in 8 session test files are deleted. Lazy-imported on macOS only, ~25ms in dist; no eager-closure growth (eager-closure-budgets.test.ts passes against both 18078a3 and the merge-base).

Symlink surviving a daemon kill with no backup: Now removed at the next daemon start, whatever it points at — matching what released v0.21.14 did on every scoped runner start. Each removal logs ios_runner_legacy_xctest_device_set_link_removed with the link target. A host-made link with a backup is still restored, not deleted.

Ignore ENOENT/EACCES so cleanup can't fail a start: Since the restore left the runner-start path, it can no longer fail a runner start at all. At daemon startup every error is caught and recorded as ios_runner_legacy_xctest_device_set_restore_failed (warn); the daemon continues. ENOENT-shaped races (another daemon already finished the step) are read from filesystem state, not error codes. Live-verified: read-only Library/Developer -> daemon stays up, EACCES logged, link untouched.

Captured (not invented) xcodebuild output: Captured live on Xcode 26.2 against a real scoped-set simulator: build-for-testing (scoped + default set) and launch, all exit 70, Unable to find a device matching the provided destination specifier. Fixtures now carry provenance: 'captured' and the real argv/Xcode version. This also surfaced that the classifier's rule matches on "device", not the old fixture's invented "destination" wording — fixed to match real Xcode text.

Non-blocking (re-export, xcodeVersion fallback): Both done — re-export deleted with the design-question fix; the hint stays a static string, and the message now says "Xcode (version unreadable)" when the version can't be read within its 5s budget.

Gates (head 9e623ba): typecheck, lint, format:check, layering, fallow — all pass. apple-runner project: 622/622. check:affected --run: 3758/3758 + 497/497 files, all runnable checks passed.

Live: daemon-startup restore verified for link+backup, link-with-no-backup, and EACCES, all non-fatal. A scoped runner start with the per-start restore removed still resolves and interacts correctly (open/press/close all exit 0), with no restore step in the request log and the host XCTestDevices directory untouched.

Replies to each inline comment follow.

@thymikee

Copy link
Copy Markdown
Member Author

Summary of this round

New head: 9e623baff2 (was 18078a3741), 4 new commits, fast-forward push.
Size vs merge-base (76dbee0), production only: +325/−509, net −184. (Before this round: +173/−501, net −328 — the daemon-startup restore adds some lines back for the single-seam design.)

Gates on 9e623ba: typecheck pass, lint pass, format:check clean, check:layering OK, check:fallow --base origin/main clean (one pre-existing inherited finding, unchanged). apple-runner project: 622/622 tests. check:affected --run: 3758/3758 tests across 497 files, command-docs 12/12, all runnable checks passed.

Live (Xcode 26.2, real device, isolated state):

  • Daemon-startup restore: three runs (link+backup, link-with-no-backup, read-only Library/Developer) — daemon stays up in all three; each outcome logged (ios_runner_legacy_xctest_device_set_link_removed / ..._restore_failed).
  • Scoped runner start with no per-start restore call: open/press/close all exit 0, xcodebuild carries -DVTSimulatorSetLocation, host XCTestDevices untouched, no restore step in the request log.
  • Captured (not invented) simulator_set_destination_not_found xcodebuild output on Xcode 26.2, used to fix the test fixtures.

Thread replies posted on all 10 inline comments (7 fixed-in-place, 1 fixed-then-superseded by the daemon-startup design, 2 declined with evidence). Top-level reply above answers the design question and the two open questions from the review.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at 9e623ba. The cleanup now runs once at daemon start instead of on every runner start, so the design question from the last review is answered. I found no blocking defect in the delta.

The restore is best effort: a missing path, a permission error, or a link with no backup does not fail daemon start, and it only unlinks a symlink, so the target keeps its data. The call goes through the platform lifecycle seam, and the Apple code stays in the root composition file. The simulator_set_destination_not_found fixtures now look like real captured Xcode 26.2 output. The live Xcode 26.2 runs are taken from the author's comment.

One behavior question for a decision: the delta drops the guard from 18078a3 (a backup exists, or the link points at our scoped set). Daemon start now removes any symlink at ~/Library/Developer/XCTestDevices on every macOS start, also for users who never used --ios-simulator-device-set (runner-device-set.ts#L43). A user who links that folder to another volume would lose the link, with only a warning in daemon.log. Should a link with no .agent-device-backup get a warning with a hint instead of an unlink, or is the removal intended (then a CHANGELOG line would help)?

CI: Smoke Tests, Repo Guards and Coverage were still running at review time, and none had failed. Smoke Tests starts the daemon and the iOS runner, so it covers the changed path; a failure there needs a look against this PR.

Not blocking: daemon-runtime-xctest-device-set.test.ts calls the wrapper directly, so deleting the call in daemon-runtime.ts would stay green; and the captured fixtures keep host-specific home and temp paths that could become placeholders.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 25, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 existing issue remains and 4 new issues found across 37 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="website/docs/docs/commands.md">

<violation number="1" location="website/docs/docs/commands.md:207">
P3: The destination error does not always include the selected Xcode version; when the version probe fails, it reports `(version unreadable)`. Qualify this documentation claim to avoid promising a detail that may be unavailable.</violation>
</file>

<file name="src/daemon/server/daemon-runtime.ts">

<violation number="1" location="src/daemon/server/daemon-runtime.ts:562">
P1: This cleanup runs after daemon readiness is published, so a client can start a default-set runner while the legacy `XCTestDevices` symlink still exists. Restore the redirect before exposing the servers, while buffering its diagnostics until after publication.</violation>
</file>

<file name="packages/platform-apple/src/runner/__tests__/runner-startup-failure-reasons.test.ts">

<violation number="1" location="packages/platform-apple/src/runner/__tests__/runner-startup-failure-reasons.test.ts:177">
P3: The shared `assertFailureEnvelope` check was relaxed from exact equality to a prefix `startsWith` for every row in the table, not just the scoped-set rows whose message now appends a suffix. Rows whose failure message stops matching the old exact text (appended detail, reworded base, wrapped text) now pass silently; the exact expected message is only pinned in the two dedicated tests. Keep a per-fixture exact expected message so the other rows retain their regression guard.</violation>
</file>

<file name="packages/platform-apple/src/runner/__tests__/runner-early-exit-diagnosis.test.ts">

<violation number="1" location="packages/platform-apple/src/runner/__tests__/runner-early-exit-diagnosis.test.ts:220">
P3: These assertions pin only the message tail via `endsWith`, so the middle of the user-visible sentence (`xcodebuild found no simulator <udid> in ...`) can drift without failing. The full message is deterministic here (udid, set path and Xcode version are all known), so assert the complete `error.message` to catch a wording regression in the sentence the user sees.</violation>
</file>

Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Fix all with cubic | Re-trigger cubic

createOwnerScopedDeviceClaimReconciler(createDaemonRecoveryPlatformScope()),
baseDir,
);
await restoreLegacyXctestDeviceSetForDaemonStartup(logPath);

@cubic-dev-ai cubic-dev-ai Bot Sep 25, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: This cleanup runs after daemon readiness is published, so a client can start a default-set runner while the legacy XCTestDevices symlink still exists. Restore the redirect before exposing the servers, while buffering its diagnostics until after publication.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/daemon/server/daemon-runtime.ts, line 562:

<comment>This cleanup runs after daemon readiness is published, so a client can start a default-set runner while the legacy `XCTestDevices` symlink still exists. Restore the redirect before exposing the servers, while buffering its diagnostics until after publication.</comment>

<file context>
@@ -559,6 +559,7 @@ export async function startDaemonRuntime(
       createOwnerScopedDeviceClaimReconciler(createDaemonRecoveryPlatformScope()),
       baseDir,
     );
+    await restoreLegacyXctestDeviceSetForDaemonStartup(logPath);
     // Arms the initial idle-reap timer: a daemon that starts and never
     // receives a request must still be able to reap itself.
</file context>
Fix with cubic

```

- `--ios-simulator-device-set <path>` constrains simulator discovery and simulator command execution via `xcrun simctl --set <path> ...`.
- The XCTest runner's `xcodebuild` phases resolve a scoped simulator in the same set through `-DVTSimulatorSetLocation=<path>`; `~/Library/Developer/XCTestDevices` is never redirected. If the selected Xcode no longer resolves the simulator that way, the runner start fails with `details.reason: "simulator_set_destination_not_found"`, and the error names the set and the selected Xcode version.

@cubic-dev-ai cubic-dev-ai Bot Sep 25, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The destination error does not always include the selected Xcode version; when the version probe fails, it reports (version unreadable). Qualify this documentation claim to avoid promising a detail that may be unavailable.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At website/docs/docs/commands.md, line 207:

<comment>The destination error does not always include the selected Xcode version; when the version probe fails, it reports `(version unreadable)`. Qualify this documentation claim to avoid promising a detail that may be unavailable.</comment>

<file context>
@@ -204,7 +204,8 @@ agent-device devices --platform android --android-device-allowlist emulator-5554
 
 - `--ios-simulator-device-set <path>` constrains simulator discovery and simulator command execution via `xcrun simctl --set <path> ...`.
-- The XCTest runner's `xcodebuild` phases resolve a scoped simulator in the same set through `-DVTSimulatorSetLocation=<path>`; `~/Library/Developer/XCTestDevices` is never redirected. If the selected Xcode no longer resolves the simulator that way, the runner start fails with `details.reason: "simulator_set_destination_not_found"`.
+- The XCTest runner's `xcodebuild` phases resolve a scoped simulator in the same set through `-DVTSimulatorSetLocation=<path>`; `~/Library/Developer/XCTestDevices` is never redirected. If the selected Xcode no longer resolves the simulator that way, the runner start fails with `details.reason: "simulator_set_destination_not_found"`, and the error names the set and the selected Xcode version.
+- On macOS, daemon startup puts back a `~/Library/Developer/XCTestDevices` that an older agent-device left redirected into a scoped set: it removes any symlink at that path, restores `XCTestDevices.agent-device-backup` when it exists, and records each step in `daemon.log`. A failed restore is recorded there too and does not stop the daemon or a runner start.
 - `--android-device-allowlist <serials>` constrains Android discovery/selection to comma or space separated serials.
</file context>
Suggested change
- The XCTest runner's `xcodebuild` phases resolve a scoped simulator in the same set through `-DVTSimulatorSetLocation=<path>`; `~/Library/Developer/XCTestDevices` is never redirected. If the selected Xcode no longer resolves the simulator that way, the runner start fails with `details.reason: "simulator_set_destination_not_found"`, and the error names the set and the selected Xcode version.
- The XCTest runner's `xcodebuild` phases resolve a scoped simulator in the same set through `-DVTSimulatorSetLocation=<path>`; `~/Library/Developer/XCTestDevices` is never redirected. If the selected Xcode no longer resolves the simulator that way, the runner start fails with `details.reason: "simulator_set_destination_not_found"`, and the error names the set and includes the selected Xcode version when it is readable.
Fix with cubic

): void {
assert.equal(envelope.code, 'COMMAND_FAILED');
assert.equal(envelope.message, 'xcodebuild build-for-testing failed');
assert.ok(envelope.message.startsWith('xcodebuild build-for-testing failed'));

@cubic-dev-ai cubic-dev-ai Bot Sep 25, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The shared assertFailureEnvelope check was relaxed from exact equality to a prefix startsWith for every row in the table, not just the scoped-set rows whose message now appends a suffix. Rows whose failure message stops matching the old exact text (appended detail, reworded base, wrapped text) now pass silently; the exact expected message is only pinned in the two dedicated tests. Keep a per-fixture exact expected message so the other rows retain their regression guard.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/platform-apple/src/runner/__tests__/runner-startup-failure-reasons.test.ts, line 177:

<comment>The shared `assertFailureEnvelope` check was relaxed from exact equality to a prefix `startsWith` for every row in the table, not just the scoped-set rows whose message now appends a suffix. Rows whose failure message stops matching the old exact text (appended detail, reworded base, wrapped text) now pass silently; the exact expected message is only pinned in the two dedicated tests. Keep a per-fixture exact expected message so the other rows retain their regression guard.</comment>

<file context>
@@ -173,7 +174,7 @@ function assertFailureEnvelope(
 ): void {
   assert.equal(envelope.code, 'COMMAND_FAILED');
-  assert.equal(envelope.message, 'xcodebuild build-for-testing failed');
+  assert.ok(envelope.message.startsWith('xcodebuild build-for-testing failed'));
   assert.equal(envelope.details?.reason, fixture.reason);
   assert.ok(
</file context>
Fix with cubic

assert.equal(error.details?.simulatorSetPath, SET_WITHOUT_UDID);
assert.equal(error.details?.xcodeVersion, STUBBED_APPLE_TOOLCHAIN.xcodeVersion);
assert.ok(
error.message.endsWith(

@cubic-dev-ai cubic-dev-ai Bot Sep 25, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: These assertions pin only the message tail via endsWith, so the middle of the user-visible sentence (xcodebuild found no simulator <udid> in ...) can drift without failing. The full message is deterministic here (udid, set path and Xcode version are all known), so assert the complete error.message to catch a wording regression in the sentence the user sees.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/platform-apple/src/runner/__tests__/runner-early-exit-diagnosis.test.ts, line 220:

<comment>These assertions pin only the message tail via `endsWith`, so the middle of the user-visible sentence (`xcodebuild found no simulator <udid> in ...`) can drift without failing. The full message is deterministic here (udid, set path and Xcode version are all known), so assert the complete `error.message` to catch a wording regression in the sentence the user sees.</comment>

<file context>
@@ -166,50 +175,76 @@ test('a session that never probed the device publishes no disk-image claim (#268
+  assert.equal(error.details?.simulatorSetPath, SET_WITHOUT_UDID);
+  assert.equal(error.details?.xcodeVersion, STUBBED_APPLE_TOOLCHAIN.xcodeVersion);
+  assert.ok(
+    error.message.endsWith(
+      `simulator set ${SET_WITHOUT_UDID} with Xcode ${STUBBED_APPLE_TOOLCHAIN.xcodeVersion}`,
+    ),
</file context>
Fix with cubic

@thymikee
thymikee merged commit 78c0ca4 into main Sep 25, 2026
20 of 21 checks passed
@thymikee
thymikee deleted the refactor/2935-xcodebuild-simulator-set branch September 25, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refuse the XCTestDevices redirect when an Xcode xcrun shim (simctl or devicectl) has an armed first-launch hook: scoped simulator set deletion

1 participant