Research agent-device CI hardening for the observed infra flakes

[thiagobrez]

Part of #53

Question

Beyond the systematic iOS 26 failure, the remaining device-job failures are infrastructure flakes, each with a different cause: iOS runner ... already owned by another agent-device daemon, RUNNER_BUSY: still finishing a previous command that exceeded its execution watchdog, artifact restored but runner did not connect: dyld: Library not loaded: /usr/lib/libcurl.4.dylib, plus adb reverse-listener and emulator-boot flakes on Android.

The repo pins agent-device 0.20.10 (ADR 0006). What do the agent-device docs/repo (github.com/callstack/agent-device) recommend for CI daemon lifecycle, runner ownership, and artifact-restore issues — and do releases newer than 0.20.10 fix any of the observed errors? Check changelogs/issues for the three iOS error signatures, recommended CI setup (daemon-per-job vs shared, cleanup hooks, watchdog config), and whether the current prepare-agent-device-ios-runner.mjs dance (alert seeding, exit-75 sentinel, simulator erase between attempts) matches upstream guidance or works around already-fixed bugs. Deliver: a findings doc mapping each observed flake signature to a recommended remedy (upgrade, config change, or genuine local workaround still needed).

[thiagobrez]

Resolution

Full findings: docs/research/agent-device-ci-hardening.md (branch research/agent-device-ci-hardening).

Headline: the pinned 0.20.10 is the latest published agent-device release — no upgrade exists today that fixes any observed flake. Upstream main has ~115 unreleased PRs, several of which matter.

Per flake signature:

Signature	Verdict
iOS runner ... already owned by another agent-device daemon	Upgrade at the next upstream release. Live-owner rejection is by design in 0.20.10 (global lease dir, takeover only for dead owners/same state dir); daemons and warm runners self-idle after 5 min — that window is the collision. Merged-but-unreleased fixes: #2160 (warm-runner reclaim under claim authority), #2162 (device release --stale), #2105 (retried open supersedes the aborted attempt's claim — literally our attempt-1/attempt-2 failure shape). Meanwhile keep the daemon-stop hygiene; optionally tune AGENT_DEVICE_DAEMON_IDLE_TIMEOUT_MS / AGENT_DEVICE_IOS_RUNNER_IDLE_STOP_MS or use close --shutdown.
RUNNER_BUSY ... exceeded its execution watchdog	Keep local retry. Fixed ~30 s runner-side watchdog, not operator-configurable; error is retriable by design and all known hardening (#1107, #1126, #1248) already shipped ≤ 0.20.10.
artifact restored but runner did not connect: dyld ... libcurl.4.dylib	Local workaround needed + file upstream (unreported: zero upstream hits). Cache key omits the macOS host build and the rebuild fallback only fires on timeout-shaped failures, so dyld failures are hard errors. Purge ~/.agent-device/apple-runner in the retry/reset path so attempt 2 rebuilds instead of restoring the bad artifact.
adb listener tcp:NNNN not found	Not agent-device. All Android scenarios route to Detox; the string is adb stderr via Detox's ADB.js. agent-device already tolerates it on its own reverse path. Existing retry covers it.
Android emulator-boot flakes	Not agent-device. Owned by reactivecircus/android-emulator-runner; keep local config/retry.

Local workaround audit: the prepare ios-runner preflight and the stop-the-prepare-daemon dance match upstream's documented CI guidance verbatim (help workflow / help prepare). The alert-seeding relaunch, exit-75 sentinel, simulator erase, PID-derived Detox ports, and IPv4 pinning have no upstream equivalent — genuinely local, none obsoleted by a released fix. There is no upstream help ci topic or daemon-per-job doc beyond the above.

[thiagobrez]

Research complete; findings committed on research/agent-device-ci-hardening.