Skip to content

test(ios-smoke): budget cold runner readiness - #3026

Merged
thymikee merged 1 commit into
mainfrom
codex/ios-smoke-cold-runner-readiness
Sep 28, 2026
Merged

thymikee merged 1 commit into
mainfrom
codex/ios-smoke-cold-runner-readiness

Conversation

@thymikee

@thymikee thymikee commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Summary

Give the first iOS smoke wait after open --relaunch a 60-second budget. That wait can start the XCTest runner, whose declared startup budget is 45 seconds, before it makes its first capture. The shared assertion helper accepts an optional budget; every other wait keeps its 10-second default. Two test-harness files changed; production behavior is unchanged.

The same smoke step timed out on #3009, #3015, and #3006. Both #3009 failures ended after one 10-second poll in runner-start with zero readable captures. A failure artifact then captured a healthy XCTest tree containing the awaited Agent Device Tester label. #3006 passed the identical job on rerun. This bounds the cold-start allowance to the actual runner startup contract while preserving a real failure if startup exceeds it.

Validation

At b3b0fed9e8fbe48cb605c4ede764f454b582339c: pnpm install --frozen-lockfile, pnpm build, pnpm format, and pnpm check:affected --run passed. The affected selector ran format, lint, typecheck, and fallow; vitest related found no test files for these live harness modules. iOS CI smoke passed on this head, including fixture-backed iOS simulator E2E smoke. First #3009 failure; second #3009 failure.

@github-actions

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.87 MB 4.87 MB 0 B
Package (unpacked) 4.87 MB 4.87 MB 0 B
Package (download) 1.46 MB 1.46 MB +3 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 27.0 ms 27.1 ms +0.1 ms
CLI --help 80.8 ms 82.5 ms +1.7 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Re-trigger cubic

@thymikee

Copy link
Copy Markdown
Member Author

b3b0fed is a small, self-contained fix: it widens the cold-runner wait in the iOS smoke test to line up with runner startup, and there's nothing in the code that needs to change before merge.

Not blocking: COLD_RUNNER_WAIT_TIMEOUT_MS at https://github.com/callstack/agent-device/blob/b3b0fed/test/integration/ios-simulator-e2e/live-automation-scenario.ts#L22 is tied to RUNNER_STARTUP_TIMEOUT_MS (https://github.com/callstack/agent-device/blob/b3b0fed/packages/platform-apple/src/runner/runner-startup-transport.ts#L44) only through a comment, so could it instead be derived from the exported constant plus a margin so the two move together, or is the comment enough — either way this can be taken or left.

Smoke Tests (iOS) was still queued when I checked this, and that job is what runs the observeFixtureHome wait this PR widens, so its green run on b3b0fed is the evidence this change actually needs. The original failure was intermittent, so one green run will show the change works but won't by itself prove the flake is gone for good. Smoke Tests going green on b3b0fed is what this now waits on.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 28, 2026
@thymikee
thymikee merged commit c8c28b0 into main Sep 28, 2026
19 checks passed
@thymikee
thymikee deleted the codex/ios-smoke-cold-runner-readiness branch September 28, 2026 13:43
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-28 13:44 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant