tvOS device.io.focus: accessibility-identity focus via bounded Siri Remote traversal#3
Closed
setoelkahfi wants to merge 4 commits into
Closed
tvOS device.io.focus: accessibility-identity focus via bounded Siri Remote traversal#3setoelkahfi wants to merge 4 commits into
setoelkahfi wants to merge 4 commits into
Conversation
The DeviceKit XCUITest runner now builds for tvOS, so the automation stack can drive the tvOS Simulator. - Add a `devicekit-tvos` scheme and tvOS target to the Xcode project, plus a `scripts/patch-tvos-runner.sh` helper and Makefile targets that build and patch the tvOS runner. - On tvOS, register a `device.io.button` handler that maps button names to `XCUIRemote.Button` presses (up, down, left, right, select, menu, home, playPause). tvOS navigates by focus, not touch. - Keep the touch, keyboard, and orientation handlers iOS-only in the JSON-RPC dispatcher, behind `#if os(iOS)`. - Read the screen size from `UIScreen.main.bounds` on tvOS, since there is no SpringBoard to query. - Guard the remaining iOS-only handlers and helpers with `#if os(iOS)` so the shared sources still compile for both platforms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a compiled brandassets catalog (imagestack/imagestacklayer) for the tvOS UI test runner, wire up ASSETCATALOG_COMPILER_APPICON_NAME and CFBundleDisplayName on the devicekit-tvosUITests target, and extend patch-tvos-runner.sh to copy the compiled Assets.car and set CFBundleIcons/CFBundleDisplayName on the Runner.app so the tvOS test runner now shows the same Device Kit icon as the iOS one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mirror the iOS real-device flow for tvOS. patch-tvos-runner.sh now takes a simulator|device argument so it can pull the Swift Testing support libraries from AppleTVOS.platform for device builds; the new tvos-ipa-unsigned Makefile target builds the devicekit-tvos runner for generic/platform=tvOS, patches in the Device Kit display name and icon, and packages an unsigned devicekit-tvos-runner.ipa for re-signing at install time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a tvOS-only device.io.focus JSON-RPC handler that selects an element by accessibility identifier/label and drives Siri Remote focus with a bounded (<=50-move) traversal, returning the focused element on success or a focusNotFound (-32002) error carrying the last focused element. - New IOFocusMethodHandler (#if os(tvOS)); registered in the dispatcher - Re-reads the target frame each move to handle tvOS focus scrolling - Early-exit on no-progress (dead end / oscillation) - Guard SpringBoard-specific DumpUI normalisation behind #if os(iOS) so tvOS dumps the foreground app's own accessibility hierarchy - Remove throwaway Viaplay exploration scaffolding + unused import Implements: SPECS/quality/real-tvos-device-support/SPEC.md (Milestone 4) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Owner
Author
|
Superseded by upstream PR mobile-next#54. Keeping the branch as the source branch for upstream review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a tvOS
device.io.focusJSON-RPC command to the DeviceKit XCTest runner so agents can deterministically move Siri Remote focus to an on-screen element selected by its accessibility identity (identifier/label), instead of guessing remote sequences or interpreting screenshots.This is part of a three-repo change (initial PRs target the
setoelkahfiforks; upstreammobile-nextPRs are deferred until the three-repo E2E is green).What's in this PR (Milestone 4)
device.io.focushandler (#if os(tvOS)): selects an element byidentifier/label, then drives focus with a bounded (≤50-move) Siri Remote traversal. The direction heuristic re-reads the target frame each move to handle tvOS focus scrolling, and exits early on no-progress (dead end / oscillation).SourceTreeElement.focusNotFounderror (JSON-RPC code-32002) whoseerror.datacarries the last focused element, plus a message naming the requested selector and the move count.#if os(iOS)so a tvOS accessibility dump uses the foreground app's own root hierarchy. iOS behaviour is unchanged.Validation
make tvos-ipa-unsignedbuilds the tvOS UITests target and producesbuild/export/devicekit-tvos-runner.ipa.project.pbxprojvalidated withplutil -lint.Guardrails
DEVICEKIT_LISTEN_HOST=127.0.0.1); never0.0.0.0/ never LAN.Implements: SPECS/quality/real-tvos-device-support/SPEC.md
Related PRs (three-repo change)