Skip to content

fix(ios): a text field under the software keyboard reports focused in the snapshot - #2925

Merged
thymikee merged 7 commits into
callstack:mainfrom
okwasniewski:oskar/ios-keyboard-focus
Sep 25, 2026
Merged

thymikee merged 7 commits into
callstack:mainfrom
okwasniewski:oskar/ios-keyboard-focus

Conversation

@okwasniewski

@okwasniewski okwasniewski commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A text field the software keyboard was typing into never reported focused in an iOS snapshot.

Both focus reads asked for hasFocus, the focus engine's attribute (tvOS, keyboard navigation); the field under a software keyboard holds hasKeyboardFocus instead, which the text-entry readiness check already consults. snapshotHasFocus (RunnerTests+SnapshotAcquisition.swift, the recursive-tree producer) and elementHasFocus (RunnerTests+TvRemote.swift, the query-sweep and collapsed-tab element read) now share one reader, focusBool, which answers hasKeyboardFocus || hasFocus through KVC and reads a missing key as false. The wire shape is unchanged: focused is still true or absent.

The private-AX bridge is untouched: the AX server declares HasNativeFocus and FocusedApplications and no keyboard-focus attribute, so that fallback cannot see keyboard focus and gets no speculative code for it.

3 files, runner only. Android already reports focus.

Validation

Tested at the pushed head, on main. pnpm check:xctest-selection, pnpm check:packaged-runner-swift, and pnpm check:affected --run --base c31cc49f6 passed.

testFocusBoolReadsKeyboardFocusBesideTheFocusEnginesFocus (host lane) drives the shared reader from KVC fixtures: both keys in every combination, each key alone with the other absent, and no key at all. It passes on the booted iPhone 17 Pro simulator via xcodebuild test-without-building.

Live on that simulator, React Native app: after tapping name-input, snapshot --json reports the keyboard visible and exactly one focused node:

name-input: { focused: true, value: 'Ada Lovelace', type: 'TextField' }

Before the change the same snapshot reported no focused node.

Risk: a field with keyboard focus gains focused, so an unchanged comparison across a focus change now reports it.

Copilot AI lite review requested due to automatic review settings September 24, 2026 16:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Medium severity · 1 Low severity

Open (2)
What changed in this PR

Updates iOS snapshot focus detection to include software-keyboard focus.

Changes:

  • Combines hasFocus and hasKeyboardFocus in XCTest snapshots.
  • Requests and merges keyboard focus in the private AX bridge.
  • Extends the AX fixture shape.
File Description
apple/​runner/​AgentDeviceRunner/​AgentDeviceRunnerUITests/​UnitTests/​RunnerTests+AXRecoveryConformanceTests.swift Updated as part of this pull request.
apple/​runner/​AgentDeviceRunner/​AgentDeviceRunnerUITests/​RunnerTests+SnapshotAcquisition.swift Updated as part of this pull request.
apple/​runner/​AgentDeviceRunner/​AgentDeviceRunnerUITests/​RunnerAXSnapshotBridge.m Updated as part of this pull request.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI review requested due to automatic review settings September 24, 2026 17:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Medium severity

Open (1)
Resolved since last review (2)

Copilot AI review requested due to automatic review settings September 24, 2026 17:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

One or more issues must be addressed before approval.

Review effort: Lite
Findings: None

Resolved since last review (1)

@thymikee

Copy link
Copy Markdown
Member

Reviewed at 8a3ae74. The fix itself is a plausible read of snapshotAttributes() in RunnerAXSnapshotBridge.m: https://github.com/callstack/agent-device/blob/8a3ae74/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerAXSnapshotBridge.m#L378 asks Apple's private axAttributesForElementSnapshotKeyPaths:isMacOS: mapper to translate the new hasKeyboardFocus keypath, and that mapper is the only place this translation happens before the AX server request goes out.

The live validation in the PR description (name-input: { focused: true } on an iPhone 17 Pro simulator) exercises .recursiveTree, which SnapshotBackendCapturePlan.regularVisiblePlan tries first; .privateAX is the last-resort fallback and wasn't reached by that run. The new unit test, testPrivateAXBridgeReportsKeyboardFocusAsFocused, does exercise the real mapper through snapshotTree(withClient:), but only when it runs inside an XCTest process on a simulator, and none of the listed validation commands run it — build:xcuitest:ios only compiles. So nothing so far confirms whether Apple's private mapper actually resolves the new keypath to a real AX attribute, versus silently dropping it the way an earlier comment in this same file describes for an unmapped raw keypath. If it drops it, .privateAX keeps reporting a focused field as unfocused, reproducing this exact bug, but only on the fallback path that engages when the primary backends are degraded.

Smoke Tests builds and selectively runs the iOS runner XCTest regressions against a booted simulator, which is the mechanism that would exercise this test — it's currently queued/in progress, not failing. Please wait for that run to go green, confirming testPrivateAXBridgeReportsKeyboardFocusAsFocused passes on a real simulator, or run it directly with xcodebuild test-without-building -only-testing:AgentDeviceRunnerUITests/RunnerTests/testPrivateAXBridgeReportsKeyboardFocusAsFocused against a booted simulator. It'd also help to force .privateAX once live (for example via preferredBackend) so the fallback path itself gets a real end-to-end check, not just .recursiveTree.

I didn't have simulator access in this pass, so I couldn't run that test myself or confirm from the code alone whether the mapper drops or maps the new keypath. I also couldn't confirm from the diff whether the affected-XCTest-selection step actually includes this test for this PR's file set, only that the changed files match what that job builds and targets.

Copilot AI review requested due to automatic review settings September 24, 2026 19:03
@okwasniewski

Copy link
Copy Markdown
Contributor Author

[claude-fable-5-1] responding on behalf of Oskar

You were right to ask. The Smoke run showed the mapper's answer for the bridge's keypaths: Identifier, ElementType, Frame, Label, Value, HasNativeFocus, so hasKeyboardFocus is dropped, and strings over XCTAutomationSupport confirms why: the AX server declares XC_kAXXCAttributeHasNativeFocus and XC_kAXXCAttributeFocusedApplications and no keyboard-focus attribute at all. So the private-AX fallback cannot see keyboard focus, and my test was asserting something the server cannot provide.

7565615 changes the test to what holds: the request names the native focus, and a snapshot that arrives with keyboard focus set reads as focused through the shared OR. The bridge keeps the keypath with a comment saying why the mapper drops it. Ran xcodebuild test-without-building against the booted iPhone 17 Pro simulator: testPrivateAXBridgeReportsKeyboardFocusAsFocused, testPrivateAXRecoveryMatchesSharedFixture, and the policy test pass. The body now states the gap: the XCTest producers (recursive tree, query sweep) carry keyboard focus, the private-AX fallback keeps reporting the focus engine's focus alone.

I did not find a CLI flag that forces private-ax on a live capture, so the fallback path has the unit test and the attribute-name evidence rather than an end-to-end run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI review requested due to automatic review settings September 24, 2026 19:27
@okwasniewski
okwasniewski force-pushed the oskar/ios-keyboard-focus branch from 7565615 to a49fee9 Compare September 24, 2026 19:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@thymikee

Copy link
Copy Markdown
Member

Reviewed at a49fee9. Findings from the earlier pass (8a3ae74, #2925 (comment)) are unresolved.

testPrivateAXBridgeReportsKeyboardFocusAsFocused (apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+AXRecoveryConformanceTests.swift#L385) still feeds a fixture snapshot with hasKeyboardFocus set directly into RunnerAXSnapshotBridge. The real AX server never supplies that value — the mapper drops the keypath, as both your CI finding and the RunnerAXSnapshotBridge.m#L372 comment say — so this test exercises an OR branch production can't reach. The actual fix lives in snapshotHasFocus (RunnerTests+SnapshotAcquisition.swift#L422) and elementHasFocus (RunnerTests+TvRemote.swift#L56), and neither has a test. Reverting both leaves this test green, and its final contains "focus" assertion also passes at base, since @"hasFocus" was already requested before this change. Could a focusBool(_ object: NSObject) helper (returning hasKeyboardFocus || hasFocus) replace both call sites, with a unit test that drives it from an NSObject fixture carrying only hasKeyboardFocus? That would let a real test cover the behavior this PR claims to fix. Is the bridge-side OR and the kept hasKeyboardFocus keypath meant to pin something for a future mapper, or is it speculative code for a state that can't currently occur — and if the latter, should it come out along with its fixture test?

All 13 checks are green. CI job 107797328001 (Smoke Tests) confirms testPrivateAXBridgeReportsKeyboardFocusAsFocused ran and passed on a real booted simulator, which verifies the test executes, not that it verifies the right thing. I didn't have live device access and relied on that CI log rather than reproducing the run myself; I also didn't run strings to independently check the claim about the private AX server's attribute set, and take it as consistent with the observed mapper behavior rather than confirmed.

Nothing else is blocking once the fixed code path has its own test.

Copilot AI review requested due to automatic review settings September 25, 2026 08:08
@okwasniewski

Copy link
Copy Markdown
Contributor Author

[claude-fable-5-1] responding on behalf of Oskar

Agreed on both counts, done in 7e6ca6a.

  • The bridge-side OR and the kept hasKeyboardFocus keypath were speculative code for a state the AX server cannot produce, so they are out: RunnerAXSnapshotBridge.m and the conformance test are back to what main has.
  • snapshotHasFocus and elementHasFocus now both call one focusBool(_ object: NSObject) (in RunnerTests+SnapshotAcquisition.swift): hasKeyboardFocus || hasFocus through KVC, a missing key reads as false.
  • testFocusBoolReadsKeyboardFocusBesideTheFocusEnginesFocus (host lane, in RunnerTests+TvRemoteTests.swift) drives that reader from NSObject fixtures: keyboard focus only (true), native focus only (true), neither (false), an object exposing only the keyboard key (true), and a bare NSObject (false). Reverting either call site to hasFocus alone fails it. It passes on the booted iPhone 17 Pro simulator via xcodebuild test-without-building.

Gate on the head: check:xctest-selection, check:packaged-runner-swift, check:affected --run --base 6428c5485 all green. The diff is now 3 runner files.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

… the snapshot

Both snapshot producers read hasFocus, the focus engine's attribute (tvOS,
keyboard navigation), so the field a software keyboard was typing into
never reported focused. Keyboard focus is hasKeyboardFocus, which the
text-entry readiness check already consults; the XCTest producer and the
AX bridge now report either as focused.
…d focus too

The query-sweep producer and the collapsed-tab nodes read focus through
elementHasFocus, which still asked for hasFocus alone; it now reads the
same pair as the snapshot producers. The AX conformance fixture can hold
keyboard focus on one node, and a test proves the bridge asks for the
attribute and reports that node focused.
…sts, and the keyboard-focus test asserts it asked for keyboard focus
XCTAutomationSupport declares HasNativeFocus and FocusedApplications and
no keyboard-focus attribute, so the mapper drops the hasKeyboardFocus
keypath and the private-AX fallback reports the focus engine's focus
alone; the XCTest producers carry keyboard focus. The test now asserts
the request names the native focus and that a snapshot arriving with
keyboard focus reads as focused, and the bridge comment says why the
keypath stays.
…re the fix is

The private-AX bridge cannot see keyboard focus (the AX server declares
no such attribute), so its OR and the extra keypath were code for a
state that cannot occur; they are gone, and the bridge is what it was on
main. The two reads that carry the fix, snapshotHasFocus and
elementHasFocus, now share one focusBool reader (hasKeyboardFocus, else
hasFocus, through KVC), and a host-lane test drives that reader from
fixtures carrying only keyboard focus, only native focus, neither, and
no key at all.
@okwasniewski
okwasniewski force-pushed the oskar/ios-keyboard-focus branch from 7e6ca6a to 2ff4dcf Compare September 25, 2026 08:10
Copilot AI review requested due to automatic review settings September 25, 2026 08:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Copilot AI review requested due to automatic review settings September 25, 2026 08:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@thymikee

Copy link
Copy Markdown
Member

This looks ready. The fix addresses the focus report correctly and the code checks out at 279ccef.

Not blocking: the focusBool doc comment says one reader serves every producer, but the AX bridge still reads hasFocus only for the snapshot focused field, so scoping the wording to the XCTest producers (and maybe noting the bridge's hasFocus-only behavior) would keep it accurate — take it or leave it.

Checks are still queued or in progress with nothing failing so far; the changed files are Swift-runner-only, which overlaps the route Smoke Tests exercises, so a failure there would need to be checked against this diff, but there is nothing to attribute yet.

Once Smoke Tests and Coverage finish, this is ready for a human review.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 25, 2026
Copilot AI review requested due to automatic review settings September 25, 2026 08:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@okwasniewski

Copy link
Copy Markdown
Contributor Author

[claude-fable-5-1] responding on behalf of Oskar

Took the wording nit in c7dece5: the focusBool comment now names the XCTest producers it serves (recursive tree, query sweep, collapsed tabs) and says the private-AX bridge reads hasFocus alone because the AX server declares no keyboard-focus attribute. Comment-only change; check:xctest-selection, check:packaged-runner-swift, and check:affected --run --base c31cc49f6 green on the head.

@thymikee
thymikee merged commit 8df1010 into callstack:main Sep 25, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants