Repository navigation
feat(ios): publish the keyboard band as a capture fact; guard prefers it over the tree heuristic #2660
Description
Why
The keyboard band is measured twice, in two different ways, and only one of them is reliable.
- The runner already has the correct band.
visibleKeyboardFrame(app:)inapple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Keyboard.swift(~line 136) readsapp.keyboards.firstMatch.frame, which XCTest reports in the app's own orientation space. The scroll planner uses it forkeyboardAvoided/keyboardMinY(RunnerScrollViewportPolicy.swift) and keyboard dismissal uses it too. PR fix(ios): publish captured keyboard geometry in the app's orientation space #2653 confirmed on device that this query answers the landscape band aty 198where the tree reported a strip down the left edge. - The daemon rebuilds the same band from tree nodes.
packages/contracts/src/tap-keyboard-occlusion.ts(339 lines) finds anchor nodes, collects key rects, checks tiling and column runs, checks the docked bottom edge, and applies a width-versus-height guard to reject geometry that arrived unnormalized.src/commands/interaction/runtime/keyboard-occlusion.ts(107 lines) wraps it for every acting path. The golden tablecontracts/fixtures/tap-keyboard-occlusion-policy.jsonpins the heuristic.
The tree-derived rule exists because a capture carries no keyboard fact. Publishing one from the producer that can measure it collapses the iOS half of the heuristic to a point-in-rect check, removes the orientation dependence from the guard entirely (the guard would no longer care whether the tree was rotated), and lets the ADR 0004 paragraph "What stays unnormalized, and why the consumer rule stays" go.
Task
- Carrier. Add an optional, producer-declared keyboard fact to the snapshot result. Suggested shape on the wire and in
@agent-device/kernel/snapshot:A producer that publishes nothing iskeyboard?: | { kind: 'visible'; frame: Rect } // app orientation space, same space as node rects | { kind: 'absent' } // producer looked and found none | { kind: 'unmeasurable'; reason: string } // producer cannot measure on this path
unmeasurableby construction. Do not make it required onSnapshotResult: three client-side fallbacks rebuild results from a bareBackendSnapshotResult(see the refactor(ios): prune converged snapshot paths and close release gates #2199 prune notes indocs/adr/0004…, "Exactly-once made unrepresentable"). - Runner. Read the band once per capture in
snapshotFast/snapshotRaw(RunnerTests+Snapshot.swift) with the existingvisibleKeyboardFrame(app:)and stamp it onDataPayload. It is one moreXCUIElementQueryper capture; measure it (the scroll path already pays it per gesture) and report the number in the PR. Read it in the same main-thread hop as the viewport where possible so a rotation cannot land between the two reads.readAppleSnapshotResultinpackages/platform-apple/src/runner/snapshot-presentation.tscarries it through. - Bridge. In portrait the keyboard window's subtree is already in the app's space, so the decoder in
packages/platform-apple/src/snapshot-source/tree.tscan derivevisiblefrom theUIRemoteKeyboardWindowkey plane exactly astap-keyboard-occlusion.tsdoes today; after fix(ios): publish captured keyboard geometry in the app's orientation space #2653 a landscape capture with the keyboard up is refused by the bridge and served by the runner, so the bridge never has to answer that case. Publishunmeasurablewhen it cannot. - Guard.
assertTapTargetClearOfVisibleKeyboardprefers the fact:visible→ point-in-rect againstframe;absent→ no occlusion;unmeasurableor no fact → today's tree-derived rule, unchanged, which remains the path for Android (isAndroidInputMethodNode) and forappium-source/limrun-ios-tree. KeepresolveKeyboardTapOcclusion's public shape so the interaction-guarantee contract (packages/contracts/src/interaction-guarantees.ts, ADR 0011) does not change. - Docs. Update ADR 0004's "What stays unnormalized" paragraph and the
tap-keyboard-occlusion.tsheader comment to say the iOS runner path no longer reaches the width rule.
Acceptance criteria
-
snapshot --jsonon the XCTest runner path carrieskeyboard: { kind: 'visible', frame }with the keyboard up and{ kind: 'absent' }with it down, in portrait and in landscape, on iPhone 17 Pro (iOS 26.2 Simulator) and on one physical iPhone if available. - The iOS reports the landscape iPhone keyboard's rects in the keyboard's own rotated space #2612 scenario (landscape, keyboard up,
press @<ref behind keyboard>) is refused withtap_keyboard_occludes_targetthrough the fact path (assert with a unit test that the tree-derived rule is not consulted when the fact isvisible), and a target above the band still presses. - Portrait behaviour is byte-identical on the runner path for
snapshot -i --jsonapart from the new field (compare againstmainon the same screen). - Android behaviour unchanged:
packages/contracts/src/tap-keyboard-occlusion.test.tsand the Android integration lanes green with no fixture edits. - Bridge path: portrait keyboard-up capture publishes
visiblewith the same band the tree rule computed onmain(assert equality in a unit test over a recorded bridge tree). - Per-capture cost of the extra keyboard query on the runner path reported in the PR (p50 over 10 warm captures, keyboard up and down).
-
pnpm check:affected --rungreen; Swift runner units green underAGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS=1(seescripts/build-xcuitest-apple.sh);pnpm test:ios-snapshot-differentialgreen.
Non-goals
Deleting the tree-derived rule. It stays for producers that publish no fact. Changing the keyboard-avoidance rule in the scroll planner (it already uses the live frame).
Related: #2612, #2653, #2589 (the guard), #2622 (find focus/type without the guard), ADR 0004, ADR 0011.
Activity
feat(ios): publish the captured keyboard band as a producer-declared …