feat(engine): pass the measured keyboard state to executors

[pvedula7]

What and why

• Follow-up to the review on fix(decision): offer dismiss_keyboard while an on-screen keyboard is showing #843: the mobile engine reads agent-device's measured keyboard (raw.keyboard.kind), but that never reached ExecutorObservation, so the decision executor could only guess from keyboard/key nodes in the tree.
• EngineSnapshot and ExecutorObservation gain an optional keyboardVisible?: boolean. true means the engine measured a keyboard on screen, false means it measured none, and absent means it did not measure. The session copies it onto the observation and the feed hands it to the executor (packages/e2e/src/engine/session.ts, packages/e2e/src/agent/observation-feed.ts).
• It's a boolean, not agent-device's band. dismiss_keyboard only needs to know whether a keyboard is up, and the band's frame comes back in the producer's own coordinate space (a 402-wide iPhone reported y: 4583), so I didn't want it in the contract.
• The mobile engine sets it from the capture's keyboard fact: visible gives true, absent gives false, and unmeasurable or no fact leaves it out. Its gesture dismissal reads the same helper (packages/mobile/src/surface.ts).
• The decision executor reads keyboardVisible first and falls back to fix(decision): offer dismiss_keyboard while an on-screen keyboard is showing #843's tree probe when it's absent:

if (verbs.has('dismissKeyboard') && (observation.keyboardVisible ?? keyboardShowing(observation.tree))) controls.add('dismiss_keyboard');

• The page fingerprint in packages/decision/src/elements.ts hashes the measured state next to the pixels, with true, false and absent kept apart. A dismissal that leaves the tree as it was still counts as a page change, so it does not feed the stall guard.

Android: needs callstack/agent-device#3240

The review guessed Android is where this matters. I checked on a Pixel 8 emulator (API 35), and the field doesn't change anything there:

• Only agent-device's iOS runner publishes the keyboard fact. Its contract says so (keyboard?: SnapshotKeyboardBandFact is documented as "iOS:"), and Android snapshots carry no keyboard key. So on Android, keyboardVisible is always absent and the executor falls back to the tree.
• On an emulator, agent-device swaps in its own headless test IME for the length of the session (testIme defaults on for kind === 'emulator'), and the mobile engine doesn't opt out. No soft keyboard is ever drawn, so nothing covers Continue and dismiss_keyboard correctly isn't offered.
• With a real IME (agent-device open --no-test-ime, the case on a physical phone), Gboard does show up in the tree: 127 nodes under com.google.android.inputmethod.latin. But they're typed android.widget.FrameLayout and TextView, never keyboard or key, so fix(decision): offer dismiss_keyboard while an on-screen keyboard is showing #843's probe misses them too. Only agent-device keyboard status (dumpsys input_method) said visible: true.

callstack/agent-device#3240 adds the fact to Android captures: the band is the TYPE_INPUT_METHOD window's bounds, which its snapshot helper already records, so there's no extra adb call. Gboard up reads visible with the window frame, a dismissed keyboard reads absent, and the headless test IME reads absent. Once that ships in an agent-device release, this PR picks it up on Android with no change here.

Verified

Ran it locally: yes

• The live runs below were on the layout before feat(decision): full action grammar, none fallback, and vision through the OpenAI Decisions API #965. The port is covered by unit tests: decision (measured state over the tree, tree fallback when absent, fingerprint moves on the measured state alone), mobile (visible, absent, unmeasurable and no fact map to true, false, absent, absent), and e2e (the executor gets what the engine reported).
• Surface: the mobile benchmark's Sequential Onboarding scenario (the email field auto-focuses, so the keyboard covers Continue), using the built packages. I wrapped decisionExecutor in a throwaway config with a scripted decision model that picks dismiss_keyboard when it's offered and done otherwise, then logged every observation's keyboardVisible and the tree's keyboard/key node count. No API key was needed. The test fills the email, runs agent.act('Hide the keyboard so the Continue button can be reached'), then taps continue-button with a locator.
• e2e run --config e2e.kbprobe.config.ts --target ios-simulator --video on an iPhone 17 Pro simulator: passed on both runs. The measured state and the tree agree on this screen, and the control is offered and dispatched:

observation   keyboardVisible: true   tree keyboard/key nodes: 34
operation     offered: tap, submit, scroll_up, scroll_down, back, dismiss_keyboard, done, failed, blocked -> dismiss_keyboard
observation   keyboardVisible: false  tree keyboard/key nodes: 0
operation     offered: tap, submit, scroll_up, scroll_down, back, done, failed, blocked -> done

• Same test with --target android-emulator on a Pixel 8 emulator: passed. No keyboard is drawn (headless test IME), the field is absent, and dismiss_keyboard isn't offered:

observation   keyboardVisible: (absent)  tree keyboard/key nodes: 0
operation     offered: tap, submit, scroll_up, scroll_down, back, done, failed, blocked -> done

• Android with a real IME, using the agent-device CLI directly (open --no-test-ime, open the scenario, snapshot --json): Gboard is on screen, the capture has no keyboard fact, and none of its 127 nodes is typed keyboard or key.

• On iOS the measured state never disagreed with the tree on this benchmark, so I don't have a before/after where fix(decision): offer dismiss_keyboard while an on-screen keyboard is showing #843 alone gets it wrong. On iOS this makes the check use the runner's measurement and stop depending on the tree. The unit tests cover the two cases the tree can't handle: a measured keyboard the tree doesn't list, and key nodes left behind after a measured absence.

Checklist

• Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
• Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).
• Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
• Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures.
• Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile.

[cubic-dev-ai]

All reported issues were addressed across 21 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[okwasniewski]

Thanks! I merged main into this branch and renamed scriptedEvaluation to scriptedDecision in the decision tests (#916 renamed the helper). Typecheck and unit tests pass in decision, e2e, mobile and web. This merges after #843.

[okwasniewski]

Thanks for this. #965 just landed and restructured the decision package, and main also changed packages/mobile/src/surface.ts, so this conflicts in both. The engine side is still wanted: the mobile engine reads agent-device's keyboard band for dismissKeyboard, but executors never see it.

Could you rebase onto main once #843 is ported, and fit the decision side to the new layout? The fingerprint in elements.ts now also hashes pixels, so add the measured keyboard state alongside that, and use it in place of the tree check for dismiss_keyboard when the engine reports it.

[pvedula7]

Rebased onto main now that #843 and #844 are in; it is just the three keyboard-state commits now. pnpm check and pnpm test pass.

[pvedula7]

Rebased onto main after #1019; only the docs needed re-fitting to the rewritten pages. pnpm check and pnpm test pass.

[okwasniewski]

hey, @pvedula7 I bumped agent-device on main - can u retest everything on a live run and report back here?

[okwasniewski]

Thanks for the rebases! A few things before this can land:

• Retest on main's agent-device (0.21.25). callstack/agent-device#3240 shipped, so Android captures now carry the keyboard band. A live iOS and Android run (a real IME on Android) would be great, with the output here.
• Docs: please drop the new paragraph in reference/engine.mdx (it explains agent-device internals and still says Android has no measurement). Keep the field's doc comment, and cut executors.mdx and decision-models.mdx to one sentence each, saying what users see. Please also merge the two changesets into a short one that skips the iOS-runner detail.
• Small: fingerprint() could take the observation instead of 5 positional args, and the keyboardVisible checks could be the same in every layer (=== undefined).
• packages/mobile/tests/unit/engine.test.ts conflicts with main again.