Android input through scrcpy's control channel #1018

Part of #1013.

Scope

After this PR, a person holding an Android lease taps, swipes, scrolls, types, and presses Home, Recents, Power and Back through WS /v1/leases/{id}/input, on an emulator and on an enrolled phone. Before it, #1017's socket answers inputUnavailable: true for every Android device. Rotation is not part of it: #1017's session drops 0x07, and agents rotate through agent-device.

In short

An Android lease's device input is one control-only scrcpy server: the same vendored server the stream uses, started with video off and control on. Simlock turns each serve-sim message into a scrcpy control packet. Positions arrive as fractions of the screen and leave as pixels, so the device input reads the screen's size as it is now, in its current rotation, at open and every 2 s after; no new read starts while one is running. When the size changes (an agent rotated the device), it reports the new size, and #1017 sends a new 0x82.

flowchart LR
  S[input module, #1017] --> P[device input port, #1017]
  P --> A[Android device input NEW]
  A -- dumpsys window displays, every 2 s --> D[(device)]
  A -- control-only scrcpy, control socket --> D

• control-only scrcpy: com.genymobile.scrcpy.Server 5.0.1 with tunnel_forward=true video=false audio=false control=true power_on=false send_device_meta=false send_dummy_byte=true cleanup=true. It reads control packets on one socket and sends no picture.
• screen size: the cur=<w>x<h> that adb shell dumpsys window displays prints for display 0, which is the size in the current rotation (F17). A portrait 1080×2400 phone turned to landscape reads cur=2400x1080.
• A tap at {"type":"begin","x":0.5,"y":0.25} on cur=1080x2400 becomes a touch-down packet at pixel (540, 600), with screen size 1080×2400 in the packet.
• scroll: wheel units are clamp(round(d × 16), −16, 16) per axis, sent at x, y when given, else at the screen centre. Sign, a choice made here: positive dy (content down, F19) sends a positive vertical scroll; positive dx (content right) sends a negative horizontal scroll. {dx:0, dy:0.1} sends vscroll +2.
• keys: the table in F21 covers letters, digits, Enter, Escape, Backspace, Tab, Space, the punctuation row, the arrows, and the modifiers 0xe0–0xe7. A modifier goes out as its own key event and is also held in the meta state (F22) of every later key until its up.
• unavailable: a phone whose adb check is closed. The device input resolves {unavailable: <the gate's message>}, and Input socket for a lease #1017 sends inputUnavailable: true. Failed: anything else that stops the open (no size, the server does not start) rejects, and Input socket for a lease #1017 closes with 1013.

If this goes wrong, a person sees taps land in the wrong place, or keys type the wrong letters.

Technical spec

Facts

Commit e1f2f43.

Modules touched

Searches: screenCapture, createAndroidScreenCapture, ScreenTarget, screenTargetFor, input, KEYCODE_ in src/; snapshot in e2e/.

• src/drivers/android/input/ (new): the Android device input: server args, packets, the key table, the size read.
• src/drivers/android/index.ts: sets the driver's input (the member Input socket for a lease #1017 adds to Driver, next to screenCapture), with the same targets as the capture (F6).
• src/drivers/android/screen/: exports what the input reuses: ScreenTarget, the push, and openDeviceSocket. No behaviour change.
• src/drivers/android/AGENTS.md, src/drivers/android/screen/AGENTS.md: "The wake is the only key sent" now holds for the stream only.
• website/src/content/docs/docs/using-devices/watching-the-screen.mdx: what input does on Android.
• e2e/slow-agent-device.test.ts: one tap case in the shared suite, skipped on iOS until iOS simulator input through the HID injector #1019 turns it on. e2e/slow-android-physical-manual.test.ts: one tap with a person at the phone.

flowchart LR
  core[core driver catalog, #1017] --> drv[android/index.ts *]
  drv --> inp[android/input NEW]
  inp --> scr[android/screen * exports]
  inp --> ph[physical-handler: 5037 check]

sequenceDiagram
  participant M as input module
  participant I as Android device input
  participant D as device
  M->>I: openInput(device)
  I->>D: dumpsys window displays (cur=WxH)
  I->>D: push jar, start control-only server
  I->>D: connect control socket (dummy byte)
  I-->>M: device input; onConfig {width, height}
  M->>I: 0x03 touch begin {x:0.5, y:0.25}
  I->>D: touch packet at pixels
  Note over I,D: phone 5037 check closed
  I-->>M: {unavailable: gate message}
  Note over I,D: server does not start in 15 s
  I-->>M: open rejects (1013)

Surface and extension points

createAndroidDeviceInput(options) returns the driver's input, the implementation of #1017's device input port (feature Shared values, "Device input port"). Its options mirror AndroidScreenCaptureOptions: clock, serverPath, targetFor. It is one of two implementations beside #1019's; the port and the Driver.input member are #1017's.

Contract and event changes

none. Searched src/contract/ for input shapes: the messages are #1017's core/input.ts. No event.

Other code on the same state

Searches: scrcpy_, simlock-scrcpy-, stay_awake, KEYCODE_, user_rotation in src/ (6 hits, all in src/drivers/android/screen/).

• The stream's scrcpy session: both run a scrcpy server on one device at once. Different scid, socket and jar (F2). Neither reads the other's socket. Both clean up their own jar.
• A phone's stay_awake: only the stream's server sets it (F1). The input server does not, so its clean-up restores nothing it did not set.
• The phone wake before a stream: the input does not wake the screen and starts with power_on=false (F10). A tap on a dark screen does nothing.
• agent-device on the same device: both inject input, and agent-device may rotate. Whichever input lands last wins; that is the take-over the feature asks for. After a rotation, taps use the old size for at most 2 s, until the next size read.

Failure modes

A worker restart mid-session leaves a control-only server that cleanup=true ends with its socket. The device input changes no device setting, so nothing is left to put back. #1017 shares one device input per lease, so two opens never race here.

Rules in play

• safety rule 1, "never runs a command on one that is not enrolled": the input reaches only a leased, enrolled phone through screenTargetFor.
• safety, physical Android: "Right before each physical command, Simlock checks that server's adb protocol version": every run, spawn and connect, the size reads included, goes through the phone's target (F4).
• architecture rule 2, "iOS and Android are encapsulated in their own driver modules": key tables and scrcpy packets stay in src/drivers/android/.
• architecture rule 11, "Every wait that crosses a process boundary is bounded": each adb call, the push, the start and the close have a limit (Failure modes).
• src/drivers/android/screen/AGENTS.md, "No other option changes a device setting": the input sets power_on=false and no stay_awake.

Tests

Seam: createAndroidDeviceInput with a fake target and the scrcpy fixtures, on a FakeClock; model on src/drivers/android/screen/session.test.ts.

• Touches and scrolls on Android reach scrcpy as pixel positions for the device's size: begin/move/end at fractions on cur=1080x2400 and on cur=2400x1080; scroll {dx:0, dy:0.1} sends vscroll +2 at the screen centre; scroll with x, y sends it at that point; a position at fraction 1 lands on the last pixel (x = width - 1); a scroll delta past 16 wheel units is clamped to 16 (decided in delivery, pending the maintainer).
• Keys and buttons on Android reach scrcpy as keycodes: home 3; app_switcher 187; lock 26; back 4; HID letters a (0x04) → 29, digits 1 (0x1e) → 8 and 0 (0x27) → 7, Enter (0x28) → 66, Backspace (0x2a) → 67; Shift held then a gives A (meta state): Shift down (0xe1) sends keycode 59 down, then a goes with metaState 0x41, and Shift up clears it; an unknown usage (0x3a, the F1 key) dropped.
• An Android device input starts a control-only scrcpy, and a phone passes the adb check first: emulator (args as In short); phone with the check open; phone with it closed → unavailable with the gate's message, nothing started; scrcpy fails to start → rejected; screen size read in the current rotation, and a change after an agent rotates reported (cur=1080x2400, then cur=2400x1080 at the next 2 s read → one onConfig); close after a successful open stops the size reads, ends the server and removes the jar; no second size read starts while one runs; a failing size read keeps the last size and is logged once (decided in delivery, pending the maintainer).

Done when

• The three Tests lines pass in pnpm test.
• On an emulator, e2e/slow-agent-device.test.ts opens a lease and the Appduct fixture through agent-device, opens WS /v1/leases/{id}/input, gets 0x83 and 0x82, taps the centre of tab-status as agent-device's snapshot places it (as a fraction of the screen size in the same snapshot --json), and the next snapshot shows the Status tab's screen (slow lane, Android emulator). The case is skipped on iOS until iOS simulator input through the HID injector #1019.
• With a person at the phone, e2e/slow-android-physical-manual.test.ts sends a tap and the person confirms it landed (slow lane, physical Android, person at the phone).
• watching-the-screen.mdx lists what input does on Android.

Out of scope

• Rotation (0x07), on Android and anywhere: agents rotate through agent-device.
• Text messages (scrcpy's inject-text); serve-sim sends keys.
• The key and shifted fields of serve-sim key messages: only usage and type are read, so a non-US layout types US keys.
• Function keys, the keypad, and HID usages outside the table in F21.
• Waking a dark phone screen.
• Multi-touch.

Depends on

• Input socket for a lease #1017

Written by an agent.