Skip to content

Let a lease holder watch and interact with its device's screen #1013

Closed
5 of 5 issues completed
Closed
Let a lease holder watch and interact with its device's screen#1013
5 of 5 issues completed
Labels
feature:plannedSplit into tasks. Never picked up itself.

Description

@V3RON
V3RON
opened on Oct 26, 2026

Request: #1010

Problem

In the reporter's words: "Only operators can watch a device's screen, and the stream is view-only. A lease holder can't see the device it leased, and nobody can interact with it through the stream."

"I run a pool with coding agents leasing devices through the agent-device front. I'd like to watch what an agent is doing on its device and sometimes take over, for example to sign in or dismiss a dialog. Today that means: giving the client an operator token, which can see and stop every device in the fleet; decoding Simlock's own stream format; still not being able to tap anything."

Who it is for

An app that shows a person the device its agent leased, holding only the agent's token: an agent app with the device beside the chat (T3 Code, pingdotgg/t3code PR #25259, paused on this), a phone checking on an agent, a CI dashboard. The person watches, and now and then takes over for a step the agent cannot do: sign in, dismiss a dialog.

Outcome

  • A lease holder watches its own lease's screen with its agent token. No operator token.
  • A lease holder taps, swipes, types and presses buttons on that screen.
  • A client that already speaks serve-sim's input messages (T3 Code's device panel) drives a Simlock lease without Simlock-specific code.
  • Both stop when the lease ends, and say why.

How it works for the user

sequenceDiagram
  participant App as Agent app (agent token)
  participant GW as Gateway
  participant W as Worker holding the lease
  App->>GW: GET /v1/leases/L-42/stream
  GW->>W: relay (owner checked)
  W-->>App: frames: meta, avcC, key, delta, alive…
  App->>GW: WS /v1/leases/L-42/input
  GW->>W: two-way relay (owner checked)
  W-->>App: 0x83 admitted, 0x82 {width,height,orientation}
  App->>W: 0x03 touch begin / move / end
  Note over W: lease released
  W-->>App: stream: 0x07 {"reason":"explicit"}, body ends
  W-->>App: input: close 4001 "explicit"
  App->>GW: GET /v1/leases/L-42/stream
  GW-->>App: 404 UNKNOWN_LEASE
Loading

Before:

$ curl -H "Authorization: Bearer $AGENT_TOKEN" https://gw:7480/v1/stream/workers/w1/devices/d7
{"error":{"code":"UNKNOWN_DEVICE","message":"…"}}            # 404: operators only

After:

$ curl -H "Authorization: Bearer $AGENT_TOKEN" https://gw:7480/v1/leases/L-42/stream -o screen.bin
# 200, application/vnd.simlock.stream+h264, frames until the lease ends
$ websocat -H "Authorization: Bearer $AGENT_TOKEN" wss://gw:7480/v1/leases/L-42/input
# binary: 0x83, then 0x82 {"width":1179,"height":2556,"orientation":"portrait"}

Examples

Request (token) Today After
GET /v1/leases/L-42/stream (agent, owns L-42) 404 (no route) 200, the picture
GET /v1/leases/L-42/stream (agent, another's lease) 404 404 UNKNOWN_LEASE
GET /v1/leases/L-99/stream (unknown lease) 404 404 UNKNOWN_LEASE
GET /v1/leases/L-42/stream (operator) 404 200
GET /v1/leases/L-42/stream, capabilities.stream.enabled false 404 404
WS /v1/leases/L-42/input (agent, owns L-42), capabilities.input.enabled true refused 0x83, then 0x82
WS /v1/leases/L-42/input, capabilities.input.enabled false on the daemon answering refused 404, refused
WS /v1/leases/L-42/input through a gateway, input off on the lease's worker refused 503 INPUT_UNAVAILABLE, refused
a 9th WS /v1/leases/L-42/input while 8 are open refused 0x83 not sent; closed 1013
WS /v1/leases/L-42/input on a physical iPhone lease refused 0x83, then 0x82 with "inputUnavailable":true
GET /v1/stream/workers/w1/devices/d7 (agent) 404 UNKNOWN_DEVICE 404 UNKNOWN_DEVICE (unchanged)
Input message iOS simulator Android emulator / phone Physical iPhone
0x03 touch, 0x0b scroll applied applied none (inputUnavailable)
0x04 home applied applied (Home) none
0x04 app_switcher, lock dropped applied (Recents, Power) none
0x04 back dropped applied (Back) none
0x06 key applied on an iOS 26 or older runtime; on iOS 27 dropped, and 0x82 says "keyboardUnavailable":true applied none
0x07 orientation dropped dropped none
0x10 hinge control answered 0x90 {ok:false} answered 0x90 {ok:false} answered 0x90 {ok:false}
bad JSON, unknown tag dropped, socket stays open dropped dropped

What could go wrong

  • A person and the agent tap at once and get in each other's way. Likely now and then; it is the take-over the request asks for.
  • A client keeps reconnecting after the lease ended and sees 404s. Likely with T3's 1 s retry; harmless.
  • A tap lands in the wrong place right after the agent rotates an Android device. Short-lived: the screen size is read again every 2 s while input is open.
  • The person cannot rotate, press the app switcher or Lock on an iOS simulator. Certain until the follow-up.
  • On iOS 27 simulator runtimes, hardware keys do nothing; the person taps the on-screen keyboard. Certain until the follow-up.

Words used

  • lease holder: the principal that requested a lease and owns it, e.g. the agent token that leased L-42.
  • serve-sim input messages: binary WebSocket messages, one tag byte then UTF-8 JSON, as expo-device-hub's serve-sim speaks them at commit 1357114, e.g. 0x03 {"type":"begin","x":0.4,"y":0.7}.
  • admitted (0x83): the one-byte message that tells the client the server accepts input from now on.
  • screen config (0x82): {width, height, orientation, inputUnavailable?, keyboardUnavailable?}, sent after 0x83 and whenever the screen changes.
  • device input: what turns messages into taps on one device, shared by all input sockets of a lease: a control-only scrcpy server on Android, a HID helper process on an iOS simulator.
  • unavailable: a device input that cannot take input on this device (a physical iPhone, a phone whose adb check is closed); the socket stays open and says so in 0x82. Failed: one that could but broke; the socket closes 1013.
  • the adb check: the check every command to a physical Android phone passes first today, comparing the adb server on port 5037 with Simlock's own adb; closed means it refused, e.g. a different adb version on 5037.
  • two-way relay: a gateway-to-worker relay that keeps both directions of its uplink byte channel open for the socket's life, unlike today's request-then-response relay.
  • heartbeat: a message the worker's input session sends every 5 s so the gateway can tell a stuck worker end from a quiet one.

Non-goals

Completion conditions

  • An agent token streams its own lease through a gateway and through a standalone worker, and gets 404 for anyone else's.
  • A test client that follows T3 Code's message order (waits for 0x83, reads 0x82, sends touches and keys) drives a faked device through a gateway, in the fast lane.
  • A real iOS simulator and an Android emulator take a tap through the input socket: a tap on a known control opens the next screen, as agent-device's snapshot shows (slow lane). A person confirms a tap on an Android phone (slow lane, person at the phone).
  • Ending the lease ends both the stream (with a 0x07 frame) and the input socket (close 4001).

Open questions

none

Decisions

Shared values

Value or state Rule Written by Read by Source
Lease route /v1/leases/{id}/<module name>: http resolves the lease as lease.get does (owner or operator, else 404 UNKNOWN_LEASE), then calls the module's lease handler (request, identity, lease) → Response #1015 #1017 src/http/app.ts:803-811, src/http/app.ts:783-784
Lease socket The same path resolution for a WebSocket upgrade, on a gateway and on a standalone worker: ports routes upgrades (only it imports ws). A module's lease socket handler has two steps: open(request, identity, lease) answers a refusal Response (sent before the upgrade) or an accept; the accept then gets the socket. A plain GET to the path answers 426 from the module's leaseRoute #1017 #1018, #1019 src/ports/uplink-websocket.ts:203-226, src/http/server.ts:43-47
ended frame Tag 0x07, JSON {"reason": <EndedLease reason>}: explicit, killed, expired or device-lost; the last frame before the body ends, on both stream routes, when a lease ends. device.absence-detected on a leased device also writes device-lost, since that path ends the lease after the stop event. Other ends (device shutdown, capture error) send none. The console shows: explicit "The lease was released.", expired "The lease expired.", killed "The lease was ended by an operator.", device-lost "The device was lost.", not as an error #1015 #1015 console, #1017 close reason src/leasing/lease-ports.ts:32-38, src/core/screen.ts:7-18
Capability input capabilities.input.enabled, default false; module name input. Off on the answering daemon: 404. Off on the lease's worker behind a gateway: 503 INPUT_UNAVAILABLE, "Worker does not take input for lease ; turn on capabilities.input.enabled there", as the stream does #1017 #1018, #1019 src/core/config.ts:256-258, src/stream/gateway-link.ts:68-72
Input messages expo serve-sim @1357114: 0x03, 0x04, 0x06, 0x07 (always dropped), 0x0b, 0x10 in; 0x83, 0x82 (with inputUnavailable?, keyboardUnavailable?), 0x90 out; core/input.ts holds the types #1017 #1018, #1019 expo device-session.ts:1451, :1487-1689
Device input port WorkerContext.inputs.openInput(platform, device, signal) resolves {unavailable: reason} or a device input (send(message), onConfig(listener), onFailed(listener), close()); it rejects only on failure. The catalog routes it to an optional driver member, as openScreen routes to screenCapture; a driver without one is unavailable. Unavailable: the socket gets 0x83, then 0x82 {"width":1,"height":1,"orientation":"portrait","inputUnavailable":true}. Failed (rejected open, or onFailed): close 1013; a failed open is remembered for 10 s, and sockets in that window are refused 1013 without a new open. One device input per lease, shared by up to 8 sockets; a 9th socket is closed 1013 before 0x83; it closes when the lease's last socket closes. Width and height are pixels of the screen as it is now. On Android, where taps are sent in pixels, the device input re-reads them every 2 s and reports a change through onConfig; on an iOS simulator the HID takes positions as fractions, and the size is read once #1017 #1018, #1019 src/modules/types.ts:121-131 (ModuleScreens, the model), expo device-session.ts:136, :1386-1392
Close codes 4001 lease ended (reason text = ended reason; through a gateway the worker's 0xF1 carries it; when the gateway sees the lease go first it waits up to 2 s for that 0xF1, then closes 4001 lease ended), 1011 worker unreachable or heartbeat silence, 1001 daemon stopping, 1013 device input failed or 8 sockets already open. A close reason is cut to 123 bytes #1017 #1018, #1019 t3code stream.ts:1133-1151
Client ping Every input socket, on a gateway and on a standalone worker: a WebSocket ping every 15 s; no pong within 15 s ends the socket #1017 — src/ports/uplink-websocket.ts:41 (the uplink's 15 s)
Channel framing On a two-way relay's channel: [len u32 big-endian][tag u8][payload], at most 64 KiB per message. Client messages keep their serve-sim tags; 0xF0 heartbeat (empty) and 0xF1 close {code, reason} never reach a client. Only the input module encodes and reads them; the relay moves bytes #1017 — src/core/screen.ts:71-77 (the stream's envelope)
Heartbeat The worker's input session sends 0xF0 every 5 s; the gateway's input module drops it; the two-way relay ends after 15 s with no bytes from the worker #1016 (silence limit), #1017 (heartbeat) #1017 src/gateway/relay.ts:31-36
Two-way relay relay.request gains optional twoWay: true (only with GET and hasBody false, else BAD_REQUEST). Ending one direction (END) ends only that direction; the relay ends when both have ended or either side aborts. The gateway input module bounds its open with a 10 s signal: no head in 10 s → 503 WORKER_UNREACHABLE before the upgrade. Gateway: GatewayContext.relayTwoWay(workerId, request, tenant, leaseId?) → a refusal Response, or {status, headers, readable, writable}. Worker: WorkerModule.relayTwoWay(request, tenant, leaseId, pipe) answers a refusal or a bodiless 2xx, then uses pipe = {readable, writable, signal}. The 15 s silence limit counts only bytes from the worker, from the head or the worker's last byte, and only while the worker's direction is open #1016 #1017 src/contract/operations.ts:1078-1095, src/daemon/relay.ts:18-22
Module lease route WorkerModule.leaseRoute and GatewayModule.leaseRoute hold a module's lease handler; a gateway's lease stream reaches the worker as /stream/leases/<workerLeaseId> with leaseId = the worker's lease id, and the worker answers 404 UNKNOWN_LEASE when that lease is no longer in its records #1015 #1017 src/modules/types.ts:84-110
ended payload {"reason"} at most 64 bytes of UTF-8 JSON; a reader that cannot parse it ends with no reason shown #1015 #1015 console src/core/screen.ts:7-18
Protocol bump relay.request gains a two-way mode: bumps the protocol by one #1016 — src/contract/operations.ts:1078-1095

Behaviours

Behaviour (invoker, action, what it sees) Seam Cases Task
A caller GETs /v1/leases/{id}/stream on a standalone worker and gets the picture or a refusal worker HTTP own lease 200; another's 404 UNKNOWN_LEASE; unknown 404; operator 200; stream capability off 404 #1015
A caller GETs /v1/leases/{id}/stream on a gateway and gets the lease's worker's picture or a refusal gateway HTTP own lease 200; another's 404 UNKNOWN_LEASE; worker unreachable 503 WORKER_UNREACHABLE; worker without the stream module 503 STREAM_UNAVAILABLE; lease gone on the worker 404 UNKNOWN_LEASE #1015
A viewer of either stream route sees a 0x07 frame with the reason, then the end, when the lease ends worker stream explicit, expired, killed; device-lost by the health monitor and by physical absence; lease route and device route #1015
A console viewer sees why the picture stopped when a 0x07 frame arrives console stream reader 0x07 with a reason; an unknown tag is still skipped #1015
A gateway module's two-way relay carries bytes both ways at once until either side ends gateway relay both directions together; worker ends its direction and the gateway still writes; gateway ends; both ended, relay done; gateway module aborts, the worker target's signal aborts #1016
A gateway module opening a two-way relay gets the worker's refusal or an error gateway relay and relay.request contract non-2xx head as a Response; no two-way target RELAY_TARGET_MISSING; twoWay with POST or a body BAD_REQUEST #1016
A two-way relay with no bytes from the worker for 15 s ends with WORKER_UNREACHABLE; bytes every 5 s keep it open gateway relay silent 15 s while the gateway writes; a byte every 5 s for 60 s; worker's direction ended, no limit #1016
An uplink that drops ends every two-way relay on both sides gateway relay gateway side gets WORKER_UNREACHABLE; worker target's signal aborts #1016
A caller opens WS /v1/leases/{id}/input and gets 0x83 then 0x82, or a refused upgrade worker and gateway HTTP upgrade own lease; another's 404; unknown 404; operator; input off on the answering daemon 404; off on the lease's worker 503 INPUT_UNAVAILABLE; worker gives no head in 10 s 503 WORKER_UNREACHABLE; plain GET 426; no token 401; a 9th socket closed 1013; an unavailable device input gets "inputUnavailable":true; a failed open closes 1013 and a reconnect within 10 s is refused 1013 without a new open #1017
A client following T3 Code's order drives a faked device through a gateway gateway HTTP upgrade waits for 0x83; touch begin/move/end, key, button arrive in order; a second socket shares the same device input; the device input closes when the last socket closes; a size change reaches every socket as 0x82 #1017
A client sending a bad message keeps its socket and nothing reaches the device worker input session bad JSON; unknown tag; x outside 0..1; 0x07 dropped; 0x10 answered 0x90 {ok:false} #1017
An input socket closes with 4001 and the reason when the lease ends, and a reconnect gets 404 worker input session and gateway explicit, expired, killed, device-lost; through a gateway the worker's reason wins; a lease that ends while the socket opens #1017
An input socket closes when either end goes quiet or away worker and gateway input client misses its pong (worker and gateway); worker heartbeat stops (1011); uplink drops (1011); daemon stops (1001); the gateway sees the lease go (4001) #1017
Touches and scrolls on Android reach scrcpy as pixel positions for the device's size Android device input begin/move/end; two screen sizes; scroll #1018
Keys and buttons on Android reach scrcpy as keycodes Android device input home 3; app_switcher 187; lock 26; back 4; HID letters, digits, Enter, Backspace; Shift held then a gives A (meta state); an unknown usage dropped #1018
An Android device input starts a control-only scrcpy, and a phone passes the adb check first Android device input emulator; phone with the check open; phone with it closed → unavailable; scrcpy fails to start → rejected; screen size read in the current rotation, and a change after an agent rotates reported #1018
Touches, buttons, scrolls and keys on an iOS simulator reach the HID helper iOS device input touch; home; lock and app_switcher dropped; scroll; key on an iOS 26 runtime; key on iOS 27 or an unknown runtime dropped with keyboardUnavailable; helper missing → rejected; no 0x82 from the helper in 10 s → rejected; helper exits → failed #1019
A physical iPhone's device input is unavailable iOS device input physical device #1019

Tasks

flowchart LR
  T1[#1015 lease stream route] --> T3
  T2[#1016 two-way relay] --> T3[#1017 input socket]
  T3 --> T4[#1018 Android input]
  T4 --> T5[#1019 iOS simulator input]
Loading

Completion conditions → tasks: streaming and 404 → #1015; T3 Code's order through a gateway → #1017; real devices take a tap → #1018 (emulator, phone), #1019 (simulator); both end with the lease → #1015, #1017.

Open work in the same area:

Written by an agent.

Activity

  1. V3RON
    added
    feature:specSpec in progress, business and technical together.
    on Oct 26, 2026
  2. V3RON
    added sub-issues on Oct 26, 2026
  3. V3RON
    added 2 commits that reference this issue on Oct 26, 2026

    docs(adr): 0049 text; 0045 shipped, its §2 narrowed by 0049 (#1013)

    e9e84cf

    docs(adr): 0049 shares one device input per lease, pings clients (#1013)

    1b33229

  4. V3RON commented on Oct 26, 2026

    @V3RON
    V3RON
    on Oct 26, 2026
    ContributorAuthor

    Spec updated: Decisions, Shared values, Behaviours and Tasks written; tasks #1015–#1020 drafted.

    Written by an agent.

  5. V3RON
    added a commit that references this issue on Oct 26, 2026

    docs(adr): 0049 drops rotation and narrows iOS buttons to Home (#1013)

    e9798fb
  6. V3RON
    removed a sub-issue on Oct 26, 2026
  7. V3RON
    added a commit that references this issue on Oct 26, 2026

    docs(adr): accept 0049 (#1013)

    7d68d26
  8. V3RON
    added
    feature:plannedSplit into tasks. Never picked up itself.
    and removed
    feature:specSpec in progress, business and technical together.
    on Oct 26, 2026
  9. V3RON
    added 2 commits that reference this issue on Oct 26, 2026

    docs(adr): 0049 narrows 0028 §4 and 0030 §3, and a module name cannot…

    23ab54b

    docs(adr): order the 0048 and 0049 index rows (#1013)

    f74651e

  10. V3RON commented on Oct 26, 2026

    @V3RON
    V3RON
    on Oct 26, 2026
    ContributorAuthor

    Delivery run

    Merged: none. The stack #1014 → #1025 is open and nothing merges until every task has joined it.
    Parked: #1016 (PR #1026). R7, a close after a peer reset loses the error reason, is still open after the last allowed review check.
    Waiting: #1015 (PR #1025), reviewed and in the stack; #1017 on #1016; #1018 on #1017; #1019 on #1017 and #1018.
    Decided: #1015: a module named renew or exec → the host refuses to load it, and the core route wins.
    Decided: #1015: audit gaps → two test rows, and the 0x07 test runs in src/stream/create.test.ts.
    Decided: #1016: audit gaps → three relay test rows.
    Asleep: 119 min

    Written by an agent.

  11. V3RON commented on Oct 26, 2026

    @V3RON
    V3RON
    on Oct 26, 2026
    ContributorAuthor

    Delivery run

    Merged: none. Every task has joined the stack #1014 → #1025 → #1026 → #1034 → #1036 → #1037, and nothing merges while two of them are parked.
    Parked: #1018 (PR #1036) needs hardware. The emulator tap and the physical-phone tap have not run: the disk was too low for the slow lane, and the phone needs a person.
    Parked: #1019 (PR #1037) needs hardware. The simulator tap has not run because the disk was too low for the slow lane.
    Waiting: none. #1015 (PR #1025), #1016 (PR #1026) and #1017 (PR #1034) passed review and are ready in the stack.
    Decided: #1016 → one more fix and check for R7 past the cap (the maintainer's call).
    Decided: #1017 → a proxy-signed identity opening the input socket without Sec-Fetch-Site: same-origin gets 403. ADR 0033 §5 now says so.
    Decided: #1018 → five audit rows added to Tests: last-pixel clamp, scroll clamp, close after open, no overlapping size reads, a failed size read logged once.
    Follow-up: #1035 (bug:new), a client reset during an uplink upgrade crashes the daemon. It predates this feature.
    Asleep: 146 min

    Written by an agent.

  12. github-actions commented on Oct 27, 2026

    @github-actions
    github-actionsBot
    on Oct 27, 2026 – with GitHub Actions

    Closing: all 5 sub-issues are closed. The PR that closed the last one walks the completion conditions (delivery rule 9); reopen this if one does not hold on main. Flip its ADRs to Accepted if delivery has not.

  13. github-actions commented on Oct 27, 2026

    @github-actions
    github-actionsBot
    on Oct 27, 2026 – with GitHub Actions

    Closing: all 5 sub-issues are closed. The PR that closed the last one walks the completion conditions (delivery rule 9); reopen this if one does not hold on main. Flip its ADRs to Accepted if delivery has not.

  14. github-actions
    closed this as completedon Oct 27, 2026
  15. V3RON
    added 2 commits that reference this issue on Oct 27, 2026

    docs(adr): 0049, a lease holder watches and drives its device's screen (

    6299f30

    feat(stream): a lease holder watches its own lease's screen (#1025)

    569b0c4
  16. juliusmarminge
    added a commit that references this issue on Oct 27, 2026

    feat(devices): Simlock leases stream and take input with the agent token

    c0c4287
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature:plannedSplit into tasks. Never picked up itself.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions