Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ S15/S16 target terms below refine product intent. Definitions explicitly marked

**Conversation:** A durable discussion with one or more Bot participants. Membership, a display label and an @mention do not grant execution or resource access.

**Computer needs-you card (S16):** An origin-only conversation handoff for sign-in, verification or CAPTCHA (ADR 0092). Inserting the card precedes yielding the computer under a new screen generation. Watch screen takes exclusive owner control after earlier operations settle. Screen events supply resumed/declined/expired outcomes; the next eligible work receives them as labelled data, and only committed result publication marks the carried set delivered. Routine occurrences use their existing needs-you state without yielding. Sensitive-site sessions remain in the protected browser.

**Shared client (S16, API16-01):** The browser/native-safe transport and ephemeral projection state over the existing protected Jit service. It submits exact owner commands, reads currently authorized snapshots/control events, recovers the original send identity and reviews exact actions. It is not a second authority, engine or store. The [client contract](docs/contracts/s16-client-v1.md) supports text/tool work and exact decisions. Explicit file composition additionally supports the first [UTF-8 file-to-artifact workflow](docs/contracts/s16-files-v1.md) and React `/s16` client; an explicit strict parser configuration additionally supports [isolated PDF/PNG/JPEG inputs](docs/contracts/s16-binary-files-v1.md) through actual guarded Schway image sends. Explicit [sourced file memory](docs/contracts/s16-file-memory-v1.md) additionally carries exact original/representation pins through later work and inference history, with withdrawal/correction/forget and current recipient/private-scope checks; live steering remains unavailable; Linux isolation and live-account vision evidence remain separate pending acceptance.

**File lineage pin (S16, MEM16-01A):** Immutable metadata naming an exact original source/resource generation and original/validated-representation hashes. MEM/TEAM context, claim and artifact ancestry retains these pins; owning FileStore status and selected bytes are rechecked before release. A pin is data, never a principal or permission. Explicit opt-in freezes new file-memory ingress; old accepted file commands retain their prior semantics.
Expand Down
47 changes: 43 additions & 4 deletions docs/adr/0092-s16-agent-use-of-the-working-computer-browser.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
status: accepted for DESK16-01 (#410), third slice; D1/D2/D3 chosen by Champ on 2026-11-12; 3a merged in #460; 3b implementation adds inputs and held cards; 3c sign-in/resume remains unimplemented; closes AUTH16-01 (#442)'s last acceptance item once implemented; live host evidence belongs to #431
status: implemented for DESK16-01 (#410), third slice; D1/D2/D3 chosen by Champ on 2026-11-12; 3a merged in #460 and 3b in #461; 3c delivers needs-you sign-in/handoff and AUTH16-01 (#442)'s remaining deterministic acceptance; live host evidence belongs to #431
date: 2026-11-12
scope: DESK16-01 (#410); AUTH16-01 (#442); S16-AC06/AC07/AC11; FR-10/14/15/35/38/40; NFR-07/14
---
Expand Down Expand Up @@ -253,9 +253,13 @@ ScreenControl tickets). The owner sees the Bot work on the screen, as in Rakazo.
- **The next run gets the outcome as data.** The next work run in the conversation that posted the card, with that
computer in its admitted resources, gets one work-prompt source with one entry per card,
`{ cardId, origin, reason, outcome, generation }`, where `generation` is the new one.
- *Which cards (the carried set).* The eligible cards are chosen once, when the prompt is built, next to the
memory prepare (`application.ts:604-606`): cards posted in this conversation that have a `yield_*` outcome
event and are not yet marked delivered, whose computer is in the request's admitted resources. Before the run
- *Which cards (the carried set).* The eligible cards are chosen once, when the prompt is built, after tools
open and the protected VM journal confirms the request's actual placement: cards posted in this conversation
that have a `yield_*` outcome event and are not yet marked delivered, whose computer is in the request's
admitted resources and is its single active placed VM. A pool resource or matching Bot alone is insufficient:
sign-in on member A must not be carried or consumed by a run placed on member B. A run with no placed VM
receives no screen outcome. This refines the initial prepare-before-tool-open ordering near memory preparation.
Before the model run
starts, their `cardId`s are saved on the request in one conversation-store transaction, the way
`current.memory` is saved at `:606`, and that same set (with each card's outcome read from its event) is what
the run carries. An empty set saves nothing and adds no source.
Expand Down Expand Up @@ -437,3 +441,38 @@ ScreenControl tickets). The owner sees the Bot work on the screen, as in Rakazo.
`parseServiceConfig`, shipped in slice 3c; and `s15-working-vm-v1` (`vmDesktopCdpStream` with its
`sudo -n -u jit-desktop` attach and handshake, relay files, composite `ready-v1` key). Live evidence on `jit-vm`
(real Chromium pipe, sign-in on a trusted non-IdP fixture site, stop/start) belongs to #431.


## Slice 3c implementation record

The protected `ComputerNeedsYou` service uses the existing conversation and VM journals with the insertion,
recovery, expiry and event-delivery order specified above. No cross-store transaction is claimed. A new additive
conversation table holds bounded origin-only metadata and delivered request IDs; the VM migration preserves the
old state/events/commands while extending the CHECK and adding card IDs. Expiry validates real epoch milliseconds
rather than the revision/counter ceiling. A protected per-VM browser-generation high-water mark makes each
screen transition exceed every previously exposed browser observation. This avoids a numerical collision when
several browser reads preceded yield; recording it does not alter screen control or interrupt existing tickets.
The mark survives restart and seeds fresh runs. Browser calls still use Schway's public host/tool surface.

PR #463 review found that a logical computer resource can name multiple shared members. Outcome preparation
therefore runs after tool placement, and the protected composition intersects its resource/private-Bot check with
the actual active VM session rows for that work. Another member cannot receive or mark the signed-in member's
outcome; it remains pending until an eligible run on that exact member commits. A two-member composed regression
forces B while A is drained, then A, and checks actual MODEL prompts and the committed delivery row in both runs.

The production MODEL and fallback Schway paths receive a separate validated/hashed `screenOutcomes` source.
Publication marks only the saved carried IDs in the conversation result transaction. Routine holds come from the
trusted browser run, through the VM/composite/file tool ports, rather than parsing model prose or tool-result JSON.
The shared React client's open screen is held outside refreshed snapshot rows, so periodic withholding does not
unmount input/control; navigation, private-content failure and session loss clear it. The existing screen component
continues to own input serialization, renewal, fresh generations and hand-back.

Deterministic evidence: `s16-computer-needs-you.test.ts` (two-store gap/fault/restart, event outcomes, exact carried
sets, tier/auth/scope, routine and no-CDP semantics), `s16-needs-you-composition.test.ts` (actual tool, owner command,
MODEL prompt, only “continue”, failed provider, completed MODEL with missing evidence, publication rollback, late
outcome, no-computer request, routine occurrence and idle sweep), `s16-computer-browser.test.ts` (fresh/stale
handoff generations and bounded unproven-drain refusal), `s15-model.test.ts` (source validation and input hash),
`s15-conversation.test.ts` (fallback), `ops-execution-config.test.ts` (startup bounds and package rendering/plans),
and the actual English/Thai client bundle in `s16-needs-you.browser.test.ts` (1440/390, keyboard Watch screen,
refresh continuity, session-loss withholding, native non-IdP fixture login/session persistence and hand-back). These do not substitute for #431 live guest/account/SSH/sudo acceptance.
Automatic continuation remains #459.
46 changes: 43 additions & 3 deletions docs/contracts/s16-computer-browser-v1.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# s16-computer-browser/v1 — watched desktop browser

Slices 3a and 3b of ADR 0092 / #410. The host offers `computer_browser` only to execute work admitted to a running desktop
Slices 3a–3c of ADR 0092 / #410. The host offers `computer_browser` only to execute work admitted to a running desktop
computer. Placement and SSH identity come from that work's VM session, not model input. Inspect work and headless
computers keep their existing file/shell tools. A private computer's browser observations retain that computer's scope.

Expand All @@ -12,9 +12,10 @@ computers keep their existing file/shell tools. A private computer's browser obs
- `{ kind: "find", generation, tab?, text }`: case-insensitive literal text matches, query at most 4096 bytes.
- `{ kind: "click" | "submit", generation, tab?, target }`: an opaque element id from the latest observation.
- `{ kind: "type" | "select", generation, tab?, target, value }`: replace a non-secret input or select an exact option value; at most 4096 UTF-8 bytes. Line breaks in `type` carry Enter/form submission.
- `{ kind: "needs_you", reason: "sign_in" | "verify" | "captcha", origin }`: public HTTP(S) origin only (no path, credentials, query or fragment); posts the conversation card and yields without a screen ticket or CDP batch.

Unknown fields/kinds are refused. There is no model-supplied JS, selector, CDP command, cookie access, upload/download,
credential entry or screenshot. `needs_you` and the trusted-site sign-in/resume flow ship in 3c. This slice does not close #442 or claim live guest evidence.
credential entry or screenshot. The needs-you contract below supplies trusted-site sign-in/handoff; live guest acceptance remains #431.

`state` returns `{ state: "ready", generation, truncated, tabs }`. Allowed tabs expose bounded URL/title; refused
tabs expose only opaque id, origin and refusal code. `navigate`/`read`/`find` return
Expand Down Expand Up @@ -121,7 +122,7 @@ Every call binds disclosure to the trusted work actor and its placed VM/session:
ancestry before guest dispatch, and `vmOutput` imports persistent shared-VM ancestry before publication, including
sources acquired after session setup. A denied input sends no CDP; a denied output publishes no observation.

All operations hold ScreenControl agent tickets and recheck them and the live work guard around each CDP command.
All operations except `needs_you` hold ScreenControl agent tickets and recheck them and the live work guard around each CDP command.
Takeover closes the channel gracefully. The relay discards wholly unsent commands, completes any partial frame,
and acknowledges only after sent commands reply and sent navigations have stopped loading (`Page.stopLoading` reply). The host waits for that positive guest-drain acknowledgement,
not just rejected host promises, before releasing the agent ticket. Unproven settlement fails closed with
Expand All @@ -134,6 +135,45 @@ Neither work-shell command/session timeout applies to this dedicated stream. The
`{ state: "denied", code }` with `isError`; unexpected transport errors are `computer_browser_unavailable`.
There is no autonomous retry of an operation whose outcome may be uncertain.

## Needs-you handoff (slice 3c)

`needs_you` refuses unless the screen is `agent` (`takeover_state_changed`). It closes its stream, inserts an
origin-only card in `conversation.sqlite`, then changes `vm.sqlite` to `yield` under a fresh generation. It returns
`{ state: "yielded", cardId, generation, expiresAt }` within the ordinary 60 s call deadline. Work can end or be
cancelled without clearing yield. Later browser/VM input on that computer refuses `computer_yielded`; idle sleep
is blocked. Existing operation tickets settle normally; Watch screen waits in `settling` until they end.

The authenticated snapshot adds `needsYouCards`. `POST /api/s16/client/needs-you/commands` (CSRF, JSON ≤ 4 KiB)
accepts exactly `{ contract: "s16-computer-needs-you/v1", requestId, conversationId, cardId, kind: "watch" | "decline", generation }`.
The response is `{ contract: "s16-computer-needs-you-result/v1", card, screen }`. Watch rechecks the current site
tier and VM generation, takes control under the exact observed generation, then opens the existing screen UI.
An old generation refuses `stale_screen`. Decline applies only to a still-yielded card. Chat text cannot call either.
Sensitive-only or sensitive-tier origins require `protected_browser`.

Expiry is fixed on insertion: top-level `needsYouMinutes` / `JIT_COMPUTER_NEEDS_YOU_MINUTES`, default 240, range
30–1440; S15 validates it at startup independently of legacy config. The optional packaged-service field is validated,
rendered only when present and included in configuration plans. Restarting with a different setting affects future
cards only. Reading an overdue `yield` lazily returns it to `agent`; card expiry never clears owner/settling/fenced control.

A failed insert cannot yield. A failed yield voids the inserted card. A crash orphan is voided on read/startup
only after 60 s when neither a control row nor terminal yield event holds its card ID. Failed repair retries on read.
Screen transitions retain `card_id` through owner leases/restarts, and each return to agent atomically clears the row
and stores its old card ID on `yield_resumed`, `yield_declined` or `yield_expired`. Cards derive outcomes from these
events, so a failed card update cannot erase an outcome. No hand-back sends a CDP batch.

After tool opening confirms the actual placed VM, the next eligible work in the same conversation on that exact
computer gets `{ cardId, origin, reason, outcome, generation }` as a separate
`screen-outcome-data-never-authority` host source in both production MODEL and fallback Schway. The exact carried IDs
are saved before sending and marked delivered only in the committed conversation result-publication transaction.
Provider failure, cancellation, publication rollback or `result_unavailable` preserves them for the next eligible run;
no-computer work, a different shared pool member and outcomes arriving after preparation do not mark them.
A logical pool resource alone never proves placement. Inputs validate and hash the outcome set.
Interpretation and memory-claim parsing receive no such source. A fresh run starts at the hand-back generation;
observations from before yield remain stale. A durable per-VM observation high-water mark makes each control transition exceed every previously exposed browser generation, including after multiple reads or restart. Automatic continuation is separately owned by #459.

Routine runs return a host-observed `computer_needs_you` completion hold, mapped to the existing occurrence
`needs_you` state. They neither create a desktop card nor yield an unattended computer.

## Verification

`tests/s16-computer-browser.test.ts` exercises handshake parsing, split/coalesced frames, site rules, opaque generations,
Expand Down
16 changes: 11 additions & 5 deletions docs/contracts/s16-computer-screen-v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@ Owner contract for watching a desktop working computer and taking exclusive cont
"state": "agent", "generation": 0, "takeoverId": null, "expiresAt": null, "agentBusy": false, "scope": { "kind": "shared" } }
```

- `state`: `agent` (work may use the computer), `settling` (takeover requested; waiting for earlier work operations, `agentBusy`), `owner` (only owner input), `fenced` (control lost; nobody writes).
- `generation` advances on takeover, lease lapse, controller restart and hand-back.
- `state`: `agent` (work may use the computer), `yield` (needs-you card waiting for the owner), `settling` (takeover requested; waiting for earlier work operations, `agentBusy`), `owner` (only owner input), `fenced` (control lost; nobody writes).
- `generation` advances on takeover, lease lapse, controller restart and hand-back. Each transition exceeds the durable browser-observation high-water mark, so a pre-handoff browser counter cannot equal the new control generation. Recording an observation changes neither control nor active tickets.
- `available`/`code`: `vm_desktop_unavailable`, `vm_not_running` or `vm_desktop_restart_required` when no frame can be read.
- `scope`: lineage of every frame (a private computer's frames are its Bot's private data).
- Reading changes nothing, except that an owner lease that already lapsed is reported (and recorded) as `fenced`.
- Reading expires a past-due `yield` to `agent` with `yield_expired`, or reports/records a lapsed owner lease as `fenced`. Card expiry never removes owner control.

## Read one frame

Expand All @@ -27,7 +27,7 @@ Owner contract for watching a desktop working computer and taking exclusive cont

| kind | fields | effect |
| --- | --- | --- |
| `takeover` | `generation` | from `agent` or `fenced`: new generation and takeover id; `owner` at once, or `settling` while work operations admitted earlier are still running |
| `takeover` | `generation` | from `agent`, `yield` or `fenced`: new generation and takeover id; `owner` at once, or `settling` while work operations admitted earlier are still running |
| `renew` | `generation`, `takeoverId` | extends owner control by the lease (30 s); not journaled |
| `input` | `generation`, `takeoverId`, `input` | one input in `owner` only (`takeover_settling` while settling) |
| `hand_back` | `generation`, `takeoverId` (null when fenced) | waits for the input being sent, then `agent` under a new generation |
Expand All @@ -54,7 +54,7 @@ Coordinates are display pixels (`0 ≤ x < 1280`, `0 ≤ y < 800`).

## Watched browser tickets (ADR 0092, slice 3a)

Every `computer_browser` call (`state`, `navigate`, `read`, `find`) holds an agent ticket from before its first wait
Every `computer_browser` call except `needs_you` holds an agent ticket from before its first wait
until it settles. The ticket and live work guard are checked immediately before and after each CDP command.
A screen generation transition synchronously invalidates observations and closes the run's stream, including an idle
or opening stream. Takeover therefore refuses new browser calls with `takeover_active`, and stays `settling` until
Expand All @@ -63,3 +63,9 @@ the outstanding browser call releases its ticket. Hand-back never revives an obs
Closing a browser channel is not settlement by itself: takeover retains the browser ticket until the relay
acknowledges that unsent frames were dropped and partial/in-flight frames finished. If the dedicated SSH drain
times out or exits without acknowledgement, `computer_browser_unsettled` leaves the ticket held (fail closed).

Slice 3c rebuilds the existing control table's state CHECK to add `yield` and `card_id`, preserving prior rows,
commands and events. Events gain `card_id` in that migration. Yield retains the card across takeover/settling,
lapsed leases and restart fencing. A return to agent stores the old card ID on its outcome event in the same VM
transaction and clears it on the new control row. See [the browser contract](s16-computer-browser-v1.md#needs-you-handoff-slice-3c)
for card insertion order, recovery, expiry and outcome delivery.
Loading
Loading