Skip to content

agent: expose the streamer's wait, record and screenshot - #17

Merged
Daviduche03 merged 1 commit into
masterfrom
agent/wait-record
Nov 6, 2026
Merged

Daviduche03 merged 1 commit into
masterfrom
agent/wait-record

Conversation

@Daviduche03

Copy link
Copy Markdown
Owner

Why

The last PR made voxl the console. It also handed us three capabilities warmbox never had — and the agent API, which is where they matter most, could not reach any of them. A computer-use loop had exactly one tool for "did my click work?": sleep, then screenshot.

What

Endpoint What it does
GET /api/desktops/{id}/wait Returns when the screen stops changing (?change=1 for the first change). idle, region, since, timeout are passed through.
POST /api/desktops/{id}/record?seconds= An MP4 written from the encode that is already running. Max 300 s.
GET /api/desktops/{id}/screenshot Now takes the streamer's last frame when there is one — a copy instead of spawning ImageMagick in the guest.

/wait is damage-driven inside the guest (XDamage), so it costs nothing while the screen is still. It is the difference between act → sleep → hope and act → wait → look.

Compatibility

An image built before voxl behaves exactly as it did: /screenshot falls back to the agent's own capture, and /wait answers 501 stream_unavailable with a reason rather than hanging. ?classic=1 still forces noVNC.

Audio: the guest used to hardcode --audio-enabled=false. It now turns on by itself when the image has a PulseAudio socket. Our images still ship no sound stack, so nothing changes today — but an image that adds one gets audio without a code change.

Test plan

Against a real xfce-voxl desktop, through the daemon:

screenshot  200  image/png, X-Voxl-Seq: 2   (came from the streamer)
wait        {"changed":false,"seq":2,"timed_out":false,"waited_ms":203}
record 3s   200  59,950 bytes, ftyp isom    (valid MP4)
seconds=999999 → clamped to 300

Against the published xfce image (no voxl), the same binary:

/d/          → /vnc/... (noVNC)
screenshot   200  image/png, no X-Voxl-Seq  (agent fallback)
wait         501  stream_unavailable
  • go build, go vet, go test ./... pass; recordSeconds is unit-tested at the bounds.

The console can already say when the screen settles, save an MP4 from the
encode it is already running, and hand out its last frame. None of that was
reachable from the agent API — which is where it is worth the most, because a
computer-use loop could only sleep and hope the UI had caught up.

- GET  /api/desktops/{id}/wait — returns when the screen stops changing (or at
  the first change with ?change=1). Damage-driven inside the guest, so waiting
  costs nothing while the screen is still: no polling, no screenshots.
- POST /api/desktops/{id}/record?seconds= — an MP4 written from the live H.264,
  so no second encoder and no extra CPU. Clamped to 300s.
- GET  /api/desktops/{id}/screenshot — the streamer's frame when there is one
  (a copy, not a process spawn), the agent's ImageMagick capture otherwise.

An image built before voxl keeps working: /screenshot falls back to the agent
and /wait answers 501 with a reason. Audio now switches on by itself for an
image that ships a PulseAudio socket, instead of being hardcoded off.
@Daviduche03
Daviduche03 merged commit b8ab8a5 into master Nov 6, 2026
6 checks passed
@Daviduche03
Daviduche03 deleted the agent/wait-record branch November 6, 2026 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant