Skip to content

feat(desktop): add opt-in Windows computer control - #300

Merged
killswitch-GUI merged 26 commits into
mainfrom
codex/windows-computer-use
Oct 19, 2026
Merged

killswitch-GUI merged 26 commits into
mainfrom
codex/windows-computer-use

Conversation

@killswitch-GUI

@killswitch-GUI killswitch-GUI commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Add opt-in Windows computer use with global/workspace settings and the bundled colossus/computer-use skill. Agents select existing ordinary-user windows, inspect accessibility or target-window PNGs, and apply mouse/keyboard input against fresh observations. Closing automation leaves the user's application running.

Expose exactly 13 computer.* tools: windows, open, status, snapshot, screenshot, focus, move, click, drag, scroll, type, press, and close. Authenticated Desktop/connector grants and attached-plus tool search now discover them outside Minimal mode. Native availability, configuration, policy/approval, ownership and per-run limits remain required; authenticated sidecar acceptance covers discovery without desktop effects.

The native card provides separate state-aware Pause, Resume and Stop buttons. A delayed Pause after human takeover cannot resume control. Physical Escape stops native control; Ctrl+Alt+P pauses only; injected events cannot activate these shortcuts. Stop revokes native input and fences same-owner reopening. The runtime cancels the owning run upon a positive native UserStopped error or validated released status; this does not promise instantaneous interruption of an in-flight provider request. Pause, stale observations and generic denial remain recoverable; possibly applied input remains OutcomeUnknown and cannot be replayed automatically.

Fix the live 128-window read failure by narrowing valid optional window/snapshot counts to resolved workspace/native ceilings before policy and dispatch. Invalid types, zero and protocol overflow still fail. Consumed, mismatched or expired observations now produce trusted categorical stale previews, alongside fixed denial/count-limit recovery guidance, without leaking arbitrary native errors.

Computer screenshots use the inline image pipeline with distinct labels. Enlarged PNGs hide the native browser surface; Escape or the close button restores the same retained page. The skill documents observation replacement, freshness, new-run IDs and recovery. Codex compatibility advances to protocol 0.163.0 with GPT-6.1 Sol coverage and bounded opt-in redacted provider diagnostics.

Validation:

  • Current head: 0299cac53c5dd19c0322cda04eeb3e6034761060. Hosted PR checks: 9/9 passed. Full platform acceptance remains pending. The local PR command passed its Rust checks, then stopped at the documented Windows limitation that SDK generation requires the repository's POSIX scripts. Exact-head hosted SDK validation passed. The required local cargo xtask check rust completed successfully on this head.
  • Clean 15a6ee7f ordinary-user native acceptance passed, including occlusion-safe capture, accessibility, Unicode, balanced drag, owned dialogs, contention, revocation and acknowledged cleanup. A real GPT-6.1 Sol development run completed 30 actions in 163 seconds with no failed native tools: 17 computer, 10 browser and three orchestration actions. It verified 128-window/1024-node requests, fixture text/Clicked, inline/enlarged PNGs, styled Google/GitHub, terminal/minimize restoration and natural-completion Last view. Long-idle retained pixels and expired human-session recovery also passed.
  • 0299 changes only the Windows Desktop CI budget 75→90 minutes and its matching contract after a cold-cache timeout; CEF's 45-minute bound, per-test limits and acceptance coverage are unchanged. The Intel macOS CLI release job also has a bounded 90-minute cold-build budget; other CLI targets retain 60 minutes.
  • Physical Escape/Ctrl+Alt+P were not live-tested. Portable routing tests passed; earlier user button feedback concerns its earlier build only.

Exact signed-installed browser acceptance remains a separate publication gate; development/source evidence does not replace it.

@killswitch-GUI
killswitch-GUI marked this pull request as ready for review October 19, 2026 07:54
@killswitch-GUI killswitch-GUI added the ci:full Run cost-bounded full pre-merge acceptance on the current PR head label Oct 19, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T07:56:03.269264Z 523f54c Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@killswitch-GUI
killswitch-GUI changed the base branch from codex/windows-browser-action-stability to main October 19, 2026 08:23
@github-actions github-actions Bot removed the ci:full Run cost-bounded full pre-merge acceptance on the current PR head label Oct 19, 2026
@killswitch-GUI killswitch-GUI added the ci:full Run cost-bounded full pre-merge acceptance on the current PR head label Oct 19, 2026
@github-actions github-actions Bot removed the ci:full Run cost-bounded full pre-merge acceptance on the current PR head label Oct 19, 2026
@killswitch-GUI killswitch-GUI added the ci:full Run cost-bounded full pre-merge acceptance on the current PR head label Oct 19, 2026 — with ChatGPT Codex Connector
@killswitch-GUI
killswitch-GUI merged commit 0187fb6 into main Oct 19, 2026
80 of 83 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:full Run cost-bounded full pre-merge acceptance on the current PR head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant