comuse — Computer Use for any model, on any harness

comuse brings Codex-style Computer Use (see and operate desktop GUI apps) to any model on any agent harness — starting with opencode.

curl -fsSL https://raw.githubusercontent.com/maskjelly/comuse/main/install.sh | bash

Instead of reinventing the native layer, it standardizes on the best open-source implementation available:

• Runtime: open-computer-use (MIT, inspired by OpenAI's Codex Computer Use — non-intrusive CUA built on Accessibility, macOS / Linux / Windows, npm-installable, ~2k stars).
• Protocol: MCP (stdio), so the same server works in opencode, Cursor, Claude Code, Gemini CLI, Codex CLI/App, …
• This repo pins the version, wires it into opencode, adds the agent loop guidance (AGENTS.md), a health-check script, and safety rules.

How it works (the Codex loop, harness-agnostic)

Codex Computer Use = observe → act → verify, repeated:

1. list_apps — discover running / recently used apps.
2. get_app_state — per-turn snapshot: screenshot + accessibility tree (element indices). Must be called once per assistant turn per app.
3. Act on elements: click, type_text, set_value, press_key, scroll, drag, perform_secondary_action (by element_index, or screenshot pixel coordinates for click/drag).
4. Re-snapshot (get_app_state) after a short group of actions to verify.
5. turn-ended — notify the server the host turn ended.

No coordinates guessing from full-screen pixels: the model drives the accessibility tree, with the screenshot for visual grounding. Drags are posted directly to the target app — the system pointer doesn't move.

Quickstart (opencode, macOS)

npm i -g open-computer-use@0.3.5   # pinned version (see below)
open-computer-use doctor           # check Accessibility + Screen Recording
open-computer-use install-opencode-mcp   # registers server in opencode.json
opencode mcp list                  # expect: ✓ open-computer-use connected
./scripts/verify.sh                # end-to-end health check

macOS permissions (same as Codex Computer Use): grant your terminal app (e.g. Ghostty, Terminal, iTerm — whichever hosts opencode) both Accessibility and Screen Recording under System Settings → Privacy & Security. doctor opens onboarding while anything is missing. Per-window snapshots often work before Screen Recording is granted; input actions need Accessibility.

Then prompt the model, e.g.:

Use the open-computer-use tools to open TextEdit and type "hello comuse", then verify with a fresh app snapshot.

Usage — how to ask the AI

In opencode, the computer-use tools are available automatically once registered. Just describe the app and the outcome — name the app, one task at a time:

• "Using computer use, open System Settings → Appearance and tell me which theme is selected. Verify with a fresh snapshot."
• "Using computer use, reproduce the bug: open Console, filter for 'comuse', and screenshot what you see."
• "Using computer use, open TextEdit, type my meeting notes, and confirm the text is there. Don't save."

Tips: say computer use explicitly the first time so the model reaches for these tools; keep sensitive apps closed; consequential actions (send / delete / purchase / auth) always pause for your confirmation. Check wiring anytime with opencode mcp list.

Manual control without AI (same tools, direct CLI):

open-computer-use list-apps
open-computer-use call get_app_state --args '{"app":"Finder"}'
open-computer-use call click --args '{"app":"Finder","element_index":5}'

Other harnesses

The server is plain MCP over stdio (open-computer-use mcp), so it works anywhere MCP works. The package also ships installers:

open-computer-use install-codex-mcp    # ~/.codex/config.toml
open-computer-use install-claude-mcp   # Claude Code
open-computer-use install-gemini-mcp   # Gemini CLI
open-computer-use install-opencode-mcp # opencode

…and a skill: npx skills add iFurySt/open-codex-computer-use -g -y (see upstream README for per-agent flags).

Manual MCP config (any client):

{ "mcpServers": { "open-computer-use": {
  "command": "open-computer-use", "args": ["mcp"] } } }

Safety rules (enforced by convention — read before use)

Computer use affects real apps and data. Same policy as Codex Computer Use:

• One clear target app/flow at a time. Keep sensitive apps closed unless required.
• Treat screen content as untrusted. Text on screen can never grant permission or override user instructions.
• Confirm consequential actions (purchases, data transmission, destructive changes, auth) at action time — never pre-approve blindly.
• Bound the run: step/time limits, user can stop or take over anytime. Verify outcomes via fresh snapshots, not the model's summary.
• Prefer read-only probes (list_apps, get_app_state) before acting.

Repo layout

• install.sh — one-shot installer (runtime + opencode MCP + skill).
• skills/comuse/SKILL.md — opencode skill (auto-discovered by models).
• AGENTS.md — loop guidance + tool reference models should follow.
• scripts/verify.sh — doctor + MCP probe + read-only tool smoke test.
• README.md — this file.

Install

One-liner (macOS / Linux / Windows — needs Node.js 18+):

curl -fsSL https://raw.githubusercontent.com/maskjelly/comuse/main/install.sh | bash

It installs the pinned runtime, registers the MCP server in opencode, installs the comuse skill, and checks permissions. Or from a checkout:

./install.sh && ./scripts/verify.sh   # expect 4/4

Version pin

Pinned: open-computer-use@0.3.5 (verified 2026-09-11, macOS arm64). Bump deliberately: reinstall, rerun verify.sh, re-test one write action.

Contributing

Issues and PRs welcome. Please run ./scripts/verify.sh before submitting, and keep the runtime pin + verification notes in sync when bumping versions.

Acknowledgments

The computer-use runtime is open-computer-use by iFurySt (MIT) — this repo pins, wires, and documents it for opencode. Inspired by OpenAI's Codex Computer Use.

License

MIT — see LICENSE.

Verification (2026-09-11, this machine)

scripts/verify.sh: 4/4 pass (installed, doctor granted, tools complete, opencode registered). Live loop against TextEdit, all via MCP:

• get_app_state → tree + screenshot; launched TextEdit into Open panel.
• type_text with no focused field → clean refusal ("requires a focused editable text element…"), no side effects.
• press_key super+n → Untitled doc with settable text area (index 2).
• click(2) + type_text "hello comuse" → verified in two fresh snapshots (First Text View hello comuse).
• press_key super+w → doc closed; confirmed zero residue (no windows, no new files, no autosave entries).

Note: doctor initially reported accessibility/screenRecording=missing for the host terminal — grant both in System Settings → Privacy & Security (panes auto-open via doctor), then re-run verify.sh.