esp

Screen hands for local agents, with a dead-man's switch.

Every computer-use agent runs into the same two problems: it pays a vision API for every screenshot, and it clicks blind pixel coordinates that break the moment a window moves. esp solves both locally. It reads the Windows screen through UI Automation, targeting elements by name, with OCR as the fallback for anything UIA cannot see (browsers, Electron apps, canvases). It clicks, types, scrolls, and presses hotkeys through one MCP server your agent already knows how to talk to. No cloud calls, no per-frame cost, no coordinates baked into your prompts. And because an agent moving your real mouse is a loaded weapon, esp ships with the M0 Guard: the instant you touch the physical mouse or keyboard, the agent yields; a cyan border shows when it has control; a hard-kill hotkey ends it unconditionally.

• UIA-by-name targeting with OCR fallback. Click "Save", not (743, 512). When the target has no accessible name (a browser button, a canvas, an Electron app), esp falls back to reading the pixels with the built-in Windows OCR engine.
• Zero-cloud vision. Local OCR and UIA element trees. No API key, no per-screenshot bill, no network round trip in the hot path.
• M0 Guard safety core. Physical input from you instantly preempts the agent. A visible cyan border shows exactly when it has control. One hotkey kills it unconditionally, no matter what it's doing.

Platform: Windows. You are on the windows branch. The Mac port lives on the mac branch. The core (agent loop, esp.op API, drawing pipeline, benchmarks) is shared; only the OS-glue differs (esp/winapi.py, esp/uia.py, esp/winevents.py, esp/hooks.py, Windows OCR).

Install

Requires Python 3.10+ on Windows.

git clone https://github.com/iamagathodamon/esp.git
cd esp
pip install -r requirements.txt

Quickstart

Start the sense daemon, then drive it like a set of hands:

python -m esp.sensed                         # the daemon, listens on 127.0.0.1:3400
python -m esp.tail --quiet-pixels             # watch the event stream (optional)

python -m esp.op look                        # foreground window + elements + screenshot
python -m esp.op click "Save"                 # click the UIA element named Save
python -m esp.op fill "Search" "weather"      # focus a field by name and type into it
python -m esp.op press enter --expect Notepad # requires a foreground-title match or refuses

# narrate in real time: a speech bubble that follows the mouse
python -m esp.op say "Checking your network settings now"
python -m esp.op click "Save" --say "Saving so we don't lose this"
python -m esp.op hush

MCP server

esp_mcp.py exposes every daemon verb as a typed MCP tool (stdio transport, mcp Python SDK), so any Claude session calls esp natively instead of shelling out to esp.op. Thin wrappers only: each tool sends the exact bus command the CLI would; the daemon stays the single brain.

Register once, user scope, from inside the cloned repo directory:

claude mcp add --scope user esp -- python "%CD%/esp_mcp.py"

Tools: read-only esp_status / esp_look / esp_ocr / esp_findtext / esp_elements / esp_focused / esp_clipboard / esp_screenshot, display-only esp_say / esp_hush, and action tools esp_click_text / esp_click_element / esp_click_xy / esp_move / esp_fill / esp_type / esp_press / esp_hotkey / esp_scroll / esp_drag. Every action tool's description warns the calling model it drives the user's real mouse/keyboard. esp_click_text and esp_click_element take dry=true to locate/resolve the target with zero input. The M0 safety core applies unchanged: physical input yields, Ctrl+Alt+Q kills.

Safety

1. Ctrl+Alt+Q on the physical keyboard hard-kills the process, always.
2. Any physical keydown or mouse motion while esp holds control yields instantly. Detection uses the Windows low-level hook INJECTED flag, so esp never mistakes its own input for yours.
3. A cyan screen border means esp has the mouse. No border, no control.
4. Nothing injects input except through Guard.checkpoint().
5. Keyboard verbs require an expect foreground-title match, or the daemon refuses to type. Text never lands in the wrong window.

esp assumes you may be at the keyboard at the same moment it is. The Guard is not a confirmation dialog you can click through; it's a low-level input hook that yields control the instant it sees real hardware input, before the agent's next action fires.

Architecture

• M0 - Guard (v0.1.0): kill-switch core. All input injection passes through a Guard; any physical user input yields instantly.
• M1 - Sense daemon (v0.2.0): continuous capture, frame diff, win-event hooks, UIA cache, JSON-lines event bus on port 3400.
• M2 - Action engine (v0.3.0): SendInput at machine speed; instant/fast /human motion; unicode typing; focus assertions.
• M3 - Drawing pipeline (v0.4.0): image -> ordered strokes with offline fidelity scoring; circle -> portrait -> Mona Lisa in Paint.
• M4 - Brain link (v0.5.0): act verbs over the bus + a help hotkey publishing a screenshot event, so a model tailing the bus can act.
• M5 - Operator (v0.6.0): click/fill UIA elements by name (not pixels), a one-shot look perceive bundle, the esp.op CLI, paced typing for real apps.
• v0.8 - Vision fallback: Windows OCR as the read path for anything UIA cannot see, plus an awareness/control benchmark index.
• v0.10 - MCP server: every daemon verb as a typed MCP tool.

Demos and verification

python demos/m0_selftest.py   # safety core checks
python demos/m0_demo.py       # circles for 10s; grab the mouse to kill it
python demos/bubble_demo.py   # narration bubble follows YOUR mouse, no daemon
python demos/m1_selftest.py   # sense daemon end to end
python demos/m2_selftest.py   # action engine (idle-gated injection)
python demos/m3_selftest.py   # drawing pipeline, fully offline
python demos/m3_paint_demo.py --image mona.jpg   # esp draws in Paint
python demos/m4_selftest.py   # brain link over TCP (idle-gated)

Dependencies

mss, numpy, opencv-python for capture and drawing; pywinauto for UIA queries; the built-in Windows OCR engine (via winrt) for vision fallback; mcp only if you run esp_mcp.py. No services, no accounts, no recurring cost.