Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ agent-client-protocol-schema = { version = "=1.9.1", default-features = false }
# no new package; it adds the seccomp filter, the host-allowlist proxy and the
# `.git` protection the built-in engine lacks.
incord-sandbox = { path = "crates/sandbox" }
# Computer use (`crates/desktop`, binary `incu`): the 16-tool MCP server that
# Computer use (`crates/desktop`, binary `incu`): the 17-tool MCP server that
# `incode --desktop-mcp` wraps, and the journal/stop-file/capture code the run
# server and the toolbox call. A path dependency, so no new package: it carries
# the screen-capture, accessibility and input dependencies that used to be
Expand Down
40 changes: 35 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -850,7 +850,7 @@ anything runs.
| `Screenshot` | observe | A picture of a public page or a workspace HTML/SVG file. Offered only when the headless browser is found. |
| `RenderPdf` | change | A public page or a workspace HTML/SVG file printed to PDF. Offered only when a printer is found (below). |
| `ConvertDocument` | change | An office document to PDF through LibreOffice. Offered only when LibreOffice is found. |
| `screen_info` · `list_windows` · `screenshot` · `ui_tree` · `cursor_position` | observe | Look at the desktop. Offered only when `INCODE_DESKTOP=1` — see [Computer use](#computer-use). |
| `screen_info` · `list_windows` · `screenshot` · `ui_tree` · `cursor_position` · `wait_for` | observe | Look at the desktop, or wait (bounded) for a control, a window or a change of the screen. Offered only when `INCODE_DESKTOP=1` — see [Computer use](#computer-use). |
| `click_element` · `set_value` · `element_action` · `focus_window` · `type_text` · `paste_text` · `key` · `scroll` · `drag` · `move_mouse` · `click` | run | Act on the desktop. Same condition; refused in read-only. |
| `Skill` | observe | Read one skill's text from the shared store. |
| `Agent` | run | Hand a self-contained task to a sub-agent (a second headless run of the session's CLI, or the API route), with the session's mode, guards and cancel; at most 2 levels deep. Takes `agent: <name>` for a [user-defined sub-agent](#user-defined-sub-agents). |
Expand Down Expand Up @@ -1075,12 +1075,13 @@ plain screenshots and pixel clicks still work.
|---|---|
| `screen_info` | Displays, their sizes and scale, and on macOS any missing grant. |
| `list_windows` | The open windows: `id`, title, app, pid, focused, rectangle, and `tree` (false when the window exposes no accessibility tree). A blocked app's window is listed without its title. |
| `ui_tree` | One window's controls as `[e<id>] Role "name" (x,y,w×h)`, with `password` on secret fields. `window` picks the window (default: the focused one); `depth` limits the walk. Controls at no place on screen are counted on one line, not listed. A window with no tree is refused with a pointer to `screenshot` and `click`. |
| `ui_tree` | One window's controls as `[e<id>] Role "name" (x,y,w×h) sel: <selector>`, with `password` on secret fields; `sel:` names the control without its id (see [Selectors](#selectors-and-wait_for)). `window` picks the window (default: the focused one); `depth` limits the walk. Controls at no place on screen are counted on one line, not listed. A window with no tree is refused with a pointer to `screenshot` and `click`. |
| `screenshot` | The primary display, or one `window`. The reply carries the region and origin, so a point in the picture maps back to the screen. |
| `cursor_position` | Where the pointer is. |
| `click_element` | Click a control by id, at its centre as it is *now*. |
| `set_value` | Set a text field's value through the accessibility interface: no typing, no focus, no clipboard. Read back and marked `(verified)`, or `(unverified: why)`. A password field is never read back. |
| `element_action` | Run one of a control's own actions without the mouse: `press`, `toggle`, `expand`, `collapse`, `select` and `focus` everywhere, plus what the platform adds (`scroll_into_view` on Windows; `show_menu`, `increment`, `decrement`, `confirm`, `cancel` on macOS). An action the control lacks is refused, and the refusal lists the ones it offers. |
| `wait_for` | Wait, bounded, until a control matching a `selector` appears or goes, a window appears or closes, or the screen changes. Returns what it found, or a `timeout` reply with what it last saw. |
| `click_element` | Click a control by id or `selector`, at its centre as it is *now*. |
| `set_value` | Set a text field's value (by id or `selector`) through the accessibility interface: no typing, no focus, no clipboard. Read back and marked `(verified)`, or `(unverified: why)`. A password field is never read back. |
| `element_action` | Run one of a control's own actions (by id or `selector`) without the mouse: `press`, `toggle`, `expand`, `collapse`, `select` and `focus` everywhere, plus what the platform adds (`scroll_into_view` on Windows; `show_menu`, `increment`, `decrement`, `confirm`, `cancel` on macOS). An action the control lacks is refused, and the refusal lists the ones it offers. |
| `focus_window` | Bring a window to the front so typing reaches it. |
| `type_text` · `key` · `paste_text` | Keystrokes, a chord (`["cmdorctrl","s"]`: Cmd on macOS, Ctrl elsewhere), or a paste. `paste_text` puts the old clipboard back after `paste_restore_delay_ms`, and refuses to paste at all when the clipboard cannot be read. |
| `scroll` | Wheel steps (`dy` down, `dx` right) over a control by `id`, or under the pointer. |
Expand All @@ -1091,6 +1092,35 @@ Every acting tool also takes `window`; typing into a window that is not the
focused one is refused rather than sent elsewhere. It also takes
`screenshot_after`, which returns a picture once the screen has settled.

### Selectors and `wait_for`

An `eN` id is a position in one `ui_tree` result. A **selector** names a
control by what it is, and is resolved against the live tree when the call is
made, so it still means something in the next call, and after the program has
been restarted. `ui_tree` prints one beside every element (`sel: …`) and the
window part once at the top; `click_element`, `set_value`, `element_action`
and `wait_for` take it as `selector` in place of `id`. It must match exactly one
control: none, or several, is an error that says which, never a guess.

```
role=button;name=Save one control of the focused window
role=edit;aid=searchBox by the program's own automation id
role=button;name=Save;nth=1 the second of two look-alikes
app=notepad;window~=Save As;role=button;name=Save in another window
```

Keys: `app`, `window` / `window~` (exact / contains), `role`, `name` / `name~`,
`aid` (UIA `AutomationId`, AX `AXIdentifier`, AT-SPI `AccessibleId`), `path`
(child indices, `0.2.1`), `nth`, `depth` (how deep to walk; `ui_tree` writes it
when asked for a depth other than the default, so `nth` counts the same
controls when resolved). `;` and `\` in a value are written `\;` and
`\\`. `wait_for` takes `condition` (`element_appears`, `element_gone`,
`window_appears`, `window_gone`, `screen_changes`), a `selector` (window
conditions take only `app`/`window`), `timeout_ms` (default 5000, held to
60000) and `poll_ms` (250, 50–2000), and answers JSON whose `result` is `found`
or `timeout`. The full grammar and what each platform supplies is in
`docs/COMPUTER_USE.md`.

### What it will not do

- **Act on a control it did not show you.** An id is a position in one
Expand Down
2 changes: 1 addition & 1 deletion crates/desktop/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ version = "0.1.0"
edition = "2021"
rust-version = "1.88"
license = "Apache-2.0"
description = "Computer use for an agent: screenshots, accessibility trees, mouse and keyboard, as a 16-tool MCP server with a kill switch and a watch journal"
description = "Computer use for an agent: screenshots, accessibility trees, mouse and keyboard, as a 17-tool MCP server with a kill switch and a watch journal"
repository = "https://github.com/incordai/Incode"
keywords = ["computer-use", "mcp", "accessibility", "agents", "automation"]
categories = ["command-line-utilities", "accessibility"]
Expand Down
166 changes: 166 additions & 0 deletions crates/desktop/src/fingerprint.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
//! A small fingerprint of a picture of the screen, for asking "did it change?"
//! without keeping or sending the picture (`wait_for` `screen_changes`).
//!
//! # Why two parts
//!
//! A difference hash (does each cell get brighter or darker than its
//! neighbour) sees edges, which is what a dialog opening or a page loading
//! changes. It is blind to a change of brightness alone: a white screen and a
//! black one have no edges, so they hash the same. So the fingerprint is 128
//! bits: that gradient hash (64) beside the average brightness of a 4x4 grid,
//! four bits a cell (64). The screens are compared by how many bits differ
//! ([`distance`]), so a blinking cursor or a clock tick that moves a bit or two
//! is below a tolerance, and a window opening is far above it.
//!
//! Pure over pixels, so tested on every platform; the capture that feeds it is
//! `mcp::wait_tool`.

use image::imageops::{grayscale, resize, FilterType};
use image::RgbaImage;

/// 128 bits: gradient hash in the high half, brightness grid in the low half.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Fingerprint(pub u128);

/// The fingerprint of `picture`.
#[must_use]
pub fn of(picture: &RgbaImage) -> Fingerprint {
let grey = grayscale(picture);

// Gradient: 9x8 cells, one bit for each of the 8x8 left-brighter-than-right.
let cells = resize(&grey, 9, 8, FilterType::Triangle);
let mut gradient = 0u64;
for y in 0..8 {
for x in 0..8 {
gradient <<= 1;
if cells.get_pixel(x, y).0[0] > cells.get_pixel(x + 1, y).0[0] {
gradient |= 1;
}
}
}

// Brightness: 4x4 cells, the top four bits of each average.
let grid = resize(&grey, 4, 4, FilterType::Triangle);
let mut levels = 0u64;
for y in 0..4 {
for x in 0..4 {
levels = (levels << 4) | u64::from(grid.get_pixel(x, y).0[0] >> 4);
}
}

Fingerprint((u128::from(gradient) << 64) | u128::from(levels))
}

/// How many of the 128 bits differ.
#[must_use]
pub fn distance(a: Fingerprint, b: Fingerprint) -> u32 {
(a.0 ^ b.0).count_ones()
}

impl Fingerprint {
/// 32 hex digits, for a reply the next call can pass back as a baseline.
#[must_use]
pub fn hex(self) -> String {
format!("{:032x}", self.0)
}

/// [`Self::hex`]'s inverse; `None` for anything that is not 32 hex digits.
#[must_use]
pub fn from_hex(text: &str) -> Option<Fingerprint> {
let text = text.trim();
(text.len() == 32 && text.chars().all(|c| c.is_ascii_hexdigit()))
.then(|| u128::from_str_radix(text, 16).ok().map(Fingerprint))
.flatten()
}
}

#[cfg(test)]
mod tests {
use super::*;
use image::Rgba;

fn flat(width: u32, height: u32, grey: u8) -> RgbaImage {
RgbaImage::from_pixel(width, height, Rgba([grey, grey, grey, 255]))
}

/// A page: light, with a dark bar across the top and a dark box inside.
fn page() -> RgbaImage {
let mut img = flat(320, 200, 235);
for y in 0..20 {
for x in 0..320 {
img.put_pixel(x, y, Rgba([40, 40, 40, 255]));
}
}
for y in 80..140 {
for x in 100..220 {
img.put_pixel(x, y, Rgba([90, 120, 200, 255]));
}
}
img
}

#[test]
fn the_same_picture_is_the_same_fingerprint() {
assert_eq!(distance(of(&page()), of(&page())), 0);
assert_eq!(of(&page()), of(&page()));
}

#[test]
fn white_and_black_differ_though_neither_has_an_edge() {
let d = distance(of(&flat(160, 90, 255)), of(&flat(160, 90, 0)));
assert!(
d >= 16,
"a whole-screen brightness change is far apart: {d}"
);
}

#[test]
fn a_dialog_opening_is_far_above_a_tolerance_of_a_few_bits() {
let before = page();
let mut after = page();
// A dark dialog over the middle third.
for y in 50..150 {
for x in 60..260 {
after.put_pixel(x, y, Rgba([20, 20, 30, 255]));
}
}
let d = distance(of(&before), of(&after));
assert!(d > 8, "{d}");
}

#[test]
fn a_blinking_cursor_is_below_the_default_tolerance() {
let before = page();
let mut after = page();
for y in 100..112 {
for x in 150..152 {
after.put_pixel(x, y, Rgba([0, 0, 0, 255]));
}
}
let d = distance(of(&before), of(&after));
assert!(d <= 4, "a 2x12 pixel mark moved {d} bits");
}

#[test]
fn the_size_of_the_picture_does_not_matter() {
let small = of(&image::imageops::resize(
&page(),
160,
100,
FilterType::Triangle,
));
let large = of(&page());
assert!(distance(small, large) <= 4);
}

#[test]
fn hex_round_trips_and_refuses_anything_else() {
let f = of(&page());
assert_eq!(f.hex().len(), 32);
assert_eq!(Fingerprint::from_hex(&f.hex()), Some(f));
assert_eq!(Fingerprint::from_hex(&format!(" {} ", f.hex())), Some(f));
assert_eq!(Fingerprint::from_hex("abc"), None);
assert_eq!(Fingerprint::from_hex(&"z".repeat(32)), None);
assert_eq!(Fingerprint(0).hex(), "0".repeat(32));
}
}
5 changes: 4 additions & 1 deletion crates/desktop/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
//!
//! - **This is offered to a DELEGATED agent over MCP.** It is not dispatched by
//! a loop, because there is no such loop: the agent runs in its own process
//! and reaches the screen only through the 16 tools of [`mcp`].
//! and reaches the screen only through the 17 tools of [`mcp`].
//!
//! - **This crate does not depend on incode.** It began as a module of incode
//! (`src/desktop/`) and moved out whole. What it used to take from the rest
Expand Down Expand Up @@ -45,14 +45,17 @@ pub mod cli;
pub mod config;
pub mod coords;
pub mod dpi;
pub mod fingerprint;
pub mod hooks;
pub mod journal;
pub mod limits;
pub mod mcp;
pub mod permission;
pub mod safety;
pub mod selector;
pub mod thread;
pub mod uia;
pub mod wait;

#[cfg(target_os = "linux")]
pub mod linux;
Expand Down
60 changes: 48 additions & 12 deletions crates/desktop/src/linux.rs
Original file line number Diff line number Diff line change
Expand Up @@ -486,9 +486,28 @@ fn describe(bus: &Connection, node: &Node, id: u32, depth: u32) -> Element {
rect,
enabled: is_enabled(&states(bus, node)),
secret: role_name == "password text",
automation_id: accessible_id(bus, node),
}
}

/// The object's `AccessibleId` property (AT-SPI 2.46 and later; GTK 4 and Qt
/// fill it from the widget's id), empty when the toolkit has none or the
/// registry is older. UNVERIFIED against a live bus: this reads the property
/// the same way [`name`] reads `Name`, and an error is the empty answer.
fn accessible_id(bus: &Connection, node: &Node) -> String {
bus.call_method(
Some(node.0.as_str()),
node.1.as_str(),
Some("org.freedesktop.DBus.Properties"),
"Get",
&(ACCESSIBLE, "AccessibleId"),
)
.and_then(|reply| reply.body().deserialize::<OwnedValue>())
.ok()
.and_then(|value| String::try_from(value).ok())
.unwrap_or_default()
}

fn extents(bus: &Connection, node: &Node) -> Rect {
bus.call_method(
Some(node.0.as_str()),
Expand Down Expand Up @@ -1124,13 +1143,18 @@ pub(crate) fn window_info(window: Option<&str>) -> Result<WindowInfo, DesktopErr
Ok(target(&desktop, window)?.info.clone())
}

pub(crate) fn tree(window: Option<&str>, max_depth: u32) -> Result<Tree, DesktopError> {
pub(crate) fn tree(
window: Option<&str>,
max_depth: u32,
keep: usize,
) -> Result<Tree, DesktopError> {
let desktop = desktop()?;
let listed = target(&desktop, window)?;
let (bus, window) = accessible(&desktop, listed)?;
let walked = uia::walk(
window,
max_depth,
keep,
&|node: &Node, id, depth| describe(bus, node, id, depth),
&|node: &Node| children(bus, node),
);
Expand Down Expand Up @@ -1207,6 +1231,13 @@ pub(crate) fn set_value(
let desktop = desktop()?;
let (bus, node, described) = control(&desktop, window, want, max_depth)?;
let what = described.label();
// What it held before (never a password field's), to tell a field that
// ignored the set from one whose read-back is unreliable.
let before = (!described.secret)
.then(|| {
ask::<String, _>(bus, &node, "org.a11y.atspi.Text", "GetText", &(0i32, -1i32)).ok()
})
.flatten();
let set: bool = ask(
bus,
&node,
Expand All @@ -1223,21 +1254,25 @@ pub(crate) fn set_value(
return Err(DesktopError::Os(format!("{what} refused the new text")));
}
let did = described.set_to(value);
let tried = described.tried_to_set(value);
if described.secret {
return Ok(Done::unverified(did, "a password field is never read back"));
return Ok(Done::unverified(
tried,
"a password field is never read back",
));
}
Ok(
match ask::<String, _>(bus, &node, "org.a11y.atspi.Text", "GetText", &(0i32, -1i32)) {
Ok(now) if now == value => Done::verified(did),
Ok(now) => Done::unverified(
did,
&format!(
"it reads back as {} characters, not the {} sent",
now.chars().count(),
value.chars().count()
),
),
Err(_) => Done::unverified(did, "its text could not be read back"),
Ok(now) => match uia::judge(value, before.as_deref(), &now, false) {
uia::ReadBack::Same => Done::verified(did),
uia::ReadBack::Unverified(why) => Done::unverified(tried, &why),
uia::ReadBack::Differs(why) => {
return Err(DesktopError::Os(format!(
"{what} did not take the value: {why}"
)))
}
},
Err(_) => Done::unverified(tried, "its text could not be read back"),
},
)
}
Expand Down Expand Up @@ -1511,6 +1546,7 @@ mod tests {
},
enabled: true,
secret: false,
automation_id: String::new(),
}
}

Expand Down
Loading
Loading