From b91904ede1cab5dbb9380de1bf559ce6d525a890 Mon Sep 17 00:00:00 2001 From: blockmandev <194144408+blockmandev@users.noreply.github.com> Date: Fri, 13 Nov 2026 18:06:15 -0600 Subject: [PATCH 1/6] desktop: routines learned from Watchman records, replayed with no model, and watch A routine is a trigger phrase and the cleaned desktop steps that carried it out, built from Watchman's own tool-call records (read from the local memory engine's /v1/memory/watchman/record/{id}, behind a StepSource trait with a Watchman, an in-memory and a no-engine implementation) or from an explicit step list. Six new tools, 23 in all: routine_save, routine_run, routine_forget (acting) and routine_list, routine_match, watch (observing). Learning follows Atticus's learned shortcuts: nothing is kept when the phrase says this/that, when a call failed or a wait_for timed out, or when a step uses an eN id, a coordinate or a window handle; looking steps are dropped, wait_for is kept. From the live test: every acting step must name its window (selector window part, or in_window for typing and keys, taken from the step before when a recording has none), scroll is never replayable. Replay runs each step through Server::gated_call, the path tools/call takes (stop file, tier, pre/post hooks, blocked apps, rate cap, journal), stops at the first failure and says which step and why, with no repair. Links and buttons are invoked through element_action invoke first; a set_value or element_action that answers unverified stops the replay when the next step acts on it. Matching is by word set, any order, filler ignored. One JSON file, written atomically, expiring after 30 unused days unless taught; path overridable in Config / INCU_ROUTINES. watch is wait_for for up to 24 hours, polled gently, that tells the operator's post-tool hook what it saw. --- Cargo.toml | 2 +- README.md | 48 +- crates/desktop/Cargo.toml | 2 +- crates/desktop/src/cli.rs | 41 + crates/desktop/src/config.rs | 21 +- crates/desktop/src/lib.rs | 5 +- crates/desktop/src/mcp.rs | 30 +- crates/desktop/src/mcp/definitions.rs | 9 +- crates/desktop/src/mcp/gate.rs | 35 +- crates/desktop/src/mcp/routine_definitions.rs | 248 +++++ crates/desktop/src/mcp/routine_tests.rs | 771 ++++++++++++++++ crates/desktop/src/mcp/routine_tool.rs | 394 ++++++++ crates/desktop/src/mcp/tools.rs | 45 +- crates/desktop/src/mcp/wait_tool.rs | 54 +- crates/desktop/src/permission.rs | 26 +- .../fixtures/record_failed_mcp_call.json | 1 + .../routine/fixtures/record_shell_call.json | 1 + .../fixtures/record_structured_output.json | 1 + crates/desktop/src/routine/learn.rs | 868 ++++++++++++++++++ crates/desktop/src/routine/mod.rs | 246 +++++ crates/desktop/src/routine/replay.rs | 455 +++++++++ crates/desktop/src/routine/source.rs | 149 +++ crates/desktop/src/routine/store.rs | 504 ++++++++++ crates/desktop/src/routine/watchman.rs | 480 ++++++++++ crates/desktop/src/routine/words.rs | 144 +++ crates/desktop/src/wait.rs | 140 ++- docs/ARCHITECTURE.md | 6 +- docs/COMPUTER_USE.md | 149 ++- src/desktop.rs | 18 +- src/permissions/mod.rs | 21 +- src/toolbox/desktop.rs | 28 +- 31 files changed, 4886 insertions(+), 56 deletions(-) create mode 100644 crates/desktop/src/mcp/routine_definitions.rs create mode 100644 crates/desktop/src/mcp/routine_tests.rs create mode 100644 crates/desktop/src/mcp/routine_tool.rs create mode 100644 crates/desktop/src/routine/fixtures/record_failed_mcp_call.json create mode 100644 crates/desktop/src/routine/fixtures/record_shell_call.json create mode 100644 crates/desktop/src/routine/fixtures/record_structured_output.json create mode 100644 crates/desktop/src/routine/learn.rs create mode 100644 crates/desktop/src/routine/mod.rs create mode 100644 crates/desktop/src/routine/replay.rs create mode 100644 crates/desktop/src/routine/source.rs create mode 100644 crates/desktop/src/routine/store.rs create mode 100644 crates/desktop/src/routine/watchman.rs create mode 100644 crates/desktop/src/routine/words.rs diff --git a/Cargo.toml b/Cargo.toml index 416241b..8fd778e 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -90,7 +90,7 @@ agent-client-protocol-schema = { version = "=1.9.1", default-features = false } # no new package; it adds the seccomp filter, the host-allowlist proxy and the # `.git` protection the built-in engine lacks. incord-sandbox = { path = "crates/sandbox" } -# Computer use (`crates/desktop`, binary `incu`): the 17-tool MCP server that +# Computer use (`crates/desktop`, binary `incu`): the 23-tool MCP server that # `incode --desktop-mcp` wraps, and the journal/stop-file/capture code the run # server and the toolbox call. A path dependency, so no new package: it carries # the screen-capture, accessibility and input dependencies that used to be diff --git a/README.md b/README.md index 2f7f781..3b1066f 100644 --- a/README.md +++ b/README.md @@ -850,8 +850,8 @@ anything runs. | `Screenshot` | observe | A picture of a public page or a workspace HTML/SVG file. Offered only when the headless browser is found. | | `RenderPdf` | change | A public page or a workspace HTML/SVG file printed to PDF. Offered only when a printer is found (below). | | `ConvertDocument` | change | An office document to PDF through LibreOffice. Offered only when LibreOffice is found. | -| `screen_info` · `list_windows` · `screenshot` · `ui_tree` · `cursor_position` · `wait_for` | observe | Look at the desktop, or wait (bounded) for a control, a window or a change of the screen. Offered only when `INCODE_DESKTOP=1` — see [Computer use](#computer-use). | -| `click_element` · `set_value` · `element_action` · `focus_window` · `type_text` · `paste_text` · `key` · `scroll` · `drag` · `move_mouse` · `click` | run | Act on the desktop. Same condition; refused in read-only. | +| `screen_info` · `list_windows` · `screenshot` · `ui_tree` · `cursor_position` · `wait_for` · `watch` · `routine_list` · `routine_match` | observe | Look at the desktop, wait (bounded) for a control, a window or a change of the screen, watch for one for hours, or look up a saved routine. Offered only when `INCODE_DESKTOP=1` — see [Computer use](#computer-use). | +| `click_element` · `set_value` · `element_action` · `focus_window` · `type_text` · `paste_text` · `key` · `scroll` · `drag` · `move_mouse` · `click` · `routine_save` · `routine_run` · `routine_forget` | run | Act on the desktop, or keep, replay and forget a routine of such actions. Same condition; refused in read-only. | | `Skill` | observe | Read one skill's text from the shared store. | | `Agent` | run | Hand a self-contained task to a sub-agent (a second headless run of the session's CLI, or the API route), with the session's mode, guards and cancel; at most 2 levels deep. Takes `agent: ` for a [user-defined sub-agent](#user-defined-sub-agents). | | `AskUserQuestion` | session | A multiple-choice question to the owner: answered in the owner's app through the memory engine, else at the terminal, else reported unanswered. | @@ -1079,6 +1079,9 @@ plain screenshots and pixel clicks still work. | `screenshot` | The primary display, or one `window`. The reply carries the region and origin, so a point in the picture maps back to the screen. | | `cursor_position` | Where the pointer is. | | `wait_for` | Wait, bounded, until a control matching a `selector` appears or goes, a window appears or closes, or the screen changes. Returns what it found, or a `timeout` reply with what it last saw. | +| `watch` | `wait_for` for up to 24 hours, looked at every second, with no model polling. When it ends, your post-tool hook for `watch` is run with the reply as `tool_response`: that is the notification or follow-up. **The call blocks** until then. See [Routines and `watch`](#routines-and-watch). | +| `routine_save` · `routine_run` · `routine_forget` | Keep what was done as a routine, run it again with no model, or forget it. | +| `routine_list` · `routine_match` | List the saved routines, or show which one a phrase would run without running it. | | `click_element` | Click a control by id or `selector`, at its centre as it is *now*. | | `set_value` | Set a text field's value (by id or `selector`) through the accessibility interface: no typing, no focus, no clipboard. Read back and marked `(verified)`, or `(unverified: why)`. A password field is never read back. | | `element_action` | Run one of a control's own actions (by id or `selector`) without the mouse: `press`, `toggle`, `expand`, `collapse`, `select` and `focus` everywhere, plus what the platform adds (`scroll_into_view` on Windows; `show_menu`, `increment`, `decrement`, `confirm`, `cancel` on macOS). An action the control lacks is refused, and the refusal lists the ones it offers. | @@ -1121,6 +1124,47 @@ conditions take only `app`/`window`), `timeout_ms` (default 5000, held to or `timeout`. The full grammar and what each platform supplies is in `docs/COMPUTER_USE.md`. +### Routines and `watch` + +A **routine** is a phrase and the desktop calls that carried it out, kept so +`routine_run` can do it again with no model. It is made from what Watchman +already recorded (`routine_save` with `records`, the `w-…` ids of the turns that +did the task, read from the local memory engine) or from an explicit `steps` +list; the engine is optional. + +- `ui_tree`, `screenshot` and the other looking calls are dropped; `wait_for` is + kept. +- **Nothing is kept** when the phrase says "this" or "that", when a call failed + or a `wait_for` timed out, or when any step uses an `eN` id, a pixel + coordinate (`click`, `move_mouse`, `drag`) or a window handle (`window`, + `focus_window`). Only selector-based steps replay, and each must say where: + a selector needs an `app=`/`window=` part, and a typing or key step carries + `in_window` (a recording has none, so it is given the window of the step before + it; the replay still refuses to type unless that window has the focus). + `scroll` is never replayable. A replay invokes links and buttons through + `element_action invoke` (no mouse, no focus) and falls back to the recorded + click, and it stops when a `set_value` or `element_action` answers + "unverified" and the next step acts on it. +- A phrase matches by its **words in any order, ignoring filler words**; a near + miss runs nothing. `routine_match` shows what a phrase would run. +- A replay runs every step through the same checks as a live call (blocked apps, + hooks, stop file, journal) and **stops at the first failure**, saying which + step and why. There is no repair: do the task yourself and `routine_save` it + under the same phrase to replace the routine. +- A routine unused for 30 days is forgotten, unless saved with `taught: true`. +- The file is `routines.json` in the per-user data folder + (`%APPDATA%\incord-desktop`, `~/Library/Application Support/incord-desktop`, + or `$XDG_DATA_HOME`/`~/.local/share/incord-desktop`), overridden by + `INCU_ROUTINES` or `routines` in the config. Typed text is stored in it as + typed: do not save a routine that types a secret. + +`watch` is `wait_for` with a long deadline (default 1 hour, at most 24, looked +at every second): it blocks until the condition holds, then runs your +post-tool hook for the tool `watch` with the reply (`result`, what it found, the +`message` you passed) as `tool_response`. Put the notification in that hook, for +example `osascript -e 'display notification "…"'` on macOS or `notify-send` +on Linux. The stop file ends a watch within a second. + ### What it will not do - **Act on a control it did not show you.** An id is a position in one diff --git a/crates/desktop/Cargo.toml b/crates/desktop/Cargo.toml index e914d57..6985eb4 100644 --- a/crates/desktop/Cargo.toml +++ b/crates/desktop/Cargo.toml @@ -5,7 +5,7 @@ version = "0.1.0" edition = "2021" rust-version = "1.88" license = "Apache-2.0" -description = "Computer use for an agent: screenshots, accessibility trees, mouse and keyboard, as a 17-tool MCP server with a kill switch and a watch journal" +description = "Computer use for an agent: screenshots, accessibility trees, mouse and keyboard, as a 23-tool MCP server with a kill switch, a watch journal and replayable routines" repository = "https://github.com/incordai/Incode" keywords = ["computer-use", "mcp", "accessibility", "agents", "automation"] categories = ["command-line-utilities", "accessibility"] diff --git a/crates/desktop/src/cli.rs b/crates/desktop/src/cli.rs index 8f91d56..60b04f3 100644 --- a/crates/desktop/src/cli.rs +++ b/crates/desktop/src/cli.rs @@ -57,6 +57,14 @@ OPTIONS for serve: $INCU_CONFIG, else the platform config folder --display Linux: \":N\" or \"xvfb[:WxH]\" (overrides the config) +ENVIRONMENT for serve: + INCU_ROUTINES the file learned routines are kept in (else the + config's `routines`, else routines.json in the + per-user data folder) + INCORD_MEMORY_URL the local memory engine routine_save reads + Watchman records from (default + http://127.0.0.1:9180; loopback only) + OPTIONS for stop: --journal stop the server journalling to PATH --workspace stop the server for DIR's .incode/ journal @@ -206,6 +214,9 @@ pub struct ServeOptions { pub display: Option, /// The operator's commands around every call, if any. pub hooks: Option>, + /// Where routines are kept; `None` is the default + /// ([`crate::routine::store::Store::default_path`]). + pub routines: Option, /// What the startup lines on stderr call this server (`incu`, or /// `incode-desktop` for incode's wrapper). pub name: &'static str, @@ -228,6 +239,7 @@ impl ServeOptions { limits, display, hooks, + routines, } = config; Ok(Self { tier: args.tier, @@ -235,11 +247,27 @@ impl ServeOptions { limits, display: args.display.or(display), hooks: shell_hooks(hooks), + routines, name: "incu", }) } } +/// The routines kit `incu` runs with: the file at `path`, else the default, and +/// Watchman's records from the local memory engine as the source. `None` when +/// there is no place to keep the file. +/// +/// The engine is only asked when a routine is saved from record ids, and an +/// engine that is not running is an error at that moment, not at startup: the +/// crate works without it. +fn routine_kit(path: Option) -> Option { + let path = path.or_else(crate::routine::store::Store::default_path)?; + Some(crate::mcp::RoutineKit::new( + crate::routine::store::Store::at(path), + Box::new(crate::routine::watchman::WatchmanSource::from_env()), + )) +} + /// [`ShellHooks`] for `config`, or none when it has nothing to run. fn shell_hooks(config: HookConfig) -> Option> { (!config.is_empty()).then(|| Box::new(ShellHooks::new(config)) as Box) @@ -269,6 +297,7 @@ pub fn serve(options: ServeOptions) -> Result<(), String> { limits, display, hooks, + routines, name, } = options; crate::limits::set( @@ -307,6 +336,18 @@ pub fn serve(options: ServeOptions) -> Result<(), String> { eprintln!("{name} journalling to {}", path.display()); server = server.with_journal(path); } + server = match routine_kit(routines) { + Some(kit) => { + eprintln!("{name}: routines in {}", kit.path().display()); + server.with_routines(kit) + } + None => { + eprintln!( + "{name}: routines are off: no data folder to keep them in (set INCU_ROUTINES)" + ); + server + } + }; // Named on stderr so an operator who asked for `read-only` can see that it // arrived. A capability restriction nobody can observe is one nobody trusts. eprintln!("{name} MCP server ready on stdio ({})", tier.label()); diff --git a/crates/desktop/src/config.rs b/crates/desktop/src/config.rs index 0371b55..de57672 100644 --- a/crates/desktop/src/config.rs +++ b/crates/desktop/src/config.rs @@ -1,5 +1,5 @@ -//! The settings the crate owns: limits, the Linux display and the operator's -//! hooks. +//! The settings the crate owns: limits, the Linux display, the operator's +//! hooks and where routines are kept. //! //! # One file for either binary //! @@ -48,6 +48,11 @@ pub struct Config { /// The operator's commands around every call. See [`crate::hooks`]. #[serde(default, alias = "desktop_hooks", alias = "hooks")] pub hooks: HookConfig, + /// Where learned routines are kept. Unset: `$INCU_ROUTINES`, else + /// `routines.json` in the per-user data folder + /// ([`crate::routine::store::Store::default_path`]). + #[serde(default, alias = "routines_file")] + pub routines: Option, } impl Config { @@ -149,6 +154,18 @@ mod tests { assert_eq!(config.display.as_deref(), Some(":99")); } + /// Where routines are kept is overridable, under either name. + #[test] + fn the_routines_file_can_be_named_in_the_config() { + let config: Config = + serde_json::from_str(r#"{"routines": "/data/r.json"}"#).expect("config"); + assert_eq!(config.routines, Some(PathBuf::from("/data/r.json"))); + let alias: Config = + serde_json::from_str(r#"{"routines_file": "/data/s.json"}"#).expect("config"); + assert_eq!(alias.routines, Some(PathBuf::from("/data/s.json"))); + assert_eq!(Config::default().routines, None); + } + #[test] fn nothing_set_is_the_default() { let config: Config = serde_json::from_str("{}").expect("config"); diff --git a/crates/desktop/src/lib.rs b/crates/desktop/src/lib.rs index 493b12f..d05ffb0 100644 --- a/crates/desktop/src/lib.rs +++ b/crates/desktop/src/lib.rs @@ -8,7 +8,9 @@ //! //! - **This is offered to a DELEGATED agent over MCP.** It is not dispatched by //! a loop, because there is no such loop: the agent runs in its own process -//! and reaches the screen only through the 17 tools of [`mcp`]. +//! and reaches the screen only through the 23 tools of [`mcp`]: the 17 +//! that look and act, `watch`, and five that keep and replay what was done +//! ([`routine`]). //! //! - **This crate does not depend on incode.** It began as a module of incode //! (`src/desktop/`) and moved out whole. What it used to take from the rest @@ -51,6 +53,7 @@ pub mod journal; pub mod limits; pub mod mcp; pub mod permission; +pub mod routine; pub mod safety; pub mod selector; pub mod thread; diff --git a/crates/desktop/src/mcp.rs b/crates/desktop/src/mcp.rs index aa07719..9c839f5 100644 --- a/crates/desktop/src/mcp.rs +++ b/crates/desktop/src/mcp.rs @@ -52,8 +52,11 @@ //! - [`tools`]: `run_tool` and one step per tool. //! - [`guard`]: the blocked-application, focus, point and region refusals a //! tool asks before it acts. -//! - `wait_tool`: `wait_for`, the one tool that polls: its arguments and the -//! live desktop, over the timing rules in [`crate::wait`]. +//! - `wait_tool`: `wait_for` and `watch`, the two tools that poll: their +//! arguments and the live desktop, over the timing rules in [`crate::wait`]. +//! - `routine_tool` and `routine_definitions`: the five routine tools over +//! [`crate::routine`], and `Server::gated_call`, the one path through every +//! gate that a live call and each step of a replayed routine both take. //! - `tests`: the tests, at their old path `mcp::tests`. //! //! # Wired to @@ -75,6 +78,8 @@ use crate::permission::Tier; mod definitions; mod gate; mod guard; +mod routine_definitions; +mod routine_tool; mod tools; mod transport; mod wait_tool; @@ -85,7 +90,11 @@ mod tests; #[cfg(test)] mod selector_tests; +#[cfg(test)] +mod routine_tests; + pub use definitions::tool_definitions; +pub use routine_tool::RoutineKit; pub use transport::run_stdio; use transport::{code, Failure}; @@ -128,6 +137,9 @@ pub struct Server { /// the contract, and for why a hook sees the real input rather than the /// redacted description the journal gets. hooks: Option>, + /// The routines file and where records come from. `None` until + /// [`Server::with_routines`]; the routine tools refuse by name without it. + routines: Option, } impl Server { @@ -142,6 +154,7 @@ impl Server { stop_file: None, tier: Tier::Act, hooks: None, + routines: None, } } @@ -227,15 +240,10 @@ impl Server { // tool refused. Reporting it as a protocol error would make a refused // click look like a broken server, and the agent would retry the // transport rather than read the reason. - self.catch_stop_file(); - let permitted = self.authorise(name); - let raw_args = args.to_string(); - let vetoed = self.hook_veto(&permitted, name, &raw_args); - let outcome = self.outcome(&permitted, vetoed, name, &args); - self.tell_post_hooks(name, &raw_args, &outcome); - self.record_call(name, &args, &outcome); - - Ok(match outcome { + // + // The gates are `gated_call` (in `routine_tool`), the one path a + // replayed routine step takes too. + Ok(match self.gated_call(name, &args) { Ok(content) => json!({"content": content, "isError": false}), Err(why) => json!({ "content": [{"type": "text", "text": why}], diff --git a/crates/desktop/src/mcp/definitions.rs b/crates/desktop/src/mcp/definitions.rs index ea208f1..d0ae337 100644 --- a/crates/desktop/src/mcp/definitions.rs +++ b/crates/desktop/src/mcp/definitions.rs @@ -21,7 +21,8 @@ use serde_json::{json, Value}; -use super::{ACTING, UNTRUSTED}; +use super::tools::looks_after; +use super::UNTRUSTED; /// Every desktop tool, in MCP's shape. The toolbox offers these same /// definitions to a model (`toolbox::desktop`), so they are written once. @@ -31,6 +32,7 @@ pub fn tool_definitions() -> Vec { "screenshot", "ui_tree", "wait_for", + "watch", "click_element", "set_value", "element_action", @@ -42,6 +44,7 @@ pub fn tool_definitions() -> Vec { "drag", ]; let mut tools = base_definitions(); + tools.extend(super::routine_definitions::definitions()); for tool in &mut tools { let name = tool["name"].as_str().unwrap_or_default().to_string(); let properties = &mut tool["inputSchema"]["properties"]; @@ -52,7 +55,7 @@ pub fn tool_definitions() -> Vec { focused window the call is REFUSED rather than typing somewhere else." } "screenshot" => "Capture only this window, from list_windows.", - "wait_for" => { + "wait_for" | "watch" => { "The window to watch, from list_windows: for element_* the one the \ selector is looked for in, for screen_changes the one whose area is \ compared. Default: the window the selector names, else the focused one \ @@ -62,7 +65,7 @@ pub fn tool_definitions() -> Vec { }; properties["window"] = json!({"type": "string", "description": what}); } - if ACTING.contains(&name.as_str()) { + if looks_after(&name) { properties["screenshot_after"] = json!({ "type": "boolean", "description": "Also return a screenshot of the window afterwards, to check \ diff --git a/crates/desktop/src/mcp/gate.rs b/crates/desktop/src/mcp/gate.rs index f25ac50..40d8961 100644 --- a/crates/desktop/src/mcp/gate.rs +++ b/crates/desktop/src/mcp/gate.rs @@ -183,8 +183,19 @@ impl Server { // It cannot deny: the action is already done, and reporting a refusal // for it would be a lie to the agent. if let Some(hooks) = self.hooks.as_ref() { - if outcome.is_ok() { - let said = hooks.run_post_tool_use(name, raw_args, "", false); + if let Ok(content) = outcome { + // Every tool's hook is told the call and not its answer, as it + // always was: a screen's text is not the operator's to be + // handed by default. `watch` is the exception, because telling + // the operator WHAT it saw is the whole point of it: its hook + // is the notification, and `tool_response` is the reply + // (found or timeout, the control or window, the `message`). + let answer = if name == "watch" { + super::tools::text_of(content) + } else { + String::new() + }; + let said = hooks.run_post_tool_use(name, raw_args, &answer, false); for message in said.messages() { // stderr, never stdout: stdout is the protocol channel. eprintln!("hook after {name}: {message}"); @@ -193,6 +204,16 @@ impl Server { } } + /// A line in the journal that is not a finished call: a `watch` that has + /// begun, so an operator following the journal sees the wait. + /// + /// Called by `Server::run_tool` before a `watch` starts to poll. + pub(super) fn journal_note(&mut self, t: &str, detail: &str) { + if let Some(journal) = self.journal.as_mut() { + let _ = journal.record(t, &format!("started: {detail}"), true); + } + } + /// Write the call, ran or refused, to the operator's journal. /// /// Called by `Server::call_tool` last, after [`Server::tell_post_hooks`]. @@ -256,13 +277,21 @@ impl Server { "element_action" => format!("{} {}", element_ref(args), s("action")), // The condition and the selector as `Selector::render_redacted` // has it (no text field's name), and how long it may wait. - "wait_for" => format!( + // `watch` is the same, and its `message` (what its hook is told to + // say) stays out of the journal like any other free text. + "wait_for" | "watch" => format!( "{} {} timeout {}", s("condition"), selector_ref(args), args.get("timeout_ms") .map_or("default".to_string(), |ms| format!("{ms} ms")) ), + // The phrase a routine is asked for by, which is the operator's own + // name for it; never its steps, which may carry typed text. + "routine_save" | "routine_run" | "routine_forget" | "routine_match" => { + let phrase: String = s("phrase").chars().take(80).collect(); + format!("\"{phrase}\"") + } "focus_window" => format!("window {}", window_arg(args).unwrap_or_default()), "click" | "move_mouse" => format!("({}, {})", n("x"), n("y")), "drag" => format!( diff --git a/crates/desktop/src/mcp/routine_definitions.rs b/crates/desktop/src/mcp/routine_definitions.rs new file mode 100644 index 0000000..59acbec --- /dev/null +++ b/crates/desktop/src/mcp/routine_definitions.rs @@ -0,0 +1,248 @@ +//! What a client is told about `watch` and the five routine tools. +//! +//! Kept apart from [`super::definitions`] for the reason that file is apart +//! from the transport: these are paragraphs a model reads, and they change for a +//! different reason than the code does. [`super::definitions::tool_definitions`] +//! appends [`definitions`] to the base list and adds `window` and +//! `screenshot_after` the same way it does for every other tool. +//! +//! # Wired to +//! +//! - **Called by** [`super::definitions::tool_definitions`]. +//! - **Calls** nothing; reads [`super::UNTRUSTED`] and [`crate::wait`]'s limits. + +use serde_json::{json, Value}; + +use super::UNTRUSTED; + +/// `watch` and the routine tools, in MCP's shape. +pub(super) fn definitions() -> Vec { + vec![ + watch_definition(), + routine_save_definition(), + routine_run_definition(), + routine_forget_definition(), + routine_list_definition(), + routine_match_definition(), + ] +} + +/// The `phrase` argument shared by four of the tools. +fn phrase_property(what: &str) -> Value { + json!({"type": "string", "description": what}) +} + +fn watch_definition() -> Value { + json!({ + "name": "watch", + "description": format!( + "Watch for something to happen and tell the operator when it does, with no \ + model polling: the same conditions as wait_for (element_appears, \ + element_gone, window_appears, window_gone, screen_changes) looked for every \ + second for up to {max_hours} hours, then the operator's post-tool hook for \ + `watch` is run with the reply as its `tool_response`, which is how it \ + notifies (a desktop notification, a message, a follow-up command: whatever \ + the operator wrote). THIS CALL BLOCKS until the condition holds, the \ + time runs out, or the operator stops the desktop (the stop file ends it \ + within a second), and your turn waits with it: use it only when waiting is \ + the task. For a few seconds use wait_for. `timeout_ms` defaults to {default} \ + ms (one hour) and is held to {max} ms; `poll_ms` {min_poll} to {max_poll}. \ + `message` is what the notification should say; it is returned in the reply. \ + The reply is JSON whose `result` is `found` or `timeout`, as wait_for's. \ + {UNTRUSTED}", + max_hours = crate::wait::MAX_WATCH_MS / 3_600_000, + default = crate::wait::DEFAULT_WATCH_MS, + max = crate::wait::MAX_WATCH_MS, + min_poll = crate::wait::MIN_WATCH_POLL_MS, + max_poll = crate::wait::MAX_WATCH_POLL_MS, + ), + "inputSchema": { + "type": "object", + "properties": { + "condition": { + "type": "string", + "enum": [ + "element_appears", "element_gone", "window_appears", "window_gone", + "screen_changes" + ], + }, + "selector": { + "type": "string", + "description": "key=value pairs joined by ';' — control: role, name \ + (name~ for contains), aid, path, nth; window: app, \ + window (window~ for contains). E.g. \ + role=button;name=Save or app=notepad;window~=Save As. \ + For element_* conditions it must name a control; for \ + window_* only app and/or window.", + }, + "timeout_ms": { + "type": "integer", + "description": format!( + "Longest to watch. Default {}, at most {}.", + crate::wait::DEFAULT_WATCH_MS, + crate::wait::MAX_WATCH_MS + ), + }, + "poll_ms": { + "type": "integer", + "description": format!( + "How often to look. Default {}, from {} to {}.", + crate::wait::DEFAULT_WATCH_POLL_MS, + crate::wait::MIN_WATCH_POLL_MS, + crate::wait::MAX_WATCH_POLL_MS + ), + }, + "baseline": { + "type": "string", + "description": "screen_changes: the `hash` of an earlier reply to compare \ + with. Default: the screen as the watch begins.", + }, + "tolerance": { + "type": "integer", + "description": format!( + "screen_changes: how many of the 128 bits may differ. Default {}.", + crate::wait::DEFAULT_TOLERANCE + ), + }, + "message": { + "type": "string", + "description": "What the notification should say, up to 200 characters. \ + Returned in the reply, so the hook can show it.", + }, + }, + "required": ["condition"], + }, + }) +} + +fn routine_save_definition() -> Value { + json!({ + "name": "routine_save", + "description": format!( + "Keep what was just done at the desktop as a routine that `routine_run` can \ + do again later with no model. Give the `phrase` that asks for it and EITHER \ + `records` (Watchman record ids of the turns that did the task, in any \ + order; needs the local memory engine) OR `steps` (a list of {{tool, args}} \ + you write). The looking calls (ui_tree, screenshot, list_windows, \ + screen_info, cursor_position) are dropped and wait_for is kept. NOTHING IS \ + KEPT, and the reason is given, when: the phrase says \"this\" or \"that\"; \ + any call failed or a wait_for timed out; or any step uses an eN id, a pixel \ + coordinate (click, move_mouse, drag) or a window handle (`window`, \ + focus_window), because those hold for one screen only — name controls with \ + a `selector` and windows inside it (app=...;window=...). Every acting step \ + must say WHERE: a selector needs an app= or window= part, and a typing or \ + key step needs `in_window` (e.g. app=notepad;window~=Untitled; a recording \ + is given the window of the step before it, and the replay refuses to type \ + unless that window has focus); scroll is never kept. A replay invokes links \ + and buttons without the mouse and stops when a set_value or element_action \ + answers \"unverified\" and the next step depends on it. Saving under a \ + phrase that means the same as a saved one (same words, any order, filler \ + words ignored) REPLACES it: that is how a routine that failed is redone. \ + Typed text is stored as it is in the routines file: do not save a routine \ + that types a secret. A routine unused for 30 days is forgotten unless \ + `taught` is true. {UNTRUSTED}" + ), + "inputSchema": { + "type": "object", + "properties": { + "phrase": phrase_property( + "The words that ask for this routine, e.g. \"open the weekly report\". \ + Not \"this\" or \"that\"." + ), + "records": { + "type": "array", + "items": {"type": "string"}, + "description": "Watchman record ids (w-...) of the turns that did the \ + task. Their desktop calls are joined in the order they \ + happened.", + }, + "steps": { + "type": "array", + "items": { + "type": "object", + "properties": { + "tool": {"type": "string"}, + "args": {"type": "object"} + }, + "required": ["tool"], + }, + "description": "The steps themselves, instead of records.", + }, + "taught": { + "type": "boolean", + "description": "The user asked for this routine by name. It then never \ + expires. Default false.", + }, + }, + "required": ["phrase"], + }, + }) +} + +fn routine_run_definition() -> Value { + json!({ + "name": "routine_run", + "description": format!( + "Do a saved routine again, on the real desktop, now: the phrase is matched by \ + its words (any order, filler ignored; see routine_match). Every step runs \ + through the same checks as a call you make yourself (blocked applications, \ + the operator's hooks, the stop file, the journal), in order, and stops at the \ + first one that fails. There is NO repair: on a failure you are told which \ + step and why, and what was already done stays done. Then do the whole task \ + yourself, from the screen as it is, and routine_save it under the same \ + phrase to replace the routine (or routine_forget it). The routine moves the \ + mouse and keyboard of the machine the operator is using. {UNTRUSTED}" + ), + "inputSchema": { + "type": "object", + "properties": { + "phrase": phrase_property("The words that ask for the routine."), + }, + "required": ["phrase"], + }, + }) +} + +fn routine_forget_definition() -> Value { + json!({ + "name": "routine_forget", + "description": "Delete the saved routine a phrase asks for. Use it for a routine \ + that no longer does the right thing.", + "inputSchema": { + "type": "object", + "properties": { + "phrase": phrase_property("The words that ask for the routine."), + }, + "required": ["phrase"], + }, + }) +} + +fn routine_list_definition() -> Value { + json!({ + "name": "routine_list", + "description": "Every saved routine that has not expired: its phrase, how many \ + steps, when it was learned and last used, whether it was taught \ + (never expires), and how many days it has left.", + "inputSchema": {"type": "object", "properties": {}}, + }) +} + +fn routine_match_definition() -> Value { + json!({ + "name": "routine_match", + "description": "Which saved routine a phrase would run, without running it: the \ + routine and its steps (typed text shown by length only), or that \ + none matches. A phrase matches a routine when it has the same \ + words in any order, ignoring filler words (a, the, to, please, \ + ...); a near miss does not match. Call this before routine_run \ + to see what it would do.", + "inputSchema": { + "type": "object", + "properties": { + "phrase": phrase_property("The words to match."), + }, + "required": ["phrase"], + }, + }) +} diff --git a/crates/desktop/src/mcp/routine_tests.rs b/crates/desktop/src/mcp/routine_tests.rs new file mode 100644 index 0000000..e2d0dd2 --- /dev/null +++ b/crates/desktop/src/mcp/routine_tests.rs @@ -0,0 +1,771 @@ +//! The routine tools and `watch` on a live server: the gates a replayed step +//! passes, stop-at-first-failure, the tiers, and what the journal and the +//! operator's hook are told. +//! +//! The "desktop" is the real one of the machine running the tests, as in +//! `selector_tests::live`, and every step is one that touches nothing: a wait +//! for a window that does not exist to be gone (found at once) or to appear (a +//! timeout of 100 ms). + +use std::path::{Path, PathBuf}; +use std::sync::{Arc, Mutex}; + +use serde_json::{json, Value}; + +use super::*; +use crate::hooks::HookOutcome; +use crate::permission::OBSERVING; +use crate::routine::source::{Call, MemorySource, NoSource, Record, StepSource}; +use crate::routine::store::{Routines, Store}; +use crate::routine::{Routine, Step}; + +/// A window no program has: waiting for it to be gone is true at once. +const NO_SUCH_WINDOW: &str = "window=zz no such window 7f3a9c1e"; + +fn folder(name: &str) -> PathBuf { + let dir = + std::env::temp_dir().join(format!("incu-routine-tools-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("temp dir"); + dir +} + +fn server_with(dir: &Path, source: Box) -> Server { + Server::new(InputThread::start().expect("input thread")).with_routines(RoutineKit::new( + Store::at(dir.join("routines.json")), + source, + )) +} + +fn server(dir: &Path) -> Server { + server_with(dir, Box::new(NoSource)) +} + +fn call(server: &mut Server, name: &str, args: Value) -> Value { + let line = json!({ + "jsonrpc": "2.0", "id": 1, "method": "tools/call", + "params": {"name": name, "arguments": args} + }) + .to_string(); + let reply = server.handle_line(&line).expect("a reply was expected"); + serde_json::from_str(&reply).expect("the reply must be valid JSON") +} + +fn said(reply: &Value) -> String { + reply["result"]["content"][0]["text"] + .as_str() + .unwrap_or_default() + .to_string() +} + +fn refused(reply: &Value) -> bool { + reply["result"]["isError"] == true +} + +/// The JSON a tool answered. +fn answer(reply: &Value) -> Value { + assert!(!refused(reply), "{reply}"); + serde_json::from_str(&said(reply)).expect("the answer is JSON") +} + +fn gone() -> Value { + json!({"condition": "window_gone", "selector": NO_SUCH_WINDOW}) +} + +fn never_appears() -> Value { + json!({"condition": "window_appears", "selector": NO_SUCH_WINDOW, "timeout_ms": 100, "poll_ms": 50}) +} + +fn step(args: Value) -> Value { + json!({"tool": "wait_for", "args": args}) +} + +fn save_steps(s: &mut Server, phrase: &str, steps: Vec) -> Value { + call(s, "routine_save", json!({"phrase": phrase, "steps": steps})) +} + +/// The tools a pre-tool hook was asked about. +type Pre = Arc>>; +/// The tools a post-tool hook was told about, and the answer it was handed. +type Post = Arc>>; + +/// What a post-tool and a pre-tool hook were told, and a veto on request. +struct Recording { + veto: Option<&'static str>, + pre: Pre, + post: Post, + /// Written to after the first `wait_for`, to stop the desktop mid-routine. + stop_after_wait: Option, +} + +impl Recording { + fn new(veto: Option<&'static str>) -> (Self, Pre, Post) { + let pre = Arc::new(Mutex::new(Vec::new())); + let post = Arc::new(Mutex::new(Vec::new())); + ( + Self { + veto, + pre: pre.clone(), + post: post.clone(), + stop_after_wait: None, + }, + pre, + post, + ) + } +} + +impl Hooks for Recording { + fn run_pre_tool_use(&self, tool_name: &str, _input: &str) -> HookOutcome { + self.pre.lock().unwrap().push(tool_name.to_string()); + if self.veto == Some(tool_name) { + HookOutcome::deny(vec![format!("the operator does not allow {tool_name}")]) + } else { + HookOutcome::allow(Vec::new()) + } + } + + fn run_post_tool_use( + &self, + tool_name: &str, + _input: &str, + output: &str, + _err: bool, + ) -> HookOutcome { + self.post + .lock() + .unwrap() + .push((tool_name.to_string(), output.to_string())); + if let (Some(stop), "wait_for") = (&self.stop_after_wait, tool_name) { + crate::safety::request_stop(stop).expect("stop file"); + } + HookOutcome::allow(Vec::new()) + } +} + +#[test] +fn the_six_new_tools_are_offered_in_their_tiers() { + let listed: Vec = tool_definitions(); + let names: Vec<&str> = listed.iter().filter_map(|t| t["name"].as_str()).collect(); + for tool in [ + "watch", + "routine_save", + "routine_run", + "routine_forget", + "routine_list", + "routine_match", + ] { + assert!(names.contains(&tool), "{tool} is not offered: {names:?}"); + } + for tool in ["watch", "routine_list", "routine_match"] { + assert!(OBSERVING.contains(&tool), "{tool}"); + } + for tool in ["routine_save", "routine_run", "routine_forget"] { + assert!(ACTING.contains(&tool), "{tool}"); + } + // The picture after an action is offered where it means something. + let has_shot = |name: &str| { + listed.iter().find(|t| t["name"] == name).is_some_and(|t| { + t["inputSchema"]["properties"] + .get("screenshot_after") + .is_some() + }) + }; + assert!(has_shot("routine_run")); + assert!(!has_shot("routine_save")); + assert!(!has_shot("routine_forget")); + assert!(!has_shot("watch")); + // `watch` takes `window` as wait_for does. + let watch = listed.iter().find(|t| t["name"] == "watch").unwrap(); + assert!(watch["inputSchema"]["properties"].get("window").is_some()); + assert!(watch["description"].as_str().unwrap().contains("BLOCKS")); +} + +#[test] +fn a_routine_is_saved_from_steps_listed_matched_and_forgotten() { + let dir = folder("lifecycle"); + let mut s = server(&dir); + let saved = answer(&save_steps(&mut s, "check the window", vec![step(gone())])); + assert_eq!(saved["saved"], "check the window"); + assert_eq!(saved["replaced_an_older_routine"], false); + + let listed = answer(&call(&mut s, "routine_list", json!({}))); + assert_eq!(listed["count"], 1); + assert_eq!(listed["routines"][0]["phrase"], "check the window"); + assert_eq!(listed["routines"][0]["expires_in_days"], 30); + + // The same words in another order, with filler, are the same request. + let matched = answer(&call( + &mut s, + "routine_match", + json!({"phrase": "Window, check it please"}), + )); + assert_eq!(matched["matched"], true); + assert_eq!(matched["routine"]["steps"].as_array().unwrap().len(), 1); + let near = answer(&call( + &mut s, + "routine_match", + json!({"phrase": "check the window and the door"}), + )); + assert_eq!(near["matched"], false); + + let forgotten = call(&mut s, "routine_forget", json!({"phrase": "window check"})); + assert!(!refused(&forgotten), "{forgotten}"); + assert_eq!(answer(&call(&mut s, "routine_list", json!({})))["count"], 0); + let again = call(&mut s, "routine_forget", json!({"phrase": "window check"})); + assert!( + refused(&again) && said(&again).contains("no routine matches"), + "{again}" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_run_that_cannot_be_replayed_is_refused_with_the_reason_and_nothing_is_written() { + let dir = folder("refused"); + let mut s = server(&dir); + let cases = [ + ("close this", vec![step(gone())], "this"), + ( + "click the button", + vec![json!({"tool": "click_element", "args": {"id": 16}})], + "eN id", + ), + ( + "click the spot", + vec![json!({"tool": "click", "args": {"x": 4, "y": 5}})], + "coordinates", + ), + ( + "do nothing", + vec![json!({"tool": "ui_tree", "args": {}})], + "no desktop action", + ), + ]; + for (phrase, steps, why) in cases { + let reply = save_steps(&mut s, phrase, steps); + assert!(refused(&reply), "{phrase}: {reply}"); + assert!(said(&reply).contains(why), "{phrase}: {}", said(&reply)); + } + assert!( + !dir.join("routines.json").exists(), + "a refusal wrote a file" + ); + let neither = call(&mut s, "routine_save", json!({"phrase": "open it"})); + assert!(said(&neither).contains("records"), "{}", said(&neither)); + let both = call( + &mut s, + "routine_save", + json!({"phrase": "open it", "steps": [], "records": []}), + ); + assert!(said(&both).contains("not both"), "{}", said(&both)); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_routine_is_learned_from_watchman_records_in_the_order_they_happened() { + let dir = folder("records"); + let source = MemorySource::new() + // Named second, happened first. + .with(Record { + id: "w-b".to_string(), + started_at_ms: 2_000, + calls: vec![Call::ok("mcp__desktop__wait_for", never_gone_second())], + }) + .with(Record { + id: "w-a".to_string(), + started_at_ms: 1_000, + calls: vec![ + Call::ok("mcp__desktop__ui_tree", json!({})), + Call::ok("mcp__desktop__wait_for", gone()), + Call::ok("Bash", json!({"command": "ls"})), + ], + }); + let mut s = server_with(&dir, Box::new(source)); + let reply = call( + &mut s, + "routine_save", + json!({"phrase": "wait for it", "records": ["w-b", "w-a"]}), + ); + let saved = answer(&reply); + assert_eq!(saved["looking_steps_dropped"], 1); + assert_eq!(saved["left_out_not_desktop_tools"], json!(["Bash"])); + assert!(saved["warning"].is_string()); + let steps = saved["steps"].as_array().unwrap(); + assert_eq!(steps.len(), 2); + assert!( + steps[0].as_str().unwrap().contains("window_gone"), + "{steps:?}" + ); + assert!(steps[1].as_str().unwrap().contains("11111111"), "{steps:?}"); + + let unknown = call( + &mut s, + "routine_save", + json!({"phrase": "wait for it", "records": ["w-zzz"]}), + ); + assert!( + refused(&unknown) && said(&unknown).contains("w-zzz"), + "{unknown}" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +fn never_gone_second() -> Value { + json!({"condition": "window_gone", "selector": "window=zz 11111111"}) +} + +#[test] +fn a_failed_recorded_run_is_not_learned() { + let dir = folder("failed-record"); + let source = MemorySource::new().with(Record { + id: "w-1".to_string(), + started_at_ms: 1, + calls: vec![Call::ok("wait_for", gone()).failed("selector matches nothing")], + }); + let mut s = server_with(&dir, Box::new(source)); + let reply = call( + &mut s, + "routine_save", + json!({"phrase": "wait for it", "records": ["w-1"]}), + ); + assert!( + refused(&reply) && said(&reply).contains("the run failed"), + "{reply}" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_saved_routine_runs_every_step_through_the_gates_and_the_journal() { + let dir = folder("run"); + let journal = dir.join("desktop.jsonl"); + let (hooks, pre, post) = Recording::new(None); + let mut s = server(&dir) + .with_journal(journal.clone()) + .with_hook_runner(Box::new(hooks)); + answer(&save_steps( + &mut s, + "check the windows", + vec![step(gone()), step(gone())], + )); + + let ran = answer(&call( + &mut s, + "routine_run", + json!({"phrase": "windows check"}), + )); + assert_eq!(ran["steps_run"], 2, "{ran}"); + assert_eq!(ran["ran"], "check the windows"); + + // The pre-tool hook saw the routine and then EACH step, as a live call. + assert_eq!( + *pre.lock().unwrap(), + ["routine_save", "routine_run", "wait_for", "wait_for"] + ); + assert_eq!( + post.lock() + .unwrap() + .iter() + .filter(|(t, _)| t == "wait_for") + .count(), + 2 + ); + // And the journal has a line for each step and one for the routine. + let lines = std::fs::read_to_string(&journal).unwrap(); + assert_eq!(lines.matches("\"t\":\"wait_for\"").count(), 2, "{lines}"); + assert!(lines.contains("\"t\":\"routine_run\""), "{lines}"); + assert!(lines.contains("check the windows"), "{lines}"); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn the_first_failing_step_stops_the_replay_and_the_rest_never_run() { + let dir = folder("stop-first"); + let journal = dir.join("desktop.jsonl"); + let (hooks, pre, _post) = Recording::new(None); + let mut s = server(&dir) + .with_journal(journal.clone()) + .with_hook_runner(Box::new(hooks)); + answer(&save_steps( + &mut s, + "three waits", + vec![step(gone()), step(never_appears()), step(gone())], + )); + let reply = call(&mut s, "routine_run", json!({"phrase": "three waits"})); + assert!(refused(&reply), "{reply}"); + let said = said(&reply); + assert!(said.contains("failed at step 2 of 3"), "{said}"); + assert!(said.contains("timed out"), "{said}"); + assert!(said.contains("nothing was retried or repaired"), "{said}"); + // Step 3 was never asked of the gate, let alone the desktop. + let waits = pre + .lock() + .unwrap() + .iter() + .filter(|t| *t == "wait_for") + .count(); + assert_eq!(waits, 2); + let lines = std::fs::read_to_string(&journal).unwrap(); + assert_eq!(lines.matches("\"t\":\"wait_for\"").count(), 2, "{lines}"); + assert!( + lines.contains("refused") && lines.contains("routine_run"), + "{lines}" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn the_operators_hook_can_refuse_a_step_of_a_replay() { + let dir = folder("veto"); + let (hooks, _pre, _post) = Recording::new(Some("wait_for")); + let mut s = server(&dir).with_hook_runner(Box::new(hooks)); + answer(&save_steps(&mut s, "check the window", vec![step(gone())])); + let reply = call(&mut s, "routine_run", json!({"phrase": "check the window"})); + assert!(refused(&reply), "a replay got past the hook: {reply}"); + assert!( + said(&reply).contains("failed at step 1 of 1"), + "{}", + said(&reply) + ); + assert!( + said(&reply).contains("does not allow wait_for"), + "{}", + said(&reply) + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn the_stop_file_ends_a_replay_between_steps() { + let dir = folder("stop-file"); + let journal = dir.join("desktop.jsonl"); + let (mut hooks, _pre, _post) = Recording::new(None); + hooks.stop_after_wait = Some(crate::safety::stop_file_beside(&journal)); + let mut s = server(&dir) + .with_journal(journal) + .with_hook_runner(Box::new(hooks)); + answer(&save_steps( + &mut s, + "two waits", + vec![step(gone()), step(gone())], + )); + let reply = call(&mut s, "routine_run", json!({"phrase": "two waits"})); + assert!(refused(&reply), "{reply}"); + let said = said(&reply); + assert!(said.contains("failed at step 2 of 2"), "{said}"); + assert!(said.to_lowercase().contains("stop"), "{said}"); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_read_only_session_may_look_at_routines_and_may_not_change_or_run_them() { + let dir = folder("tier"); + let mut s = server(&dir).with_tier(Tier::Observe); + assert!(!refused(&call(&mut s, "routine_list", json!({})))); + assert!(!refused(&call( + &mut s, + "routine_match", + json!({"phrase": "open it"}) + ))); + for tool in ["routine_save", "routine_run", "routine_forget"] { + let reply = call( + &mut s, + tool, + json!({"phrase": "open it", "steps": [step(gone())]}), + ); + assert!(refused(&reply), "{tool}: {reply}"); + assert!( + said(&reply).contains("workspace-write"), + "{tool}: {}", + said(&reply) + ); + } + assert!(!dir.join("routines.json").exists()); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_routine_unused_for_thirty_days_is_gone_unless_it_was_taught() { + let dir = folder("expiry"); + let old = |phrase: &str, taught: bool| Routine { + phrase: phrase.to_string(), + steps: vec![Step::new("wait_for", gone())], + date: "2026-01-01".to_string(), + last_used_ms: 1_000, + taught, + }; + let file = Routines { + version: 1, + routines: vec![old("forgotten task", false), old("taught task", true)], + }; + let store = Store::at(dir.join("routines.json")); + store.save(&file).unwrap(); + let mut s = server(&dir); + let listed = answer(&call(&mut s, "routine_list", json!({}))); + assert_eq!(listed["count"], 1, "{listed}"); + assert_eq!(listed["routines"][0]["phrase"], "taught task"); + assert!(listed["routines"][0].get("expires_in_days").is_none()); + let run = call(&mut s, "routine_run", json!({"phrase": "forgotten task"})); + assert!( + refused(&run) && said(&run).contains("no routine matches"), + "{run}" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_stored_step_the_rules_refuse_runs_nothing() { + let dir = folder("tampered"); + let (hooks, pre, _post) = Recording::new(None); + let file = Routines { + version: 1, + routines: vec![Routine { + phrase: "edited by hand".to_string(), + steps: vec![ + Step::new("wait_for", gone()), + Step::new("click", json!({"x": 1, "y": 1})), + ], + date: String::new(), + last_used_ms: crate::routine::now_ms(), + taught: true, + }], + }; + Store::at(dir.join("routines.json")).save(&file).unwrap(); + let mut s = server(&dir).with_hook_runner(Box::new(hooks)); + let reply = call(&mut s, "routine_run", json!({"phrase": "edited by hand"})); + assert!( + refused(&reply) && said(&reply).contains("Nothing was done"), + "{reply}" + ); + assert_eq!( + *pre.lock().unwrap(), + ["routine_run"], + "not even the first step was tried" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_server_without_routines_refuses_them_by_name() { + let mut s = Server::new(InputThread::start().expect("input thread")); + let reply = call(&mut s, "routine_list", json!({})); + assert!( + refused(&reply) && said(&reply).contains("not configured"), + "{reply}" + ); +} + +#[test] +fn a_click_on_a_link_or_button_is_replayed_as_an_invoke_and_other_clicks_are_not() { + use super::routine_tool::invoke_for; + let click = |args: Value| Step::new("click_element", args); + let selector = "app=chrome;window~=Wikipedia;role=link;name=Honshu"; + let invoked = invoke_for(&click(json!({"selector": selector}))).expect("a link is invoked"); + assert_eq!(invoked, json!({"selector": selector, "action": "invoke"})); + assert!(invoke_for(&click( + json!({"selector": "app=a;role=button;name=OK", "button": "left"}) + )) + .is_some()); + // A right click, a text field and anything else are clicked as recorded. + assert!(invoke_for(&click(json!({"selector": selector, "button": "right"}))).is_none()); + assert!(invoke_for(&click(json!({"selector": "app=a;role=edit;name=Body"}))).is_none()); + assert!( + invoke_for(&click(json!({"selector": "app=a;name=Save"}))).is_none(), + "no role" + ); + assert!(invoke_for(&Step::new("set_value", json!({"selector": selector}))).is_none()); +} + +#[test] +fn a_keyboard_steps_window_is_looked_up_at_replay_and_a_missing_one_is_a_failed_step() { + use super::routine_tool::live_args; + // Not a keyboard step: its arguments go through as they are. + let wait = Step::new("wait_for", gone()); + assert_eq!(live_args(&wait).unwrap(), gone()); + // A keyboard step whose window does not exist fails, naming it, before any + // key is sent; the tool is never asked. + let typing = Step::new( + "type_text", + json!({"text": "hi", "in_window": NO_SUCH_WINDOW}), + ); + let why = live_args(&typing).unwrap_err(); + assert!(why.contains("in_window"), "{why}"); + assert!(why.contains("zz no such window"), "{why}"); +} + +#[test] +fn a_replayed_typing_step_into_a_window_that_is_not_there_fails_and_types_nothing() { + let dir = folder("typing"); + let (hooks, pre, _post) = Recording::new(None); + let mut s = server(&dir).with_hook_runner(Box::new(hooks)); + let reply = save_steps( + &mut s, + "type a note", + vec![json!({"tool": "type_text", "args": {"text": "hi", "in_window": NO_SUCH_WINDOW}})], + ); + assert!(!refused(&reply), "{reply}"); + let run = call(&mut s, "routine_run", json!({"phrase": "type a note"})); + assert!(refused(&run), "{run}"); + assert!( + said(&run).contains("failed at step 1 of 1"), + "{}", + said(&run) + ); + assert_eq!( + *pre.lock().unwrap(), + ["routine_save", "routine_run"], + "type_text never reached the gate" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn typed_text_and_steps_never_reach_the_journal_or_a_listing() { + let dir = folder("redaction"); + let journal = dir.join("desktop.jsonl"); + let mut s = server(&dir).with_journal(journal.clone()); + let reply = call( + &mut s, + "routine_save", + json!({"phrase": "type my token", "steps": [ + {"tool": "set_value", "args": {"selector": "app=vault;role=edit;name=Token", "value": "hunter2-secret"}} + ]}), + ); + assert!(!refused(&reply), "{reply}"); + assert!(!said(&reply).contains("hunter2"), "{}", said(&reply)); + let matched = said(&call( + &mut s, + "routine_match", + json!({"phrase": "token type"}), + )); + assert!( + !matched.contains("hunter2") && matched.contains("14 characters"), + "{matched}" + ); + // A text field's name is what was typed in it: not in a step's line either. + assert!(!matched.contains("name=Token"), "{matched}"); + let lines = std::fs::read_to_string(&journal).unwrap(); + assert!(!lines.contains("hunter2"), "{lines}"); + // It is in the routines file, as the docs say. + let file = std::fs::read_to_string(dir.join("routines.json")).unwrap(); + assert!(file.contains("hunter2-secret")); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn watch_returns_at_once_for_a_condition_that_holds_and_tells_the_hook_what_it_saw() { + let dir = folder("watch"); + let journal = dir.join("desktop.jsonl"); + let (hooks, _pre, post) = Recording::new(None); + let mut s = server(&dir) + .with_journal(journal.clone()) + .with_hook_runner(Box::new(hooks)); + let reply = call( + &mut s, + "watch", + json!({"condition": "window_gone", "selector": NO_SUCH_WINDOW, "message": "the export finished"}), + ); + let text = said(&reply); + assert!(!refused(&reply), "{text}"); + let json_part = &text[..=text.rfind('}').unwrap()]; + let body: Value = serde_json::from_str(json_part).unwrap(); + assert_eq!(body["result"], "found"); + assert_eq!(body["message"], "the export finished"); + // The hook for `watch` was handed the reply; no other tool's hook gets one. + let told = post.lock().unwrap().clone(); + let (name, output) = told.iter().find(|(n, _)| n == "watch").expect("watch hook"); + assert_eq!(name, "watch"); + assert!( + output.contains("the export finished") && output.contains("found"), + "{output}" + ); + let lines = std::fs::read_to_string(&journal).unwrap(); + assert!( + lines.contains("started:"), + "the begin is journalled: {lines}" + ); + assert!( + !lines.contains("the export finished"), + "the message is free text: {lines}" + ); + + call(&mut s, "wait_for", gone()); + let told = post.lock().unwrap().clone(); + assert!(told + .iter() + .filter(|(n, _)| n == "wait_for") + .all(|(_, out)| out.is_empty())); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_watch_that_times_out_is_a_reply_and_still_notifies() { + let dir = folder("watch-timeout"); + let (hooks, _pre, post) = Recording::new(None); + let mut s = server(&dir).with_hook_runner(Box::new(hooks)); + let reply = call( + &mut s, + "watch", + json!({"condition": "window_appears", "selector": NO_SUCH_WINDOW, "timeout_ms": 300, "poll_ms": 250}), + ); + assert!(!refused(&reply), "{reply}"); + assert!( + said(&reply).contains("\"result\": \"timeout\""), + "{}", + said(&reply) + ); + let told = post.lock().unwrap().clone(); + assert!( + told.iter() + .any(|(n, out)| n == "watch" && out.contains("timeout")), + "{told:?}" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn the_stop_file_ends_a_watch_that_has_a_day_to_run() { + let dir = folder("watch-stop"); + let journal = dir.join("desktop.jsonl"); + let stop = crate::safety::stop_file_beside(&journal); + let mut s = server(&dir).with_journal(journal); + // The stop arrives while the watch is polling, not before it starts. + let stopper = std::thread::spawn(move || { + std::thread::sleep(std::time::Duration::from_millis(400)); + crate::safety::request_stop(&stop).expect("stop file"); + }); + let started = std::time::Instant::now(); + let reply = call( + &mut s, + "watch", + json!({"condition": "window_appears", "selector": NO_SUCH_WINDOW, "timeout_ms": 86_400_000}), + ); + stopper.join().expect("the stopper thread"); + assert!(refused(&reply), "{reply}"); + assert!( + started.elapsed() < std::time::Duration::from_secs(10), + "a stop took {:?}", + started.elapsed() + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_watch_message_that_is_too_long_or_not_text_is_refused() { + let dir = folder("watch-message"); + let mut s = server(&dir); + let long = "x".repeat(201); + for message in [json!(long), json!(5)] { + let reply = call( + &mut s, + "watch", + json!({"condition": "window_gone", "selector": NO_SUCH_WINDOW, "message": message}), + ); + assert!( + refused(&reply) && said(&reply).contains("message"), + "{reply}" + ); + } + let _ = std::fs::remove_dir_all(&dir); +} diff --git a/crates/desktop/src/mcp/routine_tool.rs b/crates/desktop/src/mcp/routine_tool.rs new file mode 100644 index 0000000..e921bde --- /dev/null +++ b/crates/desktop/src/mcp/routine_tool.rs @@ -0,0 +1,394 @@ +//! The five routine tools: `routine_save`, `routine_run`, `routine_forget` +//! (acting tier) and `routine_list`, `routine_match` (observing tier). +//! +//! The rules are in [`crate::routine`]; this file is where they meet the live +//! server. Two things are decided here and nowhere else: +//! +//! - **A replay runs through the server's own gates.** [`Live`] sends every +//! step to [`Server::gated_call`], the function `tools/call` itself uses, so +//! each step gets the stop file, the permission tier, the operator's pre-tool +//! hook (which sees the real arguments), the blocked-application guards, the +//! action-rate cap, the post-tool hook and a journal line of its own. The +//! routine as a whole is one more journalled call. A replay cannot do what a +//! live call would be refused. +//! - **The engine is optional.** A server with no step source still lists, +//! matches, replays and forgets routines, and saves one from an explicit list +//! of steps; only saving from Watchman record ids needs the engine. +//! +//! # Wired to +//! +//! - **Called by** `Server::run_tool` in [`super::tools`]. +//! - **Calls** [`crate::routine`] for the rules and the file, and +//! `Server::gated_call` for every replayed step. + +use serde_json::{json, Value}; + +use crate::routine::learn::{check_phrase, learn_from_calls, learn_from_steps, Cleaned, IN_WINDOW}; +use crate::routine::replay::{replay, Runner}; +use crate::routine::source::{Call, StepSource}; +use crate::routine::store::{Routines, Store}; +use crate::routine::words::word_set; +use crate::routine::{now_ms, Routine, Step}; + +use crate::selector::Selector; + +use super::tools::{text, text_of, window_id_for}; +use super::Server; + +/// The most record ids one `routine_save` reads. +const MAX_RECORDS: usize = 20; + +/// The longest phrase a routine may be named by, in characters. +const MAX_PHRASE_CHARS: usize = 200; + +/// The routines file and where records come from, held by a [`Server`]. +pub struct RoutineKit { + store: Store, + source: Box, +} + +impl RoutineKit { + /// A kit over `store`, reading records from `source`. + #[must_use] + pub fn new(store: Store, source: Box) -> Self { + Self { store, source } + } + + /// Where the routines file is. + #[must_use] + pub fn path(&self) -> &std::path::Path { + self.store.path() + } +} + +impl Server { + /// Offer the routine tools over `kit`. Without this they refuse by name: + /// a server built for a protocol test has no routines file to write. + #[must_use] + pub fn with_routines(mut self, kit: RoutineKit) -> Self { + self.routines = Some(kit); + self + } + + /// The kit, or the refusal saying there is none. + fn kit(&self) -> Result<&RoutineKit, String> { + self.routines + .as_ref() + .ok_or_else(|| "routines are not configured on this server".to_string()) + } + + /// One of the five routine tools. Called by `Server::run_tool`. + pub(super) fn routine_tool(&mut self, name: &str, args: &Value) -> Result, String> { + match name { + "routine_save" => self.routine_save(args), + "routine_run" => self.routine_run(args), + "routine_forget" => self.routine_forget(args), + "routine_list" => self.routine_list(), + "routine_match" => self.routine_match(args), + other => Err(format!("no tool named '{other}'")), + } + } + + /// `routine_save`: learn a routine from Watchman records or from steps. + fn routine_save(&mut self, args: &Value) -> Result, String> { + let phrase = phrase_arg(args)?; + check_phrase(&phrase).map_err(|r| r.to_string())?; + let taught = args.get("taught").and_then(Value::as_bool).unwrap_or(false); + let kit = self.kit()?; + let cleaned = match (args.get("records"), args.get("steps")) { + (Some(_), Some(_)) => { + return Err("give `records` or `steps`, not both".to_string()); + } + (None, None) => { + return Err( + "give `records` (Watchman record ids) or `steps` to learn from".to_string(), + ); + } + (Some(ids), None) => learn_from_calls(&phrase, &calls_of(kit.source.as_ref(), ids)?), + (None, Some(steps)) => learn_from_steps(&phrase, &steps_of(steps)?), + } + .map_err(|r| r.to_string())?; + let now = now_ms(); + let mut file = kit.store.load()?; + let saved = file.learn(&phrase, cleaned.steps.clone(), taught, now)?; + kit.store.save(&file)?; + Ok(vec![text(saved_reply( + &phrase, + &cleaned, + &saved, + kit.store.path(), + ))]) + } + + /// `routine_run`: replay the routine a phrase asks for. + fn routine_run(&mut self, args: &Value) -> Result, String> { + let phrase = phrase_arg(args)?; + let routine = self.find_routine(&phrase)?; + let outcome = replay(&routine, &mut Live { server: self }); + let report = outcome.map_err(|failure| failure.to_string())?; + // Used, so its thirty days begin again. Not a reason to fail a routine + // that has just run to the end. + let remembered = self.remember_use(&routine.phrase); + let steps: Vec = report + .ran + .iter() + .enumerate() + .map(|(i, r)| json!(format!("{}. {} -> {}", i + 1, r.summary, r.answer))) + .collect(); + let mut reply = json!({ + "ran": routine.phrase, + "steps_run": report.ran.len(), + "steps": steps, + }); + if let Err(why) = remembered { + reply["note"] = json!(format!( + "it ran, but its last-used time was not saved: {why}" + )); + } + Ok(vec![text(pretty(&reply))]) + } + + /// `routine_forget`: delete the routine a phrase asks for. + fn routine_forget(&mut self, args: &Value) -> Result, String> { + let phrase = phrase_arg(args)?; + let kit = self.kit()?; + let mut file = kit.store.load()?; + if !file.forget(&phrase, now_ms()) { + return Err(format!( + "no routine matches \"{phrase}\"; routine_list shows what is saved" + )); + } + kit.store.save(&file)?; + Ok(vec![text(format!("forgot the routine for \"{phrase}\""))]) + } + + /// `routine_list`: every routine that has not expired. + fn routine_list(&self) -> Result, String> { + let kit = self.kit()?; + let file = kit.store.load()?; + let now = now_ms(); + let routines: Vec = file.live(now).map(|r| describe(r, now)).collect(); + Ok(vec![text(pretty(&json!({ + "count": routines.len(), + "routines": routines, + "file": kit.store.path().display().to_string(), + })))]) + } + + /// `routine_match`: which routine a phrase would run, without running it. + fn routine_match(&self, args: &Value) -> Result, String> { + let phrase = phrase_arg(args)?; + let kit = self.kit()?; + let file = kit.store.load()?; + let now = now_ms(); + let words: Vec = word_set(&phrase).into_iter().collect(); + let reply = match file.matching(&phrase, now) { + Some(routine) => { + let mut shown = describe(routine, now); + shown["steps"] = json!(routine.steps.iter().map(Step::summary).collect::>()); + json!({"matched": true, "words": words, "routine": shown}) + } + None => json!({"matched": false, "words": words}), + }; + Ok(vec![text(pretty(&reply))]) + } + + /// The live routine `phrase` asks for. + fn find_routine(&self, phrase: &str) -> Result { + let file = self.kit()?.store.load()?; + file.matching(phrase, now_ms()).cloned().ok_or_else(|| { + format!("no routine matches \"{phrase}\"; routine_list shows what is saved") + }) + } + + /// Record that the routine `phrase` ran to the end, reading the file again + /// first so a routine saved meanwhile is not overwritten. + fn remember_use(&self, phrase: &str) -> Result<(), String> { + let kit = self.kit()?; + let mut file: Routines = kit.store.load()?; + file.used(phrase, now_ms()); + kit.store.save(&file) + } + + /// One tool call through every gate, as `tools/call` runs it. The content it + /// answered, or the words it was refused with. + /// + /// Called by `Server::call_tool` for the call itself, and by [`Live`] for + /// each step of a replay: one path, so a step can never be let through by a + /// door a live call does not have. + pub(super) fn gated_call(&mut self, name: &str, args: &Value) -> Result, String> { + self.catch_stop_file(); + let permitted = self.authorise(name); + let raw_args = args.to_string(); + let vetoed = self.hook_veto(&permitted, name, &raw_args); + let outcome = self.outcome(&permitted, vetoed, name, args); + self.tell_post_hooks(name, &raw_args, &outcome); + self.record_call(name, args, &outcome); + outcome + } +} + +/// The live desktop as a [`Runner`]: every step through the server's gates. +struct Live<'a> { + server: &'a mut Server, +} + +impl Runner for Live<'_> { + fn run(&mut self, _n: usize, step: &Step) -> Result { + let args = live_args(step)?; + // A link or a button is invoked through its own pattern when it has one: + // that needs no mouse, works for a control scrolled out of view and takes + // no focus. A control that will not be invoked (or a refusal of the + // invoke) leaves the recorded click to run, each through the same gates; + // the journal shows both. + if let Some(invoke) = invoke_for(step) { + if let Ok(content) = self.server.gated_call("element_action", &invoke) { + return Ok(text_of(&content)); + } + } + self.server + .gated_call(&step.tool, &args) + .map(|content| text_of(&content)) + } + + fn settle(&mut self) { + std::thread::sleep(crate::limits::get().settle); + } +} + +/// The arguments the tool is called with for `step`. A keyboard step's +/// `in_window` (a window-only selector) becomes the `window` of the window it +/// names NOW; the tool then refuses unless that window has the focus, so keys +/// never go to whatever happens to be in front. +pub(super) fn live_args(step: &Step) -> Result { + let mut args = step.args.clone(); + let Some(map) = args.as_object_mut() else { + return Ok(args); + }; + let Some(named) = map.remove(IN_WINDOW) else { + return Ok(args); + }; + let raw = named.as_str().ok_or("in_window is not text")?; + let selector = Selector::parse(raw).map_err(|e| format!("in_window: {e}"))?; + let window = window_id_for(&selector).map_err(|e| format!("in_window {raw}: {e}"))?; + map.insert("window".to_string(), Value::String(window)); + Ok(args) +} + +/// The `element_action invoke` that does what the left click of `step` would, +/// when the step is a click on a link or a button found by selector. +pub(super) fn invoke_for(step: &Step) -> Option { + if step.tool != "click_element" { + return None; + } + let left = step + .args + .get("button") + .and_then(Value::as_str) + .is_none_or(|b| b == "left"); + let raw = step.args.get("selector")?.as_str()?; + let role = Selector::parse(raw).ok()?.role?; + (left && matches!(role.word(), "button" | "link")) + .then(|| json!({"selector": raw, "action": "invoke"})) +} + +/// The phrase argument: required, text, not empty, not absurdly long. +fn phrase_arg(args: &Value) -> Result { + let phrase = args + .get("phrase") + .and_then(Value::as_str) + .map(str::trim) + .filter(|p| !p.is_empty()) + .ok_or("phrase is required: the words that ask for this routine")?; + if phrase.chars().count() > MAX_PHRASE_CHARS { + return Err(format!("phrase is over {MAX_PHRASE_CHARS} characters")); + } + Ok(phrase.to_string()) +} + +/// The recorded calls of the records `ids` names, oldest record first. +fn calls_of(source: &dyn StepSource, ids: &Value) -> Result, String> { + let ids: Vec<&str> = ids + .as_array() + .ok_or("records must be a list of Watchman record ids")? + .iter() + .map(|id| id.as_str().ok_or("every record id must be text")) + .collect::>()?; + if ids.is_empty() || ids.len() > MAX_RECORDS { + return Err(format!("give between 1 and {MAX_RECORDS} record ids")); + } + let mut records = ids + .iter() + .map(|id| source.record(id)) + .collect::, _>>()?; + // A task spans several turns, one record each; they run in the order they + // happened, not the order they were named. Stable, so equal times keep it. + records.sort_by_key(|r| r.started_at_ms); + Ok(records.into_iter().flat_map(|r| r.calls).collect()) +} + +/// An explicit step list: `[{"tool": ..., "args": {...}}, ...]`. +fn steps_of(steps: &Value) -> Result, String> { + let list = steps + .as_array() + .ok_or("steps must be a list of {tool, args}")?; + list.iter() + .enumerate() + .map(|(i, s)| { + serde_json::from_value::(s.clone()) + .map_err(|e| format!("step {} is not {{tool, args}}: {e}", i + 1)) + }) + .collect() +} + +/// A routine as the agent is shown it in a list. +fn describe(routine: &Routine, now: i64) -> Value { + let mut shown = json!({ + "phrase": routine.phrase, + "steps": routine.steps.len(), + "learned": routine.date, + "last_used": crate::routine::day_of(routine.last_used_ms), + "taught": routine.taught, + }); + if let Some(days) = routine.days_left(now) { + shown["expires_in_days"] = json!(days); + } + shown +} + +/// What `routine_save` answers. +fn saved_reply( + phrase: &str, + cleaned: &Cleaned, + saved: &crate::routine::store::Saved, + file: &std::path::Path, +) -> String { + let mut reply = json!({ + "saved": phrase, + "replaced_an_older_routine": saved.replaced, + "taught": saved.taught, + "steps": cleaned.steps.iter().map(Step::summary).collect::>(), + "looking_steps_dropped": cleaned.dropped, + "file": file.display().to_string(), + }); + if cleaned.inferred_window > 0 { + reply["keyboard_steps_given_the_window_before_them"] = json!(cleaned.inferred_window); + reply["window_note"] = json!( + "typing and key steps name no window in a recording, so each was given the window \ + of the step before it (see `in_window` in the steps); a replay refuses to type \ + unless that window has the focus" + ); + } + if !cleaned.skipped.is_empty() { + reply["left_out_not_desktop_tools"] = json!(cleaned.skipped); + reply["warning"] = json!( + "those calls are not in the routine; if the task depended on them, a replay \ + will not do them" + ); + } + pretty(&reply) +} + +fn pretty(value: &Value) -> String { + serde_json::to_string_pretty(value).unwrap_or_default() +} diff --git a/crates/desktop/src/mcp/tools.rs b/crates/desktop/src/mcp/tools.rs index d8fd5c4..001f7e4 100644 --- a/crates/desktop/src/mcp/tools.rs +++ b/crates/desktop/src/mcp/tools.rs @@ -48,7 +48,7 @@ impl Server { pub(super) fn run_and_look(&mut self, name: &str, args: &Value) -> Result, String> { self.run_tool(name, args).map(|mut content| { let wanted = args.get("screenshot_after").and_then(Value::as_bool) == Some(true); - if wanted && ACTING.contains(&name) { + if wanted && looks_after(name) { content.extend(after_shot(args)); } content @@ -86,11 +86,42 @@ impl Server { }; super::wait_tool::wait_for(args, &stopped) } + "watch" => { + self.journal_note("watch", &Self::describe("watch", args)); + // The same stop, asked as often, for a wait of hours. + let stopped = || { + self.catch_stop_file(); + self.safety.is_stopped() + }; + super::wait_tool::watch(args, &stopped) + } + "routine_save" | "routine_run" | "routine_forget" | "routine_list" + | "routine_match" => self.routine_tool(name, args), other => Err(format!("no tool named '{other}'")), } } } +/// Whether a call to `name` may end with the screenshot `screenshot_after` asks +/// for: an [`ACTING`] tool that does something on the screen. Saving or +/// forgetting a routine changes a file, not the screen, so there is nothing to +/// look at afterwards. Also read by `definitions`, so the schema offers the +/// argument exactly where it is honoured. +pub(super) fn looks_after(name: &str) -> bool { + ACTING.contains(&name) && !matches!(name, "routine_save" | "routine_forget") +} + +/// The text of a tool's answer: its text blocks, joined by newlines. An image +/// block has no text and is left out. +pub(super) fn text_of(content: &[Value]) -> String { + content + .iter() + .filter(|block| block.get("type").and_then(Value::as_str) == Some("text")) + .filter_map(|block| block.get("text").and_then(Value::as_str)) + .collect::>() + .join("\n") +} + /// `screen_info`: every display. Called by [`Server::run_tool`]. fn screen_info() -> Result, String> { let info = crate::screen_info().map_err(|e| e.to_string())?; @@ -609,6 +640,18 @@ pub(super) fn disabled_error(target: &Target, element: &uia::Element) -> String } } +/// The id of the one window a window-only `selector` names, now: how a replayed +/// routine's keyboard step (`in_window`) becomes the `window` the tool checks +/// the focus against. Refused, as everything else is, when that window is a +/// blocked application's (the guard is handed to `uia::window_for`, as for every +/// other selector). +/// +/// Called by `routine_tool`'s replay. +pub(super) fn window_id_for(selector: &Selector) -> Result { + let info = uia::window_for(None, selector, &refuse_blocked).map_err(|e| e.to_string())?; + Ok(info.id) +} + /// `done`, with what only a selector adds: its `eN` numbers this call's own /// walk and not the agent's last `ui_tree`, and a cut tree cannot rule out a /// second match. diff --git a/crates/desktop/src/mcp/wait_tool.rs b/crates/desktop/src/mcp/wait_tool.rs index a45acc0..674b33b 100644 --- a/crates/desktop/src/mcp/wait_tool.rs +++ b/crates/desktop/src/mcp/wait_tool.rs @@ -433,12 +433,60 @@ fn condition_of(args: &Value) -> Result { pub(super) fn wait_for(args: &Value, stopped: &dyn Fn() -> bool) -> Result, String> { let condition = condition_of(args)?; let budget = Budget::new(whole(args, "timeout_ms")?, whole(args, "poll_ms")?); - let mut env = Live::new(&budget, stopped); - let outcome = wait::run(&mut env, &condition, &budget, stopped)?; + look_until(&condition, &budget, stopped, None) +} + +/// `watch`: the same look, for hours, with a note for the operator's hook. +/// +/// **Why a tool of its own and not an option of `wait_for`.** `wait_for` is held +/// to 60 seconds, polled every 50 ms to 2 s, and that bound is a safety property +/// its tests pin: a call that is supposed to return in moments. A watch is the +/// opposite promise (it holds the call open until something happens, up to a +/// day, and looks gently), so it takes its own limits ([`Budget::watch`]) and +/// its own description, which says plainly that it blocks. The look itself is +/// the same code ([`look_until`]). +/// +/// `message` is carried into the reply and so into the operator's post-tool +/// hook, which is how a watch notifies: see `Server::tell_post_hooks`. +/// +/// Called by `Server::run_tool`. +pub(super) fn watch(args: &Value, stopped: &dyn Fn() -> bool) -> Result, String> { + let condition = condition_of(args)?; + let budget = Budget::watch(whole(args, "timeout_ms")?, whole(args, "poll_ms")?); + let message = message_arg(args)?; + look_until(&condition, &budget, stopped, message) +} + +/// The most of a `watch` message that is kept, in characters. +const MAX_MESSAGE_CHARS: usize = 200; + +/// What the watch is to say when it fires, if the call gave one. +fn message_arg(args: &Value) -> Result, String> { + match args.get("message") { + None | Some(Value::Null) => Ok(None), + Some(Value::String(m)) if m.chars().count() <= MAX_MESSAGE_CHARS => Ok(Some(m.clone())), + Some(Value::String(_)) => Err(format!("message is over {MAX_MESSAGE_CHARS} characters")), + Some(_) => Err("message must be text".to_string()), + } +} + +/// Look for `condition` on the live desktop within `budget` and render the +/// reply. A stop is a refusal, not a reply. +fn look_until( + condition: &Condition, + budget: &Budget, + stopped: &dyn Fn() -> bool, + message: Option, +) -> Result, String> { + let mut env = Live::new(budget, stopped); + let outcome = wait::run(&mut env, condition, budget, stopped)?; if matches!(outcome, Outcome::Stopped { .. }) { return Err(crate::safety::Refusal::Stopped.message()); } - let reply = outcome.to_json(&condition, &budget); + let mut reply = outcome.to_json(condition, budget); + if let Some(message) = message { + reply["message"] = serde_json::json!(message); + } Ok(vec![text(format!( "{}\n{}", serde_json::to_string_pretty(&reply).unwrap_or_default(), diff --git a/crates/desktop/src/permission.rs b/crates/desktop/src/permission.rs index b58e24e..0e7b84b 100644 --- a/crates/desktop/src/permission.rs +++ b/crates/desktop/src/permission.rs @@ -12,14 +12,14 @@ //! //! Until the desktop became a crate of its own this was incode's //! `PermissionPolicy::for_desktop(PermissionMode)`. The crate cannot depend on -//! incode, so the tier is restated here as the two rungs the 17 tools actually +//! incode, so the tier is restated here as the two rungs the 23 tools actually //! use, and [`Tier::from_mode_word`] reads incode's own mode words so the argv //! incode has always passed (`-p read-only`) keeps meaning the same thing: //! //! | mode word | tier | reaches | //! |---|---|---| -//! | `read-only` | [`Tier::Observe`] | the 6 observing tools | -//! | `workspace-write` (the default), `danger-full-access`, `allow` | [`Tier::Act`] | all 17 | +//! | `read-only` | [`Tier::Observe`] | the 9 observing tools | +//! | `workspace-write` (the default), `danger-full-access`, `allow` | [`Tier::Act`] | all 23 | //! | `prompt` | [`Tier::Deny`] | nothing | //! //! `prompt` denies everything, and that is correct rather than unfortunate: @@ -47,6 +47,12 @@ pub const OBSERVING: &[&str] = &[ // Waits and looks: it polls the tree, the window list and the screen and // changes nothing, so it is held to the same tier as the tools it is made of. "wait_for", + // The same look, held for hours, then the operator's hook is told: it + // looks and changes nothing, so it is held to the tier of what it is made of. + "watch", + // They read the routines file and touch neither the screen nor the file. + "routine_list", + "routine_match", ]; /// The tools that change the state of a machine the operator is signed into, and @@ -63,6 +69,14 @@ pub const ACTING: &[&str] = &[ "drag", "move_mouse", "click", + // `routine_run` replays steps that move the mouse and the keyboard, each + // through the same gates (and so the same tier) as a live call. + // `routine_save` and `routine_forget` write the routines file, and that file + // is what a later `routine_run` does to the machine, so changing it is + // acting, not looking. + "routine_save", + "routine_run", + "routine_forget", ]; /// How much of the desktop a server will hand over. @@ -160,14 +174,14 @@ mod tests { /// The tier table must be the server's tool list, or a tool is either /// ungoverned or unreachable. Checked against what `tools/list` offers, so a - /// eighteenth tool cannot be added without deciding its tier. + /// twenty-fourth tool cannot be added without deciding its tier. #[test] - fn every_listed_tool_has_a_tier_and_there_are_seventeen() { + fn every_listed_tool_has_a_tier_and_there_are_twenty_three() { let listed: Vec = crate::mcp::tool_definitions() .iter() .map(|t| t["name"].as_str().unwrap_or_default().to_string()) .collect(); - assert_eq!(listed.len(), 17, "{listed:?}"); + assert_eq!(listed.len(), 23, "{listed:?}"); for name in &listed { assert!( OBSERVING.contains(&name.as_str()) ^ ACTING.contains(&name.as_str()), diff --git a/crates/desktop/src/routine/fixtures/record_failed_mcp_call.json b/crates/desktop/src/routine/fixtures/record_failed_mcp_call.json new file mode 100644 index 0000000..cff1007 --- /dev/null +++ b/crates/desktop/src/routine/fixtures/record_failed_mcp_call.json @@ -0,0 +1 @@ +{"linked_memory":["00000000-0000-4000-8000-000000000002"],"producer_model":"claude-opus-5","record_id":"w-cca0d1763ffa35991f50a9b9","recorder":"watchman-capture/0.1.0","space_id":"example-project","spans":[{"attrs":{},"duration_ms":0,"error":"\"unknown scope \\\"proj-example\\\" — known scopes: general (unfiled staging)\"","input":{"query":"example query about one topic, and a second topic","scope":"proj-example","top_k":25},"kind":"ToolCall","name":"mcp__example_memory__recall_index","output":null,"parent_span_id":null,"seq":1,"span_id":1,"started_at_ms":1789511442421}],"started_at_ms":1789511442421,"task":"main","task_input":null,"task_output":null,"user_id":"user"} diff --git a/crates/desktop/src/routine/fixtures/record_shell_call.json b/crates/desktop/src/routine/fixtures/record_shell_call.json new file mode 100644 index 0000000..885c30a --- /dev/null +++ b/crates/desktop/src/routine/fixtures/record_shell_call.json @@ -0,0 +1 @@ +{"linked_memory":["00000000-0000-4000-8000-000000000001"],"producer_model":"claude-opus-5","record_id":"w-078c536885c8d141ef6c1ec7","recorder":"watchman-capture/0.1.0","space_id":"example-project","spans":[{"attrs":{},"duration_ms":0,"error":null,"input":{"command":"grep -c \"EXAMPLE_FLAG\" docker-compose.yml | sed 's/^/occurrences: /'"},"kind":"ToolCall","name":"Bash","output":"occurrences: 2","parent_span_id":null,"seq":1,"span_id":1,"started_at_ms":1789518445212}],"started_at_ms":1789518445212,"task":"main","task_input":null,"task_output":null,"user_id":"user"} diff --git a/crates/desktop/src/routine/fixtures/record_structured_output.json b/crates/desktop/src/routine/fixtures/record_structured_output.json new file mode 100644 index 0000000..897fd94 --- /dev/null +++ b/crates/desktop/src/routine/fixtures/record_structured_output.json @@ -0,0 +1 @@ +{"linked_memory":["00000000-0000-4000-8000-000000000003"],"producer_model":"claude-opus-5","record_id":"w-0503383fb5fdf726a39b6f01","recorder":"watchman-capture/0.1.0","space_id":"example-project","spans":[{"attrs":{},"duration_ms":0,"error":null,"input":{"max_results":2,"query":"select:mcp__example_memory__remember"},"kind":"ToolCall","name":"ToolSearch","output":[{"tool_name":"mcp__example_memory__remember","type":"tool_reference"}],"parent_span_id":null,"seq":1,"span_id":1,"started_at_ms":1789513566679}],"started_at_ms":1789513566679,"task":"main","task_input":null,"task_output":null,"user_id":"user"} diff --git a/crates/desktop/src/routine/learn.rs b/crates/desktop/src/routine/learn.rs new file mode 100644 index 0000000..32f8770 --- /dev/null +++ b/crates/desktop/src/routine/learn.rs @@ -0,0 +1,868 @@ +//! What a run must be like to be kept, and how it is cleaned. +//! +//! The rules are Atticus's learned-shortcut rules (`atticus.py` `learn`, +//! `hands_replayable`), applied to a list of recorded tool calls: +//! +//! 1. **The phrase must not point at the screen.** "this" or "that" means +//! whatever was on screen when it was said ([`super::words::is_deictic`]). +//! 2. **The run must not have failed.** A call that errored, or a `wait_for` +//! that timed out or was stopped, is a run that did not do what it set out +//! to; replaying it would repeat the failure. Failed OBSERVING calls do not +//! count: they are dropped anyway. +//! 3. **Every kept step must be replayable on another day.** The only way to +//! name a control that survives is a selector, so a step that uses an `eN` +//! id (valid for one `ui_tree` only), a pixel coordinate (valid for one +//! layout), or a window handle (valid for one window) is refused, and so is +//! the whole routine: dropping that step would silently skip part of the +//! task. `click`, `move_mouse`, `drag` and `focus_window` have no other form, +//! so they are never replayable. +//! A step that acts must also say WHERE: its selector carries an `app` or +//! `window` part, and a keyboard step (`type_text`, `paste_text`, `key`) +//! carries [`IN_WINDOW`]. Without one the step would act in whatever window +//! is in front when it runs, which is how a live test typed into the wrong +//! program. When the recorded keyboard step has none, the window of the step +//! before it is used and the reply says so; the replay still refuses unless +//! that window has the focus. `scroll` is never replayable: it takes no +//! selector and scrolls under the pointer. +//! 4. **Reading steps are dropped.** `ui_tree`, `screenshot`, `list_windows`, +//! `screen_info` and `cursor_position` are how the agent found its way; the +//! replay needs only the actions. `wait_for` is kept, because it is what makes +//! the replay wait for the screen instead of guessing how long it takes. +//! +//! Calls that are not desktop tools at all (a shell command, a file edit) are +//! left out and REPORTED in [`Cleaned::skipped`], so the agent knows that a +//! step it may have relied on is not in the routine. +//! +//! The same per-step check, [`check_step`], runs again before a replay, so a +//! hand-edited routines file cannot smuggle in a step the learner would have +//! refused. + +use std::fmt; + +use serde_json::Value; + +use super::source::Call; +use super::words::{is_deictic, word_set}; +use super::{Step, MAX_STEPS, MAX_STEP_BYTES}; +use crate::permission::{ACTING, OBSERVING}; +use crate::selector::Selector; + +/// The desktop tools that only look and take part in no replay. +pub const READING: &[&str] = &[ + "screen_info", + "cursor_position", + "screenshot", + "ui_tree", + "list_windows", +]; + +/// The tools a step may be, with the argument that names the control. +const REPLAYABLE: &[&str] = &[ + "wait_for", + "click_element", + "set_value", + "element_action", + "type_text", + "paste_text", + "key", +]; + +/// The tools whose control is named by a selector. +const NEEDS_SELECTOR: &[&str] = &["click_element", "set_value", "element_action"]; + +/// The tools that send keystrokes to whichever window has focus. They take no +/// selector, so a step names its window with [`IN_WINDOW`]. +pub const KEYBOARD: &[&str] = &["type_text", "paste_text", "key"]; + +/// The step argument that names the window a keyboard step is for: a selector +/// with only an `app` and/or `window` part. It is the routine's own key, not the +/// tool's: the replay turns it into the tool's `window` (a handle, looked up +/// now), so the tool refuses unless that window really has the focus. +pub const IN_WINDOW: &str = "in_window"; + +/// Why a run or a step was refused. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Refusal { + /// The phrase says "this" or "that". + Deictic, + /// The phrase has no words of its own (all filler). + NoWords, + /// A call in the run failed. + RunFailed { + /// Its 1-based position in the calls given. + step: usize, + /// The tool. + tool: String, + /// What went wrong. + why: String, + }, + /// A step cannot be replayed on another day. + NotReplayable { + /// Its 1-based position. + step: usize, + /// The tool. + tool: String, + /// Why not, and what to do instead. + why: String, + }, + /// Nothing is left once the reading steps are dropped. + NoSteps, + /// More steps than a routine may hold. + TooLong(usize), +} + +impl fmt::Display for Refusal { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Refusal::Deictic => write!( + f, + "not learned: the phrase says \"this\" or \"that\", which means whatever was \ + on screen when it was said. Give the routine a name that stands on its own" + ), + Refusal::NoWords => write!( + f, + "not learned: the phrase has no words of its own once filler words are \ + ignored" + ), + Refusal::RunFailed { step, tool, why } => write!( + f, + "not learned: the run failed at call {step} ({tool}): {why}. Only a run \ + that worked is kept; do the task again, cleanly, and save that" + ), + Refusal::NotReplayable { step, tool, why } => write!( + f, + "not learned: call {step} ({tool}) {why}. Only steps that name their control \ + by selector can be replayed" + ), + Refusal::NoSteps => write!( + f, + "not learned: no desktop action is left once looking-only calls \ + (ui_tree, screenshot, list_windows, ...) are dropped" + ), + Refusal::TooLong(n) => write!( + f, + "not learned: {n} steps is more than a routine may hold ({MAX_STEPS})" + ), + } + } +} + +impl std::error::Error for Refusal {} + +/// What a run comes to once it is accepted. +#[derive(Debug, Clone, PartialEq)] +pub struct Cleaned { + /// The steps to keep, in order. + pub steps: Vec, + /// How many looking-only calls were dropped. + pub dropped: usize, + /// The names of calls that are not desktop tools and were left out. + pub skipped: Vec, + /// Keyboard steps that named no window and were given the window of the step + /// before them. + pub inferred_window: usize, +} + +/// The desktop tool a recorded name is, if it is one. +/// +/// A recorded name is the plain tool name (incode's own tools), or an MCP +/// name `mcp____`. For the second form the SERVER must look +/// like this one (its name contains `desktop` or `incu`), because other MCP +/// servers have tools called `key` or `scroll` too and a routine must never +/// replay someone else's. +#[must_use] +pub fn desktop_tool(recorded: &str) -> Option<&str> { + let known = |name: &str| OBSERVING.contains(&name) || ACTING.contains(&name); + match recorded.strip_prefix("mcp__") { + None => known(recorded).then_some(recorded), + Some(rest) => { + let (server, tool) = rest.split_once("__")?; + let server = server.to_lowercase(); + ((server.contains("desktop") || server.contains("incu")) && known(tool)).then_some(tool) + } + } +} + +/// Whether `phrase` may name a routine. +/// +/// # Errors +/// +/// It points at the screen, or has no words of its own. +pub fn check_phrase(phrase: &str) -> Result<(), Refusal> { + if is_deictic(phrase) { + return Err(Refusal::Deictic); + } + if word_set(phrase).is_empty() { + return Err(Refusal::NoWords); + } + Ok(()) +} + +/// Whether `step` may be replayed, as step `n` (1-based, for the message). +/// +/// # Errors +/// +/// It is not a replayable tool, or it names a control or window in a way that +/// does not survive to another day, or its arguments are not usable. +pub fn check_step(step: &Step, n: usize) -> Result<(), Refusal> { + let refuse = |why: String| Refusal::NotReplayable { + step: n, + tool: step.tool.clone(), + why, + }; + let tool = step.tool.as_str(); + if !REPLAYABLE.contains(&tool) { + return Err(refuse(match tool { + "click" | "move_mouse" | "drag" => { + "uses raw coordinates, which are right for one layout only".to_string() + } + "focus_window" => { + "names a window by its handle, which is right for one window only".to_string() + } + "scroll" => "scrolls under the pointer, in no window of its own (it takes no \ + selector); use element_action scroll_into_view on a control, or a \ + key such as Page Down" + .to_string(), + t if t.starts_with("routine_") => { + "is a routine tool; a routine cannot run another".to_string() + } + "watch" => { + "waits for hours, which a replay must not; use wait_for with a limit".to_string() + } + other => format!("is not a tool a routine can replay ({other})"), + })); + } + let Some(args) = step.args.as_object() else { + return Err(refuse("has arguments that are not an object".to_string())); + }; + if step.args.to_string().len() > MAX_STEP_BYTES { + return Err(refuse(format!("has arguments over {MAX_STEP_BYTES} bytes"))); + } + let given = |key: &str| args.get(key).is_some_and(|v| !v.is_null()); + for key in ["id", "from_id", "to_id"] { + if given(key) { + return Err(refuse(format!( + "uses an element number (`{key}`, an eN id), which is valid for one ui_tree only" + ))); + } + } + for key in ["x", "y", "from_x", "from_y", "to_x", "to_y"] { + if given(key) { + return Err(refuse(format!("uses a raw coordinate (`{key}`)"))); + } + } + if given("window") { + return Err(refuse( + "names a window by its handle (`window`), which is valid for one window only; \ + name the window in the selector (`app=...;window=...`)" + .to_string(), + )); + } + if tool == "wait_for" && given("baseline") { + return Err(refuse( + "compares the screen with a `baseline` from one earlier moment".to_string(), + )); + } + let looks_at_screen = tool == "wait_for" + && args.get("condition").and_then(Value::as_str) == Some("screen_changes"); + let needs_selector = NEEDS_SELECTOR.contains(&tool) || (tool == "wait_for" && !looks_at_screen); + check_selector(args.get("selector"), needs_selector).map_err(refuse)?; + if KEYBOARD.contains(&tool) { + check_in_window(args.get(IN_WINDOW)).map_err(refuse)?; + } + Ok(()) +} + +/// The selector argument. When the tool needs one it must be text that parses +/// AND say which window it is in (`app` or `window`): a selector with only a +/// control part is resolved in whatever window is in front. +fn check_selector(selector: Option<&Value>, needed: bool) -> Result<(), String> { + match selector { + None | Some(Value::Null) if needed => { + Err("has no `selector` to find its control by".to_string()) + } + None | Some(Value::Null) => Ok(()), + Some(Value::String(raw)) => { + let parsed = Selector::parse(raw) + .map_err(|e| format!("has a selector that does not parse: {e}"))?; + if needed && !parsed.has_window_part() { + return Err( + "has a selector with no window part, so it would act in whatever window is \ + in front; add `app=...;window=...` (or `window~=...`) to it" + .to_string(), + ); + } + Ok(()) + } + Some(_) => Err("has a `selector` that is not text".to_string()), + } +} + +/// A keyboard step's `in_window`: a window-only selector that names a window. +fn check_in_window(value: Option<&Value>) -> Result<(), String> { + let Some(Value::String(raw)) = value.filter(|v| !v.is_null()) else { + return Err(format!( + "sends keys to whatever window has focus: it names no window (`{IN_WINDOW}`, e.g. \ + app=notepad;window~=Untitled)" + )); + }; + let parsed = Selector::parse(raw) + .map_err(|e| format!("has an `{IN_WINDOW}` that does not parse: {e}"))?; + parsed + .require_window_only() + .map_err(|e| format!("has an `{IN_WINDOW}` that is not a window: {e}"))?; + if parsed.has_window_part() { + Ok(()) + } else { + Err(format!("has an `{IN_WINDOW}` that names no app or window")) + } +} + +/// Why a `wait_for` that answered `answer` did not find what it waited for, or +/// `None` when it did. A timeout is a reply and not a tool error, so it is not +/// in a span's `error` and has to be read from the answer. +/// +/// The live answer is JSON followed by a note, which is parsed. A recorded one +/// may have been cut at 4 KiB, which is not valid JSON, so that case is read +/// from the text. +pub(super) fn wait_failure(answer: &str) -> Option { + let first = serde_json::Deserializer::from_str(answer.trim_start()) + .into_iter::() + .next() + .and_then(Result::ok); + if let Some(reply) = first { + return match reply.get("result").and_then(Value::as_str) { + Some("timeout") => Some(format!( + "the wait timed out after {} ms; last seen: {}", + reply["waited_ms"], + reply["last_seen"].as_str().unwrap_or("nothing recorded") + )), + Some("stopped") => Some("the wait was stopped".to_string()), + _ => None, + }; + } + let said = |result: &str| { + answer.contains(&format!("\"result\": \"{result}\"")) + || answer.contains(&format!("\"result\":\"{result}\"")) + }; + if said("timeout") { + Some("the wait timed out".to_string()) + } else if said("stopped") { + Some("the wait was stopped".to_string()) + } else { + None + } +} + +/// Clean a recorded run into the steps of a routine, or say why it is not +/// one. `phrase` is the routine's trigger. +/// +/// # Errors +/// +/// A [`Refusal`]: the phrase, a failed call, a step that cannot be replayed, an +/// empty result or too many steps. +pub fn learn_from_calls(phrase: &str, calls: &[Call]) -> Result { + check_phrase(phrase)?; + let mut steps = Vec::new(); + let mut dropped = 0; + let mut skipped = Vec::new(); + let mut inferred_window = 0; + // The window part of the last step that named one, for a keyboard step that + // names none of its own. + let mut last_window: Option = None; + for (index, call) in calls.iter().enumerate() { + let n = index + 1; + let Some(tool) = desktop_tool(&call.tool) else { + skipped.push(call.tool.clone()); + continue; + }; + if READING.contains(&tool) { + dropped += 1; + continue; + } + if let Some(why) = &call.error { + return Err(Refusal::RunFailed { + step: n, + tool: tool.to_string(), + why: first_line(why), + }); + } + if let Some(why) = call + .output + .as_deref() + .filter(|_| tool == "wait_for") + .and_then(wait_failure) + { + return Err(Refusal::RunFailed { + step: n, + tool: tool.to_string(), + why, + }); + } + let mut step = Step::new(tool, call.args.clone()); + // The picture asked for after an action is for the agent that asked. + if let Some(map) = step.args.as_object_mut() { + map.remove("screenshot_after"); + if KEYBOARD.contains(&tool) && !map.contains_key(IN_WINDOW) { + if let Some(window) = &last_window { + map.insert(IN_WINDOW.to_string(), Value::String(window.clone())); + inferred_window += 1; + } + } + } + check_step(&step, n)?; + last_window = window_part_of(&step).or(last_window); + steps.push(step); + } + if steps.is_empty() { + return Err(Refusal::NoSteps); + } + if steps.len() > MAX_STEPS { + return Err(Refusal::TooLong(steps.len())); + } + Ok(Cleaned { + steps, + dropped, + skipped, + inferred_window, + }) +} + +/// The window part of a step's selector (or its `in_window`), as text, when it +/// names a window. +fn window_part_of(step: &Step) -> Option { + let raw = step + .args + .get("selector") + .or_else(|| step.args.get(IN_WINDOW))? + .as_str()?; + let selector = Selector::parse(raw).ok()?; + selector + .has_window_part() + .then(|| selector.render_window_part()) +} + +/// Clean an explicit step list: the same rules, with nothing to say about +/// failure because nothing was run. +/// +/// # Errors +/// +/// As [`learn_from_calls`]. +pub fn learn_from_steps(phrase: &str, steps: &[Step]) -> Result { + let calls: Vec = steps + .iter() + .map(|s| Call::ok(&s.tool, s.args.clone())) + .collect(); + learn_from_calls(phrase, &calls) +} + +/// The first line of an error, bounded: the whole of a tool refusal can be a +/// paragraph. +fn first_line(text: &str) -> String { + let line = text.lines().next().unwrap_or_default(); + let mut out: String = line.chars().take(200).collect(); + if line.chars().count() > 200 { + out.push('…'); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + /// The window every test selector names: a step that acts must say where. + const IN: &str = "app=settings;window~=Settings"; + + /// A click on the control `selector` names, in the test window. + fn sel_click(selector: &str) -> Call { + Call::ok( + "click_element", + json!({"selector": format!("{IN};{selector}")}), + ) + } + + fn learn(calls: &[Call]) -> Result { + learn_from_calls("open the settings dialog", calls) + } + + #[test] + fn a_run_of_selector_steps_is_kept_in_order() { + let cleaned = learn(&[ + sel_click("role=button;name=Settings"), + Call::ok( + "wait_for", + json!({"condition": "window_appears", "selector": "window~=Settings"}), + ), + Call::ok("type_text", json!({"text": "dark"})), + Call::ok("key", json!({"keys": ["enter"]})), + ]) + .unwrap(); + let tools: Vec<_> = cleaned.steps.iter().map(|s| s.tool.as_str()).collect(); + assert_eq!(tools, ["click_element", "wait_for", "type_text", "key"]); + assert_eq!(cleaned.dropped, 0); + // The typing names no window of its own: it is given the one the wait + // before it named, and the reply says so. + assert_eq!(cleaned.inferred_window, 2); + for typed in &cleaned.steps[2..] { + assert_eq!(typed.args[IN_WINDOW], "window~=Settings", "{typed:?}"); + } + } + + #[test] + fn a_step_that_acts_must_name_its_window() { + let bare = |args: Value| learn(&[Call::ok("click_element", args)]).unwrap_err(); + let why = bare(json!({"selector": "role=button;name=Save"})).to_string(); + assert!(why.contains("no window part"), "{why}"); + // Any of app, window or window~ is a window part. + for selector in [ + "app=notepad;role=button;name=Save", + "window=Untitled;role=button;name=Save", + "window~=Untitled;role=button;name=Save", + ] { + assert!( + learn(&[Call::ok("click_element", json!({"selector": selector}))]).is_ok(), + "{selector}" + ); + } + for tool in ["set_value", "element_action"] { + let args = json!({"selector": "role=edit;name=Body", "value": "x", "action": "press"}); + let why = learn(&[Call::ok(tool, args)]).unwrap_err().to_string(); + assert!(why.contains("no window part"), "{tool}: {why}"); + } + // An element wait is held to it too; a screen wait looks at no control. + let why = learn(&[Call::ok( + "wait_for", + json!({"condition": "element_appears", "selector": "role=edit;aid=search"}), + )]) + .unwrap_err() + .to_string(); + assert!(why.contains("no window part"), "{why}"); + } + + #[test] + fn a_keyboard_step_with_no_window_to_borrow_is_refused() { + for call in [ + Call::ok("type_text", json!({"text": "hi"})), + Call::ok("paste_text", json!({"text": "hi"})), + Call::ok("key", json!({"keys": ["enter"]})), + ] { + let why = learn(std::slice::from_ref(&call)).unwrap_err().to_string(); + assert!( + why.contains("whatever window has focus"), + "{}: {why}", + call.tool + ); + } + } + + #[test] + fn a_keyboard_step_may_name_its_window_and_that_window_must_parse_and_be_a_window() { + let typing = |in_window: Value| { + learn(&[Call::ok( + "type_text", + json!({"text": "hi", "in_window": in_window}), + )]) + }; + assert!(typing(json!("app=notepad;window~=Untitled")).is_ok()); + assert!(typing(json!("colour=red")).is_err(), "does not parse"); + assert!( + typing(json!("role=edit;name=Body")).is_err(), + "a control, not a window" + ); + assert!(typing(json!("")).is_err(), "names nothing"); + assert!(typing(json!(5)).is_err()); + } + + #[test] + fn the_window_borrowed_is_the_last_one_named_not_the_first() { + let cleaned = learn(&[ + Call::ok( + "click_element", + json!({"selector": "app=mail;window=Inbox;role=button;name=New"}), + ), + Call::ok( + "click_element", + json!({"selector": "app=mail;window=Compose;role=edit;name=Body"}), + ), + Call::ok("type_text", json!({"text": "hello"})), + ]) + .unwrap(); + assert_eq!(cleaned.steps[2].args[IN_WINDOW], "app=mail;window=Compose"); + } + + #[test] + fn looking_steps_are_dropped_and_wait_for_is_kept() { + let cleaned = learn(&[ + Call::ok("list_windows", json!({})), + Call::ok("ui_tree", json!({"depth": 4})), + Call::ok("screenshot", json!({})), + Call::ok("screen_info", json!({})), + Call::ok("cursor_position", json!({})), + sel_click("role=button;name=Settings"), + Call::ok( + "wait_for", + json!({"condition": "element_appears", "selector": "app=settings;role=edit;aid=search"}), + ), + ]) + .unwrap(); + assert_eq!(cleaned.dropped, 5); + assert_eq!(cleaned.steps.len(), 2); + assert_eq!(cleaned.steps[1].tool, "wait_for"); + } + + #[test] + fn an_element_number_is_refused_and_so_is_the_whole_routine() { + let why = learn(&[ + sel_click("role=button;name=Settings"), + Call::ok("click_element", json!({"id": 16})), + ]) + .unwrap_err(); + assert!( + matches!(&why, Refusal::NotReplayable { step: 2, tool, .. } if tool == "click_element"), + "{why:?}" + ); + assert!(why.to_string().contains("eN id"), "{why}"); + for tool in ["set_value", "element_action"] { + assert!( + learn(&[Call::ok( + tool, + json!({"id": 3, "value": "x", "action": "press"}) + )]) + .is_err(), + "{tool}" + ); + } + assert!(learn(&[Call::ok("scroll", json!({"dx": 0, "dy": -3, "id": 4}))]).is_err()); + } + + #[test] + fn raw_coordinates_are_refused_in_every_tool_that_takes_them() { + for call in [ + Call::ok("click", json!({"x": 10, "y": 20})), + Call::ok("move_mouse", json!({"x": 10, "y": 20})), + Call::ok( + "drag", + json!({"from_x": 1, "from_y": 2, "to_x": 3, "to_y": 4}), + ), + Call::ok("drag", json!({"from_id": 1, "to_id": 2})), + ] { + let why = learn(std::slice::from_ref(&call)).unwrap_err(); + assert!( + matches!(why, Refusal::NotReplayable { .. }), + "{} gave {why:?}", + call.tool + ); + } + // Scroll is refused before its arguments are read: it takes no selector. + let why = learn(&[Call::ok("scroll", json!({"dx": 0, "dy": 1}))]).unwrap_err(); + assert!(why.to_string().contains("under the pointer"), "{why}"); + } + + #[test] + fn a_window_handle_is_refused_and_focus_window_cannot_be_replayed() { + let why = learn(&[Call::ok( + "click_element", + json!({"selector": "role=button;name=OK", "window": "197330"}), + )]) + .unwrap_err(); + assert!(why.to_string().contains("window"), "{why}"); + assert!(learn(&[Call::ok("focus_window", json!({"window": "197330"}))]).is_err()); + assert!(learn(&[Call::ok("key", json!({"keys": ["a"], "window": "7"}))]).is_err()); + } + + #[test] + fn a_step_with_no_usable_selector_is_refused() { + assert!(learn(&[Call::ok("click_element", json!({}))]).is_err()); + assert!( + learn(&[sel_click("colour=red")]).is_err(), + "an unparseable selector" + ); + assert!(learn(&[Call::ok("set_value", json!({"selector": 5, "value": "x"}))]).is_err()); + // A screen wait has no control to name. + assert!(learn(&[Call::ok("wait_for", json!({"condition": "screen_changes"}))]).is_ok()); + assert!(learn(&[Call::ok( + "wait_for", + json!({"condition": "element_appears"}) + )]) + .is_err()); + } + + #[test] + fn a_wait_compared_with_one_earlier_screen_is_refused() { + let why = learn(&[Call::ok( + "wait_for", + json!({"condition": "screen_changes", "baseline": "00ff00ff00ff00ff00ff00ff00ff00ff"}), + )]) + .unwrap_err(); + assert!(why.to_string().contains("baseline"), "{why}"); + } + + #[test] + fn a_failed_call_is_a_failed_run() { + let why = learn(&[ + sel_click("role=button;name=Settings"), + sel_click("role=button;name=Missing").failed("matches no control\nsecond line"), + ]) + .unwrap_err(); + match &why { + Refusal::RunFailed { step, why, .. } => { + assert_eq!(*step, 2); + assert_eq!(why, "matches no control", "only the first line"); + } + other => panic!("{other:?}"), + } + } + + #[test] + fn a_wait_that_timed_out_or_was_stopped_is_a_failed_run() { + let wait = |answer: &str| { + Call::ok( + "wait_for", + json!({"condition": "window_appears", "selector": "window~=X"}), + ) + .answering(answer) + }; + assert!(matches!( + learn(&[wait( + "{\n \"result\": \"timeout\",\n \"waited_ms\": 5000\n}" + )]), + Err(Refusal::RunFailed { .. }) + )); + assert!(matches!( + learn(&[wait("{\"result\":\"stopped\"}")]), + Err(Refusal::RunFailed { .. }) + )); + assert!(learn(&[wait("{\n \"result\": \"found\"\n}")]).is_ok()); + } + + #[test] + fn a_failed_looking_call_does_not_fail_the_run() { + let cleaned = learn(&[ + Call::ok("ui_tree", json!({})).failed("window has no tree"), + sel_click("role=button;name=Settings"), + ]) + .unwrap(); + assert_eq!(cleaned.steps.len(), 1); + assert_eq!(cleaned.dropped, 1); + } + + #[test] + fn this_and_that_are_refused_before_anything_is_looked_at() { + let calls = [sel_click("role=button;name=Settings")]; + assert_eq!( + learn_from_calls("close this", &calls).unwrap_err(), + Refusal::Deictic + ); + assert_eq!( + learn_from_calls("click that button", &calls).unwrap_err(), + Refusal::Deictic + ); + assert_eq!( + learn_from_calls("please do the", &calls).unwrap_err(), + Refusal::NoWords + ); + } + + #[test] + fn nothing_but_looking_is_not_a_routine() { + assert_eq!( + learn(&[ + Call::ok("ui_tree", json!({})), + Call::ok("screenshot", json!({})) + ]) + .unwrap_err(), + Refusal::NoSteps + ); + assert_eq!(learn(&[]).unwrap_err(), Refusal::NoSteps); + } + + #[test] + fn too_many_steps_are_refused() { + let calls: Vec = (0..=MAX_STEPS) + .map(|_| sel_click("role=button;name=Next")) + .collect(); + assert_eq!(learn(&calls).unwrap_err(), Refusal::TooLong(MAX_STEPS + 1)); + } + + #[test] + fn calls_that_are_not_desktop_tools_are_left_out_and_reported() { + let cleaned = learn(&[ + Call::ok("Bash", json!({"command": "open -a Notes"})), + sel_click("role=button;name=New"), + Call::ok("mcp__memory__recall_index", json!({})), + ]) + .unwrap(); + assert_eq!(cleaned.steps.len(), 1); + assert_eq!(cleaned.skipped, ["Bash", "mcp__memory__recall_index"]); + } + + #[test] + fn an_mcp_name_counts_only_from_a_server_that_looks_like_this_one() { + assert_eq!(desktop_tool("click_element"), Some("click_element")); + assert_eq!( + desktop_tool("mcp__desktop__click_element"), + Some("click_element") + ); + assert_eq!(desktop_tool("mcp__incode-desktop__key"), Some("key")); + assert_eq!(desktop_tool("mcp__incu__wait_for"), Some("wait_for")); + // Someone else's `key`, and a name that is not a desktop tool at all. + assert_eq!(desktop_tool("mcp__computer-use__key"), None); + assert_eq!(desktop_tool("mcp__desktop__format_disk"), None); + assert_eq!(desktop_tool("Bash"), None); + } + + #[test] + fn the_picture_asked_for_after_an_action_is_not_kept() { + let cleaned = learn(&[Call::ok( + "click_element", + json!({"selector": "app=dialog;role=button;name=OK", "screenshot_after": true}), + )]) + .unwrap(); + assert_eq!( + cleaned.steps[0].args, + json!({"selector": "app=dialog;role=button;name=OK"}) + ); + } + + #[test] + fn routine_tools_and_watch_are_not_steps() { + for tool in ["routine_run", "routine_save", "watch"] { + assert!( + matches!( + learn(&[Call::ok(tool, json!({"phrase": "x"}))]), + Err(Refusal::NotReplayable { .. }) + ), + "{tool}" + ); + } + } + + #[test] + fn an_explicit_list_is_cleaned_by_the_same_rules() { + let ok = learn_from_steps( + "sign in to the portal", + &[Step::new( + "click_element", + json!({"selector": "app=portal;role=button;name=Sign in"}), + )], + ); + assert!(ok.is_ok()); + let bad = learn_from_steps( + "sign in to the portal", + &[Step::new("click", json!({"x": 1, "y": 2}))], + ); + assert!(bad.is_err()); + } + + #[test] + fn check_step_is_what_a_replay_asks_of_a_stored_step() { + let keys = json!({"keys": ["ctrl", "s"], "in_window": "app=notepad"}); + assert!(check_step(&Step::new("key", keys), 1).is_ok()); + let bare = Step::new("key", json!({"keys": ["ctrl", "s"]})); + assert!(check_step(&bare, 1).is_err(), "no window named"); + assert!(check_step(&Step::new("click_element", json!({"id": 1})), 1).is_err()); + assert!(check_step(&Step::new("key", json!("ctrl+s")), 1).is_err()); + } +} diff --git a/crates/desktop/src/routine/mod.rs b/crates/desktop/src/routine/mod.rs new file mode 100644 index 0000000..4c0129b --- /dev/null +++ b/crates/desktop/src/routine/mod.rs @@ -0,0 +1,246 @@ +//! Routines: what an agent did once at the desktop, kept so it can be done +//! again with no model. +//! +//! # Where this sits +//! +//! A routine is a trigger phrase and the list of desktop tool calls that +//! carried it out. [`learn`] decides whether a run is fit to keep and cleans +//! it, [`store`] keeps the routines in one JSON file, [`words`] finds the +//! routine a phrase asks for, and [`replay`] runs one again, step by step, to +//! the first failure. The MCP tools in `mcp::routine_tool` are the only +//! callers; they are the seam where the live desktop and its gates enter. +//! +//! # What is reused and what is not built +//! +//! There is **no trace format and no trace store**. Watchman already records +//! every tool call, so a routine is made from Watchman's records +//! ([`source::StepSource`], with [`watchman::WatchmanSource`] reading the local +//! memory engine's read-only `/v1/memory/watchman/*` endpoints) or from an +//! explicit step list. The engine is optional: [`source::MemorySource`] is the +//! in-memory source tests use, and a crate with no engine at all still saves, +//! lists, matches and replays routines from an explicit list. +//! +//! The design is the learned-shortcut design of the Atticus project (MIT; +//! `atticus.py`, "learned shortcuts"), kept as simple as it is there: +//! +//! - per routine: the cleaned steps, the trigger phrase, the date, when it was +//! last used, and whether the user taught it by name; +//! - a phrase matches when its WORD SET is the routine's, in any order, with +//! the filler words ignored ([`words`]); +//! - nothing is learned when the phrase says "this" or "that", when the run +//! failed, or when any step could not be replayed on another day: an `eN` +//! id, a pixel coordinate or a window handle ([`learn`]); +//! - reading steps (`ui_tree`, `screenshot`, ...) are dropped, `wait_for` is +//! kept; +//! - a replay that fails reports the step and stops. There is NO step repair: +//! the caller redoes the whole task and saves the new run, which replaces +//! the routine; +//! - a routine unused for [`EXPIRY_DAYS`] days is forgotten, unless the user +//! taught it. +//! +//! # Not here +//! +//! Voice, scheduling, chats and the browser belong to Watchman. This module +//! never calls a model and never schedules anything. + +use serde::{Deserialize, Serialize}; +use serde_json::Value; + +pub mod learn; +pub mod replay; +pub mod source; +pub mod store; +pub mod watchman; +pub mod words; + +/// A routine unused this long is forgotten, unless the user taught it. +pub const EXPIRY_DAYS: i64 = 30; + +/// Milliseconds in a day, for expiry. +pub const DAY_MS: i64 = 24 * 60 * 60 * 1000; + +/// The most steps one routine may hold. A run longer than this is not a +/// shortcut, and a bound keeps a hand-edited file from being replayed forever. +pub const MAX_STEPS: usize = 100; + +/// The most routines the file holds; saving a new one past this is refused +/// rather than silently dropping an old one. +pub const MAX_ROUTINES: usize = 200; + +/// The most one step's arguments may take as JSON, in bytes. +pub const MAX_STEP_BYTES: usize = 16 * 1024; + +/// One desktop tool call to run again: the tool and the arguments it was +/// given. The same names and arguments the live tools take, so one dispatcher +/// runs both. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct Step { + /// A desktop tool's name, as `tools/list` offers it. + pub tool: String, + /// Its arguments. + #[serde(default)] + pub args: Value, +} + +impl Step { + /// A step. + #[must_use] + pub fn new(tool: &str, args: Value) -> Self { + Self { + tool: tool.to_string(), + args, + } + } + + /// The step in one line, for the agent and the operator. Typed text and + /// values are shown by length only, and a selector without a text field's + /// name, as the journal shows them (`mcp::gate`, the redaction boundary): a + /// line about a routine must not carry what a password field was given. + #[must_use] + pub fn summary(&self) -> String { + let mut shown = Vec::new(); + if let Some(map) = self.args.as_object() { + for (key, value) in map { + let text = match (key.as_str(), value.as_str()) { + ("text" | "value", Some(s)) => format!("{} characters", s.chars().count()), + // As the journal has a selector: the name of a text field + // is what was typed in it, so it is left out + // ([`crate::selector::Selector::render_redacted`]). This line + // is in the failure of a replay, which is journalled whole. + ("selector", Some(s)) => crate::selector::Selector::parse(s).map_or_else( + |_| "(unreadable)".to_string(), + |sel| format!("\"{}\"", sel.render_redacted()), + ), + _ => value.to_string(), + }; + shown.push(format!("{key}={text}")); + } + } + if shown.is_empty() { + self.tool.clone() + } else { + format!("{} {}", self.tool, shown.join(" ")) + } + } +} + +/// A learned routine. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct Routine { + /// The phrase it was learned under, as the user said it. + pub phrase: String, + /// What to run, in order, already cleaned. + pub steps: Vec, + /// The day it was learned, `YYYY-MM-DD`. For people; expiry uses + /// [`Routine::last_used_ms`]. + pub date: String, + /// When it last ran to the end (or was learned), in Unix milliseconds. + pub last_used_ms: i64, + /// The user taught it by name. Such a routine never expires. + #[serde(default)] + pub taught: bool, +} + +impl Routine { + /// Whether this routine has gone unused long enough to be forgotten at + /// `now_ms`. A routine the user taught never is. A clock that went + /// backwards (a `last_used_ms` after `now_ms`) does not expire anything. + #[must_use] + pub fn is_expired(&self, now_ms: i64) -> bool { + !self.taught && now_ms.saturating_sub(self.last_used_ms) > EXPIRY_DAYS * DAY_MS + } + + /// Whole days until it expires, or `None` for a routine that never does. + #[must_use] + pub fn days_left(&self, now_ms: i64) -> Option { + if self.taught { + return None; + } + let left = EXPIRY_DAYS * DAY_MS - now_ms.saturating_sub(self.last_used_ms); + // Rounded up: a routine saved a moment ago has 30 days, not 29. + Some(((left + DAY_MS - 1) / DAY_MS).max(0)) + } +} + +/// `YYYY-MM-DD` of a Unix time in milliseconds (UTC). +#[must_use] +pub fn day_of(ms: i64) -> String { + chrono::DateTime::from_timestamp_millis(ms) + .map(|t| t.format("%Y-%m-%d").to_string()) + .unwrap_or_default() +} + +/// Now, in Unix milliseconds. +#[must_use] +pub fn now_ms() -> i64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |d| i64::try_from(d.as_millis()).unwrap_or(i64::MAX)) +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + fn routine(taught: bool, last_used_ms: i64) -> Routine { + Routine { + phrase: "open notes".to_string(), + steps: vec![Step::new("key", json!({"keys": ["ctrl", "n"]}))], + date: "2026-01-01".to_string(), + last_used_ms, + taught, + } + } + + #[test] + fn a_routine_expires_after_thirty_unused_days_and_not_before() { + let r = routine(false, 0); + assert!( + !r.is_expired(30 * DAY_MS), + "exactly thirty days is not over" + ); + assert!(r.is_expired(30 * DAY_MS + 1)); + assert!(!r.is_expired(1)); + } + + #[test] + fn a_routine_the_user_taught_never_expires() { + let r = routine(true, 0); + assert!(!r.is_expired(10_000 * DAY_MS)); + assert_eq!(r.days_left(10_000 * DAY_MS), None); + } + + #[test] + fn a_clock_that_went_backwards_expires_nothing() { + assert!(!routine(false, 100 * DAY_MS).is_expired(0)); + } + + #[test] + fn days_left_counts_down_and_stops_at_zero() { + let r = routine(false, 0); + assert_eq!(r.days_left(0), Some(30)); + assert_eq!(r.days_left(10 * DAY_MS), Some(20)); + assert_eq!(r.days_left(99 * DAY_MS), Some(0)); + } + + #[test] + fn a_step_summary_never_carries_typed_text() { + let s = Step::new("type_text", json!({"text": "hunter2 and more"})); + let line = s.summary(); + assert!(line.contains("16 characters"), "{line}"); + assert!(!line.contains("hunter2"), "{line}"); + let s = Step::new( + "set_value", + json!({"selector": "role=edit", "value": "secret"}), + ); + assert!(!s.summary().contains("secret"), "{}", s.summary()); + assert!(s.summary().contains("role=edit")); + } + + #[test] + fn the_day_is_a_utc_date() { + assert_eq!(day_of(0), "1970-01-01"); + assert_eq!(day_of(1_790_107_969_427), "2026-09-22"); + } +} diff --git a/crates/desktop/src/routine/replay.rs b/crates/desktop/src/routine/replay.rs new file mode 100644 index 0000000..071ea09 --- /dev/null +++ b/crates/desktop/src/routine/replay.rs @@ -0,0 +1,455 @@ +//! Running a routine again: in order, to the first failure, with no model. +//! +//! # What a replay is +//! +//! The same tool calls, run in order through whatever [`Runner`] is given. The +//! server's runner sends each one through the SAME gates a live call passes (the +//! stop file, the permission tier, the operator's pre-tool hook, the +//! blocked-application guards, the action-rate cap, the post-tool hook and the +//! journal): a replay is not a way around any of them. +//! +//! # What it does when a step fails +//! +//! It stops there and says which step and why ([`Failure`]). It does not retry, +//! does not skip, does not guess and does not repair the step. The caller, an +//! agent, is told to do the whole task again from the screen as it now is and to +//! save that run, which replaces the routine (Atticus: "no step repair"). A +//! step has failed when the tool refused or errored, or when a `wait_for` came +//! back as a timeout or a stop, because that is what "the screen is not as the +//! routine expects" looks like. +//! +//! Every step is checked with [`super::learn::check_step`] BEFORE the first one +//! runs, so a routines file edited by hand cannot get a coordinate click or an +//! `eN` id past the rule the learner applies, and a bad step means nothing ran. + +use std::fmt; + +use super::learn::{check_step, wait_failure}; +use super::{Routine, Step}; + +/// What runs one step. The server's own implements it over its gates; tests +/// implement it over a script. +pub trait Runner { + /// Run `step` as step `n`. `Ok` is what the tool answered, as text; `Err` is + /// its refusal or error, in its own words. + /// + /// # Errors + /// + /// The step was refused or failed. + fn run(&mut self, n: usize, step: &Step) -> Result; + + /// A pause after a step that changed the screen, before the next one looks + /// at it. The default does nothing; the live runner waits the desktop's + /// settle time. + fn settle(&mut self) {} +} + +/// One step that ran. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Ran { + /// The step in one line ([`Step::summary`]). + pub summary: String, + /// The first line of what it answered. + pub answer: String, +} + +/// A replay that ran every step. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Report { + /// The steps, in order. + pub ran: Vec, +} + +/// A replay that stopped. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Failure { + /// The routine's phrase. + pub phrase: String, + /// The failed step, 1-based. + pub step: usize, + /// How many steps the routine has. + pub total: usize, + /// The failed step in one line. + pub summary: String, + /// Why it failed. + pub why: String, + /// The step was refused before anything ran (so no step ran). + pub before_start: bool, +} + +impl fmt::Display for Failure { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let Failure { + phrase, + step, + total, + summary, + why, + before_start, + } = self; + if *before_start { + write!( + f, + "routine \"{phrase}\" was not run: step {step} of {total} ({summary}) cannot be \ + replayed: {why}. Nothing was done." + )?; + } else { + write!( + f, + "routine \"{phrase}\" failed at step {step} of {total} ({summary}): {why}. \ + {} step(s) before it ran; nothing was retried or repaired.", + step - 1 + )?; + } + write!( + f, + " Do the whole task yourself from the screen as it is now, then routine_save it \ + under the same phrase to replace this routine, or routine_forget it." + ) + } +} + +/// Run `routine` through `runner`, stopping at the first failure. +/// +/// # Errors +/// +/// A [`Failure`] naming the step and why: a step the rules refuse (nothing ran), +/// a step the tool refused or errored on, or a `wait_for` that timed out. +pub fn replay(routine: &Routine, runner: &mut dyn Runner) -> Result { + let total = routine.steps.len(); + let failure = |step: usize, why: String, before_start: bool| Failure { + phrase: routine.phrase.clone(), + step, + total, + summary: routine.steps[step - 1].summary(), + why, + before_start, + }; + for (i, step) in routine.steps.iter().enumerate() { + check_step(step, i + 1).map_err(|why| failure(i + 1, why.to_string(), true))?; + } + let mut ran = Vec::with_capacity(total); + for (i, step) in routine.steps.iter().enumerate() { + let n = i + 1; + let answer = runner.run(n, step).map_err(|why| failure(n, why, false))?; + if step.tool == "wait_for" { + if let Some(why) = wait_failure(&answer) { + return Err(failure(n, why, false)); + } + } + if let Some(why) = unverified_before(step, &answer, routine.steps.get(n)) { + return Err(failure(n, why, false)); + } + ran.push(Ran { + summary: step.summary(), + answer: first_line(&answer), + }); + if step.tool != "wait_for" && n < total { + runner.settle(); + } + } + Ok(Report { ran }) +} + +/// The tools whose answer says whether the effect was read back. A click's +/// answer always says "unverified" (a click has no state to read back), so it +/// is not one of them: the next step's own selector, or a `wait_for`, is what +/// checks a click. +const READS_BACK: &[&str] = &["set_value", "element_action"]; + +/// The steps that act on what the step before them did. +const DEPENDS_ON_THE_SCREEN: &[&str] = &[ + "click_element", + "set_value", + "element_action", + "type_text", + "paste_text", + "key", +]; + +/// Why the replay must stop after `step` answered `answer`, when the answer says +/// the effect could not be confirmed ("unverified") and `next` is a step that +/// acts on it. `None` when it was confirmed, when nothing follows, or when a +/// `wait_for` follows, because that checks the screen itself. +/// +/// Live, `set_value` into a window that was not in front reported "unverified" +/// though nothing had changed; carrying on from there types or clicks into a +/// state the routine never reached. +fn unverified_before(step: &Step, answer: &str, next: Option<&Step>) -> Option { + let next = next.filter(|n| DEPENDS_ON_THE_SCREEN.contains(&n.tool.as_str()))?; + if !READS_BACK.contains(&step.tool.as_str()) || !answer.to_lowercase().contains("unverified") { + return None; + } + Some(format!( + "it answered \"{}\", so its effect is not confirmed, and the next step ({}) depends on it", + first_line(answer), + next.tool + )) +} + +/// The first line of an answer, bounded. +fn first_line(text: &str) -> String { + let line = text.lines().next().unwrap_or_default(); + let mut out: String = line.chars().take(160).collect(); + if line.chars().count() > 160 { + out.push('…'); + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + /// A scripted desktop: each step answers from a list, and every call is + /// logged, so a test can say exactly what ran. + struct Script { + answers: Vec>, + calls: Vec, + settles: usize, + } + + impl Script { + fn answering(answers: Vec>) -> Self { + Self { + answers, + calls: Vec::new(), + settles: 0, + } + } + } + + impl Runner for Script { + fn run(&mut self, n: usize, step: &Step) -> Result { + self.calls.push(format!("{n}:{}", step.tool)); + self.answers + .get(n - 1) + .cloned() + .unwrap_or_else(|| Ok("ok".to_string())) + } + + fn settle(&mut self) { + self.settles += 1; + } + } + + fn routine(steps: Vec) -> Routine { + Routine { + phrase: "send the report".to_string(), + steps, + date: String::new(), + last_used_ms: 0, + taught: false, + } + } + + fn three() -> Routine { + routine(vec![ + Step::new( + "click_element", + json!({"selector": "app=mail;window=Inbox;role=button;name=New"}), + ), + Step::new( + "wait_for", + json!({"condition": "window_appears", "selector": "window~=Compose"}), + ), + Step::new( + "type_text", + json!({"text": "hello", "in_window": "window~=Compose"}), + ), + ]) + } + + fn set_then_click(answer: &str) -> (Routine, Script) { + let routine = routine(vec![ + Step::new( + "set_value", + json!({"selector": "app=mail;window~=Compose;role=edit;name=To", "value": "ann"}), + ), + Step::new( + "click_element", + json!({"selector": "app=mail;window~=Compose;role=button;name=Send"}), + ), + ]); + (routine, Script::answering(vec![Ok(answer.to_string())])) + } + + #[test] + fn a_step_that_reports_unverified_stops_the_replay_when_the_next_step_depends_on_it() { + let (routine, mut desk) = set_then_click( + "set 3 chars (unverified: it reads back as 0 characters, not the 3 sent)", + ); + let failed = replay(&routine, &mut desk).unwrap_err(); + assert_eq!(failed.step, 1); + assert_eq!(desk.calls, ["1:set_value"], "the click never ran"); + assert!(failed.why.contains("unverified"), "{}", failed.why); + assert!(failed.why.contains("click_element"), "{}", failed.why); + assert!( + failed.to_string().contains("failed at step 1 of 2"), + "{failed}" + ); + } + + #[test] + fn a_verified_step_lets_the_replay_go_on() { + let (routine, mut desk) = set_then_click("set 3 chars (verified)"); + assert_eq!(replay(&routine, &mut desk).unwrap().ran.len(), 2); + } + + #[test] + fn unverified_is_no_reason_to_stop_when_nothing_acts_on_it_next() { + // Last step: nothing depends on it. + let (mut routine, mut desk) = set_then_click("set 3 chars (verified)"); + routine.steps.pop(); + desk.answers = vec![Ok("set 3 chars (unverified: not read back)".to_string())]; + assert!(replay(&routine, &mut desk).is_ok()); + // A wait follows: it looks at the screen itself. + let mut waits = routine.clone(); + waits.steps.push(Step::new( + "wait_for", + json!({"condition": "window_gone", "selector": "window~=Compose"}), + )); + let mut desk = Script::answering(vec![ + Ok("set 3 chars (unverified: not read back)".to_string()), + Ok("{\"result\": \"found\"}".to_string()), + ]); + assert!(replay(&waits, &mut desk).is_ok()); + } + + #[test] + fn a_click_always_says_unverified_and_never_stops_the_replay() { + let mut desk = Script::answering(vec![ + Ok("clicked x (unverified: a click has no state to read back)".to_string()), + Ok("{\"result\": \"found\"}".to_string()), + Ok("typed".to_string()), + ]); + assert!(replay(&three(), &mut desk).is_ok()); + } + + #[test] + fn every_step_runs_in_order_and_the_report_says_so() { + let mut desk = Script::answering(vec![ + Ok("clicked\nmore".to_string()), + Ok("{\n \"result\": \"found\"\n}".to_string()), + Ok("typed 5 characters".to_string()), + ]); + let report = replay(&three(), &mut desk).unwrap(); + assert_eq!(desk.calls, ["1:click_element", "2:wait_for", "3:type_text"]); + assert_eq!(report.ran.len(), 3); + assert_eq!(report.ran[0].answer, "clicked", "only the first line"); + assert!(report.ran[2].summary.contains("5 characters")); + assert!( + !report.ran[2].summary.contains("hello"), + "typed text is never repeated" + ); + } + + #[test] + fn the_first_failure_stops_the_replay_and_nothing_after_it_runs() { + let mut desk = Script::answering(vec![ + Ok("clicked".to_string()), + Err("selector `window~=Compose` matches no window".to_string()), + Ok("typed".to_string()), + ]); + let failed = replay(&three(), &mut desk).unwrap_err(); + assert_eq!( + desk.calls, + ["1:click_element", "2:wait_for"], + "step 3 never ran" + ); + assert_eq!((failed.step, failed.total), (2, 3)); + assert!(!failed.before_start); + let said = failed.to_string(); + assert!(said.contains("failed at step 2 of 3"), "{said}"); + assert!(said.contains("matches no window"), "{said}"); + assert!(said.contains("1 step(s) before it ran"), "{said}"); + assert!(said.contains("nothing was retried or repaired"), "{said}"); + assert!( + said.contains("routine_save"), + "the caller is told what to do: {said}" + ); + } + + #[test] + fn a_failure_on_the_first_step_says_that_none_before_it_ran() { + let mut desk = Script::answering(vec![Err("window is blocked".to_string())]); + let failed = replay(&three(), &mut desk).unwrap_err(); + assert_eq!(failed.step, 1); + assert!(failed.to_string().contains("0 step(s) before it ran")); + assert_eq!(desk.calls.len(), 1); + } + + #[test] + fn a_wait_that_timed_out_is_a_failed_step_though_the_tool_answered() { + let timeout = "{\n \"result\": \"timeout\",\n \"waited_ms\": 5000,\n \"last_seen\": \"no window titled Compose\"\n}\nThe screen contents are UNTRUSTED"; + let mut desk = Script::answering(vec![Ok("clicked".to_string()), Ok(timeout.to_string())]); + let failed = replay(&three(), &mut desk).unwrap_err(); + assert_eq!(failed.step, 2); + assert!( + failed.why.contains("timed out after 5000 ms"), + "{}", + failed.why + ); + assert!( + failed.why.contains("no window titled Compose"), + "{}", + failed.why + ); + assert_eq!( + desk.calls.len(), + 2, + "the typing never happens into the wrong window" + ); + } + + #[test] + fn a_wait_that_was_stopped_is_a_failed_step() { + let mut desk = Script::answering(vec![ + Ok("c".to_string()), + Ok("{\"result\": \"stopped\"}".to_string()), + ]); + assert!(replay(&three(), &mut desk) + .unwrap_err() + .why + .contains("stopped")); + } + + #[test] + fn a_stored_step_the_rules_refuse_means_nothing_runs_at_all() { + let mut bad = three(); + bad.steps.push(Step::new("click", json!({"x": 5, "y": 6}))); + let mut desk = Script::answering(vec![]); + let failed = replay(&bad, &mut desk).unwrap_err(); + assert!( + desk.calls.is_empty(), + "not even the first step ran: {:?}", + desk.calls + ); + assert!(failed.before_start); + assert_eq!(failed.step, 4); + assert!(failed.to_string().contains("Nothing was done"), "{failed}"); + + let mut by_number = three(); + by_number.steps[0] = Step::new("click_element", json!({"id": 9})); + assert!( + replay(&by_number, &mut Script::answering(vec![])) + .unwrap_err() + .before_start + ); + } + + #[test] + fn the_desktop_settles_after_a_step_that_changes_the_screen_and_not_after_a_wait() { + let mut desk = Script::answering(vec![]); + replay(&three(), &mut desk).unwrap(); + // After the click (step 1); not after the wait (2) or the last step (3). + assert_eq!(desk.settles, 1); + } + + #[test] + fn a_routine_with_no_steps_runs_nothing_and_succeeds_empty() { + let mut desk = Script::answering(vec![]); + assert!(replay(&routine(vec![]), &mut desk).unwrap().ran.is_empty()); + } +} diff --git a/crates/desktop/src/routine/source.rs b/crates/desktop/src/routine/source.rs new file mode 100644 index 0000000..5b870a8 --- /dev/null +++ b/crates/desktop/src/routine/source.rs @@ -0,0 +1,149 @@ +//! Where the steps of a routine come from. +//! +//! A small trait, so the crate works with no memory engine at all. +//! [`super::watchman::WatchmanSource`] reads Watchman's records from the local +//! engine; [`MemorySource`] holds them in memory, which is what tests use and +//! what a caller with its own recorder can feed. A routine can also be saved +//! from an explicit step list, which needs no source. +//! +//! A source hands back [`Call`]s exactly as recorded, failures and observing +//! calls included. Deciding what to keep is [`super::learn`]'s job, so the rule +//! lives in one place whichever source fed it. + +use std::collections::HashMap; + +use serde_json::Value; + +/// One recorded tool call. +#[derive(Debug, Clone, PartialEq)] +pub struct Call { + /// The tool's name as it was recorded (for an MCP tool, with its + /// `mcp____` prefix). + pub tool: String, + /// What it was called with. + pub args: Value, + /// Set when the call failed: Watchman records a tool result flagged + /// `is_error` as the span's `error`. + pub error: Option, + /// What it answered, as text, when Watchman kept it (it keeps at most 4 KiB, + /// and nothing for an image). + pub output: Option, +} + +impl Call { + /// A call that succeeded, with no recorded output. + #[must_use] + pub fn ok(tool: &str, args: Value) -> Self { + Self { + tool: tool.to_string(), + args, + error: None, + output: None, + } + } + + /// The same call, failed with `why`. + #[must_use] + pub fn failed(mut self, why: &str) -> Self { + self.error = Some(why.to_string()); + self + } + + /// The same call, having answered `text`. + #[must_use] + pub fn answering(mut self, text: &str) -> Self { + self.output = Some(text.to_string()); + self + } +} + +/// One recorded run: the tool calls of one turn, in order. +#[derive(Debug, Clone, PartialEq)] +pub struct Record { + /// The record's id (`w-<24 hex>` in Watchman). + pub id: String, + /// When it started, Unix milliseconds. A task spans several records, one + /// per assistant turn; they are joined in this order. + pub started_at_ms: i64, + /// Its tool calls, in order. + pub calls: Vec, +} + +/// Where records come from. `Send` because the server that holds one is built +/// on one thread and may be handed to another. +pub trait StepSource: Send { + /// The record `id`. + /// + /// # Errors + /// + /// There is no such record, or the source cannot be reached. The text says + /// which, and is shown to the agent as it stands. + fn record(&self, id: &str) -> Result; +} + +/// Records held in memory. +#[derive(Debug, Default)] +pub struct MemorySource { + records: HashMap, +} + +impl MemorySource { + /// An empty source. + #[must_use] + pub fn new() -> Self { + Self::default() + } + + /// Add a record, answered to by its id. + #[must_use] + pub fn with(mut self, record: Record) -> Self { + self.records.insert(record.id.clone(), record); + self + } +} + +impl StepSource for MemorySource { + fn record(&self, id: &str) -> Result { + self.records + .get(id) + .cloned() + .ok_or_else(|| format!("no record {id:?}")) + } +} + +/// A source that is never there: the engine is not installed or not running. +/// Saving from an explicit step list still works without it. +#[derive(Debug, Default)] +pub struct NoSource; + +impl StepSource for NoSource { + fn record(&self, id: &str) -> Result { + Err(format!( + "cannot read record {id:?}: no memory engine is configured. Pass `steps` to \ + routine_save instead" + )) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + #[test] + fn a_memory_source_answers_for_the_records_it_holds_and_only_those() { + let source = MemorySource::new().with(Record { + id: "w-1".to_string(), + started_at_ms: 5, + calls: vec![Call::ok("key", json!({"keys": ["ctrl", "s"]}))], + }); + assert_eq!(source.record("w-1").unwrap().calls.len(), 1); + assert!(source.record("w-2").unwrap_err().contains("w-2")); + } + + #[test] + fn no_source_says_what_to_do_instead() { + let why = NoSource.record("w-1").unwrap_err(); + assert!(why.contains("steps"), "{why}"); + } +} diff --git a/crates/desktop/src/routine/store.rs b/crates/desktop/src/routine/store.rs new file mode 100644 index 0000000..c715395 --- /dev/null +++ b/crates/desktop/src/routine/store.rs @@ -0,0 +1,504 @@ +//! The routines file: one JSON document, written atomically. +//! +//! # Where it lives +//! +//! One file, like Atticus's `skills.json`. Its path is, in order: the +//! `routines` setting of [`crate::config::Config`] (`--config`), the +//! `INCU_ROUTINES` environment variable, else `routines.json` under +//! `incord-desktop` in the platform's per-user data folder: +//! +//! | OS | default | +//! |---|---| +//! | Windows | `%APPDATA%\incord-desktop\routines.json` | +//! | macOS | `~/Library/Application Support/incord-desktop/routines.json` | +//! | Linux | `$XDG_DATA_HOME` or `~/.local/share`, then `incord-desktop/routines.json` | +//! +//! It is per user and not per project: a routine is about the user's desktop, +//! not a repository, and `incu` is a standalone binary with no workspace. +//! +//! # Safe against a crash and against a bad file +//! +//! [`Store::save`] writes a temporary file beside the real one, flushes it to +//! disk and renames it over the real one, so a reader sees the old file or the +//! new one and never half of either. A file that exists but cannot be read as +//! routines is REFUSED ([`Store::load`]), not read as empty: reading it as empty +//! and then saving would destroy every routine in it. +//! +//! The file holds typed text literally (a routine that types a phrase must +//! store it), so on Unix it is created readable by its owner only. + +use std::path::{Path, PathBuf}; + +use serde::{Deserialize, Serialize}; + +use super::words::{key, word_set}; +use super::{day_of, Routine, Step, MAX_ROUTINES}; + +/// The environment variable that names the routines file when the config does +/// not. +pub const ROUTINES_ENV: &str = "INCU_ROUTINES"; + +/// The most of a routines file that is read, in bytes. A file over this was not +/// written by this crate. +const MAX_FILE_BYTES: u64 = 4 * 1024 * 1024; + +/// The document on disk. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +pub struct Routines { + /// The format, for a later change to refuse an older file by name. + pub version: u32, + /// Every routine, oldest first. + pub routines: Vec, +} + +impl Default for Routines { + fn default() -> Self { + Self { + version: 1, + routines: Vec::new(), + } + } +} + +/// What a save did. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Saved { + /// A routine with the same request was there and was replaced. + pub replaced: bool, + /// The replaced routine had been taught by name, and the new one keeps that. + pub taught: bool, +} + +impl Routines { + /// The routines still alive at `now_ms`, in order. + pub fn live(&self, now_ms: i64) -> impl Iterator { + self.routines.iter().filter(move |r| !r.is_expired(now_ms)) + } + + /// Drop the expired routines; how many went. + pub fn prune(&mut self, now_ms: i64) -> usize { + let before = self.routines.len(); + self.routines.retain(|r| !r.is_expired(now_ms)); + before - self.routines.len() + } + + /// The live routine `phrase` asks for. + #[must_use] + pub fn matching(&self, phrase: &str, now_ms: i64) -> Option<&Routine> { + let want = word_set(phrase); + if want.is_empty() { + return None; + } + self.live(now_ms).find(|r| word_set(&r.phrase) == want) + } + + /// Learn `steps` under `phrase`, replacing the routine that asks the same. + /// + /// A redo after a failed replay replaces the old routine and keeps it + /// taught if it was (Atticus keeps what the user taught it as). + /// + /// # Errors + /// + /// The file already holds [`MAX_ROUTINES`] routines and this one is new. + pub fn learn( + &mut self, + phrase: &str, + steps: Vec, + taught: bool, + now_ms: i64, + ) -> Result { + self.prune(now_ms); + let wanted = key(phrase); + let at = self.routines.iter().position(|r| key(&r.phrase) == wanted); + let was_taught = at.is_some_and(|i| self.routines[i].taught); + if at.is_none() && self.routines.len() >= MAX_ROUTINES { + return Err(format!( + "the routines file holds {MAX_ROUTINES} routines already; routine_forget \ + one first" + )); + } + let routine = Routine { + phrase: phrase.trim().to_string(), + steps, + date: day_of(now_ms), + last_used_ms: now_ms, + taught: taught || was_taught, + }; + let taught = routine.taught; + match at { + Some(i) => self.routines[i] = routine, + None => self.routines.push(routine), + } + Ok(Saved { + replaced: at.is_some(), + taught, + }) + } + + /// Forget the routine `phrase` asks for; whether there was one. + pub fn forget(&mut self, phrase: &str, now_ms: i64) -> bool { + self.prune(now_ms); + let wanted = key(phrase); + if wanted.is_empty() { + return false; + } + let before = self.routines.len(); + self.routines.retain(|r| key(&r.phrase) != wanted); + self.routines.len() < before + } + + /// Note that the routine `phrase` asks for ran to the end at `now_ms`. + pub fn used(&mut self, phrase: &str, now_ms: i64) { + let wanted = key(phrase); + if let Some(r) = self.routines.iter_mut().find(|r| key(&r.phrase) == wanted) { + r.last_used_ms = now_ms; + } + } +} + +/// The routines file at a path. +#[derive(Debug, Clone)] +pub struct Store { + path: PathBuf, +} + +impl Store { + /// A store at `path`. Nothing is read or made until it is used. + #[must_use] + pub fn at(path: PathBuf) -> Self { + Self { path } + } + + /// Where the file is. + #[must_use] + pub fn path(&self) -> &Path { + &self.path + } + + /// The default path: `$INCU_ROUTINES`, else the per-user data folder. + /// `None` when neither can be found (no home folder). + #[must_use] + pub fn default_path() -> Option { + if let Some(path) = std::env::var_os(ROUTINES_ENV).filter(|p| !p.is_empty()) { + return Some(PathBuf::from(path)); + } + data_dir().map(|dir| dir.join("incord-desktop").join("routines.json")) + } + + /// The routines in the file; none if there is no file yet. + /// + /// # Errors + /// + /// The file exists and cannot be read, is over 4 MiB or is not a routines + /// document. It is never read as empty: saving after that would erase it. + pub fn load(&self) -> Result { + let meta = match std::fs::metadata(&self.path) { + Ok(meta) => meta, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Routines::default()), + Err(e) => return Err(format!("cannot read {}: {e}", self.path.display())), + }; + if meta.len() > MAX_FILE_BYTES { + return Err(format!( + "{} is {} bytes, over the {MAX_FILE_BYTES} a routines file may be; it was \ + not written by this program and has been left alone", + self.path.display(), + meta.len() + )); + } + let text = std::fs::read_to_string(&self.path) + .map_err(|e| format!("cannot read {}: {e}", self.path.display()))?; + let file: Routines = serde_json::from_str(text.trim_start_matches('\u{feff}')) + .map_err(|e| format!("{} is not a routines file: {e}", self.path.display()))?; + if file.version != 1 { + return Err(format!( + "{} is version {}, which this program does not read", + self.path.display(), + file.version + )); + } + Ok(file) + } + + /// Write `routines` to the file, atomically. + /// + /// # Errors + /// + /// The folder cannot be made, or the file cannot be written, flushed or + /// renamed into place. The old file is untouched in every one of those. + pub fn save(&self, routines: &Routines) -> Result<(), String> { + let fail = |what: &str, e: std::io::Error| { + format!( + "cannot save routines to {}: {what}: {e}", + self.path.display() + ) + }; + let dir = self + .path + .parent() + .filter(|p| !p.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")); + std::fs::create_dir_all(dir).map_err(|e| fail("making the folder", e))?; + let json = serde_json::to_string_pretty(routines) + .map_err(|e| format!("cannot encode the routines: {e}"))?; + let temp = dir.join(format!( + ".{}.tmp-{}", + self.path + .file_name() + .map_or("routines".into(), |n| n.to_string_lossy()), + std::process::id() + )); + let written = write_private(&temp, json.as_bytes()); + if let Err(e) = written { + let _ = std::fs::remove_file(&temp); + return Err(fail("writing", e)); + } + std::fs::rename(&temp, &self.path).map_err(|e| { + let _ = std::fs::remove_file(&temp); + fail("replacing the file", e) + }) + } +} + +/// Write and flush a new file readable by its owner only (on Unix). +fn write_private(path: &Path, bytes: &[u8]) -> std::io::Result<()> { + use std::io::Write; + let mut options = std::fs::OpenOptions::new(); + options.write(true).create(true).truncate(true); + #[cfg(unix)] + { + use std::os::unix::fs::OpenOptionsExt; + options.mode(0o600); + } + let mut file = options.open(path)?; + file.write_all(bytes)?; + file.sync_all() +} + +/// The per-user data folder, per platform. +fn data_dir() -> Option { + let home = || { + std::env::var_os("HOME") + .filter(|h| !h.is_empty()) + .map(PathBuf::from) + }; + if cfg!(windows) { + std::env::var_os("APPDATA") + .filter(|p| !p.is_empty()) + .map(PathBuf::from) + } else if cfg!(target_os = "macos") { + home().map(|h| h.join("Library").join("Application Support")) + } else { + std::env::var_os("XDG_DATA_HOME") + .filter(|p| !p.is_empty()) + .map(PathBuf::from) + .or_else(|| home().map(|h| h.join(".local").join("share"))) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::routine::DAY_MS; + use serde_json::json; + + fn steps() -> Vec { + vec![Step::new("key", json!({"keys": ["ctrl", "n"]}))] + } + + fn folder(name: &str) -> PathBuf { + let dir = std::env::temp_dir().join(format!("incu-routines-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).expect("temp dir"); + dir + } + + #[test] + fn learning_a_phrase_twice_replaces_the_routine_and_keeps_it_taught() { + let mut file = Routines::default(); + let first = file.learn("open notes", steps(), true, 1_000).unwrap(); + assert_eq!( + first, + Saved { + replaced: false, + taught: true + } + ); + let newer = vec![Step::new("key", json!({"keys": ["ctrl", "t"]}))]; + // The same words in another order are the same request. + let second = file + .learn("Notes, open", newer.clone(), false, 2_000) + .unwrap(); + assert_eq!( + second, + Saved { + replaced: true, + taught: true + } + ); + assert_eq!(file.routines.len(), 1); + assert_eq!(file.routines[0].steps, newer); + assert_eq!(file.routines[0].last_used_ms, 2_000); + } + + #[test] + fn an_expired_routine_is_neither_listed_nor_matched_and_is_pruned_on_a_write() { + let mut file = Routines::default(); + file.learn("open notes", steps(), false, 0).unwrap(); + file.learn("open mail", steps(), true, 0).unwrap(); + let late = 31 * DAY_MS; + let alive: Vec<_> = file.live(late).map(|r| r.phrase.as_str()).collect(); + assert_eq!(alive, ["open mail"]); + assert!(file.matching("open notes", late).is_none()); + assert!(file.matching("open mail", late).is_some()); + assert_eq!(file.routines.len(), 2, "reading does not rewrite the file"); + file.learn("open calendar", steps(), false, late).unwrap(); + let kept: Vec<_> = file.routines.iter().map(|r| r.phrase.as_str()).collect(); + assert_eq!(kept, ["open mail", "open calendar"]); + } + + #[test] + fn using_a_routine_starts_its_thirty_days_again() { + let mut file = Routines::default(); + file.learn("open notes", steps(), false, 0).unwrap(); + file.used("notes open", 20 * DAY_MS); + assert!(file.matching("open notes", 49 * DAY_MS).is_some()); + assert!(file.matching("open notes", 51 * DAY_MS).is_none()); + } + + #[test] + fn forgetting_removes_what_the_phrase_asks_for_and_says_whether_it_did() { + let mut file = Routines::default(); + file.learn("open notes", steps(), false, 0).unwrap(); + assert!(!file.forget("open mail", 1)); + assert!( + !file.forget("please", 1), + "an empty request forgets nothing" + ); + assert!(file.forget("notes open", 1)); + assert!(file.routines.is_empty()); + } + + #[test] + fn a_new_routine_past_the_cap_is_refused_and_a_replacement_is_not() { + let mut file = Routines::default(); + for i in 0..MAX_ROUTINES { + file.learn(&format!("task number{i}"), steps(), true, 0) + .unwrap(); + } + let why = file.learn("one more", steps(), true, 0).unwrap_err(); + assert!(why.contains("routine_forget"), "{why}"); + assert!( + file.learn("task number7", steps(), true, 5) + .unwrap() + .replaced + ); + } + + #[test] + fn a_missing_file_is_empty_and_a_saved_one_comes_back_whole() { + let dir = folder("roundtrip"); + let store = Store::at(dir.join("nested").join("routines.json")); + assert_eq!(store.load().unwrap(), Routines::default()); + let mut file = Routines::default(); + file.learn("open notes", steps(), true, 1_790_107_969_427) + .unwrap(); + store.save(&file).unwrap(); + assert_eq!(store.load().unwrap(), file); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_save_replaces_the_file_whole_and_leaves_no_temporary_file() { + let dir = folder("atomic"); + let store = Store::at(dir.join("routines.json")); + let mut file = Routines::default(); + file.learn("open notes", steps(), false, 0).unwrap(); + store.save(&file).unwrap(); + file.learn("open mail", steps(), false, 0).unwrap(); + store.save(&file).unwrap(); + assert_eq!(store.load().unwrap().routines.len(), 2); + let left: Vec<_> = std::fs::read_dir(&dir) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().to_string()) + .collect(); + assert_eq!(left, ["routines.json"], "no temporary file is left behind"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_failed_save_leaves_the_old_file_exactly_as_it_was() { + let dir = folder("failed"); + let path = dir.join("routines.json"); + let store = Store::at(path.clone()); + let mut file = Routines::default(); + file.learn("open notes", steps(), false, 0).unwrap(); + store.save(&file).unwrap(); + let before = std::fs::read_to_string(&path).unwrap(); + // A directory where the temporary file must go makes the write fail. + let temp = dir.join(format!(".routines.json.tmp-{}", std::process::id())); + std::fs::create_dir_all(&temp).unwrap(); + file.learn("open mail", steps(), false, 0).unwrap(); + let why = store.save(&file).unwrap_err(); + assert!(why.contains("cannot save routines"), "{why}"); + assert_eq!(std::fs::read_to_string(&path).unwrap(), before); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_stale_temporary_file_from_a_crash_is_overwritten_not_loaded() { + let dir = folder("stale"); + let store = Store::at(dir.join("routines.json")); + let temp = dir.join(format!(".routines.json.tmp-{}", std::process::id())); + std::fs::write(&temp, "{half a document").unwrap(); + assert_eq!( + store.load().unwrap(), + Routines::default(), + "a temp file is never read" + ); + let mut file = Routines::default(); + file.learn("open notes", steps(), false, 0).unwrap(); + store.save(&file).unwrap(); + assert_eq!(store.load().unwrap(), file); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_file_that_is_not_routines_is_refused_and_never_overwritten_by_a_load() { + let dir = folder("bad"); + let path = dir.join("routines.json"); + std::fs::write(&path, "{not json").unwrap(); + let store = Store::at(path.clone()); + let why = store.load().unwrap_err(); + assert!(why.contains("not a routines file"), "{why}"); + assert_eq!(std::fs::read_to_string(&path).unwrap(), "{not json"); + std::fs::write(&path, r#"{"version": 2, "routines": []}"#).unwrap(); + assert!(store.load().unwrap_err().contains("version 2")); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_byte_order_mark_from_a_windows_editor_is_tolerated() { + let dir = folder("bom"); + let path = dir.join("routines.json"); + std::fs::write(&path, "\u{feff}{\"version\": 1, \"routines\": []}").unwrap(); + assert!(Store::at(path).load().unwrap().routines.is_empty()); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn the_default_path_is_the_environment_variable_first() { + // Reads the process environment, so it is the one test that sets it. + let dir = folder("env"); + let wanted = dir.join("mine.json"); + std::env::set_var(ROUTINES_ENV, &wanted); + assert_eq!(Store::default_path(), Some(wanted)); + std::env::remove_var(ROUTINES_ENV); + if let Some(path) = Store::default_path() { + assert!( + path.ends_with("incord-desktop/routines.json") + || path.ends_with("incord-desktop\\routines.json") + ); + } + let _ = std::fs::remove_dir_all(&dir); + } +} diff --git a/crates/desktop/src/routine/watchman.rs b/crates/desktop/src/routine/watchman.rs new file mode 100644 index 0000000..31bcfc3 --- /dev/null +++ b/crates/desktop/src/routine/watchman.rs @@ -0,0 +1,480 @@ +//! Watchman's records, read from the local memory engine. +//! +//! # The endpoint, as it was measured +//! +//! `GET /v1/memory/watchman/record/{id}` on the engine (default +//! `http://127.0.0.1:9180`), read-only by the engine's own design. Checked +//! against a running engine on 2026-11-13. It answers `200` with +//! `content-type: application/json` and a `content-length`, and a record is: +//! +//! ```text +//! { "record_id": "w-<24 hex>", "started_at_ms": , "task": ..., "spans": [ +//! { "span_id": 1, "seq": 1, "kind": "ToolCall", "name": "Bash", +//! "input": { ... }, "output": , +//! "error": , "started_at_ms": , "attrs": {} } ], +//! ... } +//! ``` +//! +//! An id the engine does not hold answers `200` with `{"error": "no record +//! \"...\""}`, so the body, not the status, tells. `output` is NOT always text +//! (a search tool's answer was an array), and `error` is a string that is +//! itself JSON-quoted. A tool result flagged `is_error` becomes the span's +//! `error`; the capture keeps no images and cuts text at 4 KiB. +//! +//! Only `spans` with `kind` `ToolCall` are read, in `seq` order; every other +//! field is ignored, so the engine may add fields without breaking this. +//! +//! # Plain HTTP, to this machine only +//! +//! The crate has no HTTP client and the engine is local, so this speaks +//! `HTTP/1.1` over a [`std::net::TcpStream`], with `Connection: close`. It +//! refuses any host that is not a loopback address: a record becomes steps that +//! are replayed on the operator's desktop, so records may only come from the +//! operator's own machine. The engine's URL is `INCORD_MEMORY_URL` (the variable +//! incode uses), else the default above. + +use std::io::{Read, Write}; +use std::net::{TcpStream, ToSocketAddrs}; +use std::time::Duration; + +use serde_json::Value; + +use super::source::{Call, Record, StepSource}; + +/// The variable that names the engine, the one incode reads. +pub const MEMORY_URL_ENV: &str = "INCORD_MEMORY_URL"; + +/// Where the engine is when nothing says. +pub const DEFAULT_MEMORY_URL: &str = "http://127.0.0.1:9180"; + +/// The most of an answer that is read, in bytes. +const MAX_RESPONSE_BYTES: usize = 4 * 1024 * 1024; + +/// How long to wait to connect. +const CONNECT_TIMEOUT: Duration = Duration::from_secs(3); + +/// How long to wait for each read or write. +const IO_TIMEOUT: Duration = Duration::from_secs(10); + +/// Records read from the local memory engine's Watchman endpoints. +#[derive(Debug, Clone)] +pub struct WatchmanSource { + base: String, +} + +impl WatchmanSource { + /// A source for the engine at `base` (`http://host:port`). + #[must_use] + pub fn new(base: &str) -> Self { + Self { + base: base.trim_end_matches('/').to_string(), + } + } + + /// A source for the engine `INCORD_MEMORY_URL` names, else the default. + #[must_use] + pub fn from_env() -> Self { + let url = std::env::var(MEMORY_URL_ENV) + .ok() + .filter(|u| !u.trim().is_empty()) + .unwrap_or_else(|| DEFAULT_MEMORY_URL.to_string()); + Self::new(url.trim()) + } +} + +impl StepSource for WatchmanSource { + fn record(&self, id: &str) -> Result { + if !valid_id(id) { + return Err(format!( + "{id:?} is not a Watchman record id (letters, digits, `-` and `_`, at most 64)" + )); + } + let (status, body) = get(&self.base, &format!("/v1/memory/watchman/record/{id}"))?; + if status != 200 { + return Err(format!( + "the memory engine answered {status} for record {id}" + )); + } + let text = String::from_utf8(body) + .map_err(|_| format!("the memory engine's answer for record {id} is not text"))?; + parse_record(id, &text) + } +} + +/// Whether `id` can be put in a URL path as it is. +fn valid_id(id: &str) -> bool { + !id.is_empty() + && id.len() <= 64 + && id + .chars() + .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_') +} + +/// A record from the engine's answer for `id`. +/// +/// # Errors +/// +/// The answer is not JSON, is the engine's `{"error": ...}`, names another +/// record, or has no `spans`. +pub fn parse_record(id: &str, text: &str) -> Result { + let doc: Value = serde_json::from_str(text) + .map_err(|e| format!("the memory engine's answer for record {id} is not JSON: {e}"))?; + let Some(spans) = doc.get("spans").and_then(Value::as_array) else { + let why = doc + .get("error") + .and_then(Value::as_str) + .unwrap_or("it has no spans"); + return Err(format!("record {id}: {why}")); + }; + if let Some(named) = doc.get("record_id").and_then(Value::as_str) { + if named != id { + return Err(format!("asked for record {id} and was given {named}")); + } + } + let mut spans: Vec<&Value> = spans + .iter() + .filter(|s| { + s.get("kind") + .and_then(Value::as_str) + .is_none_or(|k| k == "ToolCall") + }) + .collect(); + spans.sort_by_key(|s| s.get("seq").and_then(Value::as_i64).unwrap_or(0)); + Ok(Record { + id: id.to_string(), + started_at_ms: doc + .get("started_at_ms") + .and_then(Value::as_i64) + .unwrap_or(0), + calls: spans.into_iter().filter_map(call_of).collect(), + }) +} + +/// One span as a [`Call`]; `None` for a span with no tool name. +fn call_of(span: &Value) -> Option { + let tool = span.get("name").and_then(Value::as_str)?.to_string(); + Some(Call { + tool, + args: span.get("input").cloned().unwrap_or(Value::Null), + error: span.get("error").and_then(error_text), + output: span.get("output").and_then(output_text), + }) +} + +/// A span's `error` as text: absent for null or empty, and unquoted when the +/// engine stored it JSON-quoted. +fn error_text(value: &Value) -> Option { + let text = match value { + Value::Null => return None, + Value::String(s) => match serde_json::from_str::(s) { + Ok(Value::String(inner)) => inner, + _ => s.clone(), + }, + other => other.to_string(), + }; + (!text.trim().is_empty()).then_some(text) +} + +/// A span's `output` as text, whatever shape the engine kept it in. +fn output_text(value: &Value) -> Option { + match value { + Value::Null => None, + Value::String(s) => Some(s.clone()), + other => Some(other.to_string()), + } +} + +/// `GET path` from `base` (`http://host:port`), returning the status and body. +fn get(base: &str, path: &str) -> Result<(u16, Vec), String> { + let (host, port) = split_base(base)?; + let addr = (host.as_str(), port) + .to_socket_addrs() + .map_err(|e| format!("cannot resolve {host}: {e}"))? + .find(|a| a.ip().is_loopback()) + .ok_or_else(|| { + format!( + "{host} is not a loopback address; records are only read from the memory \ + engine on this machine" + ) + })?; + let mut stream = TcpStream::connect_timeout(&addr, CONNECT_TIMEOUT) + .map_err(|e| format!("the memory engine at {host}:{port} is not answering: {e}"))?; + let _ = stream.set_read_timeout(Some(IO_TIMEOUT)); + let _ = stream.set_write_timeout(Some(IO_TIMEOUT)); + let request = format!( + "GET {path} HTTP/1.1\r\nHost: {host}:{port}\r\nAccept: application/json\r\n\ + Connection: close\r\n\r\n" + ); + stream + .write_all(request.as_bytes()) + .map_err(|e| format!("cannot ask the memory engine: {e}"))?; + let mut raw = Vec::new(); + // One byte over the cap, to tell "exactly the cap" from "more". + stream + .take(MAX_RESPONSE_BYTES as u64 + 1) + .read_to_end(&mut raw) + .map_err(|e| format!("the memory engine stopped answering: {e}"))?; + if raw.len() > MAX_RESPONSE_BYTES { + return Err("the memory engine's answer is over 4 MiB".to_string()); + } + parse_http(&raw) +} + +/// `(host, port)` of an `http://host[:port]` URL, with nothing after it. +fn split_base(base: &str) -> Result<(String, u16), String> { + let rest = base.strip_prefix("http://").ok_or_else(|| { + format!("the memory engine's URL must start with http:// (got {base:?}); it is local") + })?; + let authority = rest.trim_end_matches('/'); + if authority.is_empty() || authority.contains('/') || authority.contains('@') { + return Err(format!("{base:?} is not a plain http://host:port URL")); + } + let (host, port) = match authority.rsplit_once(':') { + Some((h, p)) if !p.is_empty() && p.chars().all(|c| c.is_ascii_digit()) => ( + h, + p.parse::() + .map_err(|_| format!("{p:?} is not a port"))?, + ), + _ => (authority, 80), + }; + let host = host.trim_start_matches('[').trim_end_matches(']'); + if host.is_empty() { + return Err(format!("{base:?} has no host")); + } + Ok((host.to_string(), port)) +} + +/// The status and the body of an HTTP/1.x response read to the end. +fn parse_http(raw: &[u8]) -> Result<(u16, Vec), String> { + let split = raw + .windows(4) + .position(|w| w == b"\r\n\r\n") + .ok_or("the memory engine's answer has no header end")?; + let head = std::str::from_utf8(&raw[..split]).map_err(|_| "the answer's header is not text")?; + let body = &raw[split + 4..]; + let mut lines = head.lines(); + let status: u16 = lines + .next() + .and_then(|l| l.split_whitespace().nth(1)) + .and_then(|s| s.parse().ok()) + .ok_or("the memory engine's answer has no status")?; + let header = |name: &str| { + lines + .clone() + .filter_map(|l| l.split_once(':')) + .find(|(k, _)| k.eq_ignore_ascii_case(name)) + .map(|(_, v)| v.trim().to_ascii_lowercase()) + }; + if header("transfer-encoding").is_some_and(|v| v.contains("chunked")) { + return Ok((status, dechunk(body)?)); + } + let length = header("content-length").and_then(|v| v.parse::().ok()); + let body = match length { + Some(n) if n <= body.len() => &body[..n], + Some(n) => { + return Err(format!( + "the answer ended after {} of {n} bytes", + body.len() + )) + } + None => body, + }; + Ok((status, body.to_vec())) +} + +/// A `Transfer-Encoding: chunked` body, joined. +fn dechunk(mut body: &[u8]) -> Result, String> { + let bad = || "the memory engine's chunked answer is malformed".to_string(); + let mut out = Vec::new(); + loop { + let line_end = body.windows(2).position(|w| w == b"\r\n").ok_or_else(bad)?; + let size_text = std::str::from_utf8(&body[..line_end]).map_err(|_| bad())?; + let size_text = size_text.split(';').next().unwrap_or_default().trim(); + let size = usize::from_str_radix(size_text, 16).map_err(|_| bad())?; + body = &body[line_end + 2..]; + if size == 0 { + return Ok(out); + } + if body.len() < size + 2 { + return Err(bad()); + } + out.extend_from_slice(&body[..size]); + body = &body[size + 2..]; + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::net::TcpListener; + + /// Real answers of `GET /v1/memory/watchman/record/{id}` from a running + /// engine, captured 2026-11-13. The project, user, memory-link and the text + /// of the commands are replaced; every key, type and nesting is as captured. + const SHELL: &str = include_str!("fixtures/record_shell_call.json"); + const FAILED: &str = include_str!("fixtures/record_failed_mcp_call.json"); + const STRUCTURED: &str = include_str!("fixtures/record_structured_output.json"); + + #[test] + fn a_real_record_is_read_into_its_calls() { + let record = parse_record("w-078c536885c8d141ef6c1ec7", SHELL).unwrap(); + assert_eq!(record.id, "w-078c536885c8d141ef6c1ec7"); + assert_eq!(record.started_at_ms, 1_789_518_445_212); + assert_eq!(record.calls.len(), 1); + let call = &record.calls[0]; + assert_eq!(call.tool, "Bash"); + let command = call.args["command"].as_str().expect("the command"); + assert!(command.starts_with("grep -c \"EXAMPLE_FLAG\""), "{command}"); + assert_eq!(call.output.as_deref(), Some("occurrences: 2")); + assert_eq!(call.error, None); + } + + #[test] + fn a_failed_call_carries_its_error_unquoted() { + let record = parse_record("w-cca0d1763ffa35991f50a9b9", FAILED).unwrap(); + let call = &record.calls[0]; + assert_eq!(call.tool, "mcp__example_memory__recall_index"); + let why = call.error.as_deref().expect("the span's error"); + assert!(why.starts_with("unknown scope"), "{why}"); + assert!(!why.starts_with('"'), "the JSON quoting is removed: {why}"); + assert_eq!(call.output, None); + } + + #[test] + fn an_output_that_is_not_text_is_kept_as_its_json() { + let record = parse_record("w-0503383fb5fdf726a39b6f01", STRUCTURED).unwrap(); + let out = record.calls[0].output.as_deref().expect("output"); + assert!(out.contains("tool_reference"), "{out}"); + assert!(serde_json::from_str::(out).unwrap().is_array()); + } + + #[test] + fn spans_are_read_in_seq_order_and_only_tool_calls() { + let doc = r#"{"record_id":"w-1","started_at_ms":9,"spans":[ + {"seq":3,"kind":"ToolCall","name":"key","input":{"keys":["b"]},"output":null,"error":null}, + {"seq":2,"kind":"ModelCall","name":"claude","input":{},"output":null,"error":null}, + {"seq":1,"kind":"ToolCall","name":"key","input":{"keys":["a"]},"output":null,"error":null}]}"#; + let record = parse_record("w-1", doc).unwrap(); + let keys: Vec<_> = record + .calls + .iter() + .map(|c| c.args["keys"][0].clone()) + .collect(); + assert_eq!(keys, [serde_json::json!("a"), serde_json::json!("b")]); + } + + #[test] + fn the_engines_own_error_and_a_mismatch_are_refusals() { + let none = parse_record("w-x", r#"{"error":"no record \"w-x\""}"#).unwrap_err(); + assert!(none.contains("no record"), "{none}"); + let wrong = parse_record("w-2", r#"{"record_id":"w-1","spans":[]}"#).unwrap_err(); + assert!(wrong.contains("w-1"), "{wrong}"); + assert!(parse_record("w-3", "not json") + .unwrap_err() + .contains("not JSON")); + assert!(parse_record("w-3", "{}").unwrap_err().contains("no spans")); + } + + #[test] + fn an_empty_error_string_is_not_a_failure() { + let doc = r#"{"spans":[{"seq":1,"kind":"ToolCall","name":"key","input":{},"error":" "}]}"#; + assert_eq!(parse_record("w-1", doc).unwrap().calls[0].error, None); + } + + #[test] + fn only_a_plain_loopback_http_url_is_spoken_to() { + assert_eq!( + split_base("http://127.0.0.1:9180").unwrap(), + ("127.0.0.1".to_string(), 9180) + ); + assert_eq!( + split_base("http://localhost/").unwrap(), + ("localhost".to_string(), 80) + ); + assert_eq!( + split_base("http://[::1]:9180").unwrap(), + ("::1".to_string(), 9180) + ); + assert!(split_base("https://127.0.0.1:9180").is_err()); + assert!(split_base("http://user@127.0.0.1").is_err()); + assert!(split_base("http://127.0.0.1:9180/v1").is_err()); + // An address literal, so no name is looked up: not this machine, not asked. + let err = get("http://192.0.2.1:9180", "/x").unwrap_err(); + assert!(err.contains("not a loopback"), "{err}"); + } + + #[test] + fn an_id_that_could_escape_the_path_is_refused_before_any_connection() { + let source = WatchmanSource::new("http://127.0.0.1:9"); + let too_long = "a".repeat(65); + for bad in [ + "", + "../x", + "w-1/../../y", + "w 1", + "w-1?x=1", + too_long.as_str(), + ] { + assert!( + source + .record(bad) + .unwrap_err() + .contains("not a Watchman record id"), + "{bad}" + ); + } + } + + #[test] + fn http_answers_are_read_by_length_by_chunks_and_to_the_end() { + let by_length = b"HTTP/1.1 200 OK\r\ncontent-length: 5\r\n\r\nhelloEXTRA"; + assert_eq!(parse_http(by_length).unwrap(), (200, b"hello".to_vec())); + let chunked = b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n5\r\nhello\r\n6;x=1\r\n world\r\n0\r\n\r\n"; + assert_eq!(parse_http(chunked).unwrap(), (200, b"hello world".to_vec())); + let to_end = b"HTTP/1.0 404 Not Found\r\n\r\nnope"; + assert_eq!(parse_http(to_end).unwrap(), (404, b"nope".to_vec())); + assert!(parse_http(b"HTTP/1.1 200 OK\r\ncontent-length: 50\r\n\r\nshort").is_err()); + assert!(parse_http(b"garbage").is_err()); + assert!(dechunk(b"zz\r\nhello").is_err()); + } + + /// The whole path, over a real socket: the request is what the engine + /// expects and the answer is parsed as the engine's real one is. + #[test] + fn a_record_is_fetched_over_a_loopback_socket() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + let served = std::thread::spawn(move || { + let (mut socket, _) = listener.accept().unwrap(); + let mut buf = [0u8; 2048]; + let n = socket.read(&mut buf).unwrap(); + let request = String::from_utf8_lossy(&buf[..n]).to_string(); + let body = SHELL; + let reply = format!( + "HTTP/1.1 200 OK\r\ncontent-type: application/json\r\ncontent-length: {}\r\n\r\n{body}", + body.len() + ); + socket.write_all(reply.as_bytes()).unwrap(); + request + }); + let source = WatchmanSource::new(&format!("http://127.0.0.1:{port}")); + let record = source.record("w-078c536885c8d141ef6c1ec7").unwrap(); + assert_eq!(record.calls[0].tool, "Bash"); + let request = served.join().unwrap(); + assert!( + request.starts_with( + "GET /v1/memory/watchman/record/w-078c536885c8d141ef6c1ec7 HTTP/1.1\r\n" + ), + "{request}" + ); + assert!(request.contains("Connection: close"), "{request}"); + } + + #[test] + fn an_engine_that_is_not_there_is_an_error_naming_it_not_a_hang() { + // Port 9 (discard) is closed on a test machine; the refusal is immediate. + let source = WatchmanSource::new("http://127.0.0.1:9"); + let why = source.record("w-1").unwrap_err(); + assert!(why.contains("not answering"), "{why}"); + } +} diff --git a/crates/desktop/src/routine/words.rs b/crates/desktop/src/routine/words.rs new file mode 100644 index 0000000..4010a47 --- /dev/null +++ b/crates/desktop/src/routine/words.rs @@ -0,0 +1,144 @@ +//! Which routine a phrase asks for: the same words, in any order. +//! +//! The rule is Atticus's, and it is deliberately blunt. A phrase is lowered, +//! cut into letters-and-digits words and stripped of the filler words +//! ([`STOP`]); two phrases are the same request when those word SETS are equal. +//! "open the notes app" and "notes app, open it please" are one routine; +//! "open notes" and "open notes and mail" are two. There is no scoring and no +//! partial match: a routine that runs on a near miss runs the wrong thing on a +//! machine the operator is signed into, and a miss only costs the agent doing +//! the task the long way. +//! +//! Direction words are NOT filler: "turn the volume up" and "turn the volume +//! down" must stay different requests, and so must "on" and "off". + +use std::collections::BTreeSet; + +use super::Routine; + +/// Words that carry no request. Atticus's list less `on`, `in`, `what` and +/// `how`, which can change what a phrase means ("turn it on", "show me how"). +/// `this` and `that` stay here for matching; [`is_deictic`] is what refuses to +/// LEARN them. +pub const STOP: &[&str] = &[ + "a", "an", "the", "to", "of", "and", "or", "for", "my", "me", "i", "you", "it", "is", "be", + "do", "can", "could", "would", "please", "with", "at", "this", "that", +]; + +/// The words of `phrase` that name the request, as a set. +#[must_use] +pub fn word_set(phrase: &str) -> BTreeSet { + phrase + .to_lowercase() + .split(|c: char| !c.is_alphanumeric()) + .filter(|w| !w.is_empty() && !STOP.contains(w)) + .map(str::to_string) + .collect() +} + +/// The phrase's identity: its word set, sorted and joined. Two phrases with the +/// same key are the same request. +#[must_use] +pub fn key(phrase: &str) -> String { + word_set(phrase).into_iter().collect::>().join(" ") +} + +/// Whether the phrase points at what is on screen right now ("close this", +/// "click that"). Such a request means something only at the moment it was +/// said, so a routine learned from it would replay against whatever is on +/// screen the next time. +#[must_use] +pub fn is_deictic(phrase: &str) -> bool { + phrase + .to_lowercase() + .split(|c: char| !c.is_alphanumeric()) + .any(|w| w == "this" || w == "that") +} + +/// The routine `phrase` asks for among `routines`, if any. A phrase with no +/// words of its own (all filler) matches nothing. +#[must_use] +pub fn find<'a>(routines: &'a [Routine], phrase: &str) -> Option<&'a Routine> { + let want = word_set(phrase); + if want.is_empty() { + return None; + } + routines.iter().find(|r| word_set(&r.phrase) == want) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn routine(phrase: &str) -> Routine { + Routine { + phrase: phrase.to_string(), + steps: Vec::new(), + date: String::new(), + last_used_ms: 0, + taught: false, + } + } + + #[test] + fn the_same_words_in_any_order_are_one_request() { + assert_eq!(key("open the notes app"), key("Notes app, open it please!")); + assert_eq!(key("put notes left"), key("left: put NOTES")); + } + + #[test] + fn filler_words_are_ignored_and_the_rest_are_not() { + assert_eq!( + word_set("could you please open the notes") + .into_iter() + .collect::>(), + ["notes", "open"] + ); + assert_ne!(key("open notes"), key("open notes and mail")); + } + + #[test] + fn direction_words_keep_opposite_requests_apart() { + assert_ne!(key("turn the volume up"), key("turn the volume down")); + assert_ne!(key("turn wifi on"), key("turn wifi off")); + assert_ne!(key("turn wifi on"), key("turn wifi")); + } + + #[test] + fn digits_and_other_scripts_are_words() { + assert_eq!(key("open tab 2"), key("tab 2 open")); + assert_ne!(key("open tab 2"), key("open tab 3")); + assert!(!word_set("खोलो नोट्स").is_empty()); + } + + #[test] + fn this_and_that_are_deictic_whatever_the_case_and_punctuation() { + assert!(is_deictic("close this window")); + assert!(is_deictic("Click THAT.")); + assert!(is_deictic("save (this)")); + assert!(!is_deictic("close the window")); + // A word that merely contains it is not the word. + assert!(!is_deictic("open thatcher notes")); + assert!(!is_deictic("open thisisfine")); + } + + #[test] + fn find_matches_by_word_set_and_never_by_a_near_miss() { + let routines = [routine("open notes"), routine("send the weekly report")]; + assert_eq!( + find(&routines, "Notes, open please").map(|r| r.phrase.as_str()), + Some("open notes") + ); + assert!(find(&routines, "open notes and mail").is_none()); + assert!(find(&routines, "open").is_none()); + assert!(find(&routines, "report weekly send").is_some()); + } + + #[test] + fn a_phrase_of_nothing_but_filler_matches_nothing() { + let routines = [routine("")]; + assert!(word_set("please do the").is_empty()); + assert!(find(&routines, "please do the").is_none()); + assert!(find(&routines, "").is_none()); + } +} diff --git a/crates/desktop/src/wait.rs b/crates/desktop/src/wait.rs index 918e082..64b4534 100644 --- a/crates/desktop/src/wait.rs +++ b/crates/desktop/src/wait.rs @@ -43,6 +43,16 @@ pub const DEFAULT_POLL_MS: u64 = 250; /// The fastest and slowest checking interval a call may ask for. pub const MIN_POLL_MS: u64 = 50; pub const MAX_POLL_MS: u64 = 2_000; +/// How long a `watch` waits when the call gives no time: one hour. +pub const DEFAULT_WATCH_MS: u64 = 60 * 60 * 1000; +/// The longest one `watch` may run: 24 hours. Longer is several watches. +pub const MAX_WATCH_MS: u64 = 24 * 60 * 60 * 1000; +/// How often a `watch` looks when the call does not say. +pub const DEFAULT_WATCH_POLL_MS: u64 = 1_000; +/// The fastest and slowest a `watch` may look. A watch is not waiting for a +/// dialog to settle, so it is held to a slower floor than `wait_for`'s. +pub const MIN_WATCH_POLL_MS: u64 = 250; +pub const MAX_WATCH_POLL_MS: u64 = 10_000; /// Bits of the 128 that may differ before the screen counts as changed. pub const DEFAULT_TOLERANCE: u32 = 4; /// Sleeps are cut into slices this long so a stop is noticed within one. @@ -59,26 +69,66 @@ pub struct Budget { pub notes: Vec, } +/// The limits a kind of wait holds a request to. +struct Bounds { + default_timeout: u64, + max_timeout: u64, + default_poll: u64, + min_poll: u64, + max_poll: u64, +} + +/// `wait_for`: a few seconds, polled often. +const WAIT_BOUNDS: Bounds = Bounds { + default_timeout: DEFAULT_TIMEOUT_MS, + max_timeout: MAX_TIMEOUT_MS, + default_poll: DEFAULT_POLL_MS, + min_poll: MIN_POLL_MS, + max_poll: MAX_POLL_MS, +}; + +/// `watch`: hours, polled gently, because nothing is waiting on the answer but +/// the person who asked to be told. +const WATCH_BOUNDS: Bounds = Bounds { + default_timeout: DEFAULT_WATCH_MS, + max_timeout: MAX_WATCH_MS, + default_poll: DEFAULT_WATCH_POLL_MS, + min_poll: MIN_WATCH_POLL_MS, + max_poll: MAX_WATCH_POLL_MS, +}; + impl Budget { /// `timeout_ms` and `poll_ms` as the call gave them (or did not), held to /// the bounds. #[must_use] pub fn new(timeout_ms: Option, poll_ms: Option) -> Budget { + Self::within(&WAIT_BOUNDS, timeout_ms, poll_ms) + } + + /// The same for a `watch`: up to [`MAX_WATCH_MS`], looking every + /// [`DEFAULT_WATCH_POLL_MS`] unless told otherwise. + #[must_use] + pub fn watch(timeout_ms: Option, poll_ms: Option) -> Budget { + Self::within(&WATCH_BOUNDS, timeout_ms, poll_ms) + } + + fn within(bounds: &Bounds, timeout_ms: Option, poll_ms: Option) -> Budget { + let (max, min_poll, max_poll) = (bounds.max_timeout, bounds.min_poll, bounds.max_poll); let mut notes = Vec::new(); - let asked = timeout_ms.unwrap_or(DEFAULT_TIMEOUT_MS); - let timeout_ms = if asked > MAX_TIMEOUT_MS { + let asked = timeout_ms.unwrap_or(bounds.default_timeout); + let timeout_ms = if asked > max { notes.push(format!( - "timeout_ms {asked} is over the cap; waited at most {MAX_TIMEOUT_MS} ms" + "timeout_ms {asked} is over the cap; waited at most {max} ms" )); - MAX_TIMEOUT_MS + max } else { asked }; - let wanted = poll_ms.unwrap_or(DEFAULT_POLL_MS); - let poll = wanted.clamp(MIN_POLL_MS, MAX_POLL_MS); + let wanted = poll_ms.unwrap_or(bounds.default_poll); + let poll = wanted.clamp(min_poll, max_poll); if poll != wanted { notes.push(format!( - "poll_ms {wanted} is outside {MIN_POLL_MS}..={MAX_POLL_MS}; polled every {poll} ms" + "poll_ms {wanted} is outside {min_poll}..={max_poll}; polled every {poll} ms" )); } Budget { @@ -922,6 +972,82 @@ mod tests { assert_eq!(Budget::new(None, Some(1)).notes.len(), 1); } + #[test] + fn a_watch_defaults_to_an_hour_every_second_and_is_held_to_a_day() { + let b = Budget::watch(None, None); + assert_eq!( + (b.timeout_ms, b.poll_ms), + (DEFAULT_WATCH_MS, DEFAULT_WATCH_POLL_MS) + ); + assert!(b.notes.is_empty()); + let long = Budget::watch(Some(MAX_WATCH_MS * 7), None); + assert_eq!(long.timeout_ms, MAX_WATCH_MS); + assert!(long.notes[0].contains("over the cap"), "{:?}", long.notes); + assert_eq!(Budget::watch(Some(MAX_WATCH_MS), None).notes.len(), 0); + assert_eq!(Budget::watch(None, Some(1)).poll_ms, MIN_WATCH_POLL_MS); + assert_eq!( + Budget::watch(None, Some(u64::MAX)).poll_ms, + MAX_WATCH_POLL_MS + ); + // The one-shot wait is untouched by any of it. + assert_eq!(Budget::new(Some(u64::MAX), None).timeout_ms, MAX_TIMEOUT_MS); + } + + #[test] + fn a_long_watch_catches_a_window_that_appears_hours_later() { + let two_hours = 2 * 60 * 60 * 1000; + let mut env = Fake::new(); + env.windows = Box::new(move |t| { + let mut all = vec![window("1", "app", "Doc", true)]; + if t >= two_hours { + all.push(window("2", "mail", "Reminder", false)); + } + all + }); + let condition = Condition::Window { + selector: sel("window~=Reminder"), + appears: true, + }; + let budget = Budget::watch(Some(3 * 60 * 60 * 1000), None); + let outcome = run(&mut env, &condition, &budget, &never).unwrap(); + match outcome { + Outcome::Met { + waited_ms, polls, .. + } => { + assert_eq!(waited_ms, two_hours); + // Once a second, not every 250 ms: 7201 looks, not 28801. + assert!((7_000..7_400).contains(&polls), "{polls} looks"); + } + other => panic!("{other:?}"), + } + assert!( + env.sleeps.iter().all(|ms| *ms <= SLICE_MS), + "a stop is still noticed within one slice" + ); + } + + #[test] + fn a_stop_ends_a_long_watch_at_once() { + let mut env = Fake::new(); + let condition = Condition::Window { + selector: sel("window~=Never"), + appears: true, + }; + let polled = Cell::new(0u32); + let stopped = || { + polled.set(polled.get() + 1); + polled.get() > 50 + }; + let budget = Budget::watch(Some(MAX_WATCH_MS), None); + let outcome = run(&mut env, &condition, &budget, &stopped).unwrap(); + assert!(matches!(outcome, Outcome::Stopped { .. }), "{outcome:?}"); + assert!( + env.now < 60_000, + "stopped within a minute of virtual time: {}", + env.now + ); + } + // ---- the loop ---- /// Checks at 0, 250, 500, 750 ...; true from `t` on. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index e6fe916..538fbbe 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1117,8 +1117,10 @@ for a patch). The ACP kind is `acp_agent::present::tool_kind`. | `WebFetch` | observe | fetch | 20 s, 512 KiB read, 20 000 chars shown; rendered: 20 s + 10 s kill grace, 2 MiB read | private addresses refused by name and by resolution; carries no credential; an HTML answer becomes markdown through Read's HTML conversion (readability when the page reads as an article, the whole page less nav/scripts otherwise), anything else is returned as sent. Where the headless browser is found: `render_js` runs the page's script first, and a page that comes back as an app shell (script, under 200 characters of text) is rendered on its own; `render_js` is not advertised where there is no browser | | `Screenshot` | observe | other | `PAGE_DEADLINE` 60 s, load 30 s, settle 3 s; viewport 200–3840 x 200–2160 CSS px; the image bounds below | OFFERED ONLY when the headless browser is found. A public `url` (WebFetch's address rules) or an HTML/SVG `path` in the workspace; `width`/`height`, `scale` 1 or 2, `full_page` (clipped at `MAX_IMAGE_EDGE` px tall, said so). The answer is an image block. See "The headless browser" below | | `RenderPdf` | change | edit | `PDF_DEADLINE` 120 s, whole-tree kill | OFFERED ONLY when a printer is found (`pdf_renderer`, below). A public `url` or a workspace HTML/SVG `path`, to `output` (must end `.pdf`); the guards and the checkpoint see `output`; the engine's `code-edit` is told it. Printed in a private temp profile and copied into place only when it exists | -| `screen_info`, `screenshot`, `ui_tree`, `cursor_position`, `wait_for` | observe | other | a call 120 s; the image bounds | OFFERED ONLY when the desktop is switched on (`INCODE_DESKTOP`). Executed by the desktop child — see "The desktop" below. `screenshot` answers with an image block | -| `click_element`, `type_text`, `paste_text`, `key`, `scroll`, `drag`, `move_mouse`, `click` | run | other | a call 120 s | the same; refused read-only, asked under prompt, refused when the engine cannot decide | +| `screen_info`, `screenshot`, `ui_tree`, `cursor_position`, `wait_for`, `routine_list`, `routine_match` | observe | other | a call 120 s; the image bounds | OFFERED ONLY when the desktop is switched on (`INCODE_DESKTOP`). Executed by the desktop child — see "The desktop" below. `screenshot` answers with an image block | +| `watch` | observe | other | up to 24 h plus a minute | the same; holds the call open until a control, window or screen change is seen, then the operator's post-tool hook for `watch` is told. See `docs/COMPUTER_USE.md`, "Routines and `watch`" | +| `click_element`, `type_text`, `paste_text`, `key`, `scroll`, `drag`, `move_mouse`, `click`, `routine_save`, `routine_forget` | run | other | a call 120 s | the same; refused read-only, asked under prompt, refused when the engine cannot decide | +| `routine_run` | run | other | 10 min | the same; replays a saved routine's steps, each through the child's own gates | | `WebSearch` | observe | fetch | 15 s, 512 KiB, 8 results | provider from `admin.json` `web_search` — contract in `web::search_with` | | `Skill` | observe | other | — | | | `Agent` | run | other | depth 2 | on the model endpoint when one is configured; otherwise on the agent CLI this toolbox's model runs in (`Toolbox::run_subagents_on`, set by the claude backend): a second headless run of it as a model only, its own toolbox one level down, the session's mode, guards, engine, person to ask, cancel and memory session id, its role routing its model (the operator's candidates, then the CLI's own ladder). Neither: refused. A worker — see "What a tool and a worker do while they run" | diff --git a/docs/COMPUTER_USE.md b/docs/COMPUTER_USE.md index 4d19cc3..bc27b6d 100644 --- a/docs/COMPUTER_USE.md +++ b/docs/COMPUTER_USE.md @@ -34,6 +34,10 @@ > computer-use plan; the first two things a flow run later without a model needs). Both are > described under "Waiting and selectors" below. > +> **Fourth round, 2026-11-13: routines and `watch`** (learned shortcuts made from Watchman's +> records, replayed with no model; "watch for me" as one long `wait_for`). Described under +> "Routines and `watch`" below. +> > **The invariant in 4c, as built:** one coordinate space at the tool surface — physical > pixels on Windows and X11, **points** on macOS (the space CGEvent and AXUIElement both > use), top-left origin. Each image carries its transform back to that space; on a @@ -359,6 +363,145 @@ timing and the conditions, over an `Env` trait; tested with a scripted desktop a clock), `fingerprint.rs`, `uia/resolve.rs` (resolving against the live tree), `mcp/wait_tool.rs` (the live `Env`). +## Routines and `watch` (fourth round, 2026-11-13) + +What the first two rounds were for: a flow done once can be done again with no model. The +design follows the owner's decisions of 2026-11-13, which supersede the trace format in the +research plan: **no trace format and no trace store**. Watchman already records every tool +call, so a routine is built from Watchman's records; and the rules are Atticus's learned +shortcuts (`atticus.py`, MIT), kept as simple as they are there. + +### What a routine is + +Per routine: the cleaned step list (`tool` + `args`, the same names and arguments the live +tools take), the trigger phrase, the date learned, the last-used time and a user-taught flag. +All in one JSON file (below). Code: `crates/desktop/src/routine/` (`words`, `learn`, `store`, +`source`, `watchman`, `replay`) and `mcp/routine_tool.rs`. + +| Tool | Tier | What it does | +|---|---|---| +| `routine_save` | acting | Learn a routine under `phrase` from `records` (Watchman record ids) or from `steps`. Saving under a phrase with the same words replaces the routine: that is how a failed one is redone. | +| `routine_run` | acting | Replay the routine a phrase matches, on the real desktop, attended. | +| `routine_forget` | acting | Delete one. | +| `routine_list` | observing | Every live routine: phrase, step count, dates, taught, days left. | +| `routine_match` | observing | The routine a phrase would run, with its steps (typed text by length only), or that none matches. Runs nothing. | +| `watch` | observing | Below. | + +`routine_save` and `routine_forget` are acting although they touch a file and not the screen, +because the file is what a later `routine_run` does to the machine. `routine_list` and +`routine_match` only read it. `permission::{OBSERVING, ACTING}`, incode's `for_desktop` and +the toolbox's tables all carry the five, and the parity test in `src/desktop.rs` covers them +(23 tools). + +### Matching + +A phrase is lowered, cut into letters-and-digits words and stripped of filler words (`a an +the to of and or for my me i you it is be do can could would please with at this that`); +two phrases are the same request when the word SETS are equal, in any order. No scoring, no +partial match: a routine that runs on a near miss runs the wrong thing on a machine the +operator is signed into. Direction words (`up`, `down`, `on`, `off`) are not filler. + +### What is refused, and what is cleaned + +Nothing is learned when: the phrase says "this" or "that"; any call failed (a tool error, or a +`wait_for` that timed out or was stopped); or any kept step cannot be replayed on another day: +an `eN` id (valid for one `ui_tree` only), a raw coordinate (`click`, `move_mouse`, `drag`), a +window handle (`window`, `focus_window`), a `wait_for` with a `baseline`, a missing or +unparseable `selector`, or another routine tool. The whole routine is refused, because dropping +the step would silently skip part of the task. + +**Every acting step must say where** (from the live test of 2026-11-13, in which a step with +no window typed into whatever was focused). A selector on `click_element`, `set_value`, +`element_action` or an element `wait_for` must carry an `app` or `window` part. `type_text`, +`paste_text` and `key` take no selector, so a step names its window with `in_window`, a +window-only selector (`app=notepad;window~=Untitled`). A recording has none, so the learner +gives each such step the window part of the nearest step before it and says so in the reply +(`keyboard_steps_given_the_window_before_them`); with no step before it the routine is +refused. `scroll` is never replayable: it takes no selector and scrolls under the pointer +(use `element_action scroll_into_view`, or Page Down then `wait_for screen_changes`). + +`ui_tree`, `screenshot`, `list_windows`, +`screen_info` and `cursor_position` are dropped (a failed one does not fail the run); +`wait_for` is kept. `screenshot_after` is removed from kept steps. Calls that are not desktop +tools (a shell command) are left out and **reported** in the reply. The same per-step check +runs again before a replay, so a hand-edited file cannot get past it. + +### Where steps come from + +A small trait, `routine::source::StepSource`: `WatchmanSource` for the engine, +`MemorySource` for tests and for a caller with its own recorder, `NoSource` when there is +none. The crate works with no engine at all: `routine_save` with `steps`, and every other +tool, never ask for it. `WatchmanSource` reads one endpoint of the local memory engine +(default `http://127.0.0.1:9180`, or `INCORD_MEMORY_URL`; loopback hosts only, because a +record becomes steps replayed on this desktop): + +`GET /v1/memory/watchman/record/{id}` -> `{record_id, started_at_ms, spans:[{seq, kind, name, +input, output, error}]}`. Only `ToolCall` spans are read, in `seq` order, and several records +are joined by `started_at_ms`. `error` set means the call failed; `output` may be text, null +or JSON. A desktop tool is recognised by its plain name or as `mcp____` with a +server name containing `desktop` or `incu`. Checked against a running engine on 2026-11-13; +the test fixtures are real answers with the project, user and text replaced. The engine had +no record of a desktop call at that date, so the desktop-call tests build their records from +that shape and the span fields above. + +### Replay + +The same tool functions in order, each through `Server::gated_call`: the path `tools/call` +itself takes, so each step gets the stop file, the tier, the operator's pre-tool hook (with +the real arguments), the blocked-application guards, the action-rate cap, the post-tool hook +and its own journal line. A replay is not a way around any live guard. It runs attended on the +real desktop (Windows and macOS have no isolated one), waits the desktop's settle time after a +step that changed the screen, and **stops at the first failure**, saying which step and why. A +`wait_for` that answers `timeout` is a failed step. + +Three rules from the live test: **links and buttons are invoked**: a recorded left click on a +control whose role is `link` or `button` runs first as `element_action invoke` (works for a +control scrolled out of view, moves no mouse, takes no focus) and only if that is refused as +the recorded `click_element`; both go through the gates and both are journalled. A keyboard +step's `in_window` is looked up **now** and passed as the tool's `window`, so the tool +refuses unless that window really has the focus (a window that is not there fails the step +before any key is sent). And the replay **stops when a `set_value` or `element_action` +answers "unverified" and the next step acts on the screen** (a click or typing; a `wait_for` +after it is allowed, because it checks the screen itself). A click always says "unverified" +and does not stop the replay: the next step's selector, or a `wait_for`, is its check. A +consequence: filling a web field with `set_value` in a browser (whose read-back is +unreliable) stops a replay at that step. **No repair**: the reply tells the caller +to do the whole task again and `routine_save` it under the same phrase, which replaces the +routine. A routine that ran to the end has its last-used time reset. + +### Storage + +One file, `routines.json`, written atomically (a temporary file beside it, flushed, renamed +over it; mode 0600 on Unix). A file that is not a routines file is refused and left alone, +never read as empty. Path: `routines` in the config (`--config`), else `$INCU_ROUTINES`, else +`%APPDATA%\incord-desktop\routines.json` (Windows), `~/Library/Application Support/ +incord-desktop/routines.json` (macOS), `$XDG_DATA_HOME` or `~/.local/share` then +`incord-desktop/routines.json` (Linux). A routine unused for 30 days expires (it is hidden at +once and removed on the next write) unless it was taught. Up to 200 routines of up to 100 +steps. **Typed text is stored literally**; do not save a routine that types a secret. + +### `watch` + +"Watch for me" is one `wait_for` with a long deadline, as a tool of its own. A separate tool +rather than an option because `wait_for`'s 60 s cap and 50 ms to 2 s polling are properties +its tests pin for a call that returns in moments, while a watch is the opposite promise: it +**blocks** the call for up to 24 hours (default 1 hour), polled every 1 s (250 ms to 10 s). +The look is the same code (`wait::run`, new `Budget::watch`). The stop file ends it within +a second. When it ends, the operator's **post-tool hook for `watch`** is run with the reply +(`result`, what it found, `message`) as `tool_response`: that hook is the notification or +the follow-up (for example `osascript -e 'display notification ...'`, `notify-send`, or a +command that starts a session). Only for `watch` is the answer passed to the hook; every +other tool's hook still gets none. No model runs and nothing polls it. A line `started: ...` +is journalled when a watch begins. Through incode's toolbox the call bound for `watch` is +24 h plus a minute and for `routine_run` ten minutes (otherwise 120 s). If the client goes +away during a watch, the server notices when the watch ends or the stop file is written. + +### Not built + +Voice, the browser, scheduling or cron, chats and step repair: Watchman has the first four, +and whole-task redo replaces the last until it proves too slow. Learned phrases do not swap an +app name (Atticus does); "close this" style phrases are refused, not interpreted. + ## 4. Coordinates — the section that decides whether this works This is where implementations fail, and they fail *silently*: the click succeeds, it @@ -597,9 +740,9 @@ argv (`incu` also takes `--tier observe|act`). | Mode | The desktop it gets | |---|---| -| `read-only` | `screen_info`, `cursor_position`, `screenshot`, `ui_tree`, `list_windows`, `wait_for`. Nothing that moves. | -| `workspace-write` (default) | all seventeen | -| `danger-full-access`, `allow` | all seventeen | +| `read-only` | `screen_info`, `cursor_position`, `screenshot`, `ui_tree`, `list_windows`, `wait_for`, `watch`, `routine_list`, `routine_match`. Nothing that moves. | +| `workspace-write` (default) | all twenty-three (adds `routine_save`, `routine_run`, `routine_forget`) | +| `danger-full-access`, `allow` | all twenty-three | | `prompt` | nothing. stdin and stdout are the protocol, so there is no channel to ask a human on, and `authorize` with no prompter denies. | The acting tools require `WorkspaceWrite` rather than `DangerFullAccess`, even diff --git a/src/desktop.rs b/src/desktop.rs index c7abe15..de1a0d6 100644 --- a/src/desktop.rs +++ b/src/desktop.rs @@ -249,6 +249,9 @@ pub fn serve( limits: admin.desktop_limits, display: admin.desktop_display, hooks, + // incode's `admin.json` has no setting for it: `$INCU_ROUTINES`, else + // the crate's per-user data folder, the same file `incu` uses. + routines: None, name: "incode-desktop", }) } @@ -329,7 +332,7 @@ mod tests { /// **The crate's tier and incode's policy must say the same thing.** The /// server now runs the tier; `PermissionPolicy::for_desktop` stays as incode's /// statement of the same table, and this is what keeps the two from - /// drifting: every mode, every one of the 17 tools, the same verdict and the + /// drifting: every mode, every one of the 23 tools, the same verdict and the /// same words in a refusal. #[test] fn the_crates_tier_agrees_with_the_desktop_permission_policy() { @@ -340,7 +343,7 @@ mod tests { .iter() .filter_map(|t| t["name"].as_str().map(str::to_string)) .collect(); - assert_eq!(offered.len(), 17, "{offered:?}"); + assert_eq!(offered.len(), 23, "{offered:?}"); for name in &offered { assert!( permission::OBSERVING.contains(&name.as_str()) @@ -348,8 +351,15 @@ mod tests { "{name} is offered by the desktop server but has no tier" ); } - // `wait_for` only looks: a read-only session may use it. - assert!(permission::OBSERVING.contains(&"wait_for")); + // `wait_for` and `watch` only look, and `routine_list` and + // `routine_match` only read the routines file: a read-only session may + // use them. Saving, forgetting or running a routine acts. + for looking in ["wait_for", "watch", "routine_list", "routine_match"] { + assert!(permission::OBSERVING.contains(&looking), "{looking}"); + } + for acting in ["routine_save", "routine_run", "routine_forget"] { + assert!(permission::ACTING.contains(&acting), "{acting}"); + } let modes = [ PermissionMode::ReadOnly, PermissionMode::WorkspaceWrite, diff --git a/src/permissions/mod.rs b/src/permissions/mod.rs index 5366269..6e23b75 100644 --- a/src/permissions/mod.rs +++ b/src/permissions/mod.rs @@ -196,8 +196,9 @@ impl PermissionPolicy { self } - /// The requirements for the seventeen tools of the desktop MCP server: six - /// that observe and eleven that move something. + /// The requirements for the twenty-three tools of the desktop MCP server: nine + /// that observe and fourteen that move something or change what a replay + /// will move. /// /// The server is the `incord-desktop` crate now, and it runs its own /// `Tier` (`crates/desktop/src/permission.rs`), which it cannot take from @@ -220,8 +221,8 @@ impl PermissionPolicy { /// /// # The split, and why it is only two rungs /// - /// The six observing tools need `ReadOnly`; the eleven that move something - /// need `WorkspaceWrite`. Not `DangerFullAccess`, even though a click + /// The nine observing tools need `ReadOnly`; the fourteen that act need + /// `WorkspaceWrite`. Not `DangerFullAccess`, even though a click /// reaches far outside any workspace — because `workspace-write` is the /// default mode, and putting the desktop above it would mean the feature /// ships switched off and the flag becomes the thing people turn off. @@ -248,8 +249,18 @@ impl PermissionPolicy { // Waits for a control, a window or a change of the screen: it // looks and changes nothing. .with_tool_requirement("wait_for", PermissionMode::ReadOnly) + // The same look held for hours; it changes nothing either. + .with_tool_requirement("watch", PermissionMode::ReadOnly) + // Read the routines file; neither the screen nor the file changes. + .with_tool_requirement("routine_list", PermissionMode::ReadOnly) + .with_tool_requirement("routine_match", PermissionMode::ReadOnly) // Acting. Each one changes the state of a machine the operator is - // signed into. + // signed into. A routine is what a replay later does to it: running + // one moves the mouse and keyboard, and saving or forgetting one + // changes what a replay will do. + .with_tool_requirement("routine_save", PermissionMode::WorkspaceWrite) + .with_tool_requirement("routine_run", PermissionMode::WorkspaceWrite) + .with_tool_requirement("routine_forget", PermissionMode::WorkspaceWrite) .with_tool_requirement("move_mouse", PermissionMode::WorkspaceWrite) .with_tool_requirement("click", PermissionMode::WorkspaceWrite) .with_tool_requirement("click_element", PermissionMode::WorkspaceWrite) diff --git a/src/toolbox/desktop.rs b/src/toolbox/desktop.rs index a15c9b5..c488f7f 100644 --- a/src/toolbox/desktop.rs +++ b/src/toolbox/desktop.rs @@ -43,6 +43,11 @@ pub(super) const OBSERVING: &[&str] = &[ // Polls up to its own cap of 60 s (`incord_desktop::wait`), well inside // `CALL_BOUND`. "wait_for", + // Holds the call open for up to a day: it has its own bound + // ([`call_bound`]), not `CALL_BOUND`. + "watch", + "routine_list", + "routine_match", ]; /// The desktop tools that act on the machine. pub(super) const ACTING: &[&str] = &[ @@ -57,10 +62,31 @@ pub(super) const ACTING: &[&str] = &[ "drag", "move_mouse", "click", + "routine_save", + // Replays every step of a routine: it has its own bound ([`call_bound`]). + "routine_run", + "routine_forget", ]; /// How long one desktop call may take. A UI tree of a large window is the /// slowest thing it does. const CALL_BOUND: Duration = Duration::from_secs(120); +/// How long a replayed routine may take: up to a hundred steps, each of which +/// can wait for the screen, and a pause after each. +const ROUTINE_BOUND: Duration = Duration::from_secs(10 * 60); + +/// How long the call to `name` may take. A `watch` is bounded by its own +/// ceiling plus a minute, so the bound is never what ends a watch that is still +/// within its time; a `routine_run` by [`ROUTINE_BOUND`]; the rest by +/// [`CALL_BOUND`]. +fn call_bound(name: &str) -> Duration { + match name { + "watch" => { + Duration::from_millis(incord_desktop::wait::MAX_WATCH_MS) + Duration::from_secs(60) + } + "routine_run" => ROUTINE_BOUND, + _ => CALL_BOUND, + } +} /// How long the child has to answer `initialize`. const START_BOUND: Duration = Duration::from_secs(20); /// What the desktop child is called in what it causes to be said. @@ -142,7 +168,7 @@ impl DesktopClient { /// Run one desktop tool. A tool's own refusal is an `Err` with its words. pub(super) fn call(&mut self, name: &str, args: &Value) -> Result { - self.client.call(name, args, CALL_BOUND, &|| false) + self.client.call(name, args, call_bound(name), &|| false) } } From c05bbb26eef5ba6b61ece0b3185fe072373b0e4f Mon Sep 17 00:00:00 2001 From: blockmandev <194144408+blockmandev@users.noreply.github.com> Date: Fri, 13 Nov 2026 22:01:04 -0600 Subject: [PATCH 2/6] desktop: routines review fixes: pictures, secrets as parameters, failure rules, locking Required behaviour that was missing - A picture before and after every replayed step, with incu's own capture under screenshot's guard (the step's window, else the screen), JPEG, in last-replay/ beside routines.json. Last run only, held to 48 MiB by dropping the oldest, so the failed step's pictures survive; the failure names them by path. Never sent to OCR. - Nothing is learned when ANY span of the records errored (a shell call too, not only a desktop call) or the record's own outcome says it failed. - "Kept" (never expires) is no longer a tool argument; passing `taught` is refused. The person sets it with `incu routine keep|unkeep|list`. Bugs - A recorded button press no longer stops a replay: only a value-setting step (set_value, element_action increment/decrement) answering "unverified" or "tried to set" does. - The recorded click runs after element_action invoke only when the invoke was refused before acting (the control does not offer it), never after an error that may have come after the control acted. - Text typed into a password or other secure field (by name, by automation id, or by the desktop's own report when saving), and text a recording kept as a placeholder or digest, is stored as a parameter {{name}}; routine_run needs params for each and refuses before step 1. Values are never written to the file or journal and are scrubbed from replay messages. Save, list and match show every step's full typed text and selector. - save refuses a file over the size load accepts, before writing; caps lowered (100 routines, 4 KiB a step, 24 KiB a routine) and a test writes the largest file they allow. - Changes take a lock file (exclusive create, 30 s stale, 10 s wait): two servers keep each other's saves. - A chunk size of ffffffffffffffff is a malformed answer, not a panic; a 15 s total deadline per request; a span with no kind is not a tool call; the Host header brackets IPv6. - Matching keeps any order but not across to/from/into/onto/over/under/before/ after: "copy backup files to desktop" is not "copy desktop files to backup". Minor - Only incu's own server names (desktop, incu, incode-desktop, incode_desktop) are taken for its tools. - A watch ends when the client disconnects (stdin is read on a thread that raises a flag at EOF), looks at the screen every 5 s, and is documented to hold the server while it runs. --- README.md | 47 +- crates/desktop/src/cli.rs | 243 ++++++ crates/desktop/src/mcp.rs | 20 + crates/desktop/src/mcp/routine_definitions.rs | 88 ++- crates/desktop/src/mcp/routine_tests.rs | 569 +++++++++++++- crates/desktop/src/mcp/routine_tool.rs | 342 ++++++++- crates/desktop/src/mcp/tools.rs | 33 +- crates/desktop/src/mcp/transport.rs | 84 +- crates/desktop/src/mcp/wait_tool.rs | 48 +- crates/desktop/src/routine/learn.rs | 215 +++++- crates/desktop/src/routine/mod.rs | 64 +- crates/desktop/src/routine/params.rs | 715 ++++++++++++++++++ crates/desktop/src/routine/pictures.rs | 236 ++++++ crates/desktop/src/routine/replay.rs | 238 +++++- crates/desktop/src/routine/source.rs | 33 +- crates/desktop/src/routine/store.rs | 376 ++++++++- crates/desktop/src/routine/watchman.rs | 285 ++++++- crates/desktop/src/routine/words.rs | 72 +- crates/desktop/src/wait.rs | 20 + docs/COMPUTER_USE.md | 140 +++- 20 files changed, 3601 insertions(+), 267 deletions(-) create mode 100644 crates/desktop/src/routine/params.rs create mode 100644 crates/desktop/src/routine/pictures.rs diff --git a/README.md b/README.md index 3b1066f..3937fc1 100644 --- a/README.md +++ b/README.md @@ -1134,8 +1134,9 @@ list; the engine is optional. - `ui_tree`, `screenshot` and the other looking calls are dropped; `wait_for` is kept. -- **Nothing is kept** when the phrase says "this" or "that", when a call failed - or a `wait_for` timed out, or when any step uses an `eN` id, a pixel +- **Nothing is kept** when the phrase says "this" or "that", when ANY call or + span in the records failed (a shell command too, not only a desktop call) or + the record says its run failed, when a `wait_for` timed out, or when any step uses an `eN` id, a pixel coordinate (`click`, `move_mouse`, `drag`) or a window handle (`window`, `focus_window`). Only selector-based steps replay, and each must say where: a selector needs an `app=`/`window=` part, and a typing or key step carries @@ -1143,27 +1144,41 @@ list; the engine is optional. it; the replay still refuses to type unless that window has the focus). `scroll` is never replayable. A replay invokes links and buttons through `element_action invoke` (no mouse, no focus) and falls back to the recorded - click, and it stops when a `set_value` or `element_action` answers - "unverified" and the next step acts on it. -- A phrase matches by its **words in any order, ignoring filler words**; a near - miss runs nothing. `routine_match` shows what a phrase would run. + click only when the invoke was refused before it did anything (the control + does not offer it), and it stops when a `set_value` (or an + increment/decrement) answers "unverified" and the next step acts on it; a + button press always says "unverified" and never stops it. +- A phrase matches by its **words in any order, ignoring filler words**, except + that the words on each side of `to`, `from`, `into`, `onto`, `over`, `under`, + `before` or `after` stay on that side ("copy a to b" is not "copy b to a"); a + near miss runs nothing. `routine_match` shows what a phrase would run. - A replay runs every step through the same checks as a live call (blocked apps, hooks, stop file, journal) and **stops at the first failure**, saying which - step and why. There is no repair: do the task yourself and `routine_save` it - under the same phrase to replace the routine. -- A routine unused for 30 days is forgotten, unless saved with `taught: true`. + step and why, with the paths of that step's before and after pictures (one of + each is taken for every step, of the last run only, beside `routines.json`; + never sent to OCR). There is no repair: do the task yourself and + `routine_save` it under the same phrase to replace the routine. +- A routine unused for 30 days is forgotten. Only **you** can keep one for good: + `incu routine keep ` (and `unkeep`, `list`) at a terminal; no tool has + an argument for it. - The file is `routines.json` in the per-user data folder (`%APPDATA%\incord-desktop`, `~/Library/Application Support/incord-desktop`, or `$XDG_DATA_HOME`/`~/.local/share/incord-desktop`), overridden by - `INCU_ROUTINES` or `routines` in the config. Typed text is stored in it as - typed: do not save a routine that types a secret. + `INCU_ROUTINES` or `routines` in the config. It holds typed text as typed, and + `routine_save`, `routine_list` and `routine_match` show every step's full + text so you can read what a routine will type. **Text typed into a password or + other secure field, and text a recording kept only as a placeholder, is never + stored**: that step is a parameter (`{{password}}`) and `routine_run` refuses + to start unless it is given `params`. Changes to the file take a lock, so two + servers do not lose each other's saves. `watch` is `wait_for` with a long deadline (default 1 hour, at most 24, looked -at every second): it blocks until the condition holds, then runs your -post-tool hook for the tool `watch` with the reply (`result`, what it found, the -`message` you passed) as `tool_response`. Put the notification in that hook, for -example `osascript -e 'display notification "…"'` on macOS or `notify-send` -on Linux. The stop file ends a watch within a second. +at every second, every five for `screen_changes`): it blocks until the condition +holds, then runs your post-tool hook for the tool `watch` with the reply +(`result`, what it found, the `message` you passed) as `tool_response`. Put the +notification in that hook, for example `osascript -e 'display notification "…"'` +on macOS or `notify-send` on Linux. The stop file, or the client going away, ends +a watch within a second; while it runs the server answers nothing else. ### What it will not do diff --git a/crates/desktop/src/cli.rs b/crates/desktop/src/cli.rs index 60b04f3..640cee8 100644 --- a/crates/desktop/src/cli.rs +++ b/crates/desktop/src/cli.rs @@ -42,6 +42,9 @@ incu - computer use for an agent, as an MCP server over stdio USAGE: incu serve [OPTIONS] speak MCP on stdin/stdout (the default command) incu stop [OPTIONS] halt the desktop a running server drives + incu routine list show every learned routine, each step in full + incu routine keep keep a routine from expiring (only you can) + incu routine unkeep let it expire again, thirty days from now incu --help | --version OPTIONS for serve: @@ -65,6 +68,11 @@ ENVIRONMENT for serve: Watchman records from (default http://127.0.0.1:9180; loopback only) +OPTIONS for routine: + --file the routines file (else the config's, else the + one serve uses) + --config the config to find it in + OPTIONS for stop: --journal stop the server journalling to PATH --workspace stop the server for DIR's .incode/ journal @@ -81,6 +89,33 @@ pub enum Command { Serve(ServeArgs), /// Ask a running server to stop. Stop(StopArgs), + /// List, keep or unkeep learned routines: what only the person does. + Routine(RoutineArgs), +} + +/// What `incu routine` does. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum RoutineAction { + /// Show every routine, with every step in full. + List, + /// Keep a routine from expiring. + Keep, + /// Let a kept routine expire again (thirty days from now). + Unkeep, +} + +/// The arguments of `incu routine`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct RoutineArgs { + /// What to do. + pub action: RoutineAction, + /// The phrase naming the routine, for keep and unkeep. Matched as a request + /// is: by its words, in any order, filler ignored. + pub phrase: String, + /// The routines file, when named; else the config's, else the default. + pub file: Option, + /// The config file named on the command line, if any. + pub config: Option, } /// The flags of `incu serve`. @@ -128,6 +163,10 @@ pub fn parse>(args: I) -> Result args.remove(0); parse_stop(args) } + Some("routine") => { + args.remove(0); + parse_routine(args) + } Some("-h" | "--help" | "help") => Ok(Command::Print(USAGE.to_string())), Some("--version" | "-V") => Ok(Command::Print(format!( "incu {}", @@ -183,6 +222,49 @@ fn parse_serve(args: Vec) -> Result { })) } +fn parse_routine(args: Vec) -> Result { + let mut it = args.into_iter(); + let action = match it.next().as_deref() { + Some("list") => RoutineAction::List, + Some("keep") => RoutineAction::Keep, + Some("unkeep") => RoutineAction::Unkeep, + Some("-h" | "--help") => return Ok(Command::Print(USAGE.to_string())), + Some(other) => { + return Err(format!( + "unknown routine command `{other}` (list, keep or unkeep)\n\n{USAGE}" + )) + } + None => return Err(format!("routine needs list, keep or unkeep\n\n{USAGE}")), + }; + let mut words = Vec::new(); + let mut file = None; + let mut config = None; + while let Some(arg) = it.next() { + match arg.as_str() { + "-h" | "--help" => return Ok(Command::Print(USAGE.to_string())), + "--file" => file = Some(PathBuf::from(value(&arg, &mut it)?)), + "--config" => config = Some(PathBuf::from(value(&arg, &mut it)?)), + flag if flag.starts_with('-') => { + return Err(format!("unknown option `{flag}` for routine\n\n{USAGE}")) + } + _ => words.push(arg), + } + } + let phrase = words.join(" "); + match (action, phrase.trim().is_empty()) { + (RoutineAction::List, false) => Err("routine list takes no phrase".to_string()), + (RoutineAction::Keep | RoutineAction::Unkeep, true) => { + Err("name the routine: incu routine keep ".to_string()) + } + _ => Ok(Command::Routine(RoutineArgs { + action, + phrase, + file, + config, + })), + } +} + fn parse_stop(args: Vec) -> Result { let mut journal = None; let mut workspace = None; @@ -354,6 +436,87 @@ pub fn serve(options: ServeOptions) -> Result<(), String> { crate::mcp::run_stdio(server).map_err(|e| e.to_string()) } +/// The routines file `incu routine` works on: `--file`, else the config's +/// `routines`, else `$INCU_ROUTINES`, else the per-user data folder: the file +/// `incu serve` uses. +/// +/// # Errors +/// +/// The config cannot be read, or there is no place to keep routines. +pub fn routine_store( + file: Option, + config: Option, +) -> Result { + use crate::routine::store::Store; + let path = match file { + Some(path) => Some(path), + None => { + let config = match &config { + Some(path) => Config::load(path)?, + None => Config::load_default()?, + }; + config.routines.or_else(Store::default_path) + } + }; + path.map(Store::at) + .ok_or_else(|| "no place to keep routines: set INCU_ROUTINES or pass --file".to_string()) +} + +/// `incu routine`: what only the person does to the routines an agent saved. +/// +/// **Why this is a command and not a tool.** A routine a person has kept never +/// expires. If an agent could say "keep this", it could exempt its own saves from +/// the thirty days that exist so that nothing it learned lives on unreviewed. So +/// keeping is done here, at a terminal, by the person, and no tool has an +/// argument for it. +/// +/// # Errors +/// +/// The file cannot be read or written, or no routine matches the phrase. +pub fn run_routine(args: RoutineArgs) -> Result { + let store = routine_store(args.file, args.config)?; + let now = crate::routine::now_ms(); + match args.action { + RoutineAction::List => { + let file = store.load()?; + let mut out = format!("routines in {}\n", store.path().display()); + let mut any = false; + for routine in file.live(now) { + any = true; + let status = match routine.days_left(now) { + Some(days) => format!("expires in {days} days unless used"), + None => "kept: never expires".to_string(), + }; + out.push_str(&format!("\n\"{}\" ({status})\n", routine.phrase)); + for (i, step) in routine.steps.iter().enumerate() { + out.push_str(&format!(" {}. {}\n", i + 1, step.review())); + } + } + if !any { + out.push_str("\n(none)\n"); + } + Ok(out.trim_end().to_string()) + } + RoutineAction::Keep | RoutineAction::Unkeep => { + let keep = args.action == RoutineAction::Keep; + let phrase = args.phrase; + store.update(|file| { + if file.set_taught(&phrase, keep, now) { + Ok(if keep { + format!("kept \"{phrase}\": it will not expire") + } else { + format!("\"{phrase}\" will expire after thirty days unused, from now") + }) + } else { + Err(format!( + "no routine matches \"{phrase}\"; `incu routine list` shows what is saved" + )) + } + }) + } + } +} + /// Ask the server whose stop file is `path` to stop, and say so on stdout. /// /// REPORTED, never swallowed: a stop request that silently failed is worse than @@ -386,6 +549,10 @@ pub fn run(command: Command) -> Result<(), String> { Ok(()) } Command::Serve(args) => serve(ServeOptions::from_args(args)?), + Command::Routine(args) => { + println!("{}", run_routine(args)?); + Ok(()) + } Command::Stop(args) => { let path = match (args.journal, args.workspace) { (Some(journal), _) => crate::safety::stop_file_beside(&journal), @@ -482,6 +649,82 @@ mod tests { )); } + fn routine_args(words: &[&str]) -> RoutineArgs { + match parse(argv(words)).expect("parses") { + Command::Routine(args) => args, + other => panic!("expected routine, got {other:?}"), + } + } + + #[test] + fn routine_keep_and_unkeep_take_the_words_that_ask_for_the_routine() { + let keep = routine_args(&["routine", "keep", "open", "the", "weekly", "report"]); + assert_eq!(keep.action, RoutineAction::Keep); + assert_eq!(keep.phrase, "open the weekly report"); + let unkeep = routine_args(&["routine", "unkeep", "report", "--file", "r.json"]); + assert_eq!(unkeep.action, RoutineAction::Unkeep); + assert_eq!(unkeep.file.as_deref(), Some(Path::new("r.json"))); + assert_eq!( + routine_args(&["routine", "list"]).action, + RoutineAction::List + ); + // A routine is named, a list is not, and a typo is not guessed. + assert!(parse(argv(&["routine", "keep"])).is_err()); + assert!(parse(argv(&["routine", "list", "extra"])).is_err()); + assert!(parse(argv(&["routine", "teach", "x"])).is_err()); + assert!(parse(argv(&["routine"])).is_err()); + assert!(parse(argv(&["routine", "keep", "x", "--taught"])).is_err()); + } + + /// Only this command sets "kept"; keeping stops the expiry and unkeeping + /// starts it again. + #[test] + fn keeping_a_routine_is_done_here_and_stops_it_expiring() { + use crate::routine::store::{Routines, Store}; + let dir = std::env::temp_dir().join(format!("incu-cli-keep-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + let file = dir.join("routines.json"); + let store = Store::at(file.clone()); + let mut routines = Routines::default(); + routines + .learn( + "open notes", + vec![crate::routine::Step::new( + "key", + serde_json::json!({"keys": ["ctrl", "n"], "in_window": "app=notes"}), + )], + false, + crate::routine::now_ms(), + ) + .unwrap(); + store.save(&routines).unwrap(); + let args = |action, phrase: &str| RoutineArgs { + action, + phrase: phrase.to_string(), + file: Some(file.clone()), + config: None, + }; + assert!(store.load().unwrap().routines[0] + .days_left(crate::routine::now_ms()) + .is_some()); + let said = run_routine(args(RoutineAction::Keep, "notes open")).unwrap(); + assert!(said.contains("will not expire"), "{said}"); + let kept = store.load().unwrap(); + assert!(kept.routines[0].taught); + assert_eq!(kept.routines[0].days_left(crate::routine::now_ms()), None); + let listed = run_routine(args(RoutineAction::List, "")).unwrap(); + assert!(listed.contains("kept: never expires"), "{listed}"); + assert!( + listed.contains("ctrl"), + "every step is shown in full: {listed}" + ); + run_routine(args(RoutineAction::Unkeep, "open notes")).unwrap(); + assert!(!store.load().unwrap().routines[0].taught); + let missing = run_routine(args(RoutineAction::Keep, "open mail")).unwrap_err(); + assert!(missing.contains("no routine matches"), "{missing}"); + let _ = std::fs::remove_dir_all(&dir); + } + /// The usage must name every flag the parser takes, or an operator cannot /// find the one that matters. #[test] diff --git a/crates/desktop/src/mcp.rs b/crates/desktop/src/mcp.rs index 9c839f5..96aa134 100644 --- a/crates/desktop/src/mcp.rs +++ b/crates/desktop/src/mcp.rs @@ -140,6 +140,10 @@ pub struct Server { /// The routines file and where records come from. `None` until /// [`Server::with_routines`]; the routine tools refuse by name without it. routines: Option, + /// Set by the stdio reader when the client's end of the pipe closes, so a + /// call that waits (`watch`, `wait_for`) ends instead of running on for + /// nobody. `None` when nothing reads stdin (a test). + client_gone: Option>, } impl Server { @@ -155,9 +159,25 @@ impl Server { tier: Tier::Act, hooks: None, routines: None, + client_gone: None, } } + /// End a waiting call when `gone` is set: the flag the stdio reader raises at + /// end of input. See [`transport::run_lines`]. + #[must_use] + pub fn with_client_gone(mut self, gone: std::sync::Arc) -> Self { + self.client_gone = Some(gone); + self + } + + /// Whether the client has gone away. + fn client_gone(&self) -> bool { + self.client_gone + .as_ref() + .is_some_and(|gone| gone.load(std::sync::atomic::Ordering::SeqCst)) + } + /// Run the operator's own commands around every tool call. /// /// An empty config is dropped rather than stored, so the ordinary case diff --git a/crates/desktop/src/mcp/routine_definitions.rs b/crates/desktop/src/mcp/routine_definitions.rs index 59acbec..54eb707 100644 --- a/crates/desktop/src/mcp/routine_definitions.rs +++ b/crates/desktop/src/mcp/routine_definitions.rs @@ -39,18 +39,22 @@ fn watch_definition() -> Value { "Watch for something to happen and tell the operator when it does, with no \ model polling: the same conditions as wait_for (element_appears, \ element_gone, window_appears, window_gone, screen_changes) looked for every \ - second for up to {max_hours} hours, then the operator's post-tool hook for \ - `watch` is run with the reply as its `tool_response`, which is how it \ - notifies (a desktop notification, a message, a follow-up command: whatever \ - the operator wrote). THIS CALL BLOCKS until the condition holds, the \ - time runs out, or the operator stops the desktop (the stop file ends it \ - within a second), and your turn waits with it: use it only when waiting is \ - the task. For a few seconds use wait_for. `timeout_ms` defaults to {default} \ + second (every {screen_poll} ms for screen_changes, which captures the whole \ + screen each look) for up to {max_hours} hours, then the operator's post-tool \ + hook for `watch` is run with the reply as its `tool_response`, which is how \ + it notifies (a desktop notification, a message, a follow-up command: \ + whatever the operator wrote). THIS CALL BLOCKS until the condition holds, \ + the time runs out, the operator stops the desktop (the stop file ends it \ + within a second) or the client disconnects, and your turn waits with it. \ + IT ALSO HOLDS THIS SERVER: while it runs no other desktop call is answered, \ + including yours. Use it only when waiting is the task. For a few seconds \ + use wait_for. `timeout_ms` defaults to {default} \ ms (one hour) and is held to {max} ms; `poll_ms` {min_poll} to {max_poll}. \ `message` is what the notification should say; it is returned in the reply. \ The reply is JSON whose `result` is `found` or `timeout`, as wait_for's. \ {UNTRUSTED}", max_hours = crate::wait::MAX_WATCH_MS / 3_600_000, + screen_poll = crate::wait::DEFAULT_WATCH_SCREEN_POLL_MS, default = crate::wait::DEFAULT_WATCH_MS, max = crate::wait::MAX_WATCH_MS, min_poll = crate::wait::MIN_WATCH_POLL_MS, @@ -86,8 +90,9 @@ fn watch_definition() -> Value { "poll_ms": { "type": "integer", "description": format!( - "How often to look. Default {}, from {} to {}.", + "How often to look. Default {} ({} for screen_changes), from {} to {}.", crate::wait::DEFAULT_WATCH_POLL_MS, + crate::wait::DEFAULT_WATCH_SCREEN_POLL_MS, crate::wait::MIN_WATCH_POLL_MS, crate::wait::MAX_WATCH_POLL_MS ), @@ -126,7 +131,9 @@ fn routine_save_definition() -> Value { you write). The looking calls (ui_tree, screenshot, list_windows, \ screen_info, cursor_position) are dropped and wait_for is kept. NOTHING IS \ KEPT, and the reason is given, when: the phrase says \"this\" or \"that\"; \ - any call failed or a wait_for timed out; or any step uses an eN id, a pixel \ + ANY call or span of the records failed (a shell command too, not only a \ + desktop call), or the record itself says its run failed, or a wait_for \ + timed out; or any step uses an eN id, a pixel \ coordinate (click, move_mouse, drag) or a window handle (`window`, \ focus_window), because those hold for one screen only — name controls with \ a `selector` and windows inside it (app=...;window=...). Every acting step \ @@ -134,13 +141,22 @@ fn routine_save_definition() -> Value { key step needs `in_window` (e.g. app=notepad;window~=Untitled; a recording \ is given the window of the step before it, and the replay refuses to type \ unless that window has focus); scroll is never kept. A replay invokes links \ - and buttons without the mouse and stops when a set_value or element_action \ - answers \"unverified\" and the next step depends on it. Saving under a \ - phrase that means the same as a saved one (same words, any order, filler \ - words ignored) REPLACES it: that is how a routine that failed is redone. \ - Typed text is stored as it is in the routines file: do not save a routine \ - that types a secret. A routine unused for 30 days is forgotten unless \ - `taught` is true. {UNTRUSTED}" + and buttons without the mouse and stops when a set_value (or an \ + increment/decrement) answers \"unverified\" and the next step depends on \ + it; a button press, which always says \"unverified\", never stops it. \ + Saving under a phrase that means the same as a saved one (same words, any \ + order, filler words ignored; but the words on each side of to, from, into, \ + onto, over, under, before or after stay on that side) REPLACES it: that is \ + how a routine that failed is redone. TYPED TEXT IS STORED AS IT IS in the \ + routines file, and the reply shows every step's full text so you and the \ + person can read it, EXCEPT text typed into a password or other secure \ + field, and text a recording kept only as a placeholder or a digest: those \ + steps are stored as a parameter, `{{{{name}}}}`, listed in the reply, and \ + `routine_run` must be given every one as `params` or it refuses before \ + step 1. You may write `{{{{name}}}}` in a typed text yourself to make \ + one. A routine unused for 30 days is forgotten; only the person can \ + keep one for good (`incu routine keep`), so there is no argument for it. \ + {UNTRUSTED}" ), "inputSchema": { "type": "object", @@ -168,11 +184,6 @@ fn routine_save_definition() -> Value { }, "description": "The steps themselves, instead of records.", }, - "taught": { - "type": "boolean", - "description": "The user asked for this routine by name. It then never \ - expires. Default false.", - }, }, "required": ["phrase"], }, @@ -190,13 +201,26 @@ fn routine_run_definition() -> Value { first one that fails. There is NO repair: on a failure you are told which \ step and why, and what was already done stays done. Then do the whole task \ yourself, from the screen as it is, and routine_save it under the same \ - phrase to replace the routine (or routine_forget it). The routine moves the \ - mouse and keyboard of the machine the operator is using. {UNTRUSTED}" + phrase to replace the routine (or routine_forget it). A picture of the \ + window each step is for (else the screen) is taken before and after the \ + step, kept for the last run only, and the paths of the failed step's two \ + pictures are in the failure. A routine whose list shows `params_needed` \ + needs every one of them in `params` (secrets it does not keep): without \ + them it refuses before step 1. A value is used in memory for that run \ + only. The routine moves the mouse and keyboard of the machine the \ + operator is using. {UNTRUSTED}" ), "inputSchema": { "type": "object", "properties": { "phrase": phrase_property("The words that ask for the routine."), + "params": { + "type": "object", + "additionalProperties": {"type": "string"}, + "description": "Values for the routine's parameters, by name, e.g. \ + {\"password\": \"...\"}: every name in the routine's \ + `params_needed` and no other.", + }, }, "required": ["phrase"], }, @@ -221,9 +245,11 @@ fn routine_forget_definition() -> Value { fn routine_list_definition() -> Value { json!({ "name": "routine_list", - "description": "Every saved routine that has not expired: its phrase, how many \ - steps, when it was learned and last used, whether it was taught \ - (never expires), and how many days it has left.", + "description": "Every saved routine that has not expired: its phrase, each step \ + with ALL the text it types and its full selector (read it: it is \ + what will run), the parameters it needs (`params_needed`), when \ + it was learned and last used, whether the person kept it (never \ + expires), and how many days it has left.", "inputSchema": {"type": "object", "properties": {}}, }) } @@ -232,11 +258,13 @@ fn routine_match_definition() -> Value { json!({ "name": "routine_match", "description": "Which saved routine a phrase would run, without running it: the \ - routine and its steps (typed text shown by length only), or that \ + routine and its steps (all the text each types, in full), or that \ none matches. A phrase matches a routine when it has the same \ - words in any order, ignoring filler words (a, the, to, please, \ - ...); a near miss does not match. Call this before routine_run \ - to see what it would do.", + words in any order, ignoring filler words (a, the, please, ...); \ + the words on each side of to, from, into, onto, over, under, \ + before or after must stay on that side (copy a to b is not copy \ + b to a); a near miss does not match. Call this before \ + routine_run to see what it would do.", "inputSchema": { "type": "object", "properties": { diff --git a/crates/desktop/src/mcp/routine_tests.rs b/crates/desktop/src/mcp/routine_tests.rs index e2d0dd2..8a541ee 100644 --- a/crates/desktop/src/mcp/routine_tests.rs +++ b/crates/desktop/src/mcp/routine_tests.rs @@ -30,11 +30,18 @@ fn folder(name: &str) -> PathBuf { dir } +/// A picture that is not one: the bytes a test's replay keeps, so that no test +/// photographs the screen of the machine that runs it (and none slows another +/// that looks at the real screen). +fn pretend_shot(_: &Step) -> Result, String> { + Ok(b"\xff\xd8 a picture".to_vec()) +} + fn server_with(dir: &Path, source: Box) -> Server { - Server::new(InputThread::start().expect("input thread")).with_routines(RoutineKit::new( - Store::at(dir.join("routines.json")), - source, - )) + Server::new(InputThread::start().expect("input thread")).with_routines( + RoutineKit::new(Store::at(dir.join("routines.json")), source) + .with_shooter(Box::new(pretend_shot)), + ) } fn server(dir: &Path) -> Server { @@ -267,20 +274,20 @@ fn a_routine_is_learned_from_watchman_records_in_the_order_they_happened() { let dir = folder("records"); let source = MemorySource::new() // Named second, happened first. - .with(Record { - id: "w-b".to_string(), - started_at_ms: 2_000, - calls: vec![Call::ok("mcp__desktop__wait_for", never_gone_second())], - }) - .with(Record { - id: "w-a".to_string(), - started_at_ms: 1_000, - calls: vec![ + .with(Record::new( + "w-b", + 2_000, + vec![Call::ok("mcp__desktop__wait_for", never_gone_second())], + )) + .with(Record::new( + "w-a", + 1_000, + vec![ Call::ok("mcp__desktop__ui_tree", json!({})), Call::ok("mcp__desktop__wait_for", gone()), Call::ok("Bash", json!({"command": "ls"})), ], - }); + )); let mut s = server_with(&dir, Box::new(source)); let reply = call( &mut s, @@ -318,11 +325,11 @@ fn never_gone_second() -> Value { #[test] fn a_failed_recorded_run_is_not_learned() { let dir = folder("failed-record"); - let source = MemorySource::new().with(Record { - id: "w-1".to_string(), - started_at_ms: 1, - calls: vec![Call::ok("wait_for", gone()).failed("selector matches nothing")], - }); + let source = MemorySource::new().with(Record::new( + "w-1", + 1, + vec![Call::ok("wait_for", gone()).failed("selector matches nothing")], + )); let mut s = server_with(&dir, Box::new(source)); let reply = call( &mut s, @@ -620,36 +627,528 @@ fn a_replayed_typing_step_into_a_window_that_is_not_there_fails_and_types_nothin let _ = std::fs::remove_dir_all(&dir); } +/// A window no program has, as an `in_window`, for steps that type. +const TYPING_WINDOW: &str = NO_SUCH_WINDOW; + #[test] -fn typed_text_and_steps_never_reach_the_journal_or_a_listing() { - let dir = folder("redaction"); +fn a_secret_typed_into_a_password_field_is_a_parameter_and_never_stored_or_journalled() { + let dir = folder("secret"); let journal = dir.join("desktop.jsonl"); - let mut s = server(&dir).with_journal(journal.clone()); + let (hooks, pre, _post) = Recording::new(None); + let mut s = server(&dir) + .with_journal(journal.clone()) + .with_hook_runner(Box::new(hooks)); let reply = call( &mut s, "routine_save", - json!({"phrase": "type my token", "steps": [ - {"tool": "set_value", "args": {"selector": "app=vault;role=edit;name=Token", "value": "hunter2-secret"}} + json!({"phrase": "unlock the vault", "steps": [ + {"tool": "wait_for", "args": gone()}, + {"tool": "type_text", "args": {"text": "plain words", "in_window": TYPING_WINDOW}}, + {"tool": "set_value", "args": {"selector": "app=vault;window=Sign in;role=edit;name=Password", "value": "hunter2-secret"}}, ]}), ); - assert!(!refused(&reply), "{reply}"); + let saved = answer(&reply); + assert_eq!(saved["parameters"][0]["name"], "password", "{saved}"); + assert_eq!(saved["parameters"][0]["step"], 3); + assert_eq!(saved["parameters"].as_array().unwrap().len(), 1, "{saved}"); assert!(!said(&reply).contains("hunter2"), "{}", said(&reply)); - let matched = said(&call( + assert!(saved["parameters_note"].is_string()); + // On disk: the placeholder, never the value. The ordinary text is kept. + let file = std::fs::read_to_string(dir.join("routines.json")).unwrap(); + assert!(!file.contains("hunter2"), "{file}"); + assert!(file.contains("{{password}}") && file.contains("plain words")); + // The listing says what the routine needs. + let listed = answer(&call(&mut s, "routine_list", json!({}))); + assert_eq!(listed["routines"][0]["params_needed"], json!(["password"])); + + // Without it, the run is refused before step 1: no step is asked of any gate. + let refusal = call(&mut s, "routine_run", json!({"phrase": "vault unlock"})); + assert!(refused(&refusal), "{refusal}"); + assert!(said(&refusal).contains("password"), "{}", said(&refusal)); + assert!( + said(&refusal).contains("Nothing was done"), + "{}", + said(&refusal) + ); + let wrong = call( &mut s, - "routine_match", - json!({"phrase": "token type"}), - )); + "routine_run", + json!({"phrase": "vault unlock", "params": {"pasword": "x"}}), + ); assert!( - !matched.contains("hunter2") && matched.contains("14 characters"), - "{matched}" + refused(&wrong) && said(&wrong).contains("takes: password"), + "{wrong}" ); - // A text field's name is what was typed in it: not in a step's line either. - assert!(!matched.contains("name=Token"), "{matched}"); + assert_eq!( + *pre.lock().unwrap(), + ["routine_save", "routine_list", "routine_run", "routine_run"], + "not even the first step was tried" + ); + + // Given, it is accepted and the run starts: it goes as far as it can on the + // test desktop (the first step passes, the second finds no such window) and + // whatever it says, the value is nowhere. + let ran = call( + &mut s, + "routine_run", + json!({"phrase": "vault unlock", "params": {"password": "hunter2-secret"}}), + ); + assert!(refused(&ran), "{ran}"); + assert!( + said(&ran).contains("failed at step 2 of 3"), + "{}", + said(&ran) + ); + assert!(!said(&ran).contains("hunter2"), "{}", said(&ran)); let lines = std::fs::read_to_string(&journal).unwrap(); assert!(!lines.contains("hunter2"), "{lines}"); - // It is in the routines file, as the docs say. let file = std::fs::read_to_string(dir.join("routines.json")).unwrap(); - assert!(file.contains("hunter2-secret")); + assert!(!file.contains("hunter2"), "{file}"); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_recordings_placeholder_is_a_parameter_and_is_not_replayed() { + let dir = folder("redacted-record"); + let source = MemorySource::new().with(Record::new( + "w-1", + 1, + vec![Call::ok( + "mcp__desktop__type_text", + json!({"text": "[REDACTED]", "in_window": TYPING_WINDOW}), + )], + )); + let mut s = server_with(&dir, Box::new(source)); + let saved = answer(&call( + &mut s, + "routine_save", + json!({"phrase": "type the code", "records": ["w-1"]}), + )); + assert_eq!(saved["parameters"][0]["name"], "text"); + assert!(saved["parameters"][0]["why"] + .as_str() + .unwrap() + .contains("placeholder")); + let file = std::fs::read_to_string(dir.join("routines.json")).unwrap(); + assert!( + !file.contains("REDACTED") && file.contains("{{text}}"), + "{file}" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn the_full_text_of_every_step_is_shown_when_saving_listing_and_matching() { + let dir = folder("review"); + let mut s = server(&dir); + let saved = answer(&call( + &mut s, + "routine_save", + json!({"phrase": "greet the room", "steps": [ + {"tool": "type_text", "args": {"text": "hello there, everyone", "in_window": TYPING_WINDOW}} + ]}), + )); + let listed = answer(&call(&mut s, "routine_list", json!({}))); + let matched = answer(&call( + &mut s, + "routine_match", + json!({"phrase": "room greet"}), + )); + for (what, shown) in [ + ("save", &saved["steps"][0]), + ("list", &listed["routines"][0]["steps"][0]), + ("match", &matched["routine"]["steps"][0]), + ] { + let line = shown.as_str().unwrap(); + assert!(line.contains("hello there, everyone"), "{what}: {line}"); + assert!( + line.contains(TYPING_WINDOW), + "{what}: the window in full: {line}" + ); + assert!( + !line.contains("characters"), + "{what}: not just a length: {line}" + ); + } + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn the_agent_cannot_make_a_routine_kept_and_a_save_never_sets_it() { + let dir = folder("taught"); + let mut s = server(&dir); + let reply = call( + &mut s, + "routine_save", + json!({"phrase": "check the window", "steps": [step(gone())], "taught": true}), + ); + assert!(refused(&reply), "{reply}"); + assert!( + said(&reply).contains("incu routine keep"), + "{}", + said(&reply) + ); + assert!(!dir.join("routines.json").exists()); + let tool = tool_definitions() + .into_iter() + .find(|t| t["name"] == "routine_save") + .unwrap(); + assert!(tool["inputSchema"]["properties"].get("taught").is_none()); + let plain = answer(&save_steps(&mut s, "check the window", vec![step(gone())])); + assert_eq!(plain["taught"], false); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_failed_shell_call_or_a_failed_record_is_not_learned() { + let dir = folder("any-failure"); + let source = MemorySource::new() + .with(Record::new( + "w-bash", + 1, + vec![ + Call::ok("Bash", json!({"command": "false"})).failed("exit code 1"), + Call::ok("mcp__desktop__wait_for", gone()), + ], + )) + .with( + Record::new( + "w-failed", + 2, + vec![Call::ok("mcp__desktop__wait_for", gone())], + ) + .failing("outcome is \"failed\""), + ) + .with(Record::new( + "w-good", + 3, + vec![Call::ok("mcp__desktop__wait_for", gone())], + )); + let mut s = server_with(&dir, Box::new(source)); + for (id, says) in [ + ("w-bash", "exit code 1"), + ("w-failed", "says its run failed"), + ] { + let reply = call( + &mut s, + "routine_save", + json!({"phrase": "wait for it", "records": [id]}), + ); + assert!( + refused(&reply) && said(&reply).contains(says), + "{id}: {reply}" + ); + } + // One failed record among several refuses the whole. + let mixed = call( + &mut s, + "routine_save", + json!({"phrase": "wait for it", "records": ["w-good", "w-failed"]}), + ); + assert!(refused(&mixed), "{mixed}"); + assert!(!dir.join("routines.json").exists()); + assert!(!refused(&call( + &mut s, + "routine_save", + json!({"phrase": "wait for it", "records": ["w-good"]}) + ))); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn the_invoke_falls_back_to_the_recorded_click_only_when_it_was_refused_before_acting() { + use super::routine_tool::{after_invoke, refused_before_acting, AfterInvoke}; + // The control does not offer the action, as each platform says it. + for refused in [ + "button \"Send\" does not support \"invoke\"; it supports: toggle, focus", + "button \"Send\" does not offer \"invoke\" (AXPress); it offers: AXShowMenu", + "button \"Send\" offers no actions; use click_element or set_value", + ] { + assert!(refused_before_acting(refused), "{refused}"); + assert!(matches!( + after_invoke(Err(refused.to_string())), + AfterInvoke::ClickAsRecorded + )); + } + // Everything else may have happened after the control acted: no second try. + for after in [ + "could not invoke button \"Send\": HRESULT 0x80004005", + "button \"Send\" refused AXPress (error -25200)", + "the desktop did not answer within 5000 ms", + "desktop stopped by the operator", + "the operator does not support \"invoke\" today", + "selector `app=mail;role=button` matches no control", + "button \"Send\" is disabled", + ] { + assert!(!refused_before_acting(after), "{after}"); + match after_invoke(Err(after.to_string())) { + AfterInvoke::Done(Err(why)) => assert_eq!(why, after), + _ => panic!("{after} fell back to a second click"), + } + } + // A good answer is the step's answer. + match after_invoke(Ok(vec![super::tools::text( + "invoked button (unverified: x)".to_string(), + )])) { + AfterInvoke::Done(Ok(said)) => assert!(said.contains("invoked")), + _ => panic!("a good answer was not taken"), + } +} + +#[test] +fn a_replay_leaves_a_folder_of_pictures_and_only_the_last_runs() { + let dir = folder("pictures"); + let mut s = server(&dir); + answer(&save_steps( + &mut s, + "check the windows", + vec![step(gone()), step(gone())], + )); + // Left by an earlier run: must be cleared. + let shots = dir.join("last-replay"); + std::fs::create_dir_all(&shots).unwrap(); + std::fs::write(shots.join("step-099-after.jpg"), "old").unwrap(); + std::fs::write(shots.join("keep-me.txt"), "mine").unwrap(); + let ran = answer(&call( + &mut s, + "routine_run", + json!({"phrase": "windows check"}), + )); + assert_eq!( + ran["pictures"]["folder"], + shots.display().to_string(), + "{ran}" + ); + assert!( + !shots.join("step-099-after.jpg").exists(), + "the last run's pictures remain" + ); + assert!( + shots.join("keep-me.txt").exists(), + "a file that is not ours was deleted" + ); + // Whether the screen could be captured here or not, the run says which. + // One before and one after each of the two steps, and no more. + assert_eq!(ran["pictures"]["kept"], 4, "{ran}"); + let mut named: Vec = std::fs::read_dir(&shots) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().to_string()) + .filter(|n| n.starts_with("step-")) + .collect(); + named.sort(); + assert_eq!( + named, + [ + "step-001-after.jpg", + "step-001-before.jpg", + "step-002-after.jpg", + "step-002-before.jpg" + ] + ); + // A second run starts again: only it is kept. + answer(&call( + &mut s, + "routine_run", + json!({"phrase": "windows check"}), + )); + assert_eq!( + std::fs::read_dir(&shots) + .unwrap() + .filter(|e| { + e.as_ref() + .unwrap() + .file_name() + .to_string_lossy() + .starts_with("step-") + }) + .count(), + 4 + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_screen_that_cannot_be_captured_is_said_and_does_not_fail_the_replay() { + let dir = folder("no-pictures"); + let kit = RoutineKit::new(Store::at(dir.join("routines.json")), Box::new(NoSource)) + .with_shooter(Box::new(|_| Err("no display in this session".to_string()))); + let mut s = Server::new(InputThread::start().expect("input thread")).with_routines(kit); + answer(&save_steps(&mut s, "check the window", vec![step(gone())])); + let ran = answer(&call( + &mut s, + "routine_run", + json!({"phrase": "check the window"}), + )); + assert_eq!(ran["steps_run"], 1); + assert_eq!(ran["pictures"]["kept"], 0); + assert!( + ran["pictures"]["trouble"] + .as_str() + .unwrap() + .contains("no display"), + "{ran}" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_failed_replay_names_the_pictures_of_the_failed_step() { + let dir = folder("failed-pictures"); + let mut s = server(&dir); + answer(&save_steps( + &mut s, + "three waits", + vec![step(gone()), step(never_appears()), step(gone())], + )); + let reply = call(&mut s, "routine_run", json!({"phrase": "three waits"})); + assert!(refused(&reply), "{reply}"); + let said = said(&reply); + let shots = dir.join("last-replay"); + // The failed step's two pictures are named, by full path, and exist. + let (before, after) = ( + shots.join("step-002-before.jpg"), + shots.join("step-002-after.jpg"), + ); + assert!(said.contains("Pictures of step 2"), "{said}"); + assert!(said.contains(&before.display().to_string()), "{said}"); + assert!(said.contains(&after.display().to_string()), "{said}"); + assert!(before.exists() && after.exists()); + assert!( + !shots.join("step-003-before.jpg").exists(), + "step 3 never ran" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn two_servers_saving_at_once_each_keep_the_others_routines() { + let dir = folder("two-servers"); + let workers: Vec<_> = (0..2) + .map(|w| { + let dir = dir.clone(); + std::thread::spawn(move || { + let mut s = server(&dir); + for i in 0..4 { + let reply = save_steps( + &mut s, + &format!("task from server{w} number{i}"), + vec![step(gone())], + ); + assert!(!refused(&reply), "{reply}"); + } + }) + }) + .collect(); + for worker in workers { + worker.join().unwrap(); + } + let kept = Store::at(dir.join("routines.json")).load().unwrap(); + assert_eq!( + kept.routines.len(), + 8, + "a save overwrote another server's routine" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn a_routine_saved_for_copying_one_way_is_not_run_for_the_other() { + let dir = folder("direction"); + let mut s = server(&dir); + answer(&save_steps( + &mut s, + "copy backup files to desktop", + vec![step(gone())], + )); + let other = answer(&call( + &mut s, + "routine_match", + json!({"phrase": "copy desktop files to backup"}), + )); + assert_eq!(other["matched"], false); + let same = answer(&call( + &mut s, + "routine_match", + json!({"phrase": "files backup copy, to the desktop"}), + )); + assert_eq!(same["matched"], true); + let run = call( + &mut s, + "routine_run", + json!({"phrase": "copy desktop files to backup"}), + ); + assert!( + refused(&run) && said(&run).contains("no routine matches"), + "{run}" + ); + let _ = std::fs::remove_dir_all(&dir); +} + +/// A reader that gives its lines when told to and then, when its sender is +/// dropped, ends. +struct Wire(std::sync::mpsc::Receiver>, Vec); + +impl std::io::Read for Wire { + fn read(&mut self, buf: &mut [u8]) -> std::io::Result { + if self.1.is_empty() { + match self.0.recv() { + Ok(more) => self.1 = more, + Err(_) => return Ok(0), + } + } + let n = self.1.len().min(buf.len()); + buf[..n].copy_from_slice(&self.1[..n]); + self.1.drain(..n); + Ok(n) + } +} + +#[test] +fn a_watch_ends_when_the_client_goes_away_instead_of_running_for_a_day() { + let dir = folder("watch-eof"); + let (to_server, from_client) = std::sync::mpsc::channel::>(); + let request = json!({ + "jsonrpc": "2.0", "id": 1, "method": "tools/call", + "params": {"name": "watch", "arguments": { + "condition": "window_appears", "selector": NO_SUCH_WINDOW, "timeout_ms": 86_400_000 + }} + }); + to_server.send(format!("{request}\n").into_bytes()).unwrap(); + // The client disconnects a moment after asking, while the watch is polling. + let hangup = std::thread::spawn(move || { + std::thread::sleep(std::time::Duration::from_millis(500)); + drop(to_server); + }); + let started = std::time::Instant::now(); + let mut out = Vec::new(); + let s = server(&dir); + super::transport::run_lines(s, Wire(from_client, Vec::new()), &mut out).unwrap(); + hangup.join().unwrap(); + assert!( + started.elapsed() < std::time::Duration::from_secs(10), + "the watch ran on for {:?}", + started.elapsed() + ); + // It answered (refused: stopped) before returning, and did not hang. + let reply = String::from_utf8(out).unwrap(); + assert!(reply.contains("\"isError\":true"), "{reply}"); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn every_line_before_the_end_of_input_is_still_answered() { + let dir = folder("eof-lines"); + let (to_server, from_client) = std::sync::mpsc::channel::>(); + for id in 1..=3 { + to_server + .send(format!("{{\"jsonrpc\":\"2.0\",\"id\":{id},\"method\":\"ping\"}}\n").into_bytes()) + .unwrap(); + } + drop(to_server); + let mut out = Vec::new(); + super::transport::run_lines(server(&dir), Wire(from_client, Vec::new()), &mut out).unwrap(); + let replies = String::from_utf8(out).unwrap(); + assert_eq!(replies.lines().count(), 3, "{replies}"); let _ = std::fs::remove_dir_all(&dir); } diff --git a/crates/desktop/src/mcp/routine_tool.rs b/crates/desktop/src/mcp/routine_tool.rs index e921bde..e489892 100644 --- a/crates/desktop/src/mcp/routine_tool.rs +++ b/crates/desktop/src/mcp/routine_tool.rs @@ -23,16 +23,23 @@ use serde_json::{json, Value}; -use crate::routine::learn::{check_phrase, learn_from_calls, learn_from_steps, Cleaned, IN_WINDOW}; +use std::cell::RefCell; +use std::collections::HashMap; +use std::path::PathBuf; + +use crate::capture::{CaptureOptions, ShotFormat}; +use crate::routine::learn::{check_phrase, check_record, learn_with, Cleaned, IN_WINDOW}; +use crate::routine::params::{self, Values, Why}; +use crate::routine::pictures::{Pictures, When}; use crate::routine::replay::{replay, Runner}; -use crate::routine::source::{Call, StepSource}; -use crate::routine::store::{Routines, Store}; +use crate::routine::source::{Call, Record, StepSource}; +use crate::routine::store::Store; use crate::routine::words::word_set; use crate::routine::{now_ms, Routine, Step}; use crate::selector::Selector; -use super::tools::{text, text_of, window_id_for}; +use super::tools::{capture, resolve_selector, text, text_of, window_id_for}; use super::Server; /// The most record ids one `routine_save` reads. @@ -45,13 +52,32 @@ const MAX_PHRASE_CHARS: usize = 200; pub struct RoutineKit { store: Store, source: Box, + /// Takes the picture of a step; incu's own capture when `None`. Replaced in + /// tests, so that what is kept and named is tested without photographing the + /// screen of the machine running them. + shooter: Option, } +/// What takes the picture of the window a step is for: the image's bytes (JPEG). +pub type Shooter = Box Result, String> + Send>; + impl RoutineKit { /// A kit over `store`, reading records from `source`. #[must_use] pub fn new(store: Store, source: Box) -> Self { - Self { store, source } + Self { + store, + source, + shooter: None, + } + } + + /// The same kit, taking its step pictures with `shooter` and not incu's own + /// capture. + #[must_use] + pub fn with_shooter(mut self, shooter: Shooter) -> Self { + self.shooter = Some(shooter); + self } /// Where the routines file is. @@ -93,9 +119,26 @@ impl Server { fn routine_save(&mut self, args: &Value) -> Result, String> { let phrase = phrase_arg(args)?; check_phrase(&phrase).map_err(|r| r.to_string())?; - let taught = args.get("taught").and_then(Value::as_bool).unwrap_or(false); + // Being kept (never expiring) is the person's to give, by `incu routine + // keep`. Said plainly, so an agent does not believe it did it. + if args.get("taught").is_some() { + return Err( + "`taught` is not an argument: only the person keeps a routine \ + from expiring, with `incu routine keep \"\"` at a terminal" + .to_string(), + ); + } let kit = self.kit()?; - let cleaned = match (args.get("records"), args.get("steps")) { + // What the desktop says is a password field, asked about each control a + // typing step might have gone into, once each. + let seen: RefCell> = RefCell::new(HashMap::new()); + let is_secret = |raw: &str| { + *seen + .borrow_mut() + .entry(raw.to_string()) + .or_insert_with(|| desktop_says_password(raw)) + }; + let calls: Vec = match (args.get("records"), args.get("steps")) { (Some(_), Some(_)) => { return Err("give `records` or `steps`, not both".to_string()); } @@ -104,14 +147,24 @@ impl Server { "give `records` (Watchman record ids) or `steps` to learn from".to_string(), ); } - (Some(ids), None) => learn_from_calls(&phrase, &calls_of(kit.source.as_ref(), ids)?), - (None, Some(steps)) => learn_from_steps(&phrase, &steps_of(steps)?), - } - .map_err(|r| r.to_string())?; + (Some(ids), None) => { + let records = records_of(kit.source.as_ref(), ids)?; + // A run that failed is not kept, whatever calls it holds. + for record in &records { + check_record(record).map_err(|r| r.to_string())?; + } + records.into_iter().flat_map(|r| r.calls).collect() + } + (None, Some(steps)) => steps_of(steps)? + .iter() + .map(|s| Call::ok(&s.tool, s.args.clone())) + .collect(), + }; + let cleaned = learn_with(&phrase, &calls, &is_secret).map_err(|r| r.to_string())?; let now = now_ms(); - let mut file = kit.store.load()?; - let saved = file.learn(&phrase, cleaned.steps.clone(), taught, now)?; - kit.store.save(&file)?; + let saved = kit + .store + .update(|file| file.learn(&phrase, cleaned.steps.clone(), false, now))?; Ok(vec![text(saved_reply( &phrase, &cleaned, @@ -124,7 +177,12 @@ impl Server { fn routine_run(&mut self, args: &Value) -> Result, String> { let phrase = phrase_arg(args)?; let routine = self.find_routine(&phrase)?; - let outcome = replay(&routine, &mut Live { server: self }); + // Every parameter must be given before anything runs: a secret the + // routine does not keep is asked for here, not half way through. + let values = Values::bind(&routine.steps, args.get("params"))?; + let mut live = Live::new(self, values); + let outcome = replay(&routine, &mut live); + let pictures = live.pictures_note(); let report = outcome.map_err(|failure| failure.to_string())?; // Used, so its thirty days begin again. Not a reason to fail a routine // that has just run to the end. @@ -139,6 +197,7 @@ impl Server { "ran": routine.phrase, "steps_run": report.ran.len(), "steps": steps, + "pictures": pictures, }); if let Err(why) = remembered { reply["note"] = json!(format!( @@ -152,13 +211,15 @@ impl Server { fn routine_forget(&mut self, args: &Value) -> Result, String> { let phrase = phrase_arg(args)?; let kit = self.kit()?; - let mut file = kit.store.load()?; - if !file.forget(&phrase, now_ms()) { - return Err(format!( - "no routine matches \"{phrase}\"; routine_list shows what is saved" - )); - } - kit.store.save(&file)?; + kit.store.update(|file| { + if file.forget(&phrase, now_ms()) { + Ok(()) + } else { + Err(format!( + "no routine matches \"{phrase}\"; routine_list shows what is saved" + )) + } + })?; Ok(vec![text(format!("forgot the routine for \"{phrase}\""))]) } @@ -184,9 +245,7 @@ impl Server { let words: Vec = word_set(&phrase).into_iter().collect(); let reply = match file.matching(&phrase, now) { Some(routine) => { - let mut shown = describe(routine, now); - shown["steps"] = json!(routine.steps.iter().map(Step::summary).collect::>()); - json!({"matched": true, "words": words, "routine": shown}) + json!({"matched": true, "words": words, "routine": describe(routine, now)}) } None => json!({"matched": false, "words": words}), }; @@ -204,10 +263,10 @@ impl Server { /// Record that the routine `phrase` ran to the end, reading the file again /// first so a routine saved meanwhile is not overwritten. fn remember_use(&self, phrase: &str) -> Result<(), String> { - let kit = self.kit()?; - let mut file: Routines = kit.store.load()?; - file.used(phrase, now_ms()); - kit.store.save(&file) + self.kit()?.store.update(|file| { + file.used(phrase, now_ms()); + Ok(()) + }) } /// One tool call through every gate, as `tools/call` runs it. The content it @@ -231,29 +290,202 @@ impl Server { /// The live desktop as a [`Runner`]: every step through the server's gates. struct Live<'a> { server: &'a mut Server, + /// What the caller gave for the routine's parameters. In memory only. + values: Values, + /// The folder this run's pictures go to, or `None` when it could not be made. + pictures: Option, + /// Why there are no pictures, or fewer: the first thing that went wrong. + picture_trouble: Option, } -impl Runner for Live<'_> { - fn run(&mut self, _n: usize, step: &Step) -> Result { +impl<'a> Live<'a> { + /// A run over `server`, with the parameters `values` and a fresh folder for + /// its pictures (the last run's are cleared). + fn new(server: &'a mut Server, values: Values) -> Self { + let (pictures, picture_trouble) = match server + .kit() + .map(|kit| kit.store.pictures_dir()) + .and_then(Pictures::begin) + { + Ok(pictures) => (Some(pictures), None), + Err(why) => (None, Some(format!("no pictures: {why}"))), + }; + Live { + server, + values, + pictures, + picture_trouble, + } + } + + /// What the reply says of the pictures: where they are and how many, or why + /// there are none. + fn pictures_note(&self) -> Value { + match &self.pictures { + Some(pictures) => { + let mut note = json!({ + "folder": pictures.dir().display().to_string(), + "kept": pictures.count(), + "note": "one picture before and one after each step, of the last run only", + }); + if let Some(trouble) = &self.picture_trouble { + note["trouble"] = json!(trouble); + } + note + } + None => json!({"kept": 0, "trouble": self.picture_trouble}), + } + } + + /// One step, as the live desktop runs it (see [`Runner::run`]). + fn step(&mut self, step: &Step) -> Result { let args = live_args(step)?; // A link or a button is invoked through its own pattern when it has one: // that needs no mouse, works for a control scrolled out of view and takes - // no focus. A control that will not be invoked (or a refusal of the - // invoke) leaves the recorded click to run, each through the same gates; - // the journal shows both. + // no focus. The recorded click runs INSTEAD only when the invoke was + // refused before it did anything; see [`after_invoke`]. if let Some(invoke) = invoke_for(step) { - if let Ok(content) = self.server.gated_call("element_action", &invoke) { - return Ok(text_of(&content)); + match after_invoke(self.server.gated_call("element_action", &invoke)) { + AfterInvoke::Done(answer) => return answer, + AfterInvoke::ClickAsRecorded => {} } } self.server .gated_call(&step.tool, &args) .map(|content| text_of(&content)) } +} + +impl Runner for Live<'_> { + fn run(&mut self, _n: usize, step: &Step) -> Result { + // The parameters go into the arguments of this one call, in memory, and + // come out of whatever it answers. + let filled = self.values.fill(step); + self.step(&filled) + .map(|answer| self.values.scrub(&answer)) + .map_err(|why| self.values.scrub(&why)) + } fn settle(&mut self) { std::thread::sleep(crate::limits::get().settle); } + + fn look(&mut self, n: usize, step: &Step, when: When) -> Option { + let Live { + server, + pictures, + picture_trouble, + .. + } = self; + let pictures = pictures.as_mut()?; + // A stop is a stop: nothing more is read off the screen either. + server.catch_stop_file(); + if server.safety.is_stopped() { + return None; + } + let mut note = |why: String| { + picture_trouble.get_or_insert(why); + }; + let taken = match server.kit().ok().and_then(|kit| kit.shooter.as_ref()) { + Some(shooter) => shooter(step), + None => picture_of(step), + }; + match taken { + Ok(bytes) => pictures + .keep(n, when, "jpg", &bytes) + .map_err(|why| note(format!("step {n}: {why}"))) + .ok(), + Err(why) => { + note(format!("step {n}: no picture: {why}")); + None + } + } + } +} + +/// What to do after the `element_action invoke` tried in place of a recorded +/// click. +pub(super) enum AfterInvoke { + /// That is the step's answer, good or bad. + Done(Result), + /// The invoke was refused before it did anything: run the recorded click. + ClickAsRecorded, +} + +/// Decide from what the invoke answered. +/// +/// The click runs after an error ONLY when the error says the control cannot be +/// invoked at all ([`refused_before_acting`]). Any other error (the pattern call +/// failed, the call timed out, the operator's hook or the stop refused it) may +/// have come after the control acted, and a second action on top would press a +/// button twice: the step fails with that error and the replay stops. +pub(super) fn after_invoke(answer: Result, String>) -> AfterInvoke { + match answer { + Ok(content) => AfterInvoke::Done(Ok(text_of(&content))), + Err(why) if refused_before_acting(&why) => AfterInvoke::ClickAsRecorded, + Err(why) => AfterInvoke::Done(Err(why)), + } +} + +/// Whether an `element_action invoke` error says the control does not offer the +/// action: nothing was done, as each platform words it (UI Automation: "does not +/// support", macOS and Linux: "does not offer", "offers no actions"). +/// +/// Narrow on purpose. It matches the whole of each platform's message (the +/// action's name in quotes AND the list of what the control does offer that +/// follows it), so a message that merely contains some of those words (an +/// operator's hook explaining a refusal) is not taken for one. +pub(super) fn refused_before_acting(why: &str) -> bool { + let said = why.to_lowercase(); + said.contains("does not support \"invoke\"; it supports:") + || (said.contains("does not offer \"invoke\"") && said.contains("; it offers:")) + || said.contains("offers no actions; use click_element or set_value") +} + +/// The window a step is for, as a selector of only its window part: the one its +/// own selector names, or its `in_window`. +fn step_window(step: &Step) -> Option { + let raw = step + .args + .get("selector") + .or_else(|| step.args.get(IN_WINDOW))? + .as_str()?; + let whole = Selector::parse(raw).ok()?; + whole + .has_window_part() + .then(|| Selector::parse(&whole.render_window_part()).ok())? +} + +/// A picture of the window `step` is for, else of the screen: incu's own capture, +/// under the same guards as `screenshot` (a blocked application's window is never +/// captured). A window that cannot be found (it may be what failed) is the +/// screen. +/// +/// JPEG at a middling quality: the pictures are for a person and a model to look +/// at, and many of them are kept. +fn picture_of(step: &Step) -> Result, String> { + let window = step_window(step).and_then(|selector| window_id_for(&selector).ok()); + let options = CaptureOptions { + format: ShotFormat::Jpeg, + jpeg_quality: 70, + ..CaptureOptions::default() + }; + capture(window.as_deref(), options).map(|(shot, _)| shot.bytes) +} + +/// Whether the desktop says the control `raw` names is a password field, now. +/// Asked when a routine is saved, of a control that no name condemns; the control +/// may not be on screen any more, and then the answer is no. +fn desktop_says_password(raw: &str) -> bool { + let Ok(selector) = Selector::parse(raw) else { + return false; + }; + // Only a selector that says which window: without one it would be looked for + // in whatever is in front. + if !selector.has_window_part() || !selector.has_element_part() { + return false; + } + resolve_selector(None, &selector).is_ok_and(|found| found.element.secret) } /// The arguments the tool is called with for `step`. A keyboard step's @@ -306,8 +538,8 @@ fn phrase_arg(args: &Value) -> Result { Ok(phrase.to_string()) } -/// The recorded calls of the records `ids` names, oldest record first. -fn calls_of(source: &dyn StepSource, ids: &Value) -> Result, String> { +/// The records `ids` names, oldest first. +fn records_of(source: &dyn StepSource, ids: &Value) -> Result, String> { let ids: Vec<&str> = ids .as_array() .ok_or("records must be a list of Watchman record ids")? @@ -324,7 +556,7 @@ fn calls_of(source: &dyn StepSource, ids: &Value) -> Result, String> { // A task spans several turns, one record each; they run in the order they // happened, not the order they were named. Stable, so equal times keep it. records.sort_by_key(|r| r.started_at_ms); - Ok(records.into_iter().flat_map(|r| r.calls).collect()) + Ok(records) } /// An explicit step list: `[{"tool": ..., "args": {...}}, ...]`. @@ -345,11 +577,18 @@ fn steps_of(steps: &Value) -> Result, String> { fn describe(routine: &Routine, now: i64) -> Value { let mut shown = json!({ "phrase": routine.phrase, - "steps": routine.steps.len(), + "step_count": routine.steps.len(), + // Everything each step types and every selector, in full: this is what + // the person reads to see what the routine will do before it runs. + "steps": routine.steps.iter().map(Step::review).collect::>(), "learned": routine.date, "last_used": crate::routine::day_of(routine.last_used_ms), "taught": routine.taught, }); + let needs = params::names(&routine.steps); + if !needs.is_empty() { + shown["params_needed"] = json!(needs); + } if let Some(days) = routine.days_left(now) { shown["expires_in_days"] = json!(days); } @@ -367,10 +606,31 @@ fn saved_reply( "saved": phrase, "replaced_an_older_routine": saved.replaced, "taught": saved.taught, - "steps": cleaned.steps.iter().map(Step::summary).collect::>(), + // Every step with the full text it types: read it, it is what will run. + "steps": cleaned.steps.iter().map(Step::review).collect::>(), "looking_steps_dropped": cleaned.dropped, "file": file.display().to_string(), }); + if !cleaned.parameters.is_empty() { + reply["parameters"] = json!(cleaned + .parameters + .iter() + .map(|p| json!({ + "name": p.name, + "step": p.step, + "why": match p.why { + Why::SecureField => "typed into a password or other secure field", + Why::Redacted => "the recording kept only a placeholder for the text", + }, + })) + .collect::>()); + reply["parameters_note"] = json!( + "those steps are NOT stored with their text. routine_run refuses to start \ + unless it is given every one, as params: {\"name\": \"...\"}. The value is \ + used in memory for that run only and is never written to the routines file \ + or the journal" + ); + } if cleaned.inferred_window > 0 { reply["keyboard_steps_given_the_window_before_them"] = json!(cleaned.inferred_window); reply["window_note"] = json!( diff --git a/crates/desktop/src/mcp/tools.rs b/crates/desktop/src/mcp/tools.rs index 001f7e4..7d8bbff 100644 --- a/crates/desktop/src/mcp/tools.rs +++ b/crates/desktop/src/mcp/tools.rs @@ -82,16 +82,18 @@ impl Server { // stop ends a wait within a fraction of a second. let stopped = || { self.catch_stop_file(); - self.safety.is_stopped() + self.safety.is_stopped() || self.client_gone() }; super::wait_tool::wait_for(args, &stopped) } "watch" => { self.journal_note("watch", &Self::describe("watch", args)); - // The same stop, asked as often, for a wait of hours. + // The same stop, asked as often, for a wait of hours, and the + // client going away (its end of stdin closing) ends it too: a + // watch nobody is listening to must not run out its day. let stopped = || { self.catch_stop_file(); - self.safety.is_stopped() + self.safety.is_stopped() || self.client_gone() }; super::wait_tool::watch(args, &stopped) } @@ -616,7 +618,10 @@ pub(super) fn element_target(args: &Value) -> Result { /// compared with a window named by id: a mismatch error describes the window, /// and for a blocked one that is its title, and a match and a mismatch would /// be told apart. -fn resolve_selector(window: Option<&str>, selector: &Selector) -> Result { +pub(super) fn resolve_selector( + window: Option<&str>, + selector: &Selector, +) -> Result { let info = uia::window_for(window, selector, &refuse_blocked).map_err(|e| e.to_string())?; uia::resolve_in(info, selector).map_err(|e| e.to_string()) } @@ -707,15 +712,29 @@ pub(super) fn guard_capture(window: Option<&str>) -> Result, Ok(Some(info)) } -/// A screenshot of one window, or of the primary display — refused when a -/// blocked application's window is in what would be captured. -fn shoot(window: Option<&str>, options: CaptureOptions) -> Result, String> { +/// The capture of one window, or of the primary display, under +/// [`guard_capture`]'s rule: refused when a blocked application's window is in +/// what would be captured. The window it was of, when one was named. +/// +/// Called by [`shoot`] and by a routine replay's before and after pictures, so +/// both are held to the one guard. +pub(super) fn capture( + window: Option<&str>, + options: CaptureOptions, +) -> Result<(capture::Screenshot, Option), String> { let of = guard_capture(window)?; let shot = match &of { Some(info) => capture::capture_region(info.rect, options), None => capture::capture_primary(options), } .map_err(|e| e.to_string())?; + Ok((shot, of)) +} + +/// A screenshot of one window, or of the primary display — refused when a +/// blocked application's window is in what would be captured. +fn shoot(window: Option<&str>, options: CaptureOptions) -> Result, String> { + let (shot, of) = capture(window, options)?; let (width, height) = shot.model_size(); let t = shot.transform; Ok(vec![ diff --git a/crates/desktop/src/mcp/transport.rs b/crates/desktop/src/mcp/transport.rs index 368089c..85b87ba 100644 --- a/crates/desktop/src/mcp/transport.rs +++ b/crates/desktop/src/mcp/transport.rs @@ -146,33 +146,85 @@ pub(super) fn error_response(id: Value, code: i64, message: &str) -> String { /// When the agent process dies its end of the pipe closes, `read_line` returns /// zero bytes, and no timer can fake that. So the loop stops the desktop on EOF /// rather than relying on a heartbeat that could keep beating into the void. -pub fn run_stdio(mut server: Server) -> std::io::Result<()> { - let stdin = std::io::stdin(); - let mut stdout = std::io::stdout(); - let mut line = String::new(); +pub fn run_stdio(server: Server) -> std::io::Result<()> { + run_lines(server, std::io::stdin(), std::io::stdout()) +} - loop { - line.clear(); - match stdin.lock().read_line(&mut line) { - // EOF: the client is gone. Halt the desktop before returning, so a - // slow in-flight action cannot outlive the thing that asked for it. - Ok(0) => { - server.safety().stop(); - return Ok(()); +/// [`run_stdio`] over any reader and writer. +/// +/// # Why input is read on a thread of its own +/// +/// The server handles one call at a time on this thread, and a `watch` holds the +/// call for up to a day. With the read inline, a client that went away in the +/// middle of one would not be noticed until the call returned: the pipe's EOF +/// sat unread behind it. So a reader thread takes the lines and the EOF, passes +/// the lines on, and raises the server's `client_gone` flag at EOF, which the +/// waiting calls look at as often as they look at the stop file. The EOF is +/// still only ACTED on here, after every line before it has been answered, +/// exactly as it always was. +/// +/// # Errors +/// +/// Reading or writing failed. +pub(crate) fn run_lines(server: Server, input: R, mut out: W) -> std::io::Result<()> +where + R: std::io::Read + Send + 'static, + W: Write, +{ + use std::sync::atomic::{AtomicBool, Ordering}; + use std::sync::{mpsc, Arc}; + + let gone = Arc::new(AtomicBool::new(false)); + let server = server.with_client_gone(gone.clone()); + let (lines, queued) = mpsc::channel::>(); + std::thread::Builder::new() + .name("incode-desktop-stdin".to_string()) + .spawn(move || { + let mut reader = std::io::BufReader::new(input); + loop { + let mut line = String::new(); + match reader.read_line(&mut line) { + Ok(0) => { + gone.store(true, Ordering::SeqCst); + return; + } + Ok(_) => { + if lines.send(Ok(line)).is_err() { + return; + } + } + Err(e) => { + gone.store(true, Ordering::SeqCst); + let _ = lines.send(Err(e)); + return; + } + } } - Ok(_) => { + })?; + let mut server = server; + loop { + match queued.recv() { + Ok(Ok(line)) => { // Every request received is evidence the client is alive. server.safety().heartbeat(); if let Some(reply) = server.handle_line(&line) { - writeln!(stdout, "{reply}")?; - stdout.flush()?; + writeln!(out, "{reply}")?; + out.flush()?; } } - Err(e) => { + // The reader failed: halt the desktop before returning. + Ok(Err(e)) => { server.safety().stop(); return Err(e); } + // EOF, every line before it answered: the client is gone. Halt the + // desktop before returning, so a slow in-flight action cannot outlive + // the thing that asked for it. + Err(_) => { + server.safety().stop(); + return Ok(()); + } } } } diff --git a/crates/desktop/src/mcp/wait_tool.rs b/crates/desktop/src/mcp/wait_tool.rs index 674b33b..e461aee 100644 --- a/crates/desktop/src/mcp/wait_tool.rs +++ b/crates/desktop/src/mcp/wait_tool.rs @@ -449,14 +449,36 @@ pub(super) fn wait_for(args: &Value, stopped: &dyn Fn() -> bool) -> Result bool) -> Result, String> { let condition = condition_of(args)?; - let budget = Budget::watch(whole(args, "timeout_ms")?, whole(args, "poll_ms")?); + let budget = watch_budget( + &condition, + whole(args, "timeout_ms")?, + whole(args, "poll_ms")?, + ); let message = message_arg(args)?; look_until(&condition, &budget, stopped, message) } +/// How long a `watch` of `condition` may run and how often it looks: a look at +/// the screen is the slow one, so it is the one given the slower default. +fn watch_budget(condition: &Condition, timeout: Option, poll: Option) -> Budget { + match condition { + Condition::Screen { .. } => Budget::watch_screen(timeout, poll), + _ => Budget::watch(timeout, poll), + } +} + /// The most of a `watch` message that is kept, in characters. const MAX_MESSAGE_CHARS: usize = 200; @@ -499,6 +521,30 @@ mod tests { use super::*; use serde_json::json; + /// A watch of the screen looks every five seconds, not every second, and + /// anything else keeps its second; a poll the caller gives is still theirs. + #[test] + fn a_watch_of_the_screen_polls_every_five_seconds_by_default() { + let screen = condition_of(&json!({"condition": "screen_changes"})).unwrap(); + let window = + condition_of(&json!({"condition": "window_gone", "selector": "app=zz;window=none"})) + .unwrap(); + assert_eq!(watch_budget(&screen, None, None).poll_ms, 5_000); + assert_eq!(watch_budget(&window, None, None).poll_ms, 1_000); + assert_eq!(watch_budget(&screen, None, Some(2_000)).poll_ms, 2_000); + // The same bounds otherwise: an hour by default, a day at most. + assert_eq!( + watch_budget(&screen, None, None).timeout_ms, + wait::DEFAULT_WATCH_MS + ); + assert_eq!( + watch_budget(&screen, Some(u64::MAX), None).timeout_ms, + wait::MAX_WATCH_MS + ); + // And wait_for is not touched: still 250 ms. + assert_eq!(Budget::new(None, None).poll_ms, wait::DEFAULT_POLL_MS); + } + /// **A look that blocks cannot outlive its limit.** The live hang was a /// call that did not return for minutes; here the work sleeps far longer /// than the limit and `bounded` must hand back long before it returns. diff --git a/crates/desktop/src/routine/learn.rs b/crates/desktop/src/routine/learn.rs index 32f8770..02ad5b9 100644 --- a/crates/desktop/src/routine/learn.rs +++ b/crates/desktop/src/routine/learn.rs @@ -5,10 +5,11 @@ //! //! 1. **The phrase must not point at the screen.** "this" or "that" means //! whatever was on screen when it was said ([`super::words::is_deictic`]). -//! 2. **The run must not have failed.** A call that errored, or a `wait_for` -//! that timed out or was stopped, is a run that did not do what it set out -//! to; replaying it would repeat the failure. Failed OBSERVING calls do not -//! count: they are dropped anyway. +//! 2. **The run must not have failed.** ANY call that errored (a shell command, +//! a refused `ui_tree`, a file that was not there: not only a desktop call), +//! a record whose own outcome says it failed ([`check_record`]), or a +//! `wait_for` that timed out or was stopped, is a run that did not do what it +//! set out to; replaying it would repeat the failure. //! 3. **Every kept step must be replayable on another day.** The only way to //! name a control that survives is a selector, so a step that uses an `eN` //! id (valid for one `ui_tree` only), a pixel coordinate (valid for one @@ -33,6 +34,10 @@ //! left out and REPORTED in [`Cleaned::skipped`], so the agent knows that a //! step it may have relied on is not in the routine. //! +//! Last, the text of a step that types into a secure field, or that a recording +//! kept only as a placeholder, is taken out and made a parameter +//! ([`super::params`]): a routine never holds a secret. +//! //! The same per-step check, [`check_step`], runs again before a replay, so a //! hand-edited routines file cannot smuggle in a step the learner would have //! refused. @@ -41,7 +46,8 @@ use std::fmt; use serde_json::Value; -use super::source::Call; +use super::params::{parameterise, Made}; +use super::source::{Call, Record}; use super::words::{is_deictic, word_set}; use super::{Step, MAX_STEPS, MAX_STEP_BYTES}; use crate::permission::{ACTING, OBSERVING}; @@ -87,6 +93,13 @@ pub enum Refusal { Deictic, /// The phrase has no words of its own (all filler). NoWords, + /// A record's own outcome says the run failed, whatever its calls say. + RecordFailed { + /// The record. + id: String, + /// What it says. + why: String, + }, /// A call in the run failed. RunFailed { /// Its 1-based position in the calls given. @@ -124,6 +137,11 @@ impl fmt::Display for Refusal { "not learned: the phrase has no words of its own once filler words are \ ignored" ), + Refusal::RecordFailed { id, why } => write!( + f, + "not learned: record {id} says its run failed ({why}). Only a run that worked \ + is kept; do the task again, cleanly, and save that" + ), Refusal::RunFailed { step, tool, why } => write!( f, "not learned: the run failed at call {step} ({tool}): {why}. Only a run \ @@ -161,15 +179,25 @@ pub struct Cleaned { /// Keyboard steps that named no window and were given the window of the step /// before them. pub inferred_window: usize, + /// The steps whose text was taken out and made a parameter the run must be + /// given ([`super::params`]). + pub parameters: Vec, } +/// The names incu's tools are recorded under as `mcp____`: the key +/// incode's client configs register it by (`desktop`), the name `incu` itself +/// answers to, and its protocol name, with the underscore a client makes of the +/// hyphen. Nothing else is taken for this server. +pub const OWN_SERVERS: &[&str] = &["desktop", "incu", "incode-desktop", "incode_desktop"]; + /// The desktop tool a recorded name is, if it is one. /// /// A recorded name is the plain tool name (incode's own tools), or an MCP -/// name `mcp____`. For the second form the SERVER must look -/// like this one (its name contains `desktop` or `incu`), because other MCP -/// servers have tools called `key` or `scroll` too and a routine must never -/// replay someone else's. +/// name `mcp____`. For the second form the SERVER must be one of +/// incu's own names ([`OWN_SERVERS`]), exactly: other MCP servers have tools +/// called `key` or `scroll` too, and a server that merely has "desktop" in its +/// name (a remote-desktop tool, a screen sharer) is not this one. A routine must +/// never replay someone else's tool. #[must_use] pub fn desktop_tool(recorded: &str) -> Option<&str> { let known = |name: &str| OBSERVING.contains(&name) || ACTING.contains(&name); @@ -177,8 +205,7 @@ pub fn desktop_tool(recorded: &str) -> Option<&str> { None => known(recorded).then_some(recorded), Some(rest) => { let (server, tool) = rest.split_once("__")?; - let server = server.to_lowercase(); - ((server.contains("desktop") || server.contains("incu")) && known(tool)).then_some(tool) + (OWN_SERVERS.contains(&server) && known(tool)).then_some(tool) } } } @@ -362,7 +389,36 @@ pub(super) fn wait_failure(answer: &str) -> Option { /// A [`Refusal`]: the phrase, a failed call, a step that cannot be replayed, an /// empty result or too many steps. pub fn learn_from_calls(phrase: &str, calls: &[Call]) -> Result { + learn_with(phrase, calls, &|_| false) +} + +/// As [`learn_from_calls`], with `is_secret` asked about the raw selector of a +/// control that no name condemns: the desktop's own report that it is a +/// password field (see [`super::params`]). +/// +/// # Errors +/// +/// As [`learn_from_calls`]. +pub fn learn_with( + phrase: &str, + calls: &[Call], + is_secret: &dyn Fn(&str) -> bool, +) -> Result { check_phrase(phrase)?; + // ANY call that failed is a run that failed: a shell command that errored, a + // file that could not be read, a looking call that was refused. Not only the + // desktop calls: what went wrong there may be why the desktop calls that + // followed did what they did, and a routine made of them is not what the + // task needed. + for (index, call) in calls.iter().enumerate() { + if let Some(why) = &call.error { + return Err(Refusal::RunFailed { + step: index + 1, + tool: call.tool.clone(), + why: first_line(why), + }); + } + } let mut steps = Vec::new(); let mut dropped = 0; let mut skipped = Vec::new(); @@ -380,13 +436,6 @@ pub fn learn_from_calls(phrase: &str, calls: &[Call]) -> Result Result MAX_STEPS { return Err(Refusal::TooLong(steps.len())); } + // After every step is known replayable, and last: the text of a secure field + // or a redacted recording is taken out of the steps before they go anywhere. + let parameters = parameterise(&mut steps, is_secret); Ok(Cleaned { steps, dropped, skipped, inferred_window, + parameters, }) } +/// Whether a record that was read says its own run did not work. +/// +/// # Errors +/// +/// [`Refusal::RecordFailed`] when it does. +pub fn check_record(record: &Record) -> Result<(), Refusal> { + match &record.failed { + Some(why) => Err(Refusal::RecordFailed { + id: record.id.clone(), + why: first_line(why), + }), + None => Ok(()), + } +} + /// The window part of a step's selector (or its `in_window`), as text, when it /// names a window. fn window_part_of(step: &Step) -> Option { @@ -737,15 +805,118 @@ mod tests { assert!(learn(&[wait("{\n \"result\": \"found\"\n}")]).is_ok()); } + /// A run is failed when ANY call in it failed: a looking call that was + /// refused, a shell command that exited non-zero, a file that was not there. #[test] - fn a_failed_looking_call_does_not_fail_the_run() { - let cleaned = learn(&[ + fn a_failed_call_of_any_kind_is_a_failed_run() { + let ok = sel_click("role=button;name=Settings"); + // The case that was missed: [Bash fails, incu key ok]. + let key = Call::ok( + "mcp__desktop__key", + json!({"keys": ["enter"], "in_window": IN}), + ); + let why = learn(&[ + Call::ok("Bash", json!({"command": "false"})).failed("exit code 1"), + key.clone(), + ]) + .unwrap_err(); + match &why { + Refusal::RunFailed { step, tool, why } => { + assert_eq!((*step, tool.as_str()), (1, "Bash")); + assert_eq!(why, "exit code 1"); + } + other => panic!("{other:?}"), + } + assert!(why.to_string().contains("call 1 (Bash)"), "{why}"); + // A looking call that failed, a call that is no desktop tool at all, and + // an error after the last desktop step. + for failing in [ Call::ok("ui_tree", json!({})).failed("window has no tree"), - sel_click("role=button;name=Settings"), + Call::ok("mcp__memory__recall_index", json!({})).failed("unknown scope"), + Call::ok("Read", json!({"file_path": "x"})).failed("no such file"), + ] { + assert!( + matches!( + learn(&[ok.clone(), failing.clone()]), + Err(Refusal::RunFailed { .. }) + ), + "{failing:?}" + ); + assert!(matches!( + learn(&[failing, ok.clone()]), + Err(Refusal::RunFailed { .. }) + )); + } + // And the same calls that all worked are learned. + assert!(learn(&[Call::ok("Bash", json!({"command": "true"})), key]).is_ok()); + } + + #[test] + fn a_record_whose_own_outcome_says_it_failed_is_not_learned() { + let record = Record::new("w-1", 1, vec![sel_click("role=button;name=Settings")]); + assert!(check_record(&record).is_ok()); + let failed = check_record(&record.failing("status is \"failed\"\nmore")).unwrap_err(); + assert_eq!( + failed, + Refusal::RecordFailed { + id: "w-1".to_string(), + why: "status is \"failed\"".to_string() + } + ); + assert!(failed.to_string().contains("w-1"), "{failed}"); + } + + /// Only incu's own servers: a name with "desktop" in it is not enough. + #[test] + fn another_servers_tools_are_never_taken_for_incus() { + for foreign in [ + "mcp__remote_desktop__key", + "mcp__desktop_sharing__click_element", + "mcp__my-incu-fork__key", + "mcp__Desktop__key", + "mcp__desktopx__wait_for", + "mcp__incu2__wait_for", + ] { + assert_eq!(desktop_tool(foreign), None, "{foreign}"); + } + for own in OWN_SERVERS { + let name = format!("mcp__{own}__wait_for"); + assert_eq!(desktop_tool(&name), Some("wait_for"), "{name}"); + } + // Left out of a routine and reported, as any other call that is not ours. + let cleaned = learn(&[ + Call::ok("mcp__remote_desktop__key", json!({"keys": ["enter"]})), + sel_click("role=button;name=New"), ]) .unwrap(); assert_eq!(cleaned.steps.len(), 1); - assert_eq!(cleaned.dropped, 1); + assert_eq!(cleaned.skipped, ["mcp__remote_desktop__key"]); + } + + #[test] + fn typing_into_a_password_field_is_learned_as_a_parameter() { + let cleaned = learn(&[ + sel_click("role=edit;name=Password"), + Call::ok( + "type_text", + json!({"text": "hunter2-secret", "in_window": IN}), + ), + ]) + .unwrap(); + assert_eq!(cleaned.parameters.len(), 1); + assert_eq!(cleaned.steps[1].args["text"], "{{password}}"); + assert!(!format!("{cleaned:?}").contains("hunter2")); + // The desktop's own word that a control is a password field counts too. + let by_desktop = learn_with( + "sign in", + &[ + sel_click("role=edit;name=Code"), + Call::ok("type_text", json!({"text": "482913", "in_window": IN})), + ], + &|raw: &str| raw.contains("name=Code"), + ) + .unwrap(); + assert_eq!(by_desktop.steps[1].args["text"], "{{code}}"); } #[test] diff --git a/crates/desktop/src/routine/mod.rs b/crates/desktop/src/routine/mod.rs index 4c0129b..d055c3a 100644 --- a/crates/desktop/src/routine/mod.rs +++ b/crates/desktop/src/routine/mod.rs @@ -24,7 +24,8 @@ //! `atticus.py`, "learned shortcuts"), kept as simple as it is there: //! //! - per routine: the cleaned steps, the trigger phrase, the date, when it was -//! last used, and whether the user taught it by name; +//! last used, and whether the person kept it (`incu routine keep`: no tool can +//! set it); //! - a phrase matches when its WORD SET is the routine's, in any order, with //! the filler words ignored ([`words`]); //! - nothing is learned when the phrase says "this" or "that", when the run @@ -47,6 +48,8 @@ use serde::{Deserialize, Serialize}; use serde_json::Value; pub mod learn; +pub mod params; +pub mod pictures; pub mod replay; pub mod source; pub mod store; @@ -65,10 +68,22 @@ pub const MAX_STEPS: usize = 100; /// The most routines the file holds; saving a new one past this is refused /// rather than silently dropping an old one. -pub const MAX_ROUTINES: usize = 200; +pub const MAX_ROUTINES: usize = 100; -/// The most one step's arguments may take as JSON, in bytes. -pub const MAX_STEP_BYTES: usize = 16 * 1024; +/// The most one step's arguments may take as JSON, in bytes. Typed text is the +/// bulk of it; a routine that types a page is not a shortcut. +pub const MAX_STEP_BYTES: usize = 4 * 1024; + +/// The most one routine may take, by [`Routine::size_bytes`]. With +/// [`MAX_ROUTINES`] of them the file stays under what it may be +/// ([`store::MAX_FILE_BYTES`]), with room for the indentation the file is +/// written with; a test writes the largest file the two allow and reads it back. +pub const MAX_ROUTINE_BYTES: usize = 24 * 1024; + +/// What a step costs beyond its own bytes: the keys, quotes and indentation of +/// the readable file. Counted so that a routine of many tiny steps, which is +/// mostly structure, is not mistaken for a small one. +const STEP_OVERHEAD_BYTES: usize = 160; /// One desktop tool call to run again: the tool and the arguments it was /// given. The same names and arguments the live tools take, so one dispatcher @@ -122,6 +137,28 @@ impl Step { format!("{} {}", self.tool, shown.join(" ")) } } + + /// The step in one line with EVERYTHING it types and every selector in + /// full: what `routine_save`, `routine_list` and `routine_match` show, so the + /// person can read exactly what a routine will type and where before it runs. + /// + /// A parameter (`{{password}}`) is shown as that name, which is all the file + /// holds of it. Not for the journal or a failure message: those take + /// [`Step::summary`], which keeps typed text out. + #[must_use] + pub fn review(&self) -> String { + let mut shown = Vec::new(); + if let Some(map) = self.args.as_object() { + for (key, value) in map { + shown.push(format!("{key}={value}")); + } + } + if shown.is_empty() { + self.tool.clone() + } else { + format!("{} {}", self.tool, shown.join(" ")) + } + } } /// A learned routine. @@ -136,12 +173,29 @@ pub struct Routine { pub date: String, /// When it last ran to the end (or was learned), in Unix milliseconds. pub last_used_ms: i64, - /// The user taught it by name. Such a routine never expires. + /// The person asked to keep it (`incu routine keep`). Such a routine never + /// expires. Only the person sets this: no tool takes it as an argument, so an + /// agent cannot exempt what it saved from expiry. A save that replaces a kept + /// routine stays kept. #[serde(default)] pub taught: bool, } impl Routine { + /// What this routine takes of the file, in bytes, as [`MAX_ROUTINE_BYTES`] + /// counts it: its phrase, each step's tool and arguments, and + /// [`STEP_OVERHEAD_BYTES`] for each step. + #[must_use] + pub fn size_bytes(&self) -> usize { + self.phrase.len() + + 64 + + self + .steps + .iter() + .map(|s| s.tool.len() + s.args.to_string().len() + STEP_OVERHEAD_BYTES) + .sum::() + } + /// Whether this routine has gone unused long enough to be forgotten at /// `now_ms`. A routine the user taught never is. A clock that went /// backwards (a `last_used_ms` after `now_ms`) does not expire anything. diff --git a/crates/desktop/src/routine/params.rs b/crates/desktop/src/routine/params.rs new file mode 100644 index 0000000..cafc855 --- /dev/null +++ b/crates/desktop/src/routine/params.rs @@ -0,0 +1,715 @@ +//! Parameters: text a routine is GIVEN each time it runs instead of keeping. +//! +//! # Why +//! +//! The routines file holds what a routine types, so that the person can read it +//! and so that a replay needs no model. Text typed into a password or other +//! secure field must not be on disk at all, and neither must a recording's +//! stand-in for text it redacted (replaying `[REDACTED]` into a login box is a +//! wrong password, not a shortcut). Such a step is made a PARAMETER when the +//! routine is learned: its text is stored as `{{name}}`, `routine_run` must be +//! given `params: {"name": "..."}` for every one, and refuses before step 1 when +//! one is missing. The value is substituted into the arguments of the one call +//! that types it, in memory. It is not written to the routines file or the +//! journal ([`crate::mcp`]'s `describe` shows the length only), and +//! [`Values::scrub`] takes it back out of any message a replay returns. +//! +//! # What makes a step a parameter +//! +//! 1. Its text is a stand-in for text that was removed: [`looks_redacted`], or a +//! value that is not text at all (a recording's `{"redacted": true}`). +//! 2. It goes into a secure field, by the field's own name or automation id +//! ([`is_secret_name`]), or because the desktop reports the control as a +//! password field when the routine is saved. A typing step goes into the +//! control the step before it named. +//! 3. The step already has `{{name}}` in its text: the agent made it one. +//! +//! This is a screen on what the learner can see, not a promise: a secure field +//! whose label does not say so, and that is no longer on screen when the routine +//! is saved, is not found. That is why `routine_save`, `routine_list` and +//! `routine_match` show the full text of every step: the person reads what is +//! kept. + +use std::collections::{BTreeMap, BTreeSet}; + +use serde_json::Value; + +use super::Step; +use crate::selector::{Selector, Text}; + +/// The arguments that carry typed text, by tool. +fn text_key(tool: &str) -> Option<&'static str> { + match tool { + "set_value" => Some("value"), + "type_text" | "paste_text" => Some("text"), + _ => None, + } +} + +/// The longest a parameter's value may be, in characters. +pub const MAX_PARAM_CHARS: usize = 1024; + +/// The most parameters one routine may take. +pub const MAX_PARAMS: usize = 16; + +/// Whether `name` can be a parameter's: a lowercase letter, then lowercase +/// letters, digits and `_`, at most 32. +fn is_name(name: &str) -> bool { + let mut chars = name.chars(); + chars.next().is_some_and(|c| c.is_ascii_lowercase()) + && name.len() <= 32 + && chars.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_') +} + +/// The parameter names in `text`, in order of appearance (`{{name}}`). +#[must_use] +pub fn tokens(text: &str) -> Vec { + let mut found = Vec::new(); + let mut rest = text; + while let Some(start) = rest.find("{{") { + let after = &rest[start + 2..]; + match after.find("}}") { + Some(end) if is_name(&after[..end]) => { + found.push(after[..end].to_string()); + rest = &after[end + 2..]; + } + _ => rest = after, + } + } + found +} + +/// The parameter names a step takes. +#[must_use] +pub fn names_of(step: &Step) -> Vec { + text_key(&step.tool) + .and_then(|key| step.args.get(key)) + .and_then(Value::as_str) + .map(tokens) + .unwrap_or_default() +} + +/// The parameter names a routine takes, each once, in order of first use. +#[must_use] +pub fn names(steps: &[Step]) -> Vec { + let mut seen = BTreeSet::new(); + steps + .iter() + .flat_map(names_of) + .filter(|n| seen.insert(n.clone())) + .collect() +} + +/// The values a run was given, by name. +#[derive(Debug, Default, Clone)] +pub struct Values(BTreeMap); + +impl Values { + /// The values `given` (the `params` argument of `routine_run`) holds for the + /// parameters `steps` take. + /// + /// # Errors + /// + /// A parameter has no value, a value is not text or is too long, or a value + /// is given for a name the routine does not take (a typo that would + /// otherwise type nothing where something was meant). + pub fn bind(steps: &[Step], given: Option<&Value>) -> Result { + let needed = names(steps); + let map = match given { + None | Some(Value::Null) => serde_json::Map::new(), + Some(Value::Object(map)) => map.clone(), + Some(_) => return Err("params must be an object of name to text".to_string()), + }; + let mut values = BTreeMap::new(); + for (name, value) in &map { + if !needed.contains(name) { + return Err(format!( + "this routine takes no parameter named {name:?}{}", + listing(&needed) + )); + } + let Some(text) = value.as_str() else { + return Err(format!("params.{name} must be text")); + }; + if text.chars().count() > MAX_PARAM_CHARS { + return Err(format!( + "params.{name} is over {MAX_PARAM_CHARS} characters" + )); + } + values.insert(name.clone(), text.to_string()); + } + let missing: Vec<&String> = needed.iter().filter(|n| !values.contains_key(*n)).collect(); + if !missing.is_empty() { + return Err(format!( + "routine_run needs params for: {}. Nothing was done. These are the secrets \ + and redacted texts the routine does not keep; pass them as \ + params: {{\"name\": \"...\"}}", + missing + .iter() + .map(|n| n.as_str()) + .collect::>() + .join(", ") + )); + } + Ok(Values(values)) + } + + /// Whether there are no values. + #[must_use] + pub fn is_empty(&self) -> bool { + self.0.is_empty() + } + + /// `step` with its `{{name}}` parts replaced by the values. In one pass: a + /// value that itself contains `{{x}}` is typed as it is. + #[must_use] + pub fn fill(&self, step: &Step) -> Step { + let Some(key) = text_key(&step.tool) else { + return step.clone(); + }; + let mut filled = step.clone(); + if let Some(text) = step.args.get(key).and_then(Value::as_str) { + filled.args[key] = Value::String(self.render(text)); + } + filled + } + + fn render(&self, template: &str) -> String { + let mut out = String::new(); + let mut rest = template; + while let Some(start) = rest.find("{{") { + out.push_str(&rest[..start]); + let after = &rest[start + 2..]; + match after.find("}}") { + Some(end) if is_name(&after[..end]) => { + let name = &after[..end]; + match self.0.get(name) { + Some(value) => out.push_str(value), + None => out.push_str(&rest[start..start + 2 + end + 2]), + } + rest = &after[end + 2..]; + } + _ => { + out.push_str("{{"); + rest = after; + } + } + } + out.push_str(rest); + out + } + + /// `text` with every value (of three characters or more) put back to its + /// `{{name}}`, for a message that is returned or journalled. + #[must_use] + pub fn scrub(&self, text: &str) -> String { + let mut out = text.to_string(); + let mut by_length: Vec<(&String, &String)> = self + .0 + .iter() + .filter(|(_, v)| v.chars().count() >= 3) + .collect(); + by_length.sort_by_key(|(_, v)| std::cmp::Reverse(v.len())); + for (name, value) in by_length { + out = out.replace(value.as_str(), &format!("{{{{{name}}}}}")); + } + out + } +} + +fn listing(needed: &[String]) -> String { + if needed.is_empty() { + "; it takes none".to_string() + } else { + format!("; it takes: {}", needed.join(", ")) + } +} + +/// Whether a control's name or automation id says it holds a secret. +/// +/// The long words match anywhere (`PasswordBox`, `confirm_password`); the short +/// ones only as whole words, so a PIN field is found and a `Pinterest` button is +/// not. +#[must_use] +pub fn is_secret_name(text: &str) -> bool { + const ANYWHERE: &[&str] = &[ + "password", + "passwd", + "passcode", + "passphrase", + "secret", + "credential", + "apikey", + "api key", + "api_key", + "private key", + "privatekey", + "security code", + "securitycode", + "verification code", + "one-time", + "one time", + "card number", + "cardnumber", + "social security", + ]; + const AS_WORDS: &[&str] = &[ + "pin", "otp", "pwd", "cvv", "cvc", "token", "ssn", "2fa", "mfa", + ]; + let lowered = text.to_lowercase(); + if ANYWHERE.iter().any(|w| lowered.contains(w)) { + return true; + } + lowered + .split(|c: char| !c.is_alphanumeric()) + .any(|w| AS_WORDS.contains(&w)) +} + +/// Whether the control a selector names is a secret field by what the selector +/// says of it: its name or automation id. +#[must_use] +pub fn selector_is_secret(raw: &str) -> bool { + let Ok(selector) = Selector::parse(raw) else { + return false; + }; + let named = match &selector.name { + Some(Text::Exact(n) | Text::Contains(n)) => is_secret_name(n), + None => false, + }; + named + || selector + .automation_id + .as_deref() + .is_some_and(is_secret_name) +} + +/// Whether `text` is a stand-in for text that was removed from a recording: +/// a placeholder (`[REDACTED]`, ``, `****`, `[12 chars]`), or a digest +/// of the value (`sha256:...`, a bare 64-digit hex). +#[must_use] +pub fn looks_redacted(text: &str) -> bool { + let trimmed = text.trim(); + let lowered = trimmed.to_lowercase(); + if lowered.is_empty() { + return false; + } + if lowered.contains("redact") { + return true; + } + if trimmed.chars().count() >= 3 && trimmed.chars().all(|c| "*•●∗·".contains(c)) { + return true; + } + for prefix in ["sha256:", "sha-256:", "sha512:", "sha1:", "md5:", "blake3:"] { + if lowered.starts_with(prefix) { + return true; + } + } + if lowered.len() == 64 && lowered.chars().all(|c| c.is_ascii_hexdigit()) { + return true; + } + let bracketed = [('[', ']'), ('<', '>'), ('(', ')')] + .iter() + .any(|(open, close)| lowered.starts_with(*open) && lowered.ends_with(*close)); + if bracketed { + let inner = lowered[1..lowered.len() - 1].trim(); + const MARKS: &[&str] = &[ + "secret", + "hidden", + "masked", + "removed", + "omitted", + "filtered", + "sensitive", + "private", + "scrubbed", + "censored", + "withheld", + "elided", + "stripped", + "digest", + "sha", + "hash", + "value", + "password", + "token", + ]; + if inner.chars().count() <= 48 && MARKS.iter().any(|m| inner.contains(m)) { + return true; + } + let words: Vec<&str> = inner.split_whitespace().collect(); + if words.len() == 2 + && words[0].chars().all(|c| c.is_ascii_digit()) + && matches!( + words[1], + "char" | "chars" | "character" | "characters" | "bytes" + ) + { + return true; + } + } + false +} + +/// Why a step was made a parameter. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Why { + /// It types into a secure field. + SecureField, + /// The recording kept a placeholder or a digest in place of the text. + Redacted, +} + +/// One step made a parameter. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Made { + /// The parameter's name. + pub name: String, + /// The step, 1-based. + pub step: usize, + /// Why. + pub why: Why, +} + +/// Make the steps that type a secret or a redacted text into parameters, in +/// place, and say which. `is_secret` is asked about the raw selector of a +/// control the name does not already condemn: the desktop's own report of a +/// password field. +pub fn parameterise(steps: &mut [Step], is_secret: &dyn Fn(&str) -> bool) -> Vec { + let mut made = Vec::new(); + let mut taken: BTreeSet = names(steps).into_iter().collect(); + // The control the step before named, for a typing step that has none of its + // own (`type_text` goes to whatever has focus, which the last click gave it). + let mut last_control: Option = None; + for (index, step) in steps.iter_mut().enumerate() { + let own = step + .args + .get("selector") + .and_then(Value::as_str) + .filter(|raw| Selector::parse(raw).is_ok_and(|s| s.has_element_part())) + .map(str::to_string); + let control = if step.tool == "set_value" { + own.clone() + } else { + last_control.clone() + }; + if let Some(own) = own { + last_control = Some(own); + } + let Some(key) = text_key(&step.tool) else { + continue; + }; + let value = step.args.get(key).cloned().unwrap_or(Value::Null); + let why = match &value { + Value::String(text) if !tokens(text).is_empty() => continue, + Value::String(text) if looks_redacted(text) => Some(Why::Redacted), + Value::String(_) => control + .as_deref() + .is_some_and(|raw| selector_is_secret(raw) || is_secret(raw)) + .then_some(Why::SecureField), + Value::Null => None, + // A recording's `{"redacted": true}` or the like: not text. + _ => Some(Why::Redacted), + }; + let Some(why) = why else { continue }; + let name = fresh_name( + match why { + Why::SecureField => "password", + Why::Redacted => "text", + }, + control.as_deref(), + &mut taken, + ); + if let Some(map) = step.args.as_object_mut() { + map.insert(key.to_string(), Value::String(format!("{{{{{name}}}}}"))); + } + made.push(Made { + name, + step: index + 1, + why, + }); + } + made +} + +/// A parameter name no other has: the control's own name when it makes a good +/// one, else `fallback`, with a number to tell it from the others. +fn fresh_name(fallback: &str, control: Option<&str>, taken: &mut BTreeSet) -> String { + let from_control = control + .and_then(|raw| Selector::parse(raw).ok()) + .and_then(|s| match s.name { + Some(Text::Exact(n) | Text::Contains(n)) => Some(n), + None => None, + }) + .map(|n| { + let mut slug: String = n + .to_lowercase() + .chars() + .map(|c| if c.is_ascii_alphanumeric() { c } else { '_' }) + .collect(); + slug.truncate(24); + slug.trim_matches('_').to_string() + }) + .filter(|s| is_name(s)); + let base = from_control.unwrap_or_else(|| fallback.to_string()); + let mut name = base.clone(); + let mut n = 2; + while !taken.insert(name.clone()) { + name = format!("{base}_{n}"); + n += 1; + } + name +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + fn no_probe(_: &str) -> bool { + false + } + + #[test] + fn placeholders_are_found_only_when_they_are_well_formed_names() { + assert_eq!(tokens("a {{password}} b {{pin_2}}"), ["password", "pin_2"]); + assert!(tokens("{{Password}} {{1x}} {{}} {{a b}} {{ x }} {{x").is_empty()); + assert_eq!(tokens("{{{{x}}"), ["x"]); + } + + #[test] + fn a_routines_parameters_are_its_placeholders_once_each_in_order() { + let steps = [ + Step::new("type_text", json!({"text": "{{user}} / {{password}}"})), + Step::new( + "set_value", + json!({"value": "{{password}}", "selector": "x"}), + ), + Step::new("key", json!({"keys": ["enter"]})), + ]; + assert_eq!(names(&steps), ["user", "password"]); + } + + #[test] + fn a_run_must_give_every_parameter_and_only_the_ones_there_are() { + let steps = [Step::new("type_text", json!({"text": "{{password}}"}))]; + let missing = Values::bind(&steps, None).unwrap_err(); + assert!( + missing.contains("password") && missing.contains("Nothing was done"), + "{missing}" + ); + let unknown = + Values::bind(&steps, Some(&json!({"password": "a", "pasword": "b"}))).unwrap_err(); + assert!( + unknown.contains("pasword") && unknown.contains("password"), + "{unknown}" + ); + assert!(Values::bind(&steps, Some(&json!({"password": 5}))).is_err()); + assert!(Values::bind(&steps, Some(&json!("x"))).is_err()); + let long = "x".repeat(MAX_PARAM_CHARS + 1); + assert!(Values::bind(&steps, Some(&json!({"password": long}))).is_err()); + let bound = Values::bind(&steps, Some(&json!({"password": "hunter2"}))).unwrap(); + assert!(!bound.is_empty()); + // A routine with none needs none, and says so when one is offered. + let plain = [Step::new("key", json!({"keys": ["enter"]}))]; + assert!(Values::bind(&plain, None).unwrap().is_empty()); + let extra = Values::bind(&plain, Some(&json!({"x": "y"}))).unwrap_err(); + assert!(extra.contains("takes none"), "{extra}"); + } + + #[test] + fn a_value_is_substituted_in_one_pass_and_scrubbed_from_what_comes_back() { + let steps = [Step::new("type_text", json!({"text": "pw={{password}}!"}))]; + let values = Values::bind(&steps, Some(&json!({"password": "a{{password}}b"}))).unwrap(); + let filled = values.fill(&steps[0]); + assert_eq!(filled.args["text"], "pw=a{{password}}b!"); + // The stored step is untouched. + assert_eq!(steps[0].args["text"], "pw={{password}}!"); + let message = "could not type hunter2 into the window"; + let values = Values::bind(&steps, Some(&json!({"password": "hunter2"}))).unwrap(); + assert_eq!( + values.scrub(message), + "could not type {{password}} into the window" + ); + // Two characters are not scrubbed: they would eat the message. + let short = Values::bind(&steps, Some(&json!({"password": "ab"}))).unwrap(); + assert_eq!(short.scrub("a cab"), "a cab"); + } + + #[test] + fn a_field_whose_name_says_secret_is_secret_and_a_lookalike_is_not() { + for yes in [ + "Password", + "Confirm password", + "PasswordBox", + "new_passcode", + "PIN", + "Enter your PIN", + "OTP code", + "API key", + "Security code", + "Token", + "CVV", + ] { + assert!(is_secret_name(yes), "{yes}"); + } + for no in [ + "Username", + "Search", + "Pinterest", + "Spinner", + "Subject", + "Tokyo", + "Body", + ] { + assert!(!is_secret_name(no), "{no}"); + } + assert!(selector_is_secret( + "app=login;window=Sign in;role=edit;name=Password" + )); + assert!(selector_is_secret( + "app=login;window=Sign in;role=edit;aid=txtPassword" + )); + assert!(!selector_is_secret( + "app=login;window=Sign in;role=edit;name=Email" + )); + assert!(!selector_is_secret("not a selector")); + } + + #[test] + fn what_a_recording_keeps_in_place_of_text_is_recognised() { + for yes in [ + "[REDACTED]", + "", + "***", + "••••••••", + "[secret]", + "", + "[12 chars]", + "<8 characters>", + "sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "[value removed]", + "(redacted by watchman)", + ] { + assert!(looks_redacted(yes), "{yes}"); + } + for no in [ + "hello", + "", + "ab", + "two words", + "[ ] todo", + "bold", + "(optional)", + "12 characters long", + "deadbeef", + ] { + assert!(!looks_redacted(no), "{no}"); + } + } + + #[test] + fn typing_into_a_password_field_becomes_a_parameter_and_the_secret_is_not_kept() { + let mut steps = vec![ + Step::new( + "set_value", + json!({"selector": "app=login;window=Sign in;role=edit;name=Email", "value": "me@example.com"}), + ), + Step::new( + "click_element", + json!({"selector": "app=login;window=Sign in;role=edit;name=Password"}), + ), + Step::new( + "type_text", + json!({"text": "hunter2-secret", "in_window": "app=login;window=Sign in"}), + ), + ]; + let made = parameterise(&mut steps, &no_probe); + assert_eq!( + made, + [Made { + name: "password".to_string(), + step: 3, + why: Why::SecureField + }] + ); + assert_eq!(steps[2].args["text"], "{{password}}"); + // The email is ordinary text and stays. + assert_eq!(steps[0].args["value"], "me@example.com"); + assert!(!serde_json::to_string(&steps).unwrap().contains("hunter2")); + assert_eq!(names(&steps), ["password"]); + } + + #[test] + fn a_set_value_on_a_secret_field_is_a_parameter_named_after_it() { + let mut steps = vec![Step::new( + "set_value", + json!({"selector": "app=vault;role=edit;name=Token", "value": "hunter2-secret"}), + )]; + let made = parameterise(&mut steps, &no_probe); + assert_eq!(made[0].name, "token"); + assert_eq!(steps[0].args["value"], "{{token}}"); + } + + #[test] + fn the_desktops_own_report_of_a_password_field_counts_when_the_name_does_not_say() { + let mut steps = vec![Step::new( + "set_value", + json!({"selector": "app=bank;window=Login;role=edit;name=Enter it here", "value": "s3cret"}), + )]; + let seen_as_password = |raw: &str| raw.contains("Enter it here"); + let made = parameterise(&mut steps, &seen_as_password); + assert_eq!(made.len(), 1); + assert_eq!(made[0].why, Why::SecureField); + assert!(!steps[0].args.to_string().contains("s3cret")); + } + + #[test] + fn a_redacted_recording_text_is_a_parameter_not_a_replayed_placeholder() { + let mut steps = vec![ + Step::new( + "set_value", + json!({"selector": "app=mail;window~=Compose;role=edit;name=Body", "value": "[REDACTED]"}), + ), + Step::new( + "type_text", + json!({"text": {"redacted": true, "sha256": "ab"}, "in_window": "app=mail"}), + ), + Step::new( + "type_text", + json!({"text": "hello", "in_window": "app=mail"}), + ), + ]; + let made = parameterise(&mut steps, &no_probe); + assert_eq!(made.len(), 2); + assert!(made.iter().all(|m| m.why == Why::Redacted)); + assert_eq!(steps[0].args["value"], "{{body}}"); + // Named after the control it went into, numbered to stay apart. + assert_eq!(steps[1].args["text"], "{{body_2}}"); + assert_eq!(steps[2].args["text"], "hello", "ordinary text stays"); + assert_eq!(names(&steps), ["body", "body_2"]); + } + + #[test] + fn two_secrets_get_two_names_and_an_agents_own_placeholder_is_left_alone() { + let mut steps = vec![ + Step::new( + "set_value", + json!({"selector": "app=a;window=W;role=edit;name=Password", "value": "one"}), + ), + Step::new( + "set_value", + json!({"selector": "app=a;window=W;role=edit;name=Password", "value": "two"}), + ), + Step::new( + "type_text", + json!({"text": "{{mine}}", "in_window": "app=a"}), + ), + ]; + let made = parameterise(&mut steps, &no_probe); + assert_eq!(made[0].name, "password"); + assert_eq!(made[1].name, "password_2"); + assert_eq!(made.len(), 2, "{{mine}} is the agent's, not made"); + assert_eq!(names(&steps), ["password", "password_2", "mine"]); + } +} diff --git a/crates/desktop/src/routine/pictures.rs b/crates/desktop/src/routine/pictures.rs new file mode 100644 index 0000000..49a6821 --- /dev/null +++ b/crates/desktop/src/routine/pictures.rs @@ -0,0 +1,236 @@ +//! The pictures of the last replay: one before and one after each step. +//! +//! # What and why +//! +//! A replay that fails is only useful if the agent that redoes the task, and the +//! person reading over its shoulder, can see what the screen looked like. So +//! each step of `routine_run` gets one picture before it and one after it, taken +//! with incu's own capture (the window the step is for, else the screen), and a +//! failure names the failed step's two pictures by path. There is no video and no +//! ffmpeg: a step is a still. +//! +//! # Kept small, last run only +//! +//! The pictures are files in one folder beside the routines file +//! ([`super::store::Store::pictures_dir`]). The next run clears the folder before +//! it starts, so only the last run is kept. Within a run the folder is held under +//! [`MAX_TOTAL_BYTES`]: when a new picture would pass it the OLDEST picture is +//! deleted, so the pictures of the step that failed (the newest) are always the +//! ones that survive. One picture over [`MAX_PICTURE_BYTES`] is not kept. +//! +//! # Never sent anywhere +//! +//! A picture is written to disk and its path returned. It is not sent to the +//! memory engine's OCR (or to anything), now or later: reading a screen with a +//! model is the job of the agent that is told the path, not of the replay. +//! +//! Called by `mcp::routine_tool`'s live runner, through +//! [`super::replay::Runner::look`]. + +use std::collections::VecDeque; +use std::path::{Path, PathBuf}; + +/// The most one picture may be, in bytes. +pub const MAX_PICTURE_BYTES: u64 = 4 * 1024 * 1024; + +/// The most the pictures of one run may take, in bytes. +pub const MAX_TOTAL_BYTES: u64 = 48 * 1024 * 1024; + +/// Which of a step's two pictures. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum When { + /// Before the step runs. + Before, + /// After it has run and the screen has had its moment to settle. + After, +} + +impl When { + fn word(self) -> &'static str { + match self { + When::Before => "before", + When::After => "after", + } + } +} + +/// The folder of one run's pictures, as it fills. +#[derive(Debug)] +pub struct Pictures { + dir: PathBuf, + kept: VecDeque<(PathBuf, u64)>, + total: u64, + max_total: u64, +} + +impl Pictures { + /// Start a run's pictures in `dir`: make it, and clear what the last run left + /// there (only the files this module names: anything else in the folder is + /// not touched). + /// + /// # Errors + /// + /// The folder cannot be made or cleared. The replay goes on without pictures. + pub fn begin(dir: PathBuf) -> Result { + Self::begin_within(dir, MAX_TOTAL_BYTES) + } + + fn begin_within(dir: PathBuf, max_total: u64) -> Result { + std::fs::create_dir_all(&dir).map_err(|e| format!("cannot make {}: {e}", dir.display()))?; + let entries = + std::fs::read_dir(&dir).map_err(|e| format!("cannot read {}: {e}", dir.display()))?; + for entry in entries.flatten() { + let name = entry.file_name().to_string_lossy().to_string(); + if is_ours(&name) { + std::fs::remove_file(entry.path()) + .map_err(|e| format!("cannot clear {}: {e}", entry.path().display()))?; + } + } + Ok(Self { + dir, + kept: VecDeque::new(), + total: 0, + max_total, + }) + } + + /// The folder. + #[must_use] + pub fn dir(&self) -> &Path { + &self.dir + } + + /// How many pictures are kept now. + #[must_use] + pub fn count(&self) -> usize { + self.kept.len() + } + + /// Keep `bytes` (an image of type `extension`) as the `when` picture of step + /// `n` (1-based), deleting the oldest pictures if the run would pass its + /// total. The path it is kept at. + /// + /// # Errors + /// + /// The picture is over [`MAX_PICTURE_BYTES`] or cannot be written. + pub fn keep( + &mut self, + n: usize, + when: When, + extension: &str, + bytes: &[u8], + ) -> Result { + let size = bytes.len() as u64; + if size > MAX_PICTURE_BYTES { + return Err(format!( + "the picture is {size} bytes, over the {MAX_PICTURE_BYTES} one may be" + )); + } + let path = self + .dir + .join(format!("step-{n:03}-{}.{extension}", when.word())); + std::fs::write(&path, bytes) + .map_err(|e| format!("cannot write {}: {e}", path.display()))?; + self.kept.push_back((path.clone(), size)); + self.total += size; + while self.total > self.max_total && self.kept.len() > 1 { + if let Some((old, old_size)) = self.kept.pop_front() { + let _ = std::fs::remove_file(old); + self.total -= old_size; + } + } + Ok(path) + } +} + +/// Whether `name` is a picture file this module made. +fn is_ours(name: &str) -> bool { + name.starts_with("step-") && (name.ends_with(".jpg") || name.ends_with(".png")) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn folder(name: &str) -> PathBuf { + let dir = std::env::temp_dir().join(format!("incu-pictures-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + dir + } + + fn files(dir: &Path) -> Vec { + let mut names: Vec = std::fs::read_dir(dir) + .unwrap() + .map(|e| e.unwrap().file_name().to_string_lossy().to_string()) + .collect(); + names.sort(); + names + } + + #[test] + fn a_step_has_a_picture_before_and_one_after_named_for_it() { + let dir = folder("names"); + let mut pictures = Pictures::begin(dir.clone()).unwrap(); + let before = pictures.keep(2, When::Before, "jpg", b"b").unwrap(); + let after = pictures.keep(2, When::After, "jpg", b"a").unwrap(); + assert!(before.ends_with("step-002-before.jpg"), "{before:?}"); + assert!(after.ends_with("step-002-after.jpg"), "{after:?}"); + assert_eq!(std::fs::read(&before).unwrap(), b"b"); + assert_eq!(pictures.count(), 2); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn only_the_last_run_is_kept_and_only_our_own_files_are_cleared() { + let dir = folder("last-run"); + let mut first = Pictures::begin(dir.clone()).unwrap(); + first.keep(1, When::Before, "jpg", b"old").unwrap(); + first.keep(7, When::After, "png", b"old").unwrap(); + std::fs::write(dir.join("notes.txt"), "mine").unwrap(); + let second = Pictures::begin(dir.clone()).unwrap(); + assert_eq!( + files(&dir), + ["notes.txt"], + "the old run is gone, the note is not" + ); + assert_eq!(second.count(), 0); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn the_run_is_held_under_its_total_by_dropping_the_oldest_pictures() { + let dir = folder("bounded"); + let mut pictures = Pictures::begin_within(dir.clone(), 10).unwrap(); + for n in 1..=5 { + pictures.keep(n, When::Before, "jpg", b"1234").unwrap(); + pictures.keep(n, When::After, "jpg", b"5678").unwrap(); + } + let left = files(&dir); + assert!(left.len() <= 2, "{left:?}"); + // What is left is the newest: the step that failed. + assert_eq!(left, ["step-005-after.jpg", "step-005-before.jpg"]); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_picture_over_its_own_cap_is_refused_and_leaves_nothing() { + let dir = folder("huge"); + let mut pictures = Pictures::begin(dir.clone()).unwrap(); + let huge = vec![0u8; MAX_PICTURE_BYTES as usize + 1]; + assert!(pictures.keep(1, When::After, "jpg", &huge).is_err()); + assert!(files(&dir).is_empty()); + assert_eq!(pictures.count(), 0); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_folder_that_cannot_be_made_is_an_error_not_a_panic() { + let dir = folder("blocked"); + std::fs::create_dir_all(&dir).unwrap(); + let file = dir.join("a-file"); + std::fs::write(&file, "x").unwrap(); + // A folder under a file cannot exist. + assert!(Pictures::begin(file.join("pictures")).is_err()); + let _ = std::fs::remove_dir_all(&dir); + } +} diff --git a/crates/desktop/src/routine/replay.rs b/crates/desktop/src/routine/replay.rs index 071ea09..7e72514 100644 --- a/crates/desktop/src/routine/replay.rs +++ b/crates/desktop/src/routine/replay.rs @@ -18,18 +18,32 @@ //! back as a timeout or a stop, because that is what "the screen is not as the //! routine expects" looks like. //! +//! Before and after each step the runner is asked for a picture +//! ([`Runner::look`], kept by [`super::pictures`]); a failure names the failed +//! step's two. A picture that cannot be taken is no reason to fail a replay. +//! //! Every step is checked with [`super::learn::check_step`] BEFORE the first one //! runs, so a routines file edited by hand cannot get a coordinate click or an //! `eN` id past the rule the learner applies, and a bad step means nothing ran. use std::fmt; +use std::path::PathBuf; use super::learn::{check_step, wait_failure}; +use super::pictures::When; use super::{Routine, Step}; /// What runs one step. The server's own implements it over its gates; tests /// implement it over a script. pub trait Runner { + /// A picture of the screen, or of the window the step is for, before or + /// after `step` (step `n`): the path it was kept at. `None` when there is no + /// picture (the default, and a screen that could not be captured: a + /// replay does not fail for want of one). + fn look(&mut self, _n: usize, _step: &Step, _when: When) -> Option { + None + } + /// Run `step` as step `n`. `Ok` is what the tool answered, as text; `Err` is /// its refusal or error, in its own words. /// @@ -75,6 +89,12 @@ pub struct Failure { pub why: String, /// The step was refused before anything ran (so no step ran). pub before_start: bool, + /// Where the picture taken before the failed step is kept, when there is + /// one. + pub before_picture: Option, + /// Where the picture taken after it is kept: what the screen showed when it + /// failed. + pub after_picture: Option, } impl fmt::Display for Failure { @@ -86,6 +106,8 @@ impl fmt::Display for Failure { summary, why, before_start, + before_picture, + after_picture, } = self; if *before_start { write!( @@ -101,6 +123,19 @@ impl fmt::Display for Failure { step - 1 )?; } + match (before_picture, after_picture) { + (None, None) => {} + (before, after) => { + write!(f, " Pictures of step {step}:")?; + if let Some(path) = before { + write!(f, " before: {path}")?; + } + if let Some(path) = after { + write!(f, " after: {path}")?; + } + write!(f, ".")?; + } + } write!( f, " Do the whole task yourself from the screen as it is now, then routine_save it \ @@ -115,47 +150,81 @@ impl fmt::Display for Failure { /// /// A [`Failure`] naming the step and why: a step the rules refuse (nothing ran), /// a step the tool refused or errored on, or a `wait_for` that timed out. -pub fn replay(routine: &Routine, runner: &mut dyn Runner) -> Result { +pub fn replay(routine: &Routine, runner: &mut dyn Runner) -> Result> { let total = routine.steps.len(); - let failure = |step: usize, why: String, before_start: bool| Failure { - phrase: routine.phrase.clone(), - step, - total, - summary: routine.steps[step - 1].summary(), - why, - before_start, + type Pictures = (Option, Option); + let failure = |step: usize, why: String, before_start: bool, pictures: &Pictures| { + Box::new(Failure { + phrase: routine.phrase.clone(), + step, + total, + summary: routine.steps[step - 1].summary(), + why, + before_start, + before_picture: pictures.0.as_ref().map(|p| p.display().to_string()), + after_picture: pictures.1.as_ref().map(|p| p.display().to_string()), + }) }; + let none: Pictures = (None, None); for (i, step) in routine.steps.iter().enumerate() { - check_step(step, i + 1).map_err(|why| failure(i + 1, why.to_string(), true))?; + check_step(step, i + 1).map_err(|why| failure(i + 1, why.to_string(), true, &none))?; } let mut ran = Vec::with_capacity(total); for (i, step) in routine.steps.iter().enumerate() { let n = i + 1; - let answer = runner.run(n, step).map_err(|why| failure(n, why, false))?; + let before = runner.look(n, step, When::Before); + let answered = runner.run(n, step); + // The screen has had its moment before the picture of it: after every + // step that changes it, the last included. A wait looks for itself. + if step.tool != "wait_for" { + runner.settle(); + } + let pictures = (before, runner.look(n, step, When::After)); + let answer = answered.map_err(|why| failure(n, why, false, &pictures))?; if step.tool == "wait_for" { if let Some(why) = wait_failure(&answer) { - return Err(failure(n, why, false)); + return Err(failure(n, why, false, &pictures)); } } if let Some(why) = unverified_before(step, &answer, routine.steps.get(n)) { - return Err(failure(n, why, false)); + return Err(failure(n, why, false, &pictures)); } ran.push(Ran { summary: step.summary(), answer: first_line(&answer), }); - if step.tool != "wait_for" && n < total { - runner.settle(); - } } Ok(Report { ran }) } -/// The tools whose answer says whether the effect was read back. A click's -/// answer always says "unverified" (a click has no state to read back), so it -/// is not one of them: the next step's own selector, or a `wait_for`, is what -/// checks a click. -const READS_BACK: &[&str] = &["set_value", "element_action"]; +/// The steps whose "unverified" is a reason to stop: the ones that SET A VALUE, +/// where "unverified" means the value may not have taken (`set_value`, and the +/// `element_action`s that set one, [`sets_a_value`]). +/// +/// NOT a click, an `invoke`, a `press` or a `scroll_into_view`: those always +/// answer "unverified" (live, on Windows, every one of them), because a button +/// press has no state to read back. Stopping on it would stop every routine that +/// presses a button; the next step's own selector, or a `wait_for`, is what +/// checks one. +fn reads_back(step: &Step) -> bool { + step.tool == "set_value" || sets_a_value(step) +} + +/// Whether `step` is an `element_action` that sets a value (a spinner's +/// `increment` or `decrement`), as opposed to pressing, invoking or scrolling. +fn sets_a_value(step: &Step) -> bool { + step.tool == "element_action" + && step + .args + .get("action") + .and_then(serde_json::Value::as_str) + .is_some_and(|a| { + matches!( + a.trim().to_ascii_lowercase().as_str(), + "increment" | "decrement" + ) + }) +} /// The steps that act on what the step before them did. const DEPENDS_ON_THE_SCREEN: &[&str] = &[ @@ -177,7 +246,8 @@ const DEPENDS_ON_THE_SCREEN: &[&str] = &[ /// state the routine never reached. fn unverified_before(step: &Step, answer: &str, next: Option<&Step>) -> Option { let next = next.filter(|n| DEPENDS_ON_THE_SCREEN.contains(&n.tool.as_str()))?; - if !READS_BACK.contains(&step.tool.as_str()) || !answer.to_lowercase().contains("unverified") { + let said = answer.to_lowercase(); + if !reads_back(step) || !(said.contains("unverified") || said.contains("tried to set")) { return None; } Some(format!( @@ -443,8 +513,130 @@ mod tests { fn the_desktop_settles_after_a_step_that_changes_the_screen_and_not_after_a_wait() { let mut desk = Script::answering(vec![]); replay(&three(), &mut desk).unwrap(); - // After the click (step 1); not after the wait (2) or the last step (3). - assert_eq!(desk.settles, 1); + // After the click (1) and the typing (3), so the picture after each shows + // the result; not after the wait (2), which looks for itself. + assert_eq!(desk.settles, 2); + } + + /// A button press is not a reason to stop: it ALWAYS says "unverified". + #[test] + fn a_recorded_button_press_does_not_stop_the_replay() { + let invoke = |action: &str| { + Step::new( + "element_action", + json!({"selector": "app=mail;window=Inbox;role=button;name=New", "action": action}), + ) + }; + let typing = Step::new( + "type_text", + json!({"text": "hello", "in_window": "window~=Compose"}), + ); + for action in ["invoke", "press", "scroll_into_view", "click"] { + let routine = routine(vec![invoke(action), typing.clone()]); + let mut desk = Script::answering(vec![Ok( + "invoked button (unverified: a press has no state to read back)".to_string(), + )]); + let report = replay(&routine, &mut desk) + .unwrap_or_else(|f| panic!("{action} stopped the replay: {f}")); + assert_eq!(report.ran.len(), 2, "{action}"); + } + } + + #[test] + fn a_value_setting_action_that_is_unverified_does_stop_the_replay() { + let spin = Step::new( + "element_action", + json!({"selector": "app=calc;window=Calc;role=edit;name=Count", "action": "increment"}), + ); + let typing = Step::new( + "type_text", + json!({"text": "1", "in_window": "app=calc;window=Calc"}), + ); + let mut desk = + Script::answering(vec![Ok("incremented (unverified: no state)".to_string())]); + let failed = replay(&routine(vec![spin, typing]), &mut desk).unwrap_err(); + assert_eq!(failed.step, 1); + assert_eq!(desk.calls, ["1:element_action"]); + // A set_value that only "tried to set" is the same. + let (routine, mut desk) = + set_then_click("tried to set e49 (Edit) to 17 chars: it reads back as nothing"); + assert!(replay(&routine, &mut desk).is_err()); + } + + /// What the scripted desktop sees, for the pictures. + struct Watching { + inner: Script, + looks: Vec, + } + + impl Runner for Watching { + fn look(&mut self, n: usize, _step: &Step, when: When) -> Option { + self.looks.push(format!("{n}:{when:?}")); + Some(PathBuf::from(format!("shots/step-{n}-{when:?}.jpg"))) + } + fn run(&mut self, n: usize, step: &Step) -> Result { + self.looks.push(format!("{n}:run")); + self.inner.run(n, step) + } + fn settle(&mut self) { + self.inner.settle(); + } + } + + #[test] + fn a_picture_is_taken_before_and_after_each_step() { + let mut desk = Watching { + inner: Script::answering(vec![]), + looks: Vec::new(), + }; + replay(&three(), &mut desk).unwrap(); + assert_eq!( + desk.looks, + [ + "1:Before", "1:run", "1:After", "2:Before", "2:run", "2:After", "3:Before", + "3:run", "3:After" + ] + ); + } + + #[test] + fn a_failure_names_the_pictures_of_the_failed_step() { + let mut desk = Watching { + inner: Script::answering(vec![ + Ok("clicked".to_string()), + Err("selector matches no window".to_string()), + ]), + looks: Vec::new(), + }; + let failed = replay(&three(), &mut desk).unwrap_err(); + assert!(failed + .before_picture + .as_deref() + .unwrap() + .contains("step-2-Before")); + assert!(failed + .after_picture + .as_deref() + .unwrap() + .contains("step-2-After")); + let said = failed.to_string(); + assert!(said.contains("Pictures of step 2"), "{said}"); + assert!(said.contains("step-2-Before.jpg"), "{said}"); + assert!(said.contains("step-2-After.jpg"), "{said}"); + // A wait that timed out is a failure with pictures too. + let timeout = "{\"result\": \"timeout\", \"waited_ms\": 1}"; + let mut desk = Watching { + inner: Script::answering(vec![Ok("c".to_string()), Ok(timeout.to_string())]), + looks: Vec::new(), + }; + assert!(replay(&three(), &mut desk) + .unwrap_err() + .to_string() + .contains("step-2-After")); + // And with no pictures at all, the message has no picture sentence. + let mut plain = Script::answering(vec![Err("nope".to_string())]); + let said = replay(&three(), &mut plain).unwrap_err().to_string(); + assert!(!said.contains("Pictures of step"), "{said}"); } #[test] diff --git a/crates/desktop/src/routine/source.rs b/crates/desktop/src/routine/source.rs index 5b870a8..d7e895c 100644 --- a/crates/desktop/src/routine/source.rs +++ b/crates/desktop/src/routine/source.rs @@ -67,6 +67,29 @@ pub struct Record { pub started_at_ms: i64, /// Its tool calls, in order. pub calls: Vec, + /// Set when the record's OWN outcome says the run failed (an error, a failed + /// status), whatever its calls say. See `watchman::outcome_failure`. + pub failed: Option, +} + +impl Record { + /// A record that did not fail. + #[must_use] + pub fn new(id: &str, started_at_ms: i64, calls: Vec) -> Self { + Self { + id: id.to_string(), + started_at_ms, + calls, + failed: None, + } + } + + /// The same record, its own outcome saying it failed with `why`. + #[must_use] + pub fn failing(mut self, why: &str) -> Self { + self.failed = Some(why.to_string()); + self + } } /// Where records come from. `Send` because the server that holds one is built @@ -132,11 +155,11 @@ mod tests { #[test] fn a_memory_source_answers_for_the_records_it_holds_and_only_those() { - let source = MemorySource::new().with(Record { - id: "w-1".to_string(), - started_at_ms: 5, - calls: vec![Call::ok("key", json!({"keys": ["ctrl", "s"]}))], - }); + let source = MemorySource::new().with(Record::new( + "w-1", + 5, + vec![Call::ok("key", json!({"keys": ["ctrl", "s"]}))], + )); assert_eq!(source.record("w-1").unwrap().calls.len(), 1); assert!(source.record("w-2").unwrap_err().contains("w-2")); } diff --git a/crates/desktop/src/routine/store.rs b/crates/desktop/src/routine/store.rs index c715395..166f1fa 100644 --- a/crates/desktop/src/routine/store.rs +++ b/crates/desktop/src/routine/store.rs @@ -24,23 +24,49 @@ //! routines is REFUSED ([`Store::load`]), not read as empty: reading it as empty //! and then saving would destroy every routine in it. //! -//! The file holds typed text literally (a routine that types a phrase must -//! store it), so on Unix it is created readable by its owner only. +//! The file holds the text a routine types, literally, EXCEPT what goes into a +//! password or other secure field and anything a recording kept only as a +//! placeholder: those are parameters (`{{name}}`, see [`super::params`]) that the +//! caller of `routine_run` supplies, and are never written here. So that the +//! person can still see everything the file does hold, on Unix it is created +//! readable by its owner only. +//! +//! # Never locked out by its own size +//! +//! [`Store::load`] refuses a file over [`MAX_FILE_BYTES`], so [`Store::save`] +//! refuses to write one: a file that can be written but not read again would +//! lose every routine in it. [`MAX_ROUTINES`] times the most one routine may take +//! ([`super::MAX_ROUTINE_BYTES`]) is held under that bound by a test. +//! +//! # Two servers, one file +//! +//! A change is load, change, save. [`Store::update`] holds a lock file beside the +//! routines file across all three, so two servers (or `incu routine keep`) that +//! change it at once each keep the other's change instead of the later save +//! writing over the earlier. use std::path::{Path, PathBuf}; +use std::time::{Duration, Instant, SystemTime}; use serde::{Deserialize, Serialize}; use super::words::{key, word_set}; -use super::{day_of, Routine, Step, MAX_ROUTINES}; +use super::{day_of, Routine, Step, MAX_ROUTINES, MAX_ROUTINE_BYTES}; /// The environment variable that names the routines file when the config does /// not. pub const ROUTINES_ENV: &str = "INCU_ROUTINES"; -/// The most of a routines file that is read, in bytes. A file over this was not -/// written by this crate. -const MAX_FILE_BYTES: u64 = 4 * 1024 * 1024; +/// The most of a routines file that is read or written, in bytes. A file over +/// this was not written by this crate. +pub const MAX_FILE_BYTES: u64 = 4 * 1024 * 1024; + +/// How long a lock file may stand before it is taken to be left by a process +/// that died. A change holds it for milliseconds. +const STALE_LOCK: Duration = Duration::from_secs(30); + +/// How long a change waits for another's lock before giving up. +const LOCK_WAIT: Duration = Duration::from_secs(10); /// The document on disk. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] @@ -85,11 +111,35 @@ impl Routines { /// The live routine `phrase` asks for. #[must_use] pub fn matching(&self, phrase: &str, now_ms: i64) -> Option<&Routine> { - let want = word_set(phrase); - if want.is_empty() { + if word_set(phrase).is_empty() { return None; } - self.live(now_ms).find(|r| word_set(&r.phrase) == want) + let want = key(phrase); + self.live(now_ms).find(|r| key(&r.phrase) == want) + } + + /// Mark the routine `phrase` asks for as taught by the person (it never + /// expires) or not; whether there was one. + /// + /// Only the person does this (`incu routine keep`); no tool takes it as an + /// argument, so an agent cannot exempt what it saved from expiry. + pub fn set_taught(&mut self, phrase: &str, taught: bool, now_ms: i64) -> bool { + self.prune(now_ms); + let wanted = key(phrase); + if word_set(phrase).is_empty() { + return false; + } + match self.routines.iter_mut().find(|r| key(&r.phrase) == wanted) { + Some(routine) => { + routine.taught = taught; + // Un-keeping starts the thirty days from now, not from long ago. + if !taught { + routine.last_used_ms = now_ms; + } + true + } + None => false, + } } /// Learn `steps` under `phrase`, replacing the routine that asks the same. @@ -124,6 +174,13 @@ impl Routines { last_used_ms: now_ms, taught: taught || was_taught, }; + let size = routine.size_bytes(); + if size > MAX_ROUTINE_BYTES { + return Err(format!( + "this routine is {size} bytes, over the {MAX_ROUTINE_BYTES} one routine may \ + take; save a shorter task (fewer or shorter typed texts)" + )); + } let taught = routine.taught; match at { Some(i) => self.routines[i] = routine, @@ -175,6 +232,103 @@ impl Store { &self.path } + /// Where the pictures of the last replay are kept: a folder beside the + /// routines file. + #[must_use] + pub fn pictures_dir(&self) -> PathBuf { + self.path + .parent() + .filter(|p| !p.as_os_str().is_empty()) + .unwrap_or_else(|| Path::new(".")) + .join("last-replay") + } + + /// Change the routines: load, `change`, save, with the lock held across all + /// three so that no other change lands in between and is lost. + /// + /// `change` answering `Err` saves nothing and is the error of the call. + /// + /// # Errors + /// + /// The lock could not be had within ten seconds, the file could not be loaded + /// or saved, or `change` refused. + pub fn update( + &self, + change: impl FnOnce(&mut Routines) -> Result, + ) -> Result { + let _held = self.lock()?; + let mut file = self.load()?; + let out = change(&mut file)?; + self.save(&file)?; + Ok(out) + } + + /// The lock file's path: the routines file's name and `.lock`. + fn lock_path(&self) -> PathBuf { + let mut name = self + .path + .file_name() + .map_or_else(|| "routines".into(), std::ffi::OsString::from); + name.push(".lock"); + self.path.with_file_name(name) + } + + /// Take the lock: an exclusive create of the lock file, retried until + /// [`LOCK_WAIT`], a file older than [`STALE_LOCK`] taken to be left by a + /// process that died and removed. + fn lock(&self) -> Result { + let path = self.lock_path(); + if let Some(dir) = path.parent().filter(|p| !p.as_os_str().is_empty()) { + std::fs::create_dir_all(dir) + .map_err(|e| format!("cannot make the folder for {}: {e}", path.display()))?; + } + let started = Instant::now(); + loop { + match std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(&path) + { + Ok(mut file) => { + use std::io::Write; + let _ = write!(file, "{}", std::process::id()); + return Ok(Lock { path }); + } + Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => { + if lock_is_stale(&path) { + // Another waiter may remove it first; that is as good. + let _ = std::fs::remove_file(&path); + continue; + } + if started.elapsed() >= LOCK_WAIT { + return Err(format!( + "another change to the routines is in progress ({} is held); \ + try again in a moment", + path.display() + )); + } + std::thread::sleep(Duration::from_millis(10)); + } + // Windows answers "access denied" to an exclusive create of a + // file another process is deleting at that moment (its release): + // a lock being let go of, which is waited for like a held one. + Err(e) + if cfg!(windows) + && e.kind() == std::io::ErrorKind::PermissionDenied + && started.elapsed() < LOCK_WAIT => + { + std::thread::sleep(Duration::from_millis(10)); + } + Err(e) => { + return Err(format!( + "cannot lock the routines ({}): {e}", + path.display() + )); + } + } + } + } + /// The default path: `$INCU_ROUTINES`, else the per-user data folder. /// `None` when neither can be found (no home folder). #[must_use] @@ -240,6 +394,16 @@ impl Store { std::fs::create_dir_all(dir).map_err(|e| fail("making the folder", e))?; let json = serde_json::to_string_pretty(routines) .map_err(|e| format!("cannot encode the routines: {e}"))?; + // The bound `load` holds the file to. A file written over it could not be + // read again, and every routine in it would be locked out. + if json.len() as u64 > MAX_FILE_BYTES { + return Err(format!( + "not saved: the routines file would be {} bytes, over the {MAX_FILE_BYTES} \ + it may be, and could not be read again. Nothing was written; routine_forget \ + a routine you no longer need", + json.len() + )); + } let temp = dir.join(format!( ".{}.tmp-{}", self.path @@ -259,6 +423,27 @@ impl Store { } } +/// The lock, held until it is dropped. +struct Lock { + path: PathBuf, +} + +impl Drop for Lock { + fn drop(&mut self) { + let _ = std::fs::remove_file(&self.path); + } +} + +/// Whether the lock file at `path` has stood longer than [`STALE_LOCK`]. A file +/// whose age cannot be read is not stale: waiting is the safe way to be wrong. +fn lock_is_stale(path: &Path) -> bool { + std::fs::metadata(path) + .and_then(|m| m.modified()) + .ok() + .and_then(|t| SystemTime::now().duration_since(t).ok()) + .is_some_and(|age| age > STALE_LOCK) +} + /// Write and flush a new file readable by its owner only (on Unix). fn write_private(path: &Path, bytes: &[u8]) -> std::io::Result<()> { use std::io::Write; @@ -485,6 +670,179 @@ mod tests { let _ = std::fs::remove_dir_all(&dir); } + #[test] + fn a_save_over_the_size_a_load_accepts_is_refused_before_anything_is_written() { + let dir = folder("toobig"); + let path = dir.join("routines.json"); + let store = Store::at(path.clone()); + let mut file = Routines::default(); + file.learn("open notes", steps(), false, 0).unwrap(); + store.save(&file).unwrap(); + let before = std::fs::read_to_string(&path).unwrap(); + // Stuff the document past the bound without going through `learn`. + file.routines[0].steps = vec![Step::new( + "type_text", + json!({"text": "x".repeat(MAX_FILE_BYTES as usize)}), + )]; + let why = store.save(&file).unwrap_err(); + assert!(why.contains("could not be read again"), "{why}"); + assert_eq!(std::fs::read_to_string(&path).unwrap(), before); + assert!(store.load().is_ok(), "the file is still readable"); + let _ = std::fs::remove_dir_all(&dir); + } + + /// The arithmetic behind "never locked out": the most the cap on routines + /// and the cap on one routine allow, written out, fits what `load` reads. + #[test] + fn the_most_the_caps_allow_is_a_file_that_can_be_read_again() { + let dir = folder("arithmetic"); + let store = Store::at(dir.join("routines.json")); + // Two shapes of a full routine: a few steps with the most text each can + // hold, and the most steps with the least in each (the most structure). + for shape in 0..2 { + let mut file = Routines::default(); + for i in 0..MAX_ROUTINES { + let steps = if shape == 0 { + // Typed text up to the routine's whole allowance. + let each = MAX_ROUTINE_BYTES / 4; + (0..3) + .map(|_| Step::new("type_text", json!({"text": "t".repeat(each - 100)}))) + .collect() + } else { + (0..crate::routine::MAX_STEPS) + .map(|_| Step::new("key", json!({"keys": ["a"]}))) + .collect::>() + }; + let mut routine = Routine { + phrase: format!("{}{i}", "p".repeat(190)), + steps, + date: "2026-01-01".to_string(), + last_used_ms: 0, + taught: true, + }; + // Grow it to the cap exactly, with filler in its last step. + let room = MAX_ROUTINE_BYTES.saturating_sub(routine.size_bytes()); + if let Some(last) = routine.steps.last_mut() { + last.args = json!({"text": "f".repeat(room.saturating_sub(10))}); + last.tool = "type_text".to_string(); + } + assert!(routine.size_bytes() <= MAX_ROUTINE_BYTES); + file.routines.push(routine); + } + store + .save(&file) + .unwrap_or_else(|why| panic!("shape {shape}: {why}")); + assert_eq!(store.load().unwrap().routines.len(), MAX_ROUTINES); + } + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn two_changes_at_once_each_keep_the_others_routine() { + let dir = folder("race"); + let path = dir.join("routines.json"); + let workers: Vec<_> = (0..8) + .map(|w| { + let path = path.clone(); + std::thread::spawn(move || { + // A store of its own, as a second server has. + let store = Store::at(path); + for i in 0..5 { + store + .update(|file| { + file.learn(&format!("task w{w} n{i}"), steps(), false, 0) + .map(|_| ()) + }) + .unwrap(); + } + }) + }) + .collect(); + for worker in workers { + worker.join().unwrap(); + } + let kept = Store::at(path).load().unwrap().routines.len(); + assert_eq!(kept, 40, "a save overwrote another's routine"); + assert!( + !dir.join("routines.json.lock").exists(), + "the lock was left behind" + ); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_change_that_refuses_saves_nothing_and_frees_the_lock() { + let dir = folder("refuse"); + let store = Store::at(dir.join("routines.json")); + let why = store + .update(|_| Err::<(), _>("no such routine".to_string())) + .unwrap_err(); + assert_eq!(why, "no such routine"); + assert!(!dir.join("routines.json").exists()); + assert!(!dir.join("routines.json.lock").exists()); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_lock_left_by_a_process_that_died_is_taken_over_and_a_live_one_is_waited_for() { + let dir = folder("stale-lock"); + let store = Store::at(dir.join("routines.json")); + let lock = dir.join("routines.json.lock"); + // Left long ago: taken over. + std::fs::write(&lock, "4242").unwrap(); + let old = SystemTime::now() - Duration::from_secs(120); + std::fs::File::options() + .write(true) + .open(&lock) + .unwrap() + .set_modified(old) + .unwrap(); + store + .update(|file| file.learn("open notes", steps(), false, 0).map(|_| ())) + .unwrap(); + assert_eq!(store.load().unwrap().routines.len(), 1); + // Held just now: not taken over (it is released here after a moment). + std::fs::write(&lock, "4243").unwrap(); + let release = lock.clone(); + let releaser = std::thread::spawn(move || { + std::thread::sleep(Duration::from_millis(300)); + std::fs::remove_file(release).unwrap(); + }); + let started = Instant::now(); + store + .update(|file| file.learn("open mail", steps(), false, 0).map(|_| ())) + .unwrap(); + assert!(started.elapsed() >= Duration::from_millis(250)); + releaser.join().unwrap(); + assert_eq!(store.load().unwrap().routines.len(), 2); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn only_the_person_can_keep_a_routine_and_unkeeping_restarts_its_thirty_days() { + let mut file = Routines::default(); + file.learn("open notes", steps(), false, 0).unwrap(); + assert!(file.set_taught("notes open", true, 1_000)); + assert!(file.matching("open notes", 400 * DAY_MS).is_some()); + assert!(file.set_taught("open notes", false, 100 * DAY_MS)); + assert!(file.matching("open notes", 120 * DAY_MS).is_some()); + assert!(file.matching("open notes", 131 * DAY_MS).is_none()); + assert!(!file.set_taught("open mail", true, 0)); + assert!(!file.set_taught("please", true, 0), "no words, no routine"); + } + + #[test] + fn a_routine_over_its_own_size_is_refused_with_the_number() { + let mut file = Routines::default(); + let big = vec![Step::new( + "type_text", + json!({"text": "x".repeat(MAX_ROUTINE_BYTES)}), + )]; + let why = file.learn("type a lot", big, false, 0).unwrap_err(); + assert!(why.contains("one routine may take"), "{why}"); + assert!(file.routines.is_empty()); + } + #[test] fn the_default_path_is_the_environment_variable_first() { // Reads the process environment, so it is the one test that sets it. diff --git a/crates/desktop/src/routine/watchman.rs b/crates/desktop/src/routine/watchman.rs index 31bcfc3..29a5f9f 100644 --- a/crates/desktop/src/routine/watchman.rs +++ b/crates/desktop/src/routine/watchman.rs @@ -35,7 +35,7 @@ use std::io::{Read, Write}; use std::net::{TcpStream, ToSocketAddrs}; -use std::time::Duration; +use std::time::{Duration, Instant}; use serde_json::Value; @@ -56,6 +56,11 @@ const CONNECT_TIMEOUT: Duration = Duration::from_secs(3); /// How long to wait for each read or write. const IO_TIMEOUT: Duration = Duration::from_secs(10); +/// How long one whole request may take, from connecting to the last byte. The +/// engine answers in milliseconds; a peer that drips a byte at a time must not +/// hold the server's one thread for as long as it likes. +const TOTAL_DEADLINE: Duration = Duration::from_secs(15); + /// Records read from the local memory engine's Watchman endpoints. #[derive(Debug, Clone)] pub struct WatchmanSource { @@ -131,23 +136,74 @@ pub fn parse_record(id: &str, text: &str) -> Result { return Err(format!("asked for record {id} and was given {named}")); } } + // A span of another kind that errored fails the run as surely as a tool + // call does, even though it is not a step: what went wrong may be why the + // tool calls after it did what they did. + let failed = outcome_failure(&doc).or_else(|| { + spans.iter().find_map(|s| { + let why = s.get("error").and_then(error_text)?; + let kind = s.get("kind").and_then(Value::as_str).unwrap_or("a span"); + (kind != "ToolCall").then(|| format!("a {kind} span errored: {why}")) + }) + }); + // `kind` must BE `ToolCall`: a span that does not say what it is is not + // known to be a tool call, and a step is never guessed. let mut spans: Vec<&Value> = spans .iter() - .filter(|s| { - s.get("kind") - .and_then(Value::as_str) - .is_none_or(|k| k == "ToolCall") - }) + .filter(|s| s.get("kind").and_then(Value::as_str) == Some("ToolCall")) .collect(); spans.sort_by_key(|s| s.get("seq").and_then(Value::as_i64).unwrap_or(0)); - Ok(Record { - id: id.to_string(), - started_at_ms: doc - .get("started_at_ms") + let mut record = Record::new( + id, + doc.get("started_at_ms") .and_then(Value::as_i64) .unwrap_or(0), - calls: spans.into_iter().filter_map(call_of).collect(), - }) + spans.into_iter().filter_map(call_of).collect(), + ); + record.failed = failed; + Ok(record) +} + +/// What the record itself says of how its run went, when it says it failed. +/// +/// The records captured on 2026-11-13 carry no outcome of their own (their keys +/// are `record_id`, `started_at_ms`, `spans`, `task`, `task_input`, +/// `task_output`, `user_id`, `space_id`, `recorder`, `producer_model`, +/// `linked_memory`), so this reads the keys an outcome would be written under, +/// should the engine add one: an `error`, an `outcome` or `status` that names a +/// failure, or a `success`/`ok` of false. Absent, the run is taken to have +/// worked, which the calls' own errors then confirm or deny. +fn outcome_failure(doc: &Value) -> Option { + const FAILED: &[&str] = &[ + "failed", + "failure", + "error", + "errored", + "cancelled", + "canceled", + "aborted", + "timeout", + "timed_out", + "crashed", + ]; + if let Some(why) = doc.get("error").and_then(error_text) { + return Some(why); + } + for key in ["outcome", "status", "result"] { + if let Some(word) = doc.get(key).and_then(Value::as_str) { + if FAILED.contains(&word.trim().to_lowercase().as_str()) { + return Some(format!("{key} is \"{word}\"")); + } + } + } + for key in ["success", "ok"] { + if doc.get(key) == Some(&Value::Bool(false)) { + return Some(format!("{key} is false")); + } + } + (doc.get("is_error") == Some(&Value::Bool(true)) + || doc.get("failed") == Some(&Value::Bool(true))) + .then(|| "it is marked as an error".to_string()) } /// One span as a [`Call`]; `None` for a span with no tool name. @@ -186,6 +242,14 @@ fn output_text(value: &Value) -> Option { /// `GET path` from `base` (`http://host:port`), returning the status and body. fn get(base: &str, path: &str) -> Result<(u16, Vec), String> { + get_within(base, path, TOTAL_DEADLINE) +} + +/// [`get`], the whole request held to `total`: the connect, the write and every +/// read together, on top of the limit each read has. A per-read limit alone does +/// not bound a peer that sends a byte at a time. +fn get_within(base: &str, path: &str, total: Duration) -> Result<(u16, Vec), String> { + let deadline = Instant::now() + total; let (host, port) = split_base(base)?; let addr = (host.as_str(), port) .to_socket_addrs() @@ -197,25 +261,52 @@ fn get(base: &str, path: &str) -> Result<(u16, Vec), String> { engine on this machine" ) })?; - let mut stream = TcpStream::connect_timeout(&addr, CONNECT_TIMEOUT) + let mut stream = TcpStream::connect_timeout(&addr, CONNECT_TIMEOUT.min(total)) .map_err(|e| format!("the memory engine at {host}:{port} is not answering: {e}"))?; - let _ = stream.set_read_timeout(Some(IO_TIMEOUT)); - let _ = stream.set_write_timeout(Some(IO_TIMEOUT)); + let _ = stream.set_write_timeout(Some(IO_TIMEOUT.min(total))); + // An IPv6 address is bracketed in a Host header (RFC 9110): `::1:9180` would + // read as an address with no port. + let authority = if host.contains(':') { + format!("[{host}]:{port}") + } else { + format!("{host}:{port}") + }; let request = format!( - "GET {path} HTTP/1.1\r\nHost: {host}:{port}\r\nAccept: application/json\r\n\ + "GET {path} HTTP/1.1\r\nHost: {authority}\r\nAccept: application/json\r\n\ Connection: close\r\n\r\n" ); stream .write_all(request.as_bytes()) .map_err(|e| format!("cannot ask the memory engine: {e}"))?; let mut raw = Vec::new(); - // One byte over the cap, to tell "exactly the cap" from "more". - stream - .take(MAX_RESPONSE_BYTES as u64 + 1) - .read_to_end(&mut raw) - .map_err(|e| format!("the memory engine stopped answering: {e}"))?; - if raw.len() > MAX_RESPONSE_BYTES { - return Err("the memory engine's answer is over 4 MiB".to_string()); + let mut piece = [0u8; 8192]; + loop { + let left = deadline.saturating_duration_since(Instant::now()); + if left.is_zero() { + return Err(format!( + "the memory engine did not finish answering within {} seconds", + total.as_secs() + )); + } + let _ = stream.set_read_timeout(Some(left.min(IO_TIMEOUT))); + match stream.read(&mut piece) { + Ok(0) => break, + Ok(n) => { + raw.extend_from_slice(&piece[..n]); + if raw.len() > MAX_RESPONSE_BYTES { + return Err("the memory engine's answer is over 4 MiB".to_string()); + } + } + // A quiet moment is not the end: the loop asks the clock. + Err(e) + if matches!( + e.kind(), + std::io::ErrorKind::TimedOut + | std::io::ErrorKind::WouldBlock + | std::io::ErrorKind::Interrupted + ) => {} + Err(e) => return Err(format!("the memory engine stopped answering: {e}")), + } } parse_http(&raw) } @@ -295,11 +386,14 @@ fn dechunk(mut body: &[u8]) -> Result, String> { if size == 0 { return Ok(out); } - if body.len() < size + 2 { + // Checked: a size of `ffffffffffffffff` must be a malformed chunk, not an + // addition that wraps (and in a debug build panics the server). + let end = size.checked_add(2).ok_or_else(bad)?; + if body.len() < end || out.len().saturating_add(size) > MAX_RESPONSE_BYTES { return Err(bad()); } out.extend_from_slice(&body[..size]); - body = &body[size + 2..]; + body = &body[end..]; } } @@ -470,6 +564,147 @@ mod tests { assert!(request.contains("Connection: close"), "{request}"); } + #[test] + fn a_chunk_size_that_overflows_is_a_malformed_answer_not_a_panic() { + for size in ["ffffffffffffffff", "fffffffffffffffe", "7fffffffffffffff"] { + let body = format!("{size}\r\nhello\r\n0\r\n\r\n"); + let why = dechunk(body.as_bytes()).unwrap_err(); + assert!(why.contains("malformed"), "{size}: {why}"); + } + let answer = b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\nffffffffffffffff\r\nx"; + assert!(parse_http(answer).unwrap_err().contains("malformed")); + // A chunk that claims more than the answer may be is refused too. + let big = format!("{:x}\r\n", MAX_RESPONSE_BYTES + 1); + assert!(dechunk(big.as_bytes()).is_err()); + } + + #[test] + fn a_peer_that_drips_a_byte_at_a_time_is_cut_off_at_the_total_deadline() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + let drip = std::thread::spawn(move || { + let (mut socket, _) = listener.accept().unwrap(); + let mut buf = [0u8; 1024]; + let _ = socket.read(&mut buf); + let _ = socket.write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 100000\r\n\r\n"); + // Faster than any per-read timeout, and for far longer than the deadline. + for _ in 0..200 { + if socket.write_all(b"x").is_err() { + break; + } + std::thread::sleep(Duration::from_millis(50)); + } + }); + let started = Instant::now(); + let why = get_within( + &format!("http://127.0.0.1:{port}"), + "/x", + Duration::from_millis(600), + ) + .unwrap_err(); + assert!(why.contains("did not finish answering"), "{why}"); + assert!( + started.elapsed() < Duration::from_secs(5), + "took {:?}", + started.elapsed() + ); + drip.join().unwrap(); + } + + #[test] + fn a_span_that_does_not_say_it_is_a_tool_call_is_not_one() { + let doc = r#"{"spans":[ + {"seq":1,"name":"key","input":{"keys":["a"]}}, + {"seq":2,"kind":null,"name":"key","input":{"keys":["b"]}}, + {"seq":3,"kind":"ToolCall","name":"key","input":{"keys":["c"]}}]}"#; + let record = parse_record("w-1", doc).unwrap(); + assert_eq!(record.calls.len(), 1); + assert_eq!(record.calls[0].args["keys"][0], "c"); + } + + #[test] + fn a_record_whose_own_outcome_says_it_failed_is_marked_failed() { + let with = |extra: &str| { + format!( + r#"{{"record_id":"w-1",{extra}"spans":[{{"seq":1,"kind":"ToolCall","name":"key","input":{{}}}}]}}"# + ) + }; + for (extra, says) in [ + (r#""error":"the turn crashed","#, "crashed"), + (r#""outcome":"Failed","#, "outcome"), + (r#""status":"cancelled","#, "status"), + (r#""success":false,"#, "success"), + (r#""is_error":true,"#, "error"), + ] { + let record = parse_record("w-1", &with(extra)).unwrap(); + let why = record + .failed + .unwrap_or_else(|| panic!("{extra} was not read")); + assert!(why.contains(says), "{extra}: {why}"); + } + for fine in [ + r#""outcome":"done","#, + r#""success":true,"#, + r#""error":null,"#, + "", + ] { + assert_eq!( + parse_record("w-1", &with(fine)).unwrap().failed, + None, + "{fine}" + ); + } + // The real, captured records say nothing of an outcome: not failed. + for real in [SHELL, FAILED, STRUCTURED] { + let id = serde_json::from_str::(real).unwrap()["record_id"] + .as_str() + .unwrap() + .to_string(); + assert_eq!(parse_record(&id, real).unwrap().failed, None, "{id}"); + } + } + + #[test] + fn an_errored_span_of_any_kind_fails_the_record() { + let doc = r#"{"spans":[ + {"seq":1,"kind":"ModelCall","name":"claude","input":{},"error":"overloaded"}, + {"seq":2,"kind":"ToolCall","name":"key","input":{"keys":["a"]},"error":null}]}"#; + let record = parse_record("w-1", doc).unwrap(); + let why = record.failed.expect("the model span errored"); + assert!( + why.contains("ModelCall") && why.contains("overloaded"), + "{why}" + ); + // A tool call's own error stays on the call, where the learner reads it. + let tool = + r#"{"spans":[{"seq":1,"kind":"ToolCall","name":"Bash","input":{},"error":"exit 1"}]}"#; + let record = parse_record("w-1", tool).unwrap(); + assert_eq!(record.calls[0].error.as_deref(), Some("exit 1")); + } + + /// The request names an IPv6 host in brackets, as HTTP requires. + #[test] + fn an_ipv6_host_header_is_bracketed() { + let Ok(listener) = TcpListener::bind("[::1]:0") else { + return; // this machine has no IPv6 loopback + }; + let port = listener.local_addr().unwrap().port(); + let served = std::thread::spawn(move || { + let (mut socket, _) = listener.accept().unwrap(); + let mut buf = [0u8; 1024]; + let n = socket.read(&mut buf).unwrap(); + let _ = socket.write_all(b"HTTP/1.1 200 OK\r\ncontent-length: 2\r\n\r\n{}"); + String::from_utf8_lossy(&buf[..n]).to_string() + }); + let (status, _) = get(&format!("http://[::1]:{port}"), "/x").unwrap(); + assert_eq!(status, 200); + let request = served.join().unwrap(); + assert!( + request.contains(&format!("Host: [::1]:{port}\r\n")), + "{request}" + ); + } + #[test] fn an_engine_that_is_not_there_is_an_error_naming_it_not_a_hang() { // Port 9 (discard) is closed on a test machine; the refusal is immediate. diff --git a/crates/desktop/src/routine/words.rs b/crates/desktop/src/routine/words.rs index 4010a47..b5c9948 100644 --- a/crates/desktop/src/routine/words.rs +++ b/crates/desktop/src/routine/words.rs @@ -11,6 +11,11 @@ //! //! Direction words are NOT filler: "turn the volume up" and "turn the volume //! down" must stay different requests, and so must "on" and "off". +//! +//! "Any order" stops at a word that says which way something goes ([`DIRECTION`]: +//! to, from, into, ...). The phrase is cut there and each side matches as its own +//! set, so "copy backup files to desktop" never runs for "copy desktop files to +//! backup": the same words, the opposite task. use std::collections::BTreeSet; @@ -36,11 +41,44 @@ pub fn word_set(phrase: &str) -> BTreeSet { .collect() } -/// The phrase's identity: its word set, sorted and joined. Two phrases with the -/// same key are the same request. +/// Words that say which way a thing goes. They are filler or ordinary words +/// elsewhere, but here the words on each side of one must stay on that side: +/// "copy backup files to desktop" is not "copy desktop files to backup", and +/// a routine that ran on the second would copy the wrong way round. +pub const DIRECTION: &[&str] = &[ + "to", "from", "into", "onto", "over", "under", "before", "after", +]; + +/// The phrase's identity. Two phrases with the same key are the same request. +/// +/// With no direction word ([`DIRECTION`]) it is the word set, sorted and +/// joined: any order. With one, the phrase is cut at each direction word and +/// each side is its own sorted word set, so the words may come in any order +/// WITHIN a side and never cross a direction word. #[must_use] pub fn key(phrase: &str) -> String { - word_set(phrase).into_iter().collect::>().join(" ") + let lowered = phrase.to_lowercase(); + let mut sides: Vec> = vec![BTreeSet::new()]; + let mut directions: Vec<&str> = Vec::new(); + for word in lowered.split(|c: char| !c.is_alphanumeric()) { + if word.is_empty() { + continue; + } + if let Some(direction) = DIRECTION.iter().find(|d| **d == word) { + directions.push(direction); + sides.push(BTreeSet::new()); + } else if !STOP.contains(&word) { + if let Some(side) = sides.last_mut() { + side.insert(word); + } + } + } + let side = |words: &BTreeSet<&str>| words.iter().copied().collect::>().join(" "); + let mut out = side(&sides[0]); + for (direction, words) in directions.iter().zip(&sides[1..]) { + out.push_str(&format!(" | {direction} | {}", side(words))); + } + out } /// Whether the phrase points at what is on screen right now ("close this", @@ -59,11 +97,11 @@ pub fn is_deictic(phrase: &str) -> bool { /// words of its own (all filler) matches nothing. #[must_use] pub fn find<'a>(routines: &'a [Routine], phrase: &str) -> Option<&'a Routine> { - let want = word_set(phrase); - if want.is_empty() { + if word_set(phrase).is_empty() { return None; } - routines.iter().find(|r| word_set(&r.phrase) == want) + let want = key(phrase); + routines.iter().find(|r| key(&r.phrase) == want) } #[cfg(test)] @@ -134,6 +172,28 @@ mod tests { assert!(find(&routines, "report weekly send").is_some()); } + #[test] + fn the_words_on_each_side_of_a_direction_word_stay_on_that_side() { + assert_ne!( + key("copy backup files to desktop"), + key("copy desktop files to backup") + ); + // Any order still holds within a side. + assert_eq!( + key("copy backup files to desktop"), + key("files backup copy, to the desktop") + ); + let routines = [routine("copy backup files to desktop")]; + assert!(find(&routines, "copy desktop files to backup").is_none()); + assert!(find(&routines, "desktop to copy files backup").is_none()); + assert!(find(&routines, "copy files backup to desktop").is_some()); + // The same words under another direction word are another request. + assert_ne!(key("move notes to trash"), key("move notes from trash")); + // Two direction words keep three sides. + assert_ne!(key("copy a from b to c"), key("copy a from c to b")); + assert_eq!(key("copy a from b to c"), key("a copy from b to c")); + } + #[test] fn a_phrase_of_nothing_but_filler_matches_nothing() { let routines = [routine("")]; diff --git a/crates/desktop/src/wait.rs b/crates/desktop/src/wait.rs index 64b4534..2e21e21 100644 --- a/crates/desktop/src/wait.rs +++ b/crates/desktop/src/wait.rs @@ -49,6 +49,11 @@ pub const DEFAULT_WATCH_MS: u64 = 60 * 60 * 1000; pub const MAX_WATCH_MS: u64 = 24 * 60 * 60 * 1000; /// How often a `watch` looks when the call does not say. pub const DEFAULT_WATCH_POLL_MS: u64 = 1_000; +/// How often a `watch` of the SCREEN looks when the call does not say. A look +/// at the screen is a capture and a hash of every pixel of it, which costs far +/// more than asking whether a window exists; for hours, once in five seconds is +/// the price a background watch should pay. +pub const DEFAULT_WATCH_SCREEN_POLL_MS: u64 = 5_000; /// The fastest and slowest a `watch` may look. A watch is not waiting for a /// dialog to settle, so it is held to a slower floor than `wait_for`'s. pub const MIN_WATCH_POLL_MS: u64 = 250; @@ -97,6 +102,13 @@ const WATCH_BOUNDS: Bounds = Bounds { max_poll: MAX_WATCH_POLL_MS, }; +/// `watch` of the screen: as [`WATCH_BOUNDS`], looking every +/// [`DEFAULT_WATCH_SCREEN_POLL_MS`] unless told otherwise. +const WATCH_SCREEN_BOUNDS: Bounds = Bounds { + default_poll: DEFAULT_WATCH_SCREEN_POLL_MS, + ..WATCH_BOUNDS +}; + impl Budget { /// `timeout_ms` and `poll_ms` as the call gave them (or did not), held to /// the bounds. @@ -105,6 +117,14 @@ impl Budget { Self::within(&WAIT_BOUNDS, timeout_ms, poll_ms) } + /// A `watch` of the screen (`screen_changes`): the same bounds as + /// [`Budget::watch`], looking every [`DEFAULT_WATCH_SCREEN_POLL_MS`] unless + /// told otherwise. + #[must_use] + pub fn watch_screen(timeout_ms: Option, poll_ms: Option) -> Budget { + Self::within(&WATCH_SCREEN_BOUNDS, timeout_ms, poll_ms) + } + /// The same for a `watch`: up to [`MAX_WATCH_MS`], looking every /// [`DEFAULT_WATCH_POLL_MS`] unless told otherwise. #[must_use] diff --git a/docs/COMPUTER_USE.md b/docs/COMPUTER_USE.md index bc27b6d..258b74e 100644 --- a/docs/COMPUTER_USE.md +++ b/docs/COMPUTER_USE.md @@ -374,19 +374,36 @@ shortcuts (`atticus.py`, MIT), kept as simple as they are there. ### What a routine is Per routine: the cleaned step list (`tool` + `args`, the same names and arguments the live -tools take), the trigger phrase, the date learned, the last-used time and a user-taught flag. -All in one JSON file (below). Code: `crates/desktop/src/routine/` (`words`, `learn`, `store`, -`source`, `watchman`, `replay`) and `mcp/routine_tool.rs`. +tools take), the trigger phrase, the date learned, the last-used time and a "kept" flag that +**only the person sets** (`incu routine keep`, below). All in one JSON file (below). Code: +`crates/desktop/src/routine/` (`words`, `learn`, `params`, `store`, `source`, `watchman`, +`replay`, `pictures`) and `mcp/routine_tool.rs`. | Tool | Tier | What it does | |---|---|---| -| `routine_save` | acting | Learn a routine under `phrase` from `records` (Watchman record ids) or from `steps`. Saving under a phrase with the same words replaces the routine: that is how a failed one is redone. | -| `routine_run` | acting | Replay the routine a phrase matches, on the real desktop, attended. | +| `routine_save` | acting | Learn a routine under `phrase` from `records` (Watchman record ids) or from `steps`. Saving under a phrase with the same words replaces the routine: that is how a failed one is redone. There is no `taught` argument (see "Keeping"). | +| `routine_run` | acting | Replay the routine a phrase matches, on the real desktop, attended. Takes `params` for the secrets a routine does not keep. | | `routine_forget` | acting | Delete one. | -| `routine_list` | observing | Every live routine: phrase, step count, dates, taught, days left. | -| `routine_match` | observing | The routine a phrase would run, with its steps (typed text by length only), or that none matches. Runs nothing. | +| `routine_list` | observing | Every live routine: phrase, **every step with its full typed text and selectors**, the parameters it needs, dates, kept, days left. | +| `routine_match` | observing | The routine a phrase would run, with every step in full, or that none matches. Runs nothing. | | `watch` | observing | Below. | +### Keeping a routine (only the person) + +A routine unused for 30 days expires, so that nothing an agent learned lives on unreviewed. A +routine the person **keeps** never expires. That is set at a terminal, never by a tool, so an +agent cannot exempt its own saves from the thirty days: + +``` +incu routine list every routine, each step in full +incu routine keep keep one (matched as a request is: by its words) +incu routine unkeep let it expire again, thirty days from now +``` + +(`--file ` or `--config ` to name the routines file; else the one `incu serve` +uses.) A `routine_save` that replaces a kept routine leaves it kept. Passing `taught` to +`routine_save` is refused with this explanation. + `routine_save` and `routine_forget` are acting although they touch a file and not the screen, because the file is what a later `routine_run` does to the machine. `routine_list` and `routine_match` only read it. `permission::{OBSERVING, ACTING}`, incode's `for_desktop` and @@ -401,10 +418,19 @@ two phrases are the same request when the word SETS are equal, in any order. No partial match: a routine that runs on a near miss runs the wrong thing on a machine the operator is signed into. Direction words (`up`, `down`, `on`, `off`) are not filler. +"Any order" stops at a word that says which way something goes: `to from into onto over under +before after`. The phrase is cut there and each side is matched as its own set, so "copy +backup files to desktop" is the same request as "files backup copy to the desktop" and never +the same as "copy desktop files to backup". + ### What is refused, and what is cleaned -Nothing is learned when: the phrase says "this" or "that"; any call failed (a tool error, or a -`wait_for` that timed out or was stopped); or any kept step cannot be replayed on another day: +Nothing is learned when: the phrase says "this" or "that"; **any call or span of the source +records failed, desktop or not** (a shell command that errored, a refused `ui_tree`, an errored +model span: `[Bash fails, incu key ok]` is refused), the record's own outcome says it failed +(an `error`, or a failed `outcome`/`status`, `success: false`; the engine's records of +2026-11-13 carry none, so today this reads only the spans), or a `wait_for` timed out or was +stopped; or any kept step cannot be replayed on another day: an `eN` id (valid for one `ui_tree` only), a raw coordinate (`click`, `move_mouse`, `drag`), a window handle (`window`, `focus_window`), a `wait_for` with a `baseline`, a missing or unparseable `selector`, or another routine tool. The whole routine is refused, because dropping @@ -421,7 +447,7 @@ refused. `scroll` is never replayable: it takes no selector and scrolls under th (use `element_action scroll_into_view`, or Page Down then `wait_for screen_changes`). `ui_tree`, `screenshot`, `list_windows`, -`screen_info` and `cursor_position` are dropped (a failed one does not fail the run); +`screen_info` and `cursor_position` are dropped (a failed one DOES fail the run: see above); `wait_for` is kept. `screenshot_after` is removed from kept steps. Calls that are not desktop tools (a shell command) are left out and **reported** in the reply. The same per-step check runs again before a replay, so a hand-edited file cannot get past it. @@ -436,10 +462,14 @@ tool, never ask for it. `WatchmanSource` reads one endpoint of the local memory record becomes steps replayed on this desktop): `GET /v1/memory/watchman/record/{id}` -> `{record_id, started_at_ms, spans:[{seq, kind, name, -input, output, error}]}`. Only `ToolCall` spans are read, in `seq` order, and several records -are joined by `started_at_ms`. `error` set means the call failed; `output` may be text, null -or JSON. A desktop tool is recognised by its plain name or as `mcp____` with a -server name containing `desktop` or `incu`. Checked against a running engine on 2026-11-13; +input, output, error}]}`. Only spans whose `kind` is exactly `ToolCall` are steps (a span +that does not say is not guessed), in `seq` order, and several records are joined by +`started_at_ms`. `error` set means the call failed; `output` may be text, null or JSON. A +desktop tool is recognised by its plain name or as `mcp____` with the server +named exactly `desktop`, `incu`, `incode-desktop` or `incode_desktop`: another server that +merely has "desktop" in its name (a remote-desktop tool) is not incu and is left out. A +request is held to 15 seconds in all (on top of 10 s per read), a malformed or overflowing +chunk is refused, and the `Host` header brackets an IPv6 address. Checked against a running engine on 2026-11-13; the test fixtures are real answers with the project, user and text replaced. The engine had no record of a desktop call at that date, so the desktop-call tests build their records from that shape and the span fields above. @@ -456,19 +486,65 @@ step that changed the screen, and **stops at the first failure**, saying which s Three rules from the live test: **links and buttons are invoked**: a recorded left click on a control whose role is `link` or `button` runs first as `element_action invoke` (works for a -control scrolled out of view, moves no mouse, takes no focus) and only if that is refused as -the recorded `click_element`; both go through the gates and both are journalled. A keyboard +control scrolled out of view, moves no mouse, takes no focus) and the recorded +`click_element` runs instead ONLY when the invoke was refused before it did anything: the +control "does not support/offer invoke" (each platform's whole message is matched, not a +word of it). Any other error from the invoke (the pattern call failed, a timeout, the +operator's hook, the stop) may have come after the control acted, and a second action would +press a button twice, so the step fails and the replay stops. A keyboard step's `in_window` is looked up **now** and passed as the tool's `window`, so the tool refuses unless that window really has the focus (a window that is not there fails the step -before any key is sent). And the replay **stops when a `set_value` or `element_action` -answers "unverified" and the next step acts on the screen** (a click or typing; a `wait_for` -after it is allowed, because it checks the screen itself). A click always says "unverified" -and does not stop the replay: the next step's selector, or a `wait_for`, is its check. A +before any key is sent). And the replay **stops when a step that SETS A VALUE answers +"unverified" or "tried to set" and the next step acts on the screen** (a click or typing; a +`wait_for` after it is allowed, because it checks the screen itself): `set_value`, and the +`element_action`s `increment` and `decrement`. A click, `invoke`, `press` or +`scroll_into_view` always says "unverified" (a press has no state to read back; live, on +Windows, every one does) and **never** stops the replay: the next step's selector, or a +`wait_for`, is its check. A consequence: filling a web field with `set_value` in a browser (whose read-back is unreliable) stops a replay at that step. **No repair**: the reply tells the caller to do the whole task again and `routine_save` it under the same phrase, which replaces the routine. A routine that ran to the end has its last-used time reset. +### Pictures of a replay + +Each step of `routine_run` gets one picture before it and one after it (after the screen has +had its settle time), taken with incu's own capture, under the same guard as `screenshot` (a +blocked application's window is never captured): of the window the step is for (its selector's +window part, or its `in_window`), else of the screen, which is also what a window that cannot +be found gets. JPEG, kept in `last-replay/` beside `routines.json` as +`step-003-before.jpg` / `step-003-after.jpg`. **Only the last run is kept** (the next run +clears the folder, touching only files of that name), the folder is held to 48 MiB by deleting +the oldest first, so the failed step's pictures, the newest, always survive, and one picture +over 4 MiB is not kept. A failure names the failed step's two pictures by path; the success +reply gives the folder and how many were kept. A screen that cannot be captured (headless, a +blocked window on screen) is said in `pictures.trouble` and does not fail the replay. There is +no video and no ffmpeg, and the pictures are **never sent to the engine's OCR** (or anywhere): +reading them is for whoever is given the path. They are written by the replay itself, not +through `screenshot`, so each is not a journal line of its own. + +### Parameters: secrets a routine does not keep + +The file holds what a routine types so that the person can read it, EXCEPT text that must not +be on disk. When a routine is learned, a step becomes a **parameter**, stored as `{{name}}`, +when: it types into a password or other secure field (the field's name or automation id says +so: `password`, `passcode`, `PIN`, `OTP`, `token`, `api key`, `security code`, ...; or, when +the control is still on screen as the routine is saved, the desktop reports it as a password +field; a typing step goes into the control the step before it named); or the recording kept +only a **placeholder or a digest** in place of the text (`[REDACTED]`, ``, `****`, +`[12 chars]`, `sha256:...`, a bare 64-digit hex, or a value that is not text): replaying a +placeholder into a login box is a wrong password, not a shortcut. An agent can also write +`{{name}}` in a text itself. `routine_save` lists the parameters it made, `routine_list` shows +`params_needed`, and `routine_run` **refuses before step 1** unless `params: {"name": "..."}` +gives every one (and no name the routine does not take). A value is substituted in memory +into the one call that types it and is taken back out of any message the replay returns; it is +not written to the file, the journal (which describes typing by length) or the reply. The +operator's own pre/post-tool hooks, as for any typing, still see the real arguments. This is +a screen on what the learner can see, not a promise: a secure field whose label does not say +so and that is gone when the routine is saved is not found, which is why `routine_save`, +`routine_list` and `routine_match` show **the full typed text of every step** (and every +selector in full): the person reads exactly what a routine will type before it runs. + ### Storage One file, `routines.json`, written atomically (a temporary file beside it, flushed, renamed @@ -477,24 +553,36 @@ never read as empty. Path: `routines` in the config (`--config`), else `$INCU_RO `%APPDATA%\incord-desktop\routines.json` (Windows), `~/Library/Application Support/ incord-desktop/routines.json` (macOS), `$XDG_DATA_HOME` or `~/.local/share` then `incord-desktop/routines.json` (Linux). A routine unused for 30 days expires (it is hidden at -once and removed on the next write) unless it was taught. Up to 200 routines of up to 100 -steps. **Typed text is stored literally**; do not save a routine that types a secret. +once and removed on the next write) unless the person kept it. Up to 100 routines of up to 100 +steps, one step's arguments at most 4 KiB and one routine at most 24 KiB (counting each +step's structure, so many tiny steps are not mistaken for a small routine). **The file can never +lock itself out:** a load refuses a file over 4 MiB, so a save refuses to write one (before +writing anything, with the old file untouched), and a test writes the largest file the caps +allow and reads it back. **Two servers do not lose each other's changes:** every change is +load, change, save under a lock file (`routines.json.lock`, an exclusive create; a lock older +than 30 s is taken to be a dead process's and removed; a change waits up to 10 s for it). ### `watch` "Watch for me" is one `wait_for` with a long deadline, as a tool of its own. A separate tool rather than an option because `wait_for`'s 60 s cap and 50 ms to 2 s polling are properties its tests pin for a call that returns in moments, while a watch is the opposite promise: it -**blocks** the call for up to 24 hours (default 1 hour), polled every 1 s (250 ms to 10 s). +**blocks** the call for up to 24 hours (default 1 hour), polled every 1 s (250 ms to 10 s), +**every 5 s for `screen_changes`**, whose look is a capture and a hash of the whole screen. The look is the same code (`wait::run`, new `Budget::watch`). The stop file ends it within -a second. When it ends, the operator's **post-tool hook for `watch`** is run with the reply +a second. **A watch holds the server for its whole duration**: the server answers one call at +a time, so while it runs no other desktop call is served, the caller's included (the tool's +description says so). When it ends, the operator's **post-tool hook for `watch`** is run with the reply (`result`, what it found, `message`) as `tool_response`: that hook is the notification or the follow-up (for example `osascript -e 'display notification ...'`, `notify-send`, or a command that starts a session). Only for `watch` is the answer passed to the hook; every other tool's hook still gets none. No model runs and nothing polls it. A line `started: ...` is journalled when a watch begins. Through incode's toolbox the call bound for `watch` is -24 h plus a minute and for `routine_run` ten minutes (otherwise 120 s). If the client goes -away during a watch, the server notices when the watch ends or the stop file is written. +24 h plus a minute and for `routine_run` ten minutes (otherwise 120 s). **If the client goes +away during a watch, the watch ends**: stdin is read on a thread of its own, which raises a +flag at end of input that waiting calls look at as often as the stop file, so the pipe's EOF +is no longer unread behind a call that runs for a day. (Every line sent before the end is +still answered, and the desktop is stopped after them, as before.) ### Not built From 561ec39d19194e83004558179329c8dd72a86113 Mon Sep 17 00:00:00 2001 From: blockmandev <194144408+blockmandev@users.noreply.github.com> Date: Fri, 13 Nov 2026 22:36:49 -0600 Subject: [PATCH 3/6] desktop: routines re-review fixes: focus-moving keys, params as references, pictures through the gate, OS lock - A typing step after a key that can move focus (Tab, Shift+Tab, Enter, the arrows, Page/Home/End) has an unknown target and becomes a {{param}} unless it names a non-secret control of its own: "click user, type alice, Tab, type hunter2, Enter" keeps only alice. - routine_run params take only references, {"password": {"env": "INCU_PARAM_PASSWORD"}}, read from incu's environment; a literal value is refused with an error that names the form. No keyring crate is in the build, so a keyring reference says so. Hooks are shown routine_run's params as "" and a parameter step's {{name}} in place of the typed text. - Replay pictures are taken by calling screenshot through gated_call (tier, hook, guards, stop file, journal), not past it. None after a step that typed a secret or before the next, recorded in pictures.skipped. Files 0600 and folders 0700 on Unix. - Each run has its own folder last-replay/run-