Skip to content

desktop: routines learned from Watchman records, replayed with no model, and watch - #95

Merged
blockmandev merged 6 commits into
mainfrom
feat/desktop-routines
Nov 15, 2026
Merged

blockmandev merged 6 commits into
mainfrom
feat/desktop-routines

Conversation

@blockmandev

Copy link
Copy Markdown
Member

Computer use: routines. A task done once is learned from Watchman's tool-call records and replayed later with no model, the way Atticus's learned shortcuts work. This adds 6 tools, for 23 in all.

Tools

  • Act tier:
    • routine_save learns from Watchman record ids (GET /v1/memory/watchman/record/{id}, loopback only) or from an explicit step list.
    • routine_run replays a routine.
    • routine_forget deletes one.
  • Observe tier:
    • routine_list and routine_match (which routine a phrase would run, without running it).
    • watch is a long wait with no model: default 1 h, at most 24 h. Its message reaches the operator's post-tool hook.
  • CLI, for the person: incu routine list | keep | unkeep | bind | unbind.

Learning rules

  • Matching uses the word set in any order, but words never cross a direction word (to, from, into…).
  • Nothing is learned from "this"/"that" phrases, failed runs (any errored span), eN ids, coordinates, or key steps that press printable characters.
  • Looking tools are stripped and wait_for is kept. Every acting step must carry a window.
  • A routine unused for 30 days expires, unless the person ran incu routine keep on it.

Replay

  • Every step goes through the same gate as a live call: stop file, tier, hooks, blocked apps, journal.
  • It stops at the first failure, with no repair; the model redoes the task and the new run replaces the routine.
  • Links and buttons are pressed with element_action invoke first. The recorded click is used only when the invoke was refused before acting.
  • A value step or state change that cannot be confirmed ("unverified") stops the replay.

Secrets

  • Typed text is stored literally only when the step's own selector names a plain, non-secret edit or combobox. Everything else becomes a {{param}}.
  • Parameters are bound only by the person: incu routine bind <phrase> <param> INCU_PARAM_<NAME>. The file stores the variable name and a digest of the steps, never a value.
  • Values never reach routines.json, the journal, hooks, pictures or replies.
  • The digest stops accidental or stale use, not an agent that can write the app-data folder. The boundary for that is running agents sandboxed (insbx); this is documented.

Pictures

  • One picture before and one after each step, taken through the screenshot gate. None is taken around parameter steps.
  • Only the last run is kept, in its own folder, owner-only on Unix and capped at 48 MiB. A failure names its two pictures.
  • No video, no ffmpeg, and no OCR during replay.

Store

  • routines.json in the user data dir, overridable with INCU_ROUTINES or Config.
  • Written atomically under an OS file lock (fd-lock, already in Cargo.lock), with a 4 MiB bound enforced on save.

Review

  • Four rounds of independent review, with fixes in c05bbb2, 561ec39, 0e83025 and 0cf65fb.
  • The final review found no remaining path from a parameter value to disk, the journal, hooks, pictures or replies.

Tests: fmt and clippy -D warnings pass on the workspace and on aarch64-apple-darwin. cargo test --workspace passes with 0 failures: 2284 in incode lib and 527 in incord-desktop.

Not verified

  • A live replay on Windows.
  • Any run on a real Mac.
  • The desktop's own password report on a real login form.
  • Lock behaviour across two separate processes (the tests use separate handles in one process).

Known trade-off: recorded type_text calls carry no field selector, so a routine learned from a recording makes all of its typing a parameter. A follow-up will let type_text take a selector, so recordings keep the field.

…el, and watch

A routine is a trigger phrase and the cleaned desktop steps that carried it
out, built from Watchman's own tool-call records (read from the local memory
engine's /v1/memory/watchman/record/{id}, behind a StepSource trait with a
Watchman, an in-memory and a no-engine implementation) or from an explicit
step list. Six new tools, 23 in all: routine_save, routine_run,
routine_forget (acting) and routine_list, routine_match, watch (observing).

Learning follows Atticus's learned shortcuts: nothing is kept when the
phrase says this/that, when a call failed or a wait_for timed out, or when a
step uses an eN id, a coordinate or a window handle; looking steps are
dropped, wait_for is kept. From the live test: every acting step must name
its window (selector window part, or in_window for typing and keys, taken
from the step before when a recording has none), scroll is never replayable.

Replay runs each step through Server::gated_call, the path tools/call takes
(stop file, tier, pre/post hooks, blocked apps, rate cap, journal), stops at
the first failure and says which step and why, with no repair. Links and
buttons are invoked through element_action invoke first; a set_value or
element_action that answers unverified stops the replay when the next step
acts on it.

Matching is by word set, any order, filler ignored. One JSON file, written
atomically, expiring after 30 unused days unless taught; path overridable in
Config / INCU_ROUTINES. watch is wait_for for up to 24 hours, polled gently,
that tells the operator's post-tool hook what it saw.
…ure rules, locking

Required behaviour that was missing
- A picture before and after every replayed step, with incu's own capture under
  screenshot's guard (the step's window, else the screen), JPEG, in last-replay/
  beside routines.json. Last run only, held to 48 MiB by dropping the oldest, so
  the failed step's pictures survive; the failure names them by path. Never sent
  to OCR.
- Nothing is learned when ANY span of the records errored (a shell call too, not
  only a desktop call) or the record's own outcome says it failed.
- "Kept" (never expires) is no longer a tool argument; passing `taught` is
  refused. The person sets it with `incu routine keep|unkeep|list`.

Bugs
- A recorded button press no longer stops a replay: only a value-setting step
  (set_value, element_action increment/decrement) answering "unverified" or
  "tried to set" does.
- The recorded click runs after element_action invoke only when the invoke was
  refused before acting (the control does not offer it), never after an error
  that may have come after the control acted.
- Text typed into a password or other secure field (by name, by automation id,
  or by the desktop's own report when saving), and text a recording kept as a
  placeholder or digest, is stored as a parameter {{name}}; routine_run needs
  params for each and refuses before step 1. Values are never written to the
  file or journal and are scrubbed from replay messages. Save, list and match
  show every step's full typed text and selector.
- save refuses a file over the size load accepts, before writing; caps lowered
  (100 routines, 4 KiB a step, 24 KiB a routine) and a test writes the largest
  file they allow.
- Changes take a lock file (exclusive create, 30 s stale, 10 s wait): two
  servers keep each other's saves.
- A chunk size of ffffffffffffffff is a malformed answer, not a panic; a 15 s
  total deadline per request; a span with no kind is not a tool call; the Host
  header brackets IPv6.
- Matching keeps any order but not across to/from/into/onto/over/under/before/
  after: "copy backup files to desktop" is not "copy desktop files to backup".

Minor
- Only incu's own server names (desktop, incu, incode-desktop, incode_desktop)
  are taken for its tools.
- A watch ends when the client disconnects (stdin is read on a thread that
  raises a flag at EOF), looks at the screen every 5 s, and is documented to
  hold the server while it runs.
…ences, pictures through the gate, OS lock

- A typing step after a key that can move focus (Tab, Shift+Tab, Enter, the
  arrows, Page/Home/End) has an unknown target and becomes a {{param}} unless it
  names a non-secret control of its own: "click user, type alice, Tab, type
  hunter2, Enter" keeps only alice.
- routine_run params take only references, {"password": {"env":
  "INCU_PARAM_PASSWORD"}}, read from incu's environment; a literal value is
  refused with an error that names the form. No keyring crate is in the build,
  so a keyring reference says so. Hooks are shown routine_run's params as
  "<param>" and a parameter step's {{name}} in place of the typed text.
- Replay pictures are taken by calling screenshot through gated_call (tier,
  hook, guards, stop file, journal), not past it. None after a step that typed a
  secret or before the next, recorded in pictures.skipped. Files 0600 and
  folders 0700 on Unix.
- Each run has its own folder last-replay/run-<time>-<pid>-<n>/; a run still
  going (an OS lock on .active) is never deleted; only the newest finished
  run is kept.
- The routines lock is an exclusive OS file lock (fd-lock, already in the
  lock file; std's File::try_lock needs 1.89 and the floor is 1.88), so a
  slow holder is never robbed and a crashed one frees it at once.
- Every element_action that changes a state (toggle, select, expand, collapse,
  increment, decrement, focus) answering "unverified" stops the replay;
  only invoke, press and scroll_into_view are exempt.
- The client counts as gone only at end of input with every sent line taken up;
  a replay checks between steps.
- Saving other steps over a kept routine un-keeps it; MAX_PARAMS enforced;
  scrub covers values of any length (short ones only where whole); docs say
  `incu routine keep` is a convenience, not a security boundary.
…d plain field, the person binds secrets, marks before folders, EOF reverted

- Typed text is stored literally ONLY when the typing step itself carries a
  selector with a window part naming an edit or combobox control, with no
  secret-looking name or automation id, that the desktop confirms is not a
  password field when saved (a control that could not be checked does not
  pass). Every other typed text, set_value included, becomes a {{param}}: no
  selector, an inherited or unknown target, a button, a secure control. The
  focus-key tracking is deleted. The replay focuses a named field before it
  types.
- routine_run takes no params. The person binds each parameter with
  `incu routine bind <phrase> <param> <ENV_VAR>` (and unbind); only the
  variable's NAME is stored, with a SHA-256 digest of the steps, and the name
  must begin INCU_PARAM_. A run refuses before step 1 when a parameter is
  unbound, the steps changed since binding, or the variable is unset. Saving
  other steps drops every binding.
- A replay's picture folder is marked before it is made: the mark is an OS-locked
  file beside the folder, taken first; cleanup deletes only a run-* folder whose
  mark it can take, so a new folder is never found unmarked.
- End of input on stdin no longer ends a watch or a replay: the server reads one
  line, answers it, then reads the next, so a pipelined last watch or routine_run
  completes. An orphaned watch lasts until its deadline (1 hour default, 24 at
  most) or `incu stop`.
…ed as no boundary, marks never deleted young

- A `key` step may hold only named keys (Enter, Tab, Escape, the arrows,
  F1-F24, Home, End, Page Up/Down, Backspace, Delete, Space) and chords with
  ctrl, alt, cmd, win or meta. One that presses a lone printable character,
  with or without shift (a PIN as {"keys": ["4"]} presses), refuses the whole
  learn, and a stored one refuses the replay, saying to write a typing step on a
  named field instead.
- The bind digest is documented, in the docs, the code and `incu --help` under
  `routine bind`, as no boundary against an agent that can write the app-data
  folder or run incu; the boundary is running agents sandboxed (insbx). No
  crypto added.
- A mark with no folder is deleted only when older than 10 minutes and free, and
  a run-* folder with no mark older than a day is deleted, so a deleted mark
  cannot leave its folder for ever.
They start the input thread, which on Linux needs a display, and CI's
Linux test job has none: 34 failed there with 'could not open an input
connection'. The five that start no server still run everywhere, and
Windows and macOS run all 39 as before.
@blockmandev
blockmandev merged commit 60a151b into main Nov 15, 2026
12 checks passed
@blockmandev
blockmandev deleted the feat/desktop-routines branch November 15, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant