Repository navigation
desktop: routines learned from Watchman records, replayed with no model, and watch - #95
Merged
Merged
Conversation
…el, and watch
A routine is a trigger phrase and the cleaned desktop steps that carried it
out, built from Watchman's own tool-call records (read from the local memory
engine's /v1/memory/watchman/record/{id}, behind a StepSource trait with a
Watchman, an in-memory and a no-engine implementation) or from an explicit
step list. Six new tools, 23 in all: routine_save, routine_run,
routine_forget (acting) and routine_list, routine_match, watch (observing).
Learning follows Atticus's learned shortcuts: nothing is kept when the
phrase says this/that, when a call failed or a wait_for timed out, or when a
step uses an eN id, a coordinate or a window handle; looking steps are
dropped, wait_for is kept. From the live test: every acting step must name
its window (selector window part, or in_window for typing and keys, taken
from the step before when a recording has none), scroll is never replayable.
Replay runs each step through Server::gated_call, the path tools/call takes
(stop file, tier, pre/post hooks, blocked apps, rate cap, journal), stops at
the first failure and says which step and why, with no repair. Links and
buttons are invoked through element_action invoke first; a set_value or
element_action that answers unverified stops the replay when the next step
acts on it.
Matching is by word set, any order, filler ignored. One JSON file, written
atomically, expiring after 30 unused days unless taught; path overridable in
Config / INCU_ROUTINES. watch is wait_for for up to 24 hours, polled gently,
that tells the operator's post-tool hook what it saw.
…ure rules, locking
Required behaviour that was missing
- A picture before and after every replayed step, with incu's own capture under
screenshot's guard (the step's window, else the screen), JPEG, in last-replay/
beside routines.json. Last run only, held to 48 MiB by dropping the oldest, so
the failed step's pictures survive; the failure names them by path. Never sent
to OCR.
- Nothing is learned when ANY span of the records errored (a shell call too, not
only a desktop call) or the record's own outcome says it failed.
- "Kept" (never expires) is no longer a tool argument; passing `taught` is
refused. The person sets it with `incu routine keep|unkeep|list`.
Bugs
- A recorded button press no longer stops a replay: only a value-setting step
(set_value, element_action increment/decrement) answering "unverified" or
"tried to set" does.
- The recorded click runs after element_action invoke only when the invoke was
refused before acting (the control does not offer it), never after an error
that may have come after the control acted.
- Text typed into a password or other secure field (by name, by automation id,
or by the desktop's own report when saving), and text a recording kept as a
placeholder or digest, is stored as a parameter {{name}}; routine_run needs
params for each and refuses before step 1. Values are never written to the
file or journal and are scrubbed from replay messages. Save, list and match
show every step's full typed text and selector.
- save refuses a file over the size load accepts, before writing; caps lowered
(100 routines, 4 KiB a step, 24 KiB a routine) and a test writes the largest
file they allow.
- Changes take a lock file (exclusive create, 30 s stale, 10 s wait): two
servers keep each other's saves.
- A chunk size of ffffffffffffffff is a malformed answer, not a panic; a 15 s
total deadline per request; a span with no kind is not a tool call; the Host
header brackets IPv6.
- Matching keeps any order but not across to/from/into/onto/over/under/before/
after: "copy backup files to desktop" is not "copy desktop files to backup".
Minor
- Only incu's own server names (desktop, incu, incode-desktop, incode_desktop)
are taken for its tools.
- A watch ends when the client disconnects (stdin is read on a thread that
raises a flag at EOF), looks at the screen every 5 s, and is documented to
hold the server while it runs.
…ences, pictures through the gate, OS lock
- A typing step after a key that can move focus (Tab, Shift+Tab, Enter, the
arrows, Page/Home/End) has an unknown target and becomes a {{param}} unless it
names a non-secret control of its own: "click user, type alice, Tab, type
hunter2, Enter" keeps only alice.
- routine_run params take only references, {"password": {"env":
"INCU_PARAM_PASSWORD"}}, read from incu's environment; a literal value is
refused with an error that names the form. No keyring crate is in the build,
so a keyring reference says so. Hooks are shown routine_run's params as
"<param>" and a parameter step's {{name}} in place of the typed text.
- Replay pictures are taken by calling screenshot through gated_call (tier,
hook, guards, stop file, journal), not past it. None after a step that typed a
secret or before the next, recorded in pictures.skipped. Files 0600 and
folders 0700 on Unix.
- Each run has its own folder last-replay/run-<time>-<pid>-<n>/; a run still
going (an OS lock on .active) is never deleted; only the newest finished
run is kept.
- The routines lock is an exclusive OS file lock (fd-lock, already in the
lock file; std's File::try_lock needs 1.89 and the floor is 1.88), so a
slow holder is never robbed and a crashed one frees it at once.
- Every element_action that changes a state (toggle, select, expand, collapse,
increment, decrement, focus) answering "unverified" stops the replay;
only invoke, press and scroll_into_view are exempt.
- The client counts as gone only at end of input with every sent line taken up;
a replay checks between steps.
- Saving other steps over a kept routine un-keeps it; MAX_PARAMS enforced;
scrub covers values of any length (short ones only where whole); docs say
`incu routine keep` is a convenience, not a security boundary.
…d plain field, the person binds secrets, marks before folders, EOF reverted
- Typed text is stored literally ONLY when the typing step itself carries a
selector with a window part naming an edit or combobox control, with no
secret-looking name or automation id, that the desktop confirms is not a
password field when saved (a control that could not be checked does not
pass). Every other typed text, set_value included, becomes a {{param}}: no
selector, an inherited or unknown target, a button, a secure control. The
focus-key tracking is deleted. The replay focuses a named field before it
types.
- routine_run takes no params. The person binds each parameter with
`incu routine bind <phrase> <param> <ENV_VAR>` (and unbind); only the
variable's NAME is stored, with a SHA-256 digest of the steps, and the name
must begin INCU_PARAM_. A run refuses before step 1 when a parameter is
unbound, the steps changed since binding, or the variable is unset. Saving
other steps drops every binding.
- A replay's picture folder is marked before it is made: the mark is an OS-locked
file beside the folder, taken first; cleanup deletes only a run-* folder whose
mark it can take, so a new folder is never found unmarked.
- End of input on stdin no longer ends a watch or a replay: the server reads one
line, answers it, then reads the next, so a pipelined last watch or routine_run
completes. An orphaned watch lasts until its deadline (1 hour default, 24 at
most) or `incu stop`.
…ed as no boundary, marks never deleted young
- A `key` step may hold only named keys (Enter, Tab, Escape, the arrows,
F1-F24, Home, End, Page Up/Down, Backspace, Delete, Space) and chords with
ctrl, alt, cmd, win or meta. One that presses a lone printable character,
with or without shift (a PIN as {"keys": ["4"]} presses), refuses the whole
learn, and a stored one refuses the replay, saying to write a typing step on a
named field instead.
- The bind digest is documented, in the docs, the code and `incu --help` under
`routine bind`, as no boundary against an agent that can write the app-data
folder or run incu; the boundary is running agents sandboxed (insbx). No
crypto added.
- A mark with no folder is deleted only when older than 10 minutes and free, and
a run-* folder with no mark older than a day is deleted, so a deleted mark
cannot leave its folder for ever.
They start the input thread, which on Linux needs a display, and CI's Linux test job has none: 34 failed there with 'could not open an input connection'. The five that start no server still run everywhere, and Windows and macOS run all 39 as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Computer use: routines. A task done once is learned from Watchman's tool-call records and replayed later with no model, the way Atticus's learned shortcuts work. This adds 6 tools, for 23 in all.
Tools
routine_savelearns from Watchman record ids (GET /v1/memory/watchman/record/{id}, loopback only) or from an explicit step list.routine_runreplays a routine.routine_forgetdeletes one.routine_listandroutine_match(which routine a phrase would run, without running it).watchis a long wait with no model: default 1 h, at most 24 h. Its message reaches the operator's post-tool hook.incu routine list | keep | unkeep | bind | unbind.Learning rules
eNids, coordinates, orkeysteps that press printable characters.wait_foris kept. Every acting step must carry a window.incu routine keepon it.Replay
element_action invokefirst. The recorded click is used only when the invoke was refused before acting.Secrets
{{param}}.incu routine bind <phrase> <param> INCU_PARAM_<NAME>. The file stores the variable name and a digest of the steps, never a value.Pictures
screenshotgate. None is taken around parameter steps.Store
routines.jsonin the user data dir, overridable withINCU_ROUTINESorConfig.Review
Tests: fmt and clippy
-D warningspass on the workspace and on aarch64-apple-darwin.cargo test --workspacepasses with 0 failures: 2284 in incode lib and 527 in incord-desktop.Not verified
Known trade-off: recorded
type_textcalls carry no field selector, so a routine learned from a recording makes all of its typing a parameter. A follow-up will lettype_texttake a selector, so recordings keep the field.