Skip to content

desktop: wait_for and stable element selectors (17 tools) - #94

Merged
blockmandev merged 9 commits into
mainfrom
feat/desktop-wait-selectors
Nov 14, 2026
Merged

blockmandev merged 9 commits into
mainfrom
feat/desktop-wait-selectors

Conversation

@blockmandev

Copy link
Copy Markdown
Member

Computer use, phase 2: wait_for and stable element selectors. The desktop server now offers 17 tools.

Selectors. A selector is key=value pairs joined by ;, using the keys app, window/window~, role, name/name~, aid, path and nth. Examples: role=button;name=Save and app=notepad;window~=Save As;role=button;name=Save.

  • ui_tree prints the shortest unique sel: for each control.
  • click_element, set_value, element_action and wait_for accept selector instead of id, and refuse a call that gives both.
  • A selector is resolved against a fresh walk of the window at call time. When there are no matches, too many, or nth is out of range, the error says which.
  • A selector never reaches a blocked app.

wait_for is in the observe tier. Its conditions are element_appears, element_gone, window_appears, window_gone and screen_changes. screen_changes compares a 128-bit fingerprint of the screen.

  • The timeout defaults to 5 s and is capped at 60 s.
  • The stop file is checked every 100 ms.
  • A timeout is an answer, not an error.

These selectors are what lets a recorded tool call mean the same control on the next run. That is the base for routines replayed from Watchman's records.

Tests: fmt and clippy (-D warnings) clean on the workspace and on the aarch64-apple-darwin and x86_64-unknown-linux-musl targets. cargo test --workspace passed with 0 failures, including 285 incord-desktop tests. The live Windows selector tests (crates/desktop/src/mcp/selector_tests.rs) passed on Windows.

Not verified:

  • macOS and Linux were compiled but not run live.
  • On Linux, reading AccessibleId and the cost of one D-Bus call per element.
  • About 450 ms per list_windows on Windows, which limits element waits to roughly one look per second.

wait_for (observing tier) waits, bounded, until a control matching a
selector appears or goes, a window appears or closes, or the screen
fingerprint changes. timeout_ms defaults to 5000 and is held to 60000;
poll_ms 50..2000; no poll starts after the deadline; the stop is asked
every 100 ms. A timeout is a JSON reply (result=timeout, last_seen), not
a tool error.

Selectors (key=value;... : app, window, role, name, aid, path, nth) are
printed by ui_tree beside every element and accepted as `selector` by
click_element, set_value, element_action and wait_for in place of `id`.
Resolved against the live tree at call time; none or several matches is
an error naming which, never a guess. eN ids are unchanged. Element
gains automation_id (UIA AutomationId, AX AXIdentifier, AT-SPI
AccessibleId).

Tier tables, incode's for_desktop, toolbox OBSERVING and the parity test
cover wait_for; docs updated.
1. A blocked application's window named by id with a selector was
   described in the "does not name" error (its title), and a match and a
   mismatch gave different refusals. window_for now takes the blocked
   guard and runs it before the selector's window part is compared, with
   the id path's own words. Tests on the selector path.

2. Errors are journalled word for word, so none quotes what it was
   given. Parse errors name the item by position; a selector in an
   error is shown redacted (resolve, require_*, the "is disabled"
   refusal); ambiguity examples name a text field by its place on every
   platform, not by its name. Tests.

3. nth/path were counted in the walk ui_tree was asked for but resolved
   in a deeper one, so nth=1 could be another control. Selectors gain
   `depth`: ui_tree stamps it when asked for a non-default depth and the
   resolver (and wait_for) walks as deep as the selector says, else to
   the default depth that ui_tree uses without being asked. Tests over
   several depths with a button nested below the default.

4. A derived name cut at a capital sigma lowercased to a different
   letter (final sigma) than inside the whole name and missed its own
   element. Contains now folds case one character at a time. Test.
…ames in messages)

1. An ambiguity error's example selectors carry the failing selector's
   depth, so their nth is counted in the walk the resolver will make.
2. fold() also maps a typed final sigma to sigma, so name~=λόγος matches
   ΛΌΓΟΣ as str::to_lowercase did.
3. Element::label (used in platform errors such as "has no value to
   set") and the id path's "is disabled" refusal no longer name an edit
   or text field: on macOS its name is its content and errors are
   journalled word for word.
Live test: after a click closed Notepad, wait_for element_gone on that
window with timeout_ms 4000 did not return for more than five minutes.

- A pinned window that no longer exists (IsWindow on Windows, asked of
  the window manager and nothing else) is element_gone: found at once,
  with no accessibility call. A closed window named by id is gone for
  element_gone and an error for element_appears.
- Every look (list windows, walk, capture, focused window) runs on a
  thread of its own and is waited for until the wait's deadline plus
  1 s. A look the desktop did not answer is a look that saw nothing
  (wait::UNANSWERED): the wait ends as `timeout` with last_seen saying
  so, never as `gone`, never as an error, and no look starts after the
  deadline. At most 4 abandoned looks are tolerated before none is
  started.
- Tests: a fake desktop where the window is destroyed mid-wait or never
  answers, and a test that a look sleeping 5 s is given up on within its
  limit plus the grace.
Live test: set_value on Notepad replied "set e2 (Edit) to 109 chars
(unverified: it reads back as 109 characters, not the 109 sent)" and, on a
Chrome input, "reads back as 0 characters" when the text was there.

- A read-back is judged by one pure function (uia/readback.rs) on every
  platform, with line endings normalised (CRLF, a RichEdit lone CR) so a
  multi-line value no longer "differs" by length alone.
- Same: verified. Empty after a non-empty set: a browser or custom control
  that does not report its value; unverified, not an error. A real value
  that is exactly what the field held before, or other text: an error that
  gives lengths and the first differing position, never the text.
- An unconfirmed reply says "tried to set eN (Role) to N chars", never
  "set". Windows adds when a program brought its window to the front while
  its value was set (Notepad does, with no incu involved).
- set_value's tool description says Notepad's ValuePattern change is not
  counted as an edit, so type_text or paste_text are the real edits.
…age time

Live test, Chrome on a long Wikipedia article: links below the 400th
element could not be found by selector (resolution shared ui_tree's print
cap), and the first look at a fresh page held only the browser's own
controls.

- A walk made to FIND a control (selector resolution, wait_for, acting on
  what a selector found) keeps up to tree_max_visited elements (default
  4000), not tree_max_elements. Printing needs a cap that fits a reply;
  finding reports one control. The visit bound already limited time and
  cost, and is now the only bound. Walk order is unchanged, so a control
  within the first 400 keeps its id, nth and path: a selector ui_tree
  printed still means the same control. The cut warning now names the real
  bound.
- A selector that matches nothing in a browser window (chrome, msedge,
  firefox, brave, vivaldi, opera, chromium) is walked once more after
  1.5 s: the first walk is what makes the browser build the page's
  accessibility tree. Only a miss is retried, once; a match, an ambiguity
  or another program is not, and the error then says the wait happened.
  wait_for polls and needs none of it. Smallest safe fix: no new tool, no
  platform code, and no cost on a hit.
- Tests: a control 600 elements past the print cap is found and the
  print-cap walk cannot see it; selectors printed within the cap resolve to
  the same control in the larger walk; the retry runs once on a miss, never
  on a hit, an ambiguity or a non-browser.
- cargo fmt over uia/readback.rs, which the previous commit left unformatted.
…g window does not block the others

Review findings 2 and 3 on eaf0942.

- A look is waited for in slices of 100 ms and the operator's stop is
  asked each time, so a stop during a look at a hung window returns within
  a slice, not at the deadline plus 1 s. wait_until asks the stop BEFORE the
  deadline, so the reply is Stopped and not a timeout.
- The cap on abandoned looks (4) is now counted per window (the id of the
  window looked at, or the kind of look when it is of none), so one hung
  program no longer makes every later wait on an unrelated window time out
  at once. A higher process-wide net (24) remains.
- Tests: a stop 300 ms into a 60 s look returns at about 300 ms; a stop
  during a blocked look is Stopped when the deadline is already past; a hung
  window's looks are refused while another window's run; the process-wide
  net holds and is above the per-window limit.
…ck and journal names

Review findings 1, 4 and 5 on 1530260, 074c6c8 and b56356a.

1. ui_tree printed selectors from its 400 elements while the resolver sees
   up to 4000, so "role=link;name=Privacy policy" (unique among the 400)
   matched two once a twin lay at element 1500. ui_tree now walks with the
   resolving bound, prints the first 400, and makes every printed selector
   unique, with its nth, among the whole walk (selectors_for_first). Only
   the printed elements are remembered as shown, so an id still names only
   what was printed. The comment on resolve_max_elements said otherwise and
   is corrected.
4. macOS set_value read "no value" (an empty AXValue or a failed read, which
   text_attribute cannot tell apart) as verified when the value sent was
   empty. judge_optional: nothing read is never verified.
5. A name that can be a value is never journalled: Role::name_may_be_content
   (edit, combo box, static text, slider, other) now governs the names in
   messages (Element::label), the look-alikes of an ambiguity error and
   Selector::render_redacted. What ui_tree prints on macOS also leaves a
   combo box's name out, as it does an edit field's, since it changes as it
   is used.

Tests: the Privacy-policy page (twin at 1500) prints nth=0 and resolves to
its own element, and only the printed part is remembered; a combo box and
text names do not reach the redacted selector, the examples or a label;
nothing read back is unverified for an empty and a non-empty value.
…; a stop while picking the window is Stopped

Re-review of df9c139.

- The wait added a given-up look to the per-window count AFTER settling
  with it, so the look's thread, returning in between, removed an entry
  that was not there yet (or another look's), and the later addition was
  never taken off: four such leaks on the windows-listing key made every
  element or window wait time out until restart. The wait now counts first,
  settles second, and takes the count back if the look had in fact
  returned, so a look's removal always follows the addition it undoes.
- A stop during the look that picks the watched window ended the wait as a
  plain error ("the wait was stopped"); it is now the Stopped outcome, as a
  stop during the polling is.
- Tests: a look made to return exactly after the wait settled, and exactly
  before, each leave no entry (a test hook in the give-up path); one
  window's returning look does not take off another's entry; a stop during
  the window-picking look is Stopped, an unanswered one without a stop is
  still an error.
@blockmandev
blockmandev merged commit c5d3aa0 into main Nov 14, 2026
12 checks passed
@blockmandev
blockmandev deleted the feat/desktop-wait-selectors branch November 15, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant