Repository navigation
desktop: wait_for and stable element selectors (17 tools) - #94
Merged
Merged
Conversation
wait_for (observing tier) waits, bounded, until a control matching a selector appears or goes, a window appears or closes, or the screen fingerprint changes. timeout_ms defaults to 5000 and is held to 60000; poll_ms 50..2000; no poll starts after the deadline; the stop is asked every 100 ms. A timeout is a JSON reply (result=timeout, last_seen), not a tool error. Selectors (key=value;... : app, window, role, name, aid, path, nth) are printed by ui_tree beside every element and accepted as `selector` by click_element, set_value, element_action and wait_for in place of `id`. Resolved against the live tree at call time; none or several matches is an error naming which, never a guess. eN ids are unchanged. Element gains automation_id (UIA AutomationId, AX AXIdentifier, AT-SPI AccessibleId). Tier tables, incode's for_desktop, toolbox OBSERVING and the parity test cover wait_for; docs updated.
1. A blocked application's window named by id with a selector was described in the "does not name" error (its title), and a match and a mismatch gave different refusals. window_for now takes the blocked guard and runs it before the selector's window part is compared, with the id path's own words. Tests on the selector path. 2. Errors are journalled word for word, so none quotes what it was given. Parse errors name the item by position; a selector in an error is shown redacted (resolve, require_*, the "is disabled" refusal); ambiguity examples name a text field by its place on every platform, not by its name. Tests. 3. nth/path were counted in the walk ui_tree was asked for but resolved in a deeper one, so nth=1 could be another control. Selectors gain `depth`: ui_tree stamps it when asked for a non-default depth and the resolver (and wait_for) walks as deep as the selector says, else to the default depth that ui_tree uses without being asked. Tests over several depths with a button nested below the default. 4. A derived name cut at a capital sigma lowercased to a different letter (final sigma) than inside the whole name and missed its own element. Contains now folds case one character at a time. Test.
…ames in messages) 1. An ambiguity error's example selectors carry the failing selector's depth, so their nth is counted in the walk the resolver will make. 2. fold() also maps a typed final sigma to sigma, so name~=λόγος matches ΛΌΓΟΣ as str::to_lowercase did. 3. Element::label (used in platform errors such as "has no value to set") and the id path's "is disabled" refusal no longer name an edit or text field: on macOS its name is its content and errors are journalled word for word.
Live test: after a click closed Notepad, wait_for element_gone on that window with timeout_ms 4000 did not return for more than five minutes. - A pinned window that no longer exists (IsWindow on Windows, asked of the window manager and nothing else) is element_gone: found at once, with no accessibility call. A closed window named by id is gone for element_gone and an error for element_appears. - Every look (list windows, walk, capture, focused window) runs on a thread of its own and is waited for until the wait's deadline plus 1 s. A look the desktop did not answer is a look that saw nothing (wait::UNANSWERED): the wait ends as `timeout` with last_seen saying so, never as `gone`, never as an error, and no look starts after the deadline. At most 4 abandoned looks are tolerated before none is started. - Tests: a fake desktop where the window is destroyed mid-wait or never answers, and a test that a look sleeping 5 s is given up on within its limit plus the grace.
Live test: set_value on Notepad replied "set e2 (Edit) to 109 chars (unverified: it reads back as 109 characters, not the 109 sent)" and, on a Chrome input, "reads back as 0 characters" when the text was there. - A read-back is judged by one pure function (uia/readback.rs) on every platform, with line endings normalised (CRLF, a RichEdit lone CR) so a multi-line value no longer "differs" by length alone. - Same: verified. Empty after a non-empty set: a browser or custom control that does not report its value; unverified, not an error. A real value that is exactly what the field held before, or other text: an error that gives lengths and the first differing position, never the text. - An unconfirmed reply says "tried to set eN (Role) to N chars", never "set". Windows adds when a program brought its window to the front while its value was set (Notepad does, with no incu involved). - set_value's tool description says Notepad's ValuePattern change is not counted as an edit, so type_text or paste_text are the real edits.
…age time Live test, Chrome on a long Wikipedia article: links below the 400th element could not be found by selector (resolution shared ui_tree's print cap), and the first look at a fresh page held only the browser's own controls. - A walk made to FIND a control (selector resolution, wait_for, acting on what a selector found) keeps up to tree_max_visited elements (default 4000), not tree_max_elements. Printing needs a cap that fits a reply; finding reports one control. The visit bound already limited time and cost, and is now the only bound. Walk order is unchanged, so a control within the first 400 keeps its id, nth and path: a selector ui_tree printed still means the same control. The cut warning now names the real bound. - A selector that matches nothing in a browser window (chrome, msedge, firefox, brave, vivaldi, opera, chromium) is walked once more after 1.5 s: the first walk is what makes the browser build the page's accessibility tree. Only a miss is retried, once; a match, an ambiguity or another program is not, and the error then says the wait happened. wait_for polls and needs none of it. Smallest safe fix: no new tool, no platform code, and no cost on a hit. - Tests: a control 600 elements past the print cap is found and the print-cap walk cannot see it; selectors printed within the cap resolve to the same control in the larger walk; the retry runs once on a miss, never on a hit, an ambiguity or a non-browser. - cargo fmt over uia/readback.rs, which the previous commit left unformatted.
…g window does not block the others Review findings 2 and 3 on eaf0942. - A look is waited for in slices of 100 ms and the operator's stop is asked each time, so a stop during a look at a hung window returns within a slice, not at the deadline plus 1 s. wait_until asks the stop BEFORE the deadline, so the reply is Stopped and not a timeout. - The cap on abandoned looks (4) is now counted per window (the id of the window looked at, or the kind of look when it is of none), so one hung program no longer makes every later wait on an unrelated window time out at once. A higher process-wide net (24) remains. - Tests: a stop 300 ms into a 60 s look returns at about 300 ms; a stop during a blocked look is Stopped when the deadline is already past; a hung window's looks are refused while another window's run; the process-wide net holds and is above the per-window limit.
…ck and journal names Review findings 1, 4 and 5 on 1530260, 074c6c8 and b56356a. 1. ui_tree printed selectors from its 400 elements while the resolver sees up to 4000, so "role=link;name=Privacy policy" (unique among the 400) matched two once a twin lay at element 1500. ui_tree now walks with the resolving bound, prints the first 400, and makes every printed selector unique, with its nth, among the whole walk (selectors_for_first). Only the printed elements are remembered as shown, so an id still names only what was printed. The comment on resolve_max_elements said otherwise and is corrected. 4. macOS set_value read "no value" (an empty AXValue or a failed read, which text_attribute cannot tell apart) as verified when the value sent was empty. judge_optional: nothing read is never verified. 5. A name that can be a value is never journalled: Role::name_may_be_content (edit, combo box, static text, slider, other) now governs the names in messages (Element::label), the look-alikes of an ambiguity error and Selector::render_redacted. What ui_tree prints on macOS also leaves a combo box's name out, as it does an edit field's, since it changes as it is used. Tests: the Privacy-policy page (twin at 1500) prints nth=0 and resolves to its own element, and only the printed part is remembered; a combo box and text names do not reach the redacted selector, the examples or a label; nothing read back is unverified for an empty and a non-empty value.
…; a stop while picking the window is Stopped Re-review of df9c139. - The wait added a given-up look to the per-window count AFTER settling with it, so the look's thread, returning in between, removed an entry that was not there yet (or another look's), and the later addition was never taken off: four such leaks on the windows-listing key made every element or window wait time out until restart. The wait now counts first, settles second, and takes the count back if the look had in fact returned, so a look's removal always follows the addition it undoes. - A stop during the look that picks the watched window ended the wait as a plain error ("the wait was stopped"); it is now the Stopped outcome, as a stop during the polling is. - Tests: a look made to return exactly after the wait settled, and exactly before, each leave no entry (a test hook in the give-up path); one window's returning look does not take off another's entry; a stop during the window-picking look is Stopped, an unanswered one without a stop is still an error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Computer use, phase 2:
wait_forand stable element selectors. The desktop server now offers 17 tools.Selectors. A selector is
key=valuepairs joined by;, using the keysapp,window/window~,role,name/name~,aid,pathandnth. Examples:role=button;name=Saveandapp=notepad;window~=Save As;role=button;name=Save.ui_treeprints the shortest uniquesel:for each control.click_element,set_value,element_actionandwait_foracceptselectorinstead ofid, and refuse a call that gives both.nthis out of range, the error says which.wait_foris in the observe tier. Its conditions areelement_appears,element_gone,window_appears,window_goneandscreen_changes.screen_changescompares a 128-bit fingerprint of the screen.These selectors are what lets a recorded tool call mean the same control on the next run. That is the base for routines replayed from Watchman's records.
Tests: fmt and clippy (
-D warnings) clean on the workspace and on the aarch64-apple-darwin and x86_64-unknown-linux-musl targets.cargo test --workspacepassed with 0 failures, including 285 incord-desktop tests. The live Windows selector tests (crates/desktop/src/mcp/selector_tests.rs) passed on Windows.Not verified:
AccessibleIdand the cost of one D-Bus call per element.list_windowson Windows, which limits element waits to roughly one look per second.