Repository navigation
feat: computer use keeps off-screen elements, marks them, and refuses to click them - #72
Merged
Merged
Conversation
… to click them (refs #56, #52) Decided in #56: an element wholly outside every display (a page's screen-reader text drawn 1x1 at y=-16079, #52) is kept for its text but marked, so the model does not take it for something to click. Root cause of the click risk: refuse_point checked only blocked windows, so a pointer action at a point on no display was sent as it was; the OS clamps such a pointer to the nearest screen edge, so it acted on whatever was there instead of what was aimed at. - uia.rs: Element::beyond_screens, and Tree::render_on(screens), which marks such an element "off-screen"; render() stays unmarked for its other callers. - mcp.rs: ui_tree renders with the displays screen_info reports; refuse_point now also refuses a point no display holds (one decision, off_every_screen), which covers click, click_element, scroll and drag by coordinates and by id. The displays failing to read refuses, as the blocked-window check already does. Tests: the tree marks the #52 element and one past the right edge, not an element on the left-hand display or overlapping a display by a pixel; the decision on points of a two-display layout with a negative origin; and a live click at 0,-16079 through the real protocol on this machine's displays is refused. Both refusal tests fail with the check disabled.
Root cause: the previous commit inserted off_every_screen and screen_rects between blocked_windows' doc comment and the function, so that comment headed the wrong function. Moved back; no code change.
3 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From #56, decided there: an element wholly outside every display is kept and marked. One example is #52's screen-reader text, drawn 1×1 at y = -16079.
Root cause of the click risk
refuse_pointchecked only blocked windows, so a pointer action at a point on no display was sent as it was. The OS clamps such a pointer to the nearest screen edge, so it would act on whatever sat there, not on what was aimed at.Change
ui_tree: marks such an elementoff-screen, using the displaysscreen_inforeports. Its text stays listed.refuse_point: now also refuses a point no display holds. This is one decision,off_every_screen, and it coversclick,click_element,scrollanddrag, by coordinates and by id. If the displays can't be read, the action is refused, as the blocked-window check already does.Verified
0,-16079is refused, through the real protocol on this machine's displays.-D warningsclean; 2347 lib + 59 bin tests passed.