Skip to content

feat: computer use keeps off-screen elements, marks them, and refuses to click them - #72

Merged
blockmandev merged 2 commits into
mainfrom
feat/off-screen-elements
Oct 24, 2026
Merged

blockmandev merged 2 commits into
mainfrom
feat/off-screen-elements

Conversation

@blockmandev

Copy link
Copy Markdown
Member

From #56, decided there: an element wholly outside every display is kept and marked. One example is #52's screen-reader text, drawn 1×1 at y = -16079.

Root cause of the click risk

refuse_point checked only blocked windows, so a pointer action at a point on no display was sent as it was. The OS clamps such a pointer to the nearest screen edge, so it would act on whatever sat there, not on what was aimed at.

Change

  • ui_tree: marks such an element off-screen, using the displays screen_info reports. Its text stays listed.
  • refuse_point: now also refuses a point no display holds. This is one decision, off_every_screen, and it covers click, click_element, scroll and drag, by coordinates and by id. If the displays can't be read, the action is refused, as the blocked-window check already does.

Verified

… to click them (refs #56, #52)

Decided in #56: an element wholly outside every display (a page's
screen-reader text drawn 1x1 at y=-16079, #52) is kept for its text but
marked, so the model does not take it for something to click.

Root cause of the click risk: refuse_point checked only blocked windows,
so a pointer action at a point on no display was sent as it was; the OS
clamps such a pointer to the nearest screen edge, so it acted on whatever
was there instead of what was aimed at.

- uia.rs: Element::beyond_screens, and Tree::render_on(screens), which
  marks such an element "off-screen"; render() stays unmarked for its
  other callers.
- mcp.rs: ui_tree renders with the displays screen_info reports;
  refuse_point now also refuses a point no display holds (one decision,
  off_every_screen), which covers click, click_element, scroll and drag
  by coordinates and by id. The displays failing to read refuses, as the
  blocked-window check already does.

Tests: the tree marks the #52 element and one past the right edge, not an
element on the left-hand display or overlapping a display by a pixel; the
decision on points of a two-display layout with a negative origin; and a
live click at 0,-16079 through the real protocol on this machine's
displays is refused. Both refusal tests fail with the check disabled.
Root cause: the previous commit inserted off_every_screen and screen_rects between blocked_windows' doc comment and the function, so that comment headed the wrong function. Moved back; no code change.
@blockmandev
blockmandev merged commit d8b81be into main Oct 24, 2026
12 checks passed
@blockmandev
blockmandev deleted the feat/off-screen-elements branch October 24, 2026 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant