[thymikee]

Summary

On web targets, coordinate presses on elements inside an iframe missed. agent-browser measures a node inside an iframe against that frame's own viewport, so presses landed on the page around the frame.

The snapshot now places each such node from its frame's unclipped viewport (page position plus border and padding, read only when a page has iframes) and clips it to the part of the frame the page shows. Content scrolled out of its frame, or inside a frame that cannot be measured, gets no rect. Frames under CSS transform: scale() or zoom are not scaled.

Reuse: one intersectRects in kernel/rect replaces copies in selectors and maestro and backs intersectScreenshotRect. Box and style reads share one batched reader.

The test-app WebView fixture gains same-origin, data: and cross-site iframes. Its whitelist adds only about:srcdoc and data:, which the default blocks for iframes.

Validation

Head cdbea80368:

• pnpm check:affected --run --head cdbea80368 passed, and so did eager-closure-budgets.test.ts.
• New tests (nested placement, clipped nested frame, clipping, unmeasurable inset or box) fail with the projection, clip, or origin fix reverted.
• Live, agent-browser 0.39 (evidence): with the page scrolled, presses in a self-scrolled same-origin frame and an out-of-process cross-site frame land on the exact point the frame records, and a cross-site link navigates its frame. agent-browser lists no nested-iframe content, so nested placement is unit-tested only.

Out of scope: web @ref presses fail with "no usable bounds" on main-frame elements too.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
agent-device-website	Ignored	Preview	Oct 28, 2026 3:39pm UTC

[cubic-dev-ai]

All reported issues were addressed across 7 files

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[github-actions]

Size Report

Metric	Base	Current	Diff
Installed (including dependencies)	5.56 MB	5.56 MB	+1.1 kB
Package (unpacked)	5.56 MB	5.56 MB	+1.1 kB
Package (download)	1.69 MB	1.69 MB	+437 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	39.3 ms	39.8 ms	+0.6 ms
CLI --help	81.1 ms	82.4 ms	+1.3 ms

[thymikee]

Thanks for this. At 759a853 the code is not ready: nested iframes that are clipped at the top or left place their children wrongly, and the live evidence does not cover the states this change targets. CI is still queued or in progress, so I can't attribute any result yet. Before merge, the fix below and the live runs must land.

The first fix is the nested-clip bug in agent-browser-snapshot.ts. Each frame should keep its unclipped page rect as the origin for its descendants, and every node should be clipped against the intersection of all ancestor frame viewports. Please add a test with a nested frame clipped at top and left.

The web snapshot and press path changed, but the only live proof is prose about one probe page. That page has no nested frame, no frame scrolled inside the page, and no frame scrolled inside its own document. Those are the states the clip and nesting code exist for, and the unit doubles assert the arithmetic, not the backend's coordinate space. Please attach agent-browser 0.39 output for (a) a nested iframe partly scrolled above its parent frame's top edge, and (b) the main page scrolled so the iframe is offset. For each case, show the press label=... point and the click event the page records.

The test-app README says the cross-site https://example.com/ frame is reachable on web. Under Chromium site isolation that frame is an out-of-process iframe. Does agent-browser 0.39 snapshot its content, and are get box rects for those refs relative to the frame viewport (as the projection assumes) or already page-relative (which would double the offset)? The live evidence covers only the same-origin srcdoc and data: frames. Please run the web target on the fixture page and show the snapshot lines with rects for the Cross-site frame subtree. Then run press on the example.com link and show that the frame navigates, with the tapped point next to the iframe's page rect. If the boxes differ, gate the projection per frame on the measured coordinate space, or drop the cross-site claim.

Not blocking, and fine to take or leave: frameViewportRect adds the frame's origin and border and padding without scale, so a frame under CSS transform: scale(...) or zoom drifts in proportion to distance from the frame origin (derive the scale from frameRect.width over CSS width plus insets, leave transformed frames unplaced, or at least state the limit in the PR body), and clipRect could reuse one exported intersectRect(a: Rect, b: Rect) from capture-kit instead of a local copy.

On the open inline threads, the P1 about the clipped rect feeding descendants (thread) still stands and is the fix above. The two P3 threads also still apply: narrowing originWhitelist and the frameGeometryExecutor double.

[thymikee]

The code fix from the earlier review is in place at 6f5e870, but two live-evidence gaps remain. The nested-clip arithmetic is fixed, and the two small test and originWhitelist notes are fixed. I could not tell which Coverage step failed. The log was not available to me. The changed-line and CRAP gates score the functions this PR touches, and that lane has a known [vitest-pool]: Worker forks emitted error flake (ci.yml:200-203). Please read the failed step's log before treating it as a code problem. I ran no tests and checked the new regression test by hand against its fixtures.

The README claim at https://github.com/callstack/agent-device/blob/6f5e870/examples/test-app/README.md#L28 is still unanswered. It says the cross-site https://example.com/ frame's headings and buttons appear inline on web. Under Chromium site isolation that frame is an out-of-process iframe. placeRect assumes every iframe child's get box is relative to the frame viewport. If agent-browser 0.39 reports these boxes relative to the page, the offset is added twice and presses land past the target. If it omits that content, the README is wrong. Could you run the web target on the test-app fixture page and attach the snapshot lines with rects for the Cross-site frame subtree plus the iframe's own page rect? Then press the example.com link and show that the frame navigates, with the printed tap point inside the link's placed rect. If the boxes are page-relative, gate the projection per frame on the measured coordinate space. If the content is absent, drop the cross-site claim.

The projection in https://github.com/callstack/agent-device/blob/6f5e870/packages/platform-web/src/agent-browser-snapshot.ts#L129 is still unproven in the live states it exists for. The claim that agent-browser 0.39 lists no nested-iframe content is prose without output. No run covers a scrolled page, where the frame's get box origin and the child's frame-relative box must agree on viewport versus document coordinates. If they differ, every press inside a frame on a scrolled page misses, and the unit doubles cannot show that. Please attach 0.39 output for two cases. First, a page with an iframe scrolled so its page rect y differs from the unscrolled value: show the snapshot rect, the press label=... point and the click the frame records. Second, a page with a nested iframe: show the raw snapshot proving nested content is not listed. If it is listed, show the press point and recorded click for a nested button partly scrolled above its parent frame's top edge.

These cubic-dev-ai threads are fixed at this commit and can be resolved: the clipped-origin bug in frameViewport (#4020 (comment)), the originWhitelist entries (#4020 (comment)), and the shared batchExecutor in the rects test (#4020 (comment)). No open thread from other reviewers still applies.

Before merge, the live runs above need to be attached, and the Coverage failure needs to be read and attributed.

[thymikee]

Live evidence at cdbea80368, agent-browser 0.39.0 (Chrome for Testing 156), viewport 1280×577. Raw output comes from a separate agent-browser session on the same page. Press output comes from agent-device.

Coverage failure: not the vitest-pool flake. eager-closure-budgets.test.ts failed because screenshot-geometry.ts started importing kernel/rect for intersectArea, which grew three eager closures. cdbea80368 restores its own arithmetic. The test passes locally (857/857).

Evidence page. Three frames sit below a 900px spacer, each with border: 3px; padding: 5px:

• a same-origin frame whose own document is scrolled to scrollY: 250;
• an outer frame scrolled to scrollY: 185, so its nested frame starts 35px above the outer frame's top edge;
• a cross-site frame served from 127.0.0.1 while the page is on localhost.

Every in-frame button reports its click as a top-level viewport point. Same-origin frames compute it by walking frameElement. For the cross-site frame, the parent adds the frame origin.

CDP /json/list shows 127.0.0.1/cross.html and example.com as separate iframe targets, so both are out-of-process frames.

Raw, page scrollY: 700:

- Iframe "Self-scrolled frame" [ref=e19]       get box: 24,240 416×166
  - button "Self-scrolled button" [ref=e23]    get box: 0,50 130×21
- Iframe "Outer frame" [ref=e20]               get box: 24,422 416×216
  - Iframe "Inner frame" [ref=e24]             get box: 0,-35 312×132
- Iframe "Cross-site local frame" [ref=e21]    get box: 24,654 416×96
    - button "Cross-site button" [ref=e25]     get box: 0,0 117×21
    - link "Cross-site link" [ref=e26]         get box: 121,1 100×18

• Frame boxes are viewport-relative: e19 is at y=940 on the unscrolled page.
• Children are frame-viewport-relative, already net of the frame's own scroll. This holds for the out-of-process frame too: e25 is at 0,0, not at the page-relative 32,662.
• The inner frame's two buttons are not listed, by agent-browser or by agent-device snapshot. Nested content is absent, so nested placement stays unit-tested only.

Presses through agent-device:

scrollY 700:  Tapped label="Self-scrolled button" (97, 309)  -> clicks: Self-scrolled button @viewport(97,309)
scrollY 1000: Tapped label="Cross-site button" (90, 373)     -> clicks: Cross-site button @viewport(90,373)
              Tapped label="Cross-site link" (203, 372)      -> cross-site loads: 1 -> 2; frame now lists heading "Cross-site next page"

Test-app fixture, cross-site example.com frame (raw):

- Iframe "Cross-site frame" [ref=e35]   get box: 24,930 304×154
  - paragraph ×6 (StaticText)
  - link "Learn more" [ref=e47]         get box: 109,855 82×18

• x=109 is (300−82)/2, so the box is frame-relative.
• At y=855, the link sits below the frame's 150px content box. It gets no rect, and press label="Learn more" reports Selector did not match. I found no way to press it through agent-device without scrolling a cross-origin document.
• The README now says the frame's content appears inline, not "headings and buttons".

Known limit: frames under a CSS transform: scale() or zoom aren't scaled, so their content is placed off.

[thymikee]

I found no code problems at cdbea80. The Coverage failure from my earlier review is no longer listed as failing, and the live evidence you posted now matches the placement code.

Smoke Tests is still running, not failed. This PR changes the web snapshot rect path, so that lane exercises it. It should finish green before merge.

The earlier inline threads on frame origin from the unclipped rect, WebViewScreen.tsx and the provider rects test are all fixed at this head, so you can resolve them.

Limits of this review: I checked the arithmetic in your live runs against the placement code but did not rerun them. Nested-iframe placement has unit-test coverage only, because agent-browser 0.39 does not list nested content. Frames under CSS transform: scale() or zoom stay unscaled, which you already state as a known limit. I did not re-check the README claim about iOS and Android inline frames, and I did not run the eager-closure-budgets test locally.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-10-28 17:08 UTC