Skip to content

fix(settle): observe a settled screen through the capture the request admitted - #3553

Merged
thymikee merged 2 commits into
mainfrom
fix/ios-fill-latency
Oct 13, 2026
Merged

thymikee merged 2 commits into
mainfrom
fix/ios-fill-latency

Conversation

@thymikee

@thymikee thymikee commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

Summary

--settle on the generic route (back, scroll) and on a coordinate press/fill polled the screen through the legacy interactor (XCTest runner tree), while snapshot, diff, wait stable and a selector --settle read it through the request-bound runtime (simulator AX bridge) — two projections of one tree. So settled: true, its diff, and the refs issued from it described a tree the caller's next command never resolves against: back --settle returning to an unchanged screen reported 5 additions/6 removals, and the next diff snapshot -i reported that churn as real.

Such a request now admits the same capture-only selector-capture plan once, and the runtime resolves one boundCapture at construction, so a request observes through one producer; no runtime-use declaration is widened. Settling stays best-effort: an owner whose facts refuse the plan (a provider device, whose only snapshot path is the interactor its own commands answer with) keeps its previous observation and discloses settle_observation_capture_unavailable with its typed reason; a throwing facts probe or bind degrades the same way. An optional observation builds no command-facing refusal response, so it spends none of that response's Apple-branch host probes, and a tap without --settle pays no admission. The iOS no-effect corroboration probe polled the legacy capture too — against a keyed baseline, a tap that landed read as a recorded failure. It now answers through the request's own capture AND still forwards the baseline's backend pin: which implementation answers and which backend it presents are options of the one capture ask, not alternatives.

Disclosed, not changed: an Apple session with no tracked app selects the without-active-app plan, which the Apple family refuses by declaration (its own snapshot refuses there too). 22 files, 258 net production lines. Closes #3510.

Validation

Tested at e39e068c3 (base 8255c735c): check:affected --run, typecheck, lint, format, check:layering, check:fallow --base origin/main clean, with the device numbers below. The review round on top (8fe9c58ad) changes tests and doc comments only — no production statement — so this evidence stays attributable; typecheck, lint, format, check:layering and fallow were rerun on that head, and CI is the authority for the rest.

iPhone 18 Pro, iOS 27.0, Settings restarted at the root before each side; 4 rounds of press 'label=General' → back --settle → diff snapshot -i plus press 201 130 --settle, one script at both SHAs, capture identity from per-request phase NDJSON.

observation base 8255c735c head e39e068c3
back --settle diff, unchanged screen +5/-6 (4/4) +0/-0 (4/4)
next diff snapshot -i phantom +6/-5 (4/4) +0/-0 (4/4)
settle captures 3–4× snapshot_capture {"backend":"xctest"} 734–775 ms; 0 bridge acquires 0 runner captures; 3–4× ios.snapshot-source.acquire {"producer":"simulator-ax-bridge"} 210–302 ms
press 201 130 --settle 4 runner captures; next diff phantom +6/-5 4 bridge acquires; next diff +0/-0
back --settle wall 2411–2597 ms 2137–2849 ms

A no---settle tap binds only ['tapPoint'], one admit-and-bind per settle request, both corroboration cells and the settled producer are asserted here; selector --settle, wait stable, snapshot and the private-ax sparse-verdict escalation ride the unchanged captureSnapshotForSession path (snapshot-diff-runtime.test.ts), which only gains a boundCapture field. Unresolved risks: the provider-refusal branch has no hardware coverage; the scroll top|bottom / scroll --until cells named in the ADR amendment are covered by the shared binding cache's own tests, not by a settle test, since these cases do not bind the action leaf; the degrade is disclosed in request diagnostics at warn (the android_ref_snapshot_refresh_failed precedent), so it is visible under --verbose/--debug but not on an ok: true payload; and a settle loop that pins private-ax mid-observation still crosses the runner/bridge boundary inside one observation — that pin belongs to the stable-capture owner, not this route.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 5.17 MB 5.18 MB +2.1 kB
Package (unpacked) 5.17 MB 5.18 MB +2.1 kB
Package (download) 1.55 MB 1.55 MB +593 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 21.9 ms 21.7 ms -0.2 ms
CLI --help 59.5 ms 59.3 ms -0.2 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 22 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread src/daemon/__tests__/generic-settle.test.ts Outdated
Comment thread src/daemon/interaction/internal/interaction-ios-tap-outcome.ts
… admitted

--settle on back/scroll and on a coordinate-targeted press/fill polled the screen
through the legacy interactor while every command the caller ran next (snapshot,
diff, a selector press --settle, wait stable) read it through the request-bound
runtime. On an iOS simulator those are two projections of one screen: the settled
diff, the refs issued from it, and the settled: true verdict described a tree the
next command could never resolve against, and a screen nobody had changed reported
five additions and six removals.

Both routes now admit the same capture-only plan the selector family polls under
and the loop polls that one binding. An owner whose facts refuse the plan keeps
observing through the interactor its own commands answer with, disclosed in the
request's diagnostics.
@thymikee
thymikee force-pushed the fix/ios-fill-latency branch from e39e068 to 8fe9c58 Compare October 13, 2026 00:43

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 8 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/daemon/generic-settle.ts">

<violation number="1" location="src/daemon/generic-settle.ts:127">
P3: This overstates the guarantee: admission or binding failure can leave the settle loop with its prior observation instead of reading through that producer. Qualify the guarantee to cases where admission and binding succeed.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic


/**
* Resolves the observation's capture before the runtime is constructed, so the loop reads the screen
* through the same producer the caller's next command reads it through (#3510). `back` and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This overstates the guarantee: admission or binding failure can leave the settle loop with its prior observation instead of reading through that producer. Qualify the guarantee to cases where admission and binding succeed.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/daemon/generic-settle.ts, line 127:

<comment>This overstates the guarantee: admission or binding failure can leave the settle loop with its prior observation instead of reading through that producer. Qualify the guarantee to cases where admission and binding succeed.</comment>

<file context>
@@ -124,15 +124,11 @@ async function observeSettled(
- * admit-and-bind for one operation, served from the gateway's binding cache rather than a second
- * device acquisition, because the observation runtime is built here and holds no reference to the
- * leaf execution's own binding.
+ * through the same producer the caller's next command reads it through (#3510). `back` and
+ * `scroll <direction>` are the cells that need it: their action use is the movement alone.
+ * `scroll top|bottom` and `scroll --until` already require `captureSnapshot` for their between-pass
</file context>
Suggested change
* through the same producer the caller's next command reads it through (#3510). `back` and
* through the request-bound producer when admission and binding succeed (#3510). `back` and

@thymikee

Copy link
Copy Markdown
Member Author

The PR is ready at 8fe9c58, and I found nothing that needs to change before merge. All 19 checks pass at this head, and there are no conflicts. Nothing else stands between this PR and merge.

Not blocking: the iOS no-effect probe at https://github.com/callstack/agent-device/blob/8fe9c58/src/daemon/interaction/internal/interaction-touch-runtime.ts#L141 gets options.settleObservationCapture, not the capture the runtime already resolved for the request. A selector-resolved tap whose baseline came from the bridge would then be compared against an unkeyed legacy capture, so the identity check fails closed and a tap that landed stays a recorded failure. This already happens on main, so it is not a regression. The rule would be that the probe reads through the same capture the runtime resolved for this request. The doc at https://github.com/callstack/agent-device/blob/8fe9c58/src/daemon/generic-settle.ts#L125 could also add "when the owner admits and binds the capture-only plan". You can take or leave both.

On the open thread, the Cubic P3 about generic-settle.ts:125 still applies (#3553 (comment)). It is comment-only, and the fallback it describes is the intended, disclosed degrade.

On evidence, the live A/B ran at e39e068. That commit is not in my local history, which was rewritten, so I could not confirm that this head differs from it only in tests and comments. The run covered back --settle and coordinate press --settle only. scroll --settle and coordinate fill --settle share the same code paths and have unit coverage, but no device run. The provider-refusal branch has no hardware coverage, as you disclosed. I did not run any tests myself. I checked the iOS probe finding by reading the code and your comment, not on a device.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Oct 13, 2026
@thymikee
thymikee merged commit 9054b86 into main Oct 13, 2026
19 checks passed
@thymikee
thymikee deleted the fix/ios-fill-latency branch October 13, 2026 05:46
@github-actions

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-10-13 05:47 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(settle): a --settle observation on the generic and coordinate routes reads the screen through a different producer than the caller's next command

1 participant