Skip to content

fix(ios): carry the snapshot viewport across the runner boundary as a declared fact - #2908

Merged
thymikee merged 7 commits into
mainfrom
fix/ios-viewport-evidence
Sep 24, 2026
Merged

thymikee merged 7 commits into
mainfrom
fix/ios-viewport-evidence

Conversation

@thymikee

@thymikee thymikee commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Closes #2891. Related: #2881 (same contracts/fixtures/ mechanism, different payload).

Summary

The runner spelled "viewport unknown" as CGRect.infinite. SnapshotGeometry.isGeometricallyActionable
read that sentinel as "everything is actionable" before it computed any center, while the host's
IosViewportEvidence does not publish the bit at all. The viewport now crosses the boundary as a
SnapshotViewport fact with three cases:

  • reported(box, interfaceOrientation)
  • derived(box)
  • missing(reason)

Only factories can construct a box-bearing case. Only reported carries an orientation, so a derived
or missing viewport cannot anchor a rotation. SnapshotGeometrySpace takes the fact, not a sentinel
CGRect.

With no box, a node whose actionability depends on containment has no hittable on the wire, which is
how the host bridge leaves it. A disabled node or a degenerate frame is still a declared false.
Clipping stays separate from containment. SnapshotPresentationInvariant reads "no box" as no root clip
to violate, not as an unbounded clip.

contracts/fixtures/snapshot-actionability-policy.json pins the shared predicate for shapes that the
fixed 320x240 fold fixture cannot reach. Every row declares both languages. A Swift-only row names its
asymmetry.

Validation

Tested SHA 771322a9e9, rebased on 86d719dc33.

  • pnpm test:ios-snapshot-differential: Swift 16 tests, node 5 tests, all pass.
  • pnpm check:affected --run: exit 0. pnpm check:xctest-selection: pass.
  • Runner build with unit tests (iOS). 86 affected RunnerTests on a new iPhone 17 Pro / iOS 26.2 simulator: Executed 86 tests, with 0 failures.
  • Host: packages/platform-apple/src/runner/__tests__/snapshot-presentation.test.ts sends a payload with hittable absent through presentAppleRunnerSnapshot, and the bit stays absent.
  • Live, same simulator, repo CLI at this head: open com.apple.Preferences, then snapshot --json. The tree backend was healthy: 73 nodes, 65 hittable: true, 8 false, none absent.
  • Earlier head e222012106: the author ran an A/B on the rnav-repro simulator against an origin/main runner. Both runners returned the same 22-node snapshot.

Not checked live: the CLI has no rotate command, and a .missing viewport cannot be forced on a device. Unit tests cover both paths. A box shaped like CGRect.infinite that reaches the TypeScript predicate is still actionable there. The refusal belongs to the producer that parses an Apple frame.

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
Installed (including dependencies) 4.81 MB 4.81 MB +1.0 kB
Package (unpacked) 4.80 MB 4.81 MB +1.0 kB
Package (download) 1.44 MB 1.44 MB +310 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.9 ms 28.3 ms -0.6 ms
CLI --help 83.8 ms 81.8 ms -2.0 ms

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at e222012. The runner now carries the snapshot viewport across the boundary as a declared fact, which is what #2891 asked for, and the approach is sound.

Does the design go far enough, though? The smaller fix looks like it's on the output side: making RawAXNode.hittable optional would make "unknown" impossible to encode as a Bool in the first place, mirroring the absent bit the host already models (projection.ts:160, noise-structural.ts). The fail-closed branch then becomes "omit", which would also resolve the f1 concern below and let most of the explanatory prose in SnapshotGeometry, rect.ts, and the fixture description go away. Before that could land, the Swift RawAXNode.hittable would need to become optional with a Codable omission, and the host runner decoder would need to treat a missing hittable the same way it treats the bridge's undefined today. Worth doing here, or a fast-follow?

The Smoke Tests failure looks unrelated to this change: the failing occupancy test goes through makeSnapshotTraversalContext, then runMainThreadWork("snapshot_viewport", ≤1s), then safeSnapshotViewport/snapshotViewport (apple/runner/AgentDeviceRunnerUITests/RunnerTests+SnapshotAcquisition.swift#L145 and #L236), and the log shows MAIN_THREAD_WORK_ABANDONED operation=snapshot_viewport slice=1.0 before the stubbed tree XPC was even reached. The only change inside that hop is an O(1) isPositiveFinite check replacing isNull/isEmpty, which shouldn't add wall time, so this reads like the cold-host timing class #2847 already covers rather than something this PR caused. Please re-run the job before treating it as a real blocker.

I could not reproduce the author's live simulator A/B (same 22-node snapshot on main and head) or the listed mutations myself, since this pass had no device. The one live route I could reach only covers the reported-viewport tree tier; the query-sweep and private-AX tiers, where a .missing or derived viewport would exercise the actual behavior change, weren't exercised live, and forcing a .missing viewport on device looks hard to arrange. The downstream effect on #2638 suppression and interactiveNodeCount is something I traced through the host code, not something a test fed a runner payload with a missing viewport through presentIosRunnerSnapshot to confirm. I also couldn't tell from the CI log alone whether this occupancy test already flakes on main, since gh run list returned no recent Smoke Tests history for main to compare against.

Not blocking: the ADR 0004 line at docs/adr/0004-ios-snapshot-backend-strategy.md#L313 should name resolveIosViewportEvidenceFromRoots (packages/platform-apple/src/runner/snapshot-presentation.ts) rather than viewportFromRoot, which only exists in the bridge decoder and isn't what the runner route calls; the RectBox fixture in apple/snapshot-presentation/Tests/AgentDeviceSnapshotPresentationTests/ActionabilityPolicyTests.swift#L21 duplicates WindowCoordinateSpaceFixture.Frame from CoordinateSpaceTests.swift and could share one decoder per #2891; the unpacked npm growth (+7.0 kB) traces mostly to paragraph-long comments starting at packages/kernel/src/rect.ts#L3 that could be trimmed to the invariant plus an ADR pointer; isUsableNavigationFrame in apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Navigation.swift#L124 could call the now-public SnapshotGeometry.isPositiveFinite instead of re-encoding the same rule; and the missing-row assertion in scripts/ios-snapshot-differential.test.ts#L225 checks the fixture against itself rather than a TypeScript predicate, so it should either drop that self-check or be marked Swift-only with asymmetry — all of these can be taken or left.

Before merge, please settle how an unknown viewport should travel on the runner wire: an absent hittable matching the host's undefined handling and #2638, or a documented declared false. Then re-run Smoke Tests so the occupancy test's cold-host timeout clears on a fresh run.

@thymikee thymikee left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at e222012. The declared viewport fact is the right direction. The remaining issue is that the migration stops just short of removing sentinel-box reachability in two adjacent places: the normalization pass still uses .null as the stand-in for no box, and derived viewport still does not encode the rotation rule it claims to have.

// No viewport box means no app frame to be quarter-turned relative to either. `.null` is the box
// `SnapshotGeometry.isPositiveFinite` refuses, so this pass turns nothing — the outcome
// `CGRect.infinite` produced before the fact carried the absence.
let appFrame = viewport.rect ?? .null

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The viewport fact is decoded straight back into a sentinel rectangle here. That works because isPositiveFinite happens to refuse .null, not because the type says so. The public space(appFrame:) and .deviceNative(appFrame:) APIs still accept any CGRect, including .infinite, and the updated coordinate-space test also switched to passing .null through that public surface, so the sentinel spelling moved rather than going away. Could space/isQuarterTurned take the viewport fact or a CGRect? so no caller has to name a box for no box?

/// A box the capture inferred for itself out of its own root element instead of a screen read. It
/// clips and contains like a reported box, and it never anchors a rotation: the tier that produces
/// it reports no interface orientation beside it (#2612).
case derived(Box)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The derived case documents that it never anchors a rotation, but that invariant lives only in prose. interfaceOrientation is a separate SnapshotAcquisition field, and SnapshotGeometrySpace.space chooses .deviceNative from orientation plus box geometry without seeing whether the viewport is reported or derived, so derived + landscape is still constructible. Folding the orientation into the viewport fact, or otherwise making derived unable to produce a rotation-capable space, would remove the same unpaired state this PR is otherwise removing.

… declared fact

The runner spelled "viewport unknown" as CGRect.infinite, which
SnapshotGeometry.isGeometricallyActionable read as "everything is
actionable" before computing any center, while the host-AX path models
the same state as IosViewportEvidence and declines to publish the bit.
One state, two encodings, opposite directions, across a language
boundary.

SnapshotViewport now carries the three cases the host already uses --
reported, derived, and missing { reason } -- and refuses an unusable box
at declaration, so no caller has to re-check what it was handed. The
unknown-viewport policy is stated once, at the Swift decision site, and
fails CLOSED: no box means no supportable claim about where a tap lands.
Clipping stays a separate question from containment; a capture with no
box has no clip and no root clip for the cumulative invariant to
violate, which is an absent answer rather than an unbounded one.
The fixed 320x240 fold fixture cannot reach an edge-of-viewport center, a
node rect neither language's null/empty check refuses, or an unknown
viewport, so the predicate's two implementations disagreed on two of
them: CGRect.infinite is built of finite Doubles and its center is (0,0),
and a negative width passes an empty check while its center lands inside
-- both actionable in Swift and not actionable in TypeScript. The private
AX bridge's JSON frame parser can hand over either.

contracts/fixtures/snapshot-actionability-policy.json is replayed by
ActionabilityPolicyTests (Swift) and by scripts/ios-snapshot-differential
.test.ts (TypeScript), which also asserts the host's own instrument:
resolveViewportEvidence refuses a regular presentation without a positive
finite viewport, which is why the TypeScript predicate has no unknown
viewport case. Rows come in pairs sharing one center so a `false` that
the guard, rather than containment, decides cannot go quiet.
…dicate

`SnapshotGeometrySpace.isPlottable` and `SnapshotGeometry.isPositiveFinite`
were the same five comparisons, written twice in one package, which is how
#2891 started: one question with two encodings that can drift apart.
`.null`, an inverted box, and an infinite box are refused identically by
both, so the coordinate-space table needs no new rows.
`SnapshotViewport.reported(box:)` normalised an unusable box to
`.missing(reason: .invalid)`, and the commit said no caller had to
re-check what it was handed — while `.reported(rect:)` stayed a public
case that accepted `CGRect.infinite` on the way past that claim. The
associated value is now a box with an internal initialiser, so outside
this package a viewport fact is only reachable through the two
factories, and the case labels cannot be named around them.
MSG2
The table replayed `{"infinite": true}` as `CGRect.infinite` in Swift and as a
rect with infinite components in TypeScript, so its infinity row never put the
same number in front of both sides. `CGRect.infinite` is made of finite Doubles:
TypeScript would call those components actionable, and only the platform that can
name the value can refuse it. The row is now Swift-only and carries `asymmetry`
naming that, every row declares both `swift` and `typescript`, and a shared
non-finite row replaces it with numbers both sides actually see.

Also states three claims the code had not been checked against: a published
`hittable: false` is retained rather than filtered out, the runner's viewport fact
decides only runner bits because the host re-derives its own from the capture's
root node, and `viewportFromRoot` — not only `resolveViewportEvidence` — is what
makes the TypeScript `viewport` argument total.
…ed viewport anchors a rotation

A capture with no viewport box can still decide a disabled or degenerate
node, but not containment, so the runner now omits hittable instead of
declaring false, matching the host bridge's undefined bit. The interface
orientation moves into SnapshotViewport.reported, so a derived or missing
viewport cannot anchor a rotation, and SnapshotGeometrySpace takes the
viewport fact rather than a sentinel CGRect.
…l lane

contracts/fixtures/snapshot-actionability-policy.json is read by both
sides of scripts/ios-snapshot-differential, and packages/kernel/src/rect.ts
holds the TypeScript predicate that table replays, so a change to either
must select that lane rather than rely on the generic golden-table rule
reaching only the unit and Swift builds.
@thymikee
thymikee force-pushed the fix/ios-viewport-evidence branch from e222012 to 771322a Compare September 24, 2026 16:22
@thymikee

Copy link
Copy Markdown
Member Author

Addressed the 14:54Z comment and the 15:22Z review. The branch is rebased onto main (86d719dc33); the only overlap was ADR 0004, and it applied cleanly. New commit: f38456de87. chore(gates) stays last.

Blocking

  • How an unknown viewport travels on the wire: hittable is now absent. I made this change here, not in a follow-up. RawAXNode.hittable and PresentedNode.hittable are now Bool?. The synthesized Codable omits the key when the value is nil. isGeometricallyActionable returns nil only when containment is the one open question and there is no viewport box. A disabled node or a degenerate frame is still a declared false. The fold and the invariant read == true. The capture plan's full-screen check reads != true. The host runner decoder needed no change, because RawSnapshotNode.hittable is already optional. A new host test sends a payload with no hittable through presentAppleRunnerSnapshot, with and without interactiveOnly, and checks that the host does not add a value.
  • Inline, SnapshotCoordinateSpace.swift:170 (the sentinel box): space(...) and normalized(...) now take the SnapshotViewport fact, not a CGRect. .deviceNative now carries SnapshotViewport.Box, and only the factories can construct a Box. The .null stand-in is removed, and so is the test that passed .null in.
  • Inline, SnapshotModels.swift:178 (derived + landscape): the orientation is now part of the fact: .reported(Box, interfaceOrientation:). .derived has no orientation, so it cannot give a rotation-capable space. SnapshotAcquisition.interfaceOrientation and SnapshotTraversalContext.interfaceOrientation are removed. The tree tier and the private-AX tier read the orientation in the same viewport hop. The query sweep reads it as unknown, which is what it did before. New tests: space gives .appOrientation for .missing and for .derived in landscape, and normalized without a box leaves a child's hittable as nil and a disabled child as false.

Non-blocking (all taken)

  • ADR 0004 now names resolveIosViewportEvidenceFromRoots. I rewrote the paragraph for the absent-bit policy and made it shorter.
  • ActionabilityPolicyTests and CoordinateSpaceTests now share one FixtureRect decoder and one contractsFixtureURL helper.
  • I cut the long comments in rect.ts, SnapshotGeometry, SnapshotModels, the invariant, the conformance tool and the fixture description to the invariant plus a pointer.
  • isUsableNavigationFrame is removed. Navigation now calls SnapshotGeometry.isPositiveFinite.
  • The self-check on the missing rows is removed. Those rows are now Swift-only, with an asymmetry note, and expect hittable: null. I added two Swift-only rows that are still false without a box: a disabled node and a zero-area node. The TypeScript check now requires its rows to cover reported and derived.

Smoke Tests: testAbandonedTreeCaptureSkipsQuerySweepAndHonorsWarmupExemption, with MAIN_THREAD_WORK_ABANDONED operation=snapshot_viewport before the stubbed tree XPC. This matches the #2847 row in #2491's table. #2847 is on this branch's base, so its warm-up read does not always prevent the failure on a cold host. This PR changes only the return type of the hop the test times, not the XCUI queries in it. The push starts a new run.

Validation at 771322a9e9

  • pnpm test:ios-snapshot-differential: Swift 16 tests, node 5 tests, all pass.
  • Runner build with unit tests (iOS), then 86 affected RunnerTests on a new iPhone 17 Pro / iOS 26.2 simulator: Executed 86 tests, with 0 failures. The list covered snapshot presentation, private AX, AX fallback, invariant, fold, hittability, capture plan, occupancy, geometry, conformance, navigation and selector policy.
  • Live check, same simulator, repo CLI: open com.apple.Preferences, then snapshot --json. The tree backend was healthy: 73 nodes, 65 hittable: true, 8 false, none absent. I could not check a live rotation, because the CLI has no rotate command. I also could not force a .missing viewport live. Unit tests cover both paths.
  • pnpm check:affected --run: exit 0. pnpm check:xctest-selection passes.

@thymikee

Copy link
Copy Markdown
Member Author

Reviewed at 771322a. The PR is ready to merge as far as the code goes.

The .missing-viewport nil-source gap from the earlier pass (e222012) is fixed: SnapshotVisibilityFold.swift now falls back to the geometric predicate when the source bit is undecided, so a clipped-to-empty frame still publishes false instead of leaving hittable absent.

Not blocking: could you take a look, whenever convenient, at whether the fold should key off node.hittable == false directly instead of flatMap before falling back to geometry (apple/snapshot-presentation/Sources/AgentDeviceSnapshotPresentation/SnapshotVisibilityFold.swift#L208), and whether the wire-shape test in apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+SnapshotPresentationTests.swift#L6 should also pin the encoded bytes for the missing-viewport/absent case so a future custom encoder emitting "hittable":null doesn't slip past both sides — neither is required for this PR.

No device or test run was performed in this pass, so the author's 86 RunnerTests, the live Preferences snapshot at 771322a, and the differential-lane pass are unverified here. A .missing viewport and a live rotation weren't exercised on device, since the CLI has no rotate command; unit tests only cover the absent-bit and derived/missing-no-rotation paths. Reachability of the non-root Window carrier that an ancestor scroll anchor clips to empty wasn't confirmed on a real tree.

Smoke Tests and Coverage haven't finished on 771322a, so nothing is failing yet. The diff touches the iOS runner snapshot_viewport hop in RunnerTests+SnapshotAcquisition.swift, which reads orientation inside safeSnapshotViewport — the same hop where the earlier MAIN_THREAD_WORK_ABANDONED failure occurred, so if Smoke Tests fail there again it needs its own look rather than being waved off as unrelated to this change.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Sep 24, 2026
@thymikee
thymikee merged commit 5dee3be into main Sep 24, 2026
19 checks passed
@thymikee
thymikee deleted the fix/ios-viewport-evidence branch September 24, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(ios): an unknown viewport is CGRect.infinite on the runner and a typed fact on the host, resolved in opposite directions

1 participant